Self-Speculation for Faster Reasoning Models

# Self-Speculation for Faster Reasoning Models
Source: [https://arxiv.org/html/2608.20359](https://arxiv.org/html/2608.20359)
Ravisri Valluri University of California, Los Angeles ravisrivk@gmail\.com &Tung Nguyen University of California, Los Angeles &Aditya Grover University of California, Los Angeles

###### Abstract

Large language models \(LLMs\) are deployed for increasingly complex tasks involving planning and multi\-step decision making, but high\-quality performance on these tasks often requires generating long reasoning traces\. This is a poor fit for latency\-sensitive and interactive applications like voice assistants or coding agents, where generation latency can strongly affect user experience\. Existing acceleration methods typically focus on token\-level generation, without utilizing the structure of reasoning workflows\. We introduce SSR: Self\-Speculation for Reasoning Models, a training\-free self\-speculative decoding method that leverages the chain\-of\-thought \(CoT\) as a source of speculation\. SSR uses the partial\-CoT answer distribution as the drafter and the full\-CoT distribution as the verifier, deriving both from the same model at different reasoning budgets\. This builds on the observation that later partial\-CoT responses often exhibit greater semantic and lexical overlap with the full\-budget response\. Due to this overlap, SSR can accept long draft prefixes at once, leading to large speedups on structured and long\-form generation tasks\. To further exploit draft\-response overlap beyond the contiguous prefix accepted by standard speculative decoding, SSR also incorporates suffix decoding, using the draft to seed a suffix cache and recover useful spans beyond the accepted prefix, further reducing latency on tasks with high lexical overlap between the draft and the final response\. We evaluate SSR on multiple structured and long\-form generation tasks where it is most useful, and demonstrate a relative improvement of up to 24\.1% on total generation latency for popular open\-source models such as Qwen3\.5 and Gemma\-4\. Code is available at[https://github\.com/Ravi\-VK/SSR/](https://github.com/Ravi-VK/SSR/)\.

## 1Introduction

Reasoning language models are pushing LLMs beyond short\-form assistance and into tasks that require sustained problem solving, including planning, coding, and decision making\. These models achieve stronger performance by generating long reasoning traces that can span over thousands of tokens before producing a final response\. While this improves answer quality, it also increases end\-to\-end generation latency, making them difficult to use in applications that require fast responses\. This is especially limiting in interactive settings such as chatbots, voice\-based assistants, and coding agents, where a long delay can degrade the user experience\. In such settings, it is often worth spending additional compute if doing so reduces end\-to\-end latency\.

Speculative decoding is a natural fit for this setting\. In standard speculative decoding, a draft model proposes multiple future tokens, which are then verified in parallel by the target model\(Leviathanet al\.,[2023](https://arxiv.org/html/2608.20359#bib.bib7); Chenet al\.,[2023](https://arxiv.org/html/2608.20359#bib.bib8)\)\. When the draft is accurate, the verifier can accept several tokens at once and reduce the number of sequential decoding steps\. The main challenge is constructing a draft that is both cheap to produce and accurate enough that the target accepts most of its tokens\.

Prior work on speculative decoding has approached draft model selection in different ways\. A fully independent draft model is the natural baseline, but it introduces substantial overhead, since the draft model must be trained separately, kept in memory alongside the target model, and carefully optimized to match the target model’s output distribution\. One line of work, including Medusa\(Caiet al\.,[2024](https://arxiv.org/html/2608.20359#bib.bib9)\)and EAGLE\(Liet al\.,[2024](https://arxiv.org/html/2608.20359#bib.bib10)\), reduces this overhead by training lightweight draft heads or small modules that reuse parts of the target model, but still requires training and architectural modifications\. Another line of work avoids auxiliary parameters entirely by drafting from the target model itself, using early layers\(Elhoushiet al\.,[2024](https://arxiv.org/html/2608.20359#bib.bib4); Liuet al\.,[2024](https://arxiv.org/html/2608.20359#bib.bib6)\), sparse attention\(Yueet al\.,[2026](https://arxiv.org/html/2608.20359#bib.bib2); Zhaoet al\.,[2025](https://arxiv.org/html/2608.20359#bib.bib3)\), quantization\(Tiwariet al\.,[2025](https://arxiv.org/html/2608.20359#bib.bib1)\), or other approximations of the target model’s internal computation\. Quantization\- and sparse\-attention\-based variants of self\-speculative decoding can often be entirely training\-free\. At the other end of the spectrum, model\-free methods\(Saxena,[2023](https://arxiv.org/html/2608.20359#bib.bib12); Oliaroet al\.,[2024](https://arxiv.org/html/2608.20359#bib.bib13); Huet al\.,[2025](https://arxiv.org/html/2608.20359#bib.bib11)\)copy text from the prompt or prior generations using n\-gram lookup or suffix structures\. These methods eliminate draft\-model calls entirely, but cannot generate new content or adapt to the model’s current reasoning state\.

While all these approaches can be effective at reducing decoding latency, they do not fully take advantage of the structure of reasoning language models\. Reasoning language models expose an additional source of signal for speculation: the chain\-of\-thought itself\. Work on test\-time scaling has shown that CoT length can be controlled through budget forcing and reflection\(Muennighoffet al\.,[2025](https://arxiv.org/html/2608.20359#bib.bib14)\), and that increasing the reasoning budget generally improves answer quality, at least up to a point\. This suggests that partial CoTs are not completely arbitrary incomplete states, and that they can induce meaningful intermediate answer distributions\. Answers sampled early in the CoT might be incomplete or inconsistent, but as reasoning progresses they improve in quality and become increasingly similar in content and surface form to the final answer\. This property offers a natural basis for drafting, and prior work has not used it so far\.

![Refer to caption](https://arxiv.org/html/2608.20359v1/self-spec-compact.png)Figure 1:Illustration of concurrent generation in self\-speculative decoding\. A lower\-budget instance drafts answer tokens while reasoning continues in parallel\. Then the verifier accepts the longest draft prefix supported by the higher\-budget distribution; after the first rejected token, generation continues under the verifier\.We take advantage of this property to propose SSR: Self\-Speculation for Reasoning Models, a self\-speculative decoding method that uses the answer distribution induced by a partial CoT as the draft distribution\. The same language model is used at two different reasoning budgets — an intermediate CoT prefix produces draft answer tokens, while the completed CoT acts as the verifier\. Draft generation and continued reasoning proceed concurrently, so the drafting overhead is hidden behind ongoing CoT generation \(Figure[1](https://arxiv.org/html/2608.20359#S1.F1)\)\. Because the draft distribution converges toward the final answer distribution as the CoT grows, SSR requires no auxiliary parameters or target\-model\-specific modifications\. Under exact verification, output quality is unchanged\. When drafts are accepted, latency decreases, and when rejected, the model falls back to standard decoding with minor overhead, making SSR a drop\-in replacement for standard decoding in any reasoning model\.

We perform left\-to\-right draft verification to accept the longest matching prefix between the draft and the final answer\. In structured tasks, the model commits to broad structure early in the trace, so a draft sampled mid\-CoT often shares substantial overlap with the final response\. For long\-form outputs, however, two responses might disagree on minor details like variable names or phrasing choices early on, while still agreeing on long spans later in the text\. Discarding the draft at the first rejected token wastes this overlap\. We therefore follow this prefix verification stage with suffix decoding, building a suffix cache from the draft and using it to propose spans that may still match beyond the first rejection, recovering useful text that standard verification would typically discard\.

If the draft acceptance rate increases monotonically with CoT length, the optimal sampling point would leave exactly enough CoT tokens remaining to mask the drafting cost in order to begin sampling the draft as late as possible\. In practice, this ideal point is not known in advance, since it depends on the total CoT length\. This motivates an iterative variant of SSR that samples drafts at multiple points during CoT generation\. Earlier drafts are not wasted — we apply the same verification procedure to bootstrap later drafts from earlier ones, reducing the overhead of multi\-point sampling and increasing the likelihood that at least one draft overlaps substantially with the final answer\.

In summary, our contributions are:

- •We propose SSR, a training\-free self\-speculative decoding method for reasoning language models that uses partial\-CoT answer distributions as drafts, requiring no auxiliary parameters or target\-model\-specific modifications\.
- •We extend standard draft verification with suffix decoding to recover useful draft spans beyond the first mismatch, improving acceptance rates for long\-form and structured outputs\.
- •We introduce an iterative variant that samples drafts at multiple points during CoT generation, using earlier drafts to bootstrap later ones and reduce multi\-point sampling overhead\.
- •We provide an implementation in vLLM that integrates SSR as a drop\-in addition to existing serving configurations, requiring no changes to model weights or inference infrastructure\.
- •We evaluate on coding and long\-form generation tasks including HumanEval, ClassEval, and LongProc, achieving latency reductions of up to 24\.1% while preserving output quality\.

## 2Related Work

### 2\.1Speculative decoding

The key driver of generation latency for autoregressive language models is the decode step, which is repeated across a large number of tokens\. Speculative decoding reduces this cost by using a cheaper draft model to propose multiple tokens in advance, which are then verified in parallel by the target model\. In its exact formulation, it uses rejection sampling to ensure that the final sampling distribution remains identical to that of the target model\. If multiple draft tokens are accepted, this can significantly reduce the number of expensive decoding steps required of the target model\. This line of work\(Leviathanet al\.,[2023](https://arxiv.org/html/2608.20359#bib.bib7); Chenet al\.,[2023](https://arxiv.org/html/2608.20359#bib.bib8)\)was developed as a lossless acceleration, but has since expanded to include lossy variants, including model\-free speculative decoding, which does not necessarily associate a sampling probability with draft tokens, and principled relaxations of verification\(Zhonget al\.,[2025](https://arxiv.org/html/2608.20359#bib.bib16)\)that can achieve even higher acceleration gains\.

#### Draft model construction\.

The speedup speculative decoding provides is a function of both drafting overhead and acceptance rate\. For speculative decoding to accelerate generation, the draft model must be cheap to run and close enough to the target model to maintain a high acceptance rate\. In the open\-source community, it is common to use a smaller, distilled variant of a flagship model as the draft model, especially when the target reaches tens or hundreds of billions of parameters\. But when resources are constrained, and because the draft model also occupies significant memory, many popular approaches share parameters and computation between draft and target to minimize this overhead\.Liet al\.\([2024](https://arxiv.org/html/2608.20359#bib.bib10)\)predicts penultimate\-layer hidden states for future positions using lightweight feature predictors, then uses the last layer of the target model to predict the corresponding tokens in parallel as the draft proposal\.Caiet al\.\([2024](https://arxiv.org/html/2608.20359#bib.bib9)\)adds additional language modeling heads to decode future positions from the same hidden state\. These approaches substantially reduce the footprint of drafting, but still require target\-model\-specific training and architectural modifications\.

#### Self\-speculative decoding\.

Another line of work eliminates the draft model entirely, using only the parameters and activations of the target model to propose tokens\. Self\-speculative decoding typically relies on a lossy approximation of the target model for draft proposals — via early exiting, sparsity, or quantization — and reruns full inference for verification\.Elhoushiet al\.\([2024](https://arxiv.org/html/2608.20359#bib.bib4)\)andLiuet al\.\([2024](https://arxiv.org/html/2608.20359#bib.bib6)\)enable self\-speculative decoding through early exiting, with the former training intermediate layers to draft tokens for full\-model verification and the latter adding a Thompson\-sampling\-based controller to decide when to exit and speculate\. While these layer\-skipping approaches typically require training, quantization and sparsity\-based approaches can be entirely training\-free\.Tiwariet al\.\([2025](https://arxiv.org/html/2608.20359#bib.bib1)\)uses a hierarchical quantized KV cache to create a cheaper self\-speculative draft pass while preserving the full\-precision cache for verification\.Yueet al\.\([2026](https://arxiv.org/html/2608.20359#bib.bib2)\)co\-designs sparse attention with self\-speculative decoding so that the draft pass reads only a subset of the KV cache before full verification\.Zhaoet al\.\([2025](https://arxiv.org/html/2608.20359#bib.bib3)\)also uses sparse attention, proposing PillarAttn, which uses attention scores to dynamically identify a small subset of important tokens for long\-reasoning workflows\. None of these approaches exploits the structure of reasoning models directly: even PillarAttn, which targets long CoT generation, treats the chain of thought as a sequence to attend over sparsely rather than exploiting how reasoning converges to a final answer\.

#### Model\-free methods\.

Self\-speculative decoding approaches still incur overhead from a relatively cheaper but nonetheless non\-trivial drafting pass\. At the extreme end of the spectrum, model\-free approaches can propose spans of text with negligible overhead\. N\-gram speculative decoding\(Saxena,[2023](https://arxiv.org/html/2608.20359#bib.bib12)\)proposes fixed\-length n\-grams drawn from the prompt and previously generated tokens, working well in domains with high lexical overlap\. Suffix decoding and related work\(Oliaroet al\.,[2024](https://arxiv.org/html/2608.20359#bib.bib13); Huet al\.,[2025](https://arxiv.org/html/2608.20359#bib.bib11)\)extend this by building a suffix tree over this text, enabling longer span proposals based on the currently matched suffix of the generated tokens\. These methods are fast but inherently passive — they can only repeat text that already exists somewhere, and cannot adapt meaningfully to the model’s current reasoning state\.

### 2\.2Inference\-time compute and budgeted reasoning

Recent work has shown a clear tradeoff between inference\-time compute and answer quality in reasoning language models, driving methods to explicitly control reasoning length at test time\. Budget forcing controls the number of thinking tokens by forcing early termination or extending reasoning — for example by appending continuation tokens — with answer quality generally improving as the reasoning budget grows\(Muennighoffet al\.,[2025](https://arxiv.org/html/2608.20359#bib.bib14)\)\. A parallel line of work studies when model outputs stabilize during a reasoning trace, using stability signals to stop early and reduce redundant computation\(Liu and Wang,[2025](https://arxiv.org/html/2608.20359#bib.bib19)\)\. More broadly, work on test\-time compute scaling examines how to allocate additional inference compute to improve answer quality\(Snellet al\.,[2024](https://arxiv.org/html/2608.20359#bib.bib17); Wellecket al\.,[2024](https://arxiv.org/html/2608.20359#bib.bib18)\)\. These methods primarily treat intermediate\-budget outputs as cheaper final predictions or early\-stop endpoints, rather than as computation that can be reused for subsequent generation\.

## 3Preliminaries

### 3\.1Autoregressive reasoning language models

We focus on applications where a language model generates a reasoning trace before producing a final response\. The chain\-of\-thought is sampled first, token by token, until an end\-of\-thinking token is reached\. The final response is then conditioned on both the user query and the reasoning trace, and is sampled token by token until an end\-of\-sequence token is reached\. In budget forcing, the language model is used to sample a response after a specified number of CoT tokens; sampling a response after partial CoT typically requires appending a special end\-of\-thinking token and, in some setups, an interrupt string\.

We usepθ​\(o∣q\)p\_\{\\theta\}\(o\\mid q\)to denote the probability of sampling output sequenceoofor user inputqqfrom a language model with parametersθ\\theta\. For autoregressive models,pθ​\(o∣q\)=∏t=1lpθ​\(ot∣x,o<t\)p\_\{\\theta\}\(o\\mid q\)=\\prod\_\{t=1\}^\{l\}p\_\{\\theta\}\(o\_\{t\}\\mid x,o\_\{<t\}\), whereo=\(o1,…,ol\)o=\(o\_\{1\},\\ldots,o\_\{l\}\)\. We denote the reasoning trace \(chain\-of\-thought\) byrrand the final answer sequence byaa: for reasoning language models, the output sequence is the concatenation of the reasoning trace and the final answer\. If a reasoning budgetbbis specified, we writepθ​\(a∣q,b\)p\_\{\\theta\}\(a\\mid q,b\)as shorthand forpθ​\(a∣q,r≤b\)p\_\{\\theta\}\(a\\mid q,r\_\{\\leq b\}\), wherer≤br\_\{\\leq b\}denotes the firstbbtokens of the reasoning trace\.

### 3\.2Speculative decoding

In speculative decoding, a draft model with distributionddproposes tokens, which are then verified by the target model with distributionpp\. The draft model proposes a sequence ofnntokensa^1,…,a^n\\hat\{a\}\_\{1\},\\ldots,\\hat\{a\}\_\{n\}before verification begins\. Tokens are accepted sequentially until the first rejection; we refer to the accepted tokens as the accepted prefix\. A draft tokena^t\\hat\{a\}\_\{t\}is accepted with probability

αt=min⁡\(1,p​\(a^t∣q,r,a<t\)d​\(a^t∣q,r,a<t\)\),\\alpha\_\{t\}=\\min\\left\(1,\\frac\{p\(\\hat\{a\}\_\{t\}\\mid q,r,a\_\{<t\}\)\}\{d\(\\hat\{a\}\_\{t\}\\mid q,r,a\_\{<t\}\)\}\\right\),
which ensures that accepted tokens are distributed according topp\. Upon rejection, a token is sampled from the residual distribution

at∼max⁡\(p−d,0\)∑vmax⁡\(p​\(v\)−d​\(v\),0\),a\_\{t\}\\sim\\frac\{\\max\(p\-d,0\)\}\{\\sum\_\{v\}\\max\(p\(v\)\-d\(v\),0\)\},
where the residual is the positive part ofp−dp\-d, renormalized to a valid distribution\. Together, acceptance and residual sampling guarantee that the final output distribution is identical to that of the target modelpp, regardless of the draft model quality\.

The draft model proposes a sequence of tokens before verification, so if the acceptance rate is high, many tokens are accepted in a single verifier forward pass, yielding a net speedup\. Exact speculative decoding requires maintaining bothppandddat each position to compute the residual upon rejection; simply resampling fromppupon rejection yields an approximate variant that is also widely used in practice\(Liet al\.,[2025](https://arxiv.org/html/2608.20359#bib.bib20)\)\.

### 3\.3Suffix decoding

Suffix decoding\(Oliaroet al\.,[2024](https://arxiv.org/html/2608.20359#bib.bib13)\)extends speculative decoding to the model\-free setting, where no draft distributionddis available\. Rather than sampling draft tokens from a model, it indexes all suffixes of the prompt and previously generated tokens into a suffix cache, and looks up continuations matching the current generation context as draft proposals\. Since no draft probabilities are associated with proposed tokens, exact rejection sampling is not applicable\. Instead, proposed tokens are verified greedily and accepted only if they match the target model’s greedy prediction, making suffix decoding an approximate rather than exact acceleration method\. It works best in settings with high lexical repetition, such as agentic loops and code editing, where matching suffixes are likely to reoccur\.

## 4Method

### 4\.1Self\-Speculative Decoding

We instantiate speculative decoding using a single reasoning model evaluated at two CoT budgets, with draft distributionpθ​\(a∣q,bd\)p\_\{\\theta\}\(a\\mid q,b\_\{d\}\)and verifier distributionpθ​\(a∣q,bv\)p\_\{\\theta\}\(a\\mid q,b\_\{v\}\), wherebd<bvb\_\{d\}<b\_\{v\}\. In practice, conditioning on a partial CoTr≤br\_\{\\leq b\}requires appending an end\-of\-thinking token and, in some setups, a budget\-forcing prefix to prompt the model to produce an answer\. The key observation is that draft generation and continued CoT reasoning can proceed concurrently: while the model continues reasoning toward budgetbvb\_\{v\}, it simultaneously produces draft answer tokens conditioned onr≤bdr\_\{\\leq b\_\{d\}\}\. The overhead of drafting is therefore largely hidden behind ongoing reasoning\.

For concurrent drafting to be feasible, the total CoT length must exceed the draft budget plus the maximum draft length,

so that draft generation completes before CoT generation ends\. Beyond this prerequisite, draft quality depends on how much reasoning has completed by budgetbdb\_\{d\}: whenbdb\_\{d\}is a small fraction of the total CoT length, the partial CoT may not yet have converged toward the final answer, producing drafts that diverge in high\-entropy tasks\. In practice, we find thatbdb\_\{d\}should be a substantial fraction of the total CoT length for drafts to be useful\.

At a high level, the procedure has four phases: \(i\) generate reasoning tokens until the draft budgetbdb\_\{d\}is reached, \(ii\) concurrently sample a draft answera^\\hat\{a\}conditioned on the partial CoTr≤bdr\_\{\\leq b\_\{d\}\}while reasoning continues, \(iii\) collect the draft once reasoning reachesbvb\_\{v\}, and \(iv\) verify the draft against the high\-budget distribution, retaining the accepted prefix and sampling a continuation\. Algorithm[1](https://arxiv.org/html/2608.20359#alg1)gives the full procedure\.

Throughout,VerifyDraft​\(a^;q,rd,rv\)\\textsc\{VerifyDraft\}\(\\hat\{a\};\\,q,r\_\{d\},r\_\{v\}\)denotes standard speculative decoding verification, treatingpθ\(⋅∣q,rd\)p\_\{\\theta\}\(\\cdot\\mid q,r\_\{d\}\)as the proposal distribution andpθ\(⋅∣q,rv\)p\_\{\\theta\}\(\\cdot\\mid q,r\_\{v\}\)as the target distribution, and returning the length of the longest accepted prefix ofa^\\hat\{a\}\.

#### Suffix decoding\.

After the accepted prefix is retained, generation continues autoregressively under the verifier distribution\. To accelerate this continuation phase, we apply suffix decoding using a cache built from the draft tokensa^1,…,a^m\\hat\{a\}\_\{1\},\\ldots,\\hat\{a\}\_\{m\}\. At each step, the current generation context is used to look up matching suffixes in the cache, proposing candidate spans that are verified greedily against the verifier\. Accepted spans advance generation by multiple tokens per verifier step, reducing the number of sequential decoding steps in the continuation phase\.

This extension is most effective when the draft and final response diverge in a minor early detail but agree on long spans later in the text\. This tends to hold in structured tasks such as coding and planning, where the model commits to high\-level structure early and surface differences are localized\.

Algorithm 1Single\-Shot Self\-Speculative Decoding from Partial Reasoning1:Prompt

qq, draft budget

bdb\_\{d\}, verifier budget

bvb\_\{v\}, model

pθp\_\{\\theta\}, suffix cache

CC
2:Generated response

yy
3:Phase 1: Concurrent CoT and Draft Generation

4:

r←∅r\\leftarrow\\emptyset⊳\\trianglerightReasoning trace

5:while

\|r\|<bv\|r\|<b\_\{v\}do

6:Sample

z∼pθ\(⋅∣q,r\)z\\sim p\_\{\\theta\}\(\\cdot\\mid q,r\)and

r←concat​\(r,z\)r\\leftarrow\\mathrm\{concat\}\(r,z\)
7:if

\|r\|=bd\|r\|=b\_\{d\}then

8:

rd←rr\_\{d\}\\leftarrow r
9:Start draft generation

a^∼pθ\(⋅∣q,rd\)\\hat\{a\}\\sim p\_\{\\theta\}\(\\cdot\\mid q,r\_\{d\}\)in parallel∥\\parallel

10:endif

11:ifdraft generation finishesthen

12:Collect draft tokens

a^=\(a^1,…,a^m\)\\hat\{a\}=\(\\hat\{a\}\_\{1\},\\ldots,\\hat\{a\}\_\{m\}\)and

C←UpdateSuffixCache​\(C,a^\)C\\leftarrow\\mathrm\{UpdateSuffixCache\}\(C,\\hat\{a\}\)
13:endif

14:endwhile

15:

rv←rr\_\{v\}\\leftarrow r
16:

17:Phase 2: Verification

18:

kacc←VerifyDraft​\(a^;q,rd,rv\)k\_\{\\mathrm\{acc\}\}\\leftarrow\\textsc\{VerifyDraft\}\(\\hat\{a\};\\,q,r\_\{d\},r\_\{v\}\)
19:

a^1:kacc\\hat\{a\}\_\{1:k\_\{\\mathrm\{acc\}\}\}is the longest accepted draft prefix\.⊳\\trianglerightdraft CoT:rdr\_\{d\}, verifier CoT:rvr\_\{v\}

20:

21:Phase 3: Continuation

22:Sample continuation

acont∼pθ\(⋅∣q,rv,a^1:kacc\)a\_\{\\mathrm\{cont\}\}\\sim p\_\{\\theta\}\(\\cdot\\mid q,r\_\{v\},\\hat\{a\}\_\{1:k\_\{\\mathrm\{acc\}\}\}\)
23:Use suffix decoding with cacheCCduring continuation\.

24:return

concat​\(rv,a^1:kacc,acont\)\\mathrm\{concat\}\\left\(r\_\{v\},\\hat\{a\}\_\{1:k\_\{\\mathrm\{acc\}\}\},a\_\{\\mathrm\{cont\}\}\\right\)

### 4\.2Serving\-Time Implementation

We implement self\-speculation by duplicating the original request inside the scheduler when the draft budget is reached\. The child request shares the same prefix as the original reasoning request, so vLLM’s prefix caching can reuse the computation already performed for the prompt and partial CoT\. The child then appends an end\-of\-thinking marker and begins answer drafting, while the parent request continues along the higher\-budget reasoning path\.

This implementation is simple and requires minimal changes to the serving stack, but it is not compute\-free\. During the overlap window, the scheduler is effectively serving both the parent and child requests, which can increase the instantaneous batch size and introduce bursts of additional compute\. The speedup therefore comes from hiding draft latency behind ongoing reasoning, not from reducing total FLOPs\. Prefix caching makes this practical by avoiding redundant computation over the shared prefix\. We provide more details in the appendix\.

### 4\.3Multi\-Stage and Iterative Variants

The framework extends naturally to multi\-stage decoding, where inference budgets increase progressively\. Given a sequence of budgetsb1<b2<⋯<bTb\_\{1\}<b\_\{2\}<\\cdots<b\_\{T\}, we apply speculative decoding iteratively: at each stage, a draft generated underbib\_\{i\}is verified and extended underbi\+1b\_\{i\+1\}\. This yields a sequence of refinements in which intermediate outputs are repeatedly reused rather than discarded\. Empirically, we observe that a substantial fraction of final\-answer tokens stabilize at relatively small budgets, which makes multi\-stage reuse effective in practice\.

The single\-stage version requires selecting a draft budget in advance for sampling an initial response\. This decision is important because if the draft budget is too small, only a short prefix of the draft may be accepted\. Since the CoT length is not known in advance, this can require domain\-specific hyperparameter tuning\. In the multi\-stage version, the draft is updated at regular intervals, making this choice less brittle: one can safely choose an early initial draft budget and rely on later stages to refine and extend the draft\. The full procedure is given in Algorithm[2](https://arxiv.org/html/2608.20359#alg2)\(Appendix[C](https://arxiv.org/html/2608.20359#A3)\)\.

## 5Experiments

Schematic accepted prefix;rejected draft;regular decoding;accepted suffix\.Examplefrom datetime import datetime, timedeltaclass CalendarUtil: \.\.\.def remove\_event\(self, event\): \.\.\.\>\>\> calendar\.events = \[\{'date': datetime\(2023, 1, 1, 0, 0\),\.\.\. 'start\_time': datetime\(2023, 1, 1, 0, 0\),\.\.\. 'end\_time': datetime\(2023, 1, 1,1, 0\),\.\.\. 'description': 'New Year'\}\]

Figure 2:Top: schematic illustration of prefix verification and suffix reuse\. Bottom: compressed real code\-generation example showing the same structure\.Table 1:Main latency results across benchmarks and 4B\-class models\. Improvement is relative latency reduction, computed from mean per\-sample speedup as100⋅\(1−1/speedup\)100\\cdot\(1\-1/\\mathrm\{speedup\}\)\.Table 2:Ablation of prefix verification and suffix decoding on ClassEval with Gemma\-4\-E4B\-it\. Speedup is relative to same\-token replay baseline\.Table 3:CoT throughput with and without concurrent drafting\. Throughput is computed as aggregate CoT tokens divided by aggregate CoT time\. Draft share is the fraction of CoT time during which a draft request was active\.Table 4:Stagewise latency breakdown on code benchmarks\. Drafting may overlap with CoT generation, so components do not necessarily sum to total latency\. Suffix decoding is included in continuation time\.Table 5:Iterative SSR reduces draft\-child wall time\. Draft wall time is the summed wall\-clock lifetime of draft child requests spawned during CoT generation\.### 5\.1Experimental Setup

We evaluate SSR in an end\-to\-end generation setting\. Each request generates a chain\-of\-thought up to a maximum reasoning budget; during reasoning, SSR launches an answer draft from a lower\-budget prefix, verifies it under the higher\-budget context, and continues generation from the accepted prefix, optionally using suffix decoding with the draft as an additional suffix\-cache source\. All latency comparisons are against naive autoregressive generation with the same model and sampling settings\.

Benchmarks\.Our methods are aimed at longer\-form generation tasks where both the reasoning trace and the answer are significant contributors to generation latency\. Coding is a strong fit, since it typically requires nontrivial reasoning and produces answers of meaningful length\. For coding, we evaluate on ClassEval\(Duet al\.,[2023](https://arxiv.org/html/2608.20359#bib.bib24)\)and HumanEval\(Chenet al\.,[2021](https://arxiv.org/html/2608.20359#bib.bib23)\): ClassEval contains longer class\-level programs while HumanEval contains shorter function\-level completions\. Another primary benchmark is LongProc\(Yeet al\.,[2025](https://arxiv.org/html/2608.20359#bib.bib25)\), which contains long structured generation tasks with substantial variation across subtasks\. We focus on thecountdown,html\_to\_tsv,path\_traversal,pseudo\_to\_code, andtom\_trackingsubtasks, evaluating primarily on the 2K variant, which targets long outputs of roughly 2K tokens\.

Models\.We evaluate two 4B\-class reasoning models:Qwen3\.5\-4B\(Team,[2026](https://arxiv.org/html/2608.20359#bib.bib21)\)andGemma\-4\-E4B\-IT\(Google DeepMind,[2026](https://arxiv.org/html/2608.20359#bib.bib22)\)\. Both are open\-source, independently developed, and deployable on a single GPU, providing a well\-rounded comparison across model families\.

Metrics\.We report end\-to\-end speedup \(baseline wall\-clock latency divided by SSR latency\) along with absolute latency\. We additionally report accepted prefix tokens and suffix\-accepted tokens to measure reuse from prefix verification and suffix decoding respectively\.

Compute\.All latency experiments run on a single machine with 10 NVIDIA RTX A6000 GPUs, with one vLLM worker per GPU and batch size 1\. Tensor parallelism is not used for the 4B models\. The machine has two 24\-core Intel Xeon Gold 6342 CPUs\.

### 5\.2Results

Table[1](https://arxiv.org/html/2608.20359#S5.T1)summarizes results across LongProc 2K, ClassEval, and HumanEval\. The primary configuration uses prefix verification with suffix decoding\. SSR gives the strongest gains on ClassEval, where Gemma\-4\-E4B\-it achieves a24\.1%24\.1\\%latency reduction and Qwen3\.5\-4B achieves18\.5%18\.5\\%\. LongProc 2K shows moderate but consistent gains across both models \(7\.1%7\.1\\%–9\.1%9\.1\\%\)\. HumanEval shows the smallest improvements \(2\.9%2\.9\\%–14\.6%14\.6\\%\), which is expected: SSR only accelerates answer generation, so when CoT generation dominates total latency — as it does for HumanEval’s relatively short completions — the gains are limited regardless of draft quality\.

Stagewise latency breakdown\.Tables[4](https://arxiv.org/html/2608.20359#S5.T4)and[3](https://arxiv.org/html/2608.20359#S5.T3)decompose SSR latency and quantify the cost of concurrent drafting\. Draft generation takes88–1212seconds but runs concurrently with CoT generation, so it does not add directly to end\-to\-end latency\. The throughput cost is small: Gemma\-4\-E4B\-it shows virtually no degradation, while Qwen3\.5\-4B incurs a5\.4%5\.4\\%reduction during the draft\-active window — but since drafting is active for only2525–30%30\\%of CoT time, the effective overhead is under2%2\\%, easily offset by accepted draft tokens\. Verification is negligible \(0\.070\.07–0\.240\.24seconds\)\. The continuation phase is the primary remaining contributor: on HumanEval, CoT generation dominates relative to continuation \(20\.920\.9vs\.4\.64\.6seconds for Qwen3\.5\-4B\), explaining the smaller gains there\.

Prefix and suffix ablation\.Table[2](https://arxiv.org/html/2608.20359#S5.T2)ablates the two reuse mechanisms on ClassEval with Gemma\-4\-E4B\-it\. Prefix\-only verification yields a modest7\.9%7\.9\\%latency reduction, while suffix\-only achieves24\.1%24\.1\\%despite accepting zero prefix tokens — tokens that would have been accepted as a prefix are often recoverable as suffix matches, since they constitute exact lexical overlaps with the final answer\.

Iterative SSR overhead\.Table[5](https://arxiv.org/html/2608.20359#S5.T5)measures the wall\-clock lifetime of draft child requests under multi\-drafting and iterative SSR, whereiiis the draft interval andmmis the maximum draft length\. Iterative SSR reduces draft child wall time by11\.9%11\.9\\%at the\(i=750,m=500\)\(i\{=\}750,m\{=\}500\)setting and3\.3%3\.3\\%at\(i=500,m=250\)\(i\{=\}500,m\{=\}250\)\. The larger reduction at the longer interval reflects that later drafts are more likely to overlap with the final answer, so the bootstrapped draft requires fewer tokens to complete\.

### 5\.3Discussion

The preliminary results support three trends\. First, SSR is most effective when the final answer is long or structured enough for accepted draft tokens to offset the overhead of drafting and verification, explaining why LongProc 2K and ClassEval show stronger speedups than HumanEval\. Second, suffix decoding is important because prefix verification alone discards useful draft content after the first mismatch, especially in long\-form outputs with localized differences\. Third, the method is not uniformly beneficial: when drafts have low overlap with the final answer, or when answer continuations are too short, the additional compute can outweigh the latency hidden by concurrency\.

## 6Limitations and Future Work

SSR is most effective when answer length is comparable to CoT length; when reasoning dominates generation time, gains are limited since CoT generation is not accelerated\. In high\-entropy tasks, drafts may be semantically similar to the final response but share little lexical overlap, limiting prefix and suffix reuse\. Natural directions for future work include extending SSR to accelerate CoT generation itself and fine\-tuning models to commit to response structure earlier in the reasoning trace\.

## 7Conclusion

We introduce SSR, a training\-free self\-speculative decoding method for reasoning language models that uses the chain\-of\-thought as a source of speculation\. By conditioning the same model at two reasoning budgets, SSR derives both draft and verifier without auxiliary models or target\-model\-specific modifications, hiding draft generation overhead behind concurrent reasoning\. Combined with suffix decoding to recover useful draft content beyond the accepted prefix, SSR achieves consistent latency reductions on structured and long\-form generation tasks while preserving output quality\. We hope this work encourages exploration of reasoning structure as a resource for efficient generation\.

## References

- T\. Cai, Y\. Li, Z\. Geng, H\. Peng, J\. D\. Lee, D\. Chen, and T\. Dao \(2024\)Medusa: simple LLM inference acceleration framework with multiple decoding heads\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,R\. Salakhutdinov, Z\. Kolter, K\. A\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research,pp\. 5209–5235\.External Links:[Link](https://proceedings.mlr.press/v235/cai24b.html)Cited by:[§1](https://arxiv.org/html/2608.20359#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.20359#S2.SS1.SSS0.Px1.p1.1)\.
- C\. Chen, S\. Borgeaud, G\. Irving, J\. Lespiau, L\. Sifre, and J\. Jumper \(2023\)Accelerating large language model decoding with speculative sampling\.CoRRabs/2302\.01318\.External Links:[Link](https://doi.org/10.48550/arXiv.2302.01318),[Document](https://dx.doi.org/10.48550/ARXIV.2302.01318),2302\.01318Cited by:[§1](https://arxiv.org/html/2608.20359#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.20359#S2.SS1.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. Pondé, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. W\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, I\. Babuschkin, S\. Balaji, S\. Jain, A\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. Zaremba \(2021\)Evaluating large language models trained on code\.ArXivabs/2107\.03374\.External Links:[Link](https://api.semanticscholar.org/CorpusID:235755472)Cited by:[§5\.1](https://arxiv.org/html/2608.20359#S5.SS1.p2.1)\.
- X\. Du, M\. Liu, K\. Wang, H\. Wang, J\. Liu, Y\. Chen, J\. Feng, C\. Sha, X\. Peng, and Y\. Lou \(2023\)ClassEval: a manually\-crafted benchmark for evaluating llms on class\-level code generation\.ArXivabs/2308\.01861\.External Links:[Link](https://api.semanticscholar.org/CorpusID:260439062)Cited by:[§5\.1](https://arxiv.org/html/2608.20359#S5.SS1.p2.1)\.
- M\. Elhoushi, A\. Shrivastava, D\. Liskovich, B\. Hosmer, B\. Wasti, L\. Lai, A\. Mahmoud, B\. Acun, S\. Agarwal, A\. Roman, A\. A\. Aly, B\. Chen, and C\. Wu \(2024\)LayerSkip: enabling early exit inference and self\-speculative decoding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2024, Bangkok, Thailand, August 11\-16, 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),pp\. 12622–12642\.External Links:[Link](https://doi.org/10.18653/v1/2024.acl-long.681),[Document](https://dx.doi.org/10.18653/V1/2024.ACL-LONG.681)Cited by:[§1](https://arxiv.org/html/2608.20359#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.20359#S2.SS1.SSS0.Px2.p1.1)\.
- Google DeepMind \(2026\)Gemma 4\.Note:[https://deepmind\.google/models/gemma/gemma\-4/](https://deepmind.google/models/gemma/gemma-4/)Accessed: 2026\-05\-06Cited by:[§5\.1](https://arxiv.org/html/2608.20359#S5.SS1.p3.1)\.
- Y\. Hu, K\. Wang, X\. Zhang, F\. Zhang, C\. Li, H\. Chen, and J\. Zhang \(2025\)SAM decoding: speculative decoding via suffix automaton\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2025, Vienna, Austria, July 27 \- August 1, 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),pp\. 12187–12204\.External Links:[Link](https://aclanthology.org/2025.acl-long.595/)Cited by:[§1](https://arxiv.org/html/2608.20359#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.20359#S2.SS1.SSS0.Px3.p1.1)\.
- Y\. Leviathan, M\. Kalman, and Y\. Matias \(2023\)Fast inference from transformers via speculative decoding\.InInternational Conference on Machine Learning, ICML 2023, 23\-29 July 2023, Honolulu, Hawaii, USA,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research,pp\. 19274–19286\.External Links:[Link](https://proceedings.mlr.press/v202/leviathan23a.html)Cited by:[§1](https://arxiv.org/html/2608.20359#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.20359#S2.SS1.p1.1)\.
- J\. Li, C\. Yu, and H\. Guo \(2025\)An introduction to speculative decoding for reducing latency in ai inference\.Note:NVIDIA Technical BlogAccessed: 2026\-03\-31External Links:[Link](https://developer.nvidia.com/blog/an-introduction-to-speculative-decoding-for-reducing-latency-in-ai-inference/)Cited by:[§3\.2](https://arxiv.org/html/2608.20359#S3.SS2.p6.3)\.
- Y\. Li, F\. Wei, C\. Zhang, and H\. Zhang \(2024\)EAGLE: speculative sampling requires rethinking feature uncertainty\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,R\. Salakhutdinov, Z\. Kolter, K\. A\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research,pp\. 28935–28948\.External Links:[Link](https://proceedings.mlr.press/v235/li24bt.html)Cited by:[§1](https://arxiv.org/html/2608.20359#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.20359#S2.SS1.SSS0.Px1.p1.1)\.
- J\. Liu, Q\. Wang, J\. Wang, and X\. Cai \(2024\)Speculative decoding via early\-exiting for faster LLM inference with thompson sampling control mechanism\.InFindings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11\-16, 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Findings of ACL,pp\. 3027–3043\.External Links:[Link](https://doi.org/10.18653/v1/2024.findings-acl.179),[Document](https://dx.doi.org/10.18653/V1/2024.FINDINGS-ACL.179)Cited by:[§1](https://arxiv.org/html/2608.20359#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.20359#S2.SS1.SSS0.Px2.p1.1)\.
- X\. Liu and L\. Wang \(2025\)Answer convergence as a signal for early stopping in reasoning\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4\-9, 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),pp\. 17896–17907\.External Links:[Link](https://doi.org/10.18653/v1/2025.emnlp-main.904),[Document](https://dx.doi.org/10.18653/V1/2025.EMNLP-MAIN.904)Cited by:[§2\.2](https://arxiv.org/html/2608.20359#S2.SS2.p1.1)\.
- N\. Muennighoff, Z\. Yang, W\. Shi, X\. L\. Li, L\. Fei\-Fei, H\. Hajishirzi, L\. Zettlemoyer, P\. Liang, E\. J\. Candès, and T\. Hashimoto \(2025\)S1: simple test\-time scaling\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4\-9, 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),pp\. 20275–20321\.External Links:[Link](https://doi.org/10.18653/v1/2025.emnlp-main.1025),[Document](https://dx.doi.org/10.18653/V1/2025.EMNLP-MAIN.1025)Cited by:[§1](https://arxiv.org/html/2608.20359#S1.p4.1),[§2\.2](https://arxiv.org/html/2608.20359#S2.SS2.p1.1)\.
- G\. Oliaro, Z\. Jia, D\. F\. Campos, and A\. Qiao \(2024\)SuffixDecoding: A model\-free approach to speeding up large language model inference\.CoRRabs/2411\.04975\.External Links:[Link](https://doi.org/10.48550/arXiv.2411.04975),[Document](https://dx.doi.org/10.48550/ARXIV.2411.04975),2411\.04975Cited by:[§1](https://arxiv.org/html/2608.20359#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.20359#S2.SS1.SSS0.Px3.p1.1),[§3\.3](https://arxiv.org/html/2608.20359#S3.SS3.p1.1)\.
- A\. Saxena \(2023\)Prompt lookup decoding\.External Links:[Link](https://github.com/apoorvumang/prompt-lookup-decoding/)Cited by:[§1](https://arxiv.org/html/2608.20359#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.20359#S2.SS1.SSS0.Px3.p1.1)\.
- C\. Snell, J\. Lee, K\. Xu, and A\. Kumar \(2024\)Scaling LLM test\-time compute optimally can be more effective than scaling model parameters\.CoRRabs/2408\.03314\.External Links:[Link](https://doi.org/10.48550/arXiv.2408.03314),[Document](https://dx.doi.org/10.48550/ARXIV.2408.03314),2408\.03314Cited by:[§2\.2](https://arxiv.org/html/2608.20359#S2.SS2.p1.1)\.
- Q\. Team \(2026\)Qwen3\.5\-omni technical report\.External Links:2604\.15804,[Link](https://arxiv.org/abs/2604.15804)Cited by:[§5\.1](https://arxiv.org/html/2608.20359#S5.SS1.p3.1)\.
- R\. Tiwari, H\. Xi, A\. Tomar, C\. R\. C\. Hooper, S\. Kim, M\. Horton, M\. Najibi, M\. W\. Mahoney, K\. Keutzer, and A\. Gholami \(2025\)QuantSpec: self\-speculative decoding with hierarchical quantized KV cache\.InForty\-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13\-19, 2025,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research\.External Links:[Link](https://proceedings.mlr.press/v267/tiwari25b.html)Cited by:[§1](https://arxiv.org/html/2608.20359#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.20359#S2.SS1.SSS0.Px2.p1.1)\.
- S\. Welleck, A\. Bertsch, M\. Finlayson, H\. Schoelkopf, A\. Xie, G\. Neubig, I\. Kulikov, and Z\. Harchaoui \(2024\)From decoding to meta\-generation: inference\-time algorithms for large language models\.Trans\. Mach\. Learn\. Res\.2024\.External Links:[Link](https://openreview.net/forum?id=eskQMcIbMS)Cited by:[§2\.2](https://arxiv.org/html/2608.20359#S2.SS2.p1.1)\.
- X\. Ye, F\. Yin, Y\. He, J\. Zhang, H\. Yen, T\. Gao, G\. Durrett, and D\. Chen \(2025\)LongProc: benchmarking long\-context language models on long procedural generation\.ArXivabs/2501\.05414\.External Links:[Link](https://api.semanticscholar.org/CorpusID:275405405)Cited by:[§5\.1](https://arxiv.org/html/2608.20359#S5.SS1.p2.1)\.
- Y\. Yue, Y\. Xue, and J\. Huang \(2026\)SpecAttn: co\-designing sparse attention with self\-speculative decoding\.CoRRabs/2602\.07223\.External Links:[Link](https://doi.org/10.48550/arXiv.2602.07223),[Document](https://dx.doi.org/10.48550/ARXIV.2602.07223),2602\.07223Cited by:[§1](https://arxiv.org/html/2608.20359#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.20359#S2.SS1.SSS0.Px2.p1.1)\.
- Y\. Zhao, J\. Tang, K\. Zhu, Z\. Ye, C\. Chang, C\. Lin, J\. Park, G\. Xiao, M\. S\. Abdelfattah, M\. Gao, B\. Kasikci, S\. Han, and I\. Stoica \(2025\)Accelerating large\-scale reasoning model inference with sparse self\-speculative decoding\.External Links:2512\.01278,[Link](https://arxiv.org/abs/2512.01278)Cited by:[§1](https://arxiv.org/html/2608.20359#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.20359#S2.SS1.SSS0.Px2.p1.1)\.
- M\. Zhong, N\. Teku, and R\. Tandon \(2025\)Speeding up speculative decoding via approximate verification\.CoRRabs/2502\.04557\.External Links:[Link](https://doi.org/10.48550/arXiv.2502.04557),[Document](https://dx.doi.org/10.48550/ARXIV.2502.04557),2502\.04557Cited by:[§2\.1](https://arxiv.org/html/2608.20359#S2.SS1.p1.1)\.

## Appendix AServing\-Time Implementation

When the reasoning request reaches the draft budget, the scheduler spawns a childDraftingrequest initialized with the same token prefix, followed by the model\-specific end\-of\-thinking tokens\. The child generates up to a configured draft\-token budget while the parent continues along the longer reasoning path\.

In multi\-drafting mode, the scheduler spawns draft children at fixed token increments, storing outputs by spawn point\. When the parent finishes, outstanding draft children are removed and the latest usable draft is selected for verification\.

Verification is implemented as aVerificationrequest whose prompt is the final reasoning prefix followed by the selected draft tokens\. Prompt log probabilities are compared against the draft’s log probabilities and a prefix is accepted using the standard speculative decoding rule\. AContinuationrequest then generates remaining answer tokens from the accepted prefix, optionally using the draft as a suffix cache source\. All stages are ordinary vLLM requests coordinated by scheduler metadata, with the final output returned under the original request id\.

## Appendix BInference Hyperparameters

Unless otherwise stated, all latency experiments use batch size 1 and run one independent vLLM worker per GPU\. Exploratory runs use eager mode, no warmup, and one measured run\. For each accelerated request, we run a separate non\-speculative vLLM instance as the baseline and generate the same number of total response tokens as the accelerated request\.

Table 6:Default inference and self\-speculative decoding hyperparameters\.For Qwen\-style thinking models, the end\-of\-thinking delimiter is</think\>\. For Gemma\-4\-E4B\-it, we use the native thinking boundary<channel\|\>\. When the model naturally emits the thinking boundary before the CoT cap, we truncate the CoT at that boundary and begin verification and continuation from the resulting prefix\. Continuation generation is allowed to terminate at EOS before reaching the cap\.

## Appendix CMulti\-Stage Self\-Speculative Decoding

Algorithm 2Multi\-Stage Self\-Speculative Decoding from Partial Reasoning1:Prompt

qq, budgets

b1<b2<⋯<bTb\_\{1\}<b\_\{2\}<\\cdots<b\_\{T\}, model

pθp\_\{\\theta\}, suffix cache

CC
2:Generated response

yy, updated suffix cache

CC
3:Generate reasoning up to budget

b1b\_\{1\}to obtain

r1r\_\{1\}
4:Start draft generation

a^1∼pθ\(⋅∣q,r1\)\\hat\{a\}\_\{1\}\\sim p\_\{\\theta\}\(\\cdot\\mid q,r\_\{1\}\)in parallel

5:for

i=1,…,T−1i=1,\\ldots,T\-1do

6:Continue reasoning to budget

bi\+1b\_\{i\+1\}to obtain

ri\+1r\_\{i\+1\}
7:Collect draft

a^i\\hat\{a\}\_\{i\}and update suffix cache

CC
8:

ki←VerifyDraft​\(a^i;q,ri,ri\+1\)k\_\{i\}\\leftarrow\\textsc\{VerifyDraft\}\(\\hat\{a\}\_\{i\};\\,q,r\_\{i\},r\_\{i\+1\}\)
9:

a^i,1:ki\\hat\{a\}\_\{i,1:k\_\{i\}\}is the longest prefix accepted under the higher\-budget contextri\+1r\_\{i\+1\}\.

10:if

i<T−1i<T\-1then

11:Start next\-stage draft generation

a^i\+1∼pθ\(⋅∣q,ri\+1,a^i,1:ki\)\\hat\{a\}\_\{i\+1\}\\sim p\_\{\\theta\}\(\\cdot\\mid q,r\_\{i\+1\},\\hat\{a\}\_\{i,1:k\_\{i\}\}\)
12:Reuse the accepted prefix rather than drafting from scratch\.

13:endif

14:endfor

15:Sample continuation

acont∼pθ\(⋅∣q,rT,a^T−1,1:kT−1\)a\_\{\\mathrm\{cont\}\}\\sim p\_\{\\theta\}\(\\cdot\\mid q,r\_\{T\},\\hat\{a\}\_\{T\-1,1:k\_\{T\-1\}\}\)
16:Use suffix decoding with cacheCCduring continuation\.

17:return

concat​\(rT,a^T−1,1:kT−1,acont\),C\\mathrm\{concat\}\\left\(r\_\{T\},\\hat\{a\}\_\{T\-1,1:k\_\{T\-1\}\},a\_\{\\mathrm\{cont\}\}\\right\),C

## NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact\. Do not remove the checklist:The papers not including the checklist will be desk rejected\.The checklist should follow the references and follow the \(optional\) supplemental material\. The checklist does NOT count towards the page limit\.

Please read the checklist guidelines carefully for information on how to answer these questions\. For each question in the checklist:

- •You should answer\[Yes\],\[No\], or\[N/A\]\.
- •\[N/A\]means either that the question is Not Applicable for that particular paper or the relevant information is Not Available\.
- •Please provide a short \(1–2 sentence\) justification right after your answer \(even for\[N/A\]\)\.

The checklist answers are an integral part of your paper submission\.They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers\. You will also be asked to include it \(after eventual revisions\) with the final version of your paper, and its final version will be published with the paper\.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation\. While\[Yes\]is generally preferable to\[No\], it is perfectly acceptable to answer\[No\]provided a proper justification is given \(e\.g\., error bars are not reported because it would be too computationally expensive” or “we were unable to find the license for the dataset we used”\)\. In general, answering\[No\]or\[N/A\]is not grounds for rejection\. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate\. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix\. If you answer\[Yes\]to a question, in the justification please point to the section\(s\) where related material for the question can be found\.

IMPORTANT, please:

- •Delete this instruction block, but keep the section heading “NeurIPS Paper Checklist",
- •Keep the checklist subsection headings, questions/answers and guidelines below\.
- •Do not modify the questions and only use the provided macros for your answers\.

1. 1\.Claims
2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
3. Answer:\[Yes\]
4. Justification: We describe the settings and scope intended for the outlined methods within the paper\.
5. Guidelines: - •The answer\[N/A\]means that the abstract and introduction do not include the claims made in the paper\. - •The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations\. A\[No\]or\[N/A\]answer to this question will not be perceived well by the reviewers\. - •The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings\. - •It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper\.
6. 2\.Limitations
7. Question: Does the paper discuss the limitations of the work performed by the authors?
8. Answer:\[Yes\]\.
9. Justification: We add a limitations section to describe settings in which our algorithm will likely struggle\.
10. Guidelines: - •The answer\[N/A\]means that the paper has no limitation while the answer\[No\]means that the paper has limitations, but those are not discussed in the paper\. - •The authors are encouraged to create a separate “Limitations” section in their paper\. - •The paper should point out any strong assumptions and how robust the results are to violations of these assumptions \(e\.g\., independence assumptions, noiseless settings, model well\-specification, asymptotic approximations only holding locally\)\. The authors should reflect on how these assumptions might be violated in practice and what the implications would be\. - •The authors should reflect on the scope of the claims made, e\.g\., if the approach was only tested on a few datasets or with a few runs\. In general, empirical results often depend on implicit assumptions, which should be articulated\. - •The authors should reflect on the factors that influence the performance of the approach\. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting\. Or a speech\-to\-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon\. - •The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size\. - •If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness\. - •While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper\. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community\. Reviewers will be specifically instructed to not penalize honesty concerning limitations\.
11. 3\.Theory assumptions and proofs
12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete \(and correct\) proof?
13. Answer:\[N/A\]\.
14. Justification: No theoretical results states in this paper\.
15. Guidelines: - •The answer\[N/A\]means that the paper does not include theoretical results\. - •All the theorems, formulas, and proofs in the paper should be numbered and cross\-referenced\. - •All assumptions should be clearly stated or referenced in the statement of any theorems\. - •The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition\. - •Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material\. - •Theorems and Lemmas that the proof relies upon should be properly referenced\.
16. 4\.Experimental result reproducibility
17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper \(regardless of whether the code and data are provided or not\)?
18. Answer:\[Yes\]\.
19. Justification: We outline the datasets and inference settings in the experiments section, and plan on releasing code upon publication\.
20. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •If the paper includes experiments, a\[No\]answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not\. - •If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable\. - •Depending on the contribution, reproducibility can be accomplished in various ways\. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model\. In general\. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model \(e\.g\., in the case of a large language model\), releasing of a model checkpoint, or other means that are appropriate to the research performed\. - •While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution\. For example 1. \(a\)If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm\. 2. \(b\)If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully\. 3. \(c\)If the contribution is a new model \(e\.g\., a large language model\), then there should either be a way to access this model for reproducing the results or a way to reproduce the model \(e\.g\., with an open\-source dataset or instructions for how to construct the dataset\)\. 4. \(d\)We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility\. In the case of closed\-source models, it may be that access to the model is limited in some way \(e\.g\., to registered users\), but it should be possible for other researchers to have some path to reproducing or verifying the results\.
21. 5\.Open access to data and code
22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
23. Answer:\[No\]
24. Justification: We intend to release our code upon publication\.
25. Guidelines: - •The answer\[N/A\]means that paper does not include experiments requiring code\. - • - •While we encourage the release of code and data, we understand that this might not be possible, so\[No\]is an acceptable answer\. Papers cannot be rejected simply for not including code, unless this is central to the contribution \(e\.g\., for a new open\-source benchmark\)\. - •The instructions should contain the exact command and environment needed to run to reproduce the results\. See the NeurIPS code and data submission guidelines \([https://neurips\.cc/public/guides/CodeSubmissionPolicy](https://neurips.cc/public/guides/CodeSubmissionPolicy)\) for more details\. - •The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc\. - •The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines\. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why\. - •At submission time, to preserve anonymity, the authors should release anonymized versions \(if applicable\)\. - •Providing as much information as possible in supplemental material \(appended to the paper\) is recommended, but including URLs to data and code is permitted\.
26. 6\.Experimental setting/details
27. Question: Does the paper specify all the training and test details \(e\.g\., data splits, hyperparameters, how they were chosen, type of optimizer\) necessary to understand the results?
28. Answer:\[Yes\]
29. Justification: Described in the experiments section, and within the evaluation scripts\.
30. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them\. - •The full details can be provided either with the code, in appendix, or as supplemental material\.
31. 7\.Experiment statistical significance
32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
33. Answer:\[No\]
34. Justification: As in other similar works, we report only the results obtained by standard evaluation scripts, which might not include statistical signficance\.
35. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The authors should answer\[Yes\]if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper\. - •The factors of variability that the error bars are capturing should be clearly stated \(for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions\)\. - •The method for calculating the error bars should be explained \(closed form formula, call to a library function, bootstrap, etc\.\) - •The assumptions made should be given \(e\.g\., Normally distributed errors\)\. - •It should be clear whether the error bar is the standard deviation or the standard error of the mean\. - •It is OK to report 1\-sigma error bars, but one should state it\. The authors should preferably report a 2\-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified\. - •For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range \(e\.g\., negative error rates\)\. - •If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text\.
36. 8\.Experiments compute resources
37. Question: For each experiment, does the paper provide sufficient information on the computer resources \(type of compute workers, memory, time of execution\) needed to reproduce the experiments?
38. Answer:\[Yes\]\.
39. Justification: We mention the resources used within the experiments section\.
40. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage\. - •The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute\. - •The paper should disclose whether the full research project required more compute than the experiments reported in the paper \(e\.g\., preliminary or failed experiments that didn’t make it into the paper\)\.
41. 9\.Code of ethics
43. Answer:\[Yes\]\.
44. Justification: We believe we have abided by the NeurIPS code of ethics in the submission of this paper\.
45. Guidelines: - •The answer\[N/A\]means that the authors have not reviewed the NeurIPS Code of Ethics\. - •If the authors answer\[No\], they should explain the special circumstances that require a deviation from the Code of Ethics\. - •The authors should make sure to preserve anonymity \(e\.g\., if there is a special consideration due to laws or regulations in their jurisdiction\)\.
46. 10\.Broader impacts
47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
48. Answer:\[N/A\]\.
49. Justification: This work will likely have no direct societal impact\.
50. Guidelines: - •The answer\[N/A\]means that there is no societal impact of the work performed\. - •If the authors answer\[N/A\]or\[No\], they should explain why their work has no societal impact or why the paper does not address societal impact\. - •Examples of negative societal impacts include potential malicious or unintended uses \(e\.g\., disinformation, generating fake profiles, surveillance\), fairness considerations \(e\.g\., deployment of technologies that could make decisions that unfairly impact specific groups\), privacy considerations, and security considerations\. - •The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments\. However, if there is a direct path to any negative applications, the authors should point it out\. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation\. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster\. - •The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from \(intentional or unintentional\) misuse of the technology\. - •If there are negative societal impacts, the authors could also discuss possible mitigation strategies \(e\.g\., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML\)\.
51. 11\.Safeguards
52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse \(e\.g\., pre\-trained language models, image generators, or scraped datasets\)?
53. Answer:\[N/A\]
54. Justification: This work has low risk of misuse, as it is an inference\-time optimization\.
55. Guidelines: - •The answer\[N/A\]means that the paper poses no such risks\. - •Released models that have a high risk for misuse or dual\-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters\. - •Datasets that have been scraped from the Internet could pose safety risks\. The authors should describe how they avoided releasing unsafe images\. - •We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort\.
56. 12\.Licenses for existing assets
57. Question: Are the creators or original owners of assets \(e\.g\., code, data, models\), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
58. Answer:\[Yes\]
59. Justification: We believe we have credited the original owners of the main code/data/models used in this work\.
60. Guidelines: - •The answer\[N/A\]means that the paper does not use existing assets\. - •The authors should cite the original paper that produced the code package or dataset\. - •The authors should state which version of the asset is used and, if possible, include a URL\. - •The name of the license \(e\.g\., CC\-BY 4\.0\) should be included for each asset\. - •For scraped data from a particular source \(e\.g\., website\), the copyright and terms of service of that source should be provided\. - •If assets are released, the license, copyright information, and terms of use in the package should be provided\. For popular datasets,[paperswithcode\.com/datasets](https://paperswithcode.com/datasets)has curated licenses for some datasets\. Their licensing guide can help determine the license of a dataset\. - •For existing datasets that are re\-packaged, both the original license and the license of the derived asset \(if it has changed\) should be provided\. - •If this information is not available online, the authors are encouraged to reach out to the asset’s creators\.
61. 13\.New assets
62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
63. Answer:\[Yes\]
64. Justification: Our code is accompanied with relevant documentation\.
65. Guidelines: - •The answer\[N/A\]means that the paper does not release new assets\. - •Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates\. This includes details about training, license, limitations, etc\. - •The paper should discuss whether and how consent was obtained from people whose asset is used\. - •At submission time, remember to anonymize your assets \(if applicable\)\. You can either create an anonymized URL or include an anonymized zip file\.
66. 14\.Crowdsourcing and research with human subjects
67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation \(if any\)?
68. Answer:\[N/A\]
69. Justification: No human subjects or sources used for this paper\.
70. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper\. - •According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector\.
71. 15\.Institutional review board \(IRB\) approvals or equivalent for research with human subjects
72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board \(IRB\) approvals \(or an equivalent approval/review based on the requirements of your country or institution\) were obtained?
73. Answer:\[N/A\]
74. Justification: No crowdsourcing with human subjects\.
75. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Depending on the country in which research is conducted, IRB approval \(or equivalent\) may be required for any human subjects research\. If you obtained IRB approval, you should clearly state this in the paper\. - •We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution\. - •For initial submissions, do not include any information that would break anonymity \(if applicable\), such as the institution conducting the review\.
76. 16\.Declaration of LLM usage
77. Question: Does the paper describe the usage of LLMs if it is an important, original, or non\-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does*not*impact the core methodology, scientific rigor, or originality of the research, declaration is not required\.
78. Answer:\[N/A\]
79. Justification: LLMs used only for writing/editing/formatting\.
80. Guidelines: - •The answer\[N/A\]means that the core method development in this research does not involve LLMs as any important, original, or non\-standard components\. - •Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described\.