The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Experts

# The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Experts
Source: [https://arxiv.org/html/2609.22156](https://arxiv.org/html/2609.22156)
###### Abstract

Speculative decoding in Mixture\-of\-Experts \(MoE\) models faces the problem of unstable verification cost caused by input\-dependent expert loading\. To study the physics of this process, we formulate speculation\-budget selection as an offline Stochastic Shortest Path \(SSP\) problem over reference sequences and build a diagnostic Oracle that uses counterfactual simulation to account for MoE verification cost\. A detailed analysis of the Oracle’s decisions on the Qwen3\-Coder and EAGLE\-3 pairing, in the space of marginal deltas \(Delta Space\), shows that rejected candidates form a strict linear boundary\. This result demonstrates that a complex global optimization is locally governed by a necessary condition balancing marginal cost against expected progress \(Δ​𝔼​\[C​o​s​t\]Δ​𝔼​\[a\]\\frac\{\\Delta\\mathbb\{E\}\[Cost\]\}\{\\Delta\\mathbb\{E\}\[a\]\}\), providing a rigorous mathematical reference point for designing future adaptive online heuristics\.

## 1Introduction

Autoregressive decoding remains sequential, and at small batch sizes its latency is often dominated by reading weights and the KV cache from memory\([Shazeer, 2019](https://arxiv.org/html/2609.22156#bib.bib20);[Pope et al\., 2023](https://arxiv.org/html/2609.22156#bib.bib17)\)\. Speculative decoding \(SD\) reduces the number of sequential calls to the target model: a cheap draft model proposes several tokens, which are verified by the target model in a single parallel pass\([Leviathan et al\., 2023](https://arxiv.org/html/2609.22156#bib.bib10);[Chen et al\., 2023](https://arxiv.org/html/2609.22156#bib.bib4)\)\. For dense models this pass partially amortizes the weight read, so generation efficiency is usually tied directly to draft\-model quality and the number of accepted tokens\.

For Mixture\-of\-Experts \(MoE\) architectures, this classical explanation is insufficient\. Because the router selects experts separately for each token\([Shazeer et al\., 2017](https://arxiv.org/html/2609.22156#bib.bib21);[Fedus et al\., 2022](https://arxiv.org/html/2609.22156#bib.bib6);[Lepikhin et al\., 2020](https://arxiv.org/html/2609.22156#bib.bib9)\), positions within a single speculative pass can access entirely different sets of weights\([Huang et al\., 2025](https://arxiv.org/html/2609.22156#bib.bib8);[Saxena et al\., 2025](https://arxiv.org/html/2609.22156#bib.bib19)\)\. A long speculative chain increases the expected number of accepted tokens, but it simultaneously widens the union of activated experts for the tail of positions that may end up being rejected\. Consequently, a high token\-acceptance probability alone no longer guarantees a speedup: in MoE models one must carefully balance the chance of a correct guess against the input\-dependent cost of verification\.

Existing approaches try to address this problem by developing various online heuristics \(e\.g\., confidence\-based cutoffs or dynamic budgets\)\. However, because of the high local variance in routing cost, such heuristics often operate blindly, without a rigorous analytical reference point\. Rather than proposing yet another empirical online policy, in this work we take a step back and ask:what is the absolute theoretical limit on the speedup of speculative decoding in MoE, and what fundamental rules does the ideal strategy obey?

To answer this question, we formulate the search for the optimal speculation budget as a Stochastic Shortest Path \(SSP\) problem over a known reference \(ground\-truth\) sequence\. We build a hybrid offline Oracle that lets us isolate algorithmic losses from systems\-level ones and study the local “physics” of decision\-making\.

##### Our Contributions\.

Our contributions are as follows:

- •We develop theSequence\-Conditioned Offline Oraclemethodology\. Using counterfactual simulation of “junk tails”, the Oracle allows an honest estimate of the cost of erroneous tokens in an MoE model without distorting the reference generation path\.
- •We empirically demonstrate thelimits of speculation in MoE\. Unlike dense models, increasing the available budget \(Lm​a​xL\_\{max\}\) in MoE leads to a strong diminishing\-returns effect: the ideal algorithm forcibly truncates budgets to avoid routing penalties for low\-probability tokens\.
- •We introduceDelta Space Analysis— an examination of the Oracle’s space of marginal decisions\. We show that the complex global search over the Bellman equation reduces in practice to maintaining a simple linear balance \(Δ​𝔼​\[C​o​s​t\]Δ​𝔼​\[a\]\\frac\{\\Delta\\mathbb\{E\}\[Cost\]\}\{\\Delta\\mathbb\{E\}\[a\]\}\)\. This demonstrates that lightweight online heuristics based on early exit, triggered once a constant cost\-slope barrier is crossed, are feasible\.

## 2Background & Related Work

### 2\.1Decode Cost as the Baseline for Speculative Decoding

Speculative decoding targets thedecodephase, in which autoregressive generation sequentially appends tokens after theprefillphase has built the KV cache\([Vaswani et al\., 2017](https://arxiv.org/html/2609.22156#bib.bib22);[Pope et al\., 2023](https://arxiv.org/html/2609.22156#bib.bib17)\)\. At small batch sizes, each decoding step can be dominated by reading model weights and KV\-cache state from HBM, so latency is not captured byFLOPsalone\([Shazeer, 2019](https://arxiv.org/html/2609.22156#bib.bib20);[Pope et al\., 2023](https://arxiv.org/html/2609.22156#bib.bib17)\)\.

### 2\.2MoE Inference

MoE models increase overall capacity through conditional computation: for each token, only a small subset of experts, chosen by the router, is activated\([Shazeer et al\., 2017](https://arxiv.org/html/2609.22156#bib.bib21);[Fedus et al\., 2022](https://arxiv.org/html/2609.22156#bib.bib6);[Lepikhin et al\., 2020](https://arxiv.org/html/2609.22156#bib.bib9)\)\. Because routing is performed independently for each token, positions verified in a single target\-model pass may access different expert identifiers, which can increase the volume of weight access and overhead on the target\-model side\([Huang et al\., 2025](https://arxiv.org/html/2609.22156#bib.bib8);[Saxena et al\., 2025](https://arxiv.org/html/2609.22156#bib.bib19)\)\. The execution path of the target MoE model remains unchanged; what is measured is the relationship between draft length and the layer\-averaged number of unique experts, on one hand, and verification cost, on the other\.

### 2\.3Speculative Decoding

Speculative decoding accelerates autoregressive generation via a “draft\-then\-verify” scheme\([Leviathan et al\., 2023](https://arxiv.org/html/2609.22156#bib.bib10);[Chen et al\., 2023](https://arxiv.org/html/2609.22156#bib.bib4)\)\. In each round, the draft model proposesKKdraft tokens, and the target model verifies them in a single verification pass\. Tokens are then accepted sequentially up to the first mismatch; under sampling, rejection sampling is used to preserve the target model’s distribution\.

EAGLE is treated here as an example of a strong contemporary draft\-side approach\([Li et al\., 2024b](https://arxiv.org/html/2609.22156#bib.bib12);[Li et al\., 2024a](https://arxiv.org/html/2609.22156#bib.bib11);[Li et al\., 2025](https://arxiv.org/html/2609.22156#bib.bib13)\)\. The key question is a different one: can the target MoE model’s verification pass become expensive enough that the expected speedup vanishes even with a sufficiently good draft model?

### 2\.4Related Work

Classical speculative decoding verifies draft\-model continuations with the target model and preserves the target model’s distribution via rejection sampling\([Leviathan et al\., 2023](https://arxiv.org/html/2609.22156#bib.bib10);[Chen et al\., 2023](https://arxiv.org/html/2609.22156#bib.bib4)\)\. This line of work has produced stronger draft\-side mechanisms, including feature\-level drafting, dynamic candidate trees, multiple output heads, architectural changes to the draft decoder, and specialized training objectives\([Li et al\., 2024b](https://arxiv.org/html/2609.22156#bib.bib12);[Li et al\., 2024a](https://arxiv.org/html/2609.22156#bib.bib11);[Li et al\., 2025](https://arxiv.org/html/2609.22156#bib.bib13);[Cai et al\., 2024](https://arxiv.org/html/2609.22156#bib.bib3);[Wang et al\., 2026](https://arxiv.org/html/2609.22156#bib.bib25);[Zhong et al\., 2025](https://arxiv.org/html/2609.22156#bib.bib31);[Samarin et al\., 2026](https://arxiv.org/html/2609.22156#bib.bib18)\)\. Such methods mainly improve candidate quality, draft\-model overhead, or tree construction prior to verification by the target model\. The present study instead isolates the configuration\-dependent cost of the target MoE model’s verification pass itself\.

Methods with adaptive lookahead depth change the length or structure of speculation at runtime\. DISCO uses a draft\-model confidence signal, GammaTune adapts the budget from the acceptance history of previous rounds, SVIP ties stopping to draft entropy, TETRIS optimizes token selection under batching, and AdaEAGLE explicitly predicts an adaptive structure\([Mamou et al\., 2024](https://arxiv.org/html/2609.22156#bib.bib14);[Gautam et al\., 2025](https://arxiv.org/html/2609.22156#bib.bib7);[Zhang et al\., 2024b](https://arxiv.org/html/2609.22156#bib.bib30);[Wu et al\., 2025](https://arxiv.org/html/2609.22156#bib.bib27);[Zhang et al\., 2024a](https://arxiv.org/html/2609.22156#bib.bib29)\)\. These works motivate dynamic control; DISCO and GammaTune, however, do not directly model the MoE\-specific signal of expert\-union size in the target model’s pass\. Consequently, adaptive or oracle\-based length selection by itself is not a contribution of the present study\.

Work on speculative decoding for MoE explicitly accounts for the cost of target verification\. MoESD introduces target\-model efficiency, and Cascade weighs expected progress against verification cost to avoid unfavorable speculation\([Huang et al\., 2025](https://arxiv.org/html/2609.22156#bib.bib8);[Saxena et al\., 2025](https://arxiv.org/html/2609.22156#bib.bib19)\)\. EVICT and MoE\-Spec directly exploit the growth of the activated\-expert union: EVICT prunes the candidate tree, and MoE\-Spec caps the per\-layer expert budget\([Pan et al\., 2026](https://arxiv.org/html/2609.22156#bib.bib16);[McDanel et al\., 2026](https://arxiv.org/html/2609.22156#bib.bib15)\)\. EcoSpec incorporates the predicted marginal cost of expert activation into draft\-token selection\([Xie et al\., 2026](https://arxiv.org/html/2609.22156#bib.bib28)\), while Sparse Verification sparsifies the computation of verification itself\([Wang et al\., 2025a](https://arxiv.org/html/2609.22156#bib.bib23)\)\. These approaches modify the verification policy or the target model’s computation; the link between the number of unique experts and cost is therefore not claimed here as novel\.

A related line of systems research treats expert loading and data movement as the binding constraint\. SpecMoE, SP\-MoE, and SpecMoEOff use self\-speculation, weight prefetching, or offloading to reduce the memory burden of MoE inference, while MoE\-SpeQ uses quantized drafting for accurate prediction of expert routing and its prefetching, hiding I/O latency\([Bang et al\., 2026](https://arxiv.org/html/2609.22156#bib.bib2);[Chen et al\., 2025](https://arxiv.org/html/2609.22156#bib.bib5);[Wang et al\., 2025c](https://arxiv.org/html/2609.22156#bib.bib26);[Wang et al\., 2025b](https://arxiv.org/html/2609.22156#bib.bib24)\)\. These systems change the expert\-loading regime or the model’s execution environment\. SPEED\-Bench further shows that results depend on workload, batching, and environment\([Abramovich et al\., 2026](https://arxiv.org/html/2609.22156#bib.bib1)\)\. By the same logic, the synthetically resampled counterfactual tail in the present work could in principle route differently than the real continuation of the same position\.

The contribution of this paper is limited to a configuration\-dependent cost calibration of an unmodified target MoE model and to a sequence\-conditioned SSP analysis\. The paper does not propose an online policy and does not claim superiority over EVICT, MoE\-Spec, EcoSpec, or other adaptive methods\.

## 3Problem Formulation & The Cost Model

### 3\.1Useful Progress and Target Model Time

Following MoESD\([Huang et al\., 2025](https://arxiv.org/html/2609.22156#bib.bib8)\), we separate three quantities: mean useful progress, draft cost, and target\-model verification time;B=1B=1in all experiments\. For a measured run ofRRrounds,GrG\_\{r\}denotes the number of useful output tokens in roundrr; in the sequence\-conditioned setting below,aadenotes the number of consecutive matches with a fixed reference and is not a count of rejection\-sampling acceptances\. LetTT\(B\)​\(K\)T\_\{T\}^\{\(B\)\}\(K\)denote the target model’s verification time forKKdraft\-token positions plus one extra target\-model position, and letTT\(B\)​\(0\)T\_\{T\}^\{\(B\)\}\(0\)denote the same pass with no draft tokens\. The normalized target\-model efficiency from MoESD isolates the ratio of these pass times:

ηB​\(K\)=TT\(B\)​\(0\)TT\(B\)​\(K\)\.\\eta\_\{B\}\(K\)=\\frac\{T\_\{T\}^\{\(B\)\}\(0\)\}\{T\_\{T\}^\{\(B\)\}\(K\)\}\.\(1\)For a fixed batch size and execution\-environment configuration, we hereafter allow the shorthandTT​\(K\)T\_\{T\}\(K\)andη⁡\(K\)\\eta\(K\); in the present analysis,η⁡\(K\)\\eta\(K\)serves as a descriptive normalization and does not enter directly into the offline policy’s objective\.

UppercaseKKhenceforth denotes the measured calibration budget, while lowercasekkdenotes the offline policy’s action \(Section[4](https://arxiv.org/html/2609.22156#S4)\)\.

Here MoE changes the problem setting: unlike dense models, whereTT​\(K\)T\_\{T\}\(K\)depends weakly onKKfrom a memory\-traffic standpoint, in MoE each verified token selects its own subset of experts, so the verification pass can increase the layer\-averaged number of unique expertsU¯r\\bar\{U\}\_\{r\}relative to a single decoding step\([Huang et al\., 2025](https://arxiv.org/html/2609.22156#bib.bib8);[Saxena et al\., 2025](https://arxiv.org/html/2609.22156#bib.bib19)\)\. This dependence amongTT​\(K\)T\_\{T\}\(K\),η⁡\(K\)\\eta\(K\), andU¯r\\bar\{U\}\_\{r\}is unpacked by the verification cost model below\.

### 3\.2Speedup and Degradation with IncreasingKK

Following MoESD\([Huang et al\., 2025](https://arxiv.org/html/2609.22156#bib.bib8)\), the realized speedup can be written as the ratio of useful progress to the round’s total cost\. Decomposing the round cost into draft cost and target\-model verification time, and dropping the small term accounting for rejection sampling and post\-processing, we obtain the approximation

SpeedUp≈𝔼⁡\[Gr\]K​TDTT​\(0\)\+1η⁡\(K\),𝔼⁡\[Gr\]=1R​∑r=1RGr,\\begin\{gathered\}\\mathrm\{SpeedUp\}\\approx\\frac\{\\mathbb\{E\}\[G\_\{r\}\]\}\{K\\dfrac\{T\_\{D\}\}\{T\_\{T\}\(0\)\}\+\\dfrac\{1\}\{\\eta\(K\)\}\},\\\\\[4\.0pt\] \\mathbb\{E\}\[G\_\{r\}\]=\\frac\{1\}\{R\}\\sum\_\{r=1\}^\{R\}G\_\{r\},\\end\{gathered\}\(2\)whereGrG\_\{r\}is the number of useful output tokens in roundrr,η⁡\(K\)\\eta\(K\)is defined by Eq\. \([1](https://arxiv.org/html/2609.22156#S3.E1)\), andTDT\_\{D\}is the cost of a single draft position; the calibrated form ofTDT\_\{D\}is given in Section[3\.5](https://arxiv.org/html/2609.22156#S3.SS5)\.

As the budgetKKgrows, the numerator can increase through larger useful progress𝔼⁡\[Gr\]\\mathbb\{E\}\[G\_\{r\}\], but the denominator also grows: the termK​TD/TT​\(0\)KT\_\{D\}/T\_\{T\}\(0\)grows linearly inKK, and1/η⁡\(K\)1/\\eta\(K\)grows if target efficiencyη⁡\(K\)\\eta\(K\)declines\. For dense models, the decline ofη⁡\(K\)\\eta\(K\)with increasingKKis typically weaker; for MoE targets, it is observably linked to growth in the layer\-averaged number of unique expertsU¯r\\bar\{U\}\_\{r\}in the verification microbatch, which the verification cost model below unpacks \(Section[3\.3](https://arxiv.org/html/2609.22156#S3.SS3)\)\. Thus, the benefit of increasingKKis not guaranteed in advance: it depends on whether the gain in𝔼⁡\[Gr\]\\mathbb\{E\}\[G\_\{r\}\]offsets the growth of the denominator\.

### 3\.3Verification Cost Model

To describe the decline in target\-model efficiency as speculation length grows, we account for the hardware constraints of MoE inference\. In dense models, the cost of a verification step is often dominated mainly by the number of tokens verified in parallel\. In MoE models, a substantial component of the forward\-pass latencyTT\(B\)​\(K\)T\_\{T\}^\{\(B\)\}\(K\)can be memory traffic tied to expert weights\.

Each token in the speculative chain is routed to its own token\-specific subset of experts\. If the corresponding weights are not reused from fast on\-chip memory, reading them from global memory can increase latency\. The layer\-averaged number of unique experts across all positions of the pass is therefore treated as an observed cost signal, not as a causally identified volume of traffic\. Within a round,U¯r\\bar\{U\}\_\{r\}is computed over the same verification positions asTT\(B\)​\(K\)T\_\{T\}^\{\(B\)\}\(K\): unique experts are counted separately in each MoE layer, and the result is then averaged across layers\. In compact form, this intuition corresponds to the dependence

TT,r≈α​U¯r\+β,T\_\{T,r\}\\approx\\alpha\\bar\{U\}\_\{r\}\+\\beta,\(3\)but the actual predictive model below additionally accounts for context and the categorical form of the microbatch\.

Letmr=Kr\+1m\_\{r\}=K\_\{r\}\+1denote the number of positions computed by the target model in roundrr:KrK\_\{r\}draft positions plus one extra target\-model position\. The batch size remainsB=1B=1throughout\. Measured latency depends on three observed channels: the expert unionU¯r\\bar\{U\}\_\{r\}, the context lengthLctx,rL\_\{\\mathrm\{ctx\},r\}, and the shape of the verification microbatchmrm\_\{r\}\. The first channel is consistent with reading expert weights from HBM, the second captures KV\-cache and attention cost, and the third accounts for block size and CUDA\-graph mode\.

For the configuration under study, we use the diagnostic regression model

TT,r=β0​\(mr\)\+α⁡\(mr\)​U¯r\+c​Lctx,r\+εr,T\_\{T,r\}=\\beta\_\{0\}\(m\_\{r\}\)\+\\alpha\(m\_\{r\}\)\\bar\{U\}\_\{r\}\+cL\_\{\\mathrm\{ctx\},r\}\+\\varepsilon\_\{r\},\(4\)whereβ0​\(m\)\\beta\_\{0\}\(m\)andα⁡\(m\)\\alpha\(m\)are free categorical coefficients for eachm=K\+1m=K\+1,ccis the shared context coefficient, andεr\\varepsilon\_\{r\}is the residual\. The categorical form describes the observed plateaus ofα⁡\(m\)\\alpha\(m\)and the non\-monotonicity betweenm=4m=4andm=5m=5better than a shared linear dependence\. The breakpoint was chosen after inspecting the full sweep, so it is a descriptive part of this calibration rather than an independently confirmed boundary of a CUDA\-graph mode\.

Formally, letℐr\\mathcal\{I\}\_\{r\}contain allmrm\_\{r\}positions of the verification pass, including the extra target\-model position, and let𝒰ℓ,r\\mathcal\{U\}\_\{\\ell,r\}be the union of expert identifiers selected for these positions in MoE layerℓ\\ell\. The routing signal we use is

U¯r=1\|ℒMoE\|​∑ℓ∈ℒMoE\|𝒰ℓ,r\|\.\\bar\{U\}\_\{r\}=\\frac\{1\}\{\|\\mathcal\{L\}\_\{\\mathrm\{MoE\}\}\|\}\\sum\_\{\\ell\\in\\mathcal\{L\}\_\{\\mathrm\{MoE\}\}\}\|\\mathcal\{U\}\_\{\\ell,r\}\|\.\(5\)Thus,U¯r\\bar\{U\}\_\{r\}has the same full\-microbatch, per\-round granularity asTT,rT\_\{T,r\}\. Expert identifiers are extracted from a per\-round trace in the vLLM fork and synchronized with CUDA\-event timings; because of double buffering, covariates are shifted by one record\. The overhead of this instrumentation itself was not separately measured\.

### 3\.4Regression Visualization byKK

Figure[1](https://arxiv.org/html/2609.22156#S3.F1)shows the scatter of roundsTT,rT\_\{T,r\}againstU¯r\\bar\{U\}\_\{r\}, together with the fitted line of model \([4](https://arxiv.org/html/2609.22156#S3.E4)\), separately for eachK∈\{1,…,10\}K\\in\\\{1,\\dots,10\\\}\. Going fromm=4m=4tom=5m=5, the slopeα⁡\(m\)\\alpha\(m\)increases roughly threefold, consistent with the breakpoint discussed above; the in\-sampleR2R^\{2\}remains high across the whole range \(0\.9670\.967–0\.9990\.999\)\.

![Refer to caption](https://arxiv.org/html/2609.22156v1/fig_perK_scatter_paper.png)Figure 1:TT,rT\_\{T,r\}againstU¯r\\bar\{U\}\_\{r\}for eachKK, with the fitted line and coefficients of model \([4](https://arxiv.org/html/2609.22156#S3.E4)\)\. The extended robustness protocol is given in Appendix[A](https://arxiv.org/html/2609.22156#A1)\.
### 3\.5Quantitative Validation and Scope of Applicability

The model was estimated on925 752925\\,752rounds from a sweep overK∈\{1,…,10\}K\\in\\\{1,\\dots,10\\\}for Qwen3\-Coder\-30B\-A3B\-Instruct and EAGLE\-3 on a single NVIDIA A100 80GB\. Under GroupKFold grouped by generation,U¯r\\bar\{U\}\_\{r\}is the strongest single predictor of verification time\. In the per\-KKruns, the in\-sampleR2R^\{2\}of model \([4](https://arxiv.org/html/2609.22156#S3.E4)\) is0\.96690\.9669–0\.99910\.9991, and OOS MAPE is0\.260\.26–0\.59%0\.59\\%\. Its standalone predictive power is uneven across domains: OOSR2R^\{2\}is about0\.600\.60–0\.640\.64for humanities, writing, and roleplay, versus0\.870\.87for code\_math\.

For the round’s total cost, a separate draft\-time model is used

T^D​\(K\)=\(0\.459​K\+0\.073\)​ms,\\widehat\{T\}\_\{D\}\(K\)=\(0\.459K\+0\.073\)\\,\\mathrm\{ms\},\(6\)with CV MedAPE around0\.8%0\.8\\%; this model supplies the draft term in the offline analysis\. The coefficients describe a predictive, not a causal, relationship, and only for the fixed model, backend, GPU, CUDA\-graph mode, andB=1B=1; the full protocol is given in Appendix[A](https://arxiv.org/html/2609.22156#A1)\.

## 4The Sequence\-Conditioned Offline Oracle \(Methodology\)

As shown in the previous section, the verification\-step time in MoE models is estimated by a linear model that accounts for prefix length and the number of unique activated experts\. However, it is precisely the input\-dependent routing — how specific generated tokens activate different sets of experts — that creates high variance in verification cost\. Because of this instability, blindly fixing a speculative\-batch sizekkleads to suboptimal outcomes: the system inevitably risks spending compute on loading experts for tokens that will end up being rejected\.

To find the absolute theoretical speedup limit and understand how the ideal strategy should balance token\-acceptance probability against verification cost, we formulate the problem as a stochastic offline process over a known reference \(ground\-truth\) sequence\. This lets us build a hybrid Oracle whose behavior serves as a reference point for designing future adaptive online systems\.

### 4\.1Formulation as a Stochastic Shortest Path \(SSP\)

For a rigorous search of the ideal speculation budget, we formulate the decoding process as a Stochastic Shortest Path \(SSP\) problem\. The Markov Decision Process \(MDP\) formalization is as follows:

- •State space \(SS\):The system’s state is the current positioniiin the reference \(ground\-truth\) sequenceX1:N=\(x1,x2,…,xN\)X\_\{1:N\}=\(x\_\{1\},x\_\{2\},\\dots,x\_\{N\}\)of lengthNN\. PositionNNacts as an absorbing state signaling the end of generation\. Reaching this state terminates the process with zero residual cost \(d​p​\[N\]=0dp\[N\]=0\)\.
- •Action space \(AA\):While in stateii, the Oracle takes an action — it chooses the speculative\-batch size \(draft length\)k∈\{1​…​Lm​a​x\}k\\in\\\{1\\dots L\_\{max\}\\\}\.
- •Transitions:Given a chosen batch sizekk, the draft model produces a hypothesis, of which the target model may acceptaatokens,0≤a≤k0\\leq a\\leq k\. Because the speculative\-decoding algorithm always computes one bonus token from the target model’s logits, the system advances even under a complete hypothesis mismatch \(a=0a=0\)\. Thus, from positioniithe system deterministically transitions to the new statemin⁡\(i\+a\+1,N\)\\min\(i\+a\+1,N\)\.
- •Termination guarantee \(Proper Policy\):A well\-posed SSP problem requires the existence of at least one policy guaranteed to reach the absorbing state\. Since the number of accepted tokens is alwaysa≥0a\\geq 0, every step advances the position by at least11\. Hence, any strategy is guaranteed to reach the end of the sequenceNNin at mostNNsteps\. This property makes the transition graph a directed acyclic graph \(DAG\) and guarantees strict convergence of the Bellman problem\.

### 4\.2Counterfactual Simulation of Cost and Transitions

Solving the SSP problem and finding the minimum expected generation time requires defining two components for each positioni∈\{0,…,N−1\}i\\in\\\{0,\\dots,N\-1\\\}and speculation budgetk∈\{1,…,Lm​a​x\}k\\in\\\{1,\\dots,L\_\{max\}\\\}: the transition probabilitiesP⁡\(a∣k\)P\(a\\mid k\)and the verification cost estimateC​o​s​t​\(i,k,a\)Cost\(i,k,a\)\. Directly sampling trajectories online would cause a “collapse of the future”, since incorrectly generated tokens would drift generation away from the reference text\. To address this, we apply counterfactual simulation\.

##### Transition probability distributionP⁡\(a∣k\)P\(a\\mid k\)\.

For positioniiand batch lengthkk, the probability that the target model accepts exactlyaatokens \(0≤a≤k0\\leq a\\leq k\) is determined by the draft model’s autoregressive confidence on the reference sequenceX1:NX\_\{1:N\}\.

Letpj=Pd​r​a​f​t​\(xi\+1\+j∣X≤i\+j\)p\_\{j\}=P\_\{draft\}\(x\_\{i\+1\+j\}\\mid X\_\{\\leq i\+j\}\)denote the probability that the draft model correctly predicts the\(j\+1\)\(j\+1\)\-th token of the speculative window \(i\.e\., the reference tokenxi\+1\+jx\_\{i\+1\+j\}, given the prefixX≤i\+jX\_\{\\leq i\+j\}, wherej∈\{0,…,k−1\}j\\in\\\{0,\\dots,k\-1\\\}\)\. Then the probability of accepting the firstaatokens and rejecting the\(a\+1\)\(a\+1\)\-th token \(fora<ka<k\) is computed as:

P⁡\(a∣k\)=\(∏j=0a−1pj\)⋅\(1−pa\)P\(a\\mid k\)=\\left\(\\prod\_\{j=0\}^\{a\-1\}p\_\{j\}\\right\)\\cdot\(1\-p\_\{a\}\)\(7\)For the case of full hypothesis acceptance \(a=ka=k\), the probability isP⁡\(k∣k\)=∏j=0k−1pjP\(k\\mid k\)=\\prod\_\{j=0\}^\{k\-1\}p\_\{j\}\.

##### Counterfactual simulation of “junk tails” and cost estimation\.

The verification costC​o​s​t​\(i,k,a\)Cost\(i,k,a\)is determined not only by the number of accepted tokens but also by the structure of experts activated by the entire candidate chain of lengthkk\. To estimate this cost without disturbing the main stateii, we model the physics of a speculative error by sampling hypothetical suffixes \(“junk tails”\):

1. 1\.For eacha∈\{0,…,k\}a\\in\\\{0,\\dots,k\\\}, a candidate chainX~\(a\)\\tilde\{X\}^\{\(a\)\}of lengthkkis formed\. Its firstaatokens exactly match the ground truth \(xi\+1,…,xi\+ax\_\{i\+1\},\\dots,x\_\{i\+a\}\)\.
2. 2\.At positionaa, an error is artificially induced: the tokenx~a\\tilde\{x\}\_\{a\}is sampled from the draft model’s distribution with the reference token’s probabilityxi\+1\+ax\_\{i\+1\+a\}zeroed out\. The remainingk−a−1k\-a\-1tokens are sampled freely\.
3. 3\.The resulting suffix batchX~\(a\)\\tilde\{X\}^\{\(a\)\}is fed to the target model using the previously cached KV cache for prefixX≤iX\_\{\\leq i\}\. The number of unique activated expertsU⁡\(i,k,a\)U\(i,k,a\)is extracted from the MoE layers’ router logits\.
4. 4\.Converting the expert countU⁡\(i,k,a\)U\(i,k,a\)into the final step time is done through the cost modelC​o​s​t​\(i,k,a\)=td​r​a​f​t​\(k\)\+tt​a​r​g​e​t​\(U,Lp​r​e​f​i​x\)Cost\(i,k,a\)=t\_\{draft\}\(k\)\+t\_\{target\}\(U,L\_\{prefix\}\), whereLp​r​e​f​i​x=Lp​r​o​m​p​t\+iL\_\{prefix\}=L\_\{prompt\}\+i\.

The simulated suffixX~\(a\)\\tilde\{X\}^\{\(a\)\}is used solely to measure the expert countU⁡\(i,k,a\)U\(i,k,a\)and is then discarded\. For computational efficiency, the Oracle uses a single\-sample Monte Carlo estimate of the junk tail, relying on the local variance in activated experts averaging out over the length of the whole sequence\. In this way, the Oracle correctly accounts for the routing overhead of incorrectly predicted tokens while keeping the state space strictly on the linear ground\-truth axis\.

### 4\.3The Bellman Oracle and Speedup Bounds

Given the probability distributionP⁡\(a∣k\)\{P\(a\\mid k\)\}and the precomputed cost matrixC​o​s​t​\(i,k,a\)Cost\(i,k,a\), the minimum expected generation time for the remaining sequence \(from positioniito the endNN\) is given by the Bellman equation:

d​p​\[i\]=mink∈\{1​…​Lm​a​x\}\[∑a=0kP\(a∣k\)⋅\(Cost\(i,k,a\)\+dp\[min\(i\+a\+1,N\)\]\)\]\\begin\{split\}dp\[i\]&=\\min\_\{k\\in\\\{1\\dots L\_\{max\}\\\}\}\\Bigg\[\\sum\_\{a=0\}^\{k\}\{P\(a\\mid k\)\}\\cdot\\\\ &\\quad\\Big\(Cost\(i,k,a\)\+dp\[\\min\(i\+a\+1,N\)\]\\Big\)\\Bigg\]\\end\{split\}\(8\)
Because every speculative step deterministically advances the position by at least one token, the state graph is strictly acyclic \(a DAG\)\. This allows the final expectationd​p​\[0\]dp\[0\]to be computed in a single pass of standard backward induction from the absorbing stated​p​\[N\]=0dp\[N\]=0\. The array ofkkvalues that minimize the equation at each stepiiforms our Oracle’s optimal offline policy\.

It is important to stress the limits of this estimate’s applicability\. Because the counterfactual simulation \(Section 4\.2\) is rigidly tied to the reference sequenceX1:NX\_\{1:N\}and ignores alternative valid generation paths, the resulting valued​p​\[0\]dp\[0\]and its corresponding speedupS=C​o​s​tA​R/d​p​\[0\]S=Cost\_\{AR\}/dp\[0\]represent an efficiency limit only for reproducing this specific text\. Nevertheless, for the sequence\-preserving speculative\-decoding problem, this estimate provides a reliable analytical reference point \(an upper bound\) against which any online heuristic lacking access to future context can be compared\.

## 5Analysis of the Oracle’s Behavior

To empirically evaluate the developed Oracle, we used the Math Reasoning subset \(80 questions\) of the SpecBench benchmark\. Qwen3\-Coder\-30B\-A3B\-Instruct served as the target model, and SGLang\-EAGLE3\-Qwen3\-Coder\-30B\-A3B\-Instruct\-SpecForge served as the draft model\. All simulations and profiling measurements were carried out on a single NVIDIA A100 80GB GPU\.

### 5\.1The Limits of Speculation \(Quantitative Bounds\)

We first examine the algorithm’s macro\-behavior as the maximum allowed speculation budgetLm​a​xL\_\{max\}increases\.

Figure 2:Quantitative bounds of the MoE Stochastic Oracle under varying speculation budgetsLm​a​xL\_\{max\}\.AsLm​a​xL\_\{max\}increases, Theoretical Maximum Speedup \(Sm​a​xS\_\{max\}\), Meankk, and Mean Accepted Length exhibit diminishing returns and saturation\. The Oracle selectively capskkto avoid costly MoE expert activations on low\-probability draft tokens\.Figure[2](https://arxiv.org/html/2609.22156#S5.F2)shows Oracle Speedup, Avg Draft Length, and Avg Accepted Length as functions ofLm​a​xL\_\{max\}\. The main result of this experiment is a strong diminishing\-returns effect\. As the plot shows, beyondLm​a​x=6L\_\{max\}=6the curves essentially stop growing: the final speedup plateaus at roughly 2\.34x, and the average accepted\-batch length \(Avg Accepted Length\) saturates at≈2\.1\\approx 2\.1tokens\. Most importantly, the Oracle forcibly caps the average generated draft length \(Avg Draft Length\) at≈2\.8\\approx 2\.8, effectively ignoring the larger budgets available up toLm​a​x=12L\_\{max\}=12\.

This dynamic is explained by the very mechanics of the DP algorithm’s decision\-making\. AsLm​a​xL\_\{max\}grows, the Oracle gains the ability to use long speculative chains, yet it deliberately declines to use it\. The reason lies in the strict accounting of acceptance probability: the draft model’s predictive power inevitably declines for farther\-out tokens\. Because in the MoE architecture processing each rejected token carries a noticeable penalty from loading unique experts, the Oracle balances the falling success probabilityP⁡\(a∣k\)P\(a\\mid k\)against the rising cost in the matrixC​o​s​t​\(i,k,a\)Cost\(i,k,a\)\. As a result, the algorithm prefers to settle on shorter but more reliable chains, forcibly stabilizing the average value ofkkregardless of how large a budgetLm​a​xL\_\{max\}it is formally allowed\.

### 5\.2Delta Space Analysis and the Linear Boundary

To understand the logic by which the Oracle truncates speculative\-batch length, we examined the space of marginal decisions \(Delta Space\) at the local level\. For each generation step, the Oracle selects the optimal draft lengthko​p​tk\_\{opt\}\. All other possible lengthsk≠ko​p​tk\\neq k\_\{opt\}are treated as rejected alternatives\.

For each such alternative, we computed two marginal deltas relative to the Oracle’s choice:

- •Δ​𝔼​\[a\]=𝔼​\[a\]k−𝔼​\[a\]ko​p​t\\Delta\\mathbb\{E\}\[a\]=\\mathbb\{E\}\[a\]\_\{k\}\-\\mathbb\{E\}\[a\]\_\{k\_\{opt\}\}— the marginal difference in the expected number of accepted tokens;
- •Δ​𝔼​\[C​o​s​t\]=𝔼​\[C​o​s​t\]k−𝔼​\[C​o​s​t\]ko​p​t\\Delta\\mathbb\{E\}\[Cost\]=\\mathbb\{E\}\[Cost\]\_\{k\}\-\\mathbb\{E\}\[Cost\]\_\{k\_\{opt\}\}— the marginal difference in expected cost \(number of activated experts\)\.

![Refer to caption](https://arxiv.org/html/2609.22156v1/delta_space_final.png)Figure 3:Delta Space Analysis \(Δ​𝔼​\[Cost\]\\Delta\\mathbb\{E\}\[\\text\{Cost\}\]vs\.Δ​𝔼​\[a\]\\Delta\\mathbb\{E\}\[a\]\) of rejected draft candidates relative to the Oracle’s choice\(0,0\)\(0,0\)\.Quadrants II and IV are strictly empty, proving strict monotonicity\. Candidates in Quadrant I \(risk zone\) and Quadrant III \(caution zone\) form a linear boundary passing through the origin\.If the Oracle’s choice is placed at the origin\(0,0\)\(0,0\), the rejected candidates form a dense cloud of points around it \(see Figure[3](https://arxiv.org/html/2609.22156#S5.F3)\)\. Analysis of the distribution of these points reveals three fundamental behavioral patterns:

##### 1\. Structural monotonicity \(empty quadrants II and IV\)\.

The upper\-left \(Δ​C​o​s​t\>0,Δ​a<0\\Delta Cost\>0,\\Delta a<0\) and lower\-right \(Δ​C​o​s​t​<0,Δ​a\>​0\\Delta Cost<0,\\Delta a\>0\) quadrants of the plot are, as expected, completely empty\. This is a direct consequence of the algorithm’s construction and the cost model: generating additional tokens structurally requires additional computation steps\. Thus, the impossibility of increasing𝔼⁡\[a\]\\mathbb\{E\}\[a\]while decreasing𝔼⁡\[C​o​s​t\]\\mathbb\{E\}\[Cost\]serves as a naturalsanity checkon the correctness of the constructed space of marginal deltas\.

##### 2\. The risk and caution thresholds \(quadrants I and III\)\.

The upper\-right quadrant \(the risk zone\) concentrates batches that were too long: they promised more tokens \(Δ​a\>0\\Delta a\>0\) but were rejected for a disproportionately high expert overpayment \(Δ​C​o​s​t\>0\\Delta Cost\>0\)\. The lower\-left quadrant \(the caution zone\) contains chains that were too short: they saved time \(Δ​C​o​s​t<0\\Delta Cost<0\) but were rejected for a substantial loss of potential tokens\.

##### 3\. The linear boundary\.

The most important observation is that the edges of the rejected\-point clouds in quadrants I and III line up along a single line passing through the origin\. This line \(constructed from the 1st percentile of the ratioΔ​C​o​s​t/Δ​a\\Delta Cost/\\Delta a\) forms the boundary of the infeasible region\. No rejected point can physically cross this line, since otherwise the algorithm would have recognized it as the optimal decision\.

The existence of this constant linear boundary has substantial practical significance\. It demonstrates that the globally optimal search over the Bellman equation, which requires knowledge of the entire future text, reduces at the local level to maintaining a simple balance: the ratioΔ​𝔼​\[C​o​s​t\]Δ​𝔼​\[a\]\\frac\{\\Delta\\mathbb\{E\}\[Cost\]\}\{\\Delta\\mathbb\{E\}\[a\]\}\. This means that building effective online speculative\-decoding algorithms for MoE does not require heavy predictive models\. It suffices to estimate the marginal cost of adding the next token to the chain\. If this ratio breaks through the constant slope barrier, the algorithm should immediately halt draft generation \(early exit\) and proceed to verification\.

## 6Conclusion

In this work, we studied the problem of input\-dependent verification cost in speculative decoding for MoE models\. To analyze the trade\-off between token\-acceptance probability and expert\-activation cost, we developed a diagnostic offline method\. By formulating the problem as a stochastic shortest path \(SSP\) over a fixed reference sequence and applying counterfactual simulation of “junk tails”, we built a Sequence\-Conditioned Oracle that provides an upper\-bound speedup estimate for a given text\.

Analysis of the Oracle’s offline decisions yields two main conclusions:

- •Diminishing returns:As the available budgetLm​a​xL\_\{max\}grows, the final speedup saturates quickly\. The Oracle deliberately limits the average speculative\-batch length, since the routing overhead of low\-probability tokens begins to outweigh the benefit of their possible acceptance\.
- •A linear boundary in delta space:Examining the rejected marginal alternatives shows that the Oracle’s choice forms a strict boundary\. Locally optimal decisions obey a necessary condition: the ratio of marginal cost to the expected gain in accepted tokens \(Δ​𝔼​\[C​o​s​t\]Δ​𝔼​\[a\]\\frac\{\\Delta\\mathbb\{E\}\[Cost\]\}\{\\Delta\\mathbb\{E\}\[a\]\}\) must not exceed a constant threshold\.

The marginal structure we uncover demonstrates that decisions requiring knowledge of the entire sequence have a clear local interpretation\. This observation does not offer a ready\-made system, but it motivates the design of future online heuristics that could use the marginal\-cost ratio for early exit from speculative generation\.

## 7Limitations

This study makes a number of assumptions that bound the applicability of the proposed offline method:

- •Batch size and configuration\.The analysis is confined to theQwen3\-Coder\-30B\-A3B\-InstructandEAGLE\-3\(SpecForge\) pairing at batch sizeB=1B=1\. Generation atB\>1B\>1introduces inter\-sequence expert sharing, which fundamentally changes the structure of the overhead and requires a different cost model\.
- •Domain effect\.The evaluation was carried out on the Math Reasoning subset\. This domain is aligned with the draft model’s training distribution, which yields high baseline confidence \(pjp\_\{j\}\) and allows the MoE\-routing penalties to be isolated honestly\. While the diminishing\-returns effect itself is general, the exact budget saturation point \(Lm​a​xL\_\{max\}\) will vary for text with different entropy\.
- •Oracle approximations\.For computational efficiency, transition probabilities are estimated via draft confidence \(surrogate probability\), and the cost of rejected tokens in the matrixC​o​s​t​\(i,k,a\)Cost\(i,k,a\)is computed via a single\-shot Monte Carlo sampling of suffixes, which introduces local noise into the point estimates\.
- •No online policy\.This work establishes the existence of a linear boundary in delta space as a necessary condition for optimality\. However, designing and end\-to\-end evaluating an adaptive online heuristic \(early exit\) that exploits this rule is left for future work\.

## References

- Abramovich et al\. \(2026\)Talor Abramovich, Maor Ashkenazi, Izzy Putterman, Benjamin Chislett, Tiyasa Mitra, Bita Darvish Rouhani, Ran Zilberstein, and Yonatan Geifman\. 2026\.[SPEED\-Bench: A unified and diverse benchmark for speculative decoding](https://doi.org/10.48550/arXiv.2604.09557)\.*Preprint*, arXiv:2604\.09557\.
- Bang et al\. \(2026\)Jehyeon Bang, Eunyeong Cho, Ranggi Hwang, Jinha Chung, and Minsoo Rhu\. 2026\.[SpecMoE: A fast and efficient mixture\-of\-experts inference via self\-assisted speculative decoding](https://arxiv.org/abs/2604.10152)\.*Preprint*, arXiv:2604\.10152\.Extended version accepted at DAC 2026\.
- Cai et al\. \(2024\)Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D\. Lee, Deming Chen, and Tri Dao\. 2024\.[Medusa: Simple LLM inference acceleration framework with multiple decoding heads](https://proceedings.mlr.press/v235/cai24b.html)\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pages 5209–5235\. PMLR\.
- Chen et al\. \(2023\)Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean\-Baptiste Lespiau, Laurent Sifre, and John Jumper\. 2023\.[Accelerating large language model decoding with speculative sampling](https://arxiv.org/abs/2302.01318)\.*Preprint*, arXiv:2302\.01318\.
- Chen et al\. \(2025\)Liangkun Chen, Zijian Wen, Tian Wu, Xiaoxi Zhang, and Chuan Wu\. 2025\.[SP\-MoE: Speculative decoding and prefetching for accelerating MoE\-based model inference](https://arxiv.org/abs/2510.10302)\.*Preprint*, arXiv:2510\.10302\.
- Fedus et al\. \(2022\)William Fedus, Barret Zoph, and Noam Shazeer\. 2022\.[Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity](https://arxiv.org/abs/2101.03961)\.*Journal of Machine Learning Research*, 23\(120\):1–39\.
- Gautam et al\. \(2025\)Aayush Gautam, Susav Shrestha, and Narasimha Reddy\. 2025\.[Token\-driven GammaTune: Adaptive calibration for enhanced speculative decoding](https://arxiv.org/abs/2504.00030)\.*Preprint*, arXiv:2504\.00030\.
- Huang et al\. \(2025\)Zongle Huang, Lei Zhu, Zongyuan Zhan, Ting Hu, Weikai Mao, Xianzhi Yu, Yongpan Liu, and Tianyu Zhang\. 2025\.[MoESD: Unveil speculative decoding’s potential for accelerating sparse MoE](https://doi.org/10.48550/arXiv.2505.19645)\.*Preprint*, arXiv:2505\.19645\.Accepted as spotlight at NeurIPS 2025\.
- Lepikhin et al\. \(2020\)Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen\. 2020\.[GShard: Scaling giant models with conditional computation and automatic sharding](https://arxiv.org/abs/2006.16668)\.*arXiv preprint arXiv:2006\.16668*\.
- Leviathan et al\. \(2023\)Yaniv Leviathan, Matan Kalman, and Yossi Matias\. 2023\.[Fast inference from transformers via speculative decoding](https://proceedings.mlr.press/v202/leviathan23a.html)\.In*Proceedings of the 40th International Conference on Machine Learning*, volume 202 of*Proceedings of Machine Learning Research*, pages 19274–19286\. PMLR\.
- Li et al\. \(2024a\)Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang\. 2024a\.[EAGLE\-2: Faster inference of language models with dynamic draft trees](https://doi.org/10.48550/arXiv.2406.16858)\.*Preprint*, arXiv:2406\.16858\.
- Li et al\. \(2024b\)Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang\. 2024b\.[EAGLE: Speculative sampling requires rethinking feature uncertainty](https://proceedings.mlr.press/v235/li24bt.html)\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pages 28935–28948\. PMLR\.
- Li et al\. \(2025\)Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang\. 2025\.[EAGLE\-3: Scaling up inference acceleration of large language models via training\-time test](https://doi.org/10.48550/arXiv.2503.01840)\.*Preprint*, arXiv:2503\.01840\.
- Mamou et al\. \(2024\)Jonathan Mamou, Oren Pereg, Daniel Korat, Moshe Berchansky, Nadav Timor, Moshe Wasserblat, and Roy Schwartz\. 2024\.[Dynamic speculation lookahead accelerates speculative decoding of large language models](https://proceedings.mlr.press/v262/mamou24a.html)\.In*Proceedings of The 4th NeurIPS Efficient Natural Language and Speech Processing Workshop*, volume 262 of*Proceedings of Machine Learning Research*, pages 456–467\. PMLR\.
- McDanel et al\. \(2026\)Bradley McDanel, Steven Li, Sruthikesh Surineni, and Harshit Khaitan\. 2026\.[MoE\-Spec: Expert budgeting for efficient speculative decoding](https://arxiv.org/abs/2602.16052)\.*Preprint*, arXiv:2602\.16052\.
- Pan et al\. \(2026\)Lehan Pan, Ziyang Tao, Ruoyu Pang, Xiao Wang, Jianjun Zhao, and Yanyong Zhang\. 2026\.[Making every verified token count: Adaptive verification for MoE speculative decoding](https://arxiv.org/abs/2605.00342)\.*Preprint*, arXiv:2605\.00342\.
- Pope et al\. \(2023\)Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean\. 2023\.[Efficiently scaling transformer inference](https://proceedings.mlsys.org/paper_files/paper/2023/file/c4be71ab8d24cdfb45e3d06dbfca2780-Paper-mlsys2023.pdf)\.In*Proceedings of Machine Learning and Systems*, volume 5, pages 606–624\.
- Samarin et al\. \(2026\)Alexander Samarin, Sergei Krutikov, Anton Shevtsov, Sergei Skvortsov, Filipp Fisin, and Alexander Golubev\. 2026\.[LK losses: Direct acceptance rate optimization for speculative decoding](https://doi.org/10.48550/arXiv.2602.23881)\.*Preprint*, arXiv:2602\.23881\.
- Saxena et al\. \(2025\)Anish Saxena, Po\-An Tsai, Hritvik Taneja, Aamer Jaleel, and Moinuddin Qureshi\. 2025\.[Utility\-driven speculative decoding for mixture\-of\-experts](https://doi.org/10.48550/arXiv.2506.20675)\.*Preprint*, arXiv:2506\.20675\.
- Shazeer \(2019\)Noam Shazeer\. 2019\.[Fast transformer decoding: One write\-head is all you need](https://arxiv.org/abs/1911.02150)\.*arXiv preprint arXiv:1911\.02150*\.
- Shazeer et al\. \(2017\)Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean\. 2017\.[Outrageously large neural networks: The sparsely\-gated mixture\-of\-experts layer](https://arxiv.org/abs/1701.06538)\.In*International Conference on Learning Representations*\.
- Vaswani et al\. \(2017\)Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N\. Gomez, Lukasz Kaiser, and Illia Polosukhin\. 2017\.[Attention is all you need](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)\.In*Advances in Neural Information Processing Systems*, volume 30\. Curran Associates, Inc\.
- Wang et al\. \(2025a\)Jikai Wang, Jianchao Tan, Yuxuan Hu, Jiayu Qin, Yerui Sun, Yuchen Xie, Xunliang Cai, Juntao Li, and Min Zhang\. 2025a\.[Accelerate speculative decoding with sparse computation in verification](https://arxiv.org/abs/2512.21911)\.*Preprint*, arXiv:2512\.21911\.
- Wang et al\. \(2025b\)Wenfeng Wang, Jiacheng Liu, Xiaofeng Hou, Xinfeng Xia, Peng Tang, Mingxuan Zhang, Chao Li, and Minyi Guo\. 2025b\.[MoE\-SpeQ: Speculative quantized decoding with proactive expert prefetching and offloading for mixture\-of\-experts](https://arxiv.org/abs/2511.14102)\.*Preprint*, arXiv:2511\.14102\.
- Wang et al\. \(2026\)Xuliang Wang, Yuetao Chen, Maochan Zhen, Fang Liu, Xinzhou Zheng, Xingwu Liu, Hong Xu, and Ming Li\. 2026\.[PRISM: Parametrically refactor inference for speculative decoding draft models](https://proceedings.mlsys.org/paper_files/paper/2026/hash/414fd191b3246a19a55741b938380136-Abstract-Conference.html)\.In*Proceedings of Machine Learning and Systems*, volume 8\.
- Wang et al\. \(2025c\)Zhibin Wang, Zhonghui Zhang, Yuhang Zhou, Zibo Wang, Mo Zhou, Peng Jiang, Weilin Cai, Chengying Huan, Rong Gu, Sheng Zhong, and Chen Tian\. 2025c\.[Accelerating mixture\-of\-experts inference by hiding offloading latency with speculative decoding](https://arxiv.org/abs/2508.21706)\.*Preprint*, arXiv:2508\.21706\.
- Wu et al\. \(2025\)Zhaoxuan Wu, Zijian Zhou, Arun Verma, Alok Prakash, Daniela Rus, and Bryan Kian Hsiang Low\. 2025\.[TETRIS: Optimal draft token selection for batch speculative decoding](https://doi.org/10.48550/arXiv.2502.15197)\.*Preprint*, arXiv:2502\.15197\.
- Xie et al\. \(2026\)Jincheng Xie, Runheng Liu, Heyan Huang, Yawen Ling, Hanbin Dai, Yu Zheng, and Wen Hu\. 2026\.[Less experts, faster decoding: Cost\-aware speculative decoding for mixture\-of\-experts](https://doi.org/10.48550/arXiv.2607.12696)\.*Preprint*, arXiv:2607\.12696\.
- Zhang et al\. \(2024a\)Situo Zhang, Hankun Wang, Da Ma, Zichen Zhu, Lu Chen, Kunyao Lan, and Kai Yu\. 2024a\.[AdaEAGLE: Optimizing speculative decoding via explicit modeling of adaptive draft structures](https://doi.org/10.48550/arXiv.2412.18910)\.*Preprint*, arXiv:2412\.18910\.
- Zhang et al\. \(2024b\)Ziyin Zhang, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Rui Wang, and Zhaopeng Tu\. 2024b\.[Draft model knows when to stop: Self\-verification speculative decoding for long\-form generation](https://doi.org/10.48550/arXiv.2411.18462)\.*Preprint*, arXiv:2411\.18462\.
- Zhong et al\. \(2025\)Wei Zhong, Manasa Bharadwaj, Yixiao Wang, Yipeng Ji, and Chul Lee\. 2025\.[Cross\-attention speculative decoding](https://doi.org/10.48550/arXiv.2505.24544)\.*Preprint*, arXiv:2505\.24544\.

## Appendix AVerification Cost Factors and the Predictive Model forTTT\_\{T\}

### A\.1Research Question and Hypotheses

The central thesis of this analysis is that for MoE targets, verification cost can be a bottleneck of speculative decoding and is linked to the activation of unique experts\. Three testable hypotheses follow from it:

1. H1\.The routing signalU¯r\\bar\{U\}\_\{r\}— the number of unique experts touched by the verification pass — predicts round latencyTTT\_\{T\}more accurately than each of the five other factors considered\.
2. H2\.Within a fixed system regime, one additional unique expert is associated with an approximately constant increment in latency\.
3. H3\.As the budgetKKgrows, the predictive association ofU¯r\\bar\{U\}\_\{r\}withTTT\_\{T\}strengthens relative to the association with context lengthLL\.

What follows describes the experimental setup and data, candidate factors, the protocol and results of the factor analysis, the specification and fit of the final model, the budget as a model parameter, the draft\-time model, and robustness checks\.

### A\.2Experimental Setup

All runs were executed on a single NVIDIA A100 80GB GPU in bfloat16\. Stage timings were captured with CUDA events inside a vLLM fork with per\-round tracing; the full sweep overK∈\{1,…,10\}K\\in\\\{1,\\dots,10\\\}took about nine GPU\-hours\.

The unit of observation is the speculative round\. Each record contains the stage timingsTTT\_\{T\}andTDT\_\{D\}, the verification\-microbatch sizem=K\+1m=K\+1, the routing signalU¯r\\bar\{U\}\_\{r\}, the context lengthLL, KV\-cache occupancy, and the CUDA\-graph bucket\. Records from the prefill phase were excluded from estimation but were used to mark generation boundaries\. The generation identifier defined the clusters for robust standard errors and the grouping used in cross\-validation\. Rounds are not independent experimental units: they are nested within generations, and the same questions recur across differentKK\. GroupKFold limits train/test leakage within each analysis but does not eliminate dependence across theKK\-slices of a single question\.

### A\.3Measurement Conventions

The definitions ofKK,m=K\+1m=K\+1, useful progress,TDT\_\{D\}, andTTT\_\{T\}are given in the main text; here we fix two boundaries of the recorded timings\. A separate quantityTreject,rT\_\{\\mathrm\{reject\},r\}would cover rejection sampling and post\-processing, which are not included in the recordedTD,rT\_\{D,r\}andTT,rT\_\{T,r\}; there is no separate measurement of this overhead in the trace\. Rejected positions are nonetheless part of the target model’s microbatch and are therefore counted inTT,rT\_\{T,r\}andU¯r\\bar\{U\}\_\{r\}\.

The extra target\-model position increases useful progress only when it becomes an emitted output token\. If this position is not emitted because the sequence ends, its computation remains part ofTT,rT\_\{T,r\}but does not increaseGrG\_\{r\}\.

Table A1:Dataset summary by run\. MedianLLranges between 586 and 606 tokens\. The non\-monotonicity ofTTT\_\{T\}atK=3→4K=3\\rightarrow 4coincides with a change in the estimatedβ0\\beta\_\{0\}; without a randomized repeat sweep it cannot be causally attributed to a CUDA\-graph mode\.
### A\.4Candidate Factors

Six measurable candidate factors were considered; each corresponds to a physical hypothesis about its mechanism of association withTTT\_\{T\}\. The speculation budgetKKis the only operator\-controlled quantity\. In the analysis it is represented by the number of positions in the verification pass,m=K\+1m=K\+1; within a single run,mmis deterministically set by the budget, so a separate factorKKis redundant\.

#### A\.4\.1Unique Expert Counting

LetGℓG\_\{\\ell\}denote the router’s scoring function in layerℓ\\ell, letkek\_\{e\}denote the number of experts selected per token, and letℒMoE\\mathcal\{L\}\_\{\\mathrm\{MoE\}\}denote the set of MoE layers of the target model\. For batch elementbb, verification positiontt, and token representationxℓ,b,tx\_\{\\ell,b,t\}, the local set of selected experts is

ℰℓ,b,t=TopK⁡\(Gℓ​\(xℓ,b,t\),ke\)\.\\mathcal\{E\}\_\{\\ell,b,t\}=\\operatorname\{TopK\}\(G\_\{\\ell\}\(x\_\{\\ell,b,t\}\),k\_\{e\}\)\.\(A1\)Herekek\_\{e\}differs fromKK: the former sets the number of experts per token, the latter the number of draft positions\.

For roundrr, the setℐr\\mathcal\{I\}\_\{r\}contains all pairs\(b,t\)\(b,t\)in the verification microbatch, including the extra target\-model position\. The union of experts in layerℓ\\ellequals

𝒰ℓ,r=⋃\(b,t\)∈ℐrℰℓ,b,t\.\\mathcal\{U\}\_\{\\ell,r\}=\\bigcup\_\{\(b,t\)\\in\\mathcal\{I\}\_\{r\}\}\\mathcal\{E\}\_\{\\ell,b,t\}\.\(A2\)The signalU¯r\\bar\{U\}\_\{r\}is defined as the mean of\|𝒰ℓ,r\|\|\\mathcal\{U\}\_\{\\ell,r\}\|across MoE layers, per Eq\. \([5](https://arxiv.org/html/2609.22156#S3.E5)\)\.

### A\.5Factor Importance

Factor importance was determined by its predictive power on held\-out generations\. Rounds were split into five folds grouped by generation: all rounds of a given response fell into the same fold\. For each fold, an OLS model was fit on the remaining four, after which out\-of\-sampleR2R^\{2\}and RMSE were computed on the held\-out fold\. Standalone factor strength was given by the across\-fold mean with a 95%tt\-interval at four degrees of freedom\. The incremental contribution beyondU¯r\\bar\{U\}\_\{r\}was measured by the paired RMSE reduction of the modelU¯r\+\\bar\{U\}\_\{r\}\+factor relative to theU¯r\\bar\{U\}\_\{r\}\-only model\.

For correlated factors, it is precisely the paired incremental estimate that is meaningful: a factor may predict well on its own by moving together withU¯r\\bar\{U\}\_\{r\}while carrying no independent information\.

Table A2:Descriptive predictive importance of factors on the pool of925 752925\\,752rounds; intervals are 95% CIs across folds\.U¯r\\bar\{U\}\_\{r\}is the strongest single predictor, andLLgives the largest additional error reduction on top of it\.Three observations follow from Table[A2](https://arxiv.org/html/2609.22156#A1.T2)\. First,U¯r\\bar\{U\}\_\{r\}is the strongest of the single predictors considered, which supports the predictive version of H1 in the studied environment\. Second, the budget predicts latency moderately on its own,R2≈0\.52R^\{2\}\\approx 0\.52, but on top ofU¯r\\bar\{U\}\_\{r\}it reduces error by only0\.014±0\.0030\.014\\pm 0\.003ms\. The budget’s predictive association with cost is mostly absorbed by its covariation withU¯r\\bar\{U\}\_\{r\}; the residual differences show up inα⁡\(m\)\\alpha\(m\)andβ0​\(m\)\\beta\_\{0\}\(m\)\. Third,LLis weak on its own but is almost uncorrelated withU¯r\\bar\{U\}\_\{r\}and gives an additional RMSE reduction of0\.146±0\.0080\.146\\pm 0\.008ms, which is consistent with, but does not identify, a separate KV\-cache cost channel\.LLand KV usage have nearly identical standalone accuracy and may encode overlapping properties of context length and memory state\. A joint VIF diagnostic or a separate ablation of their redundancy was not performed; the choice ofLLis based on interpretability and predictive contribution, not on identifying a unique causal channel\.

The standardized association ofTTT\_\{T\}withU¯r\\bar\{U\}\_\{r\}within a single run grows monotonically from0\.20​σ0\.20\\sigmaatK=1K=1to0\.93​σ0\.93\\sigmaatK=10K=10, while the association withLLdeclines from0\.96​σ0\.96\\sigmato0\.31​σ0\.31\\sigma; the width of the 95% CIs does not exceed±0\.004\\pm 0\.004\. At small microbatches the standardized association ofTTT\_\{T\}is stronger withLL; at large microbatches it is stronger withU¯r\\bar\{U\}\_\{r\}; this is descriptively consistent with H3\.

#### A\.5\.1Breakdown by Dataset Category

Within each category, the same GroupKFold\-by\-generation protocol was applied, with 95% intervals across folds\. The comparisons across the fourteen categories are descriptive; the intervals are not adjusted for multiplicity and are not used as a family of confirmatory tests\.

Table A3:Factor importance by category, sorted by the OOSR2R^\{2\}of theU¯r\\bar\{U\}\_\{r\}\-only model\. The summary row corresponds to the full pool\.Figure A1:Out\-of\-sampleR2R^\{2\}for predictingTTT\_\{T\}by category:U¯r\\bar\{U\}\_\{r\}alone versusU¯r\+L\\bar\{U\}\_\{r\}\+L\.In the descriptive comparison,U¯r\\bar\{U\}\_\{r\}remains the leading single predictor across all fourteen categories, while the additional contribution ofLLis uneven and in places has wide intervals\. The share of variance explained byU¯r\\bar\{U\}\_\{r\}alone ranges from 0\.60 for humanities and writing to 0\.87 for code\_math\. In structured domainsU¯r\\bar\{U\}\_\{r\}explains a larger share of variation, whereas in free\-form textLLprovides a markedly larger additional predictive contribution\. The wide intervals for small subcategories reflect their small sample size\.

### A\.6Predictive Model forTTT\_\{T\}

We use the specification from Eq\. \([4](https://arxiv.org/html/2609.22156#S3.E4)\): the budget enters categorically through freeβ0​\(m\)\\beta\_\{0\}\(m\)andα⁡\(m\)\\alpha\(m\)for eachm=K\+1m=K\+1, while the context coefficientccis shared\.

The choice of parameters is motivated as follows\.U¯r\\bar\{U\}\_\{r\}is an observed cost predictor consistent with reading expert weights from HBM\. A rough bandwidth\-based estimate,4848layers of9\.49\.4MB each, or about0\.450\.45GB at22TB/s, gives about0\.230\.23ms, matching the order of magnitude of the estimatedα\\alphaatm≥5m\\geq 5\. ContextLLgives the largest additional reduction in predictive error on top ofU¯r\\bar\{U\}\_\{r\}and is consistent with the KV\-cache read channel\. The categorical form describes the two observed ranges ofα⁡\(m\)\\alpha\(m\)and the shift inβ0​\(m\)\\beta\_\{0\}\(m\)betweenm=4m=4andm=5m=5\. The split point was chosen after inspecting the full dataset; the subsequent GroupKFold evaluation estimates coefficients under an already\-fixed form and is not a prospective test of the changepoint or its cause\.

Parameters were estimated by OLS separately on the subsample of eachKK, under the specificationTT∼U¯r\+LT\_\{T\}\\sim\\bar\{U\}\_\{r\}\+L\. Standard errors were clustered by generation; out\-of\-sample metrics were computed via GroupKFold\(5\) with the same grouping\.

![Refer to caption](https://arxiv.org/html/2609.22156v1/fig_perK_scatter_paper.png)Figure A2:Round scatter and fitted line for each of the ten values ofKKon shared axes\. AsKKgrows, the cloud shifts to the right, and at them=4/5m=4/5boundary the slope increases roughly threefold\.Table A4:OLS estimates of the modelTT=β0\+α​U¯r\+c​LT\_\{T\}=\\beta\_\{0\}\+\\alpha\\bar\{U\}\_\{r\}\+cLby budgetKK\. Parentheses give clustered standard errors; each run has 641 cluster\-generations\. Under these standard errors, all reported coefficients havep<0\.001p<0\.001; because of the large number of nested rounds, the emphasis is on effect sizes, intervals, and OOS error\. OOS MAPE is measured on held\-out generation groups across five folds\.In the observed sweep,α⁡\(m\)\\alpha\(m\)is0\.0780\.078–0\.0840\.084ms form≤4m\\leq 4and0\.2640\.264–0\.2680\.268ms form≥5m\\geq 5, i\.e\., the ratio of characteristic levels is about3\.23\.2\. In the small\-mmrange,β0\\beta\_\{0\}grows from 7\.3 to 10\.5 ms, after which it shifts to 4\.7 ms between adjacent configurations\. These differences describe the runs but, without interleaved repetition, do not identify a hardware cause for the boundary\.

The coefficientα\\alphais interpreted as the predictive increment associated with one additional unique expert;ccas the increment associated with a context token, about 1\.1 ms per 1000 tokens;β0\\beta\_\{0\}absorbs the baseline launch cost of dense sublayers and the router\. This interpretation is consistent with the measurements, but the regression itself does not establish causality\.

#### A\.6\.1Model by Category

The same regression was estimated within each of the fourteen categories under the specificationTT∼U¯r×C⁡\(K\)\+LT\_\{T\}\\sim\\bar\{U\}\_\{r\}\\times C\(K\)\+L\. Table[A5](https://arxiv.org/html/2609.22156#A1.T5)reports the slope atK=8K=8and the shared context coefficient\.

Table A5:Model by category, sorted byα\\alpha\. Parentheses give clustered standard errors\. Coefficients are close across all domains:α∈\[0\.262,0\.266\]\\alpha\\in\[0\.262,0\.266\]ms/expert andc∈\[1\.05,1\.19\]×10−3c\\in\[1\.05,1\.19\]\\times 10^\{\-3\}ms/token\.Figure A3:Model coefficients by category: the slopeα\\alphaatK=8K=8with 95% CI, and the error on held\-out generations\. The dashed line shows the estimate on the full pool\.Figure A4:Six dimensionless quality metrics by category:1−R21\-R^\{2\}, MedAPE, MAPE, nMAE, nRMSE, and P95 APE\. Typical error is 0\.3–0\.4%, and the 95th percentile does not exceed 1%\.

### A\.7BudgetKKas a Model Parameter

The categorical specification was compared against a shared linear model in which the budget enters as an ordinary coefficient:

TT=β0\+α​U¯r\+c​L\+γ​K\+ε\.T\_\{T\}=\\beta\_\{0\}\+\\alpha\\bar\{U\}\_\{r\}\+cL\+\\gamma K\+\\varepsilon\.\(A3\)The model was fit on the full pool of925 752925\\,752rounds; standard errors were clustered by generation, and out\-of\-sample metrics were computed via GroupKFold\(5\)\. The estimated coefficients areα=0\.2528​\(0\.0014\)\\alpha=0\.2528\\,\(0\.0014\)ms/expert,c=1\.09​\(0\.02\)×10−3c=1\.09\\,\(0\.02\)\\times 10^\{\-3\}ms/token,γ=−0\.081​\(0\.004\)\\gamma=\-0\.081\\,\(0\.004\)ms per unit ofKK, andβ0=5\.971​\(0\.035\)\\beta\_\{0\}=5\.971\\,\(0\.035\)ms\. Under the clustered standard errors used, all four coefficients havep<0\.001p<0\.001\. Because of the large number of nested rounds, the repetition of questions acrossKK, and the choice of form after inspecting the sweep, these values describe a specific fit and are not used as causal or independent confirmation of the model\.

Table A6:Budget as a linear coefficient versus the categorical form on the full pool; CV uses GroupKFold\(5\) by generation\.The linear coefficient onKKraisesR2R^\{2\}from 0\.856 to 0\.861 and reduces CV RMSE by0\.011±0\.0030\.011\\pm 0\.003ms, or 1\.6%\. The CV MAPE of the categorical form is roughly twelve times lower than that of theU¯r\+L\\bar\{U\}\_\{r\}\+Lmodel\.

![Refer to caption](https://arxiv.org/html/2609.22156v1/fig_kmodel_scatter_paper.png)Figure A5:The pooled model with a linear coefficient onKK, at the medianLLof each run\. The red line shows the pooled model; the gray dashed lines show the separate per\-KKfits\. The pooled line reproduces the large\-mmregime but is systematically off atm≤4m\\leq 4\.The pooled line tracks the large\-regime clouds closely: MAPE is 0\.6–4\.1% atK≥5K\\geq 5\. In the small regime, error reaches 7\.2% atK=1K=1and 12\.7% atK=3K=3, since the pooled slope is dominated by the larger mass of large\-regime rounds\.

The standardized contribution of the budget isβK∗=−0\.13\\beta^\{\*\}\_\{K\}=\-0\.13, versusβU¯∗=0\.99\\beta^\{\*\}\_\{\\bar\{U\}\}=0\.99andβL∗=0\.26\\beta^\{\*\}\_\{L\}=0\.26\. The negativeγ\\gammais a conditional coefficient arising from mixing regimes and is not interpreted as the marginal price of a position\.

A leave\-one\-KK\-out evaluation of the pooled form gives 5\.2% MAPE on average, versus 0\.26–0\.59% within a knownKK, with peaks of 12\.7% atK=1K=1and 15\.1% atK=3K=3\. Other smooth parameterizations examined give 6\.5–7\.4%, but their choice and breakpoint were not tuned within each training fold, so the comparison is exploratory rather than an independent validation of the categorical form\. Interpolating coefficients within the large regime achieves 0\.26–0\.65% error, but transferring across them=4/5m=4/5boundary gives 8\.8–15\.5% error\. For the configuration studied, interpolation within the large regime requires fewer anchor values ofKK, whereas transfer across the observed boundary yields markedly larger error; this observation should not be transferred to another environment without recalibration\.

### A\.8Draft Time Model

Draft time is modeled separately:

TD=a​K\+b\+εD\.T\_\{D\}=aK\+b\+\\varepsilon\_\{D\}\.\(A4\)The OLS model was fit on all925 752925\\,752rounds, with clustered standard errors and GroupKFold\(5\) by generation\. The estimates area=0\.459​\(0\.001\)a=0\.459\\,\(0\.001\)ms per step andb=0\.073​\(0\.003\)b=0\.073\\,\(0\.003\)ms\. Under the clustered standard errors used, both coefficients havep<0\.001p<0\.001; given the largenn, the coefficient magnitudes and the held\-out\-generation error are more substantively important\. MeanTDT\_\{D\}by budget deviates from the line by 0\.5% on average, and the CV MedAPE of a typical round is 0\.8%\. Per\-roundR2=0\.61R^\{2\}=0\.61; the residual distribution has a heavy tail: 0\.27% of rounds have an absolute residual greater than 1 ms, and 0\.3% of records contain a zero timing\.

![Refer to caption](https://arxiv.org/html/2609.22156v1/fig_draft_line_paper.png)Figure A6:Draft time versus budget and the modelTD=0\.459​K\+0\.073T\_\{D\}=0\.459K\+0\.073ms\. Each point corresponds to a round; points on the zero line represent records with a missing timing\.The draft head is dense, so no routing term enters the model\. The mean dependence ofTDT\_\{D\}onKKis approximately linear; the estimated slope is about 0\.46 ms per step over the rangeK=1,…,10K=1,\\dots,10\. In this sweep,TDT\_\{D\}shows no visually apparent discreteness comparable to the change inTTT\_\{T\}at them=4/5m=4/5boundary; a separate changepoint test for draft time was not performed\.

### A\.9Model Robustness

A ladder of nested specifications shows that addingLLtoU¯r\\bar\{U\}\_\{r\}alone reduces CV MAPE from 5\.6% to 4\.4% and raisesR2R^\{2\}from 0\.788 to 0\.856\. Linearmm, the interactionsm​LmLandm​U¯rm\\bar\{U\}\_\{r\}, andEmaxE\_\{\\max\}do not improve on a CV MAPE of 4\.3%\. Among the specifications examined, only the categorical budget form reduces CV MAPE to 0\.37% atR2=0\.999R^\{2\}=0\.999\.

Estimation robustness was checked on theK=8K=8subsample in five ways: clustered bootstrap over 999 generation resamples, Huber regression, trimming 1% of observations by absolute residual, separate fits on run halves, and leave\-one\-category\-out across fourteen refits\. The estimate ofα\\alphastays within 0\.2642–0\.2650 ms/expert, andccwithin 1\.100–1\.105×10−3\\times 10^\{\-3\}ms/token\.

Table A7:Model coefficients under robustness checks on theK=8K=8subsample\. For OLS and bootstrap, 95% intervals are given; for leave\-one\-category\-out, the range across fourteen refits\.Figure A7:Estimates ofα\\alphaatK=8K=8andccunder robustness checks\. The dashed line marks the baseline OLS estimate; the entire spread stays within a fraction of a percent\.The context coefficient is close across the separate per\-KKruns:c∈\[1\.098,1\.107\]×10−3c\\in\[1\.098,1\.107\]\\times 10^\{\-3\}, with variation below 0\.9%\. Mean residuals binned byLL, KV\-cache occupancy, and position within the generation show no trends\. Residual diagnostics indicate heteroscedasticity; the Breusch–Pagan test givesp≈10−28p\\approx 10^\{\-28\}\. Atn≈9×105n\\approx 9\\times 10^\{5\}, this value is treated as a diagnostic of residual shape rather than independent confirmation of a substantive effect; uncertainty is therefore assessed via clustered standard errors, with the emphasis on effect sizes, intervals, and OOS error\. Atn≈9×105n\\approx 9\\times 10^\{5\}, normality of residuals is not required, and the low pairwise correlationcorr⁡\(U¯r,L\)=0\.03\\operatorname\{corr\}\(\\bar\{U\}\_\{r\},L\)=0\.03does not indicate pronounced collinearity between these two covariates\.

All coefficients and regime boundaries pertain only to the model, draft head, GPU, backend, CUDA\-graph configuration, andB=1B=1studied\. A single seed and one full run perKKwere used; run order, cross\-session variance, and the causal status of them=4/5m=4/5boundary were not tested with a randomized or reversed sweep\. The overhead of per\-round tracing and double buffering was not measured separately\. Cross\-hardware, cross\-system, and batched transferability were not tested; the model requires recalibration when the environment changes or atB\>1B\>1\.

## Appendix BLinguistic Boundaries and Draft\-Length Selection

##### Brief descriptive summary\.

The offline policy balances a surrogate match probability of sequential matching against an estimated computational cost, rather than simply maximizing the length of the matched prefix\. In this trace, regular fragments tend to receive a long draft, while a script change tends to receive a short one\. In four purposefully selected counterfactual cases, the surrogate match probability of the added match is 84\.18–98\.45%, yet the full objective chooses a shorter boundary; after removing the expert term, the optimum shifts by one token in all four cases\. These cases were selected for exhibiting a shift and do not estimate the frequency of the effect\. Script change is also associated with the appearance of new experts in the code\-domain\-mixed subsample, whereas the first mismatch with the reference is not accompanied by a separate spike\.

The premise is that a speculative draft is more useful where the next several tokens form a predictable continuation\. This appendix tests whether such regularity is reflected in the Oracle’s decisions: whether it chooses a longer draft within stable fragments and shortens it near linguistic boundaries\.

The analysis covers 39 responses and9 5549\\,554decisions nested within them, atLm​a​x=8L\_\{max\}=8\. Decisions within a single response are serially dependent; aggregated intervals are therefore computed with a clustered bootstrap at the response level\. At each position, the offline policy chooses a lengthk⋆k^\{\\star\}from one to eight tokens that minimizes the conditional expected cost\. This objective accounts for the probability of a reference\-matching prefix and the computation of the draft and target models, including the activation of new experts\.k⋆k^\{\\star\}is the planned draft length, not the actual length of the matched prefix\. In the local examples,amodea\_\{\\mathrm\{mode\}\}denotes the modal advancement length along the reconstructed trajectory\. The number of new experts shows how many experts, not previously used within a given continuation, the target MoE model brings in; the value is averaged across layers\.

### B\.1What Is Tested

Two related but distinct questions are tested\.

1. 1\.Draft length\.If a continuation is linguistically regular — for example, it repeats a code structure or completes an already\-started construct — the probability of matching several tokens with the reference should be higher\. Length is chosen by the full Bellman equation, which includes the current cost and the continuation value; matching probability alone is not sufficient\. Cases where the surrogate match probability and the cost point to different boundaries are therefore examined separately\.
2. 2\.Expert routing\.If a transition between linguistic regimes requires a different set of computations, more new experts should appear at that position\. A stronger version of the hypothesis predicts the same kind of spike at the first position of mismatch with the reference\.

For the first question, we compare code formatting and structure, ordinary text, sentence boundaries, transitions between code and text, and a switch from Latin to Cyrillic script\. We additionally examine a series of cases with a high probability of a matching prefix but a shorter cost\-optimal length\. For the second question, we separately compare script change and the first draft mismatch\. This separation matters: a linguistic boundary, a mismatch with the reference, the surrogate match probability, and a change in the expert set are not the same event\.

Each subsequent token is assigned to one local context: formatting, vocabulary, or code structure; vocabulary or a prose boundary; markup and formulas; a mode change; a script change\. For the selected examples, we additionally distinguish the start of a function, control flow, a comment, and a word continuation\. These labels describe the observed text fragment and are used only to compare groups\.

##### Origin of the examples\.

Q1–Q6 correspond to\(q​i​d,p​o​s​i​t​i​o​n,c​a​t​e​g​o​r​y\)\(qid,\\,position,\\,category\):\(122,574,c​o​d​i​n​g\)\(122,574,coding\),\(66,408,c​o​d​e​\_​m​a​t​h\)\(66,408,code\\\_math\),\(20,492,c​o​d​e​\_​m​a​t​h\)\(20,492,code\\\_math\),\(7,38,c​o​d​e​\_​m​a​t​h\)\(7,38,code\\\_math\),\(7,420,c​o​d​e​\_​m​a​t​h\)\(7,420,code\\\_math\), and\(157,26,h​u​m​a​n​i​t​i​e​s\)\(157,26,humanities\)\. A1–A12 correspond to\(q​i​d,p​o​s​i​t​i​o​n\)\(qid,position\):\(48,66\)\(48,66\),\(70,179\)\(70,179\),\(128,9\)\(128,9\),\(135,73\)\(135,73\),\(152,355\)\(152,355\),\(121,38\)\(121,38\),\(126,20\)\(126,20\),\(154,404\)\(154,404\),\(8,100\)\(8,100\),\(127,734\)\(127,734\),\(125,516\)\(125,516\), and\(52,332\)\(52,332\); their categories in order are: code\_math, code\_math, coding, extraction, humanities, coding, coding, humanities, code\_math, coding, coding, and code\_math\.

### B\.2Local Examples

Individual decisions show what phenomena hide behind the general “boundary” category\. We compare the continuations of the target and draft models, the chosen cut pointk⋆k^\{\\star\}, the first mismatch, and the modal advancement lengthamodea\_\{\\mathrm\{mode\}\}\. In the panels, the target model’s continuation is shown on top and the draft model’s below; matches, the first mismatch, and positions past the chosen boundary are distinguished by color, and lines markk⋆k^\{\\star\}andamodea\_\{\\mathrm\{mode\}\}\.

#### B\.2\.1Predictable Continuations and Major Boundaries

Figure B1:A regular fragment and two major boundaries\. In the examples shown, a regular pattern coincides with a longk⋆k^\{\\star\}, while a structural or language transition coincides with the stopping point\.In Q1, all eight tokens of a repeating pattern match; in this example the policy also chooses the maximumk⋆=8k^\{\\star\}=8\. In Q2, the matching continuation ends right before the start of a new function, and in Q4 right before a transition from code to a Russian\-language comment\. In both cases, the first mismatch lies past the chosen boundary\. In the examples shown, a regular pattern coincides with a longk⋆k^\{\\star\}, while a structural or language transition coincides with the stopping point\.

#### B\.2\.2Low\-Cost Mismatches

Figure B2:Three limitations of the simple boundary hypothesis\. A token mismatch by itself does not imply a sharp change in routing, and the general label “linguistic boundary” does not determine the cost of a transition\.In Q3, the draft proposesifinstead ofreturn; both continuations are syntactically valid, and the number of new experts stays small\. In Q5, the mismatch occurs inside a Cyrillic word and is partly determined by tokenization\. In Q6, replacingclearlywithwellis grammatical and is accompanied by only 1\.37 new experts per layer\. Hence, a token mismatch by itself does not imply a sharp change in routing, and the general label “linguistic boundary” does not determine the cost of a transition\.

### B\.3Does the Pattern Generalize Across the Full Trace?

The local examples explain individual decisions but do not show how often each effect occurs\. A comparison across all decisions tests the first hypothesis: whether the chosen lengthk⋆k^\{\\star\}differs across linguistic contexts\.

Figure B3:Context\-wise distributions of the chosen length\. On the left, the shares ofk⋆=1k^\{\\star\}=1andk⋆≥4k^\{\\star\}\\geq 4; on the right, the meank⋆k^\{\\star\}by boundary type\. Intervals are a 95% clustered bootstrap at the response level and reflect between\-response variability\.The strongest contrast is observed between code formatting and script change\. For code formatting, the share ofk⋆≥4k^\{\\star\}\\geq 4is 0\.718\[0\.614;0\.818\]\[0\.614;0\.818\]\(455455decisions,2222responses\); for prose vocabulary it is 0\.193\[0\.119;0\.276\]\[0\.119;0\.276\]\(4383/304383/30\); and for script change it is 0\.045\[0\.010;0\.100\]\[0\.010;0\.100\]\(68/868/8\)\. At script change, the share ofk⋆=1k^\{\\star\}=1is 0\.937\[0\.867;0\.990\]\[0\.867;0\.990\]\.

However, not every boundary leads to a short draft\. At transitions between code and ordinary text, the descriptive share of decisions withk⋆≥4k^\{\\star\}\\geq 4is 0\.697\. Return and control\-flow keywords are also associated with a longer, rather than shorter, continuation\.

In the studied sample, we therefore observe a descriptive association between regular formatting and a long draft, and between script change and a short one\. But the general category “linguistic boundary” is not sufficient by itself: the specific type of transition matters\.

### B\.4When High Match Probability Does Not Imply a Long Draft

The aggregated analysis shows where the policy chooses a long draft, but it does not separate two possible reasons for stopping: a drop in match probability and a rise in computational cost\. Below we present four cases in which the adjacent longer sequence remains linguistically coherent and has a high surrogate match probability\.

For each case, we first consider the decision of the full objective function\. Then the coefficient on expert cost is zeroed out,α=0\\alpha=0, and the entire dynamic\-programming problem is solved again\. The remaining terms are kept, and the future cost is recomputed for the new policy\. The counterfactual ablation recomputes the policy value in the four selected cases and thereby tests the contribution of the expert term at the level of the Oracle’s objective function\. It is not a causal test of the hardware mechanism and does not estimate the frequency of the effect\.

Figure B4:Four cases where expert cost shifts the boundary\. In all cases, the full objective choosesk⋆k^\{\\star\}, while without expert cost the neighboring lengthk⋆\+1k^\{\\star\}\+1becomes cheaper\. The local increase in new experts serves as a diagnostic, not a standalone causal test\.C1 completes a Python condition and moves into the branch body; C2 completes an HTTP call and begins an assignment\. C3 passes through ameta viewportconstruct, and C4 completes a list and moves to a new indented line\. These descriptions capture the linguistic context of four purposefully selected examples but do not turn them into a frequency\-representative sample\.

The four cases were selected for exhibiting a shift after ablation and span the boundaries4→54\\rightarrow 5,6→76\\rightarrow 7, and7→87\\rightarrow 8\. The surrogate match probability of the added token ranges from 84\.18% to 98\.45%, and all four differencesQ0​\(k⋆\+1\)−Q0​\(k⋆\)Q^\{0\}\(k^\{\\star\}\+1\)\-Q^\{0\}\(k^\{\\star\}\)are negative\. Hence, the sum of the remaining terms favors a longer draft, and the expert term shifts the resulting boundary\.

The saved trace does not contain the full cost matrix, the value function, or all theQ⁡\(k\)Q\(k\)terms, so it cannot be used to precisely decompose the full difference between neighboring lengths or to attribute the decision to a single local jump in the expert count\. Only a local conclusion follows from them: without the expert term, the policy chooses a different length in all four illustrations; the prevalence of the effect across the full trace is not estimated\.

### B\.5Are Boundaries Associated with New Experts?

We next test the second hypothesis\. If a complex linguistic boundary changes the MoE model’s routing, more new experts should appear at that position\. If the same mechanism is directly linked to mismatches, a similar spike should be observed at the first position of mismatch with the reference\.

Figure B5:New experts at script change and at the first draft mismatch\. The left pair compares positions without a script change to Latin↔\\leftrightarrowCyrillic transitions; the right pair compares the position before a mismatch with the first mismatch position\. A higher value means the continuation brings in more previously unused experts\. Intervals are a 95% clustered bootstrap at the response level\.At a Latin↔\\leftrightarrowCyrillic transition, the number of new experts increases from 2\.66 to 3\.12 experts per layer\. The difference is\+0\.46\+0\.46, and the 95% interval\[\+0\.26;\+0\.66\]\[\+0\.26;\+0\.66\]excludes zero\. This difference is consistent with an association between script change and routing but does not separate script from the code domain\.

At the first mismatch position, the picture is different: 3\.31 experts versus 3\.30 at the preceding step\. The difference is\+0\.01\+0\.01, and the interval\[−0\.05;\+0\.07\]\[\-0\.05;\+0\.07\]includes zero\. No separate spike in new experts is observed at the first mismatch position\.

Comparisons are aligned by continuation step, and uncertainty is estimated with a clustered bootstrap at the response level\. The result is compatible with the weak associative hypothesis linking script change to routing but does not support the causal version in which the appearance of new experts directly explains mismatches\.

### B\.6Interpretation and Limitations

The descriptive analysis yields four observations\.

1. 1\.Regular formatting and repeated code structure are associated with a longer draft\.
2. 2\.A high probability of a matching prefix does not guarantee a long draft: expert cost can shift the conditional boundary\.
3. 3\.In the code\-domain\-mixed subsample, script change is associated with a shortk⋆k^\{\\star\}and the appearance of new experts\.
4. 4\.The first mismatch position by itself is not accompanied by an observable change in the number of experts\.

These observations point to a relationship between linguistic structure, draft\-length choice, and expert routing, but they do not establish a causal mechanism\. The analysis was conducted for a single target/draft model pair and 39 responses\. Cyrillic examples are concentrated mainly in code\-containing responses, so the effect of script cannot be fully separated from the domain\. Distances between hidden representations were not measured\.

Linguistic structure is linked to two different signals\. In the selected examples, regularity is accompanied by a high surrogate match probability; the aggregated effect size is not estimated\.k⋆k^\{\\star\}is determined by the full Bellman equation, which includes transition probabilities, current cost, and the continuation value; therefore, neither linguistic coherence nor a full draft match by itself determines the chosen length\. Taken together, the selected examples show the policy’s sensitivity to the type and predictability of the continuation, rather than a simple linguistic rule\. A repeated structure can go together with a long draft; a major boundary with a stop; a local mismatch with a small increase in experts with continuation; a predictable script change with either a short or a maximumk⋆k^\{\\star\}\.

### B\.7Additional Qualitative Examples

This gallery extends the local analysis and shows how different the decisions with the samek⋆k^\{\\star\}can be\. The sample is stratified, not random\. Three groups were defined first: regular continuations, structural boundaries, and counterexamples to the simple rule “a boundary or a mismatch with the reference implies early truncation and many new experts\.” The first group includes exact eight\-token matches of different types; the second includes linguistically interpretable transitions; the third includes cases that directly limit the simple hypothesis\. All 12 examples are drawn from different responses and do not repeat the six local examples above\. The gallery is used for interpretation, not to estimate the frequency or magnitude of the effect\.

For each example, we compare the continuations of the target and draft models, the cut point, and the number of new experts at that step\. The value of the first step is not used for the linguistic contrast, since it equals eight by construction\.

Figure B6:Four selected eight\-token matches from different genres and two structural boundaries\.Figure B7:Two major boundaries and four counterexamples\. Neither a formal boundary, nor a full match, nor a semantic mismatch alone determines the Oracle’s decision\.#### B\.7\.1Regular Continuations and Initial Structural Boundaries

A1–A4 show a full eight\-token match in a TypeScript import, a mathematical interval, a stable prose phrase, and a JSON schema\. These examples show that full eight\-token matches occur across several genres; their frequency is not estimated\. In A5, the Oracle retains only the general discourse openerThisand truncates the draft before the choice of predicate\. In A6, the header of a new function matches, while the choice of its signature begins already pastk⋆k^\{\\star\}\.

#### B\.7\.2Boundaries and Limitations of the Simple Hypothesis

In A7, a transition from prose to code, and in A8, a new Markdown section, both have a predictable structural scaffold; the substantive mismatch arises after the chosen boundary\. In A9, replacing/sbinwith/bin, and in A10,eventuallywithalways, occur within the draft but bring in few new experts\.

In A11, the surrogate match probability of matching at least five tokens is 99\.90%, yet the policy choosesk⋆=4k^\{\\star\}=4; the local rise in the number of new experts makes the cost\-based interpretation plausible, but theα=0\\alpha=0ablation does not change the decision here\. In A12, a switch from Cyrillic to the familiar technical tokenJSONcoincides with the maximumk⋆=8k^\{\\star\}=8\. Hence, neither a formal boundary, nor a full match, nor a semantic mismatch alone determines the Oracle’s decision\.

Additional examples illustrate the diversity of decisions but do not estimate the prevalence of the patterns\. They include repeated structures, major boundaries, low\-cost mismatches, and predictable script changes\.