Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts

# Cost-Aware Speculative Decoding for Mixture-of-Experts
Source: [https://arxiv.org/html/2607.12696](https://arxiv.org/html/2607.12696)
###### Abstract

Sparse Mixture\-of\-Experts \(MoE\) models have become an important approach for scaling Large Language Models \(LLMs\), but their inference efficiency depends strongly on expert activation patterns\. Speculative decoding \(SD\) accelerates autoregressive generation by verifying multiple draft tokens in parallel, yet existing draft selection strategies primarily optimize acceptance likelihood\. In large\-scale MoE models, however, selecting draft tokens also determines the union of experts activated during verification\. We observe that confidence\-driven SD can introduceexpert scattering: high\-probability draft tokens may route to disjoint experts, increasing expert\-weight memory traffic and reducing the speedup from speculation\. Motivated by this observation, we revisit draft\-tree selection under the non\-uniform memory\-cost structure of MoE inference\. We proposeEcoSpec, a cost\-aware speculative decoding framework that incorporates predicted marginal expert activation cost into draft selection\. With a lightweight expert predictor and a dynamic expert buffer,EcoSpecfavors draft paths that preserve high acceptance likelihood while reusing experts already covered by the current verification set, without modifying the target\-model verification rule\. We evaluateEcoSpecon three large\-scale MoE models, including DeepSeek\-V3\.1 \(671B\), Qwen3\-235B\-A22B, and GPT\-OSS\-120B, across reasoning, coding, question\-answering, and dialogue benchmarks\.EcoSpecconsistently reduces active expert footprints and improves end\-to\-end decoding speed, achieving up to1\.62×1\.62\\timesspeedup\. These results show that accounting for expert activation cost is important for efficient speculative decoding in large\-scale MoE models\.

Machine Learning, ICML

## 1Introduction

![Refer to caption](https://arxiv.org/html/2607.12696v1/x1.png)\(a\)Latency vs\. Expert Load
![Refer to caption](https://arxiv.org/html/2607.12696v1/x2.png)\(b\)Expert Activation Burden
![Refer to caption](https://arxiv.org/html/2607.12696v1/x3.png)\(c\)Single Expert Cost

Figure 1:The Bandwidth Bottleneck in MoE Speculative Decoding\.\(a\) Verification latency scales linearly with the number of active experts \(ℰ\\mathcal\{E\}\), creating a strict latency penalty for retrieving extra experts\. \(b\) Top\-K means K activated experts in each layer\. As the verification budgetγ\\gammaincreases, standard baselines \(e\.g\., Eagle\) rapidly activate disjoint sets of experts, quickly saturating the memory bandwidth\. \(c\) The physical memory footprint of asingleexpert is substantial\. For DeepSeek\-V3\.1 \(FP8\), loading just one expert consumes44\.04 MBof HBM bandwidth, implying that every mispredicted expert path incurs a massive I/O overhead\.Large Language Models \(LLMs\) have demonstrated remarkable capabilities across logic, coding, and creative tasks\(Brownet al\.,[2020](https://arxiv.org/html/2607.12696#bib.bib7); OpenAIet al\.,[2024](https://arxiv.org/html/2607.12696#bib.bib8); Grattafioriet al\.,[2024](https://arxiv.org/html/2607.12696#bib.bib9)\), yet their deployment is increasingly constrained by high inference latency and serving cost\(Rajbhandariet al\.,[2020](https://arxiv.org/html/2607.12696#bib.bib10)\)\. The standard autoregressive decoding process generates tokens sequentially, where each step requires a full pass through the target model\(Vaswaniet al\.,[2017](https://arxiv.org/html/2607.12696#bib.bib11)\)\. As models scale to hundreds of billions of parameters, this serial dependency makes inference heavily memory\-bound: the arithmetic intensity is low, and throughput is often limited by the bandwidth required to load model weights from High Bandwidth Memory \(HBM\) to on\-chip compute units\(Williamset al\.,[2009](https://arxiv.org/html/2607.12696#bib.bib13); Shazeer,[2019](https://arxiv.org/html/2607.12696#bib.bib12); Daoet al\.,[2022](https://arxiv.org/html/2607.12696#bib.bib14)\)\. Consequently, reducing decoding latency and memory traffic has become a critical priority for both real\-time user experience and infrastructure efficiency\.

To mitigate the serial decoding bottleneck, Speculative Decoding \(SD\) has been widely studied as an inference acceleration paradigm\(Leviathanet al\.,[2023](https://arxiv.org/html/2607.12696#bib.bib15); Chenet al\.,[2023](https://arxiv.org/html/2607.12696#bib.bib16); Liet al\.,[2024b](https://arxiv.org/html/2607.12696#bib.bib18),[a](https://arxiv.org/html/2607.12696#bib.bib19),[2026](https://arxiv.org/html/2607.12696#bib.bib20); Caiet al\.,[2024](https://arxiv.org/html/2607.12696#bib.bib21)\)\. SD uses a low\-cost draft mechanism to propose multiple candidate tokens, which are then verified in parallel by the target model\. When the drafts are accepted, a single target\-model forward pass can advance generation by multiple tokens, thereby amortizing parameter loading and improving hardware utilization\. For dense Transformers, where the same weight matrices are reused across all verified positions, this parallel verification can substantially increase arithmetic intensity and reduce the effective cost per generated token\(Miaoet al\.,[2024](https://arxiv.org/html/2607.12696#bib.bib17)\)\.

However, the dense\-model assumption behind this amortization does not directly extend to sparse Mixture\-of\-Experts \(MoE\) architectures\(Shazeeret al\.,[2017](https://arxiv.org/html/2607.12696#bib.bib22); Feduset al\.,[2022](https://arxiv.org/html/2607.12696#bib.bib23)\), which are increasingly adopted in large\-scale language models\(OpenAIet al\.,[2025](https://arxiv.org/html/2607.12696#bib.bib26); DeepSeek\-AIet al\.,[2025](https://arxiv.org/html/2607.12696#bib.bib27); Yanget al\.,[2025](https://arxiv.org/html/2607.12696#bib.bib29)\)\. In MoE layers, the dense feed\-forward module is replaced by a pool of experts, and a router assigns each token to a small top\-kksubset of experts\(Lepikhinet al\.,[2021](https://arxiv.org/html/2607.12696#bib.bib24); Duet al\.,[2022](https://arxiv.org/html/2607.12696#bib.bib25)\)\. Therefore, parallel verification no longer reuses a single fixed set of feed\-forward weights across all verified positions\. Its memory cost depends on the union of experts activated by the verified tokens: when different candidate tokens route to disjoint experts, the verifier must fetch additional expert weight blocks from HBM\. As a result, verification latency in MoE speculative decoding becomes highly sensitive to expert overlap and reuse, rather than being determined mainly by the number of verified tokens\.

This MoE\-specific cost structure exposes a mismatch between the acceptance\-driven selection objective of existing SD methods and the memory cost of MoE verification\. Many recent SD methods organize draft candidates as a tree and select a verification subset primarily according to confidence or acceptance likelihood\. This criterion is effective for dense models, where the main objective is to maximize the number of accepted tokens per target\-model pass\. In MoE verification, however, a high\-probability candidate can still activate experts that are disjoint from those used by other verified tokens\. Adding such a candidate may therefore enlarge the per\-step expert union, increase expert\-weight memory traffic, and reduce cache reuse\(Huanget al\.,[2026](https://arxiv.org/html/2607.12696#bib.bib30); Xueet al\.,[2024](https://arxiv.org/html/2607.12696#bib.bib31)\)\. We refer to this expansion of the per\-step expert footprint asexpert scattering\. As shown in Fig\.[1](https://arxiv.org/html/2607.12696#S1.F1), verification latency increases with the active expert footprint, while confidence\-driven selection can rapidly increase the number of unique experts touched within one verification step\. Consequently, expanding the verification set may erode end\-to\-end speedup even when acceptance rates remain comparable\. This observation motivates a cost\-aware draft selection objective that balances acceptance likelihood with the marginal cost of introducing new experts\.

To address this issue, we proposeEcoSpec, a cost\-aware speculative decoding framework for MoE models\.EcoSpecoperates at the draft\-tree selection stage, where it selects candidates for parallel verification under an acceptance–cost trade\-off\. Instead of ranking draft tokens primarily by acceptance likelihood,EcoSpecalso accounts for the marginal expert cost induced by each candidate\. This allows the selection procedure to prefer draft paths that maintain high acceptance probability while reusing experts already covered by the current verification set\. Importantly,EcoSpecdoes not modify the target\-model verification rule and therefore preserves the lossless semantics of standard speculative decoding\(Leviathanet al\.,[2023](https://arxiv.org/html/2607.12696#bib.bib15)\)\. By aligning draft selection with the memory\-cost structure of MoE inference,EcoSpecreduces unnecessary expert\-weight traffic during verification\. Empirically,EcoSpecachieves consistent speedups across multiple production\-scale MoEs, including up to1\.62×\\timeson Qwen3\-235B,1\.50×\\timeson GPT\-OSS\-120B, and1\.47×\\timeson DeepSeek\-V3\.1\.

Our contributions are summarized as follows:

- •We identify and analyzeexpert scatteringin large\-scale MoE speculative decoding: confidence\-driven draft selection can expand the per\-step union of activated experts, increasing expert\-weight memory traffic during verification\.
- •We proposeEcoSpec, a cost\-aware speculative decoding framework that incorporates marginal expert activation cost into draft\-tree selection\. With a lightweight expert predictor and a dynamic expert buffer,EcoSpecfavors draft paths that preserve high acceptance likelihood while improving expert reuse, without changing the standard lossless verification procedure\.
- •We evaluateEcoSpecon three large\-scale MoE models—DeepSeek\-V3\.1 \(671B\), Qwen3\-235B\-A22B, and GPT\-OSS\-120B—across diverse reasoning, coding, and dialogue benchmarks\.EcoSpecconsistently reduces activated experts and improves end\-to\-end decoding speed, achieving up to1\.62×1\.62\\timesspeedup over existing SD baselines\.

![Refer to caption](https://arxiv.org/html/2607.12696v1/x4.png)Figure 2:Overview of theEcoSpecframework\.The process begins with draft generation, where a draft modelMdM\_\{d\}produces a tree of candidate tokens with associated probabilities\. TheEcoSpecmodule then selects a set of draft tokens𝒮\\mathcal\{S\}for verification\. It employs a lightweight expert predictorΠθ\\Pi\_\{\\theta\}to estimate expert activations, maintains a global expert bufferℬ\\mathcal\{B\}to track experts already covered by the selected tokens, and uses cost\-aware draft selection to balance acceptance likelihood with expert activation cost\. Finally, the target MoE modelMpM\_\{p\}performs parallel verification on the selected set𝒮\\mathcal\{S\}\.
## 2Related Work

#### Speculative Decoding and Multi\-Token Prediction\.

Speculative decoding \(SD\) accelerates LLM inference by using a lower\-cost draft mechanism to propose multiple tokens, which are then verified in parallel by the target model\(Leviathanet al\.,[2023](https://arxiv.org/html/2607.12696#bib.bib15); Chenet al\.,[2023](https://arxiv.org/html/2607.12696#bib.bib16)\)\. Early approaches commonly use a separate draft model to generate candidate continuations\(Miaoet al\.,[2024](https://arxiv.org/html/2607.12696#bib.bib17)\)\. Recent methods further improve draft efficiency and acceptance by adopting tree\-structured drafting or auxiliary prediction heads, including Medusa and the EAGLE series\(Caiet al\.,[2024](https://arxiv.org/html/2607.12696#bib.bib21); Liet al\.,[2024b](https://arxiv.org/html/2607.12696#bib.bib18),[a](https://arxiv.org/html/2607.12696#bib.bib19),[2026](https://arxiv.org/html/2607.12696#bib.bib20)\)\. In parallel, Multi\-Token Prediction \(MTP\) introduces an auxiliary training objective that enables in\-model token predictors to serve as draft heads for speculative verification\(DeepSeek\-AIet al\.,[2025](https://arxiv.org/html/2607.12696#bib.bib27); Gloeckleet al\.,[2024](https://arxiv.org/html/2607.12696#bib.bib32)\)\. Despite architectural differences in how drafts are produced, these approaches typically prioritize draft tokens based on confidence \(acceptance likelihood\) and do not explicitly account for the hardware cost of activating additional experts in MoE verification\. As a result, when applied to MoE models, high\-confidence drafts may still trigger a rapidly expanding union of activated experts, corresponding to our Expert Scattering phenomenon and the associated memory inefficiency\. EcoSpec makes this explicit by optimizing a cost\-aware objective that balances acceptance likelihood against the incremental cost of activating new experts\.

#### Efficient Inference for Mixture\-of\-Experts\.

Mixture\-of\-Experts \(MoE\) models increase model capacity while reducing per\-token FLOPs, but their inference efficiency is often limited by expert\-weight memory traffic and token–expert dispatch overheads\(Shazeeret al\.,[2017](https://arxiv.org/html/2607.12696#bib.bib22); Liuet al\.,[2026](https://arxiv.org/html/2607.12696#bib.bib33)\)\. Prior work improves MoE execution under a given routing pattern through expert caching and prefetching\(Xueet al\.,[2025](https://arxiv.org/html/2607.12696#bib.bib34); Huanget al\.,[2024](https://arxiv.org/html/2607.12696#bib.bib35)\), optimized token dispatch and fused MoE kernels\(Galeet al\.,[2023](https://arxiv.org/html/2607.12696#bib.bib36)\), and routing or load\-balancing strategies that reduce uneven expert utilization\(Feduset al\.,[2022](https://arxiv.org/html/2607.12696#bib.bib23); DeepSeek\-AIet al\.,[2025](https://arxiv.org/html/2607.12696#bib.bib27)\)\. Recent systems further study MoE inference in the speculative\-decoding setting\. SP\-MoE and MoE\-SpeQ use speculative lookahead to support expert prefetching, offloading, and execution scheduling, aiming to hide or reduce expert\-movement overhead during MoE serving\(Chenet al\.,[2025](https://arxiv.org/html/2607.12696#bib.bib3); Wanget al\.,[2025](https://arxiv.org/html/2607.12696#bib.bib2)\)\. MoE\-Spec instead reduces verification\-time expert cost by imposing an expert budget and selecting only a subset of experts to load during speculative verification\(McDanelet al\.,[2026](https://arxiv.org/html/2607.12696#bib.bib1)\)\. These methods either optimize runtime execution under predicted or routed expert demand, or change the verification\-time expert budget\.EcoSpecaddresses a different stage of the decoding pipeline: it preserves the standard target\-model verifier and lossless speculative decoding semantics, and only changes which draft\-tree nodes are selected before verification\. Therefore,EcoSpecis complementary to MoE runtime optimizations, since expert caching, prefetching, offloading, or optimized dispatch can still be applied afterEcoSpecreduces the expert working set induced by the selected verification nodes\.

![Refer to caption](https://arxiv.org/html/2607.12696v1/x5.png)Figure 3:Illustration of Pre\-verification Cost\-Aware Subset Construction\. We selectγ=2\\gamma=2draft tokens before target\-model verification\. Left \(Step 1\): the root tokent0t\_\{0\}is selected according to the acceptance–cost score, and its predicted expert footprint\{E1,E2,E3,E4\}\\\{E\_\{1\},E\_\{2\},E\_\{3\},E\_\{4\}\\\}is added to the expert bufferℬ\\mathcal\{B\}for subsequent scoring\. Middle \(Step 2\): the algorithm re\-evaluates the remaining candidates using the updated buffer\. Althought1t\_\{1\}has higher cumulative draft probability \(P=0\.53P=0\.53\), it introduces three new predicted experts \(Δ​Cost=3\\Delta\\mathrm\{Cost\}=3\)\. In contrast,t2t\_\{2\}reuses experts already covered byℬ\\mathcal\{B\}and introduces only two new predicted experts \(Δ​Cost=2\\Delta\\mathrm\{Cost\}=2\), so it obtains a higher score \(S=0\.20\>0\.17S=0\.20\>0\.17\) and is selected\. Right \(Final\): the final selected set is𝒮=\{t0,t2\}\\mathcal\{S\}=\\\{t\_\{0\},t\_\{2\}\\\}, illustrating how marginal expert cost can change the ranking induced by cumulative draft probability\.

## 3MoE Speculative Decode Bottleneck

Speculative Decoding \(SD\) accelerates autoregressive generation by verifying multiple draft tokens in a single target\-model forward pass\. For dense Transformers, the memory cost of verifyingγ\\gammatokens is close to that of verifying a single token, because all verified positions reuse the same dense weight matrices within the verification batch\. Thus, increasing the verification budget mainly improves the amortization of target\-model parameter loading and can reduce the effective cost per generated token\.

This premise becomes less reliable for sparse MoE models\. For a MoE layerℓ\\ell, let𝒮ℓ​\(xt\)\\mathcal\{S\}\_\{\\ell\}\(x\_\{t\}\)denote the set of experts activated by thett\-th verified token\. Verifying a draft sequencex1:γx\_\{1:\\gamma\}requires accessing the union of experts activated by all verified tokens at that layer:

ℰverifyℓ=⋃t=1γ𝒮ℓ​\(xt\)\.\\mathcal\{E\}^\{\\ell\}\_\{\\mathrm\{verify\}\}=\\bigcup\_\{t=1\}^\{\\gamma\}\\mathcal\{S\}\_\{\\ell\}\(x\_\{t\}\)\.\(1\)The verification footprint is therefore determined by the size of this union, aggregated across MoE layers, rather than by the number of verified tokens alone\. If selected draft tokens route to overlapping experts, verification can reuse expert weights\. If they route to disjoint experts, the verifier must fetch additional expert weight blocks from HBM, increasing memory traffic within the same speculative step\.

This creates an expert\-scattering effect in MoE speculative decoding\. Standard SD selection strategies are typically designed to maximize acceptance likelihood and are not aware of expert locality\. As a result, adding more high\-confidence draft tokens can enlarge the per\-step expert union even when those tokens are likely to be accepted\. In practice,\|ℰverifyℓ\|\|\\mathcal\{E\}^\{\\ell\}\_\{\\mathrm\{verify\}\}\|can grow rapidly with the verification budgetγ\\gamma, reducing expert reuse and creating an additional memory\-traffic bottleneck\.

We profile DeepSeek\-V3\.1 \(671B\), Qwen3\-235B\-A22B, and GPT\-OSS\-120B on GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.12696#bib.bib40)\)under identical execution settings\. Fig\.[1](https://arxiv.org/html/2607.12696#S1.F1)\(a\) shows that verification latency increases with the number of unique experts activated per step, indicating that fetching additional expert weights is a major source of verification cost\. Fig\.[1](https://arxiv.org/html/2607.12696#S1.F1)\(b\) shows that standard SD selection can rapidly increase the cumulative number of unique activated experts as more tokens are selected for verification\. This means that a larger verification set provides more opportunities for token acceptance, but can also make each verification pass more expensive in MoE models\.

The cost of this effect is substantial at large scale\. As shown in Fig\.[1](https://arxiv.org/html/2607.12696#S1.F1)\(c\), a single expert can occupy tens of megabytes of memory, so even a small increase in the per\-layer expert union can translate into large HBM traffic\. For DeepSeek\-V3\.1, reducing the expected expert footprint by only0\.20\.2experts per layer per step decreases expert\-weight traffic by approximately0\.50\.5GB per speculative step\. These findings motivateEcoSpec, which incorporates expert activation cost into draft selection to reduce the verification expert footprint while maintaining high acceptance likelihood\.

## 4Methodology:EcoSpec

To reduce verification memory traffic in MoE speculative decoding,EcoSpecintroduces a cost\-aware selection mechanism between draft generation and target\-model verification\. Given a draft tree with candidate probabilities,EcoSpecselectsγ\\gammadraft tokens for parallel verification by considering both acceptance likelihood and predicted expert activation cost\. As shown in Figure[2](https://arxiv.org/html/2607.12696#S1.F2), the framework consists of three components:

\(1\) A lightweightExpert PredictorΠθ\\Pi\_\{\\theta\}that estimates the experts each draft token is likely to activate in the target MoE model\. It outputs per\-layer expert distributions, and the top\-KKpredicted experts are used as an approximation of the token’s activated expert set\.

\(2\) AGlobal Expert Bufferℬ\\mathcal\{B\}that records experts already covered by the selected verification set\. This buffer enables efficient marginal\-cost estimation by counting only the newly introduced experts when a candidate is added\.

\(3\) ACost\-Aware Draft Selectionalgorithm that selects draft tokens from the draft tree under an acceptance–cost scoring rule\. The score favors candidates with high cumulative draft probability and low marginal expert cost, encouraging selected paths to reuse experts already present inℬ\\mathcal\{B\}\.

#### Execution order\.

EcoSpecruns after draft generation and before target\-model verification\. It first constructs a verification subset𝒮\\mathcal\{S\}of sizeγ\\gammausing draft probabilities and predicted expert footprints, without querying the target router during this stage\. During subset construction, the bufferℬ\\mathcal\{B\}is updated from predicted path\-level expert sets to estimate the marginal cost of the remaining candidates\. Once𝒮\\mathcal\{S\}is constructed, the selected nodes are submitted to the standard speculative verifier and checked together in a single target\-model forward pass\.

### 4\.1Lightweight Expert Predictor

Cost\-aware selection requires estimating the expert footprint of each draft token before target\-model verification\. Directly querying the target router for every draft candidate would require running the target MoE model, which would offset the benefit of speculative decoding\. We therefore train a lightweight expert predictorΠθ\\Pi\_\{\\theta\}to approximate target\-model routing and provide a low\-cost estimate of expert activation\.

#### Architecture\.

We instantiateΠθ\\Pi\_\{\\theta\}with a small decoder\-only backbone\. For compatibility with the target MoE modelMpM\_\{p\}, each input token is first mapped by the target embedding layer and then projected into the predictor hidden space\. Given a draft tokentit\_\{i\}with contextt<it\_\{<i\}, the predictor outputs routing logits

𝐳ti=Πθ​\(ti∣t<i\)∈ℝL×E,\\mathbf\{z\}\_\{t\_\{i\}\}=\\Pi\_\{\\theta\}\(t\_\{i\}\\mid t\_\{<i\}\)\\in\\mathbb\{R\}^\{L\\times E\},\(2\)whereLLis the number of MoE layers andEEis the number of experts per layer\. We obtain a routing distribution for each MoE layer by applying softmax over experts:

𝐩^ti,ℓ=Softmax​\(𝐳ti,ℓ\)∈ℝE,ℓ∈\{1,…,L\}\.\\hat\{\\mathbf\{p\}\}\_\{t\_\{i\},\\ell\}=\\mathrm\{Softmax\}\\\!\\left\(\\mathbf\{z\}\_\{t\_\{i\},\\ell\}\\right\)\\in\\mathbb\{R\}^\{E\},\\qquad\\ell\\in\\\{1,\\dots,L\\\}\.\(3\)The predicted expert set is then constructed as a set of layer–expert pairs:

ℰpred​\(ti\)=\{\(ℓ,e\)∣e∈TopK​\(𝐩^ti,ℓ,K\),ℓ∈\{1,…,L\}\}\.\\mathcal\{E\}\_\{\\mathrm\{pred\}\}\(t\_\{i\}\)=\\left\\\{\(\\ell,e\)\\mid e\\in\\mathrm\{TopK\}\\\!\\left\(\\hat\{\\mathbf\{p\}\}\_\{t\_\{i\},\\ell\},K\\right\),\\ell\\in\\\{1,\\dots,L\\\}\\right\\\}\.\(4\)

#### Training Objective\.

During predictor training, we run the target MoE model offline and record the ground\-truth activated experts for each token\. For tokenttand MoE layerℓ\\ell, letℰgt​\(t,ℓ\)\\mathcal\{E\}\_\{\\mathrm\{gt\}\}\(t,\\ell\)denote the top\-KKexperts selected by the target router\. We define a normalized target distribution𝐪t,ℓ∈\[0,1\]E\\mathbf\{q\}\_\{t,\\ell\}\\in\[0,1\]^\{E\}by assigning uniform probability mass to the activated experts:

qt,ℓ,e=\{1K,e∈ℰgt​\(t,ℓ\),0,otherwise\.q\_\{t,\\ell,e\}=\\begin\{cases\}\\frac\{1\}\{K\},&e\\in\\mathcal\{E\}\_\{\\mathrm\{gt\}\}\(t,\\ell\),\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}\(5\)We trainΠθ\\Pi\_\{\\theta\}with layer\-wise cross\-entropy:

ℒpred​\(θ\)=−∑t∑ℓ=1L∑e=1Eqt,ℓ,e​log⁡p^t,ℓ,e\.\\mathcal\{L\}\_\{\\mathrm\{pred\}\}\(\\theta\)=\-\\sum\_\{t\}\\sum\_\{\\ell=1\}^\{L\}\\sum\_\{e=1\}^\{E\}q\_\{t,\\ell,e\}\\log\\hat\{p\}\_\{t,\\ell,e\}\.\(6\)

### 4\.2Cost\-Aware Draft Selection

Given the predicted expert sets from §[4\.1](https://arxiv.org/html/2607.12696#S4.SS1),EcoSpecselects draft\-tree nodes for verification under an acceptance–cost trade\-off\. The selection is path\-based: each candidate is scored using the cumulative draft probability and predicted expert footprint of its root\-to\-node path\. Existing draft\-tree methods primarily rank candidates by confidence or acceptance likelihood\(Liet al\.,[2024a](https://arxiv.org/html/2607.12696#bib.bib19)\)\. For MoE verification, however, a high\-confidence candidate can still introduce many new experts if its routing footprint has little overlap with the candidates already selected\. Therefore,EcoSpecaugments confidence\-based selection with the marginal expert cost induced by each candidate\.

#### Selection State\.

Let𝒯\\mathcal\{T\}denote the draft tree, where each nodetit\_\{i\}corresponds to a candidate token\. During selection,EcoSpecmaintains a selected set𝒮\\mathcal\{S\}and an expert bufferℬ\\mathcal\{B\}\. The selected set𝒮\\mathcal\{S\}contains tokens chosen for verification, whileℬ\\mathcal\{B\}contains the experts already covered by the current selected set\. The buffer is used to measure how many new experts a candidate would introduce beyond those already covered\.

#### Path\-Dependent Expert Footprint\.

Because draft\-tree verification is prefix\-dependent, selecting a node requires covering its path from the root\. For a candidate nodetit\_\{i\}, letPath​\(root→ti\)\\mathrm\{Path\}\(\\mathrm\{root\}\\\!\\to\\\!t\_\{i\}\)denote the sequence of nodes from the root totit\_\{i\}\. The predicted expert footprint of this path is

ℰt​r​a​j​\(ti\)=⋃τ∈Path​\(root→ti\)ℰp​r​e​d​\(τ\)\.\\mathcal\{E\}\_\{traj\}\(t\_\{i\}\)=\\bigcup\_\{\\tau\\in\\mathrm\{Path\}\(\\mathrm\{root\}\\to t\_\{i\}\)\}\\mathcal\{E\}\_\{pred\}\(\\tau\)\.\(7\)The marginal expert cost of selectingtit\_\{i\}under the current buffer is then defined as

Δ​Cost​\(ti∣ℬ\)=\|ℰt​r​a​j​\(ti\)∖ℬ\|\.\\Delta\\text\{Cost\}\(t\_\{i\}\\mid\\mathcal\{B\}\)=\\left\|\\mathcal\{E\}\_\{traj\}\(t\_\{i\}\)\\setminus\\mathcal\{B\}\\right\|\.\(8\)This cost counts only the additional experts that are not already covered by previously selected candidates\.

#### Acceptance–Cost Scoring\.

LetP​\(ti\)P\(t\_\{i\}\)be the cumulative draft probability along the path from the root totit\_\{i\}\.EcoSpecscores each candidate by

S​\(ti\)=P​\(ti\)Δ​Cost​\(ti∣ℬ\)\+ϵ,S\(t\_\{i\}\)=\\frac\{P\(t\_\{i\}\)\}\{\\Delta\\text\{Cost\}\(t\_\{i\}\\mid\\mathcal\{B\}\)\+\\epsilon\},\(9\)whereϵ\\epsilonavoids division by zero\. This score favors candidates with high acceptance likelihood while penalizing those that introduce many new experts\. Equivalently, it encourages the selected verification set to reuse experts already present inℬ\\mathcal\{B\}, thereby limiting the growth of the per\-step expert union\.

#### Prefix Consistency\.

The score is naturally compatible with the prefix structure of the draft tree\. For a parent nodetpt\_\{p\}and its childtct\_\{c\}, the cumulative probability satisfiesP​\(tc\)=P​\(tp\)​P​\(tc∣tp\)≤P​\(tp\)P\(t\_\{c\}\)=P\(t\_\{p\}\)P\(t\_\{c\}\\mid t\_\{p\}\)\\leq P\(t\_\{p\}\), while the path footprint satisfiesℰt​r​a​j​\(tp\)⊆ℰt​r​a​j​\(tc\)\\mathcal\{E\}\_\{traj\}\(t\_\{p\}\)\\subseteq\\mathcal\{E\}\_\{traj\}\(t\_\{c\}\)\. Therefore,Δ​Cost​\(tp∣ℬ\)≤Δ​Cost​\(tc∣ℬ\)\\Delta\\text\{Cost\}\(t\_\{p\}\\mid\\mathcal\{B\}\)\\leq\\Delta\\text\{Cost\}\(t\_\{c\}\\mid\\mathcal\{B\}\)\. Under the same bufferℬ\\mathcal\{B\}, a child node therefore cannot receive a higher score than its parent except in tie cases\. Thus, the scoring rule is aligned with prefix\-closed draft\-tree verification without requiring an additional structural penalty\.

#### Dynamic Buffer Effect\.

After a candidate path is selected, its predicted experts are added toℬ\\mathcal\{B\}\. This update reduces the marginal cost of later candidates that share experts with the selected path, increasing their scores without changing their draft probabilities\. As a result,EcoSpectends to extend high\-probability paths that also preserve expert locality, rather than expanding the draft tree solely by confidence\. The full selection procedure is provided in Appendix Alg\.[1](https://arxiv.org/html/2607.12696#alg1)\.

## 5Experiments

Table 1:Main results across target models, benchmarks, and decoding temperatures\. Each entry reports end\-to\-end speedup relative to AR decoding / mean acceptance lengthα\\alpha/ average active expertsℰ\\mathcal\{E\}\. Here,ℰ\\mathcal\{E\}denotes the average number of unique experts activated per MoE layer within one verification step\. Rows are grouped by target MoE model and target\-model decoding temperature\.### 5\.1Setup

#### Models and Benchmarks\.

We evaluateEcoSpecon three large\-scale MoE models with different sizes and routing configurations: DeepSeek\-V3\.1\(DeepSeek\-AIet al\.,[2025](https://arxiv.org/html/2607.12696#bib.bib27)\)\(671B total / 37B active, Top\-8\), Qwen3\-235B\-A22B\(Yanget al\.,[2025](https://arxiv.org/html/2607.12696#bib.bib29)\)\(235B total / 22B active, Top\-8\), and GPT\-OSS\-120B\(OpenAIet al\.,[2025](https://arxiv.org/html/2607.12696#bib.bib26)\)\(120B total / 5\.1B active, Top\-4\)\. We use seven benchmarks covering mathematical reasoning, code generation, question answering, and dialogue: GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.12696#bib.bib40)\), HumanEval\(Chenet al\.,[2021](https://arxiv.org/html/2607.12696#bib.bib41)\), AIME\-25\(MAA,[2025](https://arxiv.org/html/2607.12696#bib.bib44)\), Math500\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.12696#bib.bib42); Lightmanet al\.,[2024](https://arxiv.org/html/2607.12696#bib.bib43)\), AMC22\-24\([AI\-MO,](https://arxiv.org/html/2607.12696#bib.bib45)\), MTBench\(Zhenget al\.,[2023](https://arxiv.org/html/2607.12696#bib.bib46)\), and MMStar\(Chenet al\.,[2024](https://arxiv.org/html/2607.12696#bib.bib47)\)\.

#### Baselines and Hardware\.

#### Speculative Decoding Configuration\.

To isolate the effect of cost\-aware selection, we keep the draft\-generation configuration fixed between each speculative baseline and itsEcoSpecvariant\. The draft process runs for 3 forward steps and keeps the top\-2 tokens at each step to construct the draft tree\. The verification budgetγ\\gammais set to 4, whereγ\\gammadenotes the total number of tokens verified per speculative step, including the bonus token\. This setting follows the operating regime of the large\-scale MoE speculative baselines evaluated in this work, and the same configuration is used for the baseline andEcoSpecto ensure a controlled comparison\. Appendix[C](https://arxiv.org/html/2607.12696#A3)further evaluates different verification budgets and shows thatγ=4\\gamma=4gives the highest average end\-to\-end speedup in our setting\. All main experiments use batch size 1 unless explicitly stated otherwise\. Table[1](https://arxiv.org/html/2607.12696#S5.T1)reports both greedy target decoding \(T=0T\{=\}0\) and sampling\-based target decoding \(T=1T\{=\}1\), following common speculative decoding evaluation practice\(Liet al\.,[2024b](https://arxiv.org/html/2607.12696#bib.bib18)\)\. For the latency breakdown and HBM\-traffic analysis in §[5\.3](https://arxiv.org/html/2607.12696#S5.SS3), we use the greedy target\-decoding rows of Table[1](https://arxiv.org/html/2607.12696#S5.T1), i\.e\., batch size 1,γ=4\\gamma=4, andT=0T\{=\}0\. We further evaluate larger batch sizes, different verification budgetsγ\\gamma, and different predictor accuracy levels in Appendix[A](https://arxiv.org/html/2607.12696#A1), Appendix[C](https://arxiv.org/html/2607.12696#A3), and Appendix[B\.2](https://arxiv.org/html/2607.12696#A2.SS2), respectively\.

#### Expert Predictors\.

EcoSpecreuses the draft\-generation infrastructure of the corresponding speculative baseline and only modifies the draft selection stage\. For expert\-cost estimation, we use DeepSeek\-R1\-Distill\-Qwen\-1\.5B\(Guoet al\.,[2025](https://arxiv.org/html/2607.12696#bib.bib28)\)as the predictor backbone for DeepSeek\-V3\.1, and Qwen3\-0\.6B\(Yanget al\.,[2025](https://arxiv.org/html/2607.12696#bib.bib29)\)for Qwen3\-235B\-A22B and GPT\-OSS\-120B\. Predictor training details are provided in Appendix[B\.1](https://arxiv.org/html/2607.12696#A2.SS1)\.

#### Inference Backend\.

The main experiments are conducted with a HuggingFace Transformers\(Wolfet al\.,[2020](https://arxiv.org/html/2607.12696#bib.bib4)\)research prototype following the EAGLE\-3 inference pipeline\. We use this prototype to keep the draft\-generation, tree\-verification, and KV\-cache update workflow consistent with the released EAGLE\-3 implementation, so that the comparison isolates the effect ofEcoSpec’s cost\-aware draft selection\. During decoding, the draft model constructs the draft tree, and the selected draft nodes are verified by the target MoE model in a single forward pass with a tree\-structured attention mask\.EcoSpeconly adds the expert predictor and cost\-aware selection before this verification step; the target\-model verification rule is unchanged\.

Table 2:Estimated HBM read traffic during verification\. We report the estimated total HBM read bytes during the verification phase of one speculative step\.Table 3:Latency breakdown and throughput analysis\. We report the average wall\-clock time in seconds per speculative step\.ℰ\\mathcal\{E\}denotes the average number of unique experts activated per MoE layer within one verification step\.TpredT\_\{\\text\{pred\}\},TdraftT\_\{\\text\{draft\}\},TverifyT\_\{\\text\{verify\}\}, andTtotalT\_\{\\text\{total\}\}denote predictor overhead, draft\-generation time, target verification time, and total speculative\-step latency, respectively\.α\\alphadenotes the mean accepted tokens per step\. The speedup column reports the corresponding end\-to\-end throughput speedup under the sameT=0T=0setting as Table[1](https://arxiv.org/html/2607.12696#S5.T1)\. The latency\-breakdown columns are reported to explain the sources of the end\-to\-end speedup\.ModelMethodExpertsPredictorDraftingVerifyTotal StepAccept LenThroughput\(ℰ\\mathcal\{E\}\)↓\\downarrow\(TpredT\_\{\\text\{pred\}\}\)\(TdraftT\_\{\\text\{draft\}\}\)\(TverifyT\_\{\\text\{verify\}\}\)↓\\downarrow\(TtotalT\_\{\\text\{total\}\}\)↓\\downarrow\(α\\alpha\)↑\\uparrowSpeedup↑\\uparrowQwen3\-235B\-A22BBaseline \(AR\)8\.0–––0\.490s†1\.001\.00×\\timesEAGLE\-323\.7–0\.008s0\.832s0\.840s2\.411\.22×\\timesEcoSpec20\.50\.004s0\.008s0\.730s0\.742s2\.321\.36×\\timesΔ\\Deltavs\. EAGLE\-3\-3\.2\+0\.004s0\.000s\-0\.102s\-0\.098s\-0\.09\+0\.14×\\timesGPT\-OSS\-120BBaseline \(AR\)4\.0–––0\.081s†1\.001\.00×\\timesEAGLE\-311\.6–0\.035s0\.113s0\.148s1\.881\.14×\\timesEcoSpec10\.60\.004s0\.035s0\.090s0\.129s1\.861\.31×\\timesΔ\\Deltavs\. EAGLE\-3\-1\.0\+0\.004s0\.000s\-0\.023s\-0\.019s\-0\.02\+0\.17×\\timesDeepSeek\-V3\.1Baseline \(AR\)8\.0–––0\.369s†1\.001\.00×\\timesMTP31\.4–0\.008s0\.980s0\.988s2\.851\.10×\\timesEcoSpec31\.20\.004s0\.008s0\.930s0\.942s2\.791\.15×\\timesΔ\\Deltavs\. MTP\-0\.2\+0\.004s0\.000s\-0\.050s\-0\.046s\-0\.06\+0\.05×\\times†AR baseline time is per\-token latency\. Speculative methods report per\-step latency and generateα\\alphaaccepted tokens per step\.Delta rows compareEcoSpecwith the corresponding speculative baseline\.

### 5\.2Main Results

Table[1](https://arxiv.org/html/2607.12696#S5.T1)reports end\-to\-end speedup relative to autoregressive decoding, mean acceptance lengthα\\alpha, and the average number of unique expertsℰ\\mathcal\{E\}activated per MoE layer within one verification step\. Across three MoE backbones and seven benchmarks,EcoSpecimproves decoding speed over the corresponding speculative baseline while consistently reducing the expert footprint\. The acceptance lengths remain close to the baselines, indicating that the speedup mainly comes from reducing verification cost rather than increasing the number of accepted tokens\.

#### Greedy decoding \(T=0T\{=\}0\)\.

Under greedy decoding,EcoSpecconsistently improves speedup across all three MoE models\. For Qwen3\-235B\-A22B,EcoSpecimproves the average speedup from EAGLE\-3’s1\.22×1\.22\\timesto1\.36×1\.36\\times, while reducingℰ\\mathcal\{E\}from23\.723\.7to20\.520\.5\. The largest gain appears on MTBench, whereEcoSpecreaches1\.62×1\.62\\timesspeedup\. For GPT\-OSS\-120B,EcoSpecincreases the average speedup from1\.14×1\.14\\timesto1\.31×1\.31\\timesand reducesℰ\\mathcal\{E\}from11\.611\.6to10\.610\.6\. On MMStar and AMC22\-24, the speedup reaches1\.45×1\.45\\times\. For DeepSeek\-V3\.1,EcoSpecimproves the average speedup from MTP’s1\.10×1\.10\\timesto1\.15×1\.15\\times, while reducingℰ\\mathcal\{E\}from31\.431\.4to31\.231\.2\.

#### Sampling decoding \(T=1T\{=\}1\)\.

Under sampling, draft candidates become more diverse, butEcoSpeccontinues to reduce expert activation and improve speedup\. For Qwen3\-235B\-A22B, the average speedup increases from EAGLE\-3’s1\.28×1\.28\\timesto1\.38×1\.38\\times, whileℰ\\mathcal\{E\}decreases from24\.024\.0to20\.920\.9\. For GPT\-OSS\-120B, the average speedup improves from1\.18×1\.18\\timesto1\.30×1\.30\\times, withℰ\\mathcal\{E\}reduced from11\.911\.9to10\.910\.9\. For DeepSeek\-V3\.1,EcoSpecimproves the average speedup from MTP’s1\.28×1\.28\\timesto1\.33×1\.33\\timesand reducesℰ\\mathcal\{E\}from31\.431\.4to30\.930\.9\. These results show that the acceptance–cost trade\-off remains effective under both greedy and sampling\-based target decoding regimes\.

#### Backbone\-dependent gains\.

The magnitude of the gain varies across MoE backbones\. Qwen3\-235B\-A22B and GPT\-OSS\-120B show larger reductions in active experts, suggesting more opportunity for expert reuse during draft selection\. DeepSeek\-V3\.1 shows smaller expert\-footprint reductions, which is consistent with its more balanced routing pattern analyzed in Appendix[D](https://arxiv.org/html/2607.12696#A4)\. Nevertheless, because DeepSeek\-V3\.1 has a large per\-expert memory footprint, even a reduction of0\.20\.2experts per layer can translate into approximately0\.50\.5GB less expert\-weight traffic per speculative step\. Thus,EcoSpecremains beneficial even when the available expert\-reuse headroom is smaller\.

### 5\.3Latency Breakdown and Overhead Analysis

To understand whyEcoSpecimproves performance, we analyze the HBM traffic and latency composition of a speculative decoding step in Table[2](https://arxiv.org/html/2607.12696#S5.T2)and Table[3](https://arxiv.org/html/2607.12696#S5.T3)\. Following the evaluation scope defined in §[5\.1](https://arxiv.org/html/2607.12696#S5.SS1), this subsection analyzes the latency and HBM\-traffic behavior of theT=0T\{=\}0runs\. We decompose the total speculative\-step latency into three parts: \(1\) draft generation \(TdraftT\_\{\\text\{draft\}\}\), \(2\) expert prediction overhead introduced byEcoSpec\(TpredT\_\{\\text\{pred\}\}\), and \(3\) target\-model verification of the selected draft tokens \(TverifyT\_\{\\text\{verify\}\}\)\.EcoSpecdoes not reduce the computation required by each activated expert\. Instead, its speedup comes from reducing expert\-weight memory traffic during verification\.

#### HBM Read Traffic\.

We estimate HBM read traffic during verification from the activated expert footprint and the model\-specific expert size\. As shown in Table[2](https://arxiv.org/html/2607.12696#S5.T2),EcoSpecreduces the estimated HBM reads from99\.399\.3GB to88\.188\.1GB on Qwen3\-235B\-A22B, saving11\.211\.2GB per speculative step\. For GPT\-OSS\-120B, the estimated reads decrease from6\.06\.0GB to5\.55\.5GB\. For DeepSeek\-V3\.1, the reduction is smaller, from97\.397\.3GB to96\.896\.8GB, which is consistent with its smaller expert\-reuse headroom\. These reductions correspond to the lower verification latency reported in Table[3](https://arxiv.org/html/2607.12696#S5.T3)\.

#### Verification Latency and Predictor Overhead\.

Table[3](https://arxiv.org/html/2607.12696#S5.T3)shows thatEcoSpecreduces verification latency across all three MoE models while adding only a small predictor overhead\. On Qwen3\-235B\-A22B, the average active expert count decreases from23\.723\.7to20\.520\.5, andTverifyT\_\{\\text\{verify\}\}decreases from0\.8320\.832s to0\.7300\.730s\. On GPT\-OSS\-120B, the active expert count decreases from11\.611\.6to10\.610\.6, andTverifyT\_\{\\text\{verify\}\}decreases from0\.1130\.113s to0\.0900\.090s\. On DeepSeek\-V3\.1, the expert\-footprint reduction is smaller \(31\.4→31\.231\.4\\rightarrow 31\.2\), butTverifyT\_\{\\text\{verify\}\}still decreases from0\.9800\.980s to0\.9300\.930s\. Across all three models, the predictor overhead is about44ms per speculative step, which is small relative to the reduction in verification latency\. The acceptance length changes only slightly, so the speedup is mainly explained by reduced verification cost rather than higher acceptance length\.

### 5\.4Ablation Study: Impact of Marginal Cost Scoring

We study the effect of marginal\-cost scoring on GSM8K across all three target models\. We compareEcoSpecwith a staticGlobal Costvariant, where each node is scored by the accumulated predicted expert footprint along its path:

Sglobal​\(ti\)=P​\(ti\)\|ℰt​r​a​j​\(ti\)\|\+ϵ\.S\_\{\\mathrm\{global\}\}\(t\_\{i\}\)=\\frac\{P\(t\_\{i\}\)\}\{\|\\mathcal\{E\}\_\{traj\}\(t\_\{i\}\)\|\+\\epsilon\}\.\(10\)UnlikeEcoSpec, this variant does not update the expert buffer during selection and therefore cannot discount experts that are already covered by previously selected nodes\. As a result, deeper nodes tend to receive larger accumulated costs even when they reuse experts from earlier selected paths\.

Table[4](https://arxiv.org/html/2607.12696#S5.T4)shows that marginal\-cost scoring consistently improves speedup over the static global\-cost variant\. On Qwen3\-235B\-A22B,EcoSpecincreases speedup from1\.28×1\.28\\timesto1\.39×1\.39\\timesand mean acceptance length from2\.212\.21to2\.542\.54, while reducing the average active experts from21\.521\.5to21\.021\.0\. A similar pattern appears on GPT\-OSS\-120B, where speedup improves from1\.06×1\.06\\timesto1\.11×1\.11\\timesand acceptance length increases from1\.351\.35to1\.521\.52\. On DeepSeek\-V3\.1, the gain is smaller but still consistent\. These results indicate that updating the buffer during selection helpsEcoSpecidentify candidates that extend accepted paths while reusing already covered experts\.

Table 4:Ablation study on GSM8K\. We compare the static global\-cost variant withEcoSpec’s marginal\-cost scoring\.
### 5\.5Additional Baseline Evaluation

The main experiments compareEcoSpecwith the corresponding speculative decoding baselines for each target model: MTP for DeepSeek\-V3\.1, and EAGLE\-3 for Qwen3\-235B\-A22B and GPT\-OSS\-120B\. To provide an additional baseline comparison, we further include Group Tree Optimization \(GTO\)\(Huet al\.,[2026](https://arxiv.org/html/2607.12696#bib.bib6)\), which improves EAGLE\-style draft models by better aligning draft training with tree\-based decoding\. GTO andEcoSpecact on different stages of the speculative decoding pipeline: GTO improves draft generation, whileEcoSpecchanges the pre\-verification selection of draft nodes\.

For this experiment, we start from the released EAGLE\-3 draft model and continue training it with the GTO procedure\. The resulting GTO\-trained drafter is used to generate candidate trees under the standard speculative verification workflow\. We then evaluate GTO\+EcoSpecby applying the same cost\-aware draft\-selection strategy at the verification stage\.

Table 5:Additional baseline evaluation with GTO\. We report end\-to\-end speedup and acceptance length for GTO and GTO\+EcoSpec\.As shown in Table[5](https://arxiv.org/html/2607.12696#S5.T5), GTO\+EcoSpecachieves higher end\-to\-end speedup than GTO on all evaluated settings\. This additional baseline comparison further supports the effectiveness ofEcoSpec\.

## 6Conclusion

We presentedEcoSpec, a cost\-aware speculative decoding framework for large\-scale MoE models\.EcoSpecaddresses expert scattering during speculative verification by incorporating predicted marginal expert activation cost into draft\-tree selection\. With a lightweight expert predictor and a dynamic expert buffer,EcoSpecselects draft tokens that preserve acceptance likelihood while reducing the growth of the verification expert footprint, without modifying the target\-model verification rule\. Experiments on DeepSeek\-V3\.1, Qwen3\-235B\-A22B, and GPT\-OSS\-120B show consistent speedups and reduced active experts across reasoning, coding, and dialogue benchmarks, with up to1\.62×1\.62\\timesspeedup\. These results highlight the importance of accounting for MoE\-specific expert activation costs when applying speculative decoding to large\-scale sparse models\.

## References

- \[1\]Cited by:[Appendix E](https://arxiv.org/html/2607.12696#A5.SS0.SSS0.Px3.p1.1),[§5\.1](https://arxiv.org/html/2607.12696#S5.SS1.SSS0.Px1.p1.1)\.
- T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 1877–1901\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p1.1)\.
- T\. Cai, Y\. Li, Z\. Geng, H\. Peng, J\. D\. Lee, D\. Chen, and T\. Dao \(2024\)Medusa: simple LLM inference acceleration framework with multiple decoding heads\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=PEpbUobfJv)Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p2.1),[§2](https://arxiv.org/html/2607.12696#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Chen, S\. Borgeaud, G\. Irving, J\. Lespiau, L\. Sifre, and J\. Jumper \(2023\)Accelerating large language model decoding with speculative sampling\.External Links:2302\.01318,[Link](https://arxiv.org/abs/2302.01318)Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p2.1),[§2](https://arxiv.org/html/2607.12696#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Chen, Z\. Wen, T\. Wu, X\. Zhang, and C\. Wu \(2025\)SP\-moe: speculative decoding and prefetching for accelerating moe\-based model inference\.External Links:2510\.10302,[Link](https://arxiv.org/abs/2510.10302)Cited by:[§2](https://arxiv.org/html/2607.12696#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Chen, J\. Li, X\. Dong, P\. Zhang, Y\. Zang, Z\. Chen, H\. Duan, J\. Wang, Y\. Qiao, D\. Lin, and F\. Zhao \(2024\)Are we on the right way for evaluating large vision\-language models?\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=evP9mxNNxJ)Cited by:[Appendix E](https://arxiv.org/html/2607.12696#A5.SS0.SSS0.Px6.p1.1),[§5\.1](https://arxiv.org/html/2607.12696#S5.SS1.SSS0.Px1.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.External Links:2107\.03374,[Link](https://arxiv.org/abs/2107.03374)Cited by:[Appendix E](https://arxiv.org/html/2607.12696#A5.SS0.SSS0.Px4.p1.1),[§5\.1](https://arxiv.org/html/2607.12696#S5.SS1.SSS0.Px1.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.External Links:2110\.14168,[Link](https://arxiv.org/abs/2110.14168)Cited by:[Appendix E](https://arxiv.org/html/2607.12696#A5.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2607.12696#S3.p4.1),[§5\.1](https://arxiv.org/html/2607.12696#S5.SS1.SSS0.Px1.p1.1)\.
- T\. Dao, D\. Y\. Fu, S\. Ermon, A\. Rudra, and C\. Re \(2022\)FlashAttention: fast and memory\-efficient exact attention with IO\-awareness\.InAdvances in Neural Information Processing Systems,A\. H\. Oh, A\. Agarwal, D\. Belgrave, and K\. Cho \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=H4DqfPSibmx)Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p1.1)\.
- DeepSeek\-AI, A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang,et al\.\(2025\)DeepSeek\-v3 technical report\.External Links:2412\.19437,[Link](https://arxiv.org/abs/2412.19437)Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p3.1),[§2](https://arxiv.org/html/2607.12696#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.12696#S2.SS0.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2607.12696#S5.SS1.SSS0.Px1.p1.1)\.
- N\. Du, Y\. Huang, A\. M\. Dai, S\. Tong, D\. Lepikhin, Y\. Xu, M\. Krikun, Y\. Zhou, A\. W\. Yu, O\. Firat, B\. Zoph, L\. Fedus, M\. P\. Bosma, Z\. Zhou, T\. Wang, E\. Wang, K\. Webster, M\. Pellat, K\. Robinson, K\. Meier\-Hellstern, T\. Duke, L\. Dixon, K\. Zhang, Q\. Le, Y\. Wu, Z\. Chen, and C\. Cui \(2022\)GLaM: efficient scaling of language models with mixture\-of\-experts\.InProceedings of the 39th International Conference on Machine Learning,K\. Chaudhuri, S\. Jegelka, L\. Song, C\. Szepesvari, G\. Niu, and S\. Sabato \(Eds\.\),Proceedings of Machine Learning Research, Vol\.162,pp\. 5547–5569\.External Links:[Link](https://proceedings.mlr.press/v162/du22c.html)Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p3.1)\.
- W\. Fedus, B\. Zoph, and N\. Shazeer \(2022\)Switch transformers: scaling to trillion parameter models with simple and efficient sparsity\.J\. Mach\. Learn\. Res\.23\(1\)\.External Links:ISSN 1532\-4435Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p3.1),[§2](https://arxiv.org/html/2607.12696#S2.SS0.SSS0.Px2.p1.1)\.
- T\. Gale, D\. Narayanan, C\. Young, and M\. Zaharia \(2023\)MegaBlocks: Efficient Sparse Training with Mixture\-of\-Experts\.Proceedings of Machine Learning and Systems5\.Cited by:[§2](https://arxiv.org/html/2607.12696#S2.SS0.SSS0.Px2.p1.1)\.
- F\. Gloeckle, B\. Y\. Idrissi, B\. Rozière, D\. Lopez\-Paz, and G\. Synnaeve \(2024\)Better & faster large language models via multi\-token prediction\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[§2](https://arxiv.org/html/2607.12696#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:ISSN 1476\-4687,[Link](http://dx.doi.org/10.1038/s41586-025-09422-z),[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[§5\.1](https://arxiv.org/html/2607.12696#S5.SS1.SSS0.Px4.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the MATH dataset\.InThirty\-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track \(Round 2\),External Links:[Link](https://openreview.net/forum?id=7Bywt2mQsCe)Cited by:[Appendix E](https://arxiv.org/html/2607.12696#A5.SS0.SSS0.Px3.p1.1),[§5\.1](https://arxiv.org/html/2607.12696#S5.SS1.SSS0.Px1.p1.1)\.
- S\. Hu, J\. Li, Z\. Lu, and P\. Zhou \(2026\)Bridging draft policy misalignment: group tree optimization for speculative decoding\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=dwPdYFqVWO)Cited by:[§5\.5](https://arxiv.org/html/2607.12696#S5.SS5.p1.1)\.
- H\. Huang, N\. Ardalani, A\. Sun, L\. Ke, S\. Bhosale, H\. S\. Lee, C\. Wu, and B\. Lee \(2024\)Toward efficient inference for mixture of experts\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=stXtBqyTWX)Cited by:[§2](https://arxiv.org/html/2607.12696#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Huang, L\. Zhu, Z\. Zhan, T\. Hu, W\. Mao, X\. Yu, Y\. Liu, and T\. Zhang \(2026\)MoESD: unveil speculative decoding’s potential for accelerating sparse moe\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=FAeU7516MR)Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p4.1)\.
- D\. Lepikhin, H\. Lee, Y\. Xu, D\. Chen, O\. Firat, Y\. Huang, M\. Krikun, N\. Shazeer, and Z\. Chen \(2021\)\{gs\}hard: scaling giant models with conditional computation and automatic sharding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=qrwe7XHTmYb)Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p3.1)\.
- Y\. Leviathan, M\. Kalman, and Y\. Matias \(2023\)Fast inference from transformers via speculative decoding\.InProceedings of the 40th International Conference on Machine Learning,ICML’23\.Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p2.1),[§1](https://arxiv.org/html/2607.12696#S1.p5.3),[§2](https://arxiv.org/html/2607.12696#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Li, F\. Wei, C\. Zhang, and H\. Zhang \(2024a\)EAGLE\-2: faster inference of language models with dynamic draft trees\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 7421–7432\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.422/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.422)Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p2.1),[§2](https://arxiv.org/html/2607.12696#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2607.12696#S4.SS2.p1.1)\.
- Y\. Li, F\. Wei, C\. Zhang, and H\. Zhang \(2024b\)EAGLE: speculative sampling requires rethinking feature uncertainty\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=1NdN7eXyb4)Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p2.1),[§2](https://arxiv.org/html/2607.12696#S2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2607.12696#S5.SS1.SSS0.Px3.p1.8)\.
- Y\. Li, F\. Wei, C\. Zhang, and H\. Zhang \(2026\)EAGLE\-3: scaling up inference acceleration of large language models via training\-time test\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=4exx1hUffq)Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p2.1),[§2](https://arxiv.org/html/2607.12696#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2024\)Let’s verify step by step\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=v8L0pN6EOi)Cited by:[Appendix E](https://arxiv.org/html/2607.12696#A5.SS0.SSS0.Px3.p1.1),[§5\.1](https://arxiv.org/html/2607.12696#S5.SS1.SSS0.Px1.p1.1)\.
- J\. Liu, P\. Tang, W\. Wang, Y\. Ren, X\. Hou, P\. A\. Heng, M\. Guo, and C\. Li \(2026\)A survey on inference optimization techniques for mixture of experts models\.ACM Comput\. Surv\.58\(10\)\.External Links:ISSN 0360\-0300,[Link](https://doi.org/10.1145/3794845),[Document](https://dx.doi.org/10.1145/3794845)Cited by:[§2](https://arxiv.org/html/2607.12696#S2.SS0.SSS0.Px2.p1.1)\.
- MAA \(2025\)American invitational mathematics examination \- AIME\.Note:Hugging FaceExternal Links:[Link](https://huggingface.co/datasets/opencompass/AIME2025)Cited by:[Appendix E](https://arxiv.org/html/2607.12696#A5.SS0.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2607.12696#S5.SS1.SSS0.Px1.p1.1)\.
- B\. McDanel, S\. Li, S\. Surineni, and H\. Khaitan \(2026\)MoE\-spec: expert budgeting for efficient speculative decoding\.External Links:2602\.16052,[Link](https://arxiv.org/abs/2602.16052)Cited by:[§2](https://arxiv.org/html/2607.12696#S2.SS0.SSS0.Px2.p1.1)\.
- X\. Miao, G\. Oliaro, Z\. Zhang, X\. Cheng, Z\. Wang, Z\. Zhang, R\. Y\. Y\. Wong, A\. Zhu, L\. Yang, X\. Shi, C\. Shi, Z\. Chen, D\. Arfeen, R\. Abhyankar, and Z\. Jia \(2024\)SpecInfer: accelerating large language model serving with tree\-based speculative inference and verification\.InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3,ASPLOS ’24,New York, NY, USA,pp\. 932–949\.External Links:ISBN 9798400703867,[Link](https://doi.org/10.1145/3620666.3651335),[Document](https://dx.doi.org/10.1145/3620666.3651335)Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p2.1),[§2](https://arxiv.org/html/2607.12696#S2.SS0.SSS0.Px1.p1.1)\.
- OpenAI, :, S\. Agarwal, L\. Ahmad, J\. Ai, S\. Altman, A\. Applebaum, E\. Arbus, R\. K\. Arora, Y\. Bai,et al\.\(2025\)Gpt\-oss\-120b & gpt\-oss\-20b model card\.External Links:2508\.10925,[Link](https://arxiv.org/abs/2508.10925)Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p3.1),[§5\.1](https://arxiv.org/html/2607.12696#S5.SS1.SSS0.Px1.p1.1)\.
- OpenAI, J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman,et al\.\(2024\)GPT\-4 technical report\.External Links:2303\.08774,[Link](https://arxiv.org/abs/2303.08774)Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p1.1)\.
- S\. Rajbhandari, J\. Rasley, O\. Ruwase, and Y\. He \(2020\)ZeRO: memory optimizations toward training trillion parameter models\.InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis,SC ’20\.External Links:ISBN 9781728199986Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p1.1)\.
- N\. Shazeer, \*\. Mirhoseini, \*\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. Dean \(2017\)Outrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=B1ckMDqlg)Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p3.1),[§2](https://arxiv.org/html/2607.12696#S2.SS0.SSS0.Px2.p1.1)\.
- N\. Shazeer \(2019\)Fast transformer decoding: one write\-head is all you need\.External Links:1911\.02150,[Link](https://arxiv.org/abs/1911.02150)Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems,I\. Guyon, U\. V\. Luxburg, S\. Bengio, H\. Wallach, R\. Fergus, S\. Vishwanathan, and R\. Garnett \(Eds\.\),Vol\.30,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p1.1)\.
- W\. Wang, J\. Liu, X\. Hou, X\. Xia, P\. Tang, M\. Zhang, C\. Li, and M\. Guo \(2025\)MoE\-speq: speculative quantized decoding with proactive expert prefetching and offloading for mixture\-of\-experts\.External Links:2511\.14102,[Link](https://arxiv.org/abs/2511.14102)Cited by:[§2](https://arxiv.org/html/2607.12696#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Williams, A\. Waterman, and D\. Patterson \(2009\)Roofline: an insightful visual performance model for multicore architectures\.Commun\. ACM52\(4\),pp\. 65–76\.External Links:ISSN 0001\-0782,[Link](https://doi.org/10.1145/1498765.1498785),[Document](https://dx.doi.org/10.1145/1498765.1498785)Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p1.1)\.
- T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz, J\. Davison, S\. Shleifer, P\. von Platen, C\. Ma, Y\. Jernite, J\. Plu, C\. Xu, T\. L\. Scao, S\. Gugger, M\. Drame, Q\. Lhoest, and A\. M\. Rush \(2020\)Transformers: state\-of\-the\-art natural language processing\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,Online,pp\. 38–45\.External Links:[Link](https://aclanthology.org/2020.emnlp-demos.6/)Cited by:[§5\.1](https://arxiv.org/html/2607.12696#S5.SS1.SSS0.Px5.p1.1)\.
- F\. Xue, Z\. Zheng, Y\. Fu, J\. Ni, Z\. Zheng, W\. Zhou, and Y\. You \(2024\)OpenMoE: an early effort on open mixture\-of\-experts language models\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p4.1)\.
- L\. Xue, Y\. Fu, Z\. Lu, L\. Mai, and M\. Marina \(2025\)MoE\-infinity: efficient moe inference on personal machines with sparsity\-aware expert cache\.External Links:2401\.14361,[Link](https://arxiv.org/abs/2401.14361)Cited by:[§2](https://arxiv.org/html/2607.12696#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§1](https://arxiv.org/html/2607.12696#S1.p3.1),[§5\.1](https://arxiv.org/html/2607.12696#S5.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2607.12696#S5.SS1.SSS0.Px4.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.InThirty\-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=uccHPGDlao)Cited by:[Appendix E](https://arxiv.org/html/2607.12696#A5.SS0.SSS0.Px5.p1.1),[§5\.1](https://arxiv.org/html/2607.12696#S5.SS1.SSS0.Px1.p1.1)\.

Algorithm 1Cost\-Aware Draft Selection inEcoSpecInput:Draft Tree

𝒯\\mathcal\{T\}, Budget

γ\\gamma, Predictor

Πθ\\Pi\_\{\\theta\}, Initial Buffer

ℬ0\\mathcal\{B\}\_\{0\}
Output:Selected tokens

𝒮\\mathcal\{S\}
𝒮←∅\\mathcal\{S\}\\leftarrow\\emptyset

ℬ←ℬ0\\mathcal\{B\}\\leftarrow\\mathcal\{B\}\_\{0\}

Construct the verification subset before target\-model verification\.

while

\|𝒮\|<γ\|\\mathcal\{S\}\|<\\gammado

foreach unselected node

ti∈𝒯t\_\{i\}\\in\\mathcal\{T\}do

ℰt​r​a​j​\(ti\)←⋃τ∈Path​\(r​o​o​t→ti\)ℰp​r​e​d​\(τ\)\\mathcal\{E\}\_\{traj\}\(t\_\{i\}\)\\leftarrow\\bigcup\_\{\\tau\\in\\text\{Path\}\(root\\to t\_\{i\}\)\}\\mathcal\{E\}\_\{pred\}\(\\tau\)

C​o​s​ti←\|ℰt​r​a​j​\(ti\)∖ℬ\|Cost\_\{i\}\\leftarrow\|\\mathcal\{E\}\_\{traj\}\(t\_\{i\}\)\\setminus\\mathcal\{B\}\|

S​c​o​r​ei←P​\(ti\)/\(C​o​s​ti\+ϵ\)Score\_\{i\}\\leftarrow P\(t\_\{i\}\)/\(Cost\_\{i\}\+\\epsilon\)

endfor

t∗←arg⁡maxti⁡S​c​o​r​eit^\{\*\}\\leftarrow\\arg\\max\_\{t\_\{i\}\}Score\_\{i\}

𝒮←𝒮∪\{t∗\}\\mathcal\{S\}\\leftarrow\\mathcal\{S\}\\cup\\\{t^\{\*\}\\\}

ℬ←ℬ∪ℰt​r​a​j​\(t∗\)\\mathcal\{B\}\\leftarrow\\mathcal\{B\}\\cup\\mathcal\{E\}\_\{traj\}\(t^\{\*\}\)\{Update buffer for subsequent scoring\}

endwhile

return

𝒮\\mathcal\{S\}\{Verify all selected nodes in one target forward pass\}

## Appendix AImpact of Batch Size on Scalability

### A\.1End\-to\-End Batch\-Size Scaling

Batching is a common way to improve GPU utilization during inference\. To examine how batching affectsEcoSpec, we compare it with EAGLE\-3 under batch sizesB=\{1,2,4\}B=\\\{1,2,4\\\}and report end\-to\-end speedup together with active expert counts\.

As shown in Tables[6](https://arxiv.org/html/2607.12696#A1.T6)and[7](https://arxiv.org/html/2607.12696#A1.T7), both methods exhibit lower speedup as batch size increases\. On Qwen3\-235B\-A22B, EAGLE\-3’s average speedup drops from1\.22×1\.22\\timesatB=1B=1to1\.00×1\.00\\timesatB=4B=4, whileEcoSpecmaintains a higher average speedup of1\.09×1\.09\\timesatB=4B=4\. A similar trend appears on GPT\-OSS\-120B, where EAGLE\-3 drops from1\.14×1\.14\\timesto1\.00×1\.00\\times, whileEcoSpecretains1\.06×1\.06\\timesatB=4B=4\.

Active expert counts help explain this speedup gap\. As batch size increases, each verification step covers more candidate tokens in parallel, expanding the union of activated experts and increasing verification cost\. AtB=4B=4on Qwen3\-235B\-A22B, EAGLE\-3 activates an average of54\.954\.9experts per step, whileEcoSpecreduces this number to45\.745\.7\. On GPT\-OSS\-120B, the corresponding counts are32\.632\.6for EAGLE\-3 and28\.128\.1forEcoSpec\. This smaller expert footprint lowers verification cost relative to EAGLE\-3, allowingEcoSpecto retain higher end\-to\-end speedup under batching\.

Table 6:Batch\-size scaling on Qwen3\-235B\-A22B\. We report end\-to\-end speedup relative to AR decoding and the average number of active experts\.Table 7:Batch\-size scaling on GPT\-OSS\-120B\. We report end\-to\-end speedup relative to AR decoding and the average number of active experts\.
### A\.2Batch\-Size Scaling of Verification Cost

We further examine larger\-batch behavior using a verification\-cost\-oriented metric on 100 randomly sampled instances from the seven evaluation datasets\. In Table[8](https://arxiv.org/html/2607.12696#A1.T8), we report

ρ=TARTverify,\\rho=\\frac\{T\_\{\\mathrm\{AR\}\}\}\{T\_\{\\mathrm\{verify\}\}\},whereTART\_\{\\mathrm\{AR\}\}denotes the average AR decoding time andTverifyT\_\{\\mathrm\{verify\}\}denotes the average speculative verification time per step\. A largerρ\\rhoindicates that speculative verification is cheaper relative to AR decoding\. We also report BS=8 Spd, the throughput of each speculative method normalized by AR throughput at batch size 8\.

The ratioρ\\rhodecreases for both methods as batch size increases\. On Qwen3\-235B\-A22B,ρ\\rhodrops from0\.59×0\.59\\timesto0\.32×0\.32\\timesfor EAGLE\-3 and from0\.67×0\.67\\timesto0\.33×0\.33\\timesforEcoSpec\. On GPT\-OSS\-120B, it drops from0\.72×0\.72\\timesto0\.43×0\.43\\timesfor EAGLE\-3 and from0\.90×0\.90\\timesto0\.44×0\.44\\timesforEcoSpec\. At BS=8, both speculative methods fall below AR throughput in this prototype, with BS=8 speedup of0\.69×0\.69\\timeson Qwen3\-235B\-A22B and0\.65×0\.65\\timeson GPT\-OSS\-120B\.

These results indicate that large\-batch speculative decoding remains challenging in this prototype\. Nevertheless,EcoSpecconsistently maintains a higherTAR/TverifyT\_\{\\mathrm\{AR\}\}/T\_\{\\mathrm\{verify\}\}ratio than EAGLE\-3 across the tested batch sizes, indicating lower verification cost under the same setting\.

Table 8:Batch\-size scaling of verification cost\. We reportρ=TAR/Tverify\\rho=T\_\{\\mathrm\{AR\}\}/T\_\{\\mathrm\{verify\}\}, the ratio between average AR decoding time and average speculative verification time per step\. BS=8 Spd reports throughput normalized by AR throughput at batch size 8\.

## Appendix BPredictor Details and Analysis

### B\.1Predictor Training Details

EcoSpectrains a lightweight expert predictor for each target MoE model to estimate expert activation cost before target\-model verification\. The predictor is used only for cost\-aware draft selection\. It does not participate in token verification, does not query the target MoE model online, and does not modify the standard speculative verification rule\.

#### Predictor backbones\.

We instantiateΠθ\\Pi\_\{\\theta\}with small open\-source LMs and fine\-tune them to predict the target model’s per\-layer expert activations from token\-level inputs\. For DeepSeek\-V3\.1, we use DeepSeek\-R1\-Distill\-Qwen\-1\.5B as the predictor backbone\. For Qwen3\-235B\-A22B, we use Qwen3\-0\.6B, leveraging architectural proximity within the Qwen family\. For GPT\-OSS\-120B, since no lightweight model from the same series is publicly available, we also use Qwen3\-0\.6B\. The predictor cost is small relative to target\-model verification, as reflected in Table[3](https://arxiv.org/html/2607.12696#S5.T3)\.

#### Training data and setup\.

We collect routing traces offline by running each target MoE model on seven datasets spanning reasoning, coding, and dialogue: HumanEval, MMStar, MT\-Bench, AMC22\-24, GSM8K, AIME\-25, and Math500\. For each token and MoE layer, we record the target router’s Top\-KKselected experts as supervision, whereKKfollows the target model’s routing configuration\. We randomly split samples into 80% for training and 20% for testing\. All evaluations reported in the main paper are conducted on the held\-out test split\. We fine\-tune predictors for 100 epochs using Adam with learning rate1×10−51\\times 10^\{\-5\}and batch size 16\.

#### Routing prediction accuracy\.

We report Top\-KKrouting prediction accuracy on the test split, computed as the average overlap ratio between the predicted Top\-KKexperts and the target router’s Top\-KKexperts across tokens and MoE layers\. The resulting accuracies are 80% for DeepSeek\-V3\.1, 82% for Qwen3\-235B\-A22B, and 93% for GPT\-OSS\-120B\. The higher accuracy on GPT\-OSS\-120B is consistent with its Top\-4\-over\-128 routing setting, which yields more concentrated activation patterns than the Top\-8 routing used by DeepSeek\-V3\.1 and Qwen3\-235B\-A22B\. Since EcoSpec uses the predictor only as a cost\-estimation module, exact routing reconstruction is not required; the sensitivity to predictor quality is further evaluated in Appendix[B\.2](https://arxiv.org/html/2607.12696#A2.SS2)\.

### B\.2Impact of Predictor Accuracy

To examine the effect of predictor quality, we evaluateEcoSpecwith three predictor checkpoints: two intermediate checkpoints with Top\-KKrouting accuracy of approximately 50% and 60%, and the converged checkpoint used in the main experiments\. The decoding configuration is fixed across all runs, including the default verification budgetγ=4\\gamma=4; only the predictor checkpoint is changed\.

Table[9](https://arxiv.org/html/2607.12696#A2.T9)reports end\-to\-end speedup, acceptance length, and active expert count on Qwen3\-235B\-A22B and GPT\-OSS\-120B\. Lower\-accuracy predictors lead to lower speedup and higher active expert counts\. On Qwen3\-235B\-A22B, the converged predictor improves the speedup from1\.17×1\.17\\timesto1\.39×1\.39\\timeson GSM8K and from1\.20×1\.20\\timesto1\.62×1\.62\\timeson MT\-Bench compared with the 50%\-accuracy checkpoint\. On GPT\-OSS\-120B, the corresponding speedup improves from1\.01×1\.01\\timesto1\.11×1\.11\\timeson GSM8K and from1\.07×1\.07\\timesto1\.30×1\.30\\timeson MT\-Bench\. The active expert count also decreases consistently as predictor quality improves\.

These results show that predictor accuracy affects the quality of cost\-aware draft selection\. The converged predictor gives the best speedup in all reported settings, while intermediate predictors still retain positive speedup under the same decoding configuration\.

Table 9:Robustness to Predictor Accuracy\. Predictor\-accuracy sensitivity\. We vary the predictor checkpoint while keeping the decoding configuration fixed\.
### B\.3Oracle Expert\-Set Analysis

We further analyze whether the learned predictor provides a sufficiently accurate expert\-cost signal for cost\-aware draft selection\. In this analysis, we keep theEcoSpecselection rule unchanged and replace the predicted expert sets with ground\-truth expert sets collected offline from the target MoE model\. This isolates the effect of predictor error while keeping the selection rule fixed\.

The oracle expert sets are used only for analysis\. In normal speculative decoding, the true expert sets of candidate draft nodes are unavailable before target\-model verification unless additional target\-model computation is performed\. Therefore, this setting is reported with verification\-oriented metrics rather than end\-to\-end throughput, and is not used as an online inference baseline\.

Table 10:Oracle expert\-set analysis\. Oracle uses ground\-truth target\-router activations collected offline, while Predictor uses the learned expert predictor\. All entries report verification\-oriented metrics under the sameEcoSpecselection rule\.As shown in Table[10](https://arxiv.org/html/2607.12696#A2.T10), replacing the learned predictor with oracle expert sets produces nearly unchanged acceptance lengths, verification times, and active expert counts\. This indicates that the converged predictor provides a cost signal close to that obtained from oracle expert sets in the evaluated settings\.

This analysis complements the predictor\-accuracy study\. The accuracy study shows that lower\-quality predictors can reduce the effectiveness of cost\-aware selection, while the oracle expert\-set analysis shows that replacing the converged predictor with ground\-truth expert sets yields little additional change\. Together, these results indicate that predictor quality matters, but the converged predictor is already sufficiently accurate for the selection rule used byEcoSpec\.

## Appendix CEvaluation under Different Verification Budgets \(γ\\gamma\)

We vary the verification budgetγ∈\{3,4,5,6\}\\gamma\\in\\\{3,4,5,6\\\}to support the common default setting used in the main experiments\. Tables[11](https://arxiv.org/html/2607.12696#A3.T11)and[12](https://arxiv.org/html/2607.12696#A3.T12)report end\-to\-end speedup, acceptance length, and active expert count for EAGLE\-3 andEcoSpecunder each budget\.

We select the default verification budget according to average end\-to\-end speedup across the seven datasets\. Under this criterion,γ=4\\gamma=4gives the highest average end\-to\-end speedup for both EAGLE\-3 andEcoSpecon Qwen3\-235B\-A22B and GPT\-OSS\-120B\. Increasingγ\\gammabeyond44further increases acceptance length in many cases, but does not improve the averaged end\-to\-end speedup\.

The sweep also shows thatEcoSpecmaintains its advantage over EAGLE\-3 across the tested budgets\. On both target models,EcoSpecconsistently achieves higher average end\-to\-end speedup than EAGLE\-3 underγ∈\{3,4,5,6\}\\gamma\\in\\\{3,4,5,6\\\}\. It also uses fewer active experts on average under the same verification budget, showing that the benefit of cost\-aware draft selection is not tied to a single budget choice\.

Therefore, we useγ=4\\gamma=4as the default verification budget for both EAGLE\-3 andEcoSpecin the main experiments\.

Table 11:Verification\-budget sensitivity on Qwen3\-235B\-A22B\. Each entry reports Spd / Len / Exp, corresponding to end\-to\-end speedup, acceptance length, and active expert count\.Table 12:Verification\-budget sensitivity on GPT\-OSS\-120B\.
## Appendix DExpert Activation Pattern Analysis

We analyze expert\-activation patterns of the evaluated MoE backbones in Fig\.[4](https://arxiv.org/html/2607.12696#A4.F4)\. For each model, the left panel shows the expert\-activation heatmap across layers, and the right panel shows the layer\-wise expert\-load distribution\. This analysis characterizes the expert working sets encountered during speculative verification and provides context for why active expert count is reported alongside acceptance length\.

#### DeepSeek\-V3\.1\.

DeepSeek\-V3\.1 shows a relatively diffuse activation pattern\. The heatmap does not concentrate on a small subset of experts, and the layer\-wise load distributions are relatively narrow across most layers\. This indicates that expert usage is broadly balanced across routed experts\. For speculative verification, such balanced routing still makes the expert working set an important cost component: verifying multiple draft tokens can involve many distinct experts even when no small group of experts dominates the routing pattern\.

#### Qwen3\-235B\-A22B and GPT\-OSS\-120B\.

Qwen3\-235B\-A22B and GPT\-OSS\-120B show more concentrated expert\-activation patterns\. Their heatmaps contain clearer high\-usage regions, and their layer\-wise load distributions have wider ranges with more visible high\-load experts\. These patterns indicate stronger locality in expert usage: different candidate tokens are more likely to share parts of their expert sets, making the verified expert working set sensitive to which draft tokens are selected\.

#### Implication for cost\-aware draft selection\.

These observations support treating the expert working set as an explicit cost component in MoE speculative decoding\. Acceptance length measures how many drafted tokens are verified successfully, but it does not describe which experts are activated during verification\. Two draft sets with similar acceptance length can induce different expert working sets and therefore different verification costs\.EcoSpecincorporates this cost dimension during draft selection by favoring candidates with lower predicted marginal expert cost under the same verification budget\.

![Refer to caption](https://arxiv.org/html/2607.12696v1/src/ds_expert_load_heatmap.png)\(a\)DeepSeek\-V3\.1: Activation Heatmap
![Refer to caption](https://arxiv.org/html/2607.12696v1/src/ds_layer_load_distribution.png)\(b\)DeepSeek\-V3\.1: Load Distribution
![Refer to caption](https://arxiv.org/html/2607.12696v1/src/qwen_expert_load_heatmap.png)\(c\)Qwen3\-235B: Activation Heatmap
![Refer to caption](https://arxiv.org/html/2607.12696v1/src/qwen_layer_load_distribution.png)\(d\)Qwen3\-235B: Load Distribution
![Refer to caption](https://arxiv.org/html/2607.12696v1/src/gpt-oss_expert_load_heatmap.png)\(e\)GPT\-OSS\-120B: Activation Heatmap
![Refer to caption](https://arxiv.org/html/2607.12696v1/src/gpt-oss_layer_load_distribution.png)\(f\)GPT\-OSS\-120B: Load Distribution

Figure 4:Expert activation patterns across evaluated MoE models\. Each row corresponds to one target model\. The left panel shows the expert\-activation heatmap, and the right panel shows the layer\-wise expert\-load distribution\.

## Appendix EDatasets and Evaluation Details

We evaluateEcoSpecon seven benchmarks covering mathematical reasoning, code generation, dialogue, and text\-only inputs derived from a vision\-language benchmark\. All methods are evaluated with the same prompts and decoding settings on each dataset, so the reported speedup, acceptance length, and active expert count are computed under matched input conditions\.

#### GSM8K

GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.12696#bib.bib40)\)contains grade\-school math word problems that require multi\-step arithmetic reasoning\. We use the standard 5\-shot Chain\-of\-Thought setting, where five exemplars are prepended to the input before the test question\.

#### AIME 2025

AIME 2025\(MAA,[2025](https://arxiv.org/html/2607.12696#bib.bib44)\)contains competition\-level mathematical problems from the American Invitational Mathematics Examination\. We use a zero\-shot reasoning prompt and require the final answer to be placed in a boxed format:\{question\}\\n Please reason step by step, and put your final answer within \\boxed\{\}\.

#### MATH500 and AMC22\-24

MATH500\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.12696#bib.bib42); Lightmanet al\.,[2024](https://arxiv.org/html/2607.12696#bib.bib43)\)is a 500\-problem subset of the MATH benchmark\. AMC22\-24\([AI\-MO,](https://arxiv.org/html/2607.12696#bib.bib45)\)contains problems from the American Mathematics Competitions from 2022 to 2024\. For both datasets, we use the following zero\-shot prompt:Problem:\\n \{problem\}\\n\\n Solution:

#### HumanEval

HumanEval\(Chenet al\.,[2021](https://arxiv.org/html/2607.12696#bib.bib41)\)contains 164 Python programming problems with function signatures, docstrings, and unit tests\. We use the benchmark prompts as code\-generation inputs\.

#### MT\-Bench

MT\-Bench\(Zhenget al\.,[2023](https://arxiv.org/html/2607.12696#bib.bib46)\)is a multi\-turn instruction\-following benchmark\.

#### MMStar

MMStar\(Chenet al\.,[2024](https://arxiv.org/html/2607.12696#bib.bib47)\)is a vision\-language benchmark whose original evaluation involves both image and text inputs\. Since this work focuses on text\-only LLM inference, we remove the image inputs and provide only the textual questions to the model\. This setting is used as a text\-only generation workload and is not intended to measure vision\-language grounding ability\.

## Appendix FModel Details

Table[13](https://arxiv.org/html/2607.12696#A6.T13)summarizes the MoE configurations of the target models used in our experiments\. We include the total and active parameter counts, the number of MoE layers, the number of experts per MoE layer, and the routing Top\-kk\. These attributes determine the scale of expert activation during verification and are directly related to the cost\-aware draft selection studied in this paper\.

Table 13:Architectural summary of evaluated MoE models\. Total and active parameters follow the reporting convention of the corresponding model releases\. Top\-kkdenotes the number of selected experts per token\.