FACT:用于推测解码的固定预算分配与跨步骤训练

arXiv cs.LG 论文

摘要

FACT 是一个实验性框架,用于研究草稿模型训练、候选分配和执行条件如何共同影响推测解码的接受率与延迟。研究表明,置信度引导的分配可将 HumanEval 上的加速比从 1.46× 提升至 2.02×,而主机线程设置可能在不改变接受率的情况下,几乎完全抵消加速收益。

arXiv:2611.23463v1 Announce Type: new Abstract: Speculative decoding accelerates language-model generation by verifying multiple draft tokens in a single target-model pass. However, better draft predictions and longer accepted continuations do not necessarily produce faster generation. To examine this gap, we present FACT (Fixed-Budget Allocation and Cross-Step Training), an experimental framework for studying how drafter training, candidate allocation, and execution conditions jointly affect acceptance and latency. Using a lightweight feature-prediction drafter with a frozen Llama-3-8B-Instruct target under greedy decoding, we first examine how training translates into accepted tokens. Backpropagating through predicted-feature rollouts with greater weight on early steps improves acceptance by 15-24% over uniformly weighted detached training, yet average rollout proxy cross-entropy does not reliably rank drafters by acceptance. Beyond prediction quality, the placement of draft candidates also matters: with a fixed 256-node budget and matched per-depth widths, replacing static allocation with confidence-guided allocation increases speedup over autoregressive decoding from $1.46\times$ to $2.02\times$ on HumanEval and from $1.31\times$ to $1.75\times$ on MT-Bench. This allocation benefit extends to an independently trained drafter, but stronger training does not consistently amplify it, showing that the two improvements do not necessarily reinforce each other. Even these speed gains depend on execution conditions: changing only host-thread settings can nearly eliminate speedup without changing acceptance. Together, these findings show why training and allocation must be evaluated jointly using both acceptance and end-to-end latency under controlled execution conditions.
查看原文
查看缓存全文

缓存时间: 2026/11/19 09:00

# 1 Introduction
Source: [https://arxiv.org/html/2611.23463](https://arxiv.org/html/2611.23463)
FACT: Fixed\-Budget Allocation and Cross\-Step Training for Speculative Decoding

Hadi Fares, Mayssa Hajj Hassan, Ali Salman, Ammar Mohanna

American University of Beirut

††footnotetext:Correspondence to:\{hnf01, mhh77, aas145\}@mail\.aub\.edu\. Code:[https://github\.com/mrfares11/specdec](https://github.com/mrfares11/specdec)\.AbstractSpeculative decoding accelerates language\-model generation by verifying multiple draft tokens in a single target\-model pass\. However, better draft predictions and longer accepted continuations do not necessarily produce faster generation\. To examine this gap, we present FACT \(Fixed\-Budget Allocation and Cross\-Step Training\), an experimental framework for studying how drafter training, candidate allocation, and execution conditions jointly affect acceptance and latency\. Using a lightweight feature\-prediction drafter with a frozen Llama\-3\-8B\-Instruct target under greedy decoding, we first examine how training translates into accepted tokens\. Backpropagating through predicted\-feature rollouts with greater weight on early steps improves acceptance by 15–24% over uniformly weighted detached training, yet average rollout proxy cross\-entropy does not reliably rank drafters by acceptance\. Beyond prediction quality, the placement of draft candidates also matters: with a fixed 256\-node budget and matched per\-depth widths, replacing static allocation with confidence\-guided allocation increases speedup over autoregressive decoding from1\.46×1\.46\\timesto2\.02×2\.02\\timeson HumanEval and from1\.31×1\.31\\timesto1\.75×1\.75\\timeson MT\-Bench\. This allocation benefit extends to an independently trained drafter, but stronger training does not consistently amplify it, showing that the two improvements do not necessarily reinforce each other\. Even these speed gains depend on execution conditions: changing only host\-thread settings can nearly eliminate speedup without changing acceptance\. Together, these findings show why training and allocation must be evaluated jointly using both acceptance and end\-to\-end latency under controlled execution conditions\.

Autoregressive language models generate text one token at a time, requiring a new model evaluation for each successive token\. Speculative decoding reduces this sequential cost: a lightweight draft model proposes several candidate tokens, and the larger target model checks them in parallel\. Generation can then advance by multiple tokens per target\-model call while preserving the target’s output distribution under exact arithmetic\([Leviathan et al\., 2023](https://arxiv.org/html/2611.23463#bib.bib7);[Chen et al\., 2023](https://arxiv.org/html/2611.23463#bib.bib2)\)\.

Realizing this promise requires useful proposals at a sufficiently low cost\. EAGLE improves drafting by reusing the target model’s internal features\([Li et al\., 2024a](https://arxiv.org/html/2611.23463#bib.bib8)\), while DistillSpec and HASS improve alignment with the target’s predictions and the inputs encountered during decoding\([Zhou et al\., 2024](https://arxiv.org/html/2611.23463#bib.bib15);[Zhang et al\., 2025](https://arxiv.org/html/2611.23463#bib.bib13)\)\. Because the next token can remain uncertain, tree\-based methods explore several continuations in parallel\([Miao et al\., 2024](https://arxiv.org/html/2611.23463#bib.bib11)\)\. Building on this flexibility, EAGLE\-2 adapts candidate allocation to the context\([Li et al\., 2024b](https://arxiv.org/html/2611.23463#bib.bib9)\), while Sequoia also considers hardware when selecting tree size and depth\([Chen et al\., 2024](https://arxiv.org/html/2611.23463#bib.bib4)\)\.

Yet lower prediction loss, longer accepted continuations, and faster generation remain different outcomes\. Average loss can obscure errors at early drafting steps, even though later tokens are useful only when the preceding continuation is accepted\. Similarly, a tree that covers more useful continuations may take longer to construct and process\. Improvements therefore depend on both the tokens gained and the time spent obtaining them\.

These distinctions motivate three research questions\. RQ1\. Training: How do rollout training choices affect acceptance, and does held\-out prediction loss reflect those changes? RQ2\. Allocation: Under fixed candidate counts at each depth, how does confidence\-guided allocation affect acceptance and latency across different drafters? RQ3\. Execution: How do decode\-loop implementation and host\-thread settings affect whether acceptance gains translate into speed?

We organize these comparisons in FACT \(Fixed\-Budget Allocation and Cross\-Step Training\), a shared experimental framework built around an EAGLE\-style drafter and a frozen Llama\-3\-8B\-Instruct target\. FACT combines established drafting, rollout\-training, and cache\-reuse ideas to isolate their empirical effects; the contribution is the controlled comparison and its findings\. Our comparisons vary training, allocation, and execution settings, and test whether allocation gains transfer to an independently trained official EAGLE drafter\. As shown in Figure[1](https://arxiv.org/html/2611.23463#S1.F1), we connect these choices by measuring both useful progress and total decoding time\.

![Refer to caption](https://arxiv.org/html/2611.23463v1/fig1.png)Figure 1:From draft proposals to generation speed\. A lightweight drafter proposes candidate continuations, which are arranged in a tree and checked by the target model\. Accepted draft tokens and a target\-generated token advance the output\. We study drafter training, candidate allocation under fixed depth\-wise budgets, and execution conditions, measuring both useful progress and total decoding time\.Our contributions are threefold:

- •C1\. Rollout training improves acceptance, but proxy loss can misrank drafters\. Step\-decayed, full\-gradient rollout improves acceptance by 15–24% over the tested detached baseline in eight comparisons spanning two benchmarks, two decoding depths, and two training seeds\. Yet a teacher\-forced drafter can outperform detached rollout in acceptance despite substantially worse multi\-step proxy cross\-entropy\.
- •C2\. Allocation gains transfer under matched depth\-wise budgets\. Confidence\-guided allocation reduces latency without increasing candidate counts at any depth\. This benefit transfers to the official EAGLE drafter, with paired latency improvements whose confidence intervals exclude zero in all ten evaluated benchmark–horizon combinations\. However, stronger drafter training does not consistently increase the benefit of dynamic allocation\.
- •C3\. Execution settings can nearly eliminate speedup without changing acceptance\. Changing only the host\-thread configuration moves dynamic decoding from1\.99×1\.99\\timesto1\.03×1\.03\\timesspeedup on HumanEval, with unchanged acceptance\. Instrumented runs locate the additional cost in tree construction and the target forward pass\.

## 2 Background and Related Work

#### Speculative decoding

Speculative sampling verifies draft proposals using target probabilities and corrects rejected proposals to preserve the target distribution under exact arithmetic\([Leviathan et al\., 2023](https://arxiv.org/html/2611.23463#bib.bib7);[Chen et al\., 2023](https://arxiv.org/html/2611.23463#bib.bib2)\)\. We evaluate greedy decoding: the verifier retains the longest candidate path matching the target’s argmax choices and appends a target\-generated correction or bonus token\. Acceptingkkdraft tokens therefore advances generation byk\+1k\+1tokens, except when an end\-of\-sequence token or output budget terminates the iteration\. We report accepted draft tokens separately from total emitted tokens\. Appendix[A](https://arxiv.org/html/2611.23463#A1)gives the sampling rule for context; the reported acceleration results concern greedy decoding\.

#### Feature\-based drafting

EAGLE predicts target\-model features using a lightweight drafter conditioned on preceding features and a token sequence shifted one position ahead\([Li et al\., 2024a](https://arxiv.org/html/2611.23463#bib.bib8)\)\. Its objective combines feature regression with a token\-distribution loss through the frozen target readout\. Noise added to training features improves robustness but does not explicitly reproduce a drafter’s own feature rollouts\. We retain this feature\-prediction architecture while studying different rollout objectives\. Our feature extraction and readout convention differs from that of the official checkpoint; Section[3\.1](https://arxiv.org/html/2611.23463#S3.SS1)makes that distinction explicit\.

#### Candidate allocation

Tree attention lets candidate nodes attend to the shared prefix and their ancestors, so the target can score several continuations in one call\([Miao et al\., 2024](https://arxiv.org/html/2611.23463#bib.bib11)\)\. EAGLE\-2 uses cumulative draft confidence to expand promising paths and then reranks candidates across depths\([Li et al\., 2024b](https://arxiv.org/html/2611.23463#bib.bib9)\); Sequoia considers hardware when selecting tree size and depth\([Chen et al\., 2024](https://arxiv.org/html/2611.23463#bib.bib4)\)\. Our comparison adopts cumulative\-confidence selection but fixes the candidate count at every depth and omits final reranking across depths\. It isolates allocation among parents under matched depth budgets, rather than comparing complete native decoding systems\.

#### Training for multi\-step drafting

DistillSpec improves draft–target alignment through distillation\([Zhou et al\., 2024](https://arxiv.org/html/2611.23463#bib.bib15)\)\. HASS trains with reused predicted features, introduces a distillation loss focused on high\-probability target tokens, and investigates geometrically decaying weights on later training steps\([Zhang et al\., 2025](https://arxiv.org/html/2611.23463#bib.bib13)\)\. EAGLE\-3 incorporates training\-time drafting, removes feature reconstruction, and fuses features from multiple target layers\([Li et al\., 2025](https://arxiv.org/html/2611.23463#bib.bib10)\)\. We do not claim rollout exposure or geometric step weighting as new mechanisms\. We compare teacher forcing, uniformly weighted detached feature rollouts, and step\-decayed full\-gradient rollouts under a shared feature/readout convention\. Our detached arm is a controlled training baseline, not a reproduction of the complete HASS system or objective\. The replicated comparison tests the combined change in weighting and gradient flow; a separate exploratory control is reported in Appendix[D](https://arxiv.org/html/2611.23463#A4)\.

## 3 Methodology

FACT varies drafter training, candidate allocation, and execution within a shared speculative decoding framework\. This lets us examine how proposal quality and allocation affect useful progress, while accounting for the cost of producing and verifying candidates\.

### 3\.1 Shared drafting framework

We use a trainable fusion layer and a single decoder layer with frozen target embeddings and readout, following the EAGLE architecture\([Li et al\., 2024a](https://arxiv.org/html/2611.23463#bib.bib8)\)\. Letfif\_\{i\}denote the target feature associated with recorded tokenxix\_\{i\}\. For our trained drafters, this is the input to the target’s last decoder layer\. The readout is

H⁡\(f\)=softmax⁡\(WLM​Norm⁡\(f\)\)\.H\(f\)=\\operatorname\{softmax\}\\\!\\left\(W\_\{\\mathrm\{LM\}\}\\operatorname\{Norm\}\(f\)\\right\)\.\(1\)Becausefif\_\{i\}precedes the last decoder layer,H⁡\(fi\)H\(f\_\{i\}\)is a*proxy distribution*, not the full target’s next\-token distributionp\(⋅∣x1:i\)p\(\\cdot\\mid x\_\{1:i\}\)\. The training loss matches predictions to this proxy; verification always uses the full target\. We preserve this feature/readout convention across our training arms, making their comparison conditional on this design\. The independently trained official EAGLE checkpoint uses its own feature convention, specified in Appendix[B](https://arxiv.org/html/2611.23463#A2)\. Training rollout lengthRRand maximum decoding depthKKare independent choices\.

### 3\.2 Multi\-step drafter training

During drafting, predicted features become inputs to later predictions\. Following the feature\-alignment motivation of HASS\([Zhang et al\., 2025](https://arxiv.org/html/2611.23463#bib.bib13)\), we expose the drafter to these imperfect inputs during training\. The first rollout step pairs recorded tokens with preceding target features\. Later steps reuse predicted features, shifting the token and supervision sequences accordingly\. Conditioning tokens remain those recorded in the training data; only the feature inputs are replaced by predictions\.

At each stepss, we use a combined feature and proxy\-token objective: SmoothL1 lossℒfeat\(s\)\\mathcal\{L\}\_\{\\mathrm\{feat\}\}^\{\(s\)\}, averaged over feature dimensions and valid positions, and proxy cross\-entropyℒtok\(s\)=CE⁡\(H⁡\(f\),H⁡\(f^\)\)\\mathcal\{L\}\_\{\\mathrm\{tok\}\}^\{\(s\)\}=\\operatorname\{CE\}\(H\(f\),H\(\\hat\{f\}\)\), averaged over valid positions\. We combine them as

ℒrollout=∑s=1Rws​ℒfeat\(s\)\+λ​∑s=1Rws​ℒtok\(s\),ws=γs−1∑r=1Rγr−1,\\mathcal\{L\}\_\{\\mathrm\{rollout\}\}=\\sum\_\{s=1\}^\{R\}w\_\{s\}\\,\\mathcal\{L\}\_\{\\mathrm\{feat\}\}^\{\(s\)\}\+\\lambda\\sum\_\{s=1\}^\{R\}w\_\{s\}\\,\\mathcal\{L\}\_\{\\mathrm\{tok\}\}^\{\(s\)\},\\qquad w\_\{s\}=\\frac\{\\gamma^\{s\-1\}\}\{\\sum\_\{r=1\}^\{R\}\\gamma^\{r\-1\}\},\(2\)whereλ=0\.1\\lambda=0\.1\. Uniform weighting usesγ=1\\gamma=1, while step\-decayed weighting usesγ=0\.5\\gamma=0\.5\. In both cases, the step weights sum to one; this normalizes their total coefficient, not the resulting loss values or gradient magnitudes\. Earlier steps receive more weight because accepting later tokens depends on accepting their preceding continuation\. HASS also investigates this motivation\([Zhang et al\., 2025](https://arxiv.org/html/2611.23463#bib.bib13)\)\.

Weighting and gradient flow are separate choices\. Detached rollout stops gradients through features reused at the next step; full\-gradient rollout lets later losses influence earlier predictions\. The replicated comparison uses uniform detached rollout, step\-decayed full\-gradient rollout, and a one\-step teacher\-forced control using target input features\. An exploratory uniform full\-gradient control was evaluated at one seed on an earlier prompt set \(Appendix[D](https://arxiv.org/html/2611.23463#A4)\)\. Figure[2](https://arxiv.org/html/2611.23463#S3.F2)illustrates these distinctions\.

![Refer to caption](https://arxiv.org/html/2611.23463v1/fig2.png)Figure 2:Training on predicted features\. The drafterDDpredicts the next feature from a recorded feature and token\.1○Teacher forcing trains one step on recorded features only\.2○Detached rollout reuses each prediction as the next input but stops gradients between steps\.3○Full\-gradient rollout keeps the reuse and lets later losses reach earlier steps\.4○Step weights control each prediction’s contribution to the loss: uniform \(γ=1\\gamma=1\) or step\-decayed \(γ=0\.5\\gamma=0\.5\)\.
### 3\.3 Candidate allocation under fixed budgets

Given the drafter’s predictions, allocation determines which continuations receive candidate slots\. We fix a depth\-wise budget\(b1,…,bK\)\(b\_\{1\},\\ldots,b\_\{K\}\)and total candidate countB=∑d=1KbdB=\\sum\_\{d=1\}^\{K\}b\_\{d\}\. Both policies use the same counts\. The static policy gives parents predetermined child quotas, filled with each parent’s highest\-probability tokens\. Token identities depend on the context, but the branching structure does not\.

The dynamic policy lets extensions from all retained parents compete for the slots at each depth\. It scores a continuationy1:dy\_\{1:d\}by cumulative draft confidence:

S\(y1:d\)=∑j=1dlogq\(yj∣x1:i,y1:j−1\),S\(y\_\{1:d\}\)=\\sum\_\{j=1\}^\{d\}\\log q\\\!\\left\(y\_\{j\}\\mid x\_\{1:i\},y\_\{1:j\-1\}\\right\),\(3\)wherex1:ix\_\{1:i\}is the current prefix andqqthe draft distribution\. At depthdd, it retains thebdb\_\{d\}highest\-scoring extensions of retained depth\-\(d−1\)\(d\-1\)parents\. One parent may therefore receive several slots and another none\.

This follows EAGLE\-2’s cumulative\-confidence principle\([Li et al\., 2024b](https://arxiv.org/html/2611.23463#bib.bib9)\), but fixes the budget at every depth and omits its final reranking across depths\. Matching candidate counts and depths does not imply equal execution cost: dynamic selection and tree construction remain part of decoding time\. Appendix[B](https://arxiv.org/html/2611.23463#A2)specifies the quota rule and gives an illustrative allocation example\.

### 3\.4 Verification and cache reuse

The target checks the tree using an attention mask that exposes the shared prefix and each node’s ancestors, excluding competing branches\. Greedy verification retains the path matching the target’s highest\-probability choices and supplies a correction or bonus token, as described in Section[2](https://arxiv.org/html/2611.23463#S2.SS0.SSS0.Px1)\. We gather the accepted branch’s target key–value cache entries from this forward pass instead of recomputing them\.

The target\-generated token is then pending: its identity is known, but its cache entry is not yet computed\. We process it with the next candidate tree in one target forward pass, using a predicted feature for its position until that pass provides the true target feature\. After prompt prefill, this folded procedure uses one target forward pass per decoding iteration over the pending token andBBcandidates\.

Our dynamic allocation supports greedy decoding\. Verification preserves the target’s greedy continuation under exact arithmetic; finite\-precision agreement is evaluated separately\. Host\-thread settings can change execution cost without changing the intended acceptance rule\. Additional implementation details are given in Appendix[B](https://arxiv.org/html/2611.23463#A2)\.

## 4 Experiments

We evaluate whether improvements in speculative decoding produce both more useful draft tokens and faster generation\. We first examine how training choices affect acceptance and whether prediction loss captures these differences \(RQ1\)\. We then test confidence\-guided allocation under fixed depth\-wise budgets and examine whether its benefits transfer across drafters \(RQ2\)\. Finally, we investigate how execution conditions affect the conversion of accepted tokens into generation speed \(RQ3\)\.

### 4\.1 Experimental setup

We use a frozen Llama\-3\-8B\-Instruct target\([Grattafiori et al\., 2024](https://arxiv.org/html/2611.23463#bib.bib6)\)and evaluate greedy decoding on a single NVIDIA H100 in bfloat16\. Our workloads comprise prompts from MT\-Bench\([Zheng et al\., 2023](https://arxiv.org/html/2611.23463#bib.bib14)\), HumanEval\([Chen et al\., 2021](https://arxiv.org/html/2611.23463#bib.bib3)\), MBPP\([Austin et al\., 2021](https://arxiv.org/html/2611.23463#bib.bib1)\), GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2611.23463#bib.bib5)\), and Alpaca\([Taori et al\., 2023](https://arxiv.org/html/2611.23463#bib.bib12)\)\. Main comparisons use 50 held\-out test prompts per benchmark; the host\-thread sweep uses separate development prompts\. We use 256 candidate nodes, decoding depthsK∈\{4,8\}K\\in\\\{4,8\\\}, and a 128\-token generation budget\. Host threads are capped at eight unless varied explicitly\.

We report accepted draft tokens per verification step, excluding the target\-generated token, together with latency in milliseconds per generated token and speedup over autoregressive decoding\. Measurements include prefill but exclude model loading, and latency is normalized by the actual number of emitted tokens\. Repeated measurements are averaged within each prompt before computing paired comparisons; prompts then receive equal weight\. Allocation timings use three to ten repeats per configuration, with the mainK=4K=4comparisons on MT\-Bench and HumanEval spanning three sessions\. Confidence intervals use 10,000 paired bootstrap resamples over prompts\. Table[2](https://arxiv.org/html/2611.23463#A2.T2)consolidates the reported settings\. Speedups are paired summaries, so they need not equal ratios reconstructed from rounded mean latencies\.

### 4\.2 RQ1: How does training affect acceptance?

We compare teacher forcing, uniformly weighted detached rollout, and step\-weighted backpropagation through the rollout with decayγ=0\.5\\gamma=0\.5\. The three objectives use the same 80,000 feature rows, 10,000 optimization steps, and batch configuration\. These controls match data and optimizer steps, not training compute\. Rollout\-trained models useR=8R=8training steps, while teacher forcing uses one\. We evaluate two independently trained seeds on the same static tree, separating training effects from candidate allocation\.

As shown in Figure[3](https://arxiv.org/html/2611.23463#S4.F3), step\-weighted training improves acceptance over detached rollout in all eight combinations of benchmark, decoding depth, and seed, with gains ranging from 15% to 24%\. AtK=4K=4, acceptance averaged across seeds increases from 1\.38 to 1\.60 on MT\-Bench and from 1\.58 to 1\.89 on HumanEval\. These comparisons support the combined choice of step weighting and cross\-step backpropagation; they do not separately identify the contribution of each component\.

Figure 3:Training improvements persist across seeds\. Each marker shows mean draft acceptance over 50 test prompts for one training seed\. All configurations use a static 256\-node tree\. Percentages compare step\-weighted training with detached rollout after averaging the two seeds\.KKdenotes decoding depth, not training rollout length\.Figure[4](https://arxiv.org/html/2611.23463#S4.F4)shows where these gains arise\. We roll each drafter out along the target’s greedy continuation of each test prompt, in the same way as a chain of the draft tree at decode time, and record how often its top\-ranked token matches the target’s next token at each draft depth\. Teacher forcing agrees most often at the first depth, but its agreement falls fastest after that\. From the second depth onward, step\-weighted training agrees most often on both benchmarks, while detached rollout agrees least at every depth\.

Figure 4:Agreement with the target at each draft depth\. Fraction of positions where the drafter’s top\-ranked token equals the target’s greedy next token, measured along the target’s continuation of 50 test prompts per benchmark with decode\-time feature inputs\. Thick lines average training seeds 1 and 2, and thin lines show each seed\. The checkpoints are the 10,000\-step drafters of Figure[3](https://arxiv.org/html/2611.23463#S4.F3)\.However, held\-out rollout proxy cross\-entropy does not reproduce this ordering\. Under a common proxy cross\-entropy evaluator that weights all eight rollout steps equally, teacher forcing scores substantially worse than detached training, 7\.86 versus 3\.73, yet accepts more tokens on MT\-Bench atK=4K=4, 1\.49 versus 1\.38\. Meanwhile, detached and step\-weighted training have similar proxy cross\-entropy, 3\.73 and 3\.71, despite their different acceptance\. Thus, this proxy metric does not consistently rank the tested objectives by acceptance\. This finding is conditional on the intermediate\-feature readout and does not establish that cross\-entropy against the full target distribution is generally uninformative\.

### 4\.3 RQ2: Does candidate allocation improve generation across drafters?

Having examined training under a fixed tree, we next hold drafter weights fixed and change how candidates are allocated\. Static and dynamic trees use identical numbers of nodes at every depth\. The comparison therefore changes where candidate slots are assigned without increasing the candidate budget, although tree\-construction costs can still differ\.

We evaluate an earlier baseline drafter, our scaled drafter, and the independently trained official EAGLE drafter\([Li et al\., 2024a](https://arxiv.org/html/2611.23463#bib.bib8)\)\. The scaled checkpoint was trained for 20,000 steps and is distinct from the 10,000\-step checkpoints used in RQ1\.

Dynamic allocation improves both acceptance and generation time\. For the scaled drafter atK=4K=4, MT\-Bench acceptance rises from 1\.92 to 2\.66 while latency falls from 13\.96 to 10\.32 ms/token, increasing speedup from1\.31×1\.31\\timesto1\.75×1\.75\\times\. On HumanEval, acceptance rises from 2\.27 to 3\.22 and latency falls from 12\.47 to 8\.93 ms/token, increasing speedup from1\.46×1\.46\\timesto2\.02×2\.02\\times\.

Table 1:Main allocation comparison\. Scaled drafter,K=4K=4, 256 candidates, and 50 test prompts per workload\.τ\\tauexcludes the target\-generated token; latency includes prefill\. Figure[5](https://arxiv.org/html/2611.23463#S4.F5)reports uncertainty on paired latency savings\.Figure[5](https://arxiv.org/html/2611.23463#S4.F5)extends this comparison across all five workloads\. Applied unchanged to the official EAGLE drafter, dynamic allocation saves 3\.00–3\.92 ms/token, with confidence intervals excluding zero in all ten benchmark–depth combinations\. This demonstrates transfer across the evaluated drafters under a shared target and decoding implementation; it is not a comparison against the native EAGLE\-2 system\.

Figure 5:Allocation gains transfer across drafters\. Points show paired latency savings,Lstatic−LdynamicL\_\{\\mathrm\{static\}\}\-L\_\{\\mathrm\{dynamic\}\}, under identical depth\-wise candidate budgets\. Positive values favor dynamic allocation\. Bars show 95% bootstrap intervals over 50 test prompts after pooling repetitions within each prompt\. Both panels use the same horizontal scale\.Figure[6](https://arxiv.org/html/2611.23463#S4.F6)shows the same comparison prompt by prompt for the scaled drafter atK=4K=4\. Dynamic allocation is faster than static allocation on every one of the 250 prompts across the five workloads, so the mean gains in Table[1](https://arxiv.org/html/2611.23463#S4.T1)do not come from a few favorable prompts\.

Figure 6:Per\-prompt speedups for the scaled drafter atK=4K=4\. Each point is one of the 50 test prompts per workload, with repeated runs averaged within the prompt; HumanEval and MT\-Bench pool ten runs and the other workloads three\. Boxes show quartiles and medians\. Speedup is the per\-prompt ratio to autoregressive decoding from the same run, and the dotted line marks autoregressive speed\.We measure interaction on the absolute latency\-saving scale\. For drafterdd, defineG⁡\(d\)=Lstatic​\(d\)−Ldynamic​\(d\)G\(d\)=L\_\{\\mathrm\{static\}\}\(d\)\-L\_\{\\mathrm\{dynamic\}\}\(d\)and

I=G⁡\(scaled\)−G⁡\(baseline\)\.I=G\(\\mathrm\{scaled\}\)\-G\(\\mathrm\{baseline\}\)\.\(4\)PositiveIImeans that allocation saves more milliseconds per token for the scaled drafter\. The estimate is negative with an interval excluding zero in five of ten benchmark–depth combinations, positive only on GSM8K atK=8K=8, and inconclusive in four\. For HumanEval atK=4K=4,I=−0\.40I=\-0\.40ms/token with a 95% interval of\[−0\.64,−0\.17\]\[\-0\.64,\-0\.17\]\(Table[3](https://arxiv.org/html/2611.23463#A3.T3)\)\. Thus, a stronger drafter does not consistently receive a larger absolute latency reduction from allocation\. This statement concerns the evaluated checkpoints and this interaction scale; it does not imply an identical interaction on a speedup\-ratio scale\.

### 4\.4 RQ3: When do acceptance gains translate into speed?

The preceding results use a fixed execution configuration\. To test how much this matters, we vary the host\-thread cap while keeping the code, GPU, drafter weights, prompts, and candidate budget unchanged\.

Figure[7](https://arxiv.org/html/2611.23463#S4.F7)shows that dynamic decoding remains effective at caps between one and 32 threads but loses most of its advantage at 208 threads\. AtK=4K=4, moving from eight to 208 threads reduces speedup from1\.91×1\.91\\timesto0\.92×0\.92\\timeson GSM8K and from1\.99×1\.99\\timesto1\.03×1\.03\\timeson HumanEval\. TheK=8K=8dynamic configuration shows the same pattern, whereas the static tree remains comparatively stable\.

Crucially, draft acceptance is identical across thread settings within each configuration\. The change in speed therefore occurs without a change in accepted draft progress\.

Figure 7:Execution conditions change speed without changing acceptance\. Speedup over autoregressive decoding as host\-thread count varies\. Each point averages paired speedup ratios over 50 development prompts from one run per setting\. The dotted line denotes autoregressive speed\. Acceptance remains unchanged across caps within each configuration\.A separate campaign repeats the comparison five times on the 50 test prompts of HumanEval and MT\-Bench atK=4K=4\. Figure[8](https://arxiv.org/html/2611.23463#S4.F8)plots each prompt’s tokens per target call against its speedup\. At 208 threads every prompt produces exactly the same number of tokens per target call as at eight threads, yet the mean speedup of dynamic decoding falls from1\.99×1\.99\\timesto0\.98×0\.98\\timeson HumanEval and from1\.74×1\.74\\timesto0\.86×0\.86\\timeson MT\-Bench\. Static decoding keeps its speedup at both settings\.

Figure 8:Tokens per target call and speedup for each prompt\. Each point is one test prompt averaged over five runs per setting, using the scaled drafter\. Tokens per target call include the target\-generated token\. The static tree at 208 threads, not shown, matches the static tree at eight threads\. The dotted line marks autoregressive speed\.Separate instrumented HumanEval runs help locate the added cost\. Increasing the thread cap from eight to 208 raises tree\-construction time from 3\.19 to 6\.40 ms per step and target tree\-forward time from 20\.83 to 51\.94 ms, while acceptance checking remains near 0\.15 ms\. These phase timings are diagnostic measurements rather than the uninstrumented timings used for headline speedups\. They show why inexpensive acceptance checking alone is insufficient: execution overhead elsewhere in the loop can erase the benefit of higher acceptance\.

Figure[9](https://arxiv.org/html/2611.23463#S4.F9)shows the complete per\-step breakdown from these runs\. The static tree costs about the same per step at both settings, 37\.4 and 36\.9 ms\. For the dynamic tree, a step grows from 35\.9 to 72\.5 ms, and the target tree forward pass and tree construction account for 34\.3 ms of the 36\.6 ms increase\.

Figure 9:Time per decoding step by phase\. Means over 25 HumanEval development prompts with the scaled drafter atK=4K=4and phase timers enabled, weighting each prompt by its number of steps\. Labels give the total time and the tokens emitted per step\. Timers can perturb execution, so these runs are diagnostic and are not used for the speedups reported elsewhere\.#### Numerical fidelity

We assess output agreement separately from speed\. Float32 decoding matches the autoregressive target at all 52,224 tested positions\. In bfloat16, same\-prefix agreement is 99\.0%, so the speed measurements should not be interpreted as guaranteeing identical token sequences\. Separate quality probes on 60 prompts per benchmark yield paired differences of\+0\.033\+0\.033on HumanEval pass@1 \(95% interval\[−0\.033,0\.100\]\[\-0\.033,0\.100\]\) and0\.0000\.000on GSM8K exact match \(\[−0\.067,0\.067\]\[\-0\.067,0\.067\]\)\. These intervals do not establish quality equivalence\. Appendix[E](https://arxiv.org/html/2611.23463#A5)describes these probes and distinguishes them from the 50\-prompt timing evaluations\.

## 5 Discussion and Limitations

#### Acceptance gains are not output\-rate gains

Our acceptance metric excludes the target\-generated token\. Away from termination and output\-cap effects, increasing mean draft acceptance fromτ0\\tau\_\{0\}toτ1\\tau\_\{1\}increases emitted tokens per iteration by a factor\(τ1\+1\)/\(τ0\+1\)\(\\tau\_\{1\}\+1\)/\(\\tau\_\{0\}\+1\), before accounting for iteration cost\. For the RQ1 MT\-Bench means, the increase from 1\.38 to 1\.60 is about 16% in draft acceptance but only 9\.2% in tokens advanced per iteration\. Prefill and changes in drafting or target\-forward cost further affect measured latency\. This explains why acceptance percentages should not be read as expected speedup percentages\.

#### Proxy loss and model selection

The RQ1 result concerns ranking different training objectives using the proxy cross\-entropy defined by our intermediate\-feature readout\. That readout does not produce the full target distribution, so proxy–target mismatch is one possible contributor to the observed ranking failure\. We do not isolate it from other effects such as how errors are distributed over rollout steps\. The result also does not rule out useful within\-run correlations between loss and acceptance\. Its supported implication is to evaluate acceptance directly when choosing among these objectives\.

#### Allocation and execution

Matched depth budgets isolate changes in candidate placement while keeping dynamic selection costs inside the measurement\. The positive latency savings across the evaluated checkpoints show that the allocation benefit is not restricted to our trained weights\. The interaction remains scale\-specific: stronger training does not consistently increase absolute milliseconds saved\. The thread sweep further shows that an unchanged acceptance count can accompany a large latency change\. These results motivate recording execution settings alongside the allocation policy and checkpoint\.

#### Scope and attribution

The evaluation uses one target model, an H100 GPU platform, short generation budgets, and 50 test prompts per benchmark\. It does not establish acceleration for sampling, longer outputs, or batched serving, or superiority over native EAGLE\-2\. The replicated training comparison uses two benchmarks and two seeds and changes both weighting and gradient flow; it supports their combined configuration\. The uniform full\-gradient control is exploratory and uses an earlier prompt set\. The scaled allocation checkpoint has 20,000 training steps, whereas the RQ1 checkpoints have 10,000, so the allocation results do not directly measure the latency effect of the replicated RQ1 intervention\.

#### Measurement and numerical limits

Prompt\-level bootstrap intervals quantify variation across the evaluated prompts, not all session or deployment variability\. The thread sweep uses one run per setting, and instrumented phase timings can perturb execution\. Measurements include prefill and use a 128\-token output cap, so their ratios depend on workload lengths as well as decoding\. Exact\-arithmetic verification does not guarantee identical bfloat16 outputs\. The separate quality probes remain compatible with losses of 3\.3 percentage points on HumanEval and 6\.7 points on GSM8K; they establish neither equivalence nor broad quality preservation\.

## 6 Conclusion

We studied how drafter training, candidate allocation, and execution conditions jointly affect speculative decoding\. Step\-weighted full\-gradient rollout improves acceptance over the tested detached baseline, while rollout proxy cross\-entropy can misrank the tested objectives by acceptance\. Confidence\-guided allocation improves generation under fixed depth\-wise budgets and transfers to an independently trained drafter, but its benefit does not consistently increase with drafter strength\. Finally, execution settings can erase speed gains without changing acceptance\. The central lesson is to evaluate speculative decoding through both useful progress and total decoding time, with numerical fidelity assessed separately\.

## References

- Austin et al\. \(2021\)Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton\.Program synthesis with large language models\.*arXiv preprint arXiv:2108\.07732*, 2021\.
- Chen et al\. \(2023\)Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean\-Baptiste Lespiau, Laurent Sifre, and John Jumper\.Accelerating large language model decoding with speculative sampling\.*arXiv preprint arXiv:2302\.01318*, 2023\.
- Chen et al\. \(2021\)Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, et al\.Evaluating large language models trained on code\.*arXiv preprint arXiv:2107\.03374*, 2021\.
- Chen et al\. \(2024\)Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen\.Sequoia: Scalable, robust, and hardware\-aware speculative decoding\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.
- Cobbe et al\. \(2021\)Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman\.Training verifiers to solve math word problems\.*arXiv preprint arXiv:2110\.14168*, 2021\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al\.The Llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*, 2024\.
- Leviathan et al\. \(2023\)Yaniv Leviathan, Matan Kalman, and Yossi Matias\.Fast inference from transformers via speculative decoding\.In*International Conference on Machine Learning \(ICML\)*, 2023\.
- Li et al\. \(2024a\)Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang\.EAGLE: Speculative sampling requires rethinking feature uncertainty\.In*International Conference on Machine Learning \(ICML\)*, 2024a\.
- Li et al\. \(2024b\)Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang\.EAGLE\-2: Faster inference of language models with dynamic draft trees\.In*Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2024b\.
- Li et al\. \(2025\)Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang\.EAGLE\-3: Scaling up inference acceleration of large language models via training\-time test\.*arXiv preprint arXiv:2503\.01840*, 2025\.
- Miao et al\. \(2024\)Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia\.SpecInfer: Accelerating generative large language model serving with tree\-based speculative inference and verification\.In*International Conference on Architectural Support for Programming Languages and Operating Systems \(ASPLOS\)*, 2024\.
- Taori et al\. \(2023\)Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B\. Hashimoto\.Stanford Alpaca: An instruction\-following LLaMA model\.[https://github\.com/tatsu\-lab/stanford\_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023\.
- Zhang et al\. \(2025\)Lefan Zhang, Xiaodan Wang, Yanhua Huang, and Ruiwen Xu\.Learning harmonized representations for speculative sampling\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.
- Zheng et al\. \(2023\)Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P\. Xing, Hao Zhang, Joseph E\. Gonzalez, and Ion Stoica\.Judging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.In*Advances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track*, 2023\.
- Zhou et al\. \(2024\)Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean\-François Kagy, and Rishabh Agarwal\.DistillSpec: Improving speculative decoding via knowledge distillation\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.

## Appendix ADecoding rule and token accounting

The sampling rule is included as background, not as an acceleration result for our dynamic implementation\. Letppandqqdenote target and draft distributions conditioned on an accepted prefixx1:ix\_\{1:i\}\. A proposed tokeny∼qy\\sim qis accepted with probability

α\(y∣x1:i\)=min\(1,p\(y∣x1:i\)q\(y∣x1:i\)\)\.\\alpha\(y\\mid x\_\{1:i\}\)=\\min\\\!\\left\(1,\\frac\{p\(y\\mid x\_\{1:i\}\)\}\{q\(y\\mid x\_\{1:i\}\)\}\\right\)\.\(5\)On rejection, the replacement distribution is

r\(y∣x1:i\)=\[p\(y∣x1:i\)−q\(y∣x1:i\)\]\+∑v∈𝒱\[p\(v∣x1:i\)−q\(v∣x1:i\)\]\+,r\(y\\mid x\_\{1:i\}\)=\\frac\{\[p\(y\\mid x\_\{1:i\}\)\-q\(y\\mid x\_\{1:i\}\)\]\_\{\+\}\}\{\\sum\_\{v\\in\\mathcal\{V\}\}\[p\(v\\mid x\_\{1:i\}\)\-q\(v\\mid x\_\{1:i\}\)\]\_\{\+\}\},\(6\)where\[a\]\+=max⁡\(0,a\)\[a\]\_\{\+\}=\\max\(0,a\)\. The residual is needed only when rejection occurs; whenp=qp=q, rejection has probability zero\. If all proposals are accepted, a bonus token is drawn from the target\. This rule preserves the target distribution under exact arithmetic\([Leviathan et al\., 2023](https://arxiv.org/html/2611.23463#bib.bib7);[Chen et al\., 2023](https://arxiv.org/html/2611.23463#bib.bib2)\)\.

For the greedy experiments, the target supplies a correction at the first rejected position or a bonus after full acceptance\. We count only matching draft tokens inτ\\tau, and all emitted tokens in latency’s denominator\. A completed iteration normally emitsk\+1k\+1tokens forkkaccepted draft tokens; termination can shorten it\. Algorithmic equivalence assumes consistent target logits and tie\-breaking between the tree and autoregressive evaluations\. Finite\-precision agreement is a separate empirical check\.

## Appendix BImplementation and reported protocol

#### Feature convention

Our trained drafters use the input to the target’s last decoder layer\. The frozen final normalization and language\-model head define their proxy distribution\. The independently trained official EAGLE checkpoint uses post\-normalization target features with the output head and no additional normalization\. The full target model performs verification in both cases\.

#### Memory\-efficient training

Full\-vocabulary cross\-entropy is computed in chunks of sequence positions and its gradients are accumulated through the frozen readout\. This changes neither the mathematical objective nor its gradient, apart from floating\-point effects\. Recorded token inputs remain fixed during feature rollouts; this is exposure to predicted features, not fully generated token trajectories\.

#### Static allocation

At depthdd, thebdb\_\{d\}slots are divided as evenly as possible among the retained parents\. A fixed parent ordering determines which parents receive any remainder\. Each parent fills its quota with its highest\-probability tokens\. Dynamic allocation instead ranks extensions from all retained parents by cumulative draft confidence\. Figure[10](https://arxiv.org/html/2611.23463#A2.F10)illustrates the distinction with eight candidates; it does not specify the 256\-node experimental depth profile\.

![Refer to caption](https://arxiv.org/html/2611.23463v1/fig6.png)Figure 10:Fixed budgets, different allocation\. Illustrative widths\(3,3,2\)\(3,3,2\)and draft probabilities for the target continuation “is Paris,”\. \(a\) Static quotas omit the target’s comma continuation, so two draft tokens are accepted before the mismatch; \(b\) confidence\-guided \(dynamic\) selection places both depth\-3 slots below “Paris” and accepts three\. Accepted counts exclude the target\-generated correction or bonus token\.
#### Acceptance checking and cache commitment

Target argmaxes are computed on the device, and compact token IDs are transferred for accepted\-path selection on the host\. The path identifies target key–value entries to gather\. The target\-generated token is processed with the next tree\. The approximately 0\.15 ms acceptance\-checking measurement excludes target tree scoring and should not be read as the cost of the entire verification stage\.

Table 2:Reported settings\. Training\-step matching is not compute matching\. The 60\-prompt quality probes are separate from the timing experiments\.

## Appendix CDrafter–allocation interaction

Table[3](https://arxiv.org/html/2611.23463#A3.T3)reports paired absolute latency savings and the interaction defined in Equation[4](https://arxiv.org/html/2611.23463#S4.E4)\. Repeats are pooled within prompts\. Intervals are 95% percentile bootstrap intervals over prompts, with 10,000 resamples\. These intervals do not imply that the same interaction sign must hold for reciprocal speedup ratios\.

Table 3:Dynamic\-allocation savingsGGand interactionII, in ms/token\. Entries give estimates and 95% intervals\. NegativeIImeans smaller absolute savings for the scaled drafter\.
## Appendix DExploratory training control and training cost

Table[4](https://arxiv.org/html/2611.23463#A4.T4)reports an exploratory four\-arm comparison\. It uses one training seed and an earlier prompt set, so its acceptance values must not be pooled with the two\-seed results in Section[4\.2](https://arxiv.org/html/2611.23463#S4.SS2)\. Within this exploratory run, decayed weighting improves acceptance relative to uniform weighting when gradients cross rollout steps\. Uniform full\-gradient training does not improve on uniformly weighted detached training\. No decayed\-weight detached arm was evaluated, so the experiment does not isolate whether cross\-step gradient flow is necessary under decayed weighting\.

Table 4:Exploratory four\-arm control, seed 0, 80,000 feature rows and 10,000 optimizer steps\. MT\-Bench, static 256\-node tree, earlier prompt set\. CE is proxy cross\-entropy\. The eight\-step CE evaluation is shared across objectives\.The recorded training times for 10,000 steps on one H100 are 2\.4 h for teacher forcing, 14\.1 h for detached rollout, and 27\.0 h for step\-decayed full\-gradient rollout\. These are diagnostic costs from the original comparison, not averages over the replicated seeds\. They demonstrate that matched optimizer steps do not imply matched training time; the acceptance improvement should be interpreted with this cost difference in mind\.

Figure[11](https://arxiv.org/html/2611.23463#A4.F11)places acceptance next to training time and training data in the same exploratory setting\. The step\-weighted drafter needs about 5,000 steps, roughly 13\.5 h at its recorded rate, to reach the acceptance that teacher forcing reaches in 2\.4 h, 1\.48 against 1\.49 on MT\-Bench atK=4K=4\. Its acceptance is still rising at 20,000 steps, and at a fixed 10,000 steps it rises with the number of training feature rows on both benchmarks\. These are single\-seed results on the earlier prompt set\.

Figure 11:Acceptance against training time and training data in the exploratory run \(seed 0, static 256\-node tree, earlier prompt set\)\. The left panel shows MT\-Bench acceptance atK=4K=4\. Step\-weighted checkpoints are placed at the recorded rate of 27\.0 h per 10,000 steps, and the other objectives at their recorded 10,000\-step times, with uniform full\-gradient training taken from its training log\. The right panel shows step\-weighted drafters trained for 10,000 steps on 20,000 to 80,000 feature rows\.
## Appendix ENumerical fidelity and quality probes

Float32 speculative decoding agrees with autoregressive decoding at all 52,224 tested positions across four evaluated systems and four benchmarks\. In a separate bfloat16 same\-prefix comparison, 12,672 of 12,800 positions agree and 128 disagree\. A free\-running comparison yields 79\.9% agreement; once a token differs, the two continuations can follow different prefixes, so free\-running and same\-prefix agreement answer different questions\.

All 128 same\-prefix disagreements occur where the target’s top two logits are separated by at most two bfloat16 units in the last place; approximately 92% occur at an exact tie in the measured logits\. This concentration is consistent with finite\-precision sensitivity near ties\. It is evidence about the observed disagreements, not a proof that every implementation path is correct\. These tests do not guarantee identical bfloat16 outputs for the timed configurations\.

Table 5:Separate paired quality probes, 60 held\-out prompts per benchmark\. Differences are speculative minus autoregressive performance, on a 0–1 scale\. Intervals quantify uncertainty in the difference and do not establish equivalence\.The quality probe caps generation at 384 tokens, applies no stop\-sequence trimming, and extracts the last number as the GSM8K answer\. These conditions are shared across the paired arms, but their effects need not cancel when continuations differ\. The probe’s absolute scores should not be interpreted as a standard capability evaluation\. The intervals in Table[5](https://arxiv.org/html/2611.23463#A5.T5)allow losses of 3\.3 percentage points on HumanEval and 6\.7 points on GSM8K\. MT\-Bench task quality was not evaluated with a judge model\.

相似文章

AngelSpec:面向实际场景的高性能推测解码推理

arXiv cs.CL

AngelSpec 提出了一个统一的训练与推理框架,用于推测解码,该框架联合优化自回归多 token 预测和块并行扩散草稿模型,以处理异构实际工作负载。在 Hy3 模型系列上的实验显示,相对于自回归解码,加速比高达 2.4 倍,并且吞吐量比 DFlash 高出 11.8%。

跨语言的推测解码

arXiv cs.CL

本文比较了三种策略以提高非英语语言的推测解码效率,发现任务特定蒸馏能提高接受率但泛化性差,而n-gram草稿模型尽管接受率较低,却能提供持续的加速。

什么是推测性解码?(在paperswithco.de上热门)[R]

Reddit r/MachineLearning

推测性解码是一种推理优化技术,它使用快速草稿模型提出未来 token,并由较大模型并行验证,从而提高 LLM 的生成速度。文章强调了它在 Papers with Code 上的热门状态,以及最近的 SGLang 博客文章,该文章介绍了使用 DFlash 模型实现的最先进延迟。