Target Regeneration Shrinks the Gains of Draft-Training Objectives in Speculative Decoding

# Target Regeneration Shrinks the Gains of Draft-Training Objectivesin Speculative Decoding
Source: [https://arxiv.org/html/2610.29013](https://arxiv.org/html/2610.29013)
#### Vision study\.

The target is Qwen3\-VL\-8B\-Instruct\([Bai et al\., 2025](https://arxiv.org/html/2610.29013#bib.bib4)\)\. Apart from one control, every run continues training the released DFlash drafter for Qwen3\-8B, whose language model has the same dimensions as the target’s decoder\([Chen et al\., 2026](https://arxiv.org/html/2610.29013#bib.bib8)\)\. The training corpus is a fixed subset of 26,196 single\-turn examples from ALLaVA\([Chen et al\., 2024a](https://arxiv.org/html/2610.29013#bib.bib7)\), and a subset of the same size from ShareGPT4V\([Chen et al\., 2024b](https://arxiv.org/html/2610.29013#bib.bib9)\)serves as a second corpus\. The factorial trains for three epochs withγ=7\\gamma=7and the same learning rate for every objective and version\. Each cell is trained twice, with different anchor sampling and the same data order, and every reported MAT averages the two runs\. Appendix[A\.1](https://arxiv.org/html/2610.29013#A1.SS1)gives the remaining training details\.

#### Evaluation\.

Acceptance is measured on 99 scored prompts in each of three domains: detailed captions of COCO images\([Lin et al\., 2014](https://arxiv.org/html/2610.29013#bib.bib37)\), TextVQA questions\([Singh et al\., 2019](https://arxiv.org/html/2610.29013#bib.bib40)\)from its training split, on which no drafter is trained, and DocVQA questions\([Mathew et al\., 2021](https://arxiv.org/html/2610.29013#bib.bib39)\)\. The target decodes greedily for up to 256 tokens, and a tree verifier checks up to 63 drafted tokens in each round\. A domain’s MAT pools rounds over its prompts, and a run’s MAT is the geometric mean of its three domain MATs\. Intervals are 95% percentile intervals from 10,000 resamples of prompts within each domain, so they describe prompt variation for the trained checkpoints\.

#### Controls\.

Eight controls test the vision results on external and fully regenerated ALLaVA responses\. Six change training: random initialization;γ=2\\gamma=2, with the soft survival weighting of PARD\-2 as an extra objective; two further data orders; one third of the learning rate; an L2 anchor toward the released drafter; and KD over the target’s 64 most probable tokens\. Two change evaluation: a chain verifier that checks only the drafter’s top candidate at each of seven slots, also on the matched\-length versions, and target sampling at temperature 1 with random numbers shared by all drafters\. Appendix[A\.3](https://arxiv.org/html/2610.29013#A1.SS3)gives the objectives, epochs and runs of each control\.

#### Text replication\.

The target is Qwen3\-4B with thinking disabled\([Yang et al\., 2025](https://arxiv.org/html/2610.29013#bib.bib42)\), and the drafter continues from the released DFlash drafter for that model\. The corpus pairs 24,552 first\-turn prompts from UltraChat\([Ding et al\., 2023](https://arxiv.org/html/2610.29013#bib.bib12)\)with either their original responses or the target’s greedy responses, and CE, KD and KD \+ gate are trained on each version with the vision recipe\. Evaluation uses the public DFlash decoder, which verifies one 16\-token block in each round, on seven chat, math and code tasks, 2,103 prompts in total, at temperatures 0 and 1\. For a second drafter family, a released drafter with the EAGLE\-3 architecture\([Li et al\., 2025](https://arxiv.org/html/2610.29013#bib.bib36)\)for this target continues training with CE and KD on both versions and is evaluated at temperature 0 with its public tree decoder\. Appendix[A](https://arxiv.org/html/2610.29013#A1)gives the remaining details\.

## 4Results

### 4\.1Regeneration shrinks objective gains

Table[3](https://arxiv.org/html/2610.29013#S3)a reports the vision factorial\. On ALLaVA, distillation and the gate add large gains on external responses, but after full regeneration KD adds 3\.7% over CE and the gate no measurable gain\. The ratio of ratios is−\-9\.4% for KD and−\-5\.2% for the gate, and both are negative in every domain and training run\. Regeneration raises MAT under every objective, CE the most and KD \+ gate the least, so the three objectives end close together, and the objective that gained most on external responses gains least from the new corpus\. ShareGPT4V shows the same shrinkage: after full regeneration KD’s gain over CE falls from 18\.0% to 3\.6% and the gate’s gain over KD from 6\.1% to 0\.9%, and the ratios of ratios,−\-12\.2% for KD and−\-4\.8% for the gate, are negative in every domain and training run\.

Figure 3:Ratios of ratios with 95% prompt bootstrap intervals, comparing an increment on the given responses with the same increment on external responses\. Data\-order rows come from one training run, and best schedules use each objective’s best measured training length\.ConditionResponsesCEKDKD gain \(%\)\[0pt\]\[0pt\]Vision controlsRand\. init\., 3 ep\.External2\.066±\.0022\.259±\.002\+9\.3 \[8\.7, 9\.9\]Full2\.828±\.0022\.912±\.029\+3\.0 \[2\.4, 3\.5\]Rand\. init\., 6 ep\.External2\.071±\.0222\.349±\.020\+13\.4 \[12\.7, 14\.2\]Full3\.082±\.0143\.200±\.041\+3\.8 \[3\.2, 4\.4\]Lower LRExternal3\.099±\.0073\.474±\.016\+12\.1 \[11\.3, 12\.9\]Full3\.893±\.0094\.000±\.002\+2\.8 \[2\.2, 3\.3\]L2 anchorExternal3\.152±\.0073\.474±<\.001\+10\.2 \[9\.4, 11\.0\]Full3\.937±\.0134\.005±\.014\+1\.7 \[1\.1, 2\.3\]\[0pt\]\[0pt\]Text replicationDFlash,T=0T=0External3\.227±\.0083\.464±\.004\+7\.3 \[6\.8, 7\.8\]Regenerated4\.141±\.0164\.183±\.006\+1\.0 \[0\.7, 1\.3\]DFlash,T=1T=1External3\.018±\.0043\.225±\.017\+6\.9 \[6\.0, 7\.7\]Regenerated3\.750±\.0223\.789±\.009\+1\.0 \[0\.0, 2\.0\]EAGLE\-3,T=0T=0External3\.886±\.0174\.420±\.020\+13\.7 \[13\.2, 14\.3\]Regenerated4\.791±\.0094\.895±\.002\+2\.2 \[1\.8, 2\.5\]ConditionResponsesKDKD \+ gateGate gain \(%\)\[0pt\]\[0pt\]Text replication: gateDFlash,T=0T=0External3\.464±\.0043\.623±\.002\+4\.6 \[4\.1, 5\.0\]Regenerated4\.183±\.0064\.188±\.005\+0\.1 \[−\-0\.2, 0\.4\]DFlash,T=1T=1External3\.225±\.0173\.362±\.004\+4\.2 \[3\.2, 5\.2\]Regenerated3\.789±\.0093\.783±\.001−\-0\.2 \[−\-1\.2, 0\.9\]
Table 2:MAT after three epochs unless noted\. Subscripts show run standard deviations, brackets 95% bootstrap intervals\. Gains compare KD with CE and KD \+ gate with KD\.Table[3](https://arxiv.org/html/2610.29013#S3)b repeats the factorial comparison under three controls that cover all three objectives\. In each of them KD adds more than 10% and the gate more than 4% on external responses, whereas after full regeneration KD adds less than 5% and the gate less than 1%\. Figure[3](https://arxiv.org/html/2610.29013#S4.F3)shows that every control leaves the shrinkage in place\. Like the gate, the soft survival weighting of PARD\-2 adds little after regeneration\. Under the chain verifier the ratios of ratios stay close to those of the tree verifier, and when the target samples at temperature 1 they are−\-8\.2% for KD and−\-4\.0% for the gate\.

Table[4\.1](https://arxiv.org/html/2610.29013#S4.SS1)reports the three vision controls that train only CE and KD\. From random initialization, regenerated CE reaches 2\.828 after three epochs, close to the 2\.875 of external CE from the released drafter, yet KD adds 3\.0% over CE there against 9\.3% on external responses, so a higher baseline does not explain the smaller increment\. Training this control twice as long moves its ratio of ratios from−\-5\.8% to−\-8\.5%\. At one third of the learning rate, KD adds 12\.1% over CE on external responses, against 14\.4% in the factorial, and its ratio of ratios is−\-8\.3%\. With the L2 anchor on both objectives, KD adds 10\.2% over CE on external and 1\.7% on regenerated responses, a ratio of ratios of−\-7\.7% against−\-9\.4% without the anchor, so most of the shrinkage remains\. Appendix[C](https://arxiv.org/html/2610.29013#A3)gives every ratio of ratios, the cells with the soft weighting and the data orders\.

### 4\.2Shrinkage persists at matched length

Table[3](https://arxiv.org/html/2610.29013#S3)a also shows that most of the shrinkage remains when the supervised length is matched\. On ALLaVA, regenerated responses are 3\.5 times as long as external ones on average, so full regeneration also adds supervised positions and anchors\. After truncation and with the loss window, KD adds 6\.9% and 6\.2% over CE, less than half of its external increment, and both ratios of ratios are negative in every domain and run\.

In Table[3](https://arxiv.org/html/2610.29013#S3)a the two length controls disagree about the gate, which adds 2\.2% under truncation and 0\.2% under the loss window, a ratio of ratios between them of 2\.0%\. Relative to the loss window, truncation lowers CE by 2\.4% and KD by 1\.7% but leaves KD \+ gate within 0\.3%, so the gate recovers what truncation takes from the ungated objectives\. A length control built by cutting responses can therefore show a gate gain that the loss window does not\. Masking the turn\-closing tokens that the cut leaves as training targets removes 80% of the gap between the gate’s gains under the two treatments and leaves no measurable gain beyond the loss window’s, so turn closure explains most of the gate’s truncation\-only gain\. Under the chain verifier, the two gate ratios of ratios in Figure[3](https://arxiv.org/html/2610.29013#S4.F3)lie within one point of each other\.

Figure 4:\(a\) Vision MAT of CE and KD on ALLaVA by training length\. \(b\) KD gain over CE with 95% prompt bootstrap intervals\. Every point is a separately trained and annealed schedule, not a checkpoint of a longer run\.### 4\.3Training length moves objective gains

Figure[4](https://arxiv.org/html/2610.29013#S4.F4)varies the length of training\. On external responses the KD gain grows with training beyond half an epoch on both corpora, because CE degrades with longer training while KD changes less\. On ALLaVA, CE loses 8\.2% of its one\-epoch MAT by the third epoch and KD 3\.2%\. On regenerated ALLaVA responses the gain grows from 0\.9% after one epoch to 3\.7% after three because KD improves while CE stays within 1\.4% of its two\-epoch MAT\. It reaches 5\.6% after four epochs and 9\.6% after six because both objectives then degrade and CE degrades more, as on external responses\. A comparison at a single training length can therefore overstate or understate an objective’s gain\. Regenerated ALLaVA KD peaks at three epochs, inside the measured range\. At each objective’s best measured schedule, KD adds 8\.4% over CE on external and 2\.4% on regenerated ALLaVA responses, a ratio of ratios of−\-5\.6%\. This is about three fifths of the three\-epoch shrinkage and remains well below zero\. For the gate, the same comparison gives 2\.7% over KD on external and−\-0\.5% on regenerated ALLaVA responses, a ratio of ratios of−\-3\.1%\. Table[3](https://arxiv.org/html/2610.29013#S4.T3)lists every CE and KD schedule, and Table[B](https://arxiv.org/html/2610.29013#A2)the gate schedules\.

Table 3:Vision MAT and KD gains over CE across corpora, response sources and training lengths\. Each schedule is trained and annealed separately\.On ALLaVA, after one epoch the KD ratio of ratios is−\-7\.0% and is negative in every domain and run\. The best schedules are those with the highest MAT on the evaluation prompts\. On external ALLaVA responses they are half an epoch for CE and one epoch for KD, and on regenerated ones two epochs for CE and three for KD\. Training external ALLaVA CE for a quarter epoch lowers its MAT by 1\.2% relative to half an epoch, so half an epoch remains its best measured schedule\. When the schedules are instead selected on alternate prompts of each domain and the ratio of ratios is estimated on the remaining prompts, both halves select the same ALLaVA schedules, and the KD best\-schedule ratio of ratios averaged over the two directions is−\-5\.6%\.

From three to six epochs, regenerated ALLaVA MAT falls by 0\.35 for CE and by 0\.16 for KD\. Regenerated ALLaVA CE is best at two epochs, so both regenerated optima lie inside the measured range\. Restricted to the one\-, two\- and three\-epoch schedules that both sources share, the KD best\-schedule ratio of ratios is−\-5\.7%\. On ShareGPT4V, comparing each objective at its best measured schedule leaves KD 10\.0% ahead of CE on external and 0\.7% on regenerated responses, a ratio of ratios of−\-8\.4%\.

### 4\.4A text\-only replication

Table[4\.1](https://arxiv.org/html/2610.29013#S4.SS1)and Figure[3](https://arxiv.org/html/2610.29013#S4.F3)also include the text\-only replication\. KD adds 7\.3% over CE on external and 1\.0% on regenerated responses under greedy decoding, a ratio of ratios of−\-5\.9%, and the ratio of ratios under sampling is−\-5\.5%\. The ratio of ratios is negative in all seven tasks under greedy decoding and in six under sampling, the exception being AIME 2025 with 30 prompts at\+\+0\.6%\. The gate adds 4\.6% over KD on external responses and no measurable gain on regenerated ones under greedy decoding, a ratio of ratios of−\-4\.3%, and its ratio of ratios under sampling is−\-4\.2%\. Appendix[G](https://arxiv.org/html/2610.29013#A7)reports the results for each task\.

Table[4\.1](https://arxiv.org/html/2610.29013#S4.SS1)also lists an autoregressive EAGLE\-3 drafter trained with CE and KD on the same two versions\. KD adds 13\.7% over CE on external and 2\.2% on regenerated responses under greedy decoding, a ratio of ratios of−\-10\.2% that is negative in all seven tasks and in both training runs\.

## 5Why the Increments Shrink

#### The corpus already carries the target’s greedy token\.

The corpus columns of Table[3](https://arxiv.org/html/2610.29013#S3)a measure how often the corpus token equals the target’s greedy token\. On ALLaVA, this agreement rises from 58\.0% on external responses to 95\.2% after full regeneration, and on ShareGPT4V from 64\.7% to 94\.9%\. With greedily regenerated responses, CE already trains toward the target’s greedy prediction at nearly every slot, KD can add only the probability mass away from that token, and the gate has few continuations left to remove\. Agreement separates the external versions from the target\-written ones, but it does not order the versions within each group by their KD gains\. Training CE and KD on three target samples of every prompt at temperature 1 for the same number of updates removes most of KD’s gain on sampled responses, so more target samples also supply most of that gain\. Appendix[E](https://arxiv.org/html/2610.29013#A5)defines agreement exactly and compares it across versions, and Appendix[C](https://arxiv.org/html/2610.29013#A3)reports the three\-sample cells\.

Figure 5:Objective gains when a share of the external ALLaVA responses is replaced by the target’s greedy regenerations, with 95% intervals\.
#### The gains fall with the share the target wrote\.

We also train each objective, twice, on mixtures that replace a random quarter, half or three quarters of the external ALLaVA responses with their greedy regenerations, keeping the prompts, their order and the training length of the external cells\. As Figure[5](https://arxiv.org/html/2610.29013#S5.F5)shows, both gains decline as the regenerated share grows, while agreement rises\. Most of the KD decline comes with the first quarter: regenerating a quarter of the responses already covers about two thirds of the drop from external to full regeneration, so a KD gain measured on such a mixture is closer to the fully regenerated gain than to the external one\. Appendix[D](https://arxiv.org/html/2610.29013#A4)lists every mixture\.

#### Greedy labels carry the part of the external gain that regeneration removes\.

A label control on ALLaVA separates two parts of what KD adds\. Training on the target’s greedy token in place of the corpus token at every position, with the corpus prefix unchanged, raises MAT by 9\.3% over CE on external responses, and KD on the same prefixes adds a further 4\.7%\. On fully regenerated responses the same control adds no measurable gain over CE, and KD adds 3\.9% over it, close to the 3\.7% that KD keeps over CE\. In the vision study, regeneration thus removes the greedy\-label part of the external KD gain and leaves most of what KD adds beyond the target’s greedy token\. In the text replication under greedy decoding, the same control matches KD on external responses and adds only 0\.3% over CE on regenerated ones\. Appendix[B](https://arxiv.org/html/2610.29013#A2)lists the control for each domain and Appendix[G](https://arxiv.org/html/2610.29013#A7)for each task\.

#### The remaining KD gain lies beyond the drafter’s first choice\.

On external ALLaVA responses KD mostly corrects the drafter’s first choice, whereas after greedy regeneration most of its gain lies in how the other tree candidates are ranked\. Appendix[D](https://arxiv.org/html/2610.29013#A4)shows this by scoring each drafter on the target’s greedy text\.

#### Full regeneration outweighs the objective settings we tried\.

From the released drafter’s 2\.835, three epochs of CE on external ALLaVA responses end 1\.4% higher, whereas every regenerated cell of Table[3](https://arxiv.org/html/2610.29013#S3)a ends at least 22\.9% higher\. Fully regenerated CE also exceeds every objective setting we trained on external ALLaVA responses\. It is 8\.5% above the best of these, KD with the soft survival weighting of PARD\-2 andγ=2\\gamma=2\. Table[4\.1](https://arxiv.org/html/2610.29013#S4.SS1)and Appendix[F](https://arxiv.org/html/2610.29013#A6)list the settings\.

## 6Related Work

#### Drafters and objectives\.

Work on speculative decoding\([Leviathan et al\., 2023](https://arxiv.org/html/2610.29013#bib.bib32);[Chen et al\., 2023](https://arxiv.org/html/2610.29013#bib.bib6)\)has proposed both drafter architectures and training objectives\. Autoregressive drafters predict one token at a time\([Li et al\., 2024a](https://arxiv.org/html/2610.29013#bib.bib34);[Li et al\., 2024b](https://arxiv.org/html/2610.29013#bib.bib35);[Li et al\., 2025](https://arxiv.org/html/2610.29013#bib.bib36)\), and parallel drafters predict several future tokens in one forward pass\([Cai et al\., 2024](https://arxiv.org/html/2610.29013#bib.bib5);[An et al\., 2026a](https://arxiv.org/html/2610.29013#bib.bib2);[Chen et al\., 2026](https://arxiv.org/html/2610.29013#bib.bib8);[Lee and Eo, 2026](https://arxiv.org/html/2610.29013#bib.bib22);[Lee et al\., 2026d](https://arxiv.org/html/2610.29013#bib.bib27)\)\. Distillation objectives train the drafter to match the target’s token distributions\([Liu et al\., 2024](https://arxiv.org/html/2610.29013#bib.bib38);[Zhou et al\., 2024](https://arxiv.org/html/2610.29013#bib.bib47);[Goel et al\., 2024](https://arxiv.org/html/2610.29013#bib.bib14);[Zhang et al\., 2025](https://arxiv.org/html/2610.29013#bib.bib44)\)\. Other objectives give less weight to the positions that follow a corpus token the target or the drafter is unlikely to produce\. PARD\-2 weights tokens by the cumulative target confidence of the preceding corpus tokens\([An et al\., 2026b](https://arxiv.org/html/2610.29013#bib.bib3)\), and GRIFFIN masks the loss after the first corpus token outside the drafter’s own top\-kkpredictions\([Hu et al\., 2025](https://arxiv.org/html/2610.29013#bib.bib17)\)\. Our gate likewise masks the loss after the first corpus token that is not the target’s greedy token, so it is a hard counterpart of the PARD\-2 weighting\. We also test the PARD\-2 weighting itself in the control withγ=2\\gamma=2\.

#### Who writes the training responses\.

Training drafters on target\-generated responses is established practice\([Cai et al\., 2024](https://arxiv.org/html/2610.29013#bib.bib5);[Goel et al\., 2024](https://arxiv.org/html/2610.29013#bib.bib14);[Zafrir et al\., 2025](https://arxiv.org/html/2610.29013#bib.bib43);[Ganesan et al\., 2025](https://arxiv.org/html/2610.29013#bib.bib13)\), and DFlash, the drafter family used in both of our studies, regenerates its training responses with the target\([Chen et al\., 2026](https://arxiv.org/html/2610.29013#bib.bib8)\)\. On such responses,[Hong et al\. \(2025\)](https://arxiv.org/html/2610.29013#bib.bib16)find that distilling the target’s token distributions gives drafters for domain\-specific targets a 2% to 10% higher acceptance rate than cross\-entropy on the same responses\. Other drafters are trained and compared on fixed responses\. EAGLE uses a fixed corpus to avoid the cost of generation\([Li et al\., 2024b](https://arxiv.org/html/2610.29013#bib.bib35)\), and HASS compares its distillation loss with six alternative losses on that corpus\([Zhang et al\., 2025](https://arxiv.org/html/2610.29013#bib.bib44)\)\. Two concurrent works propose a draft\-training method that relabels an external corpus with greedy target rollouts\([Lee et al\., 2026h](https://arxiv.org/html/2610.29013#bib.bib31)\)and a vision\-language block drafter trained on the target’s greedy generations\([Lee et al\., 2026b](https://arxiv.org/html/2610.29013#bib.bib24)\)\. We propose neither and instead treat the response source as an experimental factor, train the same objectives on both sources, and measure how regeneration changes what each objective adds\. Student\-generated data with target feedback\([Agarwal et al\., 2024](https://arxiv.org/html/2610.29013#bib.bib1);[Zhou et al\., 2024](https://arxiv.org/html/2610.29013#bib.bib47)\)is a different factor, because it trains on the drafter’s own outputs\.

#### Distillation data beyond drafting\.

Sequence\-level distillation trains a student on teacher\-generated sequences instead of matching token distributions, and in translation, adding word\-level distillation on top of it changed BLEU by less than one point for every student\([Kim and Rush, 2016](https://arxiv.org/html/2610.29013#bib.bib20)\)\. Distilled corpora differ in complexity in ways whose value depends on student capacity\([Zhou et al\., 2020](https://arxiv.org/html/2610.29013#bib.bib46)\)\. In language model distillation, how the generated training data are constructed can decide whether distillation helps\([Lee et al\., 2026g](https://arxiv.org/html/2610.29013#bib.bib30);[Lee et al\., 2026a](https://arxiv.org/html/2610.29013#bib.bib23)\), and the effect of a training objective can depend on the amount of available data\([Lee et al\., 2025](https://arxiv.org/html/2610.29013#bib.bib26)\)\. Draft training on regenerated responses is sequence\-level distillation from the target, with KD as the word\-level term\.

## 7Discussion

We find that the gain of a draft\-training objective depends on who wrote the training responses\. In our vision\-language study, distillation and the survival gate add large gains on external responses of both corpora, but once the target regenerates them greedily, distillation adds a small gain and the gate little or none\. Even at matched supervised length on ALLaVA or at each objective’s best measured training length, distillation adds less than half of its external gain, and a text\-only replication shows the shrinkage too, for distillation also with an autoregressive drafter\. Because regeneration is already common in draft training, the external\-corpus gain of a new objective can overstate what it adds in such a pipeline\.

#### Which gains survive regeneration\.

We expect objectives that repair a mismatch between corpus and target to lose most of their value in pipelines that regenerate\. The gate and the soft survival weighting discount corpus continuations that the target would not produce, and greedy regeneration leaves few such continuations\. KD instead adds the target’s probability mass beyond its greedy token, which regenerated responses do not contain, and keeps a small gain\. HASS, which adds a top\-KKdistillation loss and trains the drafter on its own predicted features, likewise keeps its gain over EAGLE\-2\([Li et al\., 2024a](https://arxiv.org/html/2610.29013#bib.bib34)\)when both are trained on target\-generated rather than fixed responses\([Zhang et al\., 2025](https://arxiv.org/html/2610.29013#bib.bib44)\)\.

#### Reporting objective gains\.

For an objective meant for pipelines that regenerate their responses, we suggest also measuring its gain over CE on responses that the target regenerates with the pipeline’s own decoding\. Regenerating the ALLaVA and text corpora cost less than one training run, as Appendix[A](https://arxiv.org/html/2610.29013#A1)shows\. KD already computes the target’s distribution at every corpus position, so the agreement between corpus tokens and the target’s greedy tokens comes at no extra cost wherever KD is trained\. Reporting this agreement beside an objective gain shows how far the training responses are from the target’s own greedy predictions\. Gains on external responses should be compared across training lengths or at each objective’s best schedule, because the external KD gain roughly doubled from the shortest to the longest schedule we trained\.

## Limitations

#### Drafters and targets\.

We study two drafter families with two Qwen3 targets: a DFlash\-style parallel block drafter in both studies and an autoregressive EAGLE\-3 drafter in the text study, where it is trained only with CE and KD and evaluated only under greedy decoding\. Other drafter families, the gate on an autoregressive drafter and targets from other families may respond differently\. Apart from one control, every vision drafter continues training a released drafter\. That control shows the distillation shrinkage from random initialization at a smaller magnitude after three epochs, so the magnitudes we report describe continued training\. When the drafters of both objectives are anchored to the released drafter, most of the distillation shrinkage remains\.

#### Corpora and training lengths\.

Limited compute kept each training corpus to a fixed subset of at most 26,196 examples, and we did not extend the study to the full corpora\. The second corpus, ShareGPT4V, has only external and fully regenerated responses, and the matched\-length versions and the controls cover ALLaVA alone\. The horizon study measures the gate only on ALLaVA\. Its best schedules are selected on the evaluation prompts, although the best\-schedule ratio of ratios stays below zero on both corpora when the schedules are chosen on one half of the prompts and the ratio is estimated on the other\.

#### Measurement\.

MAT depends on the verification decoder\. We measure it with a tree and a chain verifier in the vision study and, in the text study, with a block verifier for the DFlash drafter and the EAGLE\-3 tree verifier for the EAGLE\-3 drafter, and we do not measure wall\-clock speed or answer quality\. The vision verifiers accept drafts against the target’s greedy token computed within each batched bf16 verification pass, so MAT describes acceptance under that execution\.

#### Runs and intervals\.

Each DFlash cell has two training runs with one data order and each EAGLE\-3 cell two runs in two data orders, six vision cells add one run in each of two further data orders, and each domain has 99 scored prompts\. The bootstrap intervals cover prompt variation for the trained checkpoints, and the data orders vary only for those six cells and the EAGLE\-3 cells\. Appendix[B](https://arxiv.org/html/2610.29013#A2)also resamples the two runs of each cell, and the full\-regeneration intervals of both corpora stay below zero\. Two runs give only a coarse account of run variation, and no interval covers variation across corpora\.

#### Objectives and treatments\.

The factorial uses a hard survival gate, one distillation divergence,γ=7\\gamma=7and one learning rate\. The soft survival weighting andγ=2\\gamma=2are tested only on external and fully regenerated responses, and the lower learning rate only with CE and KD on those responses\. Other divergences and position weights may shrink differently after regeneration\. The response treatments change content, length, turn boundaries and anchor support together, so the matched\-length controls bound the role of length without isolating it\.

## Ethical Considerations

This work trains and evaluates draft models with public datasets and released models within their licenses\. It involves no human subjects and collects no data from people\. A draft model changes how quickly its target generates, not what the target can produce, so the behavior and risks of the target carry over unchanged\. Appendix[A\.5](https://arxiv.org/html/2610.29013#A1.SS5)lists the licenses of all artifacts\.

## References

- Agarwal et al\. \(2024\)Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem\. 2024\.[On\-Policy Distillation of Language Models: Learning from Self\-Generated Mistakes](https://proceedings.iclr.cc/paper_files/paper/2024/hash/5be69a584901a26c521c2b51e40a4c20-Abstract-Conference.html)\.In*International Conference on Learning Representations*\.
- An et al\. \(2026a\)Zihao An, Huajun Bai, Ziqiong Liu, Dong Li, and Emad Barsoum\. 2026a\.[PARD: Accelerating LLM Inference with Low\-Cost PARallel Draft Model Adaptation](https://openreview.net/forum?id=XbOyv7iVGL)\.In*International Conference on Learning Representations*\.
- An et al\. \(2026b\)Zihao An, Taichi Liu, Ziqiong Liu, Dong Li, Ruofeng Liu, and Emad Barsoum\. 2026b\.[PARD\-2: Target\-Aligned Parallel Draft Model for Dual\-Mode Speculative Decoding](https://doi.org/10.48550/arxiv.2605.08632)\.*arXiv preprint arXiv:2605\.08632*\.
- Bai et al\. \(2025\)Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others\. 2025\.[Qwen3\-VL Technical Report](https://doi.org/10.48550/arxiv.2511.21631)\.*arXiv preprint arXiv:2511\.21631*\.
- Cai et al\. \(2024\)Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D\. Lee, Deming Chen, and Tri Dao\. 2024\.[Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://proceedings.mlr.press/v235/cai24b.html)\.In*International Conference on Machine Learning*\.
- Chen et al\. \(2023\)Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean\-Baptiste Lespiau, Laurent Sifre, and John Jumper\. 2023\.[Accelerating Large Language Model Decoding with Speculative Sampling](https://doi.org/10.48550/arxiv.2302.01318)\.*arXiv preprint arXiv:2302\.01318*\.
- Chen et al\. \(2024a\)Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang\. 2024a\.[ALLaVA: Harnessing GPT4V\-Synthesized Data for Lite Vision\-Language Models](https://doi.org/10.48550/arxiv.2402.11684)\.*arXiv preprint arXiv:2402\.11684*\.
- Chen et al\. \(2026\)Jian Chen, Yesheng Liang, and Zhijian Liu\. 2026\.[DFlash: Block Diffusion for Flash Speculative Decoding](https://openreview.net/forum?id=Oz335dV48X)\.In*International Conference on Machine Learning*\.
- Chen et al\. \(2024b\)Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin\. 2024b\.[ShareGPT4V: Improving Large Multi\-modal Models with Better Captions](https://doi.org/10.1007/978-3-031-72643-9_22)\.In*Computer Vision – ECCV 2024*, volume 15075 of*Lecture Notes in Computer Science*, pages 370–387\. Springer\.
- Chen et al\. \(2021\)Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others\. 2021\.[Evaluating Large Language Models Trained on Code](https://arxiv.org/abs/2107.03374)\.*arXiv preprint arXiv:2107\.03374*\.
- Cobbe et al\. \(2021\)Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman\. 2021\.[Training Verifiers to Solve Math Word Problems](https://arxiv.org/abs/2110.14168)\.*arXiv preprint arXiv:2110\.14168*\.
- Ding et al\. \(2023\)Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou\. 2023\.[Enhancing Chat Language Models by Scaling High\-quality Instructional Conversations](https://doi.org/10.18653/v1/2023.emnlp-main.183)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 3029–3051\. Association for Computational Linguistics\.
- Ganesan et al\. \(2025\)Mugilan Ganesan, Shane Segal, Ankur Aggarwal, Nish Sinnadurai, Sean Lie, and Vithursan Thangarasa\. 2025\.[MASSV: Multimodal Adaptation and Self\-Data Distillation for Speculative Decoding of Vision\-Language Models](https://doi.org/10.18653/v1/2025.findings-emnlp.656)\.In*Findings of the Association for Computational Linguistics: EMNLP 2025*, pages 12265–12276\. Association for Computational Linguistics\.
- Goel et al\. \(2024\)Raghavv Goel, Mukul Gagrani, Wonseok Jeon, Junyoung Park, Mingu Lee, and Christopher Lott\. 2024\.[Direct Alignment of Draft Model for Speculative Decoding with Chat\-Fine\-Tuned LLMs](https://doi.org/10.48550/arxiv.2403.00858)\.*arXiv preprint arXiv:2403\.00858*\.
- Hinton et al\. \(2015\)Geoffrey Hinton, Oriol Vinyals, and Jeff Dean\. 2015\.[Distilling the Knowledge in a Neural Network](https://doi.org/10.48550/arxiv.1503.02531)\.*arXiv preprint arXiv:1503\.02531*\.
- Hong et al\. \(2025\)Fenglu Hong, Ravi Shanker Raju, Jonathan Lingjie Li, Bo Li, Urmish Thakker, Avinash Ravichandran, Swayambhoo Jain, and Changran Hu\. 2025\.[Training Domain Draft Models for Speculative Decoding: Best Practices and Insights](https://openreview.net/forum?id=7FBJIxYXxd)\.In*First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models*\.
- Hu et al\. \(2025\)Shijing Hu, Jingyang Li, Xingyu Xie, Zhihui Lu, Kim\-chuan Toh, and Pan Zhou\. 2025\.[GRIFFIN: Effective Token Alignment for Faster Speculative Decoding](https://doi.org/10.52202/085713-4188)\.In*Advances in Neural Information Processing Systems 38*\.
- Jain et al\. \(2025\)Naman Jain, King Han, Alex Gu, Wen\-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar\-Lezama, Koushik Sen, and Ion Stoica\. 2025\.[LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://openreview.net/forum?id=chfJJYC3iL)\.In*International Conference on Learning Representations*\.
- Kang et al\. \(2026\)Myunghoon Kang, Jungseob Lee, Jaehyung Seo, and Heuiseok Lim\. 2026\.[ThinkFuse: Trajectory\-Aware Test\-Time Fusion for Small Reasoning Models](https://arxiv.org/abs/2610.07803)\.*arXiv preprint arXiv:2610\.07803*\.
- Kim and Rush \(2016\)Yoon Kim and Alexander M\. Rush\. 2016\.[Sequence\-Level Knowledge Distillation](https://doi.org/10.18653/v1/d16-1139)\.In*Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing*, pages 1317–1327\. Association for Computational Linguistics\.
- Kwon et al\. \(2023\)Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica\. 2023\.[Efficient Memory Management for Large Language Model Serving with PagedAttention](https://doi.org/10.1145/3600006.3613165)\.In*Proceedings of the 29th Symposium on Operating Systems Principles*\.
- Lee and Eo \(2026\)Jungseob Lee and Sugyeong Eo\. 2026\.[CAST: Cost\-Aware Speculative Trees from One\-Pass Block Drafters](https://arxiv.org/abs/2610.00321)\.*arXiv preprint arXiv:2610\.00321*\.
- Lee et al\. \(2026a\)Jungseob Lee, Sugyeong Eo, Seongtae Hong, Seungyoon Lee, Chanjun Park, Jaehyung Seo, and Heuiseok Lim\. 2026a\.[Distilling Directional Verification](https://arxiv.org/abs/2610.00997)\.*arXiv preprint arXiv:2610\.00997*\.
- Lee et al\. \(2026b\)Jungseob Lee, Seongtae Hong, Dongyub Jude Lee, Chanjun Park, Jaehyung Seo, Sugyeong Eo, and Heuiseok Lim\. 2026b\.[Vision Is Not Overhead: One\-Pass Block Drafting for Lossless Speculative Decoding in Vision\-Language Models](https://arxiv.org/abs/2609.00355)\.*arXiv preprint arXiv:2609\.00355*\.
- Lee et al\. \(2026c\)Jungseob Lee, Seongtae Hong, Seungjun Lee, Jaehyung Seo, Junyoung Son, Sugyeong Eo, Chanjun Park, Hyeongju Park, Hyeonseok Moon, and Heuiseok Lim\. 2026c\.[DART: Draft\-Agreement Routing for Training\-Free Adaptive Thinking Budgets in Hybrid Reasoning Models](https://arxiv.org/abs/2606.23181)\.*arXiv preprint arXiv:2606\.23181*\.
- Lee et al\. \(2025\)Jungseob Lee, Seongtae Hong, Hyeonseok Moon, and Heuiseok Lim\. 2025\.[Cross\-Lingual Optimization for Language Transfer in Large Language Models](https://doi.org/10.18653/v1/2025.acl-long.734)\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 15100–15119\. Association for Computational Linguistics\.
- Lee et al\. \(2026d\)Jungseob Lee, Dongyub Jude Lee, Chanjun Park, Sugyeong Eo, and Heuiseok Lim\. 2026d\.[Faster Block\-Diffusion Serving with Distribution\-Free Risk Guarantees](https://arxiv.org/abs/2609.33887)\.*arXiv preprint arXiv:2609\.33887*\.
- Lee et al\. \(2026e\)Jungseob Lee, Seungyoon Lee, Seongtae Hong, Sugyeong Eo, and Heuiseok Lim\. 2026e\.[Where Activation Sparsity and KV\-Cache Sparsity Cross in LLM Decoding](https://arxiv.org/abs/2609.33889)\.*arXiv preprint arXiv:2609\.33889*\.
- Lee et al\. \(2026f\)Jungseob Lee, Seungyoon Lee, Seongtae Hong, Minhyuk Kim, Chanjun Park, and Heuiseok Lim\. 2026f\.[Beyond Penalizing Mistakes: Stabilizing Efficiency Training in Large Reasoning Models via Adaptive Correct\-Only Rewards](https://arxiv.org/abs/2606.22716)\.*arXiv preprint arXiv:2606\.22716*\.
- Lee et al\. \(2026g\)Jungseob Lee, Seungyoon Lee, Suhyune Son, Dongyub Jude Lee, Sungbin Han, Sugyeong Eo, and Heuiseok Lim\. 2026g\.[Answer\-Conditioned Chains of Thought Degrade Verifiable\-Reasoning Distillation in Large Language Models](https://doi.org/10.48550/arxiv.2607.14552)\.*arXiv preprint arXiv:2607\.14552*\.
- Lee et al\. \(2026h\)Jungseob Lee, Chanjun Park, Sugyeong Eo, and Hyeonseok Moon\. 2026h\.[Recovering Off\-Policy Supervision for Speculative Decoding](https://arxiv.org/abs/2609.38795)\.*arXiv preprint arXiv:2609\.38795*\.
- Leviathan et al\. \(2023\)Yaniv Leviathan, Matan Kalman, and Yossi Matias\. 2023\.[Fast Inference from Transformers via Speculative Decoding](https://proceedings.mlr.press/v202/leviathan23a.html)\.In*Proceedings of the 40th International Conference on Machine Learning*, volume 202 of*Proceedings of Machine Learning Research*, pages 19274–19286\. PMLR\.
- Li et al\. \(2026\)Shenggui Li, Chao Wang, Yikai Zhu, Yubo Wang, Fan Yin, Shuai Shi, Yefei Chen, Xiaomin Dong, Qiaoling Chen, Jin Pan, Ji Li, Laixin Xie, Yineng Zhang, Lei Yu, Yonggang Wen, Ivor Tsang, and Tianwei Zhang\. 2026\.[SpecForge: A Flexible and Efficient Open\-Source Training Framework for Speculative Decoding](https://arxiv.org/abs/2603.18567)\.*arXiv preprint arXiv:2603\.18567*\.
- Li et al\. \(2024a\)Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang\. 2024a\.[EAGLE\-2: Faster Inference of Language Models with Dynamic Draft Trees](https://doi.org/10.18653/v1/2024.emnlp-main.422)\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*\.
- Li et al\. \(2024b\)Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang\. 2024b\.[EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://proceedings.mlr.press/v235/li24bt.html)\.In*International Conference on Machine Learning*\.
- Li et al\. \(2025\)Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang\. 2025\.[EAGLE\-3: Scaling up Inference Acceleration of Large Language Models via Training\-Time Test](https://doi.org/10.52202/085713-4562)\.In*Advances in Neural Information Processing Systems 38*\.
- Lin et al\. \(2014\)Tsung\-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C\. Lawrence Zitnick\. 2014\.[Microsoft COCO: Common Objects in Context](https://doi.org/10.1007/978-3-319-10602-1_48)\.In*Computer Vision – ECCV 2014*, volume 8693 of*Lecture Notes in Computer Science*, pages 740–755\. Springer\.
- Liu et al\. \(2024\)Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang\. 2024\.[Online Speculative Decoding](https://proceedings.mlr.press/v235/liu24y.html)\.In*International Conference on Machine Learning*\.
- Mathew et al\. \(2021\)Minesh Mathew, Dimosthenis Karatzas, and C\. V\. Jawahar\. 2021\.[DocVQA: A Dataset for VQA on Document Images](https://doi.org/10.1109/wacv48630.2021.00225)\.In*2021 IEEE Winter Conference on Applications of Computer Vision \(WACV\)*, pages 2199–2208\. IEEE\.
- Singh et al\. \(2019\)Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach\. 2019\.[Towards VQA Models That Can Read](https://doi.org/10.1109/cvpr.2019.00851)\.In*2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 8309–8318\. IEEE\.
- Taori et al\. \(2023\)Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B\. Hashimoto\. 2023\.[Stanford Alpaca: An Instruction\-following LLaMA model](https://github.com/tatsu-lab/stanford_alpaca)\.GitHub repository\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others\. 2025\.[Qwen3 Technical Report](https://arxiv.org/abs/2505.09388)\.*arXiv preprint arXiv:2505\.09388*\.
- Zafrir et al\. \(2025\)Ofir Zafrir, Igor Margulis, Dorin Shteyman, Shira Guskin, and Guy Boudoukh\. 2025\.[FastDraft: How to Train Your Draft](https://doi.org/10.18653/v1/2025.findings-acl.1156)\.In*Findings of the Association for Computational Linguistics: ACL 2025*, pages 22488–22505\. Association for Computational Linguistics\.
- Zhang et al\. \(2025\)Lefan Zhang, Xiaodan Wang, Yanhua Huang, and Ruiwen Xu\. 2025\.[Learning Harmonized Representations for Speculative Sampling](https://proceedings.iclr.cc/paper_files/paper/2025/hash/575286a73f238b6516ce0467d67eadb2-Abstract-Conference.html)\.In*International Conference on Learning Representations*\.
- Zheng et al\. \(2023\)Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph Gonzalez, and Ion Stoica\. 2023\.[Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena](https://doi.org/10.52202/075280-2020)\.In*Advances in Neural Information Processing Systems*, volume 36\.
- Zhou et al\. \(2020\)Chunting Zhou, Jiatao Gu, and Graham Neubig\. 2020\.[Understanding Knowledge Distillation in Non\-autoregressive Machine Translation](https://openreview.net/forum?id=BygFVAEKDH)\.In*International Conference on Learning Representations*\.
- Zhou et al\. \(2024\)Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean\-François Kagy, and Rishabh Agarwal\. 2024\.[DistillSpec: Improving Speculative Decoding via Knowledge Distillation](https://proceedings.iclr.cc/paper_files/paper/2024/hash/8766fbc68e1ed1cdef712ce273e0a363-Abstract-Conference.html)\.In*International Conference on Learning Representations*\.

## Appendix AProtocol Details

### A\.1Vision training

Examples come from the ALLaVA LAION instruction file in a fixed random order\. We keep the first 26,196 rows whose conversation opens with a user turn and an assistant turn of at least 40 characters and whose image belongs to the first released image archive, and we train on that first exchange\. The ShareGPT4V subset takes the 30,000 GPT\-4V captions of its 100K caption file whose images come from LAION, CC and SBU, sorts them by identifier, shuffles them with a fixed seed and keeps the first 26,196\. On ShareGPT4V, the three objectives are trained on external and fully regenerated responses, both of which also enter the horizon study, and KD and KD \+ gate withγ=2\\gamma=2are trained once on the external responses, as Appendix[F](https://arxiv.org/html/2610.29013#A6)lists\. Examples with fewer than 32 supervised assistant tokens are removed\. On ALLaVA this leaves 26,101 external, truncated and loss\-window examples and 26,193 fully regenerated examples, so three epochs take 3,261 or 3,273 updates\. No ShareGPT4V example falls below the threshold, so both ShareGPT4V versions keep all 26,196 examples and three epochs take 3,273 updates\.

The drafter has five layers and reads the hidden states of target layers 1, 9, 17, 25 and 33\. Training uses four GPUs with an effective batch of 24, AdamW with 4% warmup and cosine decay, bf16, SDPA attention, gradient clipping at norm 1 and a maximum length of 4,096 tokens, and images keep between 50,176 and 200,704 pixels\. The learning rate,1\.47×10−31\.47\\times 10^\{\-3\}, is the SpecForge\([Li et al\., 2026](https://arxiv.org/html/2610.29013#bib.bib33)\)default for DFlash training,6×10−46\\times 10^\{\-4\}, scaled by the square root of our six times larger batch on each GPU\. The KD loss uses the target distribution at temperature 1 and carries a fixed scale of 1\.013, the ratio of the median gradient norms of CE and KD on identical batches before training\.

Regeneration uses the target in vLLM\([Kwon et al\., 2023](https://arxiv.org/html/2610.29013#bib.bib21)\)with greedy decoding and the image bounds and chat template of training, and the 3\.8% of regenerated ALLaVA responses that reach the 768\-token limit are closed there\. Regenerating the ALLaVA corpus took about 5 minutes on four H100 GPUs, on which a three\-epoch ALLaVA run trains for 29 to 46 minutes\. Truncation cuts the regenerated content to the token count of the external response and closes the assistant turn\. The loss window keeps the full regenerated input and masks the loss after that count\. In the 60 retained examples whose regenerated response is shorter than the external one, both treatments keep the whole response\.

On ALLaVA, regenerated responses average 395 tokens against 112 for external ones, and a proxy for the anchors available in each example, the mean ofmin⁡\(128,L\)\\min\(128,L\)over response lengthsLL, is 106\.7 for external and 127\.3 for fully regenerated responses\. Fully regenerated ShareGPT4V responses keep the first 768 tokens of the target’s greedy response, and the 3\.9% that are longer are closed there\. They average 380 tokens against 203 for the external captions, and the anchor proxy is 127\.5 against 126\.7\.

### A\.2Vision evaluation

The captioning prompts are rows 100 to 199 of COCO val2017, the TextVQA prompts rows 0 to 99 of the public training split and the DocVQA prompts rows 400 to 499 of the validation split, each in release order\. The first prompt of each domain is a warm\-up that is not scored\. No evaluation image occurs byte for byte in the training corpus\. Images are resized to at most 896 pixels on the longer side\.

In each round the drafter predicts the 15 slots in one forward pass, and the verifier builds a new tree from these predictions\. The candidates at a slot are the drafter’s eight most probable tokens there, a path is scored by the sum of the drafter’s log\-probabilities of its tokens, and the tree keeps the 63 best nodes, each added only after its parent\. The target scores the whole tree in one forward pass with a tree attention mask and accepts the path that follows its own greedy tokens as far as the tree allows\. All vision MATs come from one GPU model, the RTX A6000, in bf16\.

Within each bootstrap resample, every run’s MAT is recomputed from the drawn prompts, and the runs of a cell are averaged before increments and ratios are formed\. MAT counts the accepted path and the target’s own token in each round before the final round is clipped at the end\-of\-sequence token or the length limit, so it can differ from delivered tokens in that last round\.

### A\.3Vision controls

Every control uses external and fully regenerated ALLaVA responses\. The controls with random initialization, one third of the learning rate and the L2 anchor train CE and KD, and the random\-initialization control trains them for three and for six epochs\. The control withγ=2\\gamma=2trains CE, KD, KD \+ gate and KD with the soft weighting, the control with two further data orders retrains the six factorial cells of the two versions once in each order, and the 64\-token control trains KD\. Apart from the data orders, control cells are trained twice\. The chain verifier re\-evaluates the 24 checkpoints of the external, fully regenerated and matched\-length versions\. The sampled evaluation re\-evaluates the 12 checkpoints of the external and fully regenerated versions with the tree verifier while the target samples at temperature 1, with random numbers fixed for each prompt and output position and shared by all drafters\.

The random\-initialization control keeps the drafter’s architecture and starts its five layers and feature projection from random weights, while the embeddings and output head come from the target as in every run\. Each six\-epoch run starts from the random weights and data order of its three\-epoch counterpart and changes only the number of epochs, so its warmup and cosine decay span 6,522 updates on external and 6,546 on fully regenerated responses\. The two further data orders shuffle the training examples into two other fixed random orders, and each of their six cells is trained once with the anchor sampling of the first factorial run, which represents the main order\. The control withγ=2\\gamma=2keeps the rest of the recipe, and its soft weighting multiplies the target’s probabilities of the preceding corpus tokens, as in PARD\-2\. The chain verifier runs the tree verifier’s routine with one candidate at each slot and seven slots in each round, on the same prompts and GPU model\. After every optimizer step, the L2 anchor moves the drafter’s weights toward those of the released drafter by the current learning rate times their difference, in both objectives\. The control with 64 distilled tokens keeps the rest of the KD recipe and reuses the factorial CE cells\.

To separate turn closure from the other differences at the cut, we also train KD and KD \+ gate twice on the truncated examples with the loss masked on the tokens that close the assistant turn after the cut, keeping the examples, their order and the training length\. The loss on these tokens stays in the examples that the cut leaves whole, whose closure is the target’s own, and in 15 cut examples that masking would leave with fewer than 32 supervised tokens\.

### A\.4Text study

The text corpus starts from the 26,196 conversations whose messages have the lowest SHA\-256 hashes in the first of the three supervised fine\-tuning files of UltraChat 200k\. Their first turns are kept when both responses leave at least 32 supervised tokens under the 1,024\-token limit\. Regenerated responses are the target’s greedy responses of at most 768 new tokens\. They are 1\.7 times as long as the original ones, 594 against 344 supervised tokens on average, and 56\.6% reach the 768\-token limit\. Regenerating the text corpus took 2\.9 GPU\-hours on one RTX A6000, against 7\.2 to 8\.9 hours for each three\-epoch DFlash text run on the same GPU model\.

Each of the eight DFlash cells, CE, KD, KD \+ gate and the label control of Section[5](https://arxiv.org/html/2610.29013#S5)on each version, is trained twice for three epochs on one RTX A6000 with the learning rate, effective batch andγ\\gammaof the vision study, using micro\-batches of 3, eight accumulation steps, 3,069 updates, a KD scale of 1 and a maximum length of 1,024 tokens\. Evaluation covers MT\-Bench\([Zheng et al\., 2023](https://arxiv.org/html/2610.29013#bib.bib45)\), Alpaca\([Taori et al\., 2023](https://arxiv.org/html/2610.29013#bib.bib41)\), GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2610.29013#bib.bib11)\), AIME 2024 and 2025, HumanEval\([Chen et al\., 2021](https://arxiv.org/html/2610.29013#bib.bib10)\)and LiveCodeBench\([Jain et al\., 2025](https://arxiv.org/html/2610.29013#bib.bib18)\)\. The seven tasks contribute 80, 80, 1,319, 30, 30, 164 and 400 prompts, with up to 256 new tokens\. AtT=1T=1the target samples and the drafter proposes its greedy tokens, and each prompt uses a fixed random seed\. Tasks take the place of domains in the aggregation and the bootstrap\.

#### EAGLE\-3 drafter\.

The second text drafter starts from a publicly released EAGLE\-3 drafter for Qwen3\-4B, one decoder layer that reads the fused hidden states of target layers 1, 17 and 32 and predicts over a fixed 32,000\-token subset of the target vocabulary, whose mapping we keep\. Each of its four cells is trained twice for three epochs with the micro\-batches, accumulation steps, updates, warmup, gradient clipping and maximum length of the DFlash text runs, with seven simulated test\-time steps and a learning rate of2\.45×10−42\.45\\times 10^\{\-4\}, the SpecForge default for EAGLE\-3 training,10−410^\{\-4\}, scaled by the same factor\. This drafter samples no anchors, so the two runs of a cell use two data orders\. CE and KD train the positions whose corpus token lies in the drafter’s vocabulary, and KD keeps the target’s eight most probable tokens within that vocabulary and merges the rest, including all probability outside the vocabulary, into a ninth outcome\. The fine\-tuned checkpoints and the released drafter are evaluated at temperature 0 on the same 2,103 prompts with the same token limit, using the public EAGLE\-3 decoder, which verifies a tree of 59 drafted tokens, at most eight deep, grown from the drafter’s ten most probable tokens at each step, so its MATs are compared only within this family\.

### A\.5Artifact licenses

Qwen3\-VL\-8B\-Instruct and Qwen3\-4B carry Apache 2\.0 and the DFlash drafters MIT\. ALLaVA, ShareGPT4V and the Alpaca prompts carry CC BY\-NC 4\.0, UltraChat 200k, GSM8K and HumanEval carry MIT, TextVQA carries CC BY 4\.0, and the MT\-Bench prompts and the formatted DocVQA release carry Apache 2\.0\. COCO images are used under the COCO terms of use, and LiveCodeBench and the AIME problems are public benchmark releases\. We use every artifact only for non\-commercial research\.

## Appendix BVision Results by Domain and Schedule

Table[B](https://arxiv.org/html/2610.29013#A2)lists MAT for every cell and domain, together with the label control of Section[5](https://arxiv.org/html/2610.29013#S5), and Table[B](https://arxiv.org/html/2610.29013#A2)lists the increments of the factorial and of ShareGPT4V for each domain\. The label control adds 9\.3% \[8\.6, 10\.0\] over CE on external responses and changes MAT by−\-0\.2% \[−\-0\.8, 0\.5\] on fully regenerated ones, a ratio of ratios of−\-8\.6% \[−\-9\.5,−\-7\.8\]\. KD adds 4\.7% \[4\.1, 5\.4\] over it on external and 3\.9% \[3\.2, 4\.6\] on fully regenerated responses\. Between the two length controls, the ratio of ratios is 0\.7% \[−\-0\.3, 1\.7\] for KD, which is not resolved\. With the turn\-closing tokens masked, MAT is 3\.779 under KD and 3\.803 under KD \+ gate, a gate gain of 0\.6%\. The ratio of ratios of the gate’s gains is−\-1\.6% \[−\-2\.3,−\-0\.8\] against truncation and 0\.4% \[−\-0\.4, 1\.3\] against the loss window\.

Table 4:Vision MAT for each domain, averaged over two training runs\. The last column is the geometric mean over domains\. Greedy labels trains CE toward the target’s greedy token\.When the two training runs of each cell are resampled along with the prompts, the intervals of the full\-regeneration ratios of ratios widen to \[−\-10\.6,−\-8\.1\] for KD and \[−\-6\.2,−\-4\.2\] for the gate\. On ShareGPT4V they are \[−\-13\.3,−\-11\.0\] for KD and \[−\-6\.0,−\-3\.7\] for the gate\. With the runs resampled, every ratio of ratios in Figure[3](https://arxiv.org/html/2610.29013#S4.F3)stays below zero, the highest upper bound being−\-1\.3%\. The gate’s ratio of ratios between truncation and the loss window becomes 2\.0% \[0\.7, 3\.4\], the gate’s increment on responses sampled at temperature 1 becomes 2\.1% \[0\.9, 3\.3\], and the KD contrast between the length controls becomes 0\.7% \[−\-0\.6, 2\.0\], which stays unresolved\.

Table[3](https://arxiv.org/html/2610.29013#S4.T3)lists the CE and KD schedules of the horizon study, Table[B](https://arxiv.org/html/2610.29013#A2)gives their domain values, and Table[B](https://arxiv.org/html/2610.29013#A2)lists the gate schedules\. The bootstrap holds each best schedule fixed\. Relative to half an epoch, a quarter epoch changes external ALLaVA CE by−\-1\.2% \[−\-2\.1,−\-0\.4\]\. On ShareGPT4V the best measured schedules are half an epoch for CE and KD on external responses and one epoch for CE and three for KD on regenerated ones\. When they are instead selected on alternate prompts of each domain and the ratio of ratios is estimated on the remaining prompts, the two halves select different regenerated KD schedules, one and three epochs, whose MATs differ by 0\.03%, and the best\-schedule ratio of ratios averaged over the two directions is−\-8\.6%\. For the gate over KD on the schedules of Table[B](https://arxiv.org/html/2610.29013#A2), both halves select the same schedules and the averaged ratio of ratios is−\-3\.2%\.

Table 5:Vision MAT for each domain, corpus, response source and training length, averaged over two training runs\. Each cell gives CE / KD\.Table 6:Vision MAT of KD and of KD with the survival gate on ALLaVA for each response source and training length, with the gate’s gain over KD\. Every row is a separately trained and annealed schedule\.Table 7:Relative MAT increments \(%\), overall with 95% prompt bootstrap intervals and for each domain\. KD compares with CE, and gate compares with KD\.## Appendix CControls of the Vision Results

Table[C](https://arxiv.org/html/2610.29013#A3)lists every ratio of ratios, including those plotted in Figure[3](https://arxiv.org/html/2610.29013#S4.F3), and each of its data\-order rows comes from one training run\. In Table[4\.1](https://arxiv.org/html/2610.29013#S4.SS1), every cell trained for three epochs from random initialization is 26% to 31% below the same cell trained from the released drafter in Table[3](https://arxiv.org/html/2610.29013#S3)a, and its KD ratio of ratios is smaller in magnitude,−\-5\.8% against−\-9\.4%, both after three epochs\. Trained twice as long, the control’s KD ratio of ratios,−\-8\.5%, is negative in every domain and training run, and its regenerated CE exceeds the released drafter by 8\.7%\.

SettingIncrementEstimate \[95% CI\]\[0pt\]\[0pt\]Vision, tree verifierFull regenerationKD−\-9\.4 \[−\-10\.2,−\-8\.5\]Full regenerationGate−\-5\.2 \[−\-6\.0,−\-4\.4\]Full regeneration,T=1T=1KD−\-8\.2 \[−\-9\.1,−\-7\.3\]Full regeneration,T=1T=1Gate−\-4\.0 \[−\-4\.8,−\-3\.1\]TruncatedKD−\-6\.6 \[−\-7\.5,−\-5\.6\]TruncatedGate−\-2\.6 \[−\-3\.4,−\-1\.8\]Loss windowKD−\-7\.2 \[−\-8\.1,−\-6\.3\]Loss windowGate−\-4\.5 \[−\-5\.3,−\-3\.7\]Sampled,T=1T=1KD−\-8\.9 \[−\-9\.7,−\-8\.2\]Sampled,T=1T=1Gate−\-2\.7 \[−\-3\.5,−\-1\.9\]Sampled,T=0\.7T=0\.7KD−\-9\.5 \[−\-10\.3,−\-8\.7\]Sampled,T=0\.7T=0\.7Gate−\-3\.7 \[−\-4\.5,−\-2\.8\]ShareGPT4V, full regenerationKD−\-12\.2 \[−\-13\.0,−\-11\.4\]ShareGPT4V, full regenerationGate−\-4\.8 \[−\-5\.7,−\-4\.0\]\[0pt\]\[0pt\]Vision, chain verifierFull regenerationKD−\-8\.9 \[−\-9\.6,−\-8\.1\]Full regenerationGate−\-4\.9 \[−\-5\.7,−\-4\.2\]TruncatedKD−\-6\.4 \[−\-7\.2,−\-5\.6\]TruncatedGate−\-4\.6 \[−\-5\.3,−\-3\.8\]Loss windowKD−\-7\.2 \[−\-8\.0,−\-6\.4\]Loss windowGate−\-5\.4 \[−\-6\.2,−\-4\.6\]\[0pt\]\[0pt\]Vision controls, full regenerationRandom initialization, three epochsKD−\-5\.8 \[−\-6\.5,−\-5\.2\]Random initialization, six epochsKD−\-8\.5 \[−\-9\.3,−\-7\.7\]γ=2\\gamma=2KD−\-10\.4 \[−\-11\.2,−\-9\.5\]γ=2\\gamma=2Gate−\-4\.6 \[−\-5\.5,−\-3\.8\]γ=2\\gamma=2Soft weight−\-4\.5 \[−\-5\.4,−\-3\.7\]Second data orderKD−\-10\.0 \[−\-11\.0,−\-8\.9\]Second data orderGate−\-3\.9 \[−\-5\.0,−\-2\.7\]Third data orderKD−\-10\.4 \[−\-11\.5,−\-9\.3\]Third data orderGate−\-3\.8 \[−\-5\.0,−\-2\.5\]Lower learning rateKD−\-8\.3 \[−\-9\.2,−\-7\.5\]L2 anchorKD−\-7\.7 \[−\-8\.5,−\-6\.9\]64 distilled tokensKD−\-9\.9 \[−\-10\.8,−\-9\.1\]Best schedulesKD−\-5\.6 \[−\-6\.4,−\-4\.8\]Best schedulesGate−\-3\.1 \[−\-4\.0,−\-2\.3\]ShareGPT4V, best schedulesKD−\-8\.4 \[−\-9\.4,−\-7\.4\]\[0pt\]\[0pt\]Text replicationDFlash,T=0T=0KD−\-5\.9 \[−\-6\.4,−\-5\.4\]DFlash,T=0T=0Gate−\-4\.3 \[−\-4\.7,−\-3\.8\]DFlash,T=1T=1KD−\-5\.5 \[−\-6\.8,−\-4\.2\]DFlash,T=1T=1Gate−\-4\.2 \[−\-5\.7,−\-2\.7\]EAGLE\-3,T=0T=0KD−\-10\.2 \[−\-10\.7,−\-9\.6\]
Table 8:Ratios of ratios \(%\) with 95% prompt bootstrap intervals, comparing each increment on the given responses with its counterpart on external responses\. KD compares with CE\. Gate and soft weight compare with KD\.Withγ=2\\gamma=2, KD with the soft survival weighting of PARD\-2 in place of the gate reaches 3\.542 on external and 4\.053 on fully regenerated responses, close to KD \+ gate in Table[3](https://arxiv.org/html/2610.29013#S3)b\. The lower learning rate,4\.9×10−44\.9\\times 10^\{\-4\}, raises external CE and KD by 7\.8% and 5\.6% over the factorial and changes fully regenerated CE and KD by 1\.3% and 0\.3%\. The L2 anchor holds every run to a relative weight drift of 0\.14 to 0\.16 from the released drafter, about half that of the factorial runs, and leaves 81% \[71, 93\] of the KD shrinkage on the log scale\. Distilling the target’s 64 most probable tokens in place of eight changes the MAT of KD trained on fully regenerated responses by−\-0\.3% \[−\-0\.9, 0\.4\] and gives a KD ratio of ratios of−\-9\.9%, against−\-9\.4% with eight tokens\.

Every chain\-verifier MAT in Table[3](https://arxiv.org/html/2610.29013#S3)b is lower than the tree\-verifier MAT of the same cell in Table[3](https://arxiv.org/html/2610.29013#S3)a because each round checks one candidate at each of seven slots\. Table[C](https://arxiv.org/html/2610.29013#A3)lists the three data orders, in which the gate changes MAT after regeneration by−\-0\.4%, 0\.2% and 0\.8%, each with an interval that includes zero\. Across the three orders, the ratios of ratios have standard deviations of 0\.5 points for KD and 0\.7 points for the gate\.

Table 9:Vision MAT and ratios of ratios with 95% intervals in three data orders, from one training run in each order\. The main order is the factorial’s, represented by its first run\.On responses sampled at temperature 1, we also train CE and KD, twice each, on three target samples of every prompt, with the recipe otherwise unchanged\. The first sample is that sampled version, and the other two are drawn in the same way: each sample draws every response with one random seed shared across prompts\. Training runs for the 3,273 updates that three epochs take on one sample\. This is one pass over the three samples, so every prompt is seen about three times, each time with a different response\. For this comparison, KD on one sample is retrained alongside the new cells\.

CE and KD reach 3\.977 and 4\.022 on three samples, against 3\.756 and 3\.926 on one\. Three samples raise CE by 5\.9% \[5\.1, 6\.6\] and KD by 2\.4% \[1\.8, 3\.1\], so KD’s gain over CE falls from 4\.5% to 1\.1% \[0\.6, 1\.6\], a ratio of ratios of−\-3\.2% \[−\-4\.0,−\-2\.5\]\. Three samples therefore remove 0\.75 \[0\.63, 0\.86\] of KD’s gain\. When the two training runs of each cell are resampled along with the prompts, the intervals of the two increases, the three\-sample gain and the ratio of ratios become \[4\.4, 7\.3\], \[1\.4, 3\.5\], \[0\.2, 2\.1\] and \[−\-4\.8,−\-1\.6\]\. The KD cell sampled atT=1T=1in Table[3](https://arxiv.org/html/2610.29013#S3)a differs from the retrained one by−\-0\.3% \[−\-0\.9, 0\.3\]; with it, the one\-sample gain is 4\.2% and the ratio of ratios is−\-3\.0% \[−\-3\.7,−\-2\.2\]\.

## Appendix DAdditional Analyses

Table[10](https://arxiv.org/html/2610.29013#A4.T10)lists the mixtures of Figure[5](https://arxiv.org/html/2610.29013#S5.F5)\. Every mixture keeps the 26,101 external examples and their order, and the fully regenerated point is the factorial cell, which keeps its own 26,193 examples and order\. Both gains fall in all three domains at the first step, and at each later step at least one of them rises in some domain\.

Table 10:Vision MAT on ALLaVA when a share of the external responses is replaced by the target’s greedy regenerations, averaged over two training runs, with agreement and the gains of KD over CE and of the gate over KD\.Table[11](https://arxiv.org/html/2610.29013#A4.T11)lists KD’s increments over CE in the top\-1 agreement and the top\-8 recall of the target’s greedy token at each block slot, scored on the target’s greedy text for the 297 MAT prompts\. Their mean difference over slots 1 to 7, top\-8 minus top\-1, is 1\.6 points \[1\.4, 1\.8\] on external responses and 0\.3 points \[0\.2, 0\.4\] after full regeneration\. The chain’s share of KD’s tree gain, the ratio oflog⁡\(1\+Ic\)\\log\(1\+I\_\{c\}\)tolog⁡\(1\+It\)\\log\(1\+I\_\{t\}\)for KD’s increments over CE scored as a seven\-slot chain,IcI\_\{c\}, and as the tree,ItI\_\{t\}, is 0\.78 \[0\.73, 0\.83\] on external responses and 0\.28 \[0\.14, 0\.43\] after full regeneration\. A chain of the drafter’s first choices thus keeps most of KD’s tree gain on external responses, where KD mostly corrects the first choice, but less than half after full regeneration, where most of the remaining gain lies in how the other tree candidates are ranked\.

Table 11:Increments of KD over CE trained on ALLaVA, in points, in the top\-1 agreement and the top\-8 recall of the target’s greedy token at each block slot, scored on the target’s greedy text, averaged over two runs and 297 prompts\.## Appendix ECorpus Diagnostics

Table[12](https://arxiv.org/html/2610.29013#A5.T12)reports how the target reads each version of the ALLaVA corpus\. We run the target again over every retained example in bf16 with the training preprocessing, and each statistic follows the definition of the training gate, with each eligible anchor weighted by its probability of being sampled in training,min⁡\(128,n\)/n\\min\(128,n\)/nfornneligible anchors\. Positions kept is the share of predicted slots that the gate keeps, and weight kept is the share of the position weight it keeps withγ=7\\gamma=7and withγ=2\\gamma=2\. The soft weight is the mean survival weight of PARD\-2, and the corpus\-token probability is the mean probability the target assigns to the corpus token\. Intervals come from 10,000 resamples of examples\.

Table 12:Corpus diagnostics over retained examples, in percent\. Trunc\. and Window denote truncation and the loss window\. Agreement is the share of slots matching the target’s greedy token\. Every 95% interval spans less than 0\.3 points\.After full regeneration the remaining disagreement concentrates where the target is uncertain\. Slots whose top\-1 probability is below 0\.5 hold 14\.5% of all slots but 74\.1% of the disagreements\. Slots where it is at least 0\.9 hold 49\.7% of all slots, and the corpus token differs from the greedy token at 0\.06% of them\. Truncation inserts a closing token that the target never predicts at the cut, and closing tokens account for 14\.1% of the disagreements in the truncated version\. Regenerated responses that reach the generation limit agree at 96\.2% against 95\.2% for the rest\.

The sampled version regenerates every response once at temperature 1 with top\-p 1, no top\-k limit, at most 768 new tokens and one random seed shared across prompts, and it retains 26,195 examples\. The second sampled version differs only in its temperature of 0\.7 and also retains 26,195 examples\.

Among the target\-written versions, agreement does not track the KD gain in Table[3](https://arxiv.org/html/2610.29013#S3)a: the matched\-length versions agree within one point of full regeneration yet keep larger KD gains, and the sampled versions agree at only 76\.4% and 84\.1% yet keep KD gains close to the greedy one, although their gate gains lie between the external and greedy values\. Nor does agreement rank the two corpora’s external versions by KD gain: ShareGPT4V’s external captions agree with the target more often than ALLaVA’s external responses, yet KD gains more over CE on them\.

## Appendix FExternal Objective Settings

Table[13](https://arxiv.org/html/2610.29013#A6.T13)lists the objective settings trained on external responses, including one sampled evaluation atT=1T=1, and Figure[6](https://arxiv.org/html/2610.29013#A6.F6)plots the position weight sweep for KD after three epochs\. Entries marked†\\daggercome from a single training run\. The sampled evaluation lets the target sample at temperature 1 with random numbers fixed for each prompt and output position, shared by all drafters\. Under sampled decoding the four settings keep their greedy\-decoding order\. KD with the soft survival weighting of PARD\-2 in place of the gate, trained for three epochs withγ=2\\gamma=2, reaches 3\.542, the highest MAT among the external ALLaVA settings\. Its gain over KD withγ=2\\gamma=2is 5\.3% in the main data order, and 4\.6% and 4\.4% when both are retrained once in each of the two further data orders\.

Table 13:Vision MAT of objective settings trained on external responses\.Figure 6:Vision MAT of KD after three epochs on external ALLaVA responses as position weightγ\\gammavaries\.†\\daggermarks a single training run, and unmarked points average two runs\.## Appendix GText Results

Table[G](https://arxiv.org/html/2610.29013#A7)lists task\-level MAT of CE and KD for both drafters, Table[15](https://arxiv.org/html/2610.29013#A7.T15)that of the gate and the label control, and Table[4\.1](https://arxiv.org/html/2610.29013#S4.SS1)gives overall MAT with intervals\. Every fine\-tuned DFlash drafter stays below its released initialization, trained on target\-generated examples including math and code\([Chen et al\., 2026](https://arxiv.org/html/2610.29013#bib.bib8)\), and the losses concentrate on these tasks, which the chat corpus barely covers\. On the two chat tasks, MT\-Bench and Alpaca, the KD ratio of ratios is−\-8\.2% under greedy decoding and−\-8\.4% under sampling, a larger shrinkage than over all seven tasks, because under greedy decoding KD adds 10\.1% over CE on external responses there and 1\.0% on regenerated ones\. These are the tasks where fine\-tuning comes closest to the released drafter, and on Alpaca the regenerated CE and KD drafters are level with it while the ratio of ratios is−\-8\.9%\.

Table 14:Text\-only MAT of CE and KD for each drafter, task and temperature\. Init\. is the released drafter, trained entries average two runs and Mean rows give geometric means over tasks\. EAGLE\-3 MAT comes from its own tree decoder\.For the DFlash drafter, the gate’s ratio of ratios is negative in all seven tasks and in both training runs at either temperature\. When the two runs of each cell are resampled along with the prompts, its interval is \[−\-4\.9,−\-3\.6\] under greedy decoding and \[−\-6\.0,−\-2\.3\] under sampling, and the interval of the gate’s gain on regenerated responses includes zero at both temperatures\. The label control of Section[5](https://arxiv.org/html/2610.29013#S5)adds 7\.4% \[6\.9, 7\.8\] over CE on external and 0\.3% \[0\.1, 0\.6\] on regenerated responses under greedy decoding, a ratio of ratios of−\-6\.5% \[−\-7\.0,−\-6\.1\], or \[−\-7\.3,−\-5\.8\] when the two training runs of each cell are resampled along with the prompts\. Under sampling it is−\-6\.3% \[−\-7\.6,−\-5\.0\], and at either temperature it is negative in all seven tasks and in both training runs\. Under greedy decoding KD adds no measurable gain over greedy labels on external responses, 0\.0% \[−\-0\.3, 0\.3\], and 0\.7% \[0\.4, 1\.0\] on regenerated ones\.

Table 15:Text\-only MAT of the DFlash drafter with the survival gate and with greedy labels for each task and temperature, averaged over two training runs\. Greedy labels trains CE toward the target’s greedy token\. Mean rows give geometric means over tasks\.
For the EAGLE\-3 drafter, KD adds 13\.7% \[13\.2, 14\.3\] over CE on external and 2\.2% \[1\.8, 2\.5\] on regenerated responses, and the ratio of ratios is−\-10\.2% \[−\-10\.7,−\-9\.6\], or \[−\-11\.0,−\-9\.3\] when the two runs of each cell are resampled along with the prompts\. In the two data orders it is−\-10\.7% and−\-9\.7%, and across tasks it ranges from−\-12\.1% on GSM8K to−\-6\.9% on HumanEval, with every task’s interval below zero, also with the runs resampled\. Every fine\-tuned EAGLE\-3 checkpoint exceeds the released drafter in every task, and its MAT over all tasks is 14\.5% to 44\.7% above the released 3\.383\.