AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction

# AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction
Source: [https://arxiv.org/html/2608.00434](https://arxiv.org/html/2608.00434)
Ziqiang Cui1,Han Shi2,Bowei He3,4,Yu Pan2,Peiyang Liu5,Shengyin Sun1, Yankai Chen3,4,Haoli Bai2,Yichun Yin2,Xue Liu3,4,Chen Ma122footnotemark:2

1City University of Hong Kong,2Huawei Technologies 3Mohamed bin Zayed University of Artificial Intelligence 4McGill University,5Peking University

###### Abstract

Multi\-Token Prediction \(MTP\) has emerged as an effective paradigm that augments a shared Large Language Model backbone with auxiliary heads, training the model to predict several future tokens in parallel to enrich its supervision signal and accelerate inference\. However, existing training frameworks adopt a rigid, fixed\-length prediction horizon, disregarding the highly non\-uniform information density of natural language and code\. Forcing the auxiliary heads to predict across high\-entropy semantic boundaries injects noisy, conflicting training signals; because these heads share the backbone’s latent representations, the resulting gradients backpropagate and interfere with the model’s core capabilities\. We proposeAdaMTP, an adaptive training paradigm that dynamically aligns the prediction horizon with the intrinsic predictability of the sequence\. At its core, an entropy\-based segmentation algorithm leverages the base model to detect sudden surges in uncertainty as semantic boundaries, partitioning sequences into variable\-length groups\. Each token is assigned an adaptive prediction depth, and a dynamically masked MTP objective suppresses the loss for predictions that cross these boundaries, attenuating the noisy gradients that degrade the backbone\. Across mathematical reasoning, code generation, and general benchmarks on three backbones \(Llama\-3\.1\-8B, Qwen\-2\.5\-7B, Gemma\-3\-12B\), AdaMTP consistently outperforms standard MTP in both task performance and inference speedup\.

AdaMTP: An Adaptive Training Paradigm for Multi\-Token Prediction

Ziqiang Cui1††thanks:Email:[ziqiang\.cui@my\.cityu\.edu\.hk](https://arxiv.org/html/2608.00434v1/mailto:[email protected]), Han Shi2††thanks:Corresponding authors\., Bowei He3,4, Yu Pan2, Peiyang Liu5, Shengyin Sun1,Yankai Chen3,4,Haoli Bai2,Yichun Yin2,Xue Liu3,4,Chen Ma122footnotemark:21City University of Hong Kong,2Huawei Technologies3Mohamed bin Zayed University of Artificial Intelligence4McGill University,5Peking University

## 1Introduction

Large Language Models \(LLMs\) have achieved remarkable success across a wide range of natural language processing, mathematical reasoning, and code generation tasks\. Currently, these models are predominantly trained using the standard Next\-Token Prediction \(NTP\) objective, where models are trained to predict the immediate next token given the preceding context\. Despite their robust generative capabilities, the strictly autoregressive nature of NTP imposes a fundamental bottleneck during inference\. Generating tokens one by one leads to high latency and computational inefficiency, especially for long\-form generation and interactive applications\.

To mitigate this inference bottleneck, Multi\-Token Prediction \(MTP\) has recently emerged as a highly effective paradigm\. By augmenting a shared LLM backbone with multiple auxiliary output heads, MTP generalizes the standard training objective to predict several future tokens simultaneously\. During training, this multi\-token objective provides richer supervision signals, equipping the model with enhanced long\-term planning capabilities\. During inference, MTP circumvents the strict autoregressive bottleneck by generating multiple tokens per step, thereby accelerating decoding speed\.

![Refer to caption](https://arxiv.org/html/2608.00434v1/x1.png)Figure 1:Token\-level predictive entropy of the LLM over an example sequence\. Within a cohesive semantic group, the entropy follows anapproximatelymonotonically decreasing trend as the local context becomes increasingly constrained; at a semantic boundary, the entropy exhibits a sudden surge\. AdaMTP exploits these entropy surges to partition the sequence into variable\-length groups\.However, existing MTP frameworks enforce a rigid, fixed\-length prediction horizon for every token, overlooking the varying predictability of the underlying context\. Natural language and code exhibit highly non\-uniform information density\. Within contiguous chunks such as common phrases or local code blocks, the sequence is highly predictable; notably, we observe that predictive uncertainty \(entropy\) approximately follows a monotonically decreasing trend in these regions \(as illustrated in Figure[1](https://arxiv.org/html/2608.00434#S1.F1)\)\. In contrast, transitions between different conceptual ideas or syntactic structures are characterized by sudden spikes in uncertainty\. Forcing the auxiliary heads to predict a fixed number of tokens across these high\-entropy boundaries inherently injects noisy, conflicting training signals\. Because the auxiliary heads and the main language modeling head share the same latent representations, these noisy gradients backpropagate and cause severe representation interference, ultimately degrading the base model’s core capabilities\.

Motivated by these limitations, we proposeAdaMTP\(Adaptive Multi\-Token Prediction\), a novel framework that dynamically aligns the multi\-token prediction horizon with the intrinsic predictability of the sequence\. At the core of AdaMTP is an entropy\-based data segmentation algorithm\. By using the base LLM to estimate token\-level predictive entropy, we identify sudden surges in uncertainty—which disrupt the aforementioned approximately monotonic decrease—as semantic boundaries, partitioning the sequence into cohesive, variable\-length groups\. Based on these partitions, we assign each token anadaptive prediction depth, defined as the distance to the end of its current group \(or the subsequent group for boundary tokens\)\. To incorporate this into training, we introduce a dynamically masked MTP loss: for any given token, the loss for future predictions that exceed its adaptive depth is masked out\. This crucial design prevents the auxiliary heads from forcibly predicting across unpredictable boundaries, thereby attenuating the noisy gradients that degrade the model’s core capabilities\. At inference time, AdaMTP provides two decoding modes\. By default, it adopts the same fixed\-horizon scheme as standard MTP, yet achieves faster inference; alternatively, an adaptive\-horizon mode prunes low\-confidence branches via the real\-time entropy signal to cut verification cost—an efficiency edge that becomes pronounced under large\-batch serving\.

We comprehensively evaluate AdaMTP on diverse benchmarks spanning mathematical reasoning, code generation, and general language proficiency\. Using three representative base models—Llama\-3\.1 \(8B\), Qwen\-2\.5 \(7B\), and Gemma\-3 \(12B\)—our extensive experiments demonstrate that AdaMTP improves on both task performance and inference efficiency\. In terms of quality, it mitigates the representation interference inherent in standard MTP and consistently surpasses both the NTP and fixed\-horizon MTP baselines in average score\. In terms of efficiency, it retains and further strengthens the self\-speculative acceleration of MTP, delivering substantial speedups over NTP while also decoding faster than standard MTP\. Ultimately, these results indicate that adapting the MTP objective to local predictability is a more reliable way to retrofit pretrained LLMs with efficient multi\-token generation\.

In summary, our main contributions are as follows:

- •We identify and empirically corroborate a key limitation of existing MTP training: uniformly predicting a fixed number of future tokens can adversely affect the pretrained backbone, which we attribute to the noisy supervision forced across high\-entropy linguistic boundaries\.
- •We propose AdaMTP, an adaptive training paradigm that uses entropy\-based data segmentation to assign each token an adaptive prediction depth, together with a dynamically masked MTP objective that suppresses training signals beyond this depth to curb the resulting noisy gradients\.
- •Extensive experiments across three backbones and eight benchmarks show that AdaMTP consistently outperforms both NTP and standard MTP in task performance while decoding faster—delivering substantial speedups over NTP and surpassing the inference speed of standard MTP\.

## 2Preliminaries

![Refer to caption](https://arxiv.org/html/2608.00434v1/x2.png)Figure 2:Overview of AdaMTP training paradigm\.\(Left\) Entropy\-Based Data Segmentation:using the base model’s next\-token entropy, the sequence is split into variable\-length groups at points of sudden entropy surge, and each token is assigned an adaptive prediction depth\.\(Right\) Dynamically Masked MTP Training:the shared LLM backbone feedsnnparallel heads that predict future tokens; losses for predictions falling within each token’s adaptive depth are retained as valid signal, while those crossing group boundaries are masked out and ignored\.### 2\.1Next\-Token Prediction \(NTP\)

The standard training paradigm for LLMs relies on NTP task\. Given a sequence of tokensx1:T=\(x1,x2,…,xT\)x\_\{1:T\}=\(x\_\{1\},x\_\{2\},\\dots,x\_\{T\}\), the primary objective is to minimize the cross\-entropy loss:

ℒNTP=−∑t=1T−1log⁡Pθ​\(xt\+1∣x1:t\)\\mathcal\{L\}\_\{\\text\{NTP\}\}=\-\\sum\_\{t=1\}^\{T\-1\}\\log P\_\{\\theta\}\(x\_\{t\+1\}\\mid x\_\{1:t\}\)\(1\)wherex1:tx\_\{1:t\}denotes the context up to steptt, andPθ​\(xt\+1∣x1:t\)P\_\{\\theta\}\(x\_\{t\+1\}\\mid x\_\{1:t\}\)represents the model’s predicted probability distribution over the vocabulary for the next token\. During inference, NTP operates in a strictly autoregressive manner\. The model generates a single token at a time by sampling from the predicted distribution\.

### 2\.2Multi\-Token Prediction \(MTP\)

MTP extends the standard NTP paradigm by training the model to predictnnfuture tokensxt\+1:t\+nx\_\{t\+1:t\+n\}at each position, using a shared backbone together with one main head andn−1n\-1auxiliary heads\.

ℒMTP=−∑t=1T−nlog⁡Pθ​\(xt\+1:t\+n∣x1:t\)\.\\mathcal\{L\}\_\{\\text\{MTP\}\}=\-\\sum\_\{t=1\}^\{T\-n\}\\log P\_\{\\theta\}\(x\_\{t\+1:t\+n\}\\mid x\_\{1:t\}\)\.\(2\)
In recent studies, the overall architectureθ\\thetais decoupled into a shared backbone with parametersθ′\\theta^\{\\prime\}, a main language modeling headθ0\\theta\_\{0\}, andn−1n\-1auxiliary output heads with parameters\{θj\}j=1n−1\\\{\\theta\_\{j\}\\\}\_\{j=1\}^\{n\-1\}\. The shared backbone maps the contextx1:tx\_\{1:t\}to a hidden representationz1:t=fθ′​\(x1:t\)z\_\{1:t\}=f\_\{\\theta^\{\\prime\}\}\(x\_\{1:t\}\)\. The main head reusesz1:tz\_\{1:t\}to predict the next tokenxt\+1x\_\{t\+1\}, while thejj\-th auxiliary head predicts the token at offsetj\+1j\{\+\}1, i\.e\.xt\+j\+1x\_\{t\+j\+1\}\. Allnnheads operate in parallel on the same representation, so thatnnfuture tokens are produced within a single forward pass\. Assuming conditional independence of the future tokens givenz1:tz\_\{1:t\}, the joint distribution of the nextnntokens is factorized as

Pθ​\(xt\+1:t\+n∣x1:t\)=Pθ0​\(xt\+1∣z1:t\)​∏j=1n−1Pθj​\(xt\+j\+1∣z1:t\),\\displaystyle P\_\{\\theta\}\(x\_\{t\+1:t\+n\}\\mid x\_\{1:t\}\)=P\_\{\\theta\_\{0\}\}\(x\_\{t\+1\}\\mid z\_\{1:t\}\)\\prod\_\{j=1\}^\{n\-1\}P\_\{\\theta\_\{j\}\}\(x\_\{t\+j\+1\}\\mid z\_\{1:t\}\),

\(3\)wherez1:tz\_\{1:t\}is a deterministic function of the context andPθj\(⋅∣z1:t\)P\_\{\\theta\_\{j\}\}\(\\cdot\\mid z\_\{1:t\}\)denotes the distribution predicted by thejj\-th auxiliary head\. In practice, rather than pretraining from scratch, these auxiliary heads are commonly attached to an already pretrained LLM and trained via fine\-tuning, retrofitting multi\-token prediction onto an existing NTP model at a fraction of the pretraining cost\.

During inference, the MTP model leverages its multiple heads to propose subsequentnncandidate tokens simultaneously\. To guarantee generation quality, this parallel drafting mechanism is paired with a verification step, effectively forming a self\-speculative decoding framework\. The model evaluates the proposed candidates in a single forward pass, accepts the valid tokens, and advances the context window accordingly\. This approach accelerates inference while strictly preserving the original output distribution of the LLM\.

## 3Methodology

In this section, we detailAdaMTP\(Adaptive Multi\-Token Prediction\), our framework that dynamically adjusts the multi\-token prediction horizon based on the intrinsic predictability of the sequence\. AdaMTP consists of an entropy\-based data segmentation algorithm, a two\-stage adaptive training pipeline, and a dual\-mode accelerated decoding mechanism for inference\.

### 3\.1Entropy\-Based Data Segmentation

During training, standard MTP forces the model to predict a fixed number of future tokens at every time step, regardless of the difficulty or predictability of the context\. This rigid approach can be highly detrimental: forcing the model to predict across high\-entropy boundaries—where future tokens are conceptually unrelated to the current context—injects noisy, conflicting training signals\. Since the auxiliary MTP heads and the main language modeling head share the same underlying hidden representations, these noisy gradients can severely interfere with the main head’s optimization, ultimately degrading the base model’s core capabilities\.

To alleviate this optimization burden and prevent representation interference, we propose an entropy\-based segmentation algorithm that groups tokens according to their predictive uncertainty\. Given a sequence of tokensX=\(x1,x2,…,xN\)X=\(x\_\{1\},x\_\{2\},\\dots,x\_\{N\}\), we use the exact same pre\-trained base model designated for supervised fine\-tuning \(SFT\) as our reference model to compute the entropy of the next\-token distribution at each positiontt:

Et=ℋ\(P\(⋅∣x<t\)\)=−∑v∈𝒱P\(v∣x<t\)logP\(v∣x<t\),\\resizebox\{385\.92152pt\}\{\}\{$\\displaystyle E\_\{t\}=\\mathcal\{H\}\(P\(\\cdot\\mid x\_\{<t\}\)\)=\-\\sum\_\{v\\in\\mathcal\{V\}\}P\(v\\mid x\_\{<t\}\)\\log P\(v\\mid x\_\{<t\}\)$\}\\,,\(4\)where𝒱\\mathcal\{V\}is the vocabulary\. Using the SFT base model as the reference ensures that the computed uncertainty perfectly aligns with the target model’s intrinsic probability distribution\. We then calculate the delta entropy between consecutive tokens asΔ​Et=Et\+1−Et\\Delta E\_\{t\}=E\_\{t\+1\}\-E\_\{t\}\. Conceptually, as the model generates tokens within a cohesive semantic chunk, the local context becomes increasingly constrained, causing the predictive entropy to exhibit a roughly monotonically decreasing trend \(i\.e\., uncertainty tends to diminish as the phrase nears completion\)\. Conversely, a large positiveΔ​Et\\Delta E\_\{t\}signifies a sudden spike in uncertainty, typically corresponding to crossing a linguistic boundary or transitioning to a new semantic unit\. This entropy dynamic, illustrated in Figure[1](https://arxiv.org/html/2608.00434#S1.F1), implies that detecting these surges alone suffices to cleanly recover coherent semantic groups\.

Therefore, we segment the sequence into contiguous groups by splitting at indices where the surge in entropy exceeds a predefined threshold \(Δ​Et\>τ\\Delta E\_\{t\}\>\\tau\)\. The thresholdτ\\tauis calibrated via a dataset\-level search so that the resulting average group size matches the total number of headsnn\(i\.e\., the maximum prediction depth\), thereby aligning the segmentation granularity with the model’s multi\-token prediction capacity\. LetGk=\[sk,ek\)G\_\{k\}=\[s\_\{k\},e\_\{k\}\)denote thekk\-th token group spanning from indexsks\_\{k\}toek−1e\_\{k\}\-1\. For a tokenxtx\_\{t\}located withinGkG\_\{k\}, we define its adaptive prediction depthdtd\_\{t\}as follows:

dt=\{ek−t−1,if​t<ek−1\|Gk\+1\|,if​t=ek−1d\_\{t\}=\\begin\{cases\}e\_\{k\}\-t\-1,&\\text\{if \}t<e\_\{k\}\-1\\\\ \|G\_\{k\+1\}\|,&\\text\{if \}t=e\_\{k\}\-1\\end\{cases\}\(5\)where\|Gk\+1\|=ek\+1−sk\+1\|G\_\{k\+1\}\|=e\_\{k\+1\}\-s\_\{k\+1\}denotes the length of the subsequent group\. Intuitively, tokens strictly inside a predictable group only predict the remainder of their current group, while the boundary token \(which precedes a high\-entropy transition\) is tasked with predicting the entirety of the next group\. This design ensures that the training prediction depth flexibly adapts to the natural chunking of language\. Throughout, we use*horizon*to denote the number of future tokens the model is asked to predict at a step, and*\(adaptive\) depth*dtd\_\{t\}to denote the per\-token training target assigned by our method\. Since the model is equipped with onlyn−1n\-1auxiliary heads, the effective supervised depth is capped at the horizonnn: for any token, wheneverdt\>nd\_\{t\}\>n, supervision is applied only up to offsetnn, i\.e\.,min⁡\(dt,n\)\\min\(d\_\{t\},n\)\.

### 3\.2AdaMTP Training

At the heart of AdaMTP is its adaptive training objective: instead of forcing every token to predict a fixed number of future tokens, we supervise each token only within its adaptive prediction depthdtd\_\{t\}, masking out any prediction that crosses a semantic boundary\. To instill this objective into a pretrained LLM while minimizing disruption to its foundational language modeling ability, we adopt a two\-stage pipeline—an auxiliary\-head warm\-up followed by joint fine\-tuning—following the standard training practice for retrofitting MTP heads onto pretrained models\(Caiet al\.,[2024](https://arxiv.org/html/2608.00434#bib.bib11); Liuet al\.,[2025](https://arxiv.org/html/2608.00434#bib.bib1)\)\.

#### Auxiliary\-Head Warm\-Up\.

Our training procedure builds upon a pretrained base language model\. To equip the model with multi\-token prediction capabilities, we augment the base LLM withn−1n\-1auxiliary output heads, each responsible for predicting a token at a specific future offset\. Architecturally, each auxiliary head is implemented as a multi\-layer perceptron \(MLP\) followed by a linear projection layer that maps the hidden states directly into the vocabulary space\. The primary objective of this initial stage is to align these newly introduced heads with the base LLM\. To achieve this, we freeze the LLM backbone and its original language modeling head, restricting parameter optimization entirely to the auxiliary heads\. During this stage, we utilize self\-distilled data for training, which is generated by feeding prompts into the base LLM and collecting its outputs\. The training objective is formulated as:

ℒwarm\-up=∑j=1n−1∑t=1T−j−1ℒCE​\(Headj​\(𝐡t\),xt\+j\+1\),\\mathcal\{L\}\_\{\\text\{warm\-up\}\}=\\sum\_\{j=1\}^\{n\-1\}\\sum\_\{t=1\}^\{T\-j\-1\}\\mathcal\{L\}\_\{\\text\{CE\}\}\\left\(\\text\{Head\}\_\{j\}\(\\mathbf\{h\}\_\{t\}\),x\_\{t\+j\+1\}\\right\),\(6\)where𝐡t\\mathbf\{h\}\_\{t\}is the frozen hidden state of the backbone at time steptt, andℒCE\\mathcal\{L\}\_\{\\text\{CE\}\}denotes the standard cross\-entropy loss\.

#### Adaptive Joint Training\.

In the second stage, we conduct joint training by applying Low\-Rank Adaptation \(LoRA\) across all model components, encompassing both the LLM backbone and the output heads\. During this joint training process, we utilize the pre\-computed adaptive depthsdtd\_\{t\}to selectively mask the MTP loss\.

Let𝐡t\\mathbf\{h\}\_\{t\}denote the hidden state at time steptt\. The base model predicts the next tokenxt\+1x\_\{t\+1\}, while thejj\-th MTP head \(1≤j<n1\\leq j<n\) predicts the future tokenxt\+j\+1x\_\{t\+j\+1\}\. Unlike traditional MTP training, which uniformly penalizes predictions up to a fixed depth regardless of context difficulty, our approach dynamically masks the loss for tokens beyond the adaptive depthdtd\_\{t\}\. Consequently, the adaptive MTP loss is defined as:

ℒMTP=∑j=1n−1∑t=1T−j−1𝕀​\(j\+1≤dt\)⋅ℒCE​\(Headj​\(𝐡t\),xt\+j\+1\),\\begin\{split\}\\mathcal\{L\}\_\{\\text\{MTP\}\}=\\sum\_\{j=1\}^\{n\-1\}\\sum\_\{t=1\}^\{T\-j\-1\}&\\mathbb\{I\}\(j\+1\\leq d\_\{t\}\)\\\\ &\\cdot\\mathcal\{L\}\_\{\\text\{CE\}\}\\left\(\\text\{Head\}\_\{j\}\(\\mathbf\{h\}\_\{t\}\),x\_\{t\+j\+1\}\\right\),\\end\{split\}\(7\)where𝕀​\(⋅\)\\mathbb\{I\}\(\\cdot\)is the indicator function andℒCE\\mathcal\{L\}\_\{\\text\{CE\}\}is the cross\-entropy loss\. To enable joint optimization, the total loss is formulated as a weighted sum of the standard language modeling lossℒLM\\mathcal\{L\}\_\{\\text\{LM\}\}\(for the base model\) and the adaptive MTP lossℒMTP\\mathcal\{L\}\_\{\\text\{MTP\}\}:

ℒtotal=ℒLM\+λ​ℒMTP,\\mathcal\{L\}\_\{\\text\{total\}\}=\\mathcal\{L\}\_\{\\text\{LM\}\}\+\\lambda\\mathcal\{L\}\_\{\\text\{MTP\}\},\(8\)whereλ\\lambdacontrols the contribution of the auxiliary heads\. This adaptive joint training prevents the model from attempting to predict excessively far into unpredictable futures, thereby concentrating its capacity on highly certain and structured predictions\.

### 3\.3Inference Procedure

During inference, the trained MTP heads are used for self\-speculative decoding: at each step they draft several candidate future tokens, and the base model verifies these candidates in a single forward pass, accepting only the prefix consistent with its own predictions\. Because every accepted token is validated by the base model, this draft\-then\-verify procedure is lossless—it yields exactly the same output as standard autoregressive decoding while reducing the number of sequential steps and thus accelerating generation\. Within this framework, we provide two generation settings:

#### Fixed\-Horizon Generation\.

In this default setting, the MTP heads always draft the full horizon ofnncandidate tokens at every step\. Following Medusa\(Caiet al\.,[2024](https://arxiv.org/html/2608.00434#bib.bib11)\), these candidates are organized into a token tree and verified by the base model in a single forward pass\. Since this verification pass is dominated by the base model’s forward computation and, at small batch sizes, is only weakly sensitive to the number of candidates it checks, drafting the maximum number of candidates maximizes the expected number of tokens accepted per step\.

#### Adaptive\-Horizon Generation\.

Alternatively, we offer an adaptive strategy that dynamically determines how many candidates to generate from the real\-time entropy delta: the MTP heads stop drafting further into the future once the entropy increase between consecutive predicted tokens exceeds a threshold\. This prunes the candidate tree and reduces the number of tokens the base model must verify\. Since it only forgoes low\-confidence tail candidates that would largely have been rejected anyway, under an appropriate threshold it incurs no loss in per\-step acceptance length relative to fixed\-horizon generation\. The resulting saving is marginal for single\-sample decoding—where verifying a few extra candidates adds negligible latency—but becomes substantial under large\-batch, compute\-bound serving, whose latency scales with the total number of candidates verified\.

Math500GSM8KMBPPMBPP\+HumanEvalHumanEval\+MMLUIFEvalAvg\.Base3\.209\.0262\.4352\.1237\.8030\.4963\.4516\.9134\.43NTP5\.6011\.3061\.3450\.9542\.2635\.3763\.6420\.0836\.32MTP5\.0011\.3060\.3850\.0041\.4635\.9863\.3319\.7435\.90Llama3\.1\-8BAdaMTP7\.2013\.1261\.6450\.2642\.6835\.9863\.6720\.2636\.85Base62\.8054\.8575\.2064\.0678\.0571\.2071\.7641\.6564\.95NTP49\.2052\.7576\.5064\.2977\.4469\.2671\.5842\.3562\.92MTP48\.6047\.6974\.3463\.7676\.2270\.1271\.7740\.2961\.60Qwen2\.5\-7BAdaMTP49\.4050\.7276\.1964\.8178\.0571\.9571\.7942\.5763\.19Base0\.009\.7672\.2258\.9945\.1235\.3724\.3929\.5234\.42NTP9\.009\.3370\.6358\.9961\.0256\.1070\.7329\.9445\.72MTP12\.6013\.8665\.3454\.5059\.1551\.2271\.8231\.4144\.99Gemma3\-12BAdaMTP16\.0014\.8166\.0854\.5060\.3753\.0572\.0432\.1346\.12

Table 1:Performance comparison among different prediction paradigms \(NTP, MTP, and our AdaMTP\) across diverse tasks and benchmarks\. For each backbone, the best average result \(Avg\.\) among NTP, MTP, and AdaMTP is highlighted in bold\.

## 4Experiments

### 4\.1Experimental Settings

- Datasets\.In alignment with the previous workLiuet al\.\([2025](https://arxiv.org/html/2608.00434#bib.bib1)\), our training corpus is assembled from the MathHendryckset al\.\([2021](https://arxiv.org/html/2608.00434#bib.bib47)\), Evol\-Instruct\-CodeLuoet al\.\([2023](https://arxiv.org/html/2608.00434#bib.bib48)\); Chaudhary \([2023](https://arxiv.org/html/2608.00434#bib.bib49)\), and Alpaca\-GPT4Penget al\.\([2023](https://arxiv.org/html/2608.00434#bib.bib25)\)datasets\. The training procedure is divided into two phases\. Initially, the entire dataset is leveraged to perform self\-distillation\. Subsequently, the second phase utilizes a randomly sampled subset of 10,000 instances, distributed across mathematical, programming, and general domains in a 4:4:2 ratio\. To rigorously assess the proposed methodologies, we employ a diverse suite of benchmarks: Math500Lightmanet al\.\([2023](https://arxiv.org/html/2608.00434#bib.bib50)\)and GSM8KCobbeet al\.\([2021](https://arxiv.org/html/2608.00434#bib.bib51)\)\(both evaluated in a 4\-shot setting\) for mathematical reasoning; MBPP, MBPP\+Austinet al\.\([2021](https://arxiv.org/html/2608.00434#bib.bib52)\); Liuet al\.\([2023](https://arxiv.org/html/2608.00434#bib.bib53)\), HumanEval, and HumanEval\+Chenet al\.\([2021](https://arxiv.org/html/2608.00434#bib.bib54)\); Liuet al\.\([2023](https://arxiv.org/html/2608.00434#bib.bib53)\)for code generation capabilities; alongside MMLUHendryckset al\.\([2020](https://arxiv.org/html/2608.00434#bib.bib55)\)and IFEvalZhouet al\.\([2023a](https://arxiv.org/html/2608.00434#bib.bib56)\)to measure general proficiency\.
- Evaluation Metrics\.To evaluate task performance, we report accuracy for mathematical and general\-domain benchmarks, while employing the pass@1 metric for code generation tasks\. Furthermore, we assess efficiency via a speedup ratio relative to standard autoregressive NTP decoding on the same model\.
- Base LLMs\.Our experimental framework employs three base large language models: Llama\-3\.1 \(8B\), Qwen\-2\.5 \(7B\), and Gemma\-3 \(12B\)\. This selection was made to ensure a comprehensive evaluation across a diverse spectrum of model architectures and parameter capacities\.
- Baselines\.To evaluate generation efficacy, we benchmark our approach against two established paradigms: standard NTP and MTP\. These baselines serve to measure the models’ capacity to produce accurate and contextually appropriate text\. For the efficiency analysis, standard autoregressive NTP decoding provides the1×1\\timesreference against which speedups are measured, while we compare AdaMTP with standard MTP, both of which employ self\-speculative decoding for lossless acceleration\.
- Implementation Details\.During the initial head warm\-up stage, we freeze the LLM backbone and exclusively train the prediction heads for 1 epoch with a learning rate of1×10−31\\times 10^\{\-3\}\. In the subsequent stage, we utilize LoRA \(rankr=32r=32,α=16\\alpha=16\) to fine\-tune the model for 3 epochs with a learning rate of1×10−51\\times 10^\{\-5\}\. We set the prediction depth ton=4n=4and the auxiliary MTP loss weight toλ=0\.1\\lambda=0\.1by default\. To ensure a fair comparison, identical training configurations are applied to the standard MTP baseline\. All experiments are conducted on four NVIDIA H800 GPUs with a total batch size of 256\.

### 4\.2Overall Performance

Table[1](https://arxiv.org/html/2608.00434#S3.T1)compares AdaMTP against NTP and standard MTP across three base models and eight benchmarks; we highlight the key findings below\.

#### AdaMTP consistently achieves the best overall performance\.

Across all three backbones, AdaMTP attains the highest average score—36\.85on Llama\-3\.1\-8B,63\.19on Qwen\-2\.5\-7B, and46\.12on Gemma\-3\-12B—surpassing both NTP and standard MTP\. This consistency across model families and scales shows that adaptively aligning the prediction horizon with sequence predictability is a robust, architecture\-agnostic strategy\. Intriguingly, on Qwen\-2\.5\-7B all fine\-tuning paradigms \(including NTP\) fall below the Base model, an observation consistent with prior findings\(Liuet al\.,[2025](https://arxiv.org/html/2608.00434#bib.bib1)\); we attribute this to the distribution shift of our fine\-tuning corpus rather than the MTP training paradigm, as plain NTP is affected identically\. Even in this regime AdaMTP still outperforms both NTP and standard MTP, indicating that the adaptive objective remains beneficial regardless of data quality\. As our focus is isolating the effect of the adaptive horizon relative to standard MTP, we leave higher\-quality data curation to future work\.

GSM8KHumanEvalIFEvalLlama3\.1NTP1\.00×\\times1\.00×\\times1\.00×\\timesMTP1\.65×\\times1\.86×\\times1\.48×\\timesAdaMTP2\.12×\\times2\.01×\\times1\.52×\\timesQwen2\.5NTP1\.00×\\times1\.00×\\times1\.00×\\timesMTP1\.84×\\times1\.55×\\times1\.40×\\timesAdaMTP1\.87×\\times1\.56×\\times1\.43×\\timesGemma3NTP1\.00×\\times1\.00×\\times1\.00×\\timesMTP2\.38×\\times1\.59×\\times1\.57×\\timesAdaMTP2\.75×\\times1\.62×\\times1\.61×\\timesTable 2:Inference speedup comparison of NTP, MTP, and AdaMTP across three backbones\.
#### Standard MTP suffers from representation interference\.

Standard MTP consistently underperforms NTP on average across all three models\. This corroborates our hypothesis: forcing the auxiliary heads to predict a rigid number of tokens across high\-entropy boundaries injects noisy, conflicting gradients into the shared backbone, corrupting its core capabilities\. AdaMTP instead masks these unpredictable predictions, suppressing the interfering signals and exceeding the performance of NTP\.

#### Gains span math reasoning, code, and general tasks\.

AdaMTP improves consistently across task categories, most notably on mathematical reasoning: since these tasks demand long\-range planning, concentrating multi\-token supervision within cohesive, predictable chunks yields cleaner training signals\. On code generation, it recovers the losses incurred by standard MTP and matches or exceeds NTP\. It also preserves general\-domain ability, achieving the best IFEval and MMLU scores among the trained paradigms\.

### 4\.3Inference Acceleration

Beyond task performance, a central promise of the MTP paradigm is its ability to accelerate decoding via self\-speculative generation\. Table[2](https://arxiv.org/html/2608.00434#S4.T2)reports the speedup ratio of NTP, standard MTP, and AdaMTP on GSM8K, HumanEval, and IFEval across all three backbones, using autoregressive NTP decoding as the reference \(1\.00×1\.00\\times\); here AdaMTP adopts its default Fixed\-Horizon decoding, the same scheme used by standard MTP\. Across every backbone and benchmark, AdaMTP delivers substantial acceleration over NTP, ranging from1\.43×1\.43\\timesup to2\.75×2\.75\\times, with the largest gains on the highly structured GSM8K task \(e\.g\.,2\.75×2\.75\\timeson Gemma\-3\-12B\)\. More importantly, AdaMTP consistently surpasses standard MTP—for instance, improving the GSM8K speedup from1\.65×1\.65\\timesto2\.12×2\.12\\timeson Llama\-3\.1\-8B—while simultaneously achieving superior task performance\. Because the two share an identical inference procedure, this acceleration gain over MTP arises purely from our adaptive training objective, which yields auxiliary heads whose drafts are more frequently accepted by the verifier, allowing AdaMTP to enhance the inference speedups that make MTP attractive in practice\.

![Refer to caption](https://arxiv.org/html/2608.00434v1/x3.png)Figure 3:Impact of the number of prediction headsnnon Llama\-3\.1\-8B \(n=1n=1corresponds to NTP\)\. AdaMTP stays consistently above standard MTP\.
### 4\.4Impact of the Number of Prediction Heads

We vary the total number of heads fromn=1n\{=\}1\(i\.e\., NTP\) ton=6n\{=\}6on GSM8K with Llama\-3\.1\-8B, holding all other settings fixed\. As shown in Figure[3](https://arxiv.org/html/2608.00434#S4.F3)\(a\), the accuracy of standard MTP declines almost monotonically asnngrows, falling from11\.6011\.60atn=2n\{=\}2to9\.689\.68atn=6n\{=\}6—well below the11\.3011\.30NTP reference\. This is a direct consequence of representation interference: each additional head forces the shared backbone to predict one token further ahead, crossing more high\-entropy semantic boundaries and injecting proportionally more noisy gradients\. AdaMTP instead masks precisely these cross\-boundary predictions, so it stays above NTP throughout and peaks atn=4n\{=\}4with13\.1213\.12; accordingly, its gap over MTP widens steadily withnn\(from\+0\.53\+0\.53atn=2n\{=\}2to\+1\.90\+1\.90atn=6n\{=\}6\)\. In terms of inference speedup \(Figure[3](https://arxiv.org/html/2608.00434#S4.F3)\(b\)\), AdaMTP likewise dominates standard MTP at every operating point and degrades more gracefully asnnincreases, since concentrating supervision within predictable chunks keeps even its deepest drafts frequently acceptable to the verifier\. Overall, across every head budgetnn, AdaMTP consistently surpasses standard MTP in both accuracy and inference speedup, confirming that its advantage is robust to the number of prediction heads rather than tied to a particular setting\.

### 4\.5Discussion on Adaptive\-Horizon Decoding

We now compare AdaMTP’s two inference modes—Fixed\-Horizon and Adaptive\-Horizon Generation—to quantify the benefit of entropy\-based candidate pruning\. To this end, we evaluate Adaptive\-Horizon against the Fixed\-Horizon baseline on GSM8K and HumanEval with Llama3\.1\-8B and Qwen2\.5\-7B, measuring the average number of tokens verified per step\. As shown in Figure[4](https://arxiv.org/html/2608.00434#S4.F4), the adaptive strategy consistently reduces the verification burden while keeping accuracy statistically indistinguishable from the baseline, confirming that the pruned tail candidates were largely redundant\. This reduction, however, yields little speedup for single\-sample inference, which is*memory\-bandwidth bound*: latency is dominated by streaming the model weights from DRAM, so trimming a few candidates neither reduces weight loading nor shortens the already\-parallel verification pass\. Under large\-batch,*compute\-bound*inference, by contrast, latency grows with the total number of candidate tokens, so pruning directly cuts the workload and yields clear throughput gains as batch size increases\.

![Refer to caption](https://arxiv.org/html/2608.00434v1/x4.png)Figure 4:Average number of candidate tokens verified per decoding step under the fixed\-horizon and adaptive\-horizon generation\. The adaptive strategy reduces the per\-step verification cost without degrading task accuracy\.

## 5Related Work

### 5\.1Multi\-Token Prediction

Qiet al\.\([2020](https://arxiv.org/html/2608.00434#bib.bib15)\)introducenn\-step\-ahead prediction in sequence\-to\-sequence pretraining to encourage future planning and reduce overfitting to local correlations\.Gloeckleet al\.\([2024](https://arxiv.org/html/2608.00434#bib.bib5)\)formalize multi\-token prediction by adding parallel heads during pretraining, improving reasoning over next\-token prediction, andBasharinet al\.\([2024](https://arxiv.org/html/2608.00434#bib.bib3)\)generalize these independent heads to a rank\-rrcanonical tensor decomposition to better capture dependencies among future tokens\. MTP has also been adopted at scale during the pretraining of industrial LLMs\(Liuet al\.,[2024](https://arxiv.org/html/2608.00434#bib.bib7); Xiaomiet al\.,[2025](https://arxiv.org/html/2608.00434#bib.bib14)\)for better data efficiency and long\-horizon planning\. Beyond pretraining from scratch, a more efficient line retrofits an existing pretrained NTP model into an MTP architecture\(Caiet al\.,[2025](https://arxiv.org/html/2608.00434#bib.bib10)\): Medusa\(Caiet al\.,[2024](https://arxiv.org/html/2608.00434#bib.bib11)\)attaches multiple lightweight heads that each forecast a token at a distinct future offset at a fraction of the pretraining cost;Samraghet al\.\([2025](https://arxiv.org/html/2608.00434#bib.bib13)\)equip the model with gated LoRA modules and a learnable sampler for simultaneous multi\-token prediction; and L\-MTP\(Liuet al\.,[2025](https://arxiv.org/html/2608.00434#bib.bib1)\)adds a leap\-based mechanism that predicts non\-sequential positions in a single forward pass to capture longer\-range dependencies\. These methods collectively show that retrofitting a pretrained NTP model with extra prediction heads is a practical, cost\-effective route to multi\-token prediction, yielding substantial inference speedups while preserving generation quality\. In addition, some approaches use MTP purely as an auxiliary training objective to improve the generation quality of NTP models, and thus provide no inference acceleration, since the model still decodes one token at a time at inference\. For instance, MuToR\(Gerontopouloset al\.,[2025](https://arxiv.org/html/2608.00434#bib.bib2)\)interleaves learnable register tokens into the training sequence to predict future targets, while TOP\(Zuhriet al\.,[2025](https://arxiv.org/html/2608.00434#bib.bib12)\)replaces exact future\-token prediction with a learning\-to\-rank loss that orders upcoming tokens by proximity\.

### 5\.2LLM Inference Acceleration

The growing cost of LLM inference has motivated acceleration methods that target different bottlenecks\. One line reduces the per\-step cost via model compression—quantization\(Hubaraet al\.,[2018](https://arxiv.org/html/2608.00434#bib.bib16); Kimet al\.,[2023](https://arxiv.org/html/2608.00434#bib.bib17); Linet al\.,[2024](https://arxiv.org/html/2608.00434#bib.bib18)\), pruning\(Frantar and Alistarh,[2023](https://arxiv.org/html/2608.00434#bib.bib21); Sunet al\.,[2023](https://arxiv.org/html/2608.00434#bib.bib22); Maet al\.,[2023](https://arxiv.org/html/2608.00434#bib.bib19); Gaoet al\.,[2024](https://arxiv.org/html/2608.00434#bib.bib20)\), and knowledge distillation\(Guet al\.,[2023](https://arxiv.org/html/2608.00434#bib.bib23); Hintonet al\.,[2015](https://arxiv.org/html/2608.00434#bib.bib24); Hsiehet al\.,[2023](https://arxiv.org/html/2608.00434#bib.bib26); Hoet al\.,[2023](https://arxiv.org/html/2608.00434#bib.bib27)\)—and efficient attention, whether linear\(Katharopouloset al\.,[2020](https://arxiv.org/html/2608.00434#bib.bib28); Yanget al\.,[2024](https://arxiv.org/html/2608.00434#bib.bib29)\), sparse\(Childet al\.,[2019](https://arxiv.org/html/2608.00434#bib.bib30); Luet al\.,[2025](https://arxiv.org/html/2608.00434#bib.bib31)\), or low\-rank\(Liuet al\.,[2024](https://arxiv.org/html/2608.00434#bib.bib7)\)\. A second line improves system\-level throughput through operator fusion\(Daoet al\.,[2022](https://arxiv.org/html/2608.00434#bib.bib33); Dao,[2023](https://arxiv.org/html/2608.00434#bib.bib34)\), KV\-cache management\(Kwonet al\.,[2023](https://arxiv.org/html/2608.00434#bib.bib35); Zhenget al\.,[2024](https://arxiv.org/html/2608.00434#bib.bib36)\), and parallelism\(NVIDIA,[2023](https://arxiv.org/html/2608.00434#bib.bib37)\)\. Orthogonally, a third line reduces the number of sequential decoding steps: speculative decoding\(Leviathanet al\.,[2023](https://arxiv.org/html/2608.00434#bib.bib38); Chenet al\.,[2023](https://arxiv.org/html/2608.00434#bib.bib39); Miaoet al\.,[2024](https://arxiv.org/html/2608.00434#bib.bib44)\)losslessly amortizes autoregressive generation by cheaply drafting candidate tokens and validating them in a single target\-model forward pass, with candidates proposed either by a separate lightweight model\(Leviathanet al\.,[2023](https://arxiv.org/html/2608.00434#bib.bib38); Yanget al\.,[2025](https://arxiv.org/html/2608.00434#bib.bib40); Zhouet al\.,[2023b](https://arxiv.org/html/2608.00434#bib.bib41)\)or by the target model itself via auxiliary prediction heads\(Sternet al\.,[2018](https://arxiv.org/html/2608.00434#bib.bib42); Caiet al\.,[2024](https://arxiv.org/html/2608.00434#bib.bib11); Liet al\.,[2024](https://arxiv.org/html/2608.00434#bib.bib43)\)\.

### 5\.3Adaptive Modeling

Motivated by the non\-uniform information density of natural language, recent work dynamically adjusts processing granularity rather than treating all tokens uniformly\(Barraultet al\.,[2024](https://arxiv.org/html/2608.00434#bib.bib46)\)\. The two most relevant to ours both adapt granularity on the*input or representation*side—the Byte Latent Transformer\(Pagnoniet al\.,[2025](https://arxiv.org/html/2608.00434#bib.bib6)\)patches bytes for the encoder, and Dynamic Large Concept Models\(Quet al\.,[2025](https://arxiv.org/html/2608.00434#bib.bib8)\)compress tokens into concepts—yet still emit a single token \(or byte\) per step and therefore cannot accelerate decoding\. AdaMTP instead applies entropy\-based, variable\-length segmentation to the*multi\-token prediction*itself: it turns each segment into an adaptive prediction depth that masks cross\-boundary supervision, simultaneously protecting the pretrained backbone and enabling accelerated multi\-token generation\.

## 6Conclusion

In this paper, we identified a key limitation of existing MTP: its fixed\-length horizon ignores the non\-uniform information density of language, forcing auxiliary heads to predict across high\-entropy boundaries and injecting noisy gradients that degrade the shared backbone\. To address this, we proposed AdaMTP, which uses entropy\-based segmentation to assign each token an adaptive prediction depth and applies a dynamically masked MTP objective that suppresses loss for cross\-boundary predictions\. Across mathematical reasoning, code, and general benchmarks on three backbones, AdaMTP consistently surpasses NTP and standard MTP in both task performance and inference speedup, showing that aligning the multi\-token objective with local predictability is a robust, architecture\-agnostic way to retrofit pretrained LLMs with efficient multi\-token generation\. Promising future directions include higher\-quality data curation and stronger adaptive\-horizon pruning for high\-throughput batched inference\.

## References

- J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le,et al\.\(2021\)Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[1st item](https://arxiv.org/html/2608.00434#S4.I1.i1.p1.1)\.
- L\. Barrault, P\. Duquenne, M\. Elbayad, A\. Kozhevnikov, B\. Alastruey, P\. Andrews, M\. Coria, G\. Couairon, M\. R\. Costa\-jussà, D\. Dale,et al\.\(2024\)Large concept models: language modeling in a sentence representation space\.arXiv preprint arXiv:2412\.08821\.Cited by:[§5\.3](https://arxiv.org/html/2608.00434#S5.SS3.p1.1)\.
- A\. Basharin, A\. Chertkov, and I\. Oseledets \(2024\)Faster language models with better multi\-token prediction using tensor decomposition\.arXiv preprint arXiv:2410\.17765\.Cited by:[§5\.1](https://arxiv.org/html/2608.00434#S5.SS1.p1.2)\.
- T\. Cai, Y\. Li, Z\. Geng, H\. Peng, J\. D\. Lee, D\. Chen, and T\. Dao \(2024\)Medusa: simple llm inference acceleration framework with multiple decoding heads\.arXiv preprint arXiv:2401\.10774\.Cited by:[§3\.2](https://arxiv.org/html/2608.00434#S3.SS2.p1.1),[§3\.3](https://arxiv.org/html/2608.00434#S3.SS3.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2608.00434#S5.SS1.p1.2),[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- Y\. Cai, X\. Liang, X\. Wang, J\. Ma, H\. Liang, J\. Luo, X\. Zuo, L\. Duan, Y\. Yin, and X\. Chen \(2025\)Fastmtp: accelerating llm inference with enhanced multi\-token prediction\.arXiv preprint arXiv:2509\.18362\.Cited by:[§5\.1](https://arxiv.org/html/2608.00434#S5.SS1.p1.2)\.
- S\. Chaudhary \(2023\)Code alpaca: an instruction\-following llama model for code generation\.Cited by:[1st item](https://arxiv.org/html/2608.00434#S4.I1.i1.p1.1)\.
- C\. Chen, S\. Borgeaud, G\. Irving, J\. Lespiau, L\. Sifre, and J\. Jumper \(2023\)Accelerating large language model decoding with speculative sampling\.arXiv preprint arXiv:2302\.01318\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. D\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[1st item](https://arxiv.org/html/2608.00434#S4.I1.i1.p1.1)\.
- R\. Child, S\. Gray, A\. Radford, and I\. Sutskever \(2019\)Generating long sequences with sparse transformers\.arXiv preprint arXiv:1904\.10509\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[1st item](https://arxiv.org/html/2608.00434#S4.I1.i1.p1.1)\.
- T\. Dao, D\. Fu, S\. Ermon, A\. Rudra, and C\. Ré \(2022\)Flashattention: fast and memory\-efficient exact attention with io\-awareness\.Advances in neural information processing systems35,pp\. 16344–16359\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- T\. Dao \(2023\)Flashattention\-2: faster attention with better parallelism and work partitioning\.arXiv preprint arXiv:2307\.08691\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- E\. Frantar and D\. Alistarh \(2023\)Sparsegpt: massive language models can be accurately pruned in one\-shot\.InInternational conference on machine learning,pp\. 10323–10337\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- S\. Gao, C\. Lin, T\. Hua, T\. Zheng, Y\. Shen, H\. Jin, and Y\. Hsu \(2024\)Disp\-llm: dimension\-independent structural pruning for large language models\.Advances in Neural Information Processing Systems37,pp\. 72219–72244\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- A\. Gerontopoulos, S\. Gidaris, and N\. Komodakis \(2025\)Multi\-token prediction needs registers\.arXiv preprint arXiv:2505\.10518\.Cited by:[§5\.1](https://arxiv.org/html/2608.00434#S5.SS1.p1.2)\.
- F\. Gloeckle, B\. Y\. Idrissi, B\. Rozière, D\. Lopez\-Paz, and G\. Synnaeve \(2024\)Better & faster large language models via multi\-token prediction\.arXiv preprint arXiv:2404\.19737\.Cited by:[§5\.1](https://arxiv.org/html/2608.00434#S5.SS1.p1.2)\.
- Y\. Gu, L\. Dong, F\. Wei, and M\. Huang \(2023\)Minillm: knowledge distillation of large language models\.arXiv preprint arXiv:2306\.08543\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2020\)Measuring massive multitask language understanding\.arXiv preprint arXiv:2009\.03300\.Cited by:[1st item](https://arxiv.org/html/2608.00434#S4.I1.i1.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the math dataset\.arXiv preprint arXiv:2103\.03874\.Cited by:[1st item](https://arxiv.org/html/2608.00434#S4.I1.i1.p1.1)\.
- G\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- N\. Ho, L\. Schmid, and S\. Yun \(2023\)Large language models are reasoning teachers\.InProceedings of the 61st annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 14852–14882\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- C\. Hsieh, C\. Li, C\. Yeh, H\. Nakhost, Y\. Fujii, A\. Ratner, R\. Krishna, C\. Lee, and T\. Pfister \(2023\)Distilling step\-by\-step\! outperforming larger language models with less training data and smaller model sizes\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 8003–8017\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- I\. Hubara, M\. Courbariaux, D\. Soudry, R\. El\-Yaniv, and Y\. Bengio \(2018\)Quantized neural networks: training neural networks with low precision weights and activations\.journal of machine learning research18\(187\),pp\. 1–30\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- A\. Katharopoulos, A\. Vyas, N\. Pappas, and F\. Fleuret \(2020\)Transformers are rnns: fast autoregressive transformers with linear attention\.InInternational conference on machine learning,pp\. 5156–5165\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- S\. Kim, C\. Hooper, A\. Gholami, Z\. Dong, X\. Li, S\. Shen, M\. W\. Mahoney, and K\. Keutzer \(2023\)Squeezellm: dense\-and\-sparse quantization\.arXiv preprint arXiv:2306\.07629\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- Y\. Leviathan, M\. Kalman, and Y\. Matias \(2023\)Fast inference from transformers via speculative decoding\.InInternational Conference on Machine Learning,pp\. 19274–19286\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- Y\. Li, F\. Wei, C\. Zhang, and H\. Zhang \(2024\)Eagle: speculative sampling requires rethinking feature uncertainty\.arXiv preprint arXiv:2401\.15077\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2023\)Let’s verify step by step\.InThe twelfth international conference on learning representations,Cited by:[1st item](https://arxiv.org/html/2608.00434#S4.I1.i1.p1.1)\.
- J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. Han \(2024\)Awq: activation\-aware weight quantization for on\-device llm compression and acceleration\.Proceedings of machine learning and systems6,pp\. 87–100\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.\(2024\)Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§5\.1](https://arxiv.org/html/2608.00434#S5.SS1.p1.2),[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- J\. Liu, C\. S\. Xia, Y\. Wang, and L\. Zhang \(2023\)Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation\.Advances in neural information processing systems36,pp\. 21558–21572\.Cited by:[1st item](https://arxiv.org/html/2608.00434#S4.I1.i1.p1.1)\.
- X\. Liu, X\. Xia, W\. Zhao, M\. Zhang, X\. Yu, X\. Su, S\. Yang, S\. Ng, and T\. Chua \(2025\)L\-mtp: leap multi\-token prediction beyond adjacent context for large language models\.arXiv preprint arXiv:2505\.17505\.Cited by:[§3\.2](https://arxiv.org/html/2608.00434#S3.SS2.p1.1),[1st item](https://arxiv.org/html/2608.00434#S4.I1.i1.p1.1),[§4\.2](https://arxiv.org/html/2608.00434#S4.SS2.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2608.00434#S5.SS1.p1.2)\.
- E\. Lu, Z\. Jiang, J\. Liu, Y\. Du, T\. Jiang, C\. Hong, S\. Liu, W\. He, E\. Yuan, Y\. Wang,et al\.\(2025\)Moba: mixture of block attention for long\-context llms\.arXiv preprint arXiv:2502\.13189\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- Z\. Luo, C\. Xu, P\. Zhao, Q\. Sun, X\. Geng, W\. Hu, C\. Tao, J\. Ma, Q\. Lin, and D\. Jiang \(2023\)Wizardcoder: empowering code large language models with evol\-instruct\.arXiv preprint arXiv:2306\.08568\.Cited by:[1st item](https://arxiv.org/html/2608.00434#S4.I1.i1.p1.1)\.
- X\. Ma, G\. Fang, and X\. Wang \(2023\)Llm\-pruner: on the structural pruning of large language models\.Advances in neural information processing systems36,pp\. 21702–21720\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- X\. Miao, G\. Oliaro, Z\. Zhang, X\. Cheng, Z\. Wang, Z\. Zhang, R\. Y\. Y\. Wong, A\. Zhu, L\. Yang, X\. Shi,et al\.\(2024\)Specinfer: accelerating large language model serving with tree\-based speculative inference and verification\.InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3,pp\. 932–949\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- NVIDIA \(2023\)TensorRT\-LLM\.External Links:[Link](https://github.com/NVIDIA/TensorRT-LLM)Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- A\. Pagnoni, R\. Pasunuru, P\. Rodriguez, J\. Nguyen, B\. Muller, M\. Li, C\. Zhou, L\. Yu, J\. E\. Weston, L\. Zettlemoyer,et al\.\(2025\)Byte latent transformer: patches scale better than tokens\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9238–9258\.Cited by:[§5\.3](https://arxiv.org/html/2608.00434#S5.SS3.p1.1)\.
- B\. Peng, C\. Li, P\. He, M\. Galley, and J\. Gao \(2023\)Instruction tuning with gpt\-4\.arXiv preprint arXiv:2304\.03277\.Cited by:[1st item](https://arxiv.org/html/2608.00434#S4.I1.i1.p1.1)\.
- W\. Qi, Y\. Yan, Y\. Gong, D\. Liu, N\. Duan, J\. Chen, R\. Zhang, and M\. Zhou \(2020\)ProphetNet: predicting future n\-gram for sequence\-to\-sequencepre\-training\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 2401–2410\.Cited by:[§5\.1](https://arxiv.org/html/2608.00434#S5.SS1.p1.2)\.
- X\. Qu, S\. Wang, Z\. Huang, K\. Hua, F\. Yin, R\. Zhu, J\. Zhou, Q\. Min, Z\. Wang, Y\. Li,et al\.\(2025\)Dynamic large concept models: latent reasoning in an adaptive semantic space\.arXiv preprint arXiv:2512\.24617\.Cited by:[§5\.3](https://arxiv.org/html/2608.00434#S5.SS3.p1.1)\.
- M\. Samragh, A\. Kundu, D\. Harrison, K\. Nishu, D\. Naik, M\. Cho, and M\. Farajtabar \(2025\)Your llm knows the future: uncovering its multi\-token prediction potential\.arXiv preprint arXiv:2507\.11851\.Cited by:[§5\.1](https://arxiv.org/html/2608.00434#S5.SS1.p1.2)\.
- M\. Stern, N\. Shazeer, and J\. Uszkoreit \(2018\)Blockwise parallel decoding for deep autoregressive models\.Advances in Neural Information Processing Systems31\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- M\. Sun, Z\. Liu, A\. Bair, and J\. Z\. Kolter \(2023\)A simple and effective pruning approach for large language models\.arXiv preprint arXiv:2306\.11695\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- L\. Xiaomi, B\. Xia, B\. Shen, D\. Zhu, D\. Zhang, G\. Wang, H\. Zhang, H\. Liu, J\. Xiao, J\. Dong,et al\.\(2025\)MiMo: unlocking the reasoning potential of language model–from pretraining to posttraining\.arXiv preprint arXiv:2505\.07608\.Cited by:[§5\.1](https://arxiv.org/html/2608.00434#S5.SS1.p1.2)\.
- S\. Yang, S\. Huang, X\. Dai, and J\. Chen \(2025\)Multi\-candidate speculative decoding\.InCCF International Conference on Natural Language Processing and Chinese Computing,pp\. 335–348\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- S\. Yang, B\. Wang, Y\. Zhang, Y\. Shen, and Y\. Kim \(2024\)Parallelizing linear transformers with the delta rule over sequence length\.Advances in neural information processing systems37,pp\. 115491–115522\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- L\. Zheng, L\. Yin, Z\. Xie, C\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. E\. Gonzalez,et al\.\(2024\)Sglang: efficient execution of structured language model programs\.Advances in neural information processing systems37,pp\. 62557–62583\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou \(2023a\)Instruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.Cited by:[1st item](https://arxiv.org/html/2608.00434#S4.I1.i1.p1.1)\.
- Y\. Zhou, K\. Lyu, A\. S\. Rawat, A\. K\. Menon, A\. Rostamizadeh, S\. Kumar, J\. Kagy, and R\. Agarwal \(2023b\)Distillspec: improving speculative decoding via knowledge distillation\.arXiv preprint arXiv:2310\.08461\.Cited by:[§5\.2](https://arxiv.org/html/2608.00434#S5.SS2.p1.1)\.
- Z\. M\. Zuhri, E\. H\. Fuadi, and A\. F\. Aji \(2025\)Predicting the order of upcoming tokens improves language modeling\.arXiv preprint arXiv:2508\.19228\.Cited by:[§5\.1](https://arxiv.org/html/2608.00434#S5.SS1.p1.2)\.