CForce: Boosting Parallel Decoding for dLLMs via Consistency Forcing

# CForce: Boosting Parallel Decoding for dLLMs via Consistency Forcing
Source: [https://arxiv.org/html/2608.13925](https://arxiv.org/html/2608.13925)
Chenkai XuAffiliation:Shanghai Jiao Tong UniversityZhuocheng GongAffiliation:Ant Group\{renyj26, 132435xck, zhijied\}@sjtu\.edu\.cn,lijg\.zero@antgroup\.comJianguo LiZhijie Deng

###### Abstract

Diffusion large language models \(dLLMs\) accelerate language generation by predicting multiple masks in a single forward pass\. However, existing dLLMs can suffer from unreliable predictions in early denoising stages under aggressive parallelism strategies, leading to errors that can propagate to later stages\. To tackle this issue, we present Consistency Forcing \(CForce\) for dLLMs, a distillation method to force the mask predictions of early stages to align with those of later stages\. CForce trains the model on pre\-collected self\-rollout trajectories, thereby improving training\-inference alignment\. We introduce Confidence Adaptive KL Divergence as a distillation objective to conjoin the merits of forward and reverse KL\. We further provide a theoretical analysis for the consistency objective to explain why CForce can approximately minimize the prediction error of early stages\. Critically, the same formulation applies to both mask\-to\-token decoding and edit\-capable decoding; in the edit\-capable case, later token\-to\-token refinements provide additional supervision for earlier masked\-state predictions\. Experiments on non\-edit and edit\-capable LLaDA models show improved speed\-quality trade\-offs, especially under high\-parallelism decoding budgets\. Code is available at:[https://github\.com/inclusionAI/dFactory](https://github.com/inclusionAI/dFactory)\.

11footnotetext:Equal contribution\.22footnotetext:Corresponding authors\.## 1Introduction

Figure 1:Speed–quality trade\-off: accuracy vs\. tokens per forward pass \(TPF\)\.Upper\-right is better\. Consistency Forcing moves both models toward a higher\-parallelism trade\-off\.Figure 2:Per\-task Accuracy Under Parallelism \(AUP\)\.AUP integrates the score–parallelism curve into a single metric\. CForce yields consistent gains across all four benchmarks\.Diffusion large language models \(dLLMs\)\([Austin et al\. 2021a](https://arxiv.org/html/2608.13925#bib.bib4);[Zheng et al\. 2023](https://arxiv.org/html/2608.13925#bib.bib46);[Lou et al\. 2024](https://arxiv.org/html/2608.13925#bib.bib27);[Sahoo et al\. 2024](https://arxiv.org/html/2608.13925#bib.bib34);[Nie et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib32);[Ye et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib43);[Arriola et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib3);[Gong et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib17)\)generate text by iteratively denoising masked sequences, committing multiple tokens per forward pass to achieve high parallelism\([Wu et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib40);[Bie et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib7);[Bie et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib6);[Wang et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib39);[Cheng et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib13)\)\. However, existing dLLMs can suffer from unreliable predictions in early denoising stages under aggressive parallelism strategies, and the resulting errors can propagate to later stages\. Thus, faster dLLM decoding depend not only on reducing denoising steps, but also on improving the reliability of early\-stage predictions under high\-parallelism decoding\.

This issue is especially important for edit\-capable dLLMs\. Conventional masked diffusion decoding is mainly mask\-to\-token \(M2T\), whereas edit\-capable models such as LLaDA2\.1\([Bie et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib6)\)also perform token\-to\-token \(T2T\) refinement, allowing later denoising states to revise earlier drafts\. These later states contain richer context and possible T2T corrections, which could provide stronger supervision for early M2T predictions before token commitment\. Existing acceleration and distillation methods do not train this early\-to\-late consistency along the model’s own threshold\-decoding trajectory: some modify decoding or caching\([Kim et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib21);[Wu et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib40);[Liu et al\. 2025b](https://arxiv.org/html/2608.13925#bib.bib26)\), while others use constructed, teacher\-generated, or privileged trajectories\([Zhang et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib44);[Kim et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib20);[Liang et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib23)\)whose information conditions differ from the student’s inference states\.

We propose Consistency Forcing \(CForce\) for dLLMs, a distillation method inspired by consistency models\([Salimans and Ho 2022](https://arxiv.org/html/2608.13925#bib.bib36);[Song et al\. 2023](https://arxiv.org/html/2608.13925#bib.bib37)\)that forces the mask predictions of early stages to align with those of later stages\. Specifically, starting from a pretrained dLLM, CForce trains the model on self\-rollout trajectories, thereby improving training\-inference alignment\. Then, CForce partitions each pre\-collected trajectory into stages\. Each stage boundary is set after a fixed number of newly revealed masked tokens, rather than at every native step, which would incur substantial trajectory\-storage overhead\. For each adjacent pair, the earlier\-stage prediction is trained to match a stop\-gradient prediction from the same model at the later stage, without relying on a frozen teacher\. Still\-masked positions are aligned withConfidenceAdaptive KLDivergence \(CAD\), which dynamically interpolates between forward and reverse KL based on later\-stage prediction confidence\. CForce also applies a cross\-entropy \(CE\) anchor to stabilize token commitment\. A curriculum over reveal difficulty further stabilizes this adjacent\-stage objective\([Xu et al\. 2025b](https://arxiv.org/html/2608.13925#bib.bib42);[Liu et al\. 2025a](https://arxiv.org/html/2608.13925#bib.bib25)\)\.

We further provide a theoretical analysis supporting this design: early\-stage prediction error is upper\-bounded by adjacent\-stage distributional drift plus reveal\-boundary token error \(Section[4\.4](https://arxiv.org/html/2608.13925#S4.SS4)\), two terms that correspond directly to the CAD and CE components of our objective\.

Empirically, the clearest result appears on the edit\-capable LLaDA2\.1\-mini\([Bie et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib6)\): Consistency Forcing increases average Tokens Per Forward \(TPF\) from 6\.94 to 9\.08 while improving average accuracy from 85\.57 to 86\.41\. On the non\-edit LLaDA2\.0\-mini\([Bie et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib7)\), the method shows an explicit speed\-quality trade\-off, increasing average TPF from 3\.60 to 6\.42 and improving few\-step accuracy at fixed TPF budgets under aggressive decoding\. Figure[1](https://arxiv.org/html/2608.13925#S1.F1)and Figure[2](https://arxiv.org/html/2608.13925#S1.F2)visualize this effect\.

Our contributions are fourfold: \(1\) we reframe dLLM acceleration around early\-stage reliability under aggressive threshold decoding, rather than focusing only on fewer sampling steps; \(2\) we introduce Consistency Forcing, which aligns adjacent stages on the model’s own decoding trajectory using CAD and a CE anchor; \(3\) we instantiate this idea in edit\-capable dLLMs, where later T2T\-refined stages supervise earlier M2T predictions; and \(4\) we show that the same recipe improves few\-step and high TPF behavior on a non\-edit dLLM, with an explicit speed\-quality trade\-off\.

## 2Related Work

#### Masked Diffusion Language Models\.

Masked diffusion language models \(MDLMs\) cast text generation as discrete denoising\([Austin et al\. 2021a](https://arxiv.org/html/2608.13925#bib.bib4);[Zheng et al\. 2023](https://arxiv.org/html/2608.13925#bib.bib46);[Lou et al\. 2024](https://arxiv.org/html/2608.13925#bib.bib27)\)\. Building on earlier masked generation methods\([Ghazvininejad et al\. 2019](https://arxiv.org/html/2608.13925#bib.bib16);[Chang et al\. 2022](https://arxiv.org/html/2608.13925#bib.bib8)\), recent dLLMs further scale this paradigm, enabling stronger text and code generation\([Nie et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib32);[Ye et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib43);[Gong et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib17)\)\. To improve generation speed, several dLLMs introduce confidence\-based parallel decoding\([Wu et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib40);[Wang et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib39);[Cheng et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib13);[Bie et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib7);[Chen et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib12)\), while edit\-capable models allow later denoising states to revise committed tokens\([Bie et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib6)\)\. These later corrections are especially relevant under aggressive parallel decoding, where early M2T predictions are made with limited context and can benefit from supervision induced by later denoising states\.

#### Acceleration Methods for dLLMs\.

Existing acceleration methods follow several complementary directions\. Inference\-time approaches improve token selection or verification without changing model parameters\([Kim et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib21);[Xu et al\. 2025a](https://arxiv.org/html/2608.13925#bib.bib41);[Agrawal et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib1);[Gao et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib15)\), while system\-level approaches reduce computation through caching\([Wu et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib40);[Ma et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib30);[Liu et al\. 2025b](https://arxiv.org/html/2608.13925#bib.bib26)\)\. Training\-based approaches instead improve few\-step generation with constructed, teacher\-generated, or privileged trajectories\([Chen et al\. 2025b](https://arxiv.org/html/2608.13925#bib.bib11);[Zhang et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib44);[Kim et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib20);[Liang et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib23)\)\. These methods accelerate decoding but do not directly train the model on the low\-context states produced by its own threshold\-decoding path\.

#### Consistency Distillation in Diffusion Models\.

Diffusion distillation reduces sampling cost by compressing slow multi\-step generation into faster samplers or models\([Luhman and Luhman 2021](https://arxiv.org/html/2608.13925#bib.bib28);[Salimans and Ho 2022](https://arxiv.org/html/2608.13925#bib.bib36);[Meng et al\. 2023](https://arxiv.org/html/2608.13925#bib.bib31)\)\. Consistency models extend this idea by enforcing agreement across trajectory states, enabling one\-step or few\-step generation\([Song et al\. 2023](https://arxiv.org/html/2608.13925#bib.bib37);[Luo et al\. 2023](https://arxiv.org/html/2608.13925#bib.bib29);[Kou et al\. 2024](https://arxiv.org/html/2608.13925#bib.bib22)\)\. Related objectives have also been studied for discrete generative models\([Hayakawa et al\. 2024](https://arxiv.org/html/2608.13925#bib.bib18);[Sahoo et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib35);[Xu et al\. 2025b](https://arxiv.org/html/2608.13925#bib.bib42)\)and, more recently, for dLLMs\([Zhang et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib44);[Kim et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib20);[Liang et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib23)\)\. These works demonstrate the usefulness of cross\-state supervision\. However, they typically depend on separate teachers, fixed representations, or teacher\-generated targets\. In contrast, Consistency Forcing constructs adjacent\-stage constraints from the model’s own threshold\-decoding trajectory without a frozen teacher, and adapts them to both masked\-token reveal and edit transitions\.

## 3Preliminaries

Figure 3:An illustration of edit\-capable dLLM decoding\. At each decoding step, high\-confidence masked positions are denoised, while already generated tokens may also be revised to correct earlier mistakes\.#### Masked Diffusion Language Models\.

![Refer to caption](https://arxiv.org/html/2608.13925v1/fig3.png)Figure 4:Overview of Consistency Forcing\.Left: self\-rollout threshold\-decoding trajectories are downsampled into adjacent stage pairs whenever the cumulative number of M2T reveals reaches the stage sizeSS; right: the earlier low\-context state is trained against same\-model stop\-gradient predictions from the later, more informative state\.Masked diffusion language models formulate text generation as denoising in a discrete token space\([Austin et al\. 2021a](https://arxiv.org/html/2608.13925#bib.bib4);[Lou et al\. 2024](https://arxiv.org/html/2608.13925#bib.bib27);[Sahoo et al\. 2024](https://arxiv.org/html/2608.13925#bib.bib34)\)\. Given a promptx∈𝒳x\\in\\mathcal\{X\}and a clean target sequencey=\(y1,…,yL\)y=\(y^\{1\},\\ldots,y^\{L\}\)over vocabulary𝒱\\mathcal\{V\}, the forward process corruptsyyby independently replacing tokens with\[MASK\]\[\\mathrm\{MASK\}\]\. Letαt∈\[0,1\]\\alpha\_\{t\}\\in\[0,1\]denote the probability that a token remains unmasked at noise timett, and letsts\_\{t\}be the corrupted sequence\. Under the absorbing\-mask process,

q⁡\(stj∣yj\)=\{αt,stj=yj,1−αt,stj=\[MASK\]\.q\(s\_\{t\}^\{j\}\\mid y^\{j\}\)=\\begin\{cases\}\\alpha\_\{t\},&s\_\{t\}^\{j\}=y^\{j\},\\\\ 1\-\\alpha\_\{t\},&s\_\{t\}^\{j\}=\[\\mathrm\{MASK\}\]\.\\end\{cases\}
The reverse model predicts a categorical distributionqθ\(⋅∣x,st,j\)q\_\{\\theta\}\(\\cdot\\mid x,s\_\{t\},j\)for each masked positionjj\.

LetMt=\{j:stj=\[MASK\]\}M\_\{t\}=\\\{j:s\_\{t\}^\{j\}=\[\\mathrm\{MASK\}\]\\\}be the masked positions\. Training can be written as the standard variational bound for masked diffusion, which reduces to a weighted cross\-entropy over these positions:

ℒMDLM\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{MDLM\}\}=𝔼\(x,y\),t,st​\[w⁡\(t\)​∑j∈MtCEθj\],\\displaystyle=\\mathbb\{E\}\_\{\(x,y\),t,s\_\{t\}\}\\left\[w\(t\)\\sum\_\{j\\in M\_\{t\}\}\\mathrm\{CE\}\_\{\\theta\}^\{j\}\\right\],\(1\)CEθj\\displaystyle\\mathrm\{CE\}\_\{\\theta\}^\{j\}=−log⁡qθ​\(yj∣x,st,j\)\.\\displaystyle=\-\\log q\_\{\\theta\}\(y^\{j\}\\mid x,s\_\{t\},j\)\.Herew\(t\)=−αt′/\(1−αt\)w\(t\)=\-\\alpha^\{\\prime\}\_\{t\}/\(1\-\\alpha\_\{t\}\)is the positive ELBO weight induced by the absorbing diffusion process\. This objective trains the model to recover clean tokens from partially observed contexts, enabling multiple positions to be denoised in parallel at inference time\.

In current large\-scale dLLMs, this masked diffusion objective is commonly used together with block diffusion\([Arriola et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib3);[Bie et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib7)\), which partitions the target sequence into blocks and denoises each block in parallel while generating blocks sequentially\. Recent edit\-capable dLLMs further augment this block\-diffusion framework with token\-to\-token refinement, allowing committed tokens to be revised in later denoising states\([Bie et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib6)\)\. Hereafter, we follow this block\-diffusion setting and discuss decoding and training within a block unless otherwise specified\.

#### Confidence\-based Parallel Decoding\.

At inference time, block diffusion generates one block by starting from an all\-mask block states0s\_\{0\}and iteratively denoising it into the all\-clean state\. This process produces a trajectory𝒯=\(s0,s1,…,sT\)\\mathcal\{T\}=\(s\_\{0\},s\_\{1\},\\ldots,s\_\{T\}\), whereTTis the number of decoding steps allocated to the block\. At steptt, letMt=\{j:stj=\[MASK\]\}M\_\{t\}=\\\{j:s\_\{t\}^\{j\}=\[\\mathrm\{MASK\}\]\\\}denote the positions that remain masked, and letptj​\(v\)=qθ​\(v∣x,st,j\)p\_\{t\}^\{j\}\(v\)=q\_\{\\theta\}\(v\\mid x,s\_\{t\},j\)be the model distribution at positionj∈Mtj\\in M\_\{t\}\.

Modern dLLMs reveal multiple positions in each forward pass according to model confidence\([Bie et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib7)\)\. The confidence of a masked position is defined asctj=maxv∈𝒱⁡ptj​\(v\)c\_\{t\}^\{j\}=\\max\_\{v\\in\\mathcal\{V\}\}p\_\{t\}^\{j\}\(v\)\. Given a thresholdτ\\tau, all positions whose confidence exceeds the threshold are selected for commitment\. To ensure progress, if no position exceeds the threshold, the most confident remaining position is selected:

ΔMt=\{\{j∈Mt:ctj\>τ\},if​\{j∈Mt:ctj\>τ\}≠∅,\{arg⁡maxj∈Mt⁡ctj\},otherwise\.\\Delta M\_\{t\}=\\begin\{cases\}\\\{j\\in M\_\{t\}:c\_\{t\}^\{j\}\>\\tau\\\},&\\text\{if \}\\\{j\\in M\_\{t\}:c\_\{t\}^\{j\}\>\\tau\\\}\\neq\\emptyset,\\\\ \\\{\\arg\\max\_\{j\\in M\_\{t\}\}c\_\{t\}^\{j\}\\\},&\\text\{otherwise\}\.\\end\{cases\}\(2\)For each selected positionj∈Δ​Mtj\\in\\Delta M\_\{t\}, the decoder commits the most probable token,

st\+1j=arg⁡maxv∈𝒱​ptj​\(v\),s\_\{t\+1\}^\{j\}=\\arg\\max\_\{v\\in\\mathcal\{V\}\}p\_\{t\}^\{j\}\(v\),\(3\)and the remaining positions stay masked\. Equivalently,

Mt\+1=Mt∖Δ​Mt\.M\_\{t\+1\}=M\_\{t\}\\setminus\\Delta M\_\{t\}\.\(4\)

#### Edit\-capable Diffusion Decoding\.

Recent edit\-capable dLLMs extend the confidence\-based decoding process by allowing T2T refinement after M2T commitment\([Bie et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib6)\)\. Given an edit thresholdτedit\\tau\_\{\\mathrm\{edit\}\}, positions selected for editing are

Et=\{j∉Mt:maxv∈𝒱,v≠stj⁡qθ​\(v∣x,st,j\)\>τedit\}\.E\_\{t\}=\\left\\\{j\\notin M\_\{t\}:\\max\_\{v\\in\\mathcal\{V\},\\,v\\neq s\_\{t\}^\{j\}\}q\_\{\\theta\}\(v\\mid x,s\_\{t\},j\)\>\\tau\_\{\\mathrm\{edit\}\}\\right\\\}\.\(5\)For eachj∈Etj\\in E\_\{t\}, the decoder replaces the current token with the most probable alternative,

st\+1j=arg⁡maxv∈𝒱,v≠stj​qθ​\(v∣x,st,j\)\.s\_\{t\+1\}^\{j\}=\\arg\\max\_\{v\\in\\mathcal\{V\},\\,v\\neq s\_\{t\}^\{j\}\}q\_\{\\theta\}\(v\\mid x,s\_\{t\},j\)\.\(6\)
As illustrated in Figure[3](https://arxiv.org/html/2608.13925#S3.F3), in edit\-capable decoding, later states can correct tokens that were committed earlier\. These T2T refinements are especially important for fast decoding when the M2T thresholdτ\\tauis low\. Consistency Forcing uses this later\-stage refinement signal to supervise earlier masked\-state predictions\.

## 4Method

### 4\.1Overview

Consistency Forcing trains a dLLM to make more reliable predictions from early, low\-context denoising states\. The method collects threshold\-decoding trajectories from the pretrained initializationqθ0q\_\{\\theta\_\{0\}\}, converts them into staged adjacent pairs, and uses the later stage as a same\-model stop\-gradient target for the earlier stage\. The training objective combines adjacent\-stage distributional alignment, a reveal\-boundary CE anchor, and a curriculum over transition difficulty; building on this, edit\-capable models can further feed later T2T signals back into earlier states to improve M2T drafting\. Figure[4](https://arxiv.org/html/2608.13925#S3.F4)illustrates the overall pipeline\.

### 4\.2Staged Trajectory Construction

We need to construct trajectories used for consistency forcing\. Specifically, we first collect the original confidence\-based decoding trajectory𝒯=\(s0,s1,…,sT\)\\mathcal\{T\}=\(s\_\{0\},s\_\{1\},\\ldots,s\_\{T\}\)using the cold\-start initializationqθ0q\_\{\\theta\_\{0\}\}as described in Section[3](https://arxiv.org/html/2608.13925#S3)\. Rather than treating every consecutive native transition in𝒯\\mathcal\{T\}as a training pair, we use the native rollout only as the source trajectory: dense native\-step supervision is inefficient and often provides only a weak training signal, because adjacent native states may differ by very few newly revealed tokens\.

Next, we maintain a reveal counter along the rollout\. Whenever the cumulative number of newly revealed tokens reaches a fixed stage sizeSS, we save the current state as a stage boundary and reset the counter\. The initial all\-mask state is always saved, and the final decoded state is appended if it has not already been saved\. This produces a staged trajectory𝒯¯=\(s¯0,s¯1,…,s¯N\)\\bar\{\\mathcal\{T\}\}=\(\\bar\{s\}\_\{0\},\\bar\{s\}\_\{1\},\\ldots,\\bar\{s\}\_\{N\}\), where adjacent stages are separated by cumulative M2T progress rather than by the raw native update index\.

We then form adjacent pairs from𝒯¯\\bar\{\\mathcal\{T\}\}\. Downsampling by cumulative M2T reveals forms coarser stages, so each adjacent pair contains a more substantial change in visible context\.

For each adjacent stage pair\(s¯i,s¯i\+1\)\(\\bar\{s\}\_\{i\},\\bar\{s\}\_\{i\+1\}\), letM¯i=\{j:s¯ij=\[MASK\]\}\\bar\{M\}\_\{i\}=\\\{j:\\bar\{s\}\_\{i\}^\{j\}=\[\\mathrm\{MASK\}\]\\\}denote the positions masked in the earlier stage\. We split these positions into

𝒰i=M¯i∩M¯i\+1,Δi=M¯i∖M¯i\+1\.\\mathcal\{U\}\_\{i\}=\\bar\{M\}\_\{i\}\\cap\\bar\{M\}\_\{i\+1\},\\qquad\\Delta\_\{i\}=\\bar\{M\}\_\{i\}\\setminus\\bar\{M\}\_\{i\+1\}\.\(7\)Here,𝒰i\\mathcal\{U\}\_\{i\}contains positions that remain masked in both stages, whileΔi\\Delta\_\{i\}contains positions that are revealed between the two stages\. These sets define the supports of the CAD and CE terms below\.

For edit\-capable decoding, we also record native edit events during the trajectory rollout and aggregate them between adjacent stage boundaries; the resulting stage\-level edit setE¯i\\bar\{E\}\_\{i\}is introduced in Section[4\.3](https://arxiv.org/html/2608.13925#S4.SS3.SSS0.Px3)\. Since all stages are obtained from the model’s own confidence\-based decoding process, the staged trajectories preserve the mask, reveal, and edit patterns that arise during fast inference\.

### 4\.3Adjacent\-stage Consistency Forcing

For each sampled adjacent stage pair\(s¯i,s¯i\+1\)\(\\bar\{s\}\_\{i\},\\bar\{s\}\_\{i\+1\}\), we use the earlier stage as the student input and the later, more informative stage as the target context\. The target branch is evaluated under stop\-gradient, denoted bysg⁡\[⋅\]\\mathrm\{sg\}\[\\cdot\], so the model learns to approximate froms¯i\\bar\{s\}\_\{i\}the prediction it would make after receiving the additional context ins¯i\+1\\bar\{s\}\_\{i\+1\}\. We instantiate this adjacent\-stage constraint with two complementary terms: CAD controls soft distributional drift on positions that remain latent across the transition, while a CE anchor stabilizes positions at the M2T reveal boundary\. This decomposition follows the error\-bound motivation in Section[4\.4](https://arxiv.org/html/2608.13925#S4.SS4)\.

#### Confidence Adaptive KL Divergence\.

On𝒰i\\mathcal\{U\}\_\{i\}, both stages are still masked, so we align the student distributionpstuj=qθ\(⋅∣x,s¯i,j\)p\_\{\\mathrm\{stu\}\}^\{j\}=q\_\{\\theta\}\(\\cdot\\mid x,\\bar\{s\}\_\{i\},j\)with the stop\-gradient target distributionptarj=sg\[qθ\(⋅∣x,s¯i\+1,j\)\]p\_\{\\mathrm\{tar\}\}^\{j\}=\\mathrm\{sg\}\[q\_\{\\theta\}\(\\cdot\\mid x,\\bar\{s\}\_\{i\+1\},j\)\]\. A standard forward KL minimizes drift but can be weak at sharpening predictions\. Conversely, reverse KL is a strong mode\-seeking objective but risks premature collapse when applied to uncertain targets\. To combine their strengths, we design Confidence Adaptive KL Divergence \(CAD\), which dynamically mixes forward and reverse KL based on the later stage’s prediction confidencecj=maxv⁡ptarj​\(v\)c\_\{j\}=\\max\_\{v\}p\_\{\\mathrm\{tar\}\}^\{j\}\(v\), defined as

ℒCAD=1\|𝒰i\|∑j∈𝒰i\[DKL\(ptarj∥pstuj\)\+cjDKL\(pstuj∥ptarj\)\],\\mathcal\{L\}\_\{\\mathrm\{CAD\}\}=\\frac\{1\}\{\|\\mathcal\{U\}\_\{i\}\|\}\\sum\_\{j\\in\\mathcal\{U\}\_\{i\}\}\\left\[D\_\{\\mathrm\{KL\}\}\(p\_\{\\mathrm\{tar\}\}^\{j\}\\\|p\_\{\\mathrm\{stu\}\}^\{j\}\)\+c\_\{j\}D\_\{\\mathrm\{KL\}\}\(p\_\{\\mathrm\{stu\}\}^\{j\}\\\|p\_\{\\mathrm\{tar\}\}^\{j\}\)\\right\],\(8\)where the forward KL term stably controls distributional drift connected to the upper bound in Section[4\.4](https://arxiv.org/html/2608.13925#S4.SS4), while the confidence\-weighted reverse KL adds mode\-seeking pressure only when the later\-stage prediction is confident\.

#### CE Anchor\.

On the newly revealed positionsΔi\\Delta\_\{i\}, we apply a cross\-entropy anchor to stabilize token commitments:

ℒCE=−1\|Δi\|∑j∈Δilogqθ\(yj∣x,s¯i,j\),\\mathcal\{L\}\_\{\\mathrm\{CE\}\}=\-\\frac\{1\}\{\|\\Delta\_\{i\}\|\}\\sum\_\{j\\in\\Delta\_\{i\}\}\\log q\_\{\\theta\}\(y^\{j\}\\mid x,\\bar\{s\}\_\{i\},j\),\(9\)whereyjy^\{j\}denotes the clean target token at positionjj\. By anchoring the student prediction immediately before reveal, the CE term discourages errors at the boundary where masked positions become committed tokens\.

#### Edit\-capable dLLMs\.

As described in Section[3](https://arxiv.org/html/2608.13925#S3), edit\-capable dLLMs provide an additional source of later\-stage supervision: T2T edits of already visible tokens\. We define the stage\-level edit setE¯i\\bar\{E\}\_\{i\}as the positions whose visible tokens differ between adjacent stagess¯i\\bar\{s\}\_\{i\}ands¯i\+1\\bar\{s\}\_\{i\+1\}:

E¯i=\{j:\\displaystyle\\bar\{E\}\_\{i\}=\\\{j:s¯ij≠\[MASK\],s¯i\+1j≠\[MASK\],\\displaystyle\\bar\{s\}\_\{i\}^\{j\}\\neq\[\\mathrm\{MASK\}\],\\;\\bar\{s\}\_\{i\+1\}^\{j\}\\neq\[\\mathrm\{MASK\}\],s¯ij≠s¯i\+1j\}\.\\displaystyle\\bar\{s\}\_\{i\}^\{j\}\\neq\\bar\{s\}\_\{i\+1\}^\{j\}\\\}\.\(10\)Edit\-capable trajectories contain both M2T reveals and later T2T corrections, so we broaden the CAD domain from𝒰i\\mathcal\{U\}\_\{i\}toM¯i∪E¯i\\bar\{M\}\_\{i\}\\cup\\bar\{E\}\_\{i\}\. TheM¯i\\bar\{M\}\_\{i\}term aligns student\-masked positions with later soft predictions under additional revealed context, whileE¯i\\bar\{E\}\_\{i\}lets visible tokens that change between stages contribute T2T correction signals\. Since these later\-stage soft distributions provide broader supervision, we also extend the CE anchor fromΔi\\Delta\_\{i\}to all student\-masked positionsM¯i\\bar\{M\}\_\{i\}\. This uses edit\-capable trajectories without changing the decoding algorithm\.

#### Curriculum Transition\.

At the beginning of optimization, the full stage gap can be difficult because the student has not yet learned to predict tokens that are revealed many positions ahead\. We therefore expose the later\-stage context gradually\. At optimization stepuu, only a fractionρu\\rho\_\{u\}of the newly revealed positions inΔi\\Delta\_\{i\}are used to form the later\-stage context:

ρu=min⁡\(1,ρ0\+\(1−ρ0\)​uTtrain\),\\rho\_\{u\}=\\min\\\!\\left\(1,\\;\\rho\_\{0\}\+\(1\-\\rho\_\{0\}\)\\frac\{u\}\{T\_\{\\mathrm\{train\}\}\}\\right\),\(11\)whereρ0=0\.1\\rho\_\{0\}=0\.1andTtrainT\_\{\\mathrm\{train\}\}is the total training steps\. Concretely, we select a subsetΓi​\(u\)⊆Δi\\Gamma\_\{i\}\(u\)\\subseteq\\Delta\_\{i\}with\|Γi​\(u\)\|=⌊ρu​\|Δi\|⌋\|\\Gamma\_\{i\}\(u\)\|=\\lfloor\\rho\_\{u\}\|\\Delta\_\{i\}\|\\rfloorand form a partially revealed later state by revealing onlyΓi​\(u\)\\Gamma\_\{i\}\(u\)while keeping the remaining positions inΔi\\Delta\_\{i\}masked\. Initially, the target state is close to the student state, stabilizing local consistency learning; asρu\\rho\_\{u\}increases, the model is gradually exposed to larger adjacent\-stage transitions and eventually learns the consistency constraint used by fast decoding\.

#### Overall Objective\.

The training loss is

ℒ=ℒCAD\+λCE​ℒCE,\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{CAD\}\}\+\\lambda\_\{\\mathrm\{CE\}\}\\mathcal\{L\}\_\{\\mathrm\{CE\}\},\(12\)whereλCE\\lambda\_\{\\mathrm\{CE\}\}balances distributional forcing and token anchoring\.

### 4\.4Theoretical Analysis

#### Setup\.

We keep the main text focused on the intuition and key conclusion, and defer the full derivation to Appendix[C](https://arxiv.org/html/2608.13925#A3)\. The goal is to justify why the Consistency Forcing objective targets early\-stage reliability\. For two categorical distributionsppandqqover vocabulary𝒱\\mathcal\{V\}, the total variation \(TV\) distance is defined as

DTV​\(p,q\)=12​∑v∈𝒱\|p⁡\(v\)−q⁡\(v\)\|\.D\_\{\\mathrm\{TV\}\}\(p,q\)=\\frac\{1\}\{2\}\\sum\_\{v\\in\\mathcal\{V\}\}\|p\(v\)\-q\(v\)\|\.\(13\)Consider a positionjjrevealed at stagerjr\_\{j\}, with clean target tokenyj=s¯Njy^\{j\}=\\bar\{s\}\_\{N\}^\{j\}, and letpij=qθ\(⋅∣x,s¯i,j\)p\_\{i\}^\{j\}=q\_\{\\theta\}\(\\cdot\\mid x,\\bar\{s\}\_\{i\},j\)\. For any earlier masked stagei<rji<r\_\{j\}, the prediction error ats¯i\\bar\{s\}\_\{i\}is measured asDTV​\(pij,δyj\)D\_\{\\mathrm\{TV\}\}\(p\_\{i\}^\{j\},\\delta\_\{y^\{j\}\}\), whereδyj\\delta\_\{y^\{j\}\}denotes the one\-hot distribution onyjy^\{j\}\.

#### Main Bound\.

The early\-stage prediction error can be bounded by adjacent\-stage distributional drift and the reveal\-boundary token error:

DTV​\(pij,δyj\)≤∑r=irj−212DKL\(pr\+1j∥prj\)−log⁡prj−1j​\(yj\)\.\\begin\{split\}D\_\{\\mathrm\{TV\}\}\(p\_\{i\}^\{j\},\\delta\_\{y^\{j\}\}\)&\\leq\\sum\_\{r=i\}^\{r\_\{j\}\-2\}\\sqrt\{\\frac\{1\}\{2\}D\_\{\\mathrm\{KL\}\}\(p\_\{r\+1\}^\{j\}\\\|p\_\{r\}^\{j\}\)\}\\\\ &\\quad\-\\log p\_\{r\_\{j\}\-1\}^\{j\}\(y^\{j\}\)\.\\end\{split\}\(14\)The first term measures how much the model prediction drifts as more context is revealed before positionjjis committed, while the second term measures the token\-level error immediately before reveal, which is controlled by the CE loss on the clean token under commitment\.

#### Connection to CForce\.

Eq\. \([14](https://arxiv.org/html/2608.13925#S4.E14)\) directly matches our objective: the forward KL component in CAD reduces adjacent\-stage drift, while the CE anchor controls the reveal\-boundary error\. The confidence\-weighted reverse KL in CAD is non\-negative and therefore preserves this forward KL alignment while sharpening predictions when the later\-stage target is confident\. Thus, Consistency Forcing optimizes a tractable surrogate for the upper bound on early\-stage prediction error\.

Diffusion typeEdit?ModelGSM8KMATH500MBPPHumanEvalScoreTPFAUPScoreTPFAUPScoreTPFAUPScoreTPFAUPPurediffusion×\\timesdUltra\-coding\-b3281\.52\*8\.40–35\.646\.72\*–37\.047\.29\*–35\.857\.23\*–d3LLM\-LLaDA73\.099\.11\*637\.65\*30\.365\.74107\.64\*40\.604\.2188\.3639\.635\.9596\.64d3LLM\-Dream81\.364\.94391\.3338\.21\*3\.9297\.5055\.60\*2\.96141\.41\*57\.10\*3\.20129\.48\*D2F\-LLaDA74\.392\.88213\.7628\.942\.6649\.0039\.002\.1352\.9640\.642\.6961\.98Blockdiffusion×\\timesLightningRL\-8B\-b3290\.305\.58492\.4063\.006\.28407\.5058\.3011\.10641\.6072\.606\.30450\.10LLaDA2\.0\-mini93\.252\.85266\.8281\.803\.46286\.7178\.693\.80305\.8483\.544\.28367\.52LLaDA2\.0\-mini\-CAP91\.744\.09374\.0381\.004\.85394\.1072\.135\.43400\.0073\.185\.89387\.79LLaDA2\.0\-mini\-CForce \(Ours\)91\.745\.36489\.0779\.206\.35509\.2373\.306\.00387\.8079\.277\.97625\.51✓\\checkmarkDMax92\.105\.48557\.0075\.405\.94507\.0079\.205\.86482\.0083\.507\.36637\.00LLaDA2\.1\-mini93\.565\.94666\.7185\.007\.44741\.4277\.757\.25636\.1685\.987\.11705\.47LLaDA2\.1\-mini\-CForce \(Ours\)92\.277\.63845\.0384\.8010\.15966\.3581\.9710\.071016\.1486\.598\.48870\.90

Table 1:Main Comparison by Diffusion Type and Edit Capability\.Each benchmark group reports Score, TPF, and AUP\. Bold and underline mark the best and second\-best metrics within each comparable block; stars mark the best pure\-diffusion results\.

## 5Experiments

### 5\.1Experimental Setup

#### Models\.

We evaluate CForce on two LLaDA variants: LLaDA2\.0\-mini for the non\-edit setting and LLaDA2\.1\-mini for the edit\-capable setting described in Section[4\.3](https://arxiv.org/html/2608.13925#S4.SS3.SSS0.Px3)\. Detailed implementation settings are provided in Appendix[B](https://arxiv.org/html/2608.13925#A2)\.

#### Training Data\.

We use queries from OpenMath\-Instruct\-2\([Toshniwal et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib38)\)and OpenCodeInstruct\([Ahmad et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib2)\)and decode responses using Algorithm[1](https://arxiv.org/html/2608.13925#alg1)\. For LLaDA2\.0, we use a confidence thresholdτ=0\.95\\tau=0\.95; for LLaDA2\.1, we useτ=0\.85\\tau=0\.85andτedit=0\.5\\tau\_\{\\mathrm\{edit\}\}=0\.5\. Responses exceeding 4096 tokens are discarded\.

#### Evaluation\.

We evaluate on four benchmarks that span mathematical reasoning and code generation: GSM8K\([Cobbe et al\. 2021](https://arxiv.org/html/2608.13925#bib.bib14)\), MATH500\([Lightman et al\. 2024](https://arxiv.org/html/2608.13925#bib.bib24)\), MBPP\([Austin et al\. 2021b](https://arxiv.org/html/2608.13925#bib.bib5)\), and HumanEval\([Chen et al\. 2021](https://arxiv.org/html/2608.13925#bib.bib9)\)\. Inference is performed with SGLang\([Zheng et al\. 2024](https://arxiv.org/html/2608.13925#bib.bib45)\)using a block size of 32 and a max generation length of 4096\. We report Score, TPF, and AUP\([Qian et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib33)\)\. Score denotes exact\-match accuracy on GSM8K and MATH500 and pass@1 on MBPP and HumanEval; higher TPF and AUP indicate greater parallelism and a better speed\-quality operating point, respectively\. Appendix[D](https://arxiv.org/html/2608.13925#A4)provides the exact operating points used to compute AUP for the LLaDA\-family rows\. Throughput results on GSM8K and HumanEval, together with the hardware setup, are reported in Appendix[E](https://arxiv.org/html/2608.13925#A5)\.

#### Baselines\.

We include LLaDA2\.0\-mini and LLaDA2\.1\-mini as the base models for the non\-edit and edit\-capable settings, respectively\([Bie et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib7);[Bie et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib6)\)\. For the LLaDA2\.0\-series comparison, we also include LLaDA2\.0\-mini\-CAP, a confidence\-aware parallel training variant of LLaDA2\.0\-mini, and refer to it as CAP\([Chen et al\. 2025b](https://arxiv.org/html/2608.13925#bib.bib11)\)\. Beyond these controlled baselines, Table[1](https://arxiv.org/html/2608.13925#S4.T1)reports representative recent dLLM acceleration methods: dUltra\-coding\-b32, d3LLM\-LLaDA, d3LLM\-Dream, D2F\-LLaDA, LightningRL\-8B\-b32, and DMax\([Chen et al\. 2025a](https://arxiv.org/html/2608.13925#bib.bib10);[Qian et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib33);[Wang et al\. 2025](https://arxiv.org/html/2608.13925#bib.bib39);[Hu et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib19);[Chen et al\. 2026](https://arxiv.org/html/2608.13925#bib.bib12)\)\. For LLaDA2\.0\-mini, CAP, and our LLaDA2\.0\-mini\-CForce, evaluation usesτ=0\.9\\tau=0\.9; for LLaDA2\.1\-mini and LLaDA2\.1\-mini\-CForce, evaluation usesτ=0\.7\\tau=0\.7andτedit=0\.5\\tau\_\{\\mathrm\{edit\}\}=0\.5\.

### 5\.2Main Results

#### Edit\-capable dLLMs\.

Table[1](https://arxiv.org/html/2608.13925#S4.T1)reports controlled comparisons between each LLaDA baseline and its Consistency Forcing variant under the corresponding decoding setting, with the strongest gains appearing in the edit\-capable case\. On LLaDA2\.1\-mini, Consistency Forcing increases average TPF from 6\.94 to 9\.08 while also improving average score from 85\.57 to 86\.41\. The improvement is strongest on the code benchmarks and comes with only minor changes on the math benchmarks\. This result is consistent with the method design: edit\-capable trajectories allow later corrections on already decoded tokens to feed back into the masked\-position drafting ability of earlier states through the stop\-gradient target\.

#### Non\-edit dLLMs\.

Under the non\-edit decoding setting, Consistency Forcing on LLaDA2\.0\-mini moves the model to a higher\-parallelism operating point: it reaches 6\.42 average TPF, compared with 3\.60 for the base model and 5\.07 for CAP\. Its average score is lower than the base model but higher than CAP, making this setting an explicit speed\-quality trade\-off\.

Figure 5:Few\-step Generation Results \(Score %\) on LLaDA2\.0 Series with Fixed TPF Budgets\.Vanilla, CAP, and Ours represent LLaDA2\.0\-mini, LLaDA2\.0\-mini\-CAP, and LLaDA2\.0\-mini\-CForce respectively\.Stage sizeSSAVG ScoreAVG TPF480\.625\.64880\.886\.421675\.057\.97
Table 2:Ablation on Stage Size for LLaDA2\.0\-mini\-CForce\.Full benchmark\-level results are in Appendix[F](https://arxiv.org/html/2608.13925#A6)\. The shaded row denotes the default setting; bold marks the best value in each metric\.TrainingAVGCurriculumFrozenTeacherScoreTPF✓\\checkmark80\.886\.42✓\\checkmark✓\\checkmark79\.495\.9179\.875\.97
Table 3:Ablation on Curriculum Learning and Target Type for LLaDA2\.0\-mini\-CForce\.The frozen\-teacher variant changes only the target predictor; the no\-curriculum variant removes the gradual exposure schedule from the same\-model stop\-gradient target\.
#### Few\-step Generation\.

Figure[5](https://arxiv.org/html/2608.13925#S5.F5)evaluates fixed TPF decoding on the LLaDA2\.0 series\. We constrain the decoding budget so that the model operates at a fixed TPF of 8 or 4, which forces substantially more tokens to be committed per forward pass than in the standard dynamic\-threshold setting\. Consistency Forcing obtains the best average score at both budgets, with the largest margin under the more aggressive TPF=8=8setting\. This supports the main hypothesis that adjacent\-stage forcing is most useful when early predictions must directly support large parallel commitments\.

### 5\.3Ablation Studies

We conduct ablations on LLaDA2\.0\-mini\-CForce, where the confidence\-threshold results expose a clear speed\-quality trade\-off\. We vary the stage size, curriculum schedule, KL divergence form, CE anchor weight, and target type while keeping the rest of the training setup fixed\. All rows are evaluated with threshold decoding atτ=0\.9\\tau=0\.9and maximum generation length 4096\.

#### Stage Size\.

Table[3](https://arxiv.org/html/2608.13925#S5.T3)studies the number of newly revealed tokens between saved trajectory stages\. A small stage size \(S=4S=4\) yields nearby state pairs and lower parallelism, reaching 80\.62 average score and 5\.64 average TPF\. Increasing the stage size to 16 raises average TPF to 7\.97 but lowers average score to 75\.05, consistent with larger transitions being harder consistency targets\. We therefore useS=8S=8, which provides the best observed balance in this ablation, with 80\.88 average score and 6\.42 average TPF\.

#### Curriculum Learning\.

Table[3](https://arxiv.org/html/2608.13925#S5.T3)isolates the curriculum schedule \(Eq\.[11](https://arxiv.org/html/2608.13925#S4.E11)\), which gradually increases the fraction of newly revealed positions exposed by the later trajectory state from 10% to 100% over training\. Removing the schedule lowers average score from 80\.88 to 79\.87 and average TPF from 6\.42 to 5\.97, confirming that gradual exposure to larger context gaps stabilizes learning\. This suggests that the model benefits from first learning local adjacent\-stage consistency before being exposed to the full transition gap\.

#### Target Type\.

Table[3](https://arxiv.org/html/2608.13925#S5.T3)also compares the same\-model stop\-gradient target used by Consistency Forcing with a frozen\-teacher variant\. The same\-model target outperforms the frozen\-teacher baseline on both score \(\+1\.39\) and TPF \(\+0\.51\), suggesting that an evolving target better tracks the student’s own decoding distribution and provides more aligned supervision for its inference states\.

ComponentVariantAVGScoreTPFKLdivergenceForward KL85\.755\.07Reverse KL65\.578\.72CAD80\.886\.42CEanchorλCE=0\\lambda\_\{\\mathrm\{CE\}\}=079\.026\.52λCE=0\.1\\lambda\_\{\\mathrm\{CE\}\}=0\.180\.886\.42λCE=2\.0\\lambda\_\{\\mathrm\{CE\}\}=2\.080\.226\.52Table 4:Ablations of the KL Divergence Form and CE Anchor Weight on LLaDA2\.0\-mini\-CForce\.
#### KL Divergence Form\.

Table[4](https://arxiv.org/html/2608.13925#S5.T4)compares forward KL, reverse KL, and our CAD\. Forward KL gives the highest average score, but it remains conservative in parallelism, with 5\.07 average TPF\. Reverse KL reaches much higher parallelism, 8\.72 average TPF, but its average score drops to 65\.57, consistent with an overly aggressive mode\-seeking objective under threshold decoding\. CAD selects an intermediate operating point: compared with forward KL, it increases average TPF from 5\.07 to 6\.42; compared with reverse KL, it avoids the large score collapse\. CAD thus balances conservative drift control with controlled sharpening\.

#### CE Anchor\.

The lower block of Table[4](https://arxiv.org/html/2608.13925#S5.T4)studies the CE anchor weight\. Removing the anchor lowers average score from 80\.88 to 79\.02 while leaving TPF nearly unchanged, suggesting that the CE term improves the score side of the trade\-off\. A larger weight,λCE=2\.0\\lambda\_\{\\mathrm\{CE\}\}=2\.0, recovers part of the score but remains below the moderate setting on average and drops noticeably on HumanEval\. We therefore useλCE=0\.1\\lambda\_\{\\mathrm\{CE\}\}=0\.1as a modest token\-level anchor: it improves average score while preserving the adjacent\-stage KL\-driven parallelism\.

## 6Conclusion

We presented Consistency Forcing, a training method for improving the speed\-quality frontier of diffusion language models under aggressive threshold decoding\. The method constructs staged trajectories from the model’s own decoding path and aligns earlier low\-context states with later, more informative states through same\-model stop\-gradient supervision\. By combining Confidence Adaptive KL Divergence, a CE anchor, and a curriculum over reveal difficulty, Consistency Forcing improves the reliability of early\-stage predictions and supports higher parallelism\. We further extended the framework to edit\-capable dLLMs, where later T2T refinements provide useful supervision for earlier M2T predictions\. Experiments on both edit\-capable and conventional dLLMs show that this trajectory\-based forcing improves high\-parallelism decoding while preserving the generation quality\.

## Limitations

For training stability, Consistency Forcing currently relies on trajectories collected in advance from a pretrained initialization\. Although these trajectories are produced by the model’s own threshold\-decoding process, they are still fixed before optimization begins\. As training changes the model, the cached trajectories may gradually differ from the actual inference trajectories visited by the updated model\. This remaining mismatch can limit how closely the training signal follows the final student distribution\. In future work, we plan to explore online trajectory collection and policy\-updating strategies so that the forcing objective can adapt to the model’s evolving inference behavior\.

## Acknowledgments

This work was supported by Ant Group Research Fund\.

## References

- Agrawal et al\. \(2025\)Sudhanshu Agrawal, Risheek Garrepalli, Raghavv Goel, Mingu Lee, Christopher Lott, and Fatih Porikli\. 2025\.Spiffy: Multiplying diffusion llm acceleration via lossless speculative decoding\.*arXiv preprint arXiv:2509\.18085*\.
- Ahmad et al\. \(2025\)Wasi Uddin Ahmad, Aleksander Ficek, Mehrzad Samadi, Jocelyn Huang, Vahid Noroozi, Somshubra Majumdar, and Boris Ginsburg\. 2025\.Opencodeinstruct: A large\-scale instruction tuning dataset for code llms\.*arXiv preprint arXiv:2504\.04030*\.
- Arriola et al\. \(2025\)Marianne Arriola, Aaron Gokaslan, Justin Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sahoo, and Volodymyr Kuleshov\. 2025\.Block diffusion: Interpolating between autoregressive and diffusion language models\.In*International Conference on Learning Representations*, volume 2025, pages 50726–50753\.
- Austin et al\. \(2021a\)Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg\. 2021a\.Structured denoising diffusion models in discrete state\-spaces\.*Advances in neural information processing systems*, 34:17981–17993\.
- Austin et al\. \(2021b\)Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others\. 2021b\.Program synthesis with large language models\.*arXiv preprint arXiv:2108\.07732*\.
- Bie et al\. \(2026\)Tiwei Bie, Maosong Cao, Xiang Cao, Bingsen Chen, Fuyuan Chen, Kun Chen, Lun Du, Daozhuo Feng, Haibo Feng, Mingliang Gong, and 1 others\. 2026\.Llada2\. 1: Speeding up text diffusion via token editing\.*arXiv preprint arXiv:2602\.08676*\.
- Bie et al\. \(2025\)Tiwei Bie, Maosong Cao, Kun Chen, Lun Du, Mingliang Gong, Zhuochen Gong, Yanmei Gu, Jiaqi Hu, Zenan Huang, Zhenzhong Lan, and 1 others\. 2025\.Llada2\. 0: Scaling up diffusion language models to 100b\.*arXiv preprint arXiv:2512\.15745*\.
- Chang et al\. \(2022\)Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman\. 2022\.Maskgit: Masked generative image transformer\.In*Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 11315–11325\.
- Chen et al\. \(2021\)Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others\. 2021\.Evaluating large language models trained on code\.*arXiv preprint arXiv:2107\.03374*\.
- Chen et al\. \(2025a\)Shirui Chen, Jiantao Jiao, Lillian J Ratliff, and Banghua Zhu\. 2025a\.dultra: Ultra\-fast diffusion language models via reinforcement learning\.*arXiv preprint arXiv:2512\.21446*\.
- Chen et al\. \(2025b\)Zigeng Chen, Gongfan Fang, Xinyin Ma, Ruonan Yu, and Xinchao Wang\. 2025b\.dparallel: Learnable parallel decoding for dllms\.*arXiv preprint arXiv:2509\.26488*\.
- Chen et al\. \(2026\)Zigeng Chen, Gongfan Fang, Xinyin Ma, Ruonan Yu, and Xinchao Wang\. 2026\.Dmax: Aggressive parallel decoding for dllms\.*arXiv preprint arXiv:2604\.08302*\.
- Cheng et al\. \(2025\)Shuang Cheng, Yihan Bian, Dawei Liu, Linfeng Zhang, Qian Yao, Zhongbo Tian, Wenhai Wang, Qipeng Guo, Kai Chen, Biqing Qi, and 1 others\. 2025\.Sdar: A synergistic diffusion\-autoregression paradigm for scalable sequence generation\.*arXiv preprint arXiv:2510\.06303*\.
- Cobbe et al\. \(2021\)Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others\. 2021\.Training verifiers to solve math word problems\.*arXiv preprint arXiv:2110\.14168*\.
- Gao et al\. \(2025\)Yifeng Gao, Ziang Ji, Yuxuan Wang, Biqing Qi, Hanlin Xu, and Linfeng Zhang\. 2025\.Self speculative decoding for diffusion large language models\.*arXiv preprint arXiv:2510\.04147*\.
- Ghazvininejad et al\. \(2019\)Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer\. 2019\.Mask\-predict: Parallel decoding of conditional masked language models\.In*Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing \(EMNLP\-IJCNLP\)*, pages 6112–6121\.
- Gong et al\. \(2025\)Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang\. 2025\.Diffucoder: Understanding and improving masked diffusion models for code generation\.*arXiv preprint arXiv:2506\.20639*\.
- Hayakawa et al\. \(2024\)Satoshi Hayakawa, Yuhta Takida, Masaaki Imaizumi, Hiromi Wakaki, and Yuki Mitsufuji\. 2024\.Distillation of discrete diffusion through dimensional correlations\.*arXiv preprint arXiv:2410\.08709*\.
- Hu et al\. \(2026\)Yanzhe Hu, Yijie Jin, Pengfei Liu, Kai Yu, and Zhijie Deng\. 2026\.Lightningrl: Breaking the accuracy\-parallelism trade\-off of block\-wise dllms via reinforcement learning\.*arXiv preprint arXiv:2603\.13319*\.
- Kim et al\. \(2025\)Minseo Kim, Chenfeng Xu, Coleman Hooper, Harman Singh, Ben Athiwaratkun, Ce Zhang, Kurt Keutzer, and Amir Gholami\. 2025\.Cdlm: Consistency diffusion language models for faster sampling\.*arXiv preprint arXiv:2511\.19269*\.
- Kim et al\. \(2026\)Seo Hyun Kim, Sunwoo Hong, Hojung Jung, Youngrok Park, and Se\-Young Yun\. 2026\.Klass: Kl\-guided fast inference in masked diffusion models\.*Advances in Neural Information Processing Systems*, 38:92267–92301\.
- Kou et al\. \(2024\)Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang\. 2024\.Cllms: Consistency large language models\.In*Forty\-first International Conference on Machine Learning*\.
- Liang et al\. \(2026\)Yihao Liang, Ze Wang, Hao Chen, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Emad Barsoum, Zicheng Liu, and Niraj K Jha\. 2026\.Cd4lm: Consistency distillation and adaptive decoding for diffusion language models\.*arXiv preprint arXiv:2601\.02236*\.
- Lightman et al\. \(2024\)Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe\. 2024\.Let’s verify step by step\.In*International Conference on Learning Representations*, volume 2024, pages 39578–39601\.
- Liu et al\. \(2025a\)Yunpeng Liu, Boxiao Liu, Yi Zhang, Xingzhong Hou, Guanglu Song, Yu Liu, and Haihang You\. 2025a\.See further when clear: Curriculum consistency model\.In*Proceedings of the Computer Vision and Pattern Recognition Conference*, pages 18103–18112\.
- Liu et al\. \(2025b\)Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang\. 2025b\.dllm\-cache: Accelerating diffusion large language models with adaptive caching\.*arXiv preprint arXiv:2506\.06295*\.
- Lou et al\. \(2024\)Aaron Lou, Chenlin Meng, and Stefano Ermon\. 2024\.Discrete diffusion modeling by estimating the ratios of the data distribution, 2024\.*URL https://arxiv\. org/abs/2310\.16834*\.
- Luhman and Luhman \(2021\)Eric Luhman and Troy Luhman\. 2021\.Knowledge distillation in iterative generative models for improved sampling speed\.*arXiv preprint arXiv:2101\.02388*\.
- Luo et al\. \(2023\)Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao\. 2023\.Latent consistency models: Synthesizing high\-resolution images with few\-step inference\.*arXiv preprint arXiv:2310\.04378*\.
- Ma et al\. \(2026\)Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang\. 2026\.dkv\-cache: The cache for diffusion language models\.*Advances in Neural Information Processing Systems*, 38:149009–149033\.
- Meng et al\. \(2023\)Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans\. 2023\.On distillation of guided diffusion models\.In*Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 14297–14306\.
- Nie et al\. \(2026\)Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji\-Rong Wen, and Chongxuan Li\. 2026\.Large language diffusion models\.*Advances in Neural Information Processing Systems*, 38:50608–50646\.
- Qian et al\. \(2026\)Yu\-Yang Qian, Junda Su, Lanxiang Hu, Peiyuan Zhang, Zhijie Deng, Peng Zhao, and Hao Zhang\. 2026\.d3llm: Ultra\-fast diffusion llm using pseudo\-trajectory distillation\.*arXiv preprint arXiv:2601\.07568*\.
- Sahoo et al\. \(2024\)Subham S Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov\. 2024\.Simple and effective masked diffusion language models\.*Advances in Neural Information Processing Systems*, 37:130136–130184\.
- Sahoo et al\. \(2025\)Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang, Justin Chiu, and Volodymyr Kuleshov\. 2025\.The diffusion duality\.*arXiv preprint arXiv:2506\.10892*\.
- Salimans and Ho \(2022\)Tim Salimans and Jonathan Ho\. 2022\.Progressive distillation for fast sampling of diffusion models\.*arXiv preprint arXiv:2202\.00512*\.
- Song et al\. \(2023\)Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever\. 2023\.Consistency models\.
- Toshniwal et al\. \(2025\)Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman\. 2025\.Openmathinstruct\-2: Accelerating ai for math with massive open\-source instruction data\.In*International Conference on Learning Representations*, volume 2025, pages 19243–19275\.
- Wang et al\. \(2025\)Xu Wang, Chenkai Xu, Yijie Jin, Jiachun Jin, Hao Zhang, and Zhijie Deng\. 2025\.Diffusion llms can do faster\-than\-ar inference via discrete diffusion forcing\.*arXiv preprint arXiv:2508\.09192*\.
- Wu et al\. \(2025\)Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie\. 2025\.Fast\-dllm: Training\-free acceleration of diffusion llm by enabling kv cache and parallel decoding\.*arXiv preprint arXiv:2505\.22618*\.
- Xu et al\. \(2025a\)Chenkai Xu, Yijie Jin, Jiajun Li, Yi Tu, Guoping Long, Dandan Tu, Mingcong Song, Hongjie Si, Tianqi Hou, Junchi Yan, and 1 others\. 2025a\.Lopa: Scaling dllm inference via lookahead parallel decoding\.*arXiv preprint arXiv:2512\.16229*\.
- Xu et al\. \(2025b\)Chenkai Xu, Xu Wang, Zhenyi Liao, Yishun Li, Tianqi Hou, and Zhijie Deng\. 2025b\.Unicms: A unified consistency model for efficient multimodal generation and understanding\.*arXiv preprint arXiv:2502\.05415*\.
- Ye et al\. \(2025\)Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong\. 2025\.Dream 7b: Diffusion large language models\.*arXiv preprint arXiv:2508\.15487*\.
- Zhang et al\. \(2026\)Tunyu Zhang, Xinxi Zhang, Ligong Han, Haizhou Shi, Xiaoxiao He, Zhuowei Li, Hao Wang, Kai Xu, Akash Srivastava, Vladimir Pavlovic, and 1 others\. 2026\.T3d: Few\-step diffusion language models via trajectory self\-distillation with direct discriminative optimization\.*arXiv preprint arXiv:2602\.12262*\.
- Zheng et al\. \(2024\)Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody H Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, and 1 others\. 2024\.Sglang: Efficient execution of structured language model programs\.*Advances in neural information processing systems*, 37:62557–62583\.
- Zheng et al\. \(2023\)Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong\. 2023\.A reparameterized discrete diffusion model for text generation\.*arXiv preprint arXiv:2302\.05737*\.

## Appendix AStaged Trajectory Construction Algorithm

Algorithm[1](https://arxiv.org/html/2608.13925#alg1)gives the complete procedure for constructing staged self\-rollout trajectories from native threshold\-decoding traces\.

Algorithm 1Staged Trajectory ConstructionPretrained dLLM

qθ0q\_\{\\theta\_\{0\}\}, prompt set

𝒳\\mathcal\{X\}, M2T threshold

τ\\tau, optional edit threshold

τedit\\tau\_\{\\mathrm\{edit\}\}, stage size

SS
Trajectory dataset

𝒟\\mathcal\{D\}
//Initialize dataset

𝒟←∅\\mathcal\{D\}\\leftarrow\\emptyset

foreach prompt

x∈𝒳x\\in\\mathcal\{X\}do

//Initialize native trajectory

Set

s0s\_\{0\}to the all\-mask initial state and save

s¯0←s0\\bar\{s\}\_\{0\}\\leftarrow s\_\{0\}
Set stage counter

n←0n\\leftarrow 0and reveal counter

R←0R\\leftarrow 0
foreach native decoding step

ttdo

Compute predictions with

qθ0\(⋅∣x,st,j\)q\_\{\\theta\_\{0\}\}\(\\cdot\\mid x,s\_\{t\},j\)
Select newly revealed positions

Δ​Mt\\Delta M\_\{t\}by Eq\. \([2](https://arxiv.org/html/2608.13925#S3.E2)\)

For edit\-capable decoding, record

EtE\_\{t\}by Eq\. \([5](https://arxiv.org/html/2608.13925#S3.E5)\)

Update the native state from

sts\_\{t\}to

st\+1s\_\{t\+1\}
Set

R←R\+\|Δ​Mt\|R\\leftarrow R\+\|\\Delta M\_\{t\}\|
if

RRreaches the stage size

SSthen

Save

s¯n\+1←st\+1\\bar\{s\}\_\{n\+1\}\\leftarrow s\_\{t\+1\}
Set

n←n\+1n\\leftarrow n\+1and

R←0R\\leftarrow 0
endif

endfor

//Finalize after full decoding

if

sTs\_\{T\}is not already savedthen

Save

s¯N←sT\\bar\{s\}\_\{N\}\\leftarrow s\_\{T\}
endif

Add the complete stage trajectory

𝒯¯=\(s¯0,…,s¯N\)\\bar\{\\mathcal\{T\}\}=\(\\bar\{s\}\_\{0\},\\ldots,\\bar\{s\}\_\{N\}\)to

𝒟\\mathcal\{D\}
endfor

return

𝒟\\mathcal\{D\}

## Appendix BImplementation Details

All models use full\-parameter fine\-tuning with the AdamW optimizer, a learning rate of1\.0×10−51\.0\\times 10^\{\-5\}, and a cosine learning rate schedule\. We set the stage size to 8, use a global batch size of 64, and train for 5 epochs\. The CE anchor loss weightλCE\\lambda\_\{\\mathrm\{CE\}\}is set to 0\.1, and the block size is 32\. During KL distillation, we retain the top\-20 tokens ranked by the target distributionptarjp\_\{\\mathrm\{tar\}\}^\{j\}and renormalize the retained probabilities before computing the KL terms\. This top\-kktruncation both reduces the computation cost of the softmax and KL terms and focuses the student on the semantically meaningful region of the distribution\. All training is conducted on 64 NVIDIA H800 GPUs\.

## Appendix CFull Derivation of the Theoretical Analysis

This appendix provides the full derivation of the bound used in Section[4\.4](https://arxiv.org/html/2608.13925#S4.SS4)\. Using the TV distance defined in Section[4\.4](https://arxiv.org/html/2608.13925#S4.SS4), consider a positionjjthat is revealed at stagerjr\_\{j\}, i\.e\.,j∈M¯rj−1j\\in\\bar\{M\}\_\{r\_\{j\}\-1\}andj∉M¯rjj\\notin\\bar\{M\}\_\{r\_\{j\}\}, with final tokenyj=s¯Njy^\{j\}=\\bar\{s\}\_\{N\}^\{j\}\. Letpij=qθ\(⋅∣x,s¯i,j\)p\_\{i\}^\{j\}=q\_\{\\theta\}\(\\cdot\\mid x,\\bar\{s\}\_\{i\},j\)denote the prediction at stageii\. For any earlier masked stagei<rji<r\_\{j\}, the triangle inequality gives

DTV​\(pij,δyj\)≤∑r=irj−2DTV​\(prj,pr\+1j\)\+DTV​\(prj−1j,δyj\),D\_\{\\mathrm\{TV\}\}\(p\_\{i\}^\{j\},\\delta\_\{y^\{j\}\}\)\\leq\\sum\_\{r=i\}^\{r\_\{j\}\-2\}D\_\{\\mathrm\{TV\}\}\(p\_\{r\}^\{j\},p\_\{r\+1\}^\{j\}\)\+D\_\{\\mathrm\{TV\}\}\(p\_\{r\_\{j\}\-1\}^\{j\},\\delta\_\{y^\{j\}\}\),\(15\)whereδyj\\delta\_\{y^\{j\}\}is the one\-hot distribution onyjy^\{j\}\. The first term accumulates distributional drift across adjacent stages, and the second term is the masked\-state error immediately before token reveal\.

Pinsker’s inequality bounds each adjacent\-stage TV term by forward KL:

DTV​\(prj,pr\+1j\)≤12DKL\(pr\+1j∥prj\)\.D\_\{\\mathrm\{TV\}\}\(p\_\{r\}^\{j\},p\_\{r\+1\}^\{j\}\)\\leq\\sqrt\{\\frac\{1\}\{2\}D\_\{\\mathrm\{KL\}\}\(p\_\{r\+1\}^\{j\}\\\|p\_\{r\}^\{j\}\)\}\.\(16\)For the reveal\-boundary term, we have

DTV​\(prj−1j,δyj\)=1−prj−1j​\(yj\)≤−log⁡prj−1j​\(yj\),\\begin\{split\}D\_\{\\mathrm\{TV\}\}\(p\_\{r\_\{j\}\-1\}^\{j\},\\delta\_\{y^\{j\}\}\)&=1\-p\_\{r\_\{j\}\-1\}^\{j\}\(y^\{j\}\)\\\\ &\\leq\-\\log p\_\{r\_\{j\}\-1\}^\{j\}\(y^\{j\}\),\\end\{split\}\(17\)using1−x≤−log⁡x1\-x\\leq\-\\log xforx∈\(0,1\]x\\in\(0,1\]\. Substituting Eqs\. \([16](https://arxiv.org/html/2608.13925#A3.E16)\) and \([17](https://arxiv.org/html/2608.13925#A3.E17)\) into Eq\. \([15](https://arxiv.org/html/2608.13925#A3.E15)\) yields

DTV​\(pij,δyj\)≤∑r=irj−212DKL\(pr\+1j∥prj\)−log⁡prj−1j​\(yj\)\.\\begin\{split\}D\_\{\\mathrm\{TV\}\}\(p\_\{i\}^\{j\},\\delta\_\{y^\{j\}\}\)&\\leq\\sum\_\{r=i\}^\{r\_\{j\}\-2\}\\sqrt\{\\frac\{1\}\{2\}D\_\{\\mathrm\{KL\}\}\(p\_\{r\+1\}^\{j\}\\\|p\_\{r\}^\{j\}\)\}\\\\ &\\quad\-\\log p\_\{r\_\{j\}\-1\}^\{j\}\(y^\{j\}\)\.\\end\{split\}\(18\)
This bound matches the structure of Consistency Forcing\. The forward KL component in Eq\. \([8](https://arxiv.org/html/2608.13925#S4.E8)\) directly targets the adjacent\-stage drift terms in Eq\. \([18](https://arxiv.org/html/2608.13925#A3.E18)\)\. The CE anchor in Eq\. \([9](https://arxiv.org/html/2608.13925#S4.E9)\) includes the reveal\-boundary term in Eq\. \([17](https://arxiv.org/html/2608.13925#A3.E17)\) and supplies token\-level supervision at commitment boundaries\. The confidence\-weighted reverse KL further encourages mode\-seeking predictions when the later\-stage target is reliable\. Sincecj≥0c\_\{j\}\\geq 0,

ℒCADj=DKL\(ptarj∥pstuj\)\+cjDKL\(pstuj∥ptarj\)≥DKL\(ptarj∥pstuj\),\\begin\{split\}\\mathcal\{L\}\_\{\\mathrm\{CAD\}\}^\{j\}&=D\_\{\\mathrm\{KL\}\}\(p\_\{\\mathrm\{tar\}\}^\{j\}\\\|p\_\{\\mathrm\{stu\}\}^\{j\}\)\+c\_\{j\}D\_\{\\mathrm\{KL\}\}\(p\_\{\\mathrm\{stu\}\}^\{j\}\\\|p\_\{\\mathrm\{tar\}\}^\{j\}\)\\\\ &\\geq D\_\{\\mathrm\{KL\}\}\(p\_\{\\mathrm\{tar\}\}^\{j\}\\\|p\_\{\\mathrm\{stu\}\}^\{j\}\),\\end\{split\}\(19\)so optimizingℒCAD\\mathcal\{L\}\_\{\\mathrm\{CAD\}\}preserves the forward KL alignment term while sharpening the student distribution toward the cleaner state’s dominant mode\. Therefore, Consistency Forcing reduces a tractable surrogate of the early\-stage prediction error bound by aligning adjacent\-stage distributions and anchoring masked\-token predictions along the trajectory\.

## Appendix DAUP Evaluation Details

ModelDecode configAVGGSM8KMATH500MBPPHumanEvalScoreTPFScoreTPFScoreTPFScoreTPFScoreTPFLLaDA2\.0\-miniτ=0\.95\\tau=0\.9586\.523\.0993\.782\.4284\.202\.9981\.503\.2186\.593\.74τ=0\.9\\tau=0\.984\.323\.6093\.252\.8581\.803\.4678\.693\.8083\.544\.28LLaDA2\.0\-mini\-CAPτ=0\.95\\tau=0\.9582\.704\.2692\.343\.4083\.004\.1576\.814\.5778\.664\.93τ=0\.9\\tau=0\.979\.515\.0791\.744\.0981\.004\.8572\.135\.4373\.185\.89LLaDA2\.0\-mini\-CForce \(Ours\)τ=0\.95\\tau=0\.9583\.905\.3292\.274\.3382\.605\.3479\.634\.8781\.106\.73τ=0\.9\\tau=0\.980\.886\.4291\.745\.3679\.206\.3573\.306\.0079\.277\.97LLaDA2\.1\-miniτ=0\.7\\tau=0\.7,τedit=0\.5\\tau\_\{\\mathrm\{edit\}\}=0\.585\.576\.9493\.565\.9485\.007\.4477\.757\.2585\.987\.11τ=0\.5\\tau=0\.5,τedit=0\.0\\tau\_\{\\mathrm\{edit\}\}=0\.085\.778\.1192\.957\.1586\.208\.7479\.168\.3084\.768\.26LLaDA2\.1\-mini\-CForce \(Ours\)τ=0\.7\\tau=0\.7,τedit=0\.5\\tau\_\{\\mathrm\{edit\}\}=0\.586\.419\.0892\.277\.6384\.8010\.1581\.9710\.0786\.598\.48τ=0\.5\\tau=0\.5,τedit=0\.0\\tau\_\{\\mathrm\{edit\}\}=0\.085\.0310\.8691\.519\.2684\.4011\.4778\.2212\.6185\.9810\.08

Table 5:Benchmark\-level Operating Points Used to Compute AUP for the LLaDA\-family Rows in Table[1](https://arxiv.org/html/2608.13925#S4.T1)\.Each benchmark group reports the Score/TPF pair used by the two\-point AUP frontier\. Bold and underline mark the best and second\-best metrics within each LLaDA\-series block\. All evaluations use maximum generation length 4096; LLaDA2\.0 models are M2T\-only and therefore do not use an edit threshold\.AUP is computed from a small score–parallelism frontier rather than from only the single Score/TPF pair shown in Table[1](https://arxiv.org/html/2608.13925#S4.T1)\. For each benchmark, an operating point is denoted by\(ai,yi\)\(a\_\{i\},y\_\{i\}\), whereaia\_\{i\}is TPF andyiy\_\{i\}is the task score in percentage points\. Following the public AUP implementation, we sort operating points by TPF and compute

AUP=a1​y1\+∑i=2mai−ai−12​\(yi​W​\(yi\)\+yi−1​W​\(yi−1\)\),\\mathrm\{AUP\}=a\_\{1\}y\_\{1\}\+\\sum\_\{i=2\}^\{m\}\\frac\{a\_\{i\}\-a\_\{i\-1\}\}\{2\}\\left\(y\_\{i\}W\(y\_\{i\}\)\+y\_\{i\-1\}W\(y\_\{i\-1\}\)\\right\),\(20\)whereW⁡\(y\)=min⁡\(exp⁡\(−3​\(1−y/ymax\)\),1\)W\(y\)=\\min\(\\exp\(\-3\(1\-y/y\_\{\\max\}\)\),1\)andymaxy\_\{\\max\}is the maximum score for that benchmark among the compared methods\. We keep operating points within five score points of the first, lowest\-TPF point, matching the default AUP filtering threshold\. This weighting penalizes operating points that gain parallelism by substantially degrading accuracy\.

Table[5](https://arxiv.org/html/2608.13925#A4.T5)lists the exact decoding configurations and benchmark\-level Score/TPF pairs used to construct the LLaDA\-family AUP frontiers in Table[1](https://arxiv.org/html/2608.13925#S4.T1)\. For the LLaDA2\.0 series, including LLaDA2\.0\-mini, LLaDA2\.0\-mini\-CAP, and LLaDA2\.0\-mini\-CForce, we evaluate two threshold\-decoding points,τ=0\.95\\tau=0\.95andτ=0\.9\\tau=0\.9\. For the edit\-capable LLaDA2\.1 series, including LLaDA2\.1\-mini and LLaDA2\.1\-mini\-CForce, we use the official quality and speed modes: quality mode setsτ=0\.7\\tau=0\.7andτedit=0\.5\\tau\_\{\\mathrm\{edit\}\}=0\.5, while speed mode setsτ=0\.5\\tau=0\.5andτedit=0\.0\\tau\_\{\\mathrm\{edit\}\}=0\.0\. The Score and TPF columns in Table[1](https://arxiv.org/html/2608.13925#S4.T1)report the main comparison operating point, while the AUP column is computed from the corresponding two\-point frontier\.

## Appendix EWall\-clock Throughput

We report wall\-clock throughput on two benchmarks, GSM8K and HumanEval, in Table[6](https://arxiv.org/html/2608.13925#A5.T6)\. All measurements use two NVIDIA H20 GPUs with tensor parallelismTP=2\\mathrm\{TP\}=2and batch size 1\. TPS denotes generated tokens per second\.

ModelGSM8K TPSHumanEval TPSAVG TPSNon\-edit dLLMsLLaDA2\.0\-mini489\.87711\.59600\.73LLaDA2\.0\-mini\-CAP710\.88891\.94801\.41LLaDA2\.0\-mini\-CForce \(Ours\)783\.09932\.82857\.96Edit\-capable dLLMsLLaDA2\.1\-mini987\.781116\.101051\.94LLaDA2\.1\-mini\-CForce \(Ours\)1231\.521352\.861292\.19

Table 6:Wall\-clock Throughput on GSM8K and HumanEval\.Bold marks the best throughput within each edit\-capability group\.In the non\-edit LLaDA2\.0 setting, CForce reaches 857\.96 average TPS, improving over the base model by 42\.82% and over CAP by 7\.06%\. In the edit\-capable LLaDA2\.1 setting, CForce reaches 1292\.19 average TPS, improving over the base model by 22\.84%\.

## Appendix FFull Ablation Results

Tables[7](https://arxiv.org/html/2608.13925#A6.T7),[8](https://arxiv.org/html/2608.13925#A6.T8), and[9](https://arxiv.org/html/2608.13925#A6.T9)provide the benchmark\-level results corresponding to the compact ablation tables in Section[5\.3](https://arxiv.org/html/2608.13925#S5.SS3)\. Each benchmark group reports Score and TPF under threshold decoding atτ=0\.9\\tau=0\.9and maximum generation length 4096\.

Stage sizeSSAVGGSM8KMATH500MBPPHumanEvalScoreTPFScoreTPFScoreTPFScoreTPFScoreTPF480\.625\.6491\.514\.5679\.805\.6474\.945\.2276\.227\.14880\.886\.4291\.745\.3679\.206\.3573\.306\.0079\.277\.971675\.057\.9790\.307\.6175\.209\.8662\.766\.7171\.957\.70

Table 7:Full benchmark\-level ablation on stage size for LLaDA2\.0\-mini\-CForce\.Each benchmark group reports Score and TPF\.CurriculumTarget typeAVGGSM8KMATH500MBPPHumanEvalScoreTPFScoreTPFScoreTPFScoreTPFScoreTPF✓\\checkmarkSame\-model target80\.886\.4291\.745\.3679\.206\.3573\.306\.0079\.277\.97✓\\checkmarkFrozen teacher79\.495\.9191\.214\.6779\.205\.5770\.735\.6376\.837\.75×\\timesSame\-model target79\.875\.9791\.664\.8977\.405\.6772\.375\.3078\.058\.00

Table 8:Full benchmark\-level ablation on curriculum learning and target type for LLaDA2\.0\-mini\-CForce\.The frozen\-teacher variant changes only the later\-state target predictor, while the no\-curriculum row keeps the same\-model stop\-gradient target and removes the gradual exposure schedule\. Each benchmark group reports Score and TPF\.ComponentVariantAVGGSM8KMATH500MBPPHumanEvalScoreTPFScoreTPFScoreTPFScoreTPFScoreTPFKL divergenceForward KL85\.755\.0792\.954\.0082\.804\.7281\.264\.7485\.986\.81Reverse KL65\.578\.7287\.647\.0270\.208\.1551\.998\.7852\.4410\.92CAD80\.886\.4291\.745\.3679\.206\.3573\.306\.0079\.277\.97CE anchorλCE=0\\lambda\_\{\\mathrm\{CE\}\}=079\.026\.5291\.585\.3478\.806\.3668\.856\.0676\.838\.32λCE=0\.1\\lambda\_\{\\mathrm\{CE\}\}=0\.180\.886\.4291\.745\.3679\.206\.3573\.306\.0079\.277\.97λCE=2\.0\\lambda\_\{\\mathrm\{CE\}\}=2\.080\.226\.5292\.495\.3478\.606\.0675\.416\.3474\.398\.35

Table 9:Full benchmark\-level ablations of the KL Divergence Form and CE Anchor Weight on LLaDA2\.0\-mini\-CForce\.Each benchmark group reports Score and TPF\.
## Appendix GCase Study

We present qualitative examples comparing the generation quality of LLaDA2\.0\-mini\-CForce \(Ours\), LLaDA2\.0\-mini\-CAP, and the LLaDA2\.0\-mini baseline\. All outputs are generated with a fixed TPF of 8, i\.e\., each forward pass produces exactly 8 tokens\. These examples illustrate that CForce produces more coherent and accurate outputs under few\-step generation compared to CAP and the baseline\.

### G\.1Example 1 \(Reference: 140\)

Prompt:

Question:Andyplants90geraniumsand40fewerpetuniasthatgeraniums\.Howmanyflowersdoesheplanttotal?

Pleasereasonstepbystep,andputyourfinalanswerwithin\\boxed\{\}\.

LLaDA2\.0\-mini\-CForce \(Ours\):

Let’ssolvetheproblemstepbystep:

1\.\*\*Numberofgeraniums\*\*:

Andyplants90geraniums\.

2\.\*\*Numberofpetunias\*\*:

Heplants40fewerpetuniasthangeraniums\.

So,numberofpetunias=90\-40=50\.

3\.\*\*Totalnumberofflowers\*\*:

Addgergeraniumandgerpetunias:

Total=geraniums\+petunias=90\+50=140\.

Finalanswer:

\\boxed\{140\}

LLaDA2\.0\-mini\-CAP:

Let’sbreakthisdownstepbystep:

1\.\*\*Identifythenumberofgeraniumsplanted:\*\*

Andyplants\*\*90geraniums\*\*\.

2\.\*\*Determinethenumberofpetuniasplanted:\*\*

Heplants\*\*40fewerpetuniasthangeraniums\*\*\.

So,numberofpetunias=90geraniumaniums\)0=\*\*50\*\*\.

3\.\*\*Calculatethetotalnumberofflowers:\*\*

Totalflowers=geraniums\+petunias

=900\+50=

\*\*120

\\boxed\{Rightarrow\{1\}0\}

LLaDA2\.0\-mini \(Baseline\):

Let’ssolvethisproblemstepbystep\.

1\.\*\*Identifythenumberofgeraniumsplants\*\*:

Andyplants\*\*90geraniums\*\*\.

2\.\*\*Determinethenumberofpetuniasheplants\*\*:

Heplants\*\*40fewerpetununiasthangeraniums\*\*\.

So,numberofpetunias=

$$

90\-040=50

$$

3\.\*\*Calculatethetotalnumberofflowers\*\*:

Addflowersnumberofgeraniumsandpetunias

$$

90\+50=140

$$

Wait\-\-thetotalwritten\.:

\\$$

\\boxed\-50=130\\\\

$$

No\-\-let’sre\-check\.

\*\*90\-40=50\*\*,correct\.5

iastotal\.

$$

\\\{90\}=\\1100

$$

No\-\-restart\.

Correct:

$$

90\+50=140

$$

Correcting\\boxed\{140\}

AfterFinalstep:

\\boxed$$

140

$$

### G\.2Example 2 \(Reference: 160\)

Prompt:

Question:Tarabought8packsof5canvasbagsfor$4each\.Shepaintedthemandsoldthematacraftfairfor$8each\.Howmuchprofitdidsheearnonherbags?

Pleasereasonstepbystep,andputyourfinalanswerwithin\\boxed\{\}\.

LLaDA2\.0\-mini\-CForce \(Ours\):

Let’ssolvethisstepbystep\.

\*\*Step1:DeterminethetotalnumberofcanvasbagsTarabought\.\*\*

Tarabought8packs,andeachpackcontains5canvasbags:

$$

8\\text\{packs\}\\times5\\text\{bagsperpack\}=40\\text\{canvasbags\}

$$

\*\*Step2:Calculatethetotalcostofthecanvasbags\.\*\*

Eachcanvasbagcosts$4:

$$

40\\text\{bags\}\\times\\$4=\\$160

$$

\*\*Step3:Calculatethetotalrevenuefromsellingthebags\.\*\*

Shesoldeachbagfor$8:

$$

40\\text\{bags\}\\times\\$8=\\$320

$$

\*\*Step4:Calculatetheprofit\.\*\*

Profit=Revenue\-TotalCost

$$

\\$320\-\\$160=\\$160

$$

\*\*FinalAnswer:\*\*

$$

\\boxed\{160\}

$$

Taraearnedaprofitof\*\*$160\*\*\.

LLaDA2\.0\-mini\-CAP:

Let’ssolvethisstepstepbystep:

\*\*Step1:CalculatethetotalcostTarathebags\.\*\*

boughtbought8packsof5canvasbags,

pack\.Eachpackpackcosts$4\.

So,totalnumberofbagsbags=

8packsx\{5bagsperpack\}=\{4\{text\{bags\}

$$

\*\*Totalcost\*\*=

8x\{44perpack\}=\\\{32\}

$$

\-\-\-

\*\*Step2:Calculatethemuchrevenuesellingthethebagsbags

$$

Shesoldeach40bagsfor$8each\.

Total,fromallbags=

40bagsx\{$8perbag\}\\\\\{320\}

$$

\-\-\-

\*\*Step3:Calculatetheprofit

$$ProfitProfit=TotalRevenue\-Totalcost

\\\{320\-\\$32\}=\\\{288\}

\-\-\-

Wait\-\-correction:

$$

$$

boxedboxed2

LLaDA2\.0\-mini \(Baseline\):

Let’sbreakthisdownstepbystep:

\*\*Step1:CalculatehowtotalcanvasbagsTarabought\.\*\*

Tboughtbought8packs,eachof5canvasbags\.

So,

Total\\text\{5\}\-canvasbags\}=8xpacks5==text40bags\.

\*\*Step2:CalculatehowTaracostforboughtoncanvasbags\.\*\*

Eachpack$44,andshebought8\.\.

So,

Cost=8x$4=\\text3$2\.

\*\*Step3CalculatehowmuchTaraearnedfromsellingthebags\.\*\*

Shesoldeachbagbagforforfor88,andhad40bags\.

\.

So,totalRevenue=40x$8=\\text320\.

\*\*Step4:profitearnedvs\.\*\*

Profit=Totalrevenue\-TotalCost\.

Profit=$320\-32=\\text288\.\.

Wait\-\-let’scorrectandrecalvaluesaccurately\.

### G\.3Example 3

Prompt:

YouareanexpertPythonprogrammer,andhereisyourtask:Writeafunctiontofindthemedianofthreenumbers\.Yourcodeshouldpassthesetests:

assertmedian\_numbers\(25,55,65\)==55\.0

assertmedian\_numbers\(20,10,30\)==20\.0

assertmedian\_numbers\(15,45,75\)==45\.0

Pleaseencloseyourcodewithindelimitersasfollows:

‘‘‘python

\#YOURCODEHERE

‘‘‘

LLaDA2\.0\-mini\-CForce \(Ours\):

‘‘‘python

defmedian\_numbers\(a,b,c\):

"""

Findthemedianofthreenumbers\.

Args:

a,b,c:threenumbers

Returns:

The:ofafloat

"""

\#Sortathreenumberslistandreturnthemedian

sorted\_numbers=sorted\(\[a,b,c\]\)

returnfloat\(sorted\_numbers\[1\]\)

‘‘‘

LLaDA2\.0\-mini\-CAP:

‘‘‘python

defmedian\_numbers\(a,,,c\):

"""

Findthemedianofthreenumbers\.

Args:

a,b,c:Threenumbers

Returns:

Themedianvalueasanumbersorafloat

place

numbers=\[a,b,c\]

numbers\.sort\(\)

returnfloat\(numbers\[1\]\)

‘‘‘

LLaDA2\.0\-mini \(Baseline\):

‘‘‘python

\#median\_numbers\(a,b,c\):

"""

Findthemedianofthreenumbers\.

Args:

a,b,c:Threenumbers

Returns:

Themedianasathree\-digitfloat

float

\#Createathreenumbersnumbersandsort

numbers=\[\]\+\[b\]\+cc

numbers\.sort\(\)

\#Returnthemedian\(middlevalue\)

returnfloat\(numbers\[1\]\)

‘‘‘