Trace-Based On-Policy Distillation for Masked Diffusion Language Models

# Trace-Based On-Policy Distillation for Masked Diffusion Language Models
Source: [https://arxiv.org/html/2607.16872](https://arxiv.org/html/2607.16872)
Haolin Ren, Ziyang Huang∗, Chenhao Yuan, Jun Zhao, Kang Liu Institute of Automation, Chinese Academy of Sciences, Beijing, China University of Chinese Academy of Sciences, Beijing, China renhaolin2026@ia\.ac\.cn

###### Abstract

Diffusion large language models \(dLLMs\) are a promising alternative to autoregressive generation\. However, reasoning\-oriented post\-training for dLLMs remains challenging\. Supervised fine\-tuning \(SFT\) for dLLMs requires dense but often off\-policy masked states, while reinforcement learning \(RL\) relies on sparse rewards or value modeling\. This paper proposestrace\-based on\-policy distillation \(TOPD\), a teacher\-supervised framework that transfers reasoning ability to a target dLLM without reward estimation\. The key idea is to supervise a dLLM on its own denoising trajectory, focusing on the trace\-aligned token decisions that form the final response\. Specifically, TOPD samples on\-policy diffusion trajectories from the target dLLM, obtains teacher token distributions from a teacher model on the corresponding partially denoised states, and updates the target dLLM with a token\-level Reverse Kullback\-Leibler \(Reverse\-KL\) objective\. This design preserves dense teacher supervision while aligning training with the model’s own denoising states\. On mathematical reasoning benchmarks, TOPD enables SDAR\-4B\-Chat to match the MATH500 accuracy of its RL\-trained counterpart TraDo\-4B\-Instruct, with gains of \+5\.7 under static evaluation and \+4\.5 under dynamic evaluation\. Compared with the RL\-trained counterpart, TOPD achieves this with 4×\\timesfewer rollout rounds, corresponding to an estimated 96\.0×\\timesto\-accuracy model\-compute speedup\.

Trace\-Based On\-Policy Distillation for Masked Diffusion Language Models

Haolin Ren††thanks:Equal contribution\., Ziyang Huang∗, Chenhao Yuan, Jun Zhao, Kang LiuInstitute of Automation, Chinese Academy of Sciences, Beijing, ChinaUniversity of Chinese Academy of Sciences, Beijing, Chinarenhaolin2026@ia\.ac\.cn

## 1Introduction

![Refer to caption](https://arxiv.org/html/2607.16872v1/figs/diffopd_comparison.png)Figure 1:Conceptual comparison between trace\-based on\-policy distillation \(TOPD\) and random\-mask reinforcement learning for dLLMs\.QuestionA ticket costs $80\. It is discounted by 25%,then taxed by 10%\. What is the final price?Full answer80×\(1−25%\)=6080\\times\(1\-25\\%\)=6060×\(1\+10%\)=6660\\times\(1\+10\\%\)=66Final answer: $66On\-policy state80×\(1−25%\)=\_80\\times\(1\-25\\%\)=\\\_\_×\(1\+10%\)=\_\\\_\\times\(1\+10\\%\)=\\\_Final answer: $\_Preserves causal direction:subtotals are filled frompreceding computationsRandom\-mask state\_×\(1−25%\)=\_\\\_\\times\(1\-25\\%\)=\\\_60×\(1\+\_%\)=6660\\times\(1\+\\\_\\%\)=66Final answer: $\_Breaks causal reasoning:60 is supervised fromlater evidence

Figure 2:A synthetic question\-answering example illustrating trajectory mismatch\. Random masking can expose later results while hiding earlier reasoning variables, forcing backward reconstruction rather than forward computation\.Post\-training has driven large gains in autoregressive large language models \(LLMs\) on reasoning tasks\.\(Ouyanget al\.,[2022](https://arxiv.org/html/2607.16872#bib.bib7); Shaoet al\.,[2024](https://arxiv.org/html/2607.16872#bib.bib18); Guoet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib17); Chuet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib21)\)\. Diffusion large language models \(dLLMs\) offer a competitive non\-autoregressive alternative based on iterative denoising\(Liet al\.,[2022](https://arxiv.org/html/2607.16872#bib.bib6); Sahooet al\.,[2024](https://arxiv.org/html/2607.16872#bib.bib15); Nieet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib16); Chenget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib30); Arriolaet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib23); Gonget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib29); Yeet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib44)\)\. However, reasoning\-oriented post\-training for dLLMs remains less well studied, particularly for distilling reasoning ability from stronger dLLMs into weaker student dLLMs\. This paper studies how to efficiently adapt student dLLMs for reasoning under iterative denoising generation, using stronger dLLMs as supervision\.

Current dLLM post\-training pipelines mainly follow two routes: supervised fine\-tuning \(SFT\) and reinforcement learning \(RL\)\(Zhaoet al\.,[2025b](https://arxiv.org/html/2607.16872#bib.bib10); Wanget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib11); Zhuet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib41); Heet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib42); Ouet al\.,[2026](https://arxiv.org/html/2607.16872#bib.bib25); Wanget al\.,[2026](https://arxiv.org/html/2607.16872#bib.bib26); Zhaoet al\.,[2025a](https://arxiv.org/html/2607.16872#bib.bib27); Zhonget al\.,[2026](https://arxiv.org/html/2607.16872#bib.bib14)\)\. SFT trains the student on fixed targets from external sources, such as human annotations and pre\-generated samples, making the supervision inherently off\-policy\. In this way, the student is optimized on static targets or idealized teacher states rather than the states induced by its own generation policy, leading to exposure bias and weaker generalization\(Agarwalet al\.,[2024](https://arxiv.org/html/2607.16872#bib.bib19); Chuet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib21)\)\. RL, on the other hand, often relies on sparse, delayed, and high\-variance rewards, which makes long\-horizon credit assignment difficult and optimization costly\. These limitations motivate on\-policy distillation \(OPD\)\(Agarwalet al\.,[2024](https://arxiv.org/html/2607.16872#bib.bib19); Lu and Lab,[2025](https://arxiv.org/html/2607.16872#bib.bib13)\), where the student samples from its own policy while a teacher provides dense feedback on those student\-generated outputs\.

However, adapting OPD to dLLMs is not a direct transplant from autoregressive models\. In autoregressive LLMs, each action is a committed next\-token decision, so teacher feedback naturally attaches to the generated prefix\. In dLLMs, the model iteratively updates a partially masked sequence, making trajectory alignment central\. A key obstacle is the random\-mask mismatch in many recent dLLM RL pipelines: training states are often built by randomly corrupting subsets of model\-generated responses\(Zhaoet al\.,[2025b](https://arxiv.org/html/2607.16872#bib.bib10); Zhuet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib41); Ouet al\.,[2026](https://arxiv.org/html/2607.16872#bib.bib25); Wanget al\.,[2026](https://arxiv.org/html/2607.16872#bib.bib26); Zhonget al\.,[2026](https://arxiv.org/html/2607.16872#bib.bib14)\)\. As illustrated in Figure[2](https://arxiv.org/html/2607.16872#S1.F2), such states can expose downstream answers while hiding upstream reasoning variables, creating backward reconstruction contexts that differ from the student’s forward denoising trajectory\. Reveal order and inference schedules materially affect masked\-diffusion behavior\(Heet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib42); Wanget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib11)\)\. Teacher supervision on random or trajectory\-agnostic states therefore weakens alignment with the decisions that form the final response\.

To this end, this paper proposestrace\-based on\-policy distillation \(TOPD\), a teacher\-supervised post\-training framework for dLLMs\. TOPD samples the student’s own inference\-time denoising trajectories and converts them into dense distillation targets by identifying the token predictions that directly determine the final response\. This trace\-based formulation avoids supervising arbitrary random\-mask configurations, which may expose later answer tokens while hiding earlier reasoning tokens and thus create conditional dependencies inconsistent with inference\-time denoising\. TOPD then uses a frozen teacher to provide token distributions on the student’s on\-policy partially denoised states and optimizes the student with step\-wise token\-distribution supervision\. In this process, a token\-level Reverse Kullback\-Leibler \(Reverse\-KL\) is exploited\. As a result, TOPD preserves trajectory\-aligned state coverage while replacing sparse\-reward credit assignment and value\-model overhead with dense teacher feedback\. Figure[1](https://arxiv.org/html/2607.16872#S1.F1)provides a conceptual comparison between TOPD and random\-mask RL\.

Experiments on mathematical reasoning benchmarks show that TOPD can effectively transfer reasoning ability from a dLLM teacher to a smaller student with substantially lower training cost\. Specifically, using TraDo\-8B\-Instruct\(Wanget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib11)\)as the teacher, TOPD brings the base SDAR\-4B\-Chat\(Chenget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib30)\)student to the same MATH500\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.16872#bib.bib37)\)accuracy as TraDo\-4B\-Instruct, an RL\-trained version of SDAR\-4B\-Chat, with gains of \+5\.7 under static evaluation and \+4\.5 under dynamic evaluation\. Moreover, compared with TraceRL\(Wanget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib11)\), TOPD achieves this with 4×\\timesfewer rollout rounds, yielding an estimated 96\.0×\\timesto\-accuracy model\-compute speedup under our parameter\-scaled accounting\.

Our contributions are summarized as follows:

- •We formulate teacher\-supervised dLLM post\-training as a trace\-based on\-policy distillation problem, showing that random\-mask supervision can induce conditional dependencies misaligned with the student’s inference\-time denoising trajectory\.
- •This paper proposes TOPD, an efficient post\-training paradigm for dLLMs, which samples on\-policy diffusion trajectories from the student and identifies the trace\-aligned denoising decisions that directly determine the final response, replacing reward\-based updates with step\-wise teacher token\-distribution supervision\.

## 2Background

### 2\.1Masked Diffusion Language Models

Masked diffusion language models \(MDLMs\) generate text by iteratively denoising masked token sequences rather than by committing tokens from left to right\(Sahooet al\.,[2024](https://arxiv.org/html/2607.16872#bib.bib15); Nieet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib16)\)\. Given a clean sequencex∈𝒱nx\\in\\mathcal\{V\}^\{n\}, training corrupts tokens into\[MASK\]at noise levelttand learns a denoiserπθ\(⋅∣z\)\\pi\_\{\\theta\}\(\\cdot\\mid z\)for the masked positions, usually with a schedule\-weighted masked reconstruction objective\(Sahooet al\.,[2024](https://arxiv.org/html/2607.16872#bib.bib15); Shiet al\.,[2024](https://arxiv.org/html/2607.16872#bib.bib24)\)\. Full\-attention dLLMs allow response tokens to attend bidirectionally to the whole corrupted response, while block\-attention dLLMs partition the response into blocks and use a block\-causal mask, e\.g\., a token in blockkkattends only to blocks≤k\\leq k\(Arriolaet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib23); Chenget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib30)\)\. At inference time, however, the model follows its own denoising trajectoryτ=\(s0,a0,…,sT\)\\tau=\(s\_\{0\},a\_\{0\},\\ldots,s\_\{T\}\)from a fully masked response, and the resulting reveal order can affect generation behavior\(Heet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib42); Wanget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib11)\)\. Since intermediate decisions may later be revised or removed, we use*trace*to denote the subset of decisions inτ\\tauthat survives intosTs\_\{T\}\.

### 2\.2Post\-Training Signals for dLLMs

Post\-training for dLLMs typically relies on SFT or RL\(Zhaoet al\.,[2025b](https://arxiv.org/html/2607.16872#bib.bib10); Wanget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib11); Zhuet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib41); Heet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib42); Ouet al\.,[2026](https://arxiv.org/html/2607.16872#bib.bib25); Wanget al\.,[2026](https://arxiv.org/html/2607.16872#bib.bib26); Zhaoet al\.,[2025a](https://arxiv.org/html/2607.16872#bib.bib27); Zhonget al\.,[2026](https://arxiv.org/html/2607.16872#bib.bib14)\)\. In SFT, a target response is partially corrupted intoy~\\tilde\{y\}and the model is trained to reconstruct the masked positions:

ℒSFT​\(θ\)=𝔼q,y,y~​\[∑i∈ℳ​\(y~\)−log⁡πθ​\(yi∣q,y~\)\],\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}\(\\theta\)=\\mathbb\{E\}\_\{q,y,\\tilde\{y\}\}\\left\[\\sum\_\{i\\in\\mathcal\{M\}\(\\tilde\{y\}\)\}\-\\log\\pi\_\{\\theta\}\(y\_\{i\}\\mid q,\\tilde\{y\}\)\\right\],\(1\)whereℳ​\(y~\)\\mathcal\{M\}\(\\tilde\{y\}\)is the masked\-position set and the expectation overy~\\tilde\{y\}denotes the random masking process applied toyy\. RL instead samples from the current student and maximizes

JRL​\(θ\)=𝔼q,τ∼Pθ\(⋅∣q\)​\[R​\(q,sT\)\]\.J\_\{\\mathrm\{RL\}\}\(\\theta\)=\\mathbb\{E\}\_\{q,\\,\\tau\\sim P\_\{\\theta\}\(\\cdot\\mid q\)\}\\bigl\[R\(q,s\_\{T\}\)\\bigr\]\.\(2\)HerePθ​\(τ∣q\)P\_\{\\theta\}\(\\tau\\mid q\)denotes the trajectory distribution induced by the student’s iterative denoising policy, andR​\(q,sT\)R\(q,s\_\{T\}\)is the scalar reward assigned to the final response\. SFT provides dense supervision but trains on fixed target responses rather than states induced by the current student\. RL uses student samples, but its sparse and delayed rewards must be assigned across many non\-autoregressive denoising decisions, making credit assignment costly and high variance\. Teacher supervision offers a middle ground: dense token distributions on student\-visited states without delayed reward estimation\.

### 2\.3Challenges in Adapting OPD to dLLMs

OPD trains a student on states sampled from its current policy while a teacher supplies dense target distributions on those same states\(Agarwalet al\.,[2024](https://arxiv.org/html/2607.16872#bib.bib19); Lu and Lab,[2025](https://arxiv.org/html/2607.16872#bib.bib13)\)\. Adapting this idea to dLLMs is non\-trivial because a diffusion step operates on a partially masked sequence and may propose multiple token decisions in parallel\. Unlike autoregressive OPD, where each generated prefix defines a single committed next\-token context, dLLM distillation must decide which partially denoised states and token positions should receive teacher feedback along a multi\-step reveal process\. Random\-mask supervision provides dense masked\-token targets, but these artificial states may not match the states or reveal order encountered by the current student during generation\(Heet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib42); Wanget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib11)\)\.

The core challenge is therefore to make OPD both on\-policy and trajectory\-aware\. The student should be supervised on states sampled from its own denoising process, and the teacher signal should be organized according to the trace construction induced by that process rather than by independently sampled masks\. TOPD addresses this by sampling on\-policy diffusion trajectories and distilling teacher distributions on trace\-aligned denoising decisions, which we formalize next\.

## 3Method

![Refer to caption](https://arxiv.org/html/2607.16872v1/figs/diffopd_full_pipeline.png)Figure 3:Overview of TOPD\. The student samples an on\-policy denoising trajectory, the teacher provides step\-wise token\-distribution supervision, and updates are applied only to trace\-aligned decisionsa~t\\tilde\{a\}\_\{t\}retained in the final output\.We introduceTOPD, a trace\-based on\-policy distillation framework for post\-training masked diffusion language models\. Given a prompt, TOPD samples a student denoising trajectory, keeps the token decisions that survive into the final response, and distills a frozen teacher distribution on those student\-visited states\.

Figure[3](https://arxiv.org/html/2607.16872#S3.F3)summarizes the TOPD pipeline\. Section[3\.1](https://arxiv.org/html/2607.16872#S3.SS1)defines the on\-policy diffusion trajectory that supplies training states\. Section[3\.2](https://arxiv.org/html/2607.16872#S3.SS2)describes how TOPD extracts trace\-aligned decisions from that trajectory\. Section[3\.3](https://arxiv.org/html/2607.16872#S3.SS3)constructs the teacher–student distribution pairs on the retained state\-position pairs\. Section[3\.4](https://arxiv.org/html/2607.16872#S3.SS4)gives the Reverse\-KL objective and its sampled\-token estimator\.

### 3\.1On\-Policy Diffusion Trajectories

The first challenge is to supervise states that match the student’s diffusion\-time behavior\. TOPD therefore samples trajectories from the current student rather than constructing states by randomly masking completed answers\. Letπθ\\pi\_\{\\theta\}denote the student dLLM and letqqbe a prompt\. Generation starts from a fully masked response states0s\_\{0\}and proceeds forTTdiffusion steps; at steptt, the student predicts tokens for selected masked positions and updates the response tost\+1s\_\{t\+1\}\. We record the rollout as

τ=\(s0,a0,s1,a1,…,sT−1,aT−1,sT\),\\tau=\(s\_\{0\},a\_\{0\},s\_\{1\},a\_\{1\},\\ldots,s\_\{T\-1\},a\_\{T\-1\},s\_\{T\}\),\(3\)wheresTs\_\{T\}is the final response andat=\{\(j,xj\)\}a\_\{t\}=\\\{\(j,x\_\{j\}\)\\\}is the set of token decisions proposed and accepted at steptt\.

Samplingτ\\taufrom the current student makes TOPD on\-policy for dLLMs\. Unlike random masks, which can expose later answer tokens while hiding earlier reasoning tokens,τ\\taufollows the student’s own reveal order and supplies the states encountered during inference\.

### 3\.2Trace\-Aligned Decision Selection

A second challenge is that not every token proposal in a diffusion rollout is a final decision\. Because later denoising may revise or overwrite provisional proposals, TOPD adopts the trace\-construction principle of TraceRL\(Wanget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib11)\), where the trace keeps only token decisions that appear in the final response:

a~t=\{\(j,xj\)∈at:xj=sT,j\}\.\\tilde\{a\}\_\{t\}=\\\{\(j,x\_\{j\}\)\\in a\_\{t\}:x\_\{j\}=s\_\{T,j\}\\\}\.\(4\)Herejjindexes a response position, anda~t\\tilde\{a\}\_\{t\}is the trace\-aligned subset at diffusion steptt\.

Trace alignment turns a non\-autoregressive rollout into supervised state\-position pairs\. Each retained pair\(st,j\)\(s\_\{t\},j\)marks the context in which the student committed final tokenxjx\_\{j\}, so teacher feedback is attached to output\-forming decisions rather than arbitrary masked positions in a completed answer\.

### 3\.3Step\-Wise Teacher Distribution Matching

Teacher feedback must use the same conditional context in which the student made each retained decision\. TOPD therefore evaluates teacher and student distributions on the same trace\-aligned state\-position pairs\. Letπtea\\pi\_\{\\text\{tea\}\}be a frozen teacher dLLM\. For each\(j,xj\)∈a~t\(j,x\_\{j\}\)\\in\\tilde\{a\}\_\{t\}, the teacher distribution is

pt,jtea\(⋅\)=πtea\(⋅∣q,st,j\)\.p^\{\\text\{tea\}\}\_\{t,j\}\(\\cdot\)=\\pi\_\{\\text\{tea\}\}\(\\cdot\\mid q,s\_\{t\},j\)\.\(5\)The corresponding student distribution is evaluated at the same prompt, state, and position:

pt,jθ\(⋅\)=πθ\(⋅∣q,st,j\)\.p^\{\\theta\}\_\{t,j\}\(\\cdot\)=\\pi\_\{\\theta\}\(\\cdot\\mid q,s\_\{t\},j\)\.\(6\)For block\-wise implementations,sts\_\{t\}denotes the schedule\-consistent model input at that step\.

This paired evaluation makes supervision state\-matched rather than answer\-matched\. The teacher signal is dense at the token level, on\-policy at the state level, and aligned with the student’s diffusion trace\.

### 3\.4Reverse\-KL Training Objective

The final challenge is to turn trace\-aligned teacher distributions into stable token\-level updates\. TOPD matches each retained student distribution to the frozen teacher with Reverse\-KL\. ForBBprompts andGGsampled trajectories per prompt, the loss is

ℒTOPD​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\text\{TOPD\}\}\(\\theta\)=1B​G​∑i=1B​G∑t:\|a~t,i\|\>01\|a~t,i\|\\displaystyle=\\frac\{1\}\{BG\}\\sum\_\{i=1\}^\{BG\}\\sum\_\{t:\\,\|\\tilde\{a\}\_\{t,i\}\|\>0\}\\frac\{1\}\{\|\\tilde\{a\}\_\{t,i\}\|\}\(7\)∑\(j,xj\)∈a~t,iDKL\(pt,jθ\(⋅\)∥pt,jtea\(⋅\)\)\.\\displaystyle\\quad\\sum\_\{\(j,x\_\{j\}\)\\in\\tilde\{a\}\_\{t,i\}\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(p^\{\\theta\}\_\{t,j\}\(\\cdot\)\\,\\middle\\\|\\,p^\{\\text\{tea\}\}\_\{t,j\}\(\\cdot\)\\right\)\.The normalization averages over sampled trajectories and then over retained positions within each diffusion step, preventing steps with many revealed tokens from dominating the update\.

Reverse\-KL concentrates the student on teacher\-preferred modes at the same state\. This is useful in noisy partially denoised contexts, where many alternatives are possible but only a few continue the student’s reasoning path\. The divergence ablation in Section[4\.4](https://arxiv.org/html/2607.16872#S4.SS4)supports this choice\.

In implementation, TOPD uses a sampled\-token score\-function estimator for the negative Reverse\-KL objective\. For each retained tokenxjx\_\{j\}, define

rt,j=sg​\(log⁡pt,jtea​\(xj\)−log⁡pt,jθ​\(xj\)\),r\_\{t,j\}=\\mathrm\{sg\}\\\!\\left\(\\log p^\{\\text\{tea\}\}\_\{t,j\}\(x\_\{j\}\)\-\\log p^\{\\theta\}\_\{t,j\}\(x\_\{j\}\)\\right\),\(8\)wheresg​\(⋅\)\\mathrm\{sg\}\(\\cdot\)denotes stop\-gradient, so the log\-probability gap is treated as a scalar coefficient\. The corresponding ascent estimator is

∇θ𝒥TOPD​\(θ\)\\displaystyle\\nabla\_\{\\theta\}\\mathcal\{J\}\_\{\\text\{TOPD\}\}\(\\theta\)≈1B​G​∑i=1B​G∑t:\|a~t,i\|\>01\|a~t,i\|\\displaystyle\\approx\\frac\{1\}\{BG\}\\sum\_\{i=1\}^\{BG\}\\sum\_\{t:\\,\|\\tilde\{a\}\_\{t,i\}\|\>0\}\\frac\{1\}\{\|\\tilde\{a\}\_\{t,i\}\|\}\(9\)∑\(j,xj\)∈a~t,i∇θlog⁡pt,jθ​\(xj\)​rt,j\.\\displaystyle\\quad\\sum\_\{\(j,x\_\{j\}\)\\in\\tilde\{a\}\_\{t,i\}\}\\nabla\_\{\\theta\}\\log p^\{\\theta\}\_\{t,j\}\(x\_\{j\}\)\\,r\_\{t,j\}\.A retained token receives a positive update when the teacher assigns it higher probability than the student and a negative update otherwise\. When full vocabularies are materialized, the same objective can be evaluated by summing over tokens\. The sampled form avoids reward models, value heads, old\-policy ratios, and sequence\-level credit assignment\.

## 4Experiments

We evaluate TOPD along three axes: effectiveness, mechanism, and practical value\. First, we test whether TOPD can match RL\-based post\-training on mathematical reasoning benchmarks under both static and dynamic decoding settings\. Second, we isolate whether on\-policy states, trace\-aligned token selection, and Reverse\-KL matching each contribute to the gain\. Third, we measure whether dense teacher supervision reduces training cost without introducing training instability\.

### 4\.1Experimental Setup

Unless otherwise noted, experiments use SDAR\-4B\-Chat\(Chenget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib30)\)as the student and TraDo\-8B\-Instruct\(Wanget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib11)\)as the teacher\. For mathematical post\-training, we use the MATH training set\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.16872#bib.bib37)\), retaining level 3–5 problems followingHuet al\.\([2026](https://arxiv.org/html/2607.16872#bib.bib38)\), yielding 8K tasks\. We use block\-wise diffusion with 4\-token blocks, 4 denoising steps per block, and 64 prompts per rollout round\. A*rollout step*denotes one rollout round plus its associated training update, not a single optimizer step\.

Training rollouts use dynamic decoding\. TOPD samples one response per prompt, whereas TraceRL samples 16 for reward normalization and value\-based credit assignment\. SFT uses TraDo\-8B\-Instruct\-generated responses from the same data source with the semi\-autoregressive objective for block\-attention dLLMs\(Arriolaet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib23)\)\. We evaluate every 5 rollout steps and report avg@3 for MATH500\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.16872#bib.bib37)\)and GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.16872#bib.bib32)\), and avg@20 for AIME2024\(Mathematical Association of America, American Mathematics Competitions,[2024](https://arxiv.org/html/2607.16872#bib.bib39)\), under both static and dynamic decoding\.

### 4\.2Main Results

Table 1:Main results on MATH500, AIME2024, and GSM8K\. “Avg\.” is the arithmetic mean over the three benchmarks for each decoding strategy\. For TOPD, scores are reported from the best evaluation checkpoint\. Method prefixes denote post\-training applied to the base model; TraceRL\-trained SDAR models correspond to TraDo checkpoints\. “Per\-Round Speedup” reports rollout\-round model\-compute speedup from Section[4\.5](https://arxiv.org/html/2607.16872#S4.SS5)and Appendix[F](https://arxiv.org/html/2607.16872#A6), with ESPO and TraceRL as the LLaDA and SDAR references \(1×\\times\)\. “To\-Accuracy Speedup” further accounts for rollout rounds to the comparable MATH500 comparison point\.![Refer to caption](https://arxiv.org/html/2607.16872v1/for_paper_final_run_result/table_vs_softadv_comparison.png)Figure 4:Learning curves of TOPD vs\. TraceRL on MATH500 under static \(solid\) and dynamic \(dashed\) masking\. TOPD reaches strong performance by rollout step 30, while TraceRL requires 120 rollout steps for comparable performance\.TOPD matches the strongest SDAR\-4B\-Chat TraceRL gains on MATH500 while using far fewer rollouts\. Table[1](https://arxiv.org/html/2607.16872#S4.T1)shows that TOPD improves SDAR\-4B\-Chat by \+5\.7 under static decoding and \+4\.5 under dynamic decoding, slightly exceeding TraceRL on MATH500 in both settings\. SFT also improves the base model, but its gains are smaller than TOPD, especially under dynamic decoding\. The improvements from TOPD extend to AIME2024 and GSM8K, showing that the transferred reasoning ability is not limited to the primary benchmark\.

TOPD reaches strong performance substantially earlier than reward\-based post\-training\. Figure[4](https://arxiv.org/html/2607.16872#S4.F4)shows that TOPD reaches TraceRL\-level MATH500 accuracy in roughly 4×\\timesfewer rollout rounds, supporting the claim that dense teacher supervision reduces the amount of sampled interaction needed for post\-training\.

### 4\.3Generalization to Full\-Attention Models

TOPD also transfers beyond the block\-attention setting where it is primarily evaluated\. In this experiment, the student is the original LLaDA\-8B\-Instruct\(Nieet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib16)\), a full\-attention masked diffusion language model, and the teacher is the ESPO\-trained LLaDA\-8B\-Instruct checkpoint\(Ouet al\.,[2026](https://arxiv.org/html/2607.16872#bib.bib25)\)\. As shown in Table[1](https://arxiv.org/html/2607.16872#S4.T1), TOPD improves LLaDA\-8B\-Instruct on MATH500 and GSM8K, and its average gains are comparable to ESPO\. The absolute gains are smaller than in the SDAR setting, likely because the LLaDA teacher is the student’s RL\-trained same\-size counterpart rather than a larger model, but the positive trend suggests that trace\-aligned teacher supervision is not specific to block\-attention generation\. Under the same teacher accounting, ESPO’s reported policy\-update budget and rollout grouping imply an estimated 10\.9×\\timesper\-round and 511\.6×\\timesto\-accuracy model\-compute speedup for TOPD at a comparable MATH500 point \(Appendix[F](https://arxiv.org/html/2607.16872#A6)\)\.

### 4\.4Ablation Studies

The ablation studies test whether each TOPD design choice is responsible for the observed gains\. They follow the default SDAR setup in Section[4\.1](https://arxiv.org/html/2607.16872#S4.SS1)and evaluate each variant on MATH500 under both static and dynamic decoding\. We vary only the source of supervised states, the token decisions that receive gradients, or the divergence objective\.

For multi\-run ablation summaries, we report*peak mean accuracy*\. At each evaluated training rollout step, we compute the mean accuracy and standard deviation across runs\. We then choose the training rollout step whose across\-run mean is highest and report that step’s mean±\\pmstandard deviation\. This statistic is therefore the peak of the mean learning curve\.

#### On\-policy vs\. off\-policy supervision\.

![Refer to caption](https://arxiv.org/html/2607.16872v1/for_paper_final_run_result/ablation_on_off_policy_sft_static_dynamic.png)Figure 5:On\-policy, off\-policy, and semi\-AR SFT supervision on MATH500\. Curves report mean accuracy over 3 runs with±1\\pm 1standard deviation shaded\.Table 2:On\-policy, off\-policy, and semi\-AR SFT supervision on MATH500\. Values are peak mean accuracy±\\pm1 standard deviation over 3 runs, as defined above; training rollout steps are shown in parentheses\. On\-policy and off\-policy variants use the same Reverse\-KL objective; semi\-AR SFT follows the supervised setting in Section[4\.1](https://arxiv.org/html/2607.16872#S4.SS1)\.Student\-sampled states are more useful than fixed or purely supervised states\. This ablation isolates the source of supervised states along an off\-policy\-to\-on\-policy spectrum\. Semi\-AR SFT uses fixed teacher\-generated responses with a semi\-autoregressive objective, off\-policy distillation uses the same teacher\-generated data as fixed trajectories, and TOPD samples trajectories from the current student\. Thus, semi\-AR SFT serves as the fixed\-target supervised reference point, while off\-policy distillation shares the Reverse\-KL objective with TOPD but not its student\-sampled states\. Figure[5](https://arxiv.org/html/2607.16872#S4.F5)shows the full across\-run mean curves, and Table[2](https://arxiv.org/html/2607.16872#S4.T2)shows an on\-policy\>\>off\-policy\>\>semi\-AR SFT ranking under both decoding strategies\.

#### Trace\-based updates vs\. random\-mask supervision\.

![Refer to caption](https://arxiv.org/html/2607.16872v1/for_paper_final_run_result/ablation_random_masking_vs_trace_aligned_static_dynamic_same_axis_with_sdar4b_baseline.png)Figure 6:Trace\-based supervision vs\. random\-mask supervision on MATH500\. Curves report mean accuracy over 3 runs with±1\\pm 1standard deviation shaded\.Table 3:Trace\-based versus random\-mask supervision on MATH500\. Values are peak mean accuracy±\\pm1 standard deviation over 3 runs, as defined above; training rollout steps are shown in parentheses\.Trace\-aligned token selection makes dense teacher feedback more targeted than random corruption\. This ablation keeps the on\-policy state distribution fixed and varies which denoising decisions receive teacher supervision\. Random\-mask supervision applies the teacher signal to randomly corrupted positions, whereas trace\-aligned supervision restricts updates to token decisions that survive into the final response\. Table[3](https://arxiv.org/html/2607.16872#S4.T3)shows that trace alignment improves peak mean accuracy under both decoding strategies, indicating that where the dense supervision is attached matters in addition to which states are sampled\.

#### Divergence objectives: Reverse\-KL, Forward Kullback–Leibler \(Forward\-KL\), and Jensen–Shannon divergence \(JSD\)\.

Table 4:Divergence objectives on MATH500\. Single\-run accuracy at key checkpoints\. Step 30 and Step 60 are evaluated checkpoints, andΔbase\\Delta\_\{\\mathrm\{base\}\}is the peak improvement over SDAR\-4B\-Chat base accuracy \(70\.2 static, 67\.4 dynamic\)\. All variants use the same on\-policy trace\-based setting\.Reverse\-KL is the most reliable divergence objective for trace\-based distillation\. This ablation keeps the on\-policy, trace\-aligned setting fixed and varies only the token\-distribution matching objective\. Table[4](https://arxiv.org/html/2607.16872#S4.T4)shows that JSD is competitive, especially early in training, but Reverse\-KL achieves the strongest peak overall\. Forward\-KL substantially underperforms, consistent with the analysis in Section[3\.4](https://arxiv.org/html/2607.16872#S3.SS4): covering the teacher’s full support in noisy states is less effective than concentrating on teacher\-preferred modes\.

Together, the ablations support the three core TOPD design choices\. On\-policy states improve over fixed targets, trace\-aligned decisions improve over random masks, and Reverse\-KL improves over Forward\-KL and slightly over JSD at peak\. These results explain why TOPD differs from SFT, fixed\-trajectory distillation, and generic masked\-token supervision\.

### 4\.5Efficiency and Stability

TOPD is more compute\-efficient than RL because it removes group\-rollout reward optimization\. TraceRL samples 16 responses per prompt and requires old\-logprob and value passes, whereas TOPD samples one response and performs a frozen\-teacher forward plus one student update\. Under our parameter\-scaled accounting in Appendix[F](https://arxiv.org/html/2607.16872#A6), this yields a 24\.0×\\timesper\-round model\-compute reduction for the SDAR\-4B\-Chat setting with value model\.

This per\-round saving compounds with faster convergence\. TOPD reaches comparable MATH500 accuracy in 4×\\timesfewer rollout rounds, yielding an estimated to\-accuracy speedup of 96\.0×\\timesover TraceRL\+V\. For LLaDA, explicit rollout\-round accounting gives a 10\.9×\\timesper\-round reduction; ESPO’s 3K policy\-update budget corresponds to 375 rollout steps, yielding a 511\.6×\\timesspeedup to the selected comparison checkpoint\. We did not observe training collapse; the main late\-training behavior is mild static overfitting after the peak checkpoint\.

## 5Related Work

#### Diffusion language models\.

Diffusion models generate by iterative denoising and have been adapted from continuous data to discrete language through categorical or masked corruption processes\(Sohl\-Dicksteinet al\.,[2015](https://arxiv.org/html/2607.16872#bib.bib2); Hoet al\.,[2020](https://arxiv.org/html/2607.16872#bib.bib3); Songet al\.,[2021](https://arxiv.org/html/2607.16872#bib.bib4); Austinet al\.,[2021](https://arxiv.org/html/2607.16872#bib.bib5); Shiet al\.,[2024](https://arxiv.org/html/2607.16872#bib.bib24)\)\. Recent masked and block diffusion language models scale this paradigm to instruction following and reasoning with partially parallel generation\(Nieet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib16); Chenget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib30); Arriolaet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib23); Gonget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib29); Labset al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib22); Yeet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib44); Bieet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib43)\)\. Because diffusion decoding produces provisional token decisions that may be revised before the final answer, trajectory structure matters\(Wanget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib11); Heet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib42)\)\. TOPD therefore trains on the student’s actual denoising trajectories rather than synthetic masked states\.

#### Reinforcement learning for dLLM post\-training\.

Reinforcement learning improves LLM reasoning through proximal policy optimization \(PPO\), reinforcement learning from human feedback \(RLHF\), preference optimization, and group\-relative policy optimization\(Schulmanet al\.,[2015](https://arxiv.org/html/2607.16872#bib.bib28),[2017](https://arxiv.org/html/2607.16872#bib.bib8); Ouyanget al\.,[2022](https://arxiv.org/html/2607.16872#bib.bib7); Rafailovet al\.,[2023](https://arxiv.org/html/2607.16872#bib.bib9); Shaoet al\.,[2024](https://arxiv.org/html/2607.16872#bib.bib18); Guoet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib17); Yuet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib40)\)\. Recent work extends these ideas to dLLMs with masked SFT, trajectory\-aware value modeling and sequence\-level policy optimization\(Zhaoet al\.,[2025b](https://arxiv.org/html/2607.16872#bib.bib10); Wanget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib11); Zhonget al\.,[2026](https://arxiv.org/html/2607.16872#bib.bib14); Ouet al\.,[2026](https://arxiv.org/html/2607.16872#bib.bib25); Zhuet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib41); Heet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib42); Wanget al\.,[2026](https://arxiv.org/html/2607.16872#bib.bib26); Zhaoet al\.,[2025a](https://arxiv.org/html/2607.16872#bib.bib27)\)\. These methods show the value of on\-policy rollouts and trace\-aware trajectory construction, but sparse final rewards must be converted into useful step\-level signals through likelihood ratios, value models, trajectory estimators, or stabilization heuristics\. TOPD keeps on\-policy trajectory coverage while replacing reward\-based credit assignment with dense teacher supervision on the student’s visited denoising states\.

#### On\-policy distillation\.

Knowledge distillation transfers behavior from a stronger teacher to a student by matching predictions or softened distributions\(Hintonet al\.,[2015](https://arxiv.org/html/2607.16872#bib.bib20); Sanhet al\.,[2020](https://arxiv.org/html/2607.16872#bib.bib33); Guet al\.,[2024](https://arxiv.org/html/2607.16872#bib.bib36); Xuet al\.,[2024](https://arxiv.org/html/2607.16872#bib.bib34)\)\. Standard offline distillation trains on fixed teacher data and may not cover states caused by student errors, a classic exposure\-bias issue in imitation learning\(Rosset al\.,[2011](https://arxiv.org/html/2607.16872#bib.bib31)\)\. On\-policy distillation instead samples states from the current student and queries teacher distributions on those states, combining on\-policy coverage with dense low\-variance supervision\(Agarwalet al\.,[2024](https://arxiv.org/html/2607.16872#bib.bib19); Lu and Lab,[2025](https://arxiv.org/html/2607.16872#bib.bib13); Xuet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib35); Song and Zheng,[2026](https://arxiv.org/html/2607.16872#bib.bib45)\)\. TOPD extends this idea to diffusion generation, where supervision must be aligned with the final denoising trace rather than committed autoregressive prefixes\.

## 6Conclusion

We introduced TOPD, a trace\-based on\-policy distillation framework for post\-training masked diffusion language models\. TOPD samples the student’s own denoising trajectories, queries a frozen teacher on the corresponding partially denoised states, and applies Reverse\-KL updates to the trace\-aligned token decisions that form the final response\. This preserves dense teacher supervision while avoiding sparse reward assignment, value modeling, and likelihood\-ratio machinery\. On mathematical reasoning, TOPD recovers TraceRL\-level MATH500 gains for SDAR\-4B\-Chat with 4×\\timesfewer rollout rounds and an estimated 96\.0×\\timesto\-accuracy model\-compute speedup; additional results on AIME2024, GSM8K, full\-attention LLaDA, and ablations support the roles of on\-policy states, trace alignment, and Reverse\-KL matching\.

## Limitations

Although TOPD provides an efficient way to transfer reasoning ability from a stronger dLLM to a weaker student through dense on\-policy teacher supervision, it still has several limitations\. First, our experiments use an 8B dLLM as the teacher and a 4B dLLM as the student, because stronger and larger publicly available dLLM teachers are currently limited\. For the full\-attention experiments, the lack of model families with multiple parameter scales further requires using an RL\-trained same\-size model as the teacher\. Therefore, it remains unclear whether the observed efficiency gains generalize when substantially larger or more capable teacher models become available\. Second, the evaluation is limited to mathematical reasoning benchmarks, including MATH500, AIME2024, and GSM8K, and does not cover broader reasoning tasks such as code generation, tool use, or open\-ended instruction following\. Third, TOPD relies on a frozen teacher for token\-level distributional supervision, making its effectiveness dependent on the teacher’s capability and alignment with the target domain\.

## Acknowledgments

The authors used AI assistants, including ChatGPT, for language polishing, wording suggestions, and coding assistance\. All research ideas, experimental design, analyses, and final manuscript content were reviewed and verified by the authors\.

## References

- R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. R\. Garea, M\. Geist, and O\. Bachem \(2024\)On\-policy distillation of language models: learning from self\-generated mistakes\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=3zKtaqxLhW)Cited by:[§A\.5](https://arxiv.org/html/2607.16872#A1.SS5.p1.2),[§1](https://arxiv.org/html/2607.16872#S1.p2.1),[§2\.3](https://arxiv.org/html/2607.16872#S2.SS3.p1.1),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px3.p1.1)\.
- M\. Arriola, A\. Gokaslan, J\. T\. Chiu, Z\. Yang, Z\. Qi, J\. Han, S\. S\. Sahoo, and V\. Kuleshov \(2025\)Block diffusion: interpolating between autoregressive and diffusion language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2503.09573)Cited by:[§A\.1](https://arxiv.org/html/2607.16872#A1.SS1.p1.4),[§A\.2](https://arxiv.org/html/2607.16872#A1.SS2.p2.5),[§C\.4](https://arxiv.org/html/2607.16872#A3.SS4.SSS0.Px2.p1.3),[§1](https://arxiv.org/html/2607.16872#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.16872#S2.SS1.p1.8),[§4\.1](https://arxiv.org/html/2607.16872#S4.SS1.p2.1),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px1.p1.1)\.
- J\. Austin, D\. D\. Johnson, J\. Ho, D\. Tarlow, and R\. van den Berg \(2021\)Structured denoising diffusion models in discrete state\-spaces\.InAdvances in Neural Information Processing Systems,A\. Beygelzimer, Y\. Dauphin, P\. Liang, and J\. W\. Vaughan \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=h7-XixPCAL)Cited by:[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px1.p1.1)\.
- T\. Bie, M\. Cao, K\. Chen, L\. Du, M\. Gong, Z\. Gong, Y\. Gu, J\. Hu, Z\. Huang, Z\. Lan, C\. Li, C\. Li, J\. Li, Z\. Li, H\. Liu, L\. Liu, G\. Lu, X\. Lu, Y\. Ma, J\. Tan, L\. Wei, J\. Wen, Y\. Xing, X\. Zhang, J\. Zhao, D\. Zheng, J\. Zhou, J\. Zhou, Z\. Zhou, L\. Zhu, and Y\. Zhuang \(2025\)LLaDA2\.0: scaling up diffusion language models to 100b\.External Links:2512\.15745,[Link](https://arxiv.org/abs/2512.15745)Cited by:[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px1.p1.1)\.
- S\. Cheng, Y\. Bian, D\. Liu, L\. Zhang, Q\. Yao, Z\. Tian, W\. Wang, Q\. Guo, K\. Chen, B\. Qi, and B\. Zhou \(2025\)SDAR: a synergistic diffusion\-autoregression paradigm for scalable sequence generation\.External Links:2510\.06303,[Link](https://arxiv.org/abs/2510.06303)Cited by:[§A\.1](https://arxiv.org/html/2607.16872#A1.SS1.p1.4),[§C\.1](https://arxiv.org/html/2607.16872#A3.SS1.SSS0.Px1.p1.1),[§C\.5](https://arxiv.org/html/2607.16872#A3.SS5.p1.8),[§1](https://arxiv.org/html/2607.16872#S1.p1.1),[§1](https://arxiv.org/html/2607.16872#S1.p5.2),[§2\.1](https://arxiv.org/html/2607.16872#S2.SS1.p1.8),[§4\.1](https://arxiv.org/html/2607.16872#S4.SS1.p1.1),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px1.p1.1)\.
- T\. Chu, Y\. Zhai, J\. Yang, S\. Tong, S\. Xie, D\. Schuurmans, Q\. V\. Le, S\. Levine, and Y\. Ma \(2025\)SFT memorizes, RL generalizes: a comparative study of foundation model post\-training\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=dYur3yabMj)Cited by:[§1](https://arxiv.org/html/2607.16872#S1.p1.1),[§1](https://arxiv.org/html/2607.16872#S1.p2.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.External Links:2110\.14168,[Link](https://arxiv.org/abs/2110.14168)Cited by:[§4\.1](https://arxiv.org/html/2607.16872#S4.SS1.p2.1)\.
- S\. Gong, S\. Agarwal, Y\. Zhang, J\. Ye, L\. Zheng, M\. Li, C\. An, P\. Zhao, W\. Bi, J\. Han, H\. Peng, and L\. Kong \(2025\)Scaling diffusion language models via adaptation from autoregressive models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=j1tSLYKwg8)Cited by:[§1](https://arxiv.org/html/2607.16872#S1.p1.1),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px1.p1.1)\.
- Y\. Gu, L\. Dong, F\. Wei, and M\. Huang \(2024\)MiniLLM: knowledge distillation of large language models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=5h0qf7IBZZ)Cited by:[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px3.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi, X\. Zhang, X\. Yu, Y\. Wu, Z\. F\. Wu, Z\. Gou, Z\. Shao, Z\. Li, Z\. Gao, A\. Liu, B\. Xue, B\. Wang, B\. Wu, B\. Feng, C\. Lu, C\. Zhao, C\. Deng, C\. Ruan, D\. Dai, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Dai, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Xu, H\. Ding, H\. Gao, H\. Qu, H\. Li, J\. Guo, J\. Li, J\. Chen, J\. Yuan, J\. Tu, J\. Qiu, J\. Li, J\. L\. Cai, J\. Ni, J\. Liang, J\. Chen, K\. Dong, K\. Hu, K\. You, K\. Gao, K\. Guan, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Zhao, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, M\. Zhang, M\. Zhang, M\. Tang, M\. Zhou, M\. Li, M\. Wang, M\. Li, N\. Tian, P\. Huang, P\. Zhang, Q\. Wang, Q\. Chen, Q\. Du, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. J\. Chen, R\. L\. Jin, R\. Chen, S\. Lu, S\. Zhou, S\. Chen, S\. Ye, S\. Wang, S\. Yu, S\. Zhou, S\. Pan, S\. S\. Li, S\. Zhou, S\. Wu, T\. Yun, T\. Pei, T\. Sun, T\. Wang, W\. Zeng, W\. Liu, W\. Liang, W\. Gao, W\. Yu, W\. Zhang, W\. L\. Xiao, W\. An, X\. Liu, X\. Wang, X\. Chen, X\. Nie, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Yang, X\. Li, X\. Su, X\. Lin, X\. Q\. Li, X\. Jin, X\. Shen, X\. Chen, X\. Sun, X\. Wang, X\. Song, X\. Zhou, X\. Wang, X\. Shan, Y\. K\. Li, Y\. Q\. Wang, Y\. X\. Wei, Y\. Zhang, Y\. Xu, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Wang, Y\. Yu, Y\. Zhang, Y\. Shi, Y\. Xiong, Y\. He, Y\. Piao, Y\. Wang, Y\. Tan, Y\. Ma, Y\. Liu, Y\. Guo, Y\. Ou, Y\. Wang, Y\. Gong, Y\. Zou, Y\. He, Y\. Xiong, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Y\. X\. Zhu, Y\. Huang, Y\. Li, Y\. Zheng, Y\. Zhu, Y\. Ma, Y\. Tang, Y\. Zha, Y\. Yan, Z\. Z\. Ren, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Xu, Z\. Xie, Z\. Zhang, Z\. Hao, Z\. Ma, Z\. Yan, Z\. Wu, Z\. Gu, Z\. Zhu, Z\. Liu, Z\. Li, Z\. Xie, Z\. Song, Z\. Pan, Z\. Huang, Z\. Xu, Z\. Zhang, and Z\. Zhang \(2025\)DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:ISSN 1476\-4687,[Link](http://dx.doi.org/10.1038/s41586-025-09422-z),[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[§1](https://arxiv.org/html/2607.16872#S1.p1.1),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px2.p1.1)\.
- H\. He, K\. Renz, Y\. Cao, and A\. Geiger \(2025\)MDPO: overcoming the training\-inference divide of masked diffusion language models\.arXiv preprint arXiv:2508\.13148\.External Links:2508\.13148,[Document](https://dx.doi.org/10.48550/arXiv.2508.13148)Cited by:[§A\.3](https://arxiv.org/html/2607.16872#A1.SS3.p1.3),[§1](https://arxiv.org/html/2607.16872#S1.p2.1),[§1](https://arxiv.org/html/2607.16872#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.16872#S2.SS1.p1.8),[§2\.2](https://arxiv.org/html/2607.16872#S2.SS2.p1.1),[§2\.3](https://arxiv.org/html/2607.16872#S2.SS3.p1.1),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px2.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the MATH dataset\.InThirty\-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track \(Round 2\),External Links:[Link](https://openreview.net/forum?id=7Bywt2mQsCe)Cited by:[§1](https://arxiv.org/html/2607.16872#S1.p5.2),[§4\.1](https://arxiv.org/html/2607.16872#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.16872#S4.SS1.p2.1)\.
- G\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.External Links:1503\.02531,[Link](https://arxiv.org/abs/1503.02531)Cited by:[§A\.5](https://arxiv.org/html/2607.16872#A1.SS5.p1.2),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px3.p1.1)\.
- J\. Ho, A\. Jain, and P\. Abbeel \(2020\)Denoising diffusion probabilistic models\.InAdvances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6\-12, 2020, virtual,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\. Balcan, and H\. Lin \(Eds\.\),External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/4c5bcfec8584af0d967f1ab10179ca4b-Abstract.html)Cited by:[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px1.p1.1)\.
- J\. Hu, Y\. Zhang, Q\. Han, D\. Jiang, X\. Zhang, and H\. Shum \(2026\)Open\-reasoner\-zero: an open source approach to scaling up reinforcement learning on the base model\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=NFM8F5cV0V)Cited by:[§4\.1](https://arxiv.org/html/2607.16872#S4.SS1.p1.1)\.
- I\. Labs, S\. Khanna, S\. Kharbanda, S\. Li, H\. Varma, E\. Wang, S\. Birnbaum, Z\. Luo, Y\. Miraoui, A\. Palrecha, S\. Ermon, A\. Grover, and V\. Kuleshov \(2025\)Mercury: ultra\-fast language models based on diffusion\.External Links:2506\.17298,[Link](https://arxiv.org/abs/2506.17298)Cited by:[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px1.p1.1)\.
- X\. L\. Li, J\. Thickstun, I\. Gulrajani, P\. Liang, and T\. Hashimoto \(2022\)Diffusion\-LM improves controllable text generation\.InAdvances in Neural Information Processing Systems,A\. H\. Oh, A\. Agarwal, D\. Belgrave, and K\. Cho \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=3s9IrEsjLyk)Cited by:[§1](https://arxiv.org/html/2607.16872#S1.p1.1)\.
- K\. Lu and T\. M\. Lab \(2025\)On\-policy distillation\.Thinking Machines Lab: Connectionism\.Note:https://thinkingmachines\.ai/blog/on\-policy\-distillationExternal Links:[Document](https://dx.doi.org/10.64434/tml.20251026)Cited by:[§A\.5](https://arxiv.org/html/2607.16872#A1.SS5.p1.2),[§1](https://arxiv.org/html/2607.16872#S1.p2.1),[§2\.3](https://arxiv.org/html/2607.16872#S2.SS3.p1.1),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px3.p1.1)\.
- Mathematical Association of America, American Mathematics Competitions \(2024\)American invitational mathematics examination \(aime\) 2024: aime i and aime ii\.Mathematical Association of America \(AMC\)\.Note:[https://artofproblemsolving\.com/wiki/index\.php/AIME\_Problems\_and\_Solutions](https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions)Competition problems used as an evaluation dataset; original problems by MAA AMCCited by:[§4\.1](https://arxiv.org/html/2607.16872#S4.SS1.p2.1)\.
- S\. Nie, F\. Zhu, Z\. You, X\. Zhang, J\. Ou, J\. Hu, J\. Zhou, Y\. Lin, J\. Wen, and C\. Li \(2025\)Large language diffusion models\.External Links:2502\.09992,[Link](https://arxiv.org/abs/2502.09992)Cited by:[§A\.1](https://arxiv.org/html/2607.16872#A1.SS1.p1.4),[§1](https://arxiv.org/html/2607.16872#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.16872#S2.SS1.p1.8),[§4\.3](https://arxiv.org/html/2607.16872#S4.SS3.p1.2),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px1.p1.1)\.
- J\. Ou, J\. Han, M\. Xu, S\. Xu, J\. Xie, S\. Ermon, Y\. Wu, and C\. Li \(2026\)Principled RL for diffusion LLMs emerges from a sequence\-level perspective\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=S5YeC9llIL)Cited by:[§C\.4](https://arxiv.org/html/2607.16872#A3.SS4.SSS0.Px3.p1.6),[§1](https://arxiv.org/html/2607.16872#S1.p2.1),[§1](https://arxiv.org/html/2607.16872#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.16872#S2.SS2.p1.1),[§4\.3](https://arxiv.org/html/2607.16872#S4.SS3.p1.2),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px2.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Gray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. Lowe \(2022\)Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,A\. H\. Oh, A\. Agarwal, D\. Belgrave, and K\. Cho \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=TG8KACxEON)Cited by:[§1](https://arxiv.org/html/2607.16872#S1.p1.1),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px2.p1.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=HPuSIXJaa9)Cited by:[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px2.p1.1)\.
- S\. Ross, G\. J\. Gordon, and J\. A\. Bagnell \(2011\)A reduction of imitation learning and structured prediction to no\-regret online learning\.External Links:1011\.0686,[Link](https://arxiv.org/abs/1011.0686)Cited by:[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px3.p1.1)\.
- S\. S\. Sahoo, M\. Arriola, A\. Gokaslan, E\. M\. Marroquin, A\. M\. Rush, Y\. Schiff, J\. T\. Chiu, and V\. Kuleshov \(2024\)Simple and effective masked diffusion language models\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=L4uaAR4ArM)Cited by:[§A\.1](https://arxiv.org/html/2607.16872#A1.SS1.p1.4),[§1](https://arxiv.org/html/2607.16872#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.16872#S2.SS1.p1.8)\.
- V\. Sanh, L\. Debut, J\. Chaumond, and T\. Wolf \(2020\)DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter\.External Links:1910\.01108,[Link](https://arxiv.org/abs/1910.01108)Cited by:[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px3.p1.1)\.
- J\. Schulman, S\. Levine, P\. Moritz, M\. I\. Jordan, and P\. Abbeel \(2015\)Trust region policy optimization\.arXiv preprint arXiv:1502\.05477\.External Links:1502\.05477Cited by:[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px2.p1.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.External Links:1707\.06347,[Link](https://arxiv.org/abs/1707.06347)Cited by:[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px2.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.External Links:2402\.03300,[Link](https://arxiv.org/abs/2402.03300)Cited by:[§1](https://arxiv.org/html/2607.16872#S1.p1.1),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px2.p1.1)\.
- J\. Shi, K\. Han, Z\. Wang, A\. Doucet, and M\. Titsias \(2024\)Simplified and generalized masked diffusion for discrete data\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=xcqSOfHt4g)Cited by:[§2\.1](https://arxiv.org/html/2607.16872#S2.SS1.p1.8),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px1.p1.1)\.
- J\. Sohl\-Dickstein, E\. Weiss, N\. Maheswaranathan, and S\. Ganguli \(2015\)Deep unsupervised learning using nonequilibrium thermodynamics\.InProceedings of the 32nd International Conference on Machine Learning,F\. Bach and D\. Blei \(Eds\.\),Proceedings of Machine Learning Research, Vol\.37,Lille, France,pp\. 2256–2265\.External Links:[Link](https://proceedings.mlr.press/v37/sohl-dickstein15.html)Cited by:[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px1.p1.1)\.
- M\. Song and M\. Zheng \(2026\)A survey of on\-policy distillation for large language models\.External Links:2604\.00626,[Link](https://arxiv.org/abs/2604.00626)Cited by:[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px3.p1.1)\.
- Y\. Song, J\. Sohl\-Dickstein, D\. P\. Kingma, A\. Kumar, S\. Ermon, and B\. Poole \(2021\)Score\-based generative modeling through stochastic differential equations\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=PxTIG12RRHS)Cited by:[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px1.p1.1)\.
- C\. Wang, P\. Rashidinejad, D\. Su, S\. Jiang, S\. Wang, S\. Zhao, C\. Zhou, S\. Z\. Shen, F\. Chen, T\. Jaakkola, Y\. Tian, and B\. Liu \(2026\)SPG: sandwiched policy gradient for masked diffusion language models\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=18j5Q49GwN)Cited by:[§1](https://arxiv.org/html/2607.16872#S1.p2.1),[§1](https://arxiv.org/html/2607.16872#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.16872#S2.SS2.p1.1),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px2.p1.1)\.
- Y\. Wang, L\. Yang, B\. Li, Y\. Tian, K\. Shen, and M\. Wang \(2025\)Revolutionizing reinforcement learning framework for diffusion large language models\.arXiv preprint arXiv:2509\.06949\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2509.06949)Cited by:[§A\.3](https://arxiv.org/html/2607.16872#A1.SS3.p1.3),[Appendix B](https://arxiv.org/html/2607.16872#A2.p1.1),[§C\.1](https://arxiv.org/html/2607.16872#A3.SS1.SSS0.Px1.p1.1),[§C\.4](https://arxiv.org/html/2607.16872#A3.SS4.SSS0.Px1.p1.4),[§1](https://arxiv.org/html/2607.16872#S1.p2.1),[§1](https://arxiv.org/html/2607.16872#S1.p3.1),[§1](https://arxiv.org/html/2607.16872#S1.p5.2),[§2\.1](https://arxiv.org/html/2607.16872#S2.SS1.p1.8),[§2\.2](https://arxiv.org/html/2607.16872#S2.SS2.p1.1),[§2\.3](https://arxiv.org/html/2607.16872#S2.SS3.p1.1),[§3\.2](https://arxiv.org/html/2607.16872#S3.SS2.p1.4),[§4\.1](https://arxiv.org/html/2607.16872#S4.SS1.p1.1),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px2.p1.1)\.
- C\. Wu, H\. Zhang, S\. Xue, Z\. Liu, S\. Diao, L\. Zhu, P\. Luo, S\. Han, and E\. Xie \(2026\)Fast\-dLLM: training\-free acceleration of diffusion LLM by enabling KV cache and parallel decoding\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=3Z3Is6hnOT)Cited by:[§C\.1](https://arxiv.org/html/2607.16872#A3.SS1.SSS0.Px1.p1.1)\.
- W\. Xu, R\. Han, Z\. Wang, L\. Le, D\. Madeka, L\. Li, W\. Y\. Wang, R\. Agarwal, C\. Lee, and T\. Pfister \(2025\)Speculative knowledge distillation: bridging the teacher\-student gap through interleaved sampling\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=EgJhwYR2tB)Cited by:[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px3.p1.1)\.
- X\. Xu, M\. Li, C\. Tao, T\. Shen, R\. Cheng, J\. Li, C\. Xu, D\. Tao, and T\. Zhou \(2024\)A survey on knowledge distillation of large language models\.External Links:2402\.13116,[Link](https://arxiv.org/abs/2402.13116)Cited by:[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px3.p1.1)\.
- J\. Ye, Z\. Xie, L\. Zheng, J\. Gao, Z\. Wu, X\. Jiang, Z\. Li, and L\. Kong \(2025\)Dream 7b: diffusion large language models\.External Links:2508\.15487,[Link](https://arxiv.org/abs/2508.15487)Cited by:[§1](https://arxiv.org/html/2607.16872#S1.p1.1),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px1.p1.1)\.
- Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, M\. Qiao, Y\. Wu, and M\. Wang \(2025\)DAPO: an open\-source llm reinforcement learning system at scale\.External Links:2503\.14476,[Link](https://arxiv.org/abs/2503.14476)Cited by:[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px2.p1.1)\.
- H\. Zhao, D\. Liang, W\. Tang, D\. D\. Yao, and N\. Kallus \(2025a\)DiFFPO: training diffusion llms to reason fast and furious via reinforcement learning\.arXiv preprint arXiv:2510\.02212\.External Links:2510\.02212Cited by:[§1](https://arxiv.org/html/2607.16872#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.16872#S2.SS2.p1.1),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px2.p1.1)\.
- S\. Zhao, D\. Gupta, Q\. Zheng, and A\. Grover \(2025b\)D1: scaling reasoning in diffusion large language models via reinforcement learning\.arXiv preprint arXiv:2504\.12216\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2504.12216)Cited by:[§1](https://arxiv.org/html/2607.16872#S1.p2.1),[§1](https://arxiv.org/html/2607.16872#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.16872#S2.SS2.p1.1),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px2.p1.1)\.
- J\. Zhong, K\. Wang, D\. Ding, Z\. Feng, H\. Bai, Y\. Xiang, J\. Sun, and Q\. Xu \(2026\)Stabilizing reinforcement learning for diffusion language models\.arXiv preprint arXiv:2603\.06743\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2603.06743)Cited by:[§1](https://arxiv.org/html/2607.16872#S1.p2.1),[§1](https://arxiv.org/html/2607.16872#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.16872#S2.SS2.p1.1),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px2.p1.1)\.
- F\. Zhu, R\. Wang, S\. Nie, X\. Zhang, C\. Wu, J\. Hu, J\. Zhou, J\. Chen, Y\. Lin, J\. Wen, and C\. Li \(2025\)LLaDA 1\.5: variance\-reduced preference optimization for large language diffusion models\.arXiv preprint arXiv:2505\.19223\.External Links:2505\.19223Cited by:[§1](https://arxiv.org/html/2607.16872#S1.p2.1),[§1](https://arxiv.org/html/2607.16872#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.16872#S2.SS2.p1.1),[§5](https://arxiv.org/html/2607.16872#S5.SS0.SSS0.Px2.p1.1)\.

## Appendix AAdditional Method Details

This section formalizes the auxiliary definitions used by the method and ablation studies\. We first define the attention patterns for full\-attention and block\-attention dLLMs, then specify the supervised, decoding, random\-mask, and distribution\-matching objectives referenced in the main text\.

### A\.1Attention Patterns

Let a response have lengthLLand letBBdenote the block size\. We writeb​\(i\)=⌈i/B⌉b\(i\)=\\lceil i/B\\rceilfor the response block containing positionii\. Prompt tokens are visible to all response positions in both model families, so the distinction below concerns response\-to\-response attention\. Following prior masked diffusion language models and block diffusion models\(Sahooet al\.,[2024](https://arxiv.org/html/2607.16872#bib.bib15); Nieet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib16); Arriolaet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib23); Chenget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib30)\), a full\-attention dLLM uses a bidirectional response mask

Ai​jfull=1,1≤i,j≤L,A^\{\\mathrm\{full\}\}\_\{ij\}=1,\\qquad 1\\leq i,j\\leq L,\(10\)which allows every response position to condition on every other response position in the corrupted sequence\. A block\-attention dLLM uses a block\-causal response mask

Ai​jblock=𝟏​\[b​\(j\)≤b​\(i\)\],1≤i,j≤L,A^\{\\mathrm\{block\}\}\_\{ij\}=\\mathbf\{1\}\\\!\\left\[b\(j\)\\leq b\(i\)\\right\],\\qquad 1\\leq i,j\\leq L,\(11\)so a token in blockb​\(i\)b\(i\)can attend to tokens in the same block and earlier blocks, but not to future blocks\. This is the architectural difference behind the “full attention” and “block attention” groupings in Table[1](https://arxiv.org/html/2607.16872#S4.T1)\. In the SDAR setting,B=4B=4\.

### A\.2Masked and Semi\-Autoregressive SFT

For a prompt\-response pair\(q,y\)\(q,y\), wherey=\(y1,…,yL\)y=\(y\_\{1\},\\ldots,y\_\{L\}\), the full\-response masked SFT objective samples a corrupted responsey~\\tilde\{y\}fromyyand predicts the masked positions\. For compactness, writehjfull=\(q,y~,j\)h^\{\\mathrm\{full\}\}\_\{j\}=\(q,\\tilde\{y\},j\):

ℒfull​\-​SFT​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{full\\text\{\-\}SFT\}\}\(\\theta\)=𝔼q,y,y~1\|ℳ​\(y~\)\|\[\\displaystyle=\\mathbb\{E\}\_\{q,y,\\tilde\{y\}\}\\frac\{1\}\{\|\\mathcal\{M\}\(\\tilde\{y\}\)\|\}\\Bigg\[\(12\)∑j∈ℳ​\(y~\)−logπθ\(yj∣hjfull\)\]\.\\displaystyle\\quad\\sum\_\{j\\in\\mathcal\{M\}\(\\tilde\{y\}\)\}\-\\log\\pi\_\{\\theta\}\(y\_\{j\}\\mid h^\{\\mathrm\{full\}\}\_\{j\}\)\\Bigg\]\.whereℳ​\(y~\)=\{j:y~j=\[MASK\]\}\\mathcal\{M\}\(\\tilde\{y\}\)=\\\{j:\\tilde\{y\}\_\{j\}=\\texttt\{\[MASK\]\}\\\}is the masked\-position set\. This objective is appropriate for full\-attention masked diffusion models because all response positions share one bidirectional corrupted context\.

For block\-attention dLLMs, the SFT baseline in Section[4\.1](https://arxiv.org/html/2607.16872#S4.SS1)uses a semi\-autoregressive objective following block diffusion training\(Arriolaet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib23)\)\. Let

Ck=\{\(k−1\)​B\+1,…,min⁡\(k​B,L\)\}C\_\{k\}=\\\{\(k\-1\)B\+1,\\ldots,\\min\(kB,L\)\\\}\(13\)be thekk\-th response block, and lety~Ck\\tilde\{y\}\_\{C\_\{k\}\}be a corrupted copy of the current block\. The clean prefixy<Cky\_\{<C\_\{k\}\}is provided as context, while the loss is applied only inside the current block\. Lethk,jsemi=\(q,y<Ck,y~Ck,j\)h^\{\\mathrm\{semi\}\}\_\{k,j\}=\(q,y\_\{<C\_\{k\}\},\\tilde\{y\}\_\{C\_\{k\}\},j\):

ℒsemi​\-​AR​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{semi\\text\{\-\}AR\}\}\(\\theta\)=𝔼q,y∑k=1⌈L/B⌉𝔼y~Ck1\|ℳ​\(y~Ck\)\|\[\\displaystyle=\\mathbb\{E\}\_\{q,y\}\\sum\_\{k=1\}^\{\\lceil L/B\\rceil\}\\mathbb\{E\}\_\{\\tilde\{y\}\_\{C\_\{k\}\}\}\\frac\{1\}\{\|\\mathcal\{M\}\(\\tilde\{y\}\_\{C\_\{k\}\}\)\|\}\\Bigg\[\(14\)∑j∈ℳ​\(y~Ck\)−logπθ\(yj∣hk,jsemi\)\]\.\\displaystyle\\quad\\sum\_\{j\\in\\mathcal\{M\}\(\\tilde\{y\}\_\{C\_\{k\}\}\)\}\-\\log\\pi\_\{\\theta\}\\bigl\(y\_\{j\}\\mid h^\{\\mathrm\{semi\}\}\_\{k,j\}\\bigr\)\\Bigg\]\.Equation[14](https://arxiv.org/html/2607.16872#A1.E14)is the supervised baseline denoted “Semi\-AR SFT” in the ablations\. It differs from TOPD in two ways: the target responses are fixed teacher\-generated samples, and the supervised states are produced by an SFT masking process rather than by the current student’s rollout trajectory\.

### A\.3Decoding Schedules

At diffusion steptt, letMt=\{j:st,j=\[MASK\]\}M\_\{t\}=\\\{j:s\_\{t,j\}=\\texttt\{\[MASK\]\}\\\}be the currently masked response positions, and let

ct,j=maxv∈𝒱⁡πθ​\(v∣q,st,j\)c\_\{t,j\}=\\max\_\{v\\in\\mathcal\{V\}\}\\pi\_\{\\theta\}\(v\\mid q,s\_\{t\},j\)\(15\)be the model confidence at positionjj\. We use confidence\-based remasking schedules following recent dLLM decoding and trajectory\-optimization work\(Heet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib42); Wanget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib11)\)\. Static decoding reveals a fixed number of positions per step, usually the highest\-confidence positions needed to satisfy the block schedule:

Utstatic=TopKj∈Mt⁡\(ct,j;mt\),U\_\{t\}^\{\\mathrm\{static\}\}=\\operatorname\{TopK\}\_\{j\\in M\_\{t\}\}\(c\_\{t,j\};\\,m\_\{t\}\),\(16\)wheremtm\_\{t\}is determined by the block size and remaining denoising steps\. Dynamic decoding instead reveals positions whose confidence exceeds a thresholdτ\\tau:

Utdynamic=\{j∈Mt:ct,j≥τ\},U\_\{t\}^\{\\mathrm\{dynamic\}\}=\\\{j\\in M\_\{t\}:c\_\{t,j\}\\geq\\tau\\\},\(17\)with a fallback to at least one revealed position when the set is empty\. Both schedules then sample or select token values forj∈Utj\\in U\_\{t\}and keep the remaining positions masked\. These definitions correspond to the static and dynamic evaluation columns in Section[4\.1](https://arxiv.org/html/2607.16872#S4.SS1)\.

### A\.4Random\-Mask Supervision

The random\-mask ablation uses the same sampled final responsesTs\_\{T\}as TOPD but changes where teacher supervision is attached\. We estimate this objective withMMindependently sampled random masks\. For each sampler∈\{1,…,M\}r\\in\\\{1,\\ldots,M\\\}, a random mask setR\(r\)⊆\{1,…,L\}R^\{\(r\)\}\\subseteq\\\{1,\\ldots,L\\\}is drawn and a corrupted responses¯\(r\)\\bar\{s\}^\{\(r\)\}is formed by replacing positions inR\(r\)R^\{\(r\)\}with\[MASK\]\. In contrast to Semi\-AR SFT, which applies one random masking instance to each fixed teacher response during a training pass, random\-mask distillation expands each sampled response with enough independently drawn masks to match the number of trace rows used by TOPD\. This keeps the amount of distillation supervision comparable to TOPD while removing trace\-aligned position selection\. Leth¯j\(r\)=\(q,s¯\(r\),j\)\\bar\{h\}^\{\(r\)\}\_\{j\}=\(q,\\bar\{s\}^\{\(r\)\},j\),pr,jθ=πθ\(⋅∣h¯j\(r\)\)p^\{\\theta\}\_\{r,j\}=\\pi\_\{\\theta\}\(\\cdot\\mid\\bar\{h\}^\{\(r\)\}\_\{j\}\), andpr,jtea=πtea\(⋅∣h¯j\(r\)\)p^\{\\mathrm\{tea\}\}\_\{r,j\}=\\pi\_\{\\mathrm\{tea\}\}\(\\cdot\\mid\\bar\{h\}^\{\(r\)\}\_\{j\}\)\. The teacher and student are then matched on those random positions:

ℒRandom​\-​Mask​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{Random\\text\{\-\}Mask\}\}\(\\theta\)=1M​∑r=1M1\|R\(r\)\|​∑j∈R\(r\)\\displaystyle=\\frac\{1\}\{M\}\\sum\_\{r=1\}^\{M\}\\frac\{1\}\{\|R^\{\(r\)\}\|\}\\sum\_\{j\\in R^\{\(r\)\}\}\(18\)D\(pr,jθ∥pr,jtea\)\.\\displaystyle\\quad D\\\!\\left\(p^\{\\theta\}\_\{r,j\}\\,\\middle\\\|\\,p^\{\\mathrm\{tea\}\}\_\{r,j\}\\right\)\.HereDDis the chosen distribution\-matching objective\. Unlike TOPD, this objective does not use the trace\-aligned retained seta~t\\tilde\{a\}\_\{t\}and therefore may supervise positions unrelated to the student’s actual reveal decisions\.

### A\.5Distribution\-Matching Objectives

For a retained or masked state\-position pair, letp=pt,jθp=p^\{\\theta\}\_\{t,j\}denote the student distribution andq=pt,jteaq=p^\{\\mathrm\{tea\}\}\_\{t,j\}denote the teacher distribution\. Distribution matching is the standard signal in knowledge distillation and on\-policy distillation\(Hintonet al\.,[2015](https://arxiv.org/html/2607.16872#bib.bib20); Agarwalet al\.,[2024](https://arxiv.org/html/2607.16872#bib.bib19); Lu and Lab,[2025](https://arxiv.org/html/2607.16872#bib.bib13)\)\. The three objectives in Section[4\.4](https://arxiv.org/html/2607.16872#S4.SS4)are

DRKL​\(p,q\)\\displaystyle D\_\{\\mathrm\{RKL\}\}\(p,q\)=DKL​\(p∥q\)\\displaystyle=D\_\{\\mathrm\{KL\}\}\(p\\\|q\)=∑v∈𝒱p​\(v\)​log⁡p​\(v\)q​\(v\),\\displaystyle=\\sum\_\{v\\in\\mathcal\{V\}\}p\(v\)\\log\\frac\{p\(v\)\}\{q\(v\)\},\(19\)DFKL​\(p,q\)\\displaystyle D\_\{\\mathrm\{FKL\}\}\(p,q\)=DKL​\(q∥p\)\\displaystyle=D\_\{\\mathrm\{KL\}\}\(q\\\|p\)=∑v∈𝒱q​\(v\)​log⁡q​\(v\)p​\(v\),\\displaystyle=\\sum\_\{v\\in\\mathcal\{V\}\}q\(v\)\\log\\frac\{q\(v\)\}\{p\(v\)\},\(20\)and aβ\\beta\-weighted Jensen–Shannon objective\. For0<β<10<\\beta<1, letmβ=\(1−β\)​p\+β​qm\_\{\\beta\}=\(1\-\\beta\)p\+\\beta qand define

DJSD,β​\(p,q\)\\displaystyle D\_\{\\mathrm\{JSD\},\\beta\}\(p,q\)=\(1−β\)​DKL​\(p∥mβ\)\\displaystyle=\(1\-\\beta\)D\_\{\\mathrm\{KL\}\}\(p\\\|m\_\{\\beta\}\)\(21\)\+β​DKL​\(q∥mβ\)\.\\displaystyle\\quad\+\\beta D\_\{\\mathrm\{KL\}\}\(q\\\|m\_\{\\beta\}\)\.The ablation withβ=0\.5\\beta=0\.5uses the symmetric JSD\. The implementation also permits endpoint settings by dispatchingβ=0\\beta=0to Forward\-KL andβ=1\\beta=1to Reverse\-KL\. In the main experiments, SDAR TOPD uses full\-vocabulary KL, while LLaDA TOPD uses top\-k=10k\{=\}10token KL for the teacher–student distribution match; Table[6](https://arxiv.org/html/2607.16872#A3.T6)reports this LLaDA\-specific truncation\.

## Appendix BAlgorithm Details

Algorithm[1](https://arxiv.org/html/2607.16872#alg1)instantiates the TOPD loop from Section[3](https://arxiv.org/html/2607.16872#S3)\. TOPD adopts the trace\-construction principle of TraceRL\(Wanget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib11)\): after a diffusion rollout is completed, only token decisions that survive into the final response are retained as trace decisions\. Each TOPD rollout round has three stages: sample a minibatch of prompts and responses from the current student, convert the resulting trajectories into trace\-aligned distillation rows, and optimize on minibatches of those rows with frozen\-teacher supervision\.

Algorithm 1TOPD: Trace\-Based On\-Policy Distillation0:Student

πθ\\pi\_\{\\theta\}, frozen teacher

πtea\\pi\_\{\\text\{tea\}\}, prompt set

𝒬\\mathcal\{Q\}, rollout prompt batch size

NN, responses per prompt

GG, update minibatch size

mm, block size

BB, denoising steps per block

KK, maximum response length

LmaxL\_\{\\max\}
1:foreach rollout rounddo

2:Sample a prompt minibatch

ℬ=\{qi\}i=1N∼𝒬\\mathcal\{B\}=\\\{q\_\{i\}\\\}\_\{i=1\}^\{N\}\\sim\\mathcal\{Q\}
3:Initialize trace\-row dataset

𝒟TOPD←∅\\mathcal\{D\}\_\{\\mathrm\{TOPD\}\}\\leftarrow\\varnothing
4:Sample student rollouts

5:foreach prompt

qi∈ℬq\_\{i\}\\in\\mathcal\{B\}do

6:for

g=1g=1to

GGdo

7:

s0i,g←s\_\{0\}^\{i,g\}\\leftarrowfully masked response of length

LmaxL\_\{\\max\}
8:

τi,g←\(s0i,g\)\\tau\_\{i,g\}\\leftarrow\(s\_\{0\}^\{i,g\}\)
9:for

t=0t=0to

T−1T\-1do

10:

ati,g∼πθ\(⋅∣qi,sti,g\)a\_\{t\}^\{i,g\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid q\_\{i\},s\_\{t\}^\{i,g\}\)
11:

st\+1i,g←unmask​\(sti,g,ati,g\)s\_\{t\+1\}^\{i,g\}\\leftarrow\\text\{unmask\}\(s\_\{t\}^\{i,g\},a\_\{t\}^\{i,g\}\)
12:Append

\(ati,g,st\+1i,g\)\(a\_\{t\}^\{i,g\},s\_\{t\+1\}^\{i,g\}\)to

τi,g\\tau\_\{i,g\}
13:endfor

14:endfor

15:endfor

16:Build trace\-aligned training rows

17:foreach trajectory

τi,g=\(s0,a0,…,sT\)\\tau\_\{i,g\}=\(s\_\{0\},a\_\{0\},\\ldots,s\_\{T\}\)do

18:for

t=0t=0to

T−1T\-1do

19:

a~ti,g←\{\(j,xj\)∈at:xj=sT,j\}\\tilde\{a\}\_\{t\}^\{i,g\}\\leftarrow\\\{\(j,x\_\{j\}\)\\in a\_\{t\}:x\_\{j\}=s\_\{T,j\}\\\}
20:foreach

\(j,xj\)∈a~ti,g\(j,x\_\{j\}\)\\in\\tilde\{a\}\_\{t\}^\{i,g\}do

21:Add row

\(qi,s^ti,g,j,xj\)\(q\_\{i\},\\hat\{s\}\_\{t\}^\{i,g\},j,x\_\{j\}\)to

𝒟TOPD\\mathcal\{D\}\_\{\\mathrm\{TOPD\}\}
22:endfor

23:endfor

24:endfor

25:Teacher\-supervised minibatch updates

26:foreach update minibatch

ℳ⊂𝒟TOPD\\mathcal\{M\}\\subset\\mathcal\{D\}\_\{\\mathrm\{TOPD\}\}with

\|ℳ\|=m\|\\mathcal\{M\}\|=mdo

27:foreach row

\(q,s^,j,xj\)∈ℳ\(q,\\hat\{s\},j,x\_\{j\}\)\\in\\mathcal\{M\}do

28:

pq,s^,jtea←πtea\(⋅∣q,s^,j\)p^\{\\text\{tea\}\}\_\{q,\\hat\{s\},j\}\\leftarrow\\pi\_\{\\text\{tea\}\}\(\\cdot\\mid q,\\hat\{s\},j\)
29:

pq,s^,jθ←πθ\(⋅∣q,s^,j\)p^\{\\theta\}\_\{q,\\hat\{s\},j\}\\leftarrow\\pi\_\{\\theta\}\(\\cdot\\mid q,\\hat\{s\},j\)
30:

ℓ​\(q,s^,j\)←DKL​\(pq,s^,jθ∥pq,s^,jtea\)\\ell\(q,\\hat\{s\},j\)\\leftarrow D\_\{\\mathrm\{KL\}\}\\\!\\left\(p^\{\\theta\}\_\{q,\\hat\{s\},j\}\\,\\\|\\,p^\{\\text\{tea\}\}\_\{q,\\hat\{s\},j\}\\right\)
31:endfor

32:

ℒℳ←1\|ℳ\|​∑\(q,s^,j,xj\)∈ℳℓ​\(q,s^,j\)\\mathcal\{L\}\_\{\\mathcal\{M\}\}\\leftarrow\\frac\{1\}\{\|\\mathcal\{M\}\|\}\\sum\_\{\(q,\\hat\{s\},j,x\_\{j\}\)\\in\\mathcal\{M\}\}\\ell\(q,\\hat\{s\},j\)
33:

θ←θ−η​∇θℒℳ\\theta\\leftarrow\\theta\-\\eta\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathcal\{M\}\}
34:endfor

35:endfor

TOPD does not ask the teacher to supervise arbitrary corruptions of a completed answer; it supervises only the predictions that survive intosTs\_\{T\}\. Consequently, teacher feedback is attached to decisions that actually form the sampled response\. The states^t\\hat\{s\}\_\{t\}is reconstructed with the same schedule constraints as the student’s denoising step: positions beyond the current progress point are reset to\[MASK\], so teacher and student are evaluated under the same conditional context\.

In practice, we implement Reverse\-KL via a sampled\-token score\-function estimator:

∇θℒt,j≈−∇θlog⁡pt,jθ​\(xj\)⋅sg​\(log⁡pt,jtea​\(xj\)pt,jθ​\(xj\)\)\.\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{t,j\}\\approx\-\\nabla\_\{\\theta\}\\log p^\{\\theta\}\_\{t,j\}\(x\_\{j\}\)\\cdot\\mathrm\{sg\}\\\!\\left\(\\log\\tfrac\{p^\{\\text\{tea\}\}\_\{t,j\}\(x\_\{j\}\)\}\{p^\{\\theta\}\_\{t,j\}\(x\_\{j\}\)\}\\right\)\.\(22\)The estimator updates only the realized tokenxjx\_\{j\}and uses the teacher–student log\-probability gap as a stopped scalar coefficient\. If the teacher assigns higher probability to the retained token than the student, the update increases the student’s probability of that token; if the teacher assigns lower probability, the update suppresses it\. In implementation, teacher logits are computed once for each reconstructed trace state and reused for all retained positions at that step\.

## Appendix CExperimental Details

This section collects the experimental choices and supplementary ablation evidence needed to reproduce and interpret the main TOPD runs\. We separate default SDAR settings, prompt construction, the full\-attention LLaDA variant, ablation settings, baseline settings, evaluation rules, and extended run\-level analyses\.

### C\.1Training Configuration

Table[5](https://arxiv.org/html/2607.16872#A3.T5)gives the default SDAR\-4B\-Chat TOPD configuration\. Unless an ablation explicitly changes a component, the experiments in Section[4\.4](https://arxiv.org/html/2607.16872#S4.SS4)inherit these model, rollout, optimization, and data settings\.

Table 5:Training hyperparameters for TOPD on SDAR\-4B\-Chat\.#### Training infrastructure\.

All SDAR TOPD experiments use 4 NVIDIA A100\-80G or A800 GPUs\. We train with DeepSpeed ZeRO\-2, parameter CPU offload, bfloat16 mixed precision, and TF32 enabled\. A rollout round consists of student sampling followed by one training pass over the resulting traces; for SDAR\-4B, one round takes approximately 8–12 minutes on 4 A100\-80G GPUs, and the 30\-step run used for the main comparison finishes in about 5 hours\. For rollout and evaluation inference, we use an extended Fast\-dLLM implementation\(Wuet al\.,[2026](https://arxiv.org/html/2607.16872#bib.bib12); Wanget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib11)\)for LLaDA and JetEngine\(Chenget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib30)\)for SDAR\.

### C\.2Prompt Template and Rollout Records

TOPD uses the same prompt surface for student rollouts, teacher supervision, and evaluation\. The student first generates an on\-policy denoising trace from the model\-specific prompt below; the teacher is then queried on the reconstructed partially denoised states from that same trace\. The teacher receives the original problem and the student’s current state\.

Math prompt templatesSDAR<\|im\_start\|\>user\{\{problem\}\}Please reason step by step, and put your final answer within \\boxed\{\}\.<\|im\_end\|\><\|im\_start\|\>assistantLLaDA<\|startoftext\|\><\|start\_header\_id\|\>user<\|end\_header\_id\|\> You need to put your final answer in \\boxed\{\}\. This is the problem:\{\{problem\}\} <\|eot\_id\|\><\|startoftext\|\><\|start\_header\_id\|\>assistant<\|end\_header\_id\|\>

Training uses the same problem statement and sampled response format as rollout generation for each model family\. This keeps the training\-time prompt identical to the rollout\-time prompt and avoids introducing a separate reference\-answer formatting path\. Evaluation on MATH500, AIME2024, and GSM8K uses the corresponding model\-specific template, with the benchmark problem substituted for\{\{problem\}\}\. For MATH500 and AIME2024, the final`\\boxed\{\}`expression is used for answer extraction; for GSM8K, we also parse the final numeric answer to match the standard GSM8K evaluation convention\.

### C\.3Full\-Attention TOPD Configuration

Table[6](https://arxiv.org/html/2607.16872#A3.T6)specifies the full\-attention LLaDA run used in Section[4\.3](https://arxiv.org/html/2607.16872#S4.SS3)\. This experiment keeps the TOPD data flow fixed but changes the model family, generation budget, block size, and objective variant to match the LLaDA recipe and its ESPO comparison point\.

Table 6:Training hyperparameters for TOPD on LLaDA\-8B\-Instruct\.
### C\.4Baseline Configurations

#### TraceRL\.

We implement TraceRL\(Wanget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib11)\)with the official training recipe\. Each rollout round samplesG=16G\{=\}16responses per prompt, scores completed responses with the reward model, recomputes old token log\-probabilities, and applies a PPO\-style update with clippingϵ=0\.2\\epsilon\{=\}0\.2and KL penaltyβ=0\.01\\beta\{=\}0\.01\. In the SDAR\-4B comparison, TraceRL also uses a value model, which adds value inference and value optimization to each training cycle\. The learning rate is1×10−61\\times 10^\{\-6\}with cosine decay, and the effective batch size is matched to TOPD at 16\.

#### Semi\-AR SFT\.

The supervised SDAR baseline fine\-tunes SDAR\-4B\-Chat on the 8,523\-problem MATH training subset using fixed teacher\-generated responses\. Following block diffusion training\(Arriolaet al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib23)\), it uses semi\-autoregressive masking with block sizeB=4B\{=\}4, a mask rate uniformly sampled from\[0\.1,0\.9\]\[0\.1,0\.9\], and cross\-entropy on masked response tokens only\. Training runs for one epoch with AdamW, learning rate1×10−51\\times 10^\{\-5\}, effective batch size 16, and gradient clipping at 1\.0\.

#### ESPO\.

We evaluate ESPO\(Ouet al\.,[2026](https://arxiv.org/html/2607.16872#bib.bib25)\)on LLaDA\-8B\-Instruct with LoRA adaptation on attention and MLP projection matrices \(r=128r\{=\}128,α=64\\alpha\{=\}64, dropout 0\.05\)\. Following the ESPO MATH setting, the policy update uses group sizeG=16G\{=\}16, policy\-update valueμ=8\\mu\{=\}8,M=2M\{=\}2Monte Carlo samples, bfloat16 precision, learning rate1×10−61\\times 10^\{\-6\}, weight decay 0\.1, and gradient clipping at 0\.2\. Generation uses random masking with 256 diffusion steps, maximum prompt length 400, maximum completion length 256, and temperature 1\.0\. TOPD\-LLaDA uses the same prompt set and a 512\-step decoding budget\.

### C\.5Evaluation Protocol

We evaluate SDAR checkpoints under both static and dynamic decoding\. Dynamic decoding uses a confidence threshold ofτ=0\.9\\tau=0\.9andt​o​ptop\-k=0k=0, i\.e\., all tokens are retained\. Static decoding follows Cheng et al\.\(Chenget al\.,[2025](https://arxiv.org/html/2607.16872#bib.bib30)\)and usest​o​ptop\-k=1k=1; both use block size 4 and 4 denoising steps per block\. We sample with temperature 1\.0 and top\-pp1\.0, reporting avg@3 for MATH500 and GSM8K and avg@20 for AIME2024, where avg@kkis the average accuracy overkksampled responses per problem\. MATH500 contains 500 problems, AIME2024 contains 30 problems, and GSM8K uses the 1,319\-example test split\. For MATH500 and AIME2024, we extract the final`\\boxed\{\}`answer and compare it with SymPy\-based normalization; for GSM8K, we extract the final numeric answer following the standard protocol\.

#### Checkpoint selection\.

SDAR learning curves and ablations are evaluated every 5 rollout steps\. Main\-table accuracies are taken from the best single run for each method, using the checkpoint selected on MATH500 under the corresponding decoding setting and then evaluated on the remaining benchmarks\. Multi\-run ablation summaries report peak mean accuracy: at each evaluated step, we compute the across\-run mean and standard deviation, then select the step with the highest across\-run mean\. Parenthesized step numbers in ablation tables denote the selected rollout step\. Table[7](https://arxiv.org/html/2607.16872#A3.T7)reports the MATH500 checkpoint choices for the main SDAR comparison and, for TOPD and TraceRL, the rollout\-step contrast used in the compute\-to\-accuracy calculation\.

Table 7:MATH500 checkpoint summary for the main SDAR comparison\. Static and dynamic columns report avg@3 accuracy\. “Selection” describes how the checkpoint is used in the main comparison\.

### C\.6Response Length Analysis

We additionally track generated response length for the SDAR\-4B\-Chat \+ TOPD runs to check whether TOPD’s accuracy gains are accompanied by a large change in output budget\. Figure[7](https://arxiv.org/html/2607.16872#A3.F7)plots the MATH500 average response length over the three SDAR TOPD runs used in the on\-policy ablation, with shaded regions showing±1\\pm 1standard deviation across runs\. Response length rises during the first few rollout rounds and then stays in a narrow range under both static and dynamic decoding, suggesting that the main MATH500 improvements are not driven by an uncontrolled expansion of generated answers\.

![Refer to caption](https://arxiv.org/html/2607.16872v1/figs/response_length_math500.png)Figure 7:Average response length on MATH500 for SDAR\-4B\-Chat \+ TOPD across three runs\. Curves report the across\-run mean, and shaded regions show±1\\pm 1standard deviation\.Table[8](https://arxiv.org/html/2607.16872#A3.T8)reports the average response length for the SDAR TOPD checkpoint used in Table[1](https://arxiv.org/html/2607.16872#S4.T1)\.

Table 8:Average response length of SDAR\-4B\-Chat \+ TOPD corresponding to Table[1](https://arxiv.org/html/2607.16872#S4.T1)\.
### C\.7Additional Ablation Diagnostics

This subsection provides an additional diagnostic view of the state\-source ablation in Section[4\.4](https://arxiv.org/html/2607.16872#S4.SS4)\. All experiments use SDAR\-4B\-Chat as the student, TraDo\-8B\-Instruct as the frozen teacher, MATH500 as the benchmark, and the same training budget as the main SDAR setting\.

#### Derived diagnostics for the state\-source ablation\.

![Refer to caption](https://arxiv.org/html/2607.16872v1/figs/appendix_deep_ablation_analysis.png)Figure 8:Derived diagnostics for the MATH500 state\-source ablation\. Left: the static–dynamic accuracy gap over training, where smaller gaps indicate less sensitivity to the evaluation remasking schedule\. Right: accuracy–length operating points for all evaluation steps with response\-length logs; filled markers denote static decoding and open markers denote dynamic decoding\.Figure[8](https://arxiv.org/html/2607.16872#A3.F8)analyzes the same on\-policy, off\-policy, and Semi\-AR SFT runs using two derived quantities rather than another copy of the learning curves\. First, the static–dynamic gap shows that TOPD keeps the two evaluation schedules closer than Semi\-AR SFT for most of training, indicating that trace\-aligned on\-policy supervision improves the sampled states\. Second, the accuracy–length scatter shows that TOPD reaches the best operating region without moving to systematically longer responses: its high\-accuracy static points cluster around the same response lengths as off\-policy distillation, while Semi\-AR SFT often produces longer responses without matching TOPD’s dynamic accuracy\. These diagnostics support the interpretation that TOPD’s gains come from better state\-matched supervision, not from a larger generation budget or a decoding\-specific artifact\.

## Appendix DCase Study

We include three representative MATH500 generations from the SDAR\-4B\-Chat TOPD checkpoint at step 30 under static decoding\. The first two are correct solutions that illustrate the model’s ability to sustain multi\-step, calculation\-grounded reasoning over long trajectories\. The third is a characteristic failure showing how a coherent\-looking solution can go wrong through an early algebraic slip\.

Case 1: Coin Redistribution \(MATH500 \#50\)Problem\.You have seven bags of gold coins\. Each bag has the same number of gold coins\. One day, you find a bag of 53 coins\. You decide to redistribute the number of coins you have so that all eight bags you hold have the same number of coins\. You successfully manage to redistribute all the coins, and you also note that you have more than 200 coins\. What is the smallest number of coins you could have had before finding the bag of 53 coins?TOPD output \(correct, 203\)\.Let the number of coins in each of the original seven bags bexx, so the total is7​x7x\. After finding the bag of 53 coins, the total becomes7​x\+537x\+53\. After redistribution into eight equal bags, let each bag containyycoins:8​y=7​x\+538y=7x\+53We also know the total exceeds 200:8​y\>2008y\>200, soy\>25y\>25\. Sinceyyis integer,y≥26y\\geq 26\.Substitutingy=26y=26:208=7​x\+53⇒155=7​x⇒x=155/7∉ℤ208=7x\+53\\Rightarrow 155=7x\\Rightarrow x=155/7\\not\\in\\mathbb\{Z\}\.Continuing the search:y=27\\displaystyle y=27:8⋅27=7​x\+53⟹163/7∉ℤ\\displaystyle:8\\\!\\cdot\\\!27=7x\\\!\+\\\!53\\;\\Longrightarrow\\;163/7\\notin\\mathbb\{Z\}y=28\\displaystyle y=28:8⋅28=7​x\+53⟹171/7∉ℤ\\displaystyle:8\\\!\\cdot\\\!28=7x\\\!\+\\\!53\\;\\Longrightarrow\\;171/7\\notin\\mathbb\{Z\}y=29\\displaystyle y=29:8⋅29=7​x\+53⟹179/7∉ℤ\\displaystyle:8\\\!\\cdot\\\!29=7x\\\!\+\\\!53\\;\\Longrightarrow\\;179/7\\notin\\mathbb\{Z\}y=30\\displaystyle y=30:8⋅30=7​x\+53⟹187/7∉ℤ\\displaystyle:8\\\!\\cdot\\\!30=7x\\\!\+\\\!53\\;\\Longrightarrow\\;187/7\\notin\\mathbb\{Z\}y=31\\displaystyle y=31:8⋅31=7​x\+53⟹195/7∉ℤ\\displaystyle:8\\\!\\cdot\\\!31=7x\\\!\+\\\!53\\;\\Longrightarrow\\;195/7\\notin\\mathbb\{Z\}y=32\\displaystyle y=32:8⋅32=7​x\+53⟹203/7=29∈ℤ\.\\displaystyle:8\\\!\\cdot\\\!32=7x\\\!\+\\\!53\\;\\Longrightarrow\\;203/7=29\\in\\mathbb\{Z\}\.The first integer solution isx=29x=29, giving the original total7​x=2037x=203\. Hence203\\boxed\{203\}\.

Case 2: Unfair Die Probability \(MATH500 \#475\)Problem\.When rolling a certain unfair six\-sided die with faces numbered 1, 2, 3, 4, 5, and 6, the probability of obtaining faceFFis greater than1/61/6, the probability of obtaining the face opposite faceFFis less than1/61/6, the probability of obtaining each of the other faces is1/61/6, and the sum of the numbers on each pair of opposite faces is 7\. When two such dice are rolled, the probability of obtaining a sum of 7 is47288\\frac\{47\}\{288\}\. Given that the probability of obtaining faceFFism/nm/n, wheremmandnnare relatively prime positive integers, findm\+nm\+n\.TOPD output \(correct, 29\)\.Letp=P​\(F\)p=P\(F\)andq=P​\(face opposite​F\)q=P\(\\text\{face opposite \}F\)\. The four remaining faces each have probability1/61/6\. Since total probability sums to 1:p\+q\+4⋅16=1⟹p\+q=13\.p\+q\+4\\cdot\\frac\{1\}\{6\}=1\\;\\Longrightarrow\\;p\+q=\\frac\{1\}\{3\}\.The opposite\-face pairs are\(1,6\),\(2,5\),\(3,4\)\(1,6\),\(2,5\),\(3,4\)\. Without loss of generality assumeF=1F=1, soP​\(1\)=pP\(1\)=pandP​\(6\)=qP\(6\)=q, whileP​\(2\)=P​\(3\)=P​\(4\)=P​\(5\)=1/6P\(2\)=P\(3\)=P\(4\)=P\(5\)=1/6\.The probability of rolling sum 7 with two dice comes from six ordered pairs:\(1,6\),\(6,1\),\(2,5\),\(5,2\),\(3,4\),\(4,3\)\(1,6\),\(6,1\),\(2,5\),\(5,2\),\(3,4\),\(4,3\)\. Therefore:P​\(sum=7\)\\displaystyle P\(\\text\{sum\}=7\)=2​p​q\+236\+236=2​\(p​q\+118\)\.\\displaystyle=2pq\+\\tfrac\{2\}\{36\}\+\\tfrac\{2\}\{36\}=2\\bigl\(pq\+\\tfrac\{1\}\{18\}\\bigr\)\.GivenP​\(sum=7\)=47/288P\(\\text\{sum\}=7\)=47/288:2​\(p​q\+118\)\\displaystyle 2\\bigl\(pq\+\\tfrac\{1\}\{18\}\\bigr\)=47288\\displaystyle=\\frac\{47\}\{288\}⇒p​q\+118\\displaystyle\\Rightarrow\\;pq\+\\frac\{1\}\{18\}=47576\\displaystyle=\\frac\{47\}\{576\}⇒p​q\\displaystyle\\Rightarrow\\;pq=47576−32576=15576=5192\.\\displaystyle=\\frac\{47\}\{576\}\-\\frac\{32\}\{576\}=\\frac\{15\}\{576\}=\\frac\{5\}\{192\}\.We now have the systemp\+q=1/3p\+q=1/3andp​q=5/192pq=5/192\. Solving the quadratict2−13​t\+5192=0t^\{2\}\-\\frac\{1\}\{3\}t\+\\frac\{5\}\{192\}=0:192​t2−64​t\+5\\displaystyle 192t^\{2\}\-64t\+5=0\\displaystyle=0t\\displaystyle t=64±642−4⋅192⋅52⋅192\\displaystyle=\\frac\{64\\pm\\sqrt\{64^\{2\}\-4\\cdot 192\\cdot 5\}\}\{2\\cdot 192\}=64±16384\\displaystyle=\\frac\{64\\pm 16\}\{384\}=524​or​18\.\\displaystyle=\\tfrac\{5\}\{24\}\\text\{ or \}\\tfrac\{1\}\{8\}\.Sincep\>1/6p\>1/6, we havep=5/24p=5/24andq=1/8q=1/8\. Thusm=5m=5,n=24n=24, andm\+n=29m\+n=29\. Hence29\\boxed\{29\}\.

Case 3 \(Failure\): Parallelogram Area \(MATH500 \#239\)Problem\.The parallelogram bounded byy=a​x\+cy=ax\+c,y=a​x\+dy=ax\+d,y=b​x\+cy=bx\+c,y=b​x\+dy=bx\+dhas area 18\. The parallelogram bounded byy=a​x\+cy=ax\+c,y=a​x−dy=ax\-d,y=b​x\+cy=bx\+c,y=b​x−dy=bx\-dhas area 72\. Given thata,b,c,da,b,c,dare positive integers, find the smallest possible value ofa\+b\+c\+da\+b\+c\+d\.TOPD output \(incorrect;predicts 20, ground truth is 16\)\.To solve the problem, we start by analyzing the given parallelograms and their areas\. The first parallelogram is bounded by the linesy=a​x\+cy=ax\+c,y=a​x\+dy=ax\+d,y=b​x\+cy=bx\+c, andy=b​x\+dy=bx\+d\. The second parallelogram is bounded by the linesy=a​x\+cy=ax\+c,y=a​x−dy=ax\-d,y=b​x\+cy=bx\+c, andy=b​x−dy=bx\-d\.#### Step 1: Determine the vertices\. For the first parallelogram:•a​x\+c=b​x\+c⇒x=0ax\+c=bx\+c\\Rightarrow x=0, giving\(0,c\)\(0,c\)\.•a​x\+d=b​x\+d⇒x=0ax\+d=bx\+d\\Rightarrow x=0, giving\(0,d\)\(0,d\)\.•a​x\+c=b​x\+d⇒x=d−ca−bax\+c=bx\+d\\Rightarrow x=\\frac\{d\-c\}\{a\-b\}, yielding\(d−ca−b,a​d−b​ca−b\)\\bigl\(\\frac\{d\-c\}\{a\-b\},\\;\\frac\{ad\-bc\}\{a\-b\}\\bigr\)\.•a​x\+d=b​x\+c⇒x=c−da−bax\+d=bx\+c\\Rightarrow x=\\frac\{c\-d\}\{a\-b\}, yielding\(c−da−b,b​c−a​da−b\)\\bigl\(\\frac\{c\-d\}\{a\-b\},\\;\\frac\{bc\-ad\}\{a\-b\}\\bigr\)\.For the second parallelogram:•a​x\+c=a​x−d⇒x=−d/cax\+c=ax\-d\\Rightarrow x=\-d/c, giving\(−d/c,c−d\)\(\-d/c,\\;c\-d\)\.•b​x\+c=b​x−d⇒x=−d/cbx\+c=bx\-d\\Rightarrow x=\-d/c, giving\(−d/c,c−d\)\(\-d/c,\\;c\-d\)\.•a​x\+c=b​x−d⇒x=c\+da−bax\+c=bx\-d\\Rightarrow x=\\frac\{c\+d\}\{a\-b\}, yielding\(c\+da−b,a​c−b​da−b\)\\bigl\(\\frac\{c\+d\}\{a\-b\},\\;\\frac\{ac\-bd\}\{a\-b\}\\bigr\)\.•a​x−d=b​x\+c⇒x=c\+da−bax\-d=bx\+c\\Rightarrow x=\\frac\{c\+d\}\{a\-b\}, yielding\(c\+da−b,b​c−a​da−b\)\\bigl\(\\frac\{c\+d\}\{a\-b\},\\;\\frac\{bc\-ad\}\{a\-b\}\\bigr\)\. #### Step 2: Calculate the areas\. For the first parallelogram, the distance betweeny=a​x\+cy=ax\+candy=a​x\+dy=ax\+dis\|d−c\|\|d\-c\|, and betweeny=b​x\+cy=bx\+candy=b​x\+dy=bx\+dis also\|d−c\|\|d\-c\|\. The area is\|d−c\|1\+a2⋅\|d−c\|1\+b2=\(d−c\)2\(1\+a2\)​\(1\+b2\)=18\.\\frac\{\|d\-c\|\}\{\\sqrt\{1\+a^\{2\}\}\}\\cdot\\frac\{\|d\-c\|\}\{\\sqrt\{1\+b^\{2\}\}\}=\\frac\{\(d\-c\)^\{2\}\}\{\\sqrt\{\(1\+a^\{2\}\)\(1\+b^\{2\}\)\}\}=18\.For the second parallelogram, each distance is2​d2d, giving2​d1\+a2⋅2​d1\+b2=4​d2\(1\+a2\)​\(1\+b2\)=72\.\\frac\{2d\}\{\\sqrt\{1\+a^\{2\}\}\}\\cdot\\frac\{2d\}\{\\sqrt\{1\+b^\{2\}\}\}=\\frac\{4d^\{2\}\}\{\\sqrt\{\(1\+a^\{2\}\)\(1\+b^\{2\}\)\}\}=72\. #### Step 3: Solve\. Divide the second equation by the first:4​d2\(d−c\)2\\displaystyle\\frac\{4d^\{2\}\}\{\(d\-c\)^\{2\}\}=4⟹d2=\(d−c\)2\\displaystyle=4\\;\\Longrightarrow\\;d^\{2\}=\(d\-c\)^\{2\}⟹d\\displaystyle\\;\\Longrightarrow\\;d=d−c​or​d=−\(d−c\)\.\\displaystyle=d\-c\\;\\text\{or\}\\;d=\-\(d\-c\)\.Sinceddandccare positive,d=d−cd=d\-cis impossible, sod=−\(d−c\)⇒2​d=c⇒c=2​dd=\-\(d\-c\)\\Rightarrow 2d=c\\Rightarrow c=2d\.Substitutingc=2​dc=2dback:d2\(1\+a2\)​\(1\+b2\)=18\.\\frac\{d^\{2\}\}\{\\sqrt\{\(1\+a^\{2\}\)\(1\+b^\{2\}\)\}\}=18\.Squaring both sides:d4=324​\(1\+a2\)​\(1\+b2\)\.d^\{4\}=324\\,\(1\+a^\{2\}\)\(1\+b^\{2\}\)\.Testing small values:d=3d=3gives\(1\+a2\)​\(1\+b2\)=1/4\(1\+a^\{2\}\)\(1\+b^\{2\}\)=1/4, impossible\. Ford=6d=6:64=12966^\{4\}=1296, so\(1\+a2\)​\(1\+b2\)=4\(1\+a^\{2\}\)\(1\+b^\{2\}\)=4, satisfied by\(a,b\)=\(1,1\)\(a,b\)=\(1,1\)\. Withc=2​d=12c=2d=12, we takea=b=1a=b=1and computea\+b\+c\+d=1\+1\+12\+6=20a\+b\+c\+d=1\+1\+12\+6=20\.Hence20\\boxed\{20\}\.

The failure case above \(Case 3\) illustrates the dominant error pattern we observed when inspecting incorrect TOPD generations: the model follows the requested step\-by\-step style, produces a boxed final answer, and the solution appears structurally sound—yet an early algebraic or arithmetic slip propagates unchecked through the derivation\. Across the broader set of incorrect solutions, the most common failures are not formatting failures but*local reasoning slips inside otherwise coherent solutions*: carrying an algebraic sign incorrectly, simplifying a fraction or exponent expression too early, mishandling a geometric constraint, or selecting a plausible intermediate result as the final answer before verifying it against the problem’s exact requested quantity\.

These failure modes are consistent with the aggregate results: TOPD improves the student’s reasoning trajectory distribution, but it does not add an external verifier or symbolic correction mechanism\. This observation motivates a natural extension: because TOPD already supplies dense teacher feedback on trace\-aligned decisions, it could be combined with answer verification or process\-level consistency checks—the teacher distribution would shape local denoising decisions, while a verifier would reject globally inconsistent solutions\. We leave this combination to future work\.

## Appendix ETrace Mechanism: Progressive Denoising Visualization

The following examples make trace\-aligned supervision concrete\. Each trajectory is drawn from MATH training data and decoded with block diffusion using block sizeB=4B\{=\}4andK=4K\{=\}4denoising steps per block\. Within a block, positions are progressively revealed; positions that have not yet been produced are displayed as\[M\]\. TOPD reconstructs these same partially denoised states for teacher evaluation, so the teacher supervises the student’s actual reveal order rather than a randomly corrupted final answer\.

### E\.1Trace 1: Quadratic Equation \(567 tokens, 48 blocks, 189 steps\)

Trace 1: Within\-Block Diffusion \(Quadratic Equation\)Problem\.Letppandqqbe the two distinct solutions to\(x−5\)​\(2​x\+9\)=x2−13​x\+40\(x\-5\)\(2x\+9\)=x^\{2\}\-13x\+40\. What is\(p\+3\)​\(q\+3\)\(p\+3\)\(q\+3\)?Block 1 produces “To solve the” over steps 1–4\. Blocks 2–4 are shown below with one column per block position;\[M\]marks a position that remains masked at that denoising substep\.

### E\.2Trace 2: Diophantine Equation \(880 tokens, 104 blocks, 413 steps\)

Trace 2: Within\-Block Diffusion \(Diophantine Equation\)Problem\.Find the number of ordered triples\(x,y,z\)\(x,y,z\)of real numbers such thatx4\+y4\+z4−4​x​y​z=−1x^\{4\}\+y^\{4\}\+z^\{4\}\-4xyz=\-1\.Block 1 produces “To solve the equation” over steps 1–4\. Blocks 2–4 use the same 4\-column format, showing that even within a short prefix the reveal order can differ across positions\.

## Appendix FTraining Compute Accounting

This appendix gives the update\-normalized model\-compute accounting used in Section[4\.5](https://arxiv.org/html/2607.16872#S4.SS5)\. The goal is not to predict wall\-clock runtime, but to compare the dominant neural\-model forward and backward work required by TOPD and the RL baselines under the same rollout\-round abstraction\. We assume matched batch size, matched prompt budget, and the same effective trace\-row length across methods, so common multiplicative factors are omitted\.

#### FLOPs proxy\.

For a student model withNsN\_\{s\}parameters and effective trace\-row lengthDD, we use

Cfwd=2​Ns​D,Cbwd=4​Ns​D\.C\_\{\\mathrm\{fwd\}\}=2N\_\{s\}D,\\qquad C\_\{\\mathrm\{bwd\}\}=4N\_\{s\}D\.\(23\)Thus one trainable forward–backward update costs6​Ns​D6N\_\{s\}D\. LetFs=2​Ns​DF\_\{s\}=2N\_\{s\}Ddenote one student forward\-equivalent unit\. A trainable student update costs3​Fs3F\_\{s\}, and a frozen teacher forward costsα​Fs\\alpha F\_\{s\}, whereα=Nt/Ns\\alpha=N\_\{t\}/N\_\{s\}\. For the SDAR\-4B experiments with an 8B teacher,α≈2\\alpha\\approx 2; for same\-size teacher accounting,α=1\\alpha=1\.

We use a full\-model forward/backward proxy for all methods, including LoRA\-based baselines\. This keeps the comparison at the level of dominant model evaluations rather than trainable\-parameter count or implementation\-specific optimizer savings\.

#### Update\-normalized costs\.

In the update\-normalized proxy, response generation is counted as one student forward\-equivalent unit per sampled response\. This abstracts away the identical denoising schedule details and isolates the algorithmic difference in the number of sampled responses and update\-side model passes\.

TOPD samples one response per prompt, queries the frozen teacher once, and performs one student update:

CTOPD=Fs\+α​Fs\+3​Fs=\(4\+α\)​Fs\.C\_\{\\mathrm\{TOPD\}\}=F\_\{s\}\+\\alpha F\_\{s\}\+3F\_\{s\}=\(4\+\\alpha\)F\_\{s\}\.\(24\)
TraceRL samplesG=16G=16responses per prompt\. Without a value model, each response requires response generation, old\-log\-probability recomputation, and one policy update:

CRL​\-​noV=G​\(Fs\+Fs\+3​Fs\)=80​Fs\.C\_\{\\mathrm\{RL\\text\{\-\}noV\}\}=G\(F\_\{s\}\+F\_\{s\}\+3F\_\{s\}\)=80F\_\{s\}\.\(25\)Since the original TraceRL recipe uses a value model, we use the value\-model setting as the main comparison\. The no\-value path is reported only as an auxiliary compute reference\. With a value model trained everykkrollout rounds, TraceRL additionally uses value inference and an amortized value update:

CRL​\-​V​\(k\)\\displaystyle C\_\{\\mathrm\{RL\\text\{\-\}V\}\}\(k\)=G​\(Fs\+Fs\+Fs\+3​Fs\+3​Fsk\)\\displaystyle=G\\left\(F\_\{s\}\+F\_\{s\}\+F\_\{s\}\+3F\_\{s\}\+\\frac\{3F\_\{s\}\}\{k\}\\right\)\(26\)=16​\(6\+3k\)​Fs\.\\displaystyle=6\\left\(6\+\\frac\{3\}\{k\}\\right\)F\_\{s\}\.For the SDAR\-4B comparison, the value model is trained every round \(k=1k=1\), giving

CRL​\-​V​\(1\)=144​Fs\.C\_\{\\mathrm\{RL\\text\{\-\}V\}\}\(1\)=144F\_\{s\}\.\(27\)
Therefore the per\-round update\-normalized reductions are

Sround,noV​\(α\)\\displaystyle S\_\{\\mathrm\{round,noV\}\}\(\\alpha\)=804\+α,\\displaystyle=\\frac\{80\}\{4\+\\alpha\},\(28\)Sround,V​\(α,k\)\\displaystyle S\_\{\\mathrm\{round,V\}\}\(\\alpha,k\)=16​\(6\+3/k\)4\+α\.\\displaystyle=\\frac\{16\(6\+3/k\)\}\{4\+\\alpha\}\.\(29\)For the SDAR\-4B comparison withα=2\\alpha=2andk=1k=1, this gives

Sround,V\(2,1\)=1446=24\.0×\.S\_\{\\mathrm\{round,V\}\}\(2,1\)=\\frac\{144\}\{6\}=24\.0\\times\.\(30\)With same\-size teacher accounting, it gives28\.8×28\.8\\times\.

Table 9:Update\-normalized per\-round model\-compute accounting for TraceRL versus TOPD\.α=1\\alpha=1counts the teacher as one same\-size frozen forward;α=2\\alpha=2charges the 8B teacher as twice the 4B student forward cost\.
#### Compute to reach comparable MATH500 accuracy\.

The learning\-curve comparison compounds the per\-round reduction with the number of rollout rounds needed to reach comparable MATH500 accuracy\. TOPD reaches the comparison point at step 30, while TraceRL reaches it at step 120, giving a4\.0×4\.0\\timesstep ratio\. For the value\-model setting used in the SDAR\-4B comparison,

24\.0×4\.0=96\.0×\.24\.0\\times 4\.0=96\.0\\times\.\(31\)Under same\-size teacher accounting, the corresponding estimate is115\.2×115\.2\\times; against the no\-value TraceRL path, it is53\.3×53\.3\\times\.

Table 10:Estimated training speedups to reach comparable MATH500 accuracy under parameter\-scaled teacher accounting for the SDAR\-4B comparison \(α=2\\alpha=2,k=1k=1\)\.
#### ESPO versus TOPD on LLaDA\.

For the LLaDA comparison, we use a stricter rollout\-round accounting so that ESPO and TOPD are compared under the same data\-collection unit\. ESPO reports a coupled\-sampling FLOPs proxy in which one sampled response costs

\(KESPO\+6​μ​M\)​Fs,\(K\_\{\\mathrm\{ESPO\}\}\+6\\mu M\)F\_\{s\},\(32\)whereKESPOK\_\{\\mathrm\{ESPO\}\}is the number of diffusion sampling steps,μ\\muis the number of policy updates per data\-collection round, andMMis the number of Monte Carlo ELBO samples\. For the ESPO MATH setting,G=16G=16,KESPO=256K\_\{\\mathrm\{ESPO\}\}=256,μ=8\\mu=8, andM=2M=2, giving

CESPO\\displaystyle C\_\{\\mathrm\{ESPO\}\}=G​\(KESPO\+6​μ​M\)​Fs\\displaystyle=G\(K\_\{\\mathrm\{ESPO\}\}\+6\\mu M\)F\_\{s\}=16​\(256\+6×8×2\)​Fs=5632​Fs\.\\displaystyle=16\(256\+6\\times 8\\times 2\)F\_\{s\}=5632F\_\{s\}\.\(33\)TOPD\-LLaDA samples one response per prompt withKTOPD=512K\_\{\\mathrm\{TOPD\}\}=512diffusion steps, queries a same\-size frozen teacher, and performs one student update:

CTOPD\\displaystyle C\_\{\\mathrm\{TOPD\}\}=\(KTOPD\+α\+3\)​Fs\\displaystyle=\(K\_\{\\mathrm\{TOPD\}\}\+\\alpha\+3\)F\_\{s\}=\(512\+1\+3\)​Fs=516​Fs\.\\displaystyle=\(512\+1\+3\)F\_\{s\}=516F\_\{s\}\.\(34\)The resulting rollout\-round model\-compute speedup is

Sround,ESPO/TOPD=5632516=10\.9×\.S\_\{\\mathrm\{round,ESPO/TOPD\}\}=\\frac\{5632\}\{516\}=10\.9\\times\.\(35\)
Table 11:Rollout\-round model\-compute accounting for LLaDA ESPO and TOPD\. ESPO usesG=16G=16,KESPO=256K\_\{\\mathrm\{ESPO\}\}=256,μ=8\\mu=8, andM=2M=2; TOPD\-LLaDA usesKTOPD=512K\_\{\\mathrm\{TOPD\}\}=512and a same\-size frozen teacher\.For the LLaDA comparison, ESPO is trained for 3K policy\-update steps under its official recipe\. Since ESPO usesμ=8\\mu=8policy updates per data\-collection round, this corresponds to3000/8=3753000/8=375rollout rounds\. TOPD\-LLaDA selects the step\-8 checkpoint from a 30\-round training run, giving a rollout\-step ratio of375/8=46\.875375/8=46\.875\. Combining this with the unrounded per\-round ratio gives

5632516×46\.875=511\.6×\.\\frac\{5632\}\{516\}\\times 46\.875=511\.6\\times\.\(36\)We therefore report10\.9×10\.9\\timesper\-round and511\.6×511\.6\\timesto\-accuracy model\-compute speedups for the LLaDA comparison\. This same\-size teacher accounting differs from the SDAR setting, where the teacher is larger than the student; for LLaDA, the teacher is the student’s ESPO\-trained counterpart\.

#### Interpretation\.

These estimates are theoretical model\-compute comparisons, not measured wall\-clock speedups\. They intentionally abstract away implementation details such as cached generation, synchronization, model placement, rollout filtering, and dataloader overhead\. The measured GPU\-hour results in Section[F](https://arxiv.org/html/2607.16872#A6.SS0.SSS0.Px6)therefore provide a complementary end\-to\-end runtime view\.

#### Measured GPU\-hour comparison and gap analysis\.

Table[12](https://arxiv.org/html/2607.16872#A6.T12)reports empirical GPU\-hour measurements from the SDAR\-4B comparison on 4×\\timesA100\-80G GPUs\. TraceRL with a value model \(G=16G\{=\}16,k=1k\{=\}1\) consumes approximately 1\.9–2\.0 GPU\-hours per rollout round, while TOPD consumes 0\.53–0\.80 GPU\-hours per round\. Combined with the faster convergence of TOPD, 30 versus 120 rollout rounds to reach comparable MATH500 accuracy, this yields a measured end\-to\-end speedup of1010–15×15\\times\.

Table 12:Measured GPU\-hour comparison between TOPD and TraceRL\+V on SDAR\-4B using 4×\\timesA100\-80G GPUs\. “To comparable acc\.” denotes GPU\-hours to reach the accuracy points in Section[4\.5](https://arxiv.org/html/2607.16872#S4.SS5)\.The measured speedup is lower than the theoretical to\-accuracy estimate of96\.0×96\.0\\timesbecause the model\-compute proxy abstracts away implementation\-dependent costs\. In practice, cached denoising forwards, full\-sequence training forwards, model placement, synchronization, and pipeline overhead have different runtime profiles\. TraceRL also updates only on prompt groups that contain both correct and incorrect responses, so not every sampled prompt contributes a policy\-gradient update\. These factors reduce the realized per\-round ratio, but the measured GPU\-hour results still show that TOPD reaches comparable MATH500 accuracy with substantially less end\-to\-end training compute\.