From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models

# From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models
Source: [https://arxiv.org/html/2607.26504](https://arxiv.org/html/2607.26504)
Seunggeun Kim1⋆Jaeyeon Kim2⋆Taekyun Lee1⋆Yuyuan Chen2⋆Yilun Du2Sham Kakade2Sitan Chen21University of Texas at Austin,2Harvard University,⋆Co\-first authors

###### Abstract

Many discrete reasoning tasks, such as code generation, are inherently non\-causal: programmers move between high\-level structure and local details, a process we call*any\-order inference*\. For autoregressive language models, which lack a native any\-order interface, non\-causal abilities such as infilling and next\-edit prediction require hand\-designed mechanisms\. Can we instead design models that natively support any\-order inference? Masked diffusion models have recently emerged as compelling candidates, as their any\-order training objective naturally offers an any\-order prediction interface\. This*interface*, however, does not automatically yield any\-order*inference*\. We demonstrate that this interface\-inference gap stems from*positional uncertainty*: fixed\-canvas, token\-level models may know*what*semantic component should appear without knowing*where*to place it\. In light of this, we propose two complementary approaches: \(1\)*Insertion\-based masked diffusion*, building on FlexMDM\(Kimet al\.,[2025a](https://arxiv.org/html/2607.26504#bib.bib14)\), relaxes fixed\-position commitments via insertions, enabling generation across non\-contiguous regions\. \(2\)*Latent\-space masked diffusion*shifts prediction to coarser semantic segments, enabling search over latent generation orders\. Empirically, we train a 7B FlexMDM for Python coding and a 125M LatentMDM for GSM8K and show that both approaches induce distinct any\-order inference behaviors and improve downstream performance\. We release our codebase at[https://github\.com/SeunggeunKimkr/genuine\-any\-order](https://github.com/SeunggeunKimkr/genuine-any-order)\.

![Refer to caption](https://arxiv.org/html/2607.26504v1/x1.png)Figure 1:Illustrative overview of our work\.Masked diffusion models cannot perform genuine*any\-order*inference as they operate on a fixed canvas at the token level\. We demonstrate that insertion\-based and latent\-space masked diffusion modeling overcome this limitation, enabling genuine*any\-order*inference\.## 1Introduction

Many real\-world discrete reasoning tasks are inherently non\-causal: when humans construct complex discrete objects such as code, they revise earlier decisions, insert missing pieces, and alternate between unfolding high\-level structure and filling in local details\. Autoregressive language models, the dominant paradigm in discrete generative modeling, decode left\-to\-right and thus do not natively support such trajectories\. A substantial line of work has nevertheless extended them with non\-causal mechanisms such as infilling\(Bavarianet al\.,[2022](https://arxiv.org/html/2607.26504#bib.bib96)\), reordering\(Shahet al\.,[2024](https://arxiv.org/html/2607.26504#bib.bib98)\), and next\-edit prediction\(Luet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib99)\)\. While this shows that any\-order inference is a desirable capability, the task\-specific nature of these approaches motivates more native and principled generative paradigms\.

We focus on masked diffusion models \(MDMs\)\(Shiet al\.,[2024](https://arxiv.org/html/2607.26504#bib.bib5); Sahooet al\.,[2024](https://arxiv.org/html/2607.26504#bib.bib16)\), a popular instantiation of diffusion language models\(Nieet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib12); Xieet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib11); Gonget al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib3); Bieet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib26)\)\. MDMs are natural candidates for any\-order inference as their training objective endows them with an any\-order prediction*interface*\(Ouet al\.,[2024](https://arxiv.org/html/2607.26504#bib.bib9); Zhenget al\.,[2024](https://arxiv.org/html/2607.26504#bib.bib10); Kimet al\.,[2025c](https://arxiv.org/html/2607.26504#bib.bib15)\): given a partially masked sequence, they predict clean tokens at arbitrary masked positions\. This interface makes any\-order inference possible in principle, but not automatic: the decoding policy must still choose which positions to reveal at each step, and poor choices can collapse generation order back toward left\-to\-right completion\. We therefore ask whether MDMs actually deliver on this promise:*can an any\-order prediction interface yield inference that is genuinely any\-order in nature?*

Our contributions\.In this work, we focus on code generation, a domain that would naturally benefit from genuine any\-order inference\. Our first finding is negative\. We show that under confidence\-based decoding, the dominant inference\-time strategy for improving downstream performance, MDMs produce code that is*algorithmically close*to what they would have generated via strict left\-to\-right decoding \(Section[3\.1](https://arxiv.org/html/2607.26504#S3.SS1)\)\. This resonates with recent observations of*causal collapse*\(Niet al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib43); Gonget al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib3); Liet al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib41)\), but we identify a more fundamental bottleneck underlying this phenomenon:*positional uncertainty*\.

Positional uncertainty arises when the model knows*what*semantic component should be generated next, but remains uncertain about*where*to place it: its probability mass is spread across multiple plausible locations on the fixed masked canvas\. Since confidence\-based unmasking ranks individual positions, these globally meaningful but*positionally dispersed*components appear less confident than locally determined continuations, pushing decoding back toward left\-to\-right completion\. These findings \(Section[3\.2](https://arxiv.org/html/2607.26504#S3.SS2)\) suggest that the any\-order training objective of masked diffusion models does not guarantee genuine any\-order inference*per se*\. This motivates a broader question:

What generative models can achieve genuine any\-order inference?

In this work, we propose two complementary remedies, each sidestepping positional uncertainty in a different way and inducing*distinct forms of any\-order inference*\. Our first solution,*insertion\-based masked diffusion*\(Section[4\.1](https://arxiv.org/html/2607.26504#S4.SS1)\), builds on FlexMDM\(Kimet al\.,[2025a](https://arxiv.org/html/2607.26504#bib.bib14)\), which equips MDMs with the ability to insert tokens\. This circumvents the fixed\-canvas bottleneck: unmasking no longer commits a token to a specific final position, since later insertions can shift its location in the final sequence\. With this reduced token\-commitment burden, we demonstrate that FlexMDM yields a structural form of any\-order inference: under the parsed program tree, generation*moves back and forth across different nodes*rather than completing one contiguous region at a time\. Empirically, we achieve this by significantly extending the practical scope ofKimet al\.\([2025a](https://arxiv.org/html/2607.26504#bib.bib14)\): we fine\-tune Dream\-Coder 7B\(Xieet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib11)\)into a general\-purpose Python\-code\-generating FlexMDM, which outperforms Dream\-Coder 7B on HumanEval and HumanEval\+\(Chenet al\.,[2021](https://arxiv.org/html/2607.26504#bib.bib23)\)at Pass@16, and moves ahead of it at Pass@1 on MBPP and MBPP\+\(Austinet al\.,[2021](https://arxiv.org/html/2607.26504#bib.bib24)\)while remaining on par overall\.

Our second solution,*latent\-space masked diffusion*\(LatentMDM\)\(Section[4\.2](https://arxiv.org/html/2607.26504#S4.SS2)\), overcomes positional uncertainty by moving masked diffusion from token space to a continuous latent space\. LatentMDM is distinct from prior embedding\-space approaches, such as the Flow Map Language Model\(Leeet al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib77)\), and constitutes our main modeling contribution\. Rather than selecting individual token positions, the model decides which masked latent segment to decode next, shifting the interface to a coarser semantic granularity\. This enables a distinct form of any\-order inference:*searching over semantic generation orders in latent space*, allowing the model to discover orderings beyond the left\-to\-right token order\. Unlike FlexMDM, LatentMDM has no off\-the\-shelf pretrained base model\. We therefore pretrain a 125M\-parameter latent masked diffusion model on TinyGSM\(Liuet al\.,[2023](https://arxiv.org/html/2607.26504#bib.bib64)\)and show that it outperforms standard MDMs, embedding\-space models such as DUO\(Sahooet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib80)\)and𝕊\\mathbb\{S\}\-FLM\(Deschenaux and Gulcehre,[2026](https://arxiv.org/html/2607.26504#bib.bib59)\), and even an autoregressive model with KV caching under the same inference budget\.

## 2Preliminaries

In this section, we review masked diffusion models \(MDMs\)\(Shiet al\.,[2024](https://arxiv.org/html/2607.26504#bib.bib5); Sahooet al\.,[2024](https://arxiv.org/html/2607.26504#bib.bib16)\)\.

Notation\.Suppose our goal is to learn to generate samples from the data distribution𝐱∼pdata\{\\mathbf\{x\}\}\\sim p\_\{\\mathrm\{data\}\}over length\-LLdiscrete sequences with a finite vocabulary𝒱\\mathcal\{V\}\. Let𝐱i\{\\mathbf\{x\}\}^\{i\}denote theii\-th element of a given sequence𝐱=\(𝐱1,…,𝐱L\)\{\\mathbf\{x\}\}=\(\{\\mathbf\{x\}\}^\{1\},\\dots,\{\\mathbf\{x\}\}^\{L\}\)andΔ​\(𝒱\)\\Delta\(\\mathcal\{V\}\)indicate the simplex of probability distributions over𝒱\\mathcal\{V\}\.

Training\.Although MDMs admit several interpretations, we adopt an*any\-order*language model interpretation\(Ouet al\.,[2024](https://arxiv.org/html/2607.26504#bib.bib9); Zhenget al\.,[2024](https://arxiv.org/html/2607.26504#bib.bib10)\), which streamlines the prior account of MDM\(Sahooet al\.,[2024](https://arxiv.org/html/2607.26504#bib.bib16); Shiet al\.,[2024](https://arxiv.org/html/2607.26504#bib.bib5)\)\. Roughly speaking, MDMs introduce an auxiliary mask token𝐦\\mathbf\{m\}and learn, for each masked position, the posterior marginals of clean tokens conditioned on a masked sequence\. To learn this posterior during training, one draws a clean sequence𝐱∼pdata\{\\mathbf\{x\}\}\\sim p\_\{\\mathrm\{data\}\}and constructs a partially masked sequence𝐳\{\\mathbf\{z\}\}as follows: Samplen∼Unif​\{0,…,L\}n\\sim\\mathrm\{Unif\}\\\{0,\\dots,L\\\}and replace the tokens at uniformly selectednnindices in𝐱\{\\mathbf\{x\}\}with𝐦\\mathbf\{m\}\. Hence, a resulting𝐳\{\\mathbf\{z\}\}hasnn\(randomly drawn\) masked indices\.

This masking procedure induces a joint distribution over\(𝐱,𝐳\)\(\{\\mathbf\{x\}\},\{\\mathbf\{z\}\}\)and we refer to the conditional marginallaw​\(𝐱i∣𝐳\)\\mathrm\{law\}\(\{\\mathbf\{x\}\}^\{i\}\\mid\{\\mathbf\{z\}\}\)as the*unmasking posterior*\. This unmasking posterior is the central object in MDMs and is modeled by a neural networkfθf\_\{\\theta\}that takes𝐳\{\\mathbf\{z\}\}as input and outputs a\|𝒱\|×L\|\\mathcal\{V\}\|\\times Lmatrix\. Concretely, itsii\-th column,fθi\(⋅\|𝐳\)∈Δ\(𝒱\)f\_\{\\theta\}^\{i\}\(\\cdot\\,\|\\,\{\\mathbf\{z\}\}\)\\in\\Delta\(\\mathcal\{V\}\), models the unmasking posteriorfθi​\(v\|𝐳\)≈p​\(𝐱i=v\|𝐳\)f\_\{\\theta\}^\{i\}\(v\\,\|\\,\{\\mathbf\{z\}\}\)\\approx p\(\{\\mathbf\{x\}\}^\{i\}=v\\,\|\\,\{\\mathbf\{z\}\}\)\. To trainfθf\_\{\\theta\}, we minimize cross\-entropy loss summed over all masked indices\.

ℒ​\(θ\):=𝔼𝐱,𝐳​\[1n​∑i:𝐳i=𝐦−log⁡fθi​\(𝐱i\|𝐳\)\]\.\\mathcal\{L\}\(\\theta\)\\colon=\\mathbb\{E\}\_\{\{\\mathbf\{x\}\},\{\\mathbf\{z\}\}\}\\left\[\\frac\{1\}\{n\}\\sum\_\{i\\colon\{\\mathbf\{z\}\}^\{i\}=\\mathbf\{m\}\}\-\\log f\_\{\\theta\}^\{i\}\(\{\\mathbf\{x\}\}^\{i\}\\,\|\\,\{\\mathbf\{z\}\}\)\\right\]\.
MDM inference\.A key feature of MDMs is their*any\-order objective*: a pretrained MDM has learned to predict the posterior of*any masked position*in a sequence\. This any\-order property builds upon earlier work in masked language modeling\(Devlinet al\.,[2019](https://arxiv.org/html/2607.26504#bib.bib8); Ghazvininejadet al\.,[2019](https://arxiv.org/html/2607.26504#bib.bib66); Wang and Cho,[2019](https://arxiv.org/html/2607.26504#bib.bib67)\)\. Next, we explain how this any\-order property translates to the flexibility at inference time\.

MDM inference starts from a length\-LLmasked sequence𝐱1=\(𝐦,…,𝐦\)\{\\mathbf\{x\}\}\_\{1\}=\(\\mathbf\{m\},\\dots,\\mathbf\{m\}\)or generally a given prompt𝐱1=\(\[prompt\],𝐦,…,𝐦\)\{\\mathbf\{x\}\}\_\{1\}=\(\[\\texttt\{prompt\}\],\\mathbf\{m\},\\dots,\\mathbf\{m\}\), and proceeds over a monotonically decreasing time gridt0=1\>⋯\>tN=0t\_\{0\}=1\>\\dots\>t\_\{N\}=0\. At each steptℓt\_\{\\ell\}, given a partially masked sequence𝐱tℓ∈\(𝒱∪\{𝐦\}\)L\{\\mathbf\{x\}\}\_\{t\_\{\\ell\}\}\\in\(\\mathcal\{V\}\\cup\\\{\\mathbf\{m\}\\\}\)^\{L\}, we proceed in two steps to obtain𝐱tℓ\+1\{\\mathbf\{x\}\}\_\{t\_\{\\ell\+1\}\}:\(a\)Choose a subset of masked positions𝒮⊆\{i\|𝐱tℓi=𝐦\}\\mathcal\{S\}\\subseteq\\\{i\\,\|\\,\{\\mathbf\{x\}\}\_\{t\_\{\\ell\}\}^\{i\}=\\mathbf\{m\}\\\}and\(b\)For eachi∈𝒮i\\in\\mathcal\{S\}, unmask𝐱tℓi\{\\mathbf\{x\}\}\_\{t\_\{\\ell\}\}^\{i\}to a clean tokenvvsampled fromv∼fθi\(⋅\|𝐱tℓ\)∈Δ\(𝒱\)v\\sim f\_\{\\theta\}^\{i\}\(\\cdot\\,\|\\,\{\\mathbf\{x\}\}\_\{t\_\{\\ell\}\}\)\\in\\Delta\(\\mathcal\{V\}\)\. Notably, as the MDM training is any\-order, i\.e\.,fθf\_\{\\theta\}predicts the clean token distribution over all masked positions, there is flexibility in the choice of𝒮\\mathcal\{S\}, which is central to the downstream performance\. Below, we describe several strategies for choosing𝒮\\mathcal\{S\}in practice\.

Training\-free methods, includingZhenget al\.\([2024](https://arxiv.org/html/2607.26504#bib.bib10)\); Kimet al\.\([2025c](https://arxiv.org/html/2607.26504#bib.bib15)\); Penget al\.\([2025](https://arxiv.org/html/2607.26504#bib.bib6)\); Ben\-Hamuet al\.\([2025](https://arxiv.org/html/2607.26504#bib.bib21)\); Nieet al\.\([2025](https://arxiv.org/html/2607.26504#bib.bib12)\); Wuet al\.\([2025b](https://arxiv.org/html/2607.26504#bib.bib29)\); Hayakawaet al\.\([2025](https://arxiv.org/html/2607.26504#bib.bib31)\), typically first compute a confidence score for each masked position and select a set𝒮\\mathcal\{S\}of the positions with the highest\-scoring indices, i\.e\.,𝒮←TopKi:𝐱ti=𝐦​\[score​\(i\)\]\\mathcal\{S\}\\leftarrow\\mathrm\{TopK\}\_\{i:\\,\{\\mathbf\{x\}\}\_\{t\}^\{i\}=\\mathbf\{m\}\}\\left\[\\mathrm\{score\}\(i\)\\right\], wherescore​\(i\)\\mathrm\{score\}\(i\)quantifies how certain the model prediction is at positionii\.

Common recipes forscore​\(i\)\\mathrm\{score\}\(i\)include the maximum predicted probabilitymaxv∈𝒱⁡fθi​\(v\|𝐱t\)\\max\_\{v\\in\\mathcal\{V\}\}f\_\{\\theta\}^\{i\}\(v\\,\|\\,\{\\mathbf\{x\}\}\_\{t\}\), the margin between the top two probabilitiesfθi​\(v1\|𝐱t\)−fθi​\(v2\|𝐱t\)f\_\{\\theta\}^\{i\}\(v\_\{1\}\\,\|\\,\{\\mathbf\{x\}\}\_\{t\}\)\-f\_\{\\theta\}^\{i\}\(v\_\{2\}\\,\|\\,\{\\mathbf\{x\}\}\_\{t\}\)\(wherev1,v2v\_\{1\},v\_\{2\}are the top\-2 tokens with the highest predicted probabilities\), and the negative entropy of the categorical distribution\. Some variants further incorporate an explicit position\-dependent bias toward leftmost positions, often referred to as semi\-autoregressive decoding\. We refer to this broad family of training\-free rules as*confidence\-based decoding*\.

Diffusion large language models\.After the advent of MDMs, they have been effectively scaled from 7B\-parameter\(Nieet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib12); Yeet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib60); Xieet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib11); Songet al\.,[2025b](https://arxiv.org/html/2607.26504#bib.bib50); Gonget al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib3)\)to 100B\(Bieet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib26),[2026](https://arxiv.org/html/2607.26504#bib.bib53)\)and industry\-scale\(DeepMind,[2025](https://arxiv.org/html/2607.26504#bib.bib51); Inception–Labet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib52)\)and are often referred to as diffusion language models\. Although the exact details of closed\-source industrial models are not fully disclosed, all open\-source diffusion language models considered in this work instantiate the masked diffusion modeling framework\.

## 3The Interface\-to\-Inference Gap

As discussed in the introduction, many discrete objects, such as code, admit natural non\-left\-to\-right generation orders\. MDMs are appealing candidates for modeling such non\-causal procedures due to their*any\-order*interface\. So far, however, this inference\-time flexibility has primarily been used for efficient parallel decoding\(DeepMind,[2025](https://arxiv.org/html/2607.26504#bib.bib51); Inception–Labet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib52); Wuet al\.,[2025a](https://arxiv.org/html/2607.26504#bib.bib61),[b](https://arxiv.org/html/2607.26504#bib.bib29); Songet al\.,[2025b](https://arxiv.org/html/2607.26504#bib.bib50)\)\. These works show that MDMs can exploit parallelism, but leave open a more fundamental question:*do MDMs actually exploit any\-order flexibility to reason more effectively?*

Prior work gives positive evidence in restricted settings: on small, highly structured domains such as logical puzzles, MDMs can exploit their any\-order interface to follow non\-autoregressive generation orders and outperform autoregressive baselines\(Yeet al\.,[2024a](https://arxiv.org/html/2607.26504#bib.bib13); Kimet al\.,[2025c](https://arxiv.org/html/2607.26504#bib.bib15); Traininet al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib62)\)\. At larger scales, however, recent empirical work paints a less optimistic picture: with confidence\-based decoding, the dominant inference strategy in practice, MDMs exhibit a form of*causal collapse*on standard math and coding benchmarks\. In particular, the token generation order remains close to autoregressive\. These findings suggest that any\-order training alonedoes not guarantee genuinely any\-order inferenceonce MDMs are scaled\.

Comparison to prior work\.It is well known that, despite its flexibility in generation order, confidence\-based MDM sampling often produces less diverse outputs than fixed left\-to\-right sampling with a nonzero token\-sampling temperature\(Niet al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib43); Olaussonet al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib42); Shenet al\.,[2026b](https://arxiv.org/html/2607.26504#bib.bib49); Lamontet al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib44); Fanget al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib46)\)\. Degradation of diversity is harmful in Pass@K evaluation and post\-training, where multiple diverse rollouts provide valuable search coverage or learning signals\.

Most directly,Niet al\.\([2026](https://arxiv.org/html/2607.26504#bib.bib43)\)studies why confidence\-based any\-order decoding has lower entropy than AR\-like decoding\. Their explanation is that the flexibility of any\-order interface allows the model to repeatedly select low\-entropy tokens, thereby committing early to low\-entropy trajectories\. This question, therefore, is distinct from the causal\-collapse phenomenon studied in our work: they ask why confidence\-based decoding is less diverse than autoregressive, whereas we ask why confidence\-based decoding, despite its*any\-order interface*, fails to go meaningfully beyond AR\-like decoding\.

A separate line of work more directly investigates causal collapse by examining why MDM generation orders frequently become left\-to\-right\(Gonget al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib3); Liet al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib41)\)\. These works attribute the phenomenon to causal biases in the pretraining data: under this view, confidence\-based decoding recovers a*natural*ordering already present in the data, which then appears causal\. Although plausible, this explanation remains incomplete\. In particular, it does not readily explain why the same collapse persists in code generation, where Python programs often do not possess a unique inherent causal ordering\.

Our work provides a mechanistic account of causal collapse\. We quantify precisely how closely confidence\-based MDM inference structurally approximates causal decoding on Python programs and investigate why this approximation arises\. We trace the collapse not merely to a preference inherited from the training distribution, but to a more fundamental limitation of token\-level, fixed\-canvas MDM inference:*positional uncertainty*\.

Summary of the results\.In Section[3\.1](https://arxiv.org/html/2607.26504#S3.SS1), we first characterize the causal collapse from a*structural*perspective\. Beyond analyzing the generation order itself, using tree\-based parsing of generated code, we show that confidence\-based MDM decoding produces samples that arestructurally similarto those generated in a strictly left\-to\-right manner, and that MDM’s apparent flexible interface does not translate into inference that exploits the hierarchical structure of programs\. In Section[3\.2](https://arxiv.org/html/2607.26504#S3.SS2), we then formalize*positional uncertainty*as the inevitable bottleneck that arises when token\-level, fixed\-canvas decoding must decide both*what*should be generatedand*where*it should be placed\. We show that this uncertainty persists across different MDMs and tasks\.

### 3\.1Semantic collapse in Python code generation

In this section, we demonstrate that confidence\-based MDM decoding produces programs that are structurally similar to those generated by strict left\-to\-right \(L2R\) decoding\. For each pretrained MDM, we hold the model fixed and vary only the unmasking policy\. The L2R baseline reveals the leftmost remaining masked position at each step, while sampling its token from the same learned posteriorfθi\(⋅∣𝐱t\)f\_\{\\theta\}^\{i\}\(\\cdot\\mid\{\\mathbf\{x\}\}\_\{t\}\)\. We then compare the final programs generated by each non\-L2R policy against the corresponding L2R generations, using a structural similarity metric and best\-match aggregation described below and detailed in Appendix[B](https://arxiv.org/html/2607.26504#A2)\.

![Refer to caption](https://arxiv.org/html/2607.26504v1/x2.png)Figure 2:\(Left\)Confidence\-based decoding yields code structurally close to left\-to\-right decoding\.\(Right\)A simplified AST of the python function that returns\-xifx < 0andxotherwise\.Code similarity metrics\.To compare program structure, we use a standard AST\-based tree\-edit comparison\. We parse each generated program with Python’s built\-in ast module, yielding an abstract syntax tree \(see Figure[2](https://arxiv.org/html/2607.26504#S3.F2)for example\) whose nodes represent syntactic components such as functions, branches, loops, assignments, and returns\. We then compute normalized tree\-edit similarity between ASTs; higher values indicate more similar program structure\. We refer to this score as*abstract syntax tree distance*\(ASTD\) and provide precise definitions, variants, and additional results in Appendix[B](https://arxiv.org/html/2607.26504#A2)\.

Experiment setup\.We take several large\-scale pretrained MDMs, including their base and instruction\- or RL\-fine\-tuned variants; Dream\-7B\-Base/Instruct\(Xieet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib11)\), DiffuCoder\-7B\-Base/Instruct/cpGRPO\(Gonget al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib3)\), and Dream\-Coder\-7B\-Instruct\(Xieet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib11)\)\. On problems from HumanEval\(Chenet al\.,[2021](https://arxiv.org/html/2607.26504#bib.bib23)\), MBPP\(Austinet al\.,[2021](https://arxiv.org/html/2607.26504#bib.bib24)\), and LiveCodeBench\(Jainet al\.,[2024](https://arxiv.org/html/2607.26504#bib.bib7)\), we generate samples with sequence length256256and256256steps, using token sampling temperature0\.20\.2and nucleus sampling withp=0\.95p=0\.95, which exactly match the configurations fromGonget al\.\([2025](https://arxiv.org/html/2607.26504#bib.bib3)\)\. We compare three confidence\-based policies—Top\-KK, Top\-KKmargin, and entropy\-based decoding—against strict L2R decoding\. All generations use temperature 0\.2\. For each prompt, we draw one sample from each confidence\-based policy, since at this low temperature, confidence\-based decoding produces little sample diversity\. In contrast, we draw 32 samples from L2R and random decoding\. For L2R, this gives a reference set for asking whether samples from other policies are already covered by L2R\-biased generation; for random decoding, it provides a calibration baseline for how structurally diverse valid or correct programs can be when the reveal order has no causal bias\. The larger sample budget also offsets the lower valid/correct rate of L2R and random decoding\.

Best\-match aggregation and filters\.Since L2R is our reference for structurally left\-to\-right\-biased generation, we examine whether each sample produced by a given policy has a close counterpart among the L2R samples for the same prompt\. We thus use*best\-match aggregation*: for each generated program, we compute its similarity to the most similar L2R program, and then average these best\-match scores across prompts\. A high score means that the policy mostly discovers structures already covered by L2R decoding; a low score means that it finds programs structurally outside the L2R reference set\. We compute this under two filters: valid–valid, where both programs parse, and success–success, where both programs pass the unit tests\. Appendix[B](https://arxiv.org/html/2607.26504#A2)gives the precise definition\.

Results\.As shown in Figure[2](https://arxiv.org/html/2607.26504#S3.F2), confidence\-based decoding methods \(blue bars\) remain substantially closer to the L2R reference set than the Random\-vs\-L2R baseline \(red bar\)\. This trend holds across models and tasks, as shown in additional results in Appendix[B](https://arxiv.org/html/2607.26504#A2)\. Thus,the programs produced by confidence\-based decoding share the same global skeleton and hierarchical structureas L2R\-generated code\. By contrast, the lower Random\-vs\-L2R similarity shows that the benchmarks admit structurally different valid or correct programs, but confidence\-based MDM decoding tends not to discover them\.

### 3\.2Diagnosing Positional Uncertainty in MDMs

The observation in Section[3\.1](https://arxiv.org/html/2607.26504#S3.SS1)raises the next question about*why*this causal collapse occurs\. In this section, we argue that the key bottleneck lies in the fixed\-canvas, token\-level nature of MDMs\.

Example: the*what*and*where*of a return node\.Consider a simple code snippet with return statements, illustrated in Figure[1](https://arxiv.org/html/2607.26504#S0.F1)\. In many coding tasks, the model may infer early that the program needs a fallback return, such asreturn False\. However, the correct location of this return statement can remain ambiguous before the surrounding structure is resolved: placing it inside a loop, inside a branch, or after the loop can lead to different programs\. Thus, a model may be confident about*what*semantic component should appear—for example, a fallback return node—while remaining uncertain about*where*it should be placed\. In a fixed\-canvas MDM, this uncertainty*spreads*the overall high probability mass for the return node across several plausible locations, resulting in relatively low mass at each of those locations\. Therefore, confidence\-based decoding tends to select nearby local continuations first and collapse toward a left\-to\-right ordering\.

![Refer to caption](https://arxiv.org/html/2607.26504v1/x3.png)
![Refer to caption](https://arxiv.org/html/2607.26504v1/x4.png)

Figure 3:Positional Uncertainty\.Positional uncertainty\.We now formalize the intuition for*positional uncertainty*stated above\. For a tokenvv, we define its*aggregate mass*asm​\(v;𝐱t\)=∑i:𝐱ti=𝐦fθi​\(v\|𝐱t\)m\(v;\{\\mathbf\{x\}\}\_\{t\}\)=\\sum\_\{i:\\,\{\\mathbf\{x\}\}\_\{t\}^\{i\}=\\mathbf\{m\}\}f\_\{\\theta\}^\{i\}\(v\\,\|\\,\{\\mathbf\{x\}\}\_\{t\}\)\. This measures the total probability assigned to tokenvvacross all currently masked positions\. To measure whether this mass is concentrated at a single location or dispersed across many plausible locations, we define the*localization score*

LOC​\(v;𝐱t\):=maxi:𝐱ti=𝐦⁡\[fθi​\(v\|𝐱t\)m​\(v;𝐱t\)\]\.\\mathrm\{LOC\}\(v;\{\\mathbf\{x\}\}\_\{t\}\)\\colon=\\max\_\{i:\\,\{\\mathbf\{x\}\}\_\{t\}^\{i\}=\\mathbf\{m\}\}\\left\[\\frac\{f\_\{\\theta\}^\{i\}\(v\\,\|\\,\{\\mathbf\{x\}\}\_\{t\}\)\}\{m\(v;\{\\mathbf\{x\}\}\_\{t\}\)\}\\right\]\.Thus,LOC​\(v;⋅\)\\mathrm\{LOC\}\(v;\\cdot\)is the largest fraction of tokenvv’s aggregate mass assigned to any one masked position\. High values indicate a clear placement; low values imply dispersal\. The case where the model knows*what to generate but not where to place it*corresponds to a tokenvvwith \(1\) high aggregated massm​\(v;𝐱t\)m\(v;\{\\mathbf\{x\}\}\_\{t\}\)but \(2\) lowLOC​\(v;⋅\)\\mathrm\{LOC\}\(v;\\cdot\)\.

Results\.We examine whether positional uncertainty appears in practice\. For each prompt and each partial sequence𝐱t\{\\mathbf\{x\}\}\_\{t\}, we collect two types of points\. First, to visualize tokens the model assigns high overall probability to, we take the top two tokensvvwith the largest aggregate massm​\(v;𝐱t\)m\(v;\{\\mathbf\{x\}\}\_\{t\}\)and plot\(m​\(v;𝐱t\),LOC​\(v;𝐱t\)\)\(m\(v;\{\\mathbf\{x\}\}\_\{t\}\),\\mathrm\{LOC\}\(v;\{\\mathbf\{x\}\}\_\{t\}\)\)ingreen\. Second, to visualize what the model actually commits, for each revealed tokenvvat positionii, we plot its aggregate mass together with its committed\-position localizationfθi​\(v\|𝐱t\)/m​\(v;𝐱t\)f\_\{\\theta\}^\{i\}\(v\\,\|\\,\{\\mathbf\{x\}\}\_\{t\}\)/m\(v;\{\\mathbf\{x\}\}\_\{t\}\)inred\. We aggregate these points across prompts and provide additional plots in Appendix[C](https://arxiv.org/html/2607.26504#A3)\.

The resulting pattern in Figure[3](https://arxiv.org/html/2607.26504#S3.F3)supports our claim\. Many high\-mass tokens lie in thelower\-rightregion of the plot, indicating that the model assigns them substantial total probability, but does not decode them due to low localization\. In contrast, committed tokens tend to have lower aggregate mass but higher localization, appearing toward theupper\-leftregion\.

Similarity analysis for learned decoding strategies\.A natural follow\-up question is whether this bottleneck can be mitigated by post\-training the unmasking policy of an MDM, as explored in\(Honget al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib27); Jazbecet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib20); Chenet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib28); Yeet al\.,[2024b](https://arxiv.org/html/2607.26504#bib.bib75)\)\. To examine this possibility, we take a checkpoint fromChenet al\.\([2025](https://arxiv.org/html/2607.26504#bib.bib28)\), which trains an unmasking policy model with reinforcement learning\. We then repeat our similarity analysis using the planner\-induced unmasking policy\. Although the learned planner changes the distribution over decoding orders, our results in Appendix[B\.2](https://arxiv.org/html/2607.26504#A2.SS2)show that the same trend persists: generated programs remain structurally close to those produced by strict left\-to\-right decoding\. This suggests that, within fixed\-canvas token\-level MDMs, learning a decoding policy alone may be insufficient to escape causal collapse\.

We do not claim this as an exhaustive evaluation of all planner\-learning methods, since some of the public checkpoints are not available\. Nevertheless, this result supports our broader view that positional uncertainty is a more fundamental limitation of fixed\-canvas token\-level MDMs for enabling genuine any\-order inference, rather than merely a weakness of a particular confidence score\.

## 4Circumventing Positional Uncertainty

We identified positional uncertainty as a key bottleneck of MDMs\. This suggests that genuine any\-order inference may require relaxing at least one of two aspects of MDM generation:*token\-level*and*fixed\-canvas*\. In this section, we present two alternative generative models that each relax one of these aspects\. We first summarize their modeling principles, then provide detailed descriptions and empirical evidence in Section[4\.1](https://arxiv.org/html/2607.26504#S4.SS1)and Section[4\.2](https://arxiv.org/html/2607.26504#S4.SS2)\.

FlexMDM: relaxing fixed\-canvas\.Our first approach is to employ FlexMDM\(Kimet al\.,[2025a](https://arxiv.org/html/2607.26504#bib.bib14)\), an insertion\-based masked diffusion model that not only decodes mask tokens but also inserts new masks during inference\. This relaxes positional uncertainty since revealing a token no longer irrevocably commits it to a fixed position: later insertions can shift its location in the final sequence\.

This relaxation induces a structural form of*any\-order inference*\. When we view generated code through its parsed algorithmic tree, we find that FlexMDM generation moves back and forth across different semantic nodes, rather than completing one contiguous region before moving to the next, thereby going beyond left\-to\-right generation\. In this sense, insertion gives a mechanism for drafting multiple semantic regions and refining them later, closer to how human programmers develop code\.

LatentMDM: relaxing token\-level\.Our second approach moves the masked\-diffusion interface from individual tokens to segment\-level latent variables\. We split each sequence into coarse segments, such as lines of code, and jointly train a lightweight encoder\-decoder pair that maps between segments and latent vectors, together with a masked diffusion model over the sequence of segment latents\.

At inference time, the resulting LatentMDM chooses which*masked latent segment*to reveal next, and the decoder realizes the chosen latent as tokens\. This switches the object of any\-order inference from token positions to latent segments\. By absorbing token\-level positional uncertainty into segment\-level latent variables, the revealed confidence becomes less tied to local token\-position certainty and more tied to the compatibility of higher\-level semantic components\.Thus, the form of any\-order reasoning realized by LatentMDM is semantic\-order search in a more compact exploration space:deciding which high\-level latent segment to instantiate next\.

Experiments\.We present a suite of experiments to answer the following questions:

Q1\.*Do our solutions mitigate causal collapse by relaxing positional uncertainty?*

Q2\.*What new forms of any\-order inference do our solutions enable?*

### 4\.1FlexMDM

Now we describe the training and inference procedures of FlexMDM and present empirical results obtained by fine\-tuning Dream\-Coder 7B\(Xieet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib11)\)\.

Training\.To enable insertion during training, FlexMDM augments the standard MDM masking procedure \(recall Section[2](https://arxiv.org/html/2607.26504#S2)\) with*deletion*\. As a result, during training, FlexMDM observes not only masked but also deleted versions of sequences𝐱∼pdata\{\\mathbf\{x\}\}\\sim p\_\{\\mathrm\{data\}\}\. The masking and deletion rates are determined by two schedules: an unmasking and an insertion schedule\. Given a corrupted sequence𝐳\{\\mathbf\{z\}\}, a FlexMDM is trained to jointly predict the posterior distribution over masked tokens and the expected number of tokens to insert, conditioned on the current sequence𝐳\{\\mathbf\{z\}\}\.

Inference\.At inference time, given a trained FlexMDM and the corresponding schedules, generation proceeds by alternating between two operations: \(1\) selecting which masked tokens to reveal according to the unmasking schedule and model’s predicted unmasking posterior, and \(2\) inserting additional mask tokens according to the model’s predicted insertion counts\. Importantly, FlexMDM retains the any\-order flexibility of standard MDM: for unmasking, one can still use confidence\-based decoding to decide which masked tokens to reveal\. For insertion, we sample the number of mask tokens to insert from a Poisson distribution whose rate is determined by the training\-time insertion schedule together with the model’s prediction\. Like unmasking, insertion carries its own inference\-time temperature: the per\-gap insertion rates factorize into a total count and a placement distribution, and tempering only the placement controls*where*new masks open while preserving the expected generation length \(Appendix[D\.5](https://arxiv.org/html/2607.26504#A4.SS5)\)\. We defer technical details to Appendix[D\.1](https://arxiv.org/html/2607.26504#A4.SS1)\.

![Refer to caption](https://arxiv.org/html/2607.26504v1/x5.png)

Figure 4:\(Left\)We quantify any\-order inference by tracking how generation moves across the tree\.\(Right\)FlexMDM outperforms Dream\-Coder 7B on HumanEval and HumanEval\+, at Pass@16\.Experimental setup\.Rather than training FlexMDM from scratch, we initialize from Dream\-Coder 7B\(Xieet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib11)\), while attaching an auxiliary head and an AdaLN time embedding\(Peebles and Xie,[2023](https://arxiv.org/html/2607.26504#bib.bib70)\)for insertion prediction\. We construct the fine\-tuning corpus by post\-processing Python data from OpenCodeInstruct\(Ahmadet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib71)\), rStar\-Coder\(Liuet al\.,[2025b](https://arxiv.org/html/2607.26504#bib.bib72)\), KodCode\(Xuet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib73)\), and opc\-stage\-2\-educational\(Huanget al\.,[2025a](https://arxiv.org/html/2607.26504#bib.bib74)\), resulting in≈\\approx2\.6M training sequences\. We then fine\-tune the model using the FlexMDM training loss for≈\\approx1150 GPU hours\.

Results\.FlexMDM by construction addressesQ1by decoupling unmasking from fixed\-position commitment\. This relaxation induces an*any\-order inference behavior*\(Q2\): FlexMDM generates drafts of non\-contiguous regions of the generated code\. We quantify this*any\-order inference*by parsing generated code into an AST \(as in Section[3\.1](https://arxiv.org/html/2607.26504#S3.SS1)\) and tracking how generation moves across the tree\. We use three complementary metrics: Coverage Before Commitment \(CBC\), the average number of sibling nodes previewed before completing the first child block; Return to Unfinished Blocks \(RUB\), which measures back\-and\-forth revisits to unfinished nodes; and Open\-Block Width \(OBW\), the maximum number of sibling blocks kept open simultaneously\. FlexMDM yields substantially higher values than Dream\-Coder’s standard inference across all metrics \(Figure[4](https://arxiv.org/html/2607.26504#S4.F4), left\), supporting our claim of achieving any\-order inference with insertion\-based models: they generate code across non\-contiguous semantic components, whereas an MDM with confidence\-based decoding largely collapses to an autoregressive order\. We defer precise definitions and additional results to Appendix[D\.3](https://arxiv.org/html/2607.26504#A4.SS3)\.

Next, we test whether this any\-order behavior translates into downstream gains\. On HumanEval and HumanEval\+, FlexMDM outperforms Dream\-Coder, our primary baseline, at Pass@16 \(Figure[4](https://arxiv.org/html/2607.26504#S4.F4), right\)\. This suggests that any\-order inference may help by enabling more diverse algorithmic solution paths\. On MBPP and MBPP\+, lowering the insertion temperature to0\.60\.6puts FlexMDM ahead of Dream\-Coder at Pass@1, with overall performance matching the baseline \(Table[5](https://arxiv.org/html/2607.26504#A4.T5)\); Appendix[D\.5](https://arxiv.org/html/2607.26504#A4.SS5)ablates this knob\.

### 4\.2LatentMDM

In this section, we explain the training and inference recipes of LatentMDM, and present empirical results on pre\-training a 125M\-scale LatentMDM on TinyGSM\(Liuet al\.,[2023](https://arxiv.org/html/2607.26504#bib.bib64)\)\.

Training\.We partition each sequence𝐱\{\\mathbf\{x\}\}into at mostLsL\_\{s\}\(variable\-length\) segments,𝐱=\(𝐲1,…,𝐲Ls\)\{\\mathbf\{x\}\}=\(\{\\mathbf\{y\}\}^\{1\},\\dots,\{\\mathbf\{y\}\}^\{L\_\{s\}\}\), where each segment𝐲i\{\\mathbf\{y\}\}^\{i\}belongs to the segment space𝒴\\mathcal\{Y\}\. An encoderEϕ:𝒴→ℝHE\_\{\\phi\}:\\mathcal\{Y\}\\rightarrow\\mathbb\{R\}^\{H\}maps each segment to a fixed\-dimensional latent vector; therefore, a full sequence is represented as a tensor inℝLs×H\\mathbb\{R\}^\{L\_\{s\}\\times H\}\. We train a masked diffusion model in this latent space by applying masking*at the segment level*\. Concretely, we draw a partially masked segment sequence𝐳=\(𝐳1,…,𝐳Ls\)\{\\mathbf\{z\}\}=\(\{\\mathbf\{z\}\}^\{1\},\\dots,\{\\mathbf\{z\}\}^\{L\_\{s\}\}\)with masked segment indicesℳ​\(𝐳\)⊆\{1,…,Ls\}\\mathcal\{M\}\(\{\\mathbf\{z\}\}\)\\subseteq\\\{1,\\dots,L\_\{s\}\\\}\. For eachi∈ℳ​\(𝐳\)i\\in\\mathcal\{M\}\(\{\\mathbf\{z\}\}\), the segment is replaced by𝐦\\mathbf\{m\}, while unmasked segments remain unchanged, i\.e\.,𝐳i=𝐲i\{\\mathbf\{z\}\}^\{i\}=\{\\mathbf\{y\}\}^\{i\}fori∉ℳ​\(𝐳\)i\\notin\\mathcal\{M\}\(\{\\mathbf\{z\}\}\)\. The latent input is then formed by encoding unmasked segments withEϕE\_\{\\phi\}and replacing each masked segment with a learnable mask embedding𝐞𝐦∈ℝH\\mathbf\{e\}\_\{\\mathbf\{m\}\}\\in\\mathbb\{R\}^\{H\}\.

The LatentMDM, parameterized asfθ:ℝLs×H→ℝLs×H′f\_\{\\theta\}:\\mathbb\{R\}^\{L\_\{s\}\\times H\}\\to\\mathbb\{R\}^\{L\_\{s\}\\times H^\{\\prime\}\}, takes this partially masked latent sequence, conditioned on the prompt, and predicts a latent vector inℝH′\\mathbb\{R\}^\{H^\{\\prime\}\}for every masked segment\. The LatentMDM preserves and projects each segment intoℝH′\\mathbb\{R\}^\{H^\{\\prime\}\}, which serves as the conditioning space for the decoder\. Finally, an*autoregressive*decoderDψD\_\{\\psi\}reconstructs each latent representation inℝH′\\mathbb\{R\}^\{H^\{\\prime\}\}token by token\. The training objective is the average autoregressive cross\-entropy loss over the masked segments:

ℒ​\(θ,ϕ,ψ\)=𝔼𝐱,𝐳​\[1\|ℳ​\(𝐳\)\|​∑i∈ℳ​\(𝐳\)∑j=1len​\(𝐲i\)−log⁡Dψ​\(𝐲ji∣𝐲<ji,𝐡θ,ϕi​\(𝐳\)\)\],\\mathcal\{L\}\(\\theta,\\phi,\\psi\)=\\mathbb\{E\}\_\{\{\\mathbf\{x\}\},\{\\mathbf\{z\}\}\}\\left\[\\frac\{1\}\{\|\\mathcal\{M\}\(\{\\mathbf\{z\}\}\)\|\}\\sum\_\{i\\in\\mathcal\{M\}\(\{\\mathbf\{z\}\}\)\}\\sum\_\{j=1\}^\{\\mathrm\{len\}\(\{\\mathbf\{y\}\}^\{i\}\)\}\-\\log D\_\{\\psi\}\\\!\\left\(\{\\mathbf\{y\}\}^\{i\}\_\{j\}\\mid\{\\mathbf\{y\}\}^\{i\}\_\{<j\},\\mathbf\{h\}\_\{\\theta,\\phi\}^\{i\}\(\{\\mathbf\{z\}\}\)\\right\)\\right\],
where𝐡θ,ϕ​\(𝐳\)∈ℝLs×H′\\mathbf\{h\}\_\{\\theta,\\phi\}\(\{\\mathbf\{z\}\}\)\\in\\mathbb\{R\}^\{L\_\{s\}\\times H^\{\\prime\}\}denotes the latent conditioning produced by LatentMDM,len​\(𝐲i\)\\mathrm\{len\}\(\{\\mathbf\{y\}\}^\{i\}\)denotes the length of𝐲i\{\\mathbf\{y\}\}^\{i\}, and the expectation is taken over the masking procedure defined above\. Importantly, we use an autoregressive decoder for local segment sampling, since next\-token prediction modeling remains effective for local sequence modeling\(Kimet al\.,[2025c](https://arxiv.org/html/2607.26504#bib.bib15)\)\.

Inference\.At inference time, the model is given only the prompt and maintains a partially filled segment sequence𝐳\{\\mathbf\{z\}\}\. Initially, all segment positions are masked\. At each iteration, the revealed segments are encoded byEϕE\_\{\\phi\}, while masked positions are represented by the learned mask embedding\. The LatentMDM then predicts a contextual latent𝐡θ,ϕi​\(𝐳\)\\mathbf\{h\}\_\{\\theta,\\phi\}^\{i\}\(\{\\mathbf\{z\}\}\)for each masked segment positionii\.

The key design knob is which segment to reveal next: the autoregressive decoderDψD\_\{\\psi\}tentatively generates a candidate segment𝐲^i\\hat\{\{\\mathbf\{y\}\}\}^\{i\}token\-by\-token from𝐡θ,ϕi​\(𝐳\)\\mathbf\{h\}\_\{\\theta,\\phi\}^\{i\}\(\{\\mathbf\{z\}\}\)\. This tentative decoding is performed in parallel across all masked segments\. We score each candidate by its length\-normalized token log\-likelihood underDψD\_\{\\psi\}, commit the highest\-scoring segment, re\-encode it withEϕE\_\{\\phi\}, and update the corresponding masked position in𝐳\{\\mathbf\{z\}\}for the next iteration\. Thus, the inference\-time decoding strategy is driven by*segment\-level confidence*, which we view as*any\-order inference*over semantic space: the model reveals the latent segment whose prediction can be most confidently realized\.

![Refer to caption](https://arxiv.org/html/2607.26504v1/x6.png)Figure 5:\(Left\)LatentMDM outperforms autoregressive and other baselines at matched wall\-clock time\.\(Right\)It mitigates positional uncertainty, decoding tokens with high aggregate mass\.Experimental setup\.Since LatentMDM introduces fundamentally different modeling from MDMs, we had to train it from scratch\. In particular, we target a setup where \(1\) models pretrained from scratch can achieve reasonable performance on downstream benchmarks, e\.g\., GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.26504#bib.bib22)\), while \(2\) the compute budget remains feasible for running several controlled experiments\. We adopt TinyGSM\(Liuet al\.,[2023](https://arxiv.org/html/2607.26504#bib.bib64)\)as the pretraining corpus\. TinyGSM converts GSM8K\-style natural\-language solutions into structured Python programs, making the task substantially more learnable at small model scales\. We train LatentMDM and all baselines from scratch, using the same 125M\-parameter scale with 500k optimizer steps and a global batch size of 256\.

A1\.To answerQ1, we revisit thepositional\-uncertaintydiagnosis in Section[3\.2](https://arxiv.org/html/2607.26504#S3.SS2)\. We examine the same mass\-localization space defined bym​\(v;𝐱t\)m\(v;\{\\mathbf\{x\}\}\_\{t\}\)andLOC​\(v;𝐱t\)\\mathrm\{LOC\}\(v;\{\\mathbf\{x\}\}\_\{t\}\)\. Figure[5](https://arxiv.org/html/2607.26504#S4.F5)\(right\) shows that LatentMDM commits a large fraction of high\-mass but low\-localization tokens \(brown\)\. This contrasts with MDM, where tokens in the same region remain undecoded \(Figure[3](https://arxiv.org/html/2607.26504#S3.F3)\)\. This demonstrates that moving the any\-order interface from token positions to segment\-level latents relaxes the positional bottleneck of fixed\-canvas MDMs\.

A2\.To answerQ2, we scrutinize whether LatentMDM enables a useful form of any\-order inference over semantic segments, in particular on GSM8K\. We compare LatentMDM against representative discrete generative models: masked diffusion models \(MDMs\), autoregressive models, hyperspherical flow language model\(Deschenaux and Gulcehre,[2026](https://arxiv.org/html/2607.26504#bib.bib59)\)\(𝕊\\mathbb\{S\}\-FLM\), and DUO\(Sahooet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib80)\)\.111We omit Flow Map Language ModelsLeeet al\.\([2026](https://arxiv.org/html/2607.26504#bib.bib77)\), asDeschenaux and Gulcehre \([2026](https://arxiv.org/html/2607.26504#bib.bib59)\)benchmark against it under an almost identical experimental setup\.

For hyperspherical flow language model and DUO, we largely follow the implementation details in the codebase ofDeschenaux and Gulcehre \([2026](https://arxiv.org/html/2607.26504#bib.bib59)\)\. Both methods support few\-step sampling, which reduces latency but may come at the cost of lower accuracy\. We therefore evaluate their latency–accuracy trade\-offs across sampling steps in\{8,16,…,512\}\\\{8,16,\\ldots,512\\\}and report the Pareto\-optimal configurations: 16 steps for Spherical Flow Map Language Models and 8 steps for DUO\. Moreover, we include two decoding\-policy variants with the same LatentMDM: LatentMDM\-random, which selects segments uniformly at random, and LatentMDM\-L2R, which reveals segments in left\-to\-right order\.

Fig\.[5](https://arxiv.org/html/2607.26504#S4.F5)\(left\) reports pass@K evaluation results, together with the corresponding wall\-clock decoding time\. The results show that LatentMDM consistently outperforms both LatentMDM\-L2R and LatentMDM\-random, by more than 6%\\%\. Since these variants differ only in their segment\-selection policy, this result shows that the gain does not come merely from latent modeling, but from the ability to*choose which semantic segment to reveal next*, indicating that LatentMDM performs genuinely any\-order decoding over segments\.

We next compare against modeling baselines\. Under a matched wall\-clock budget, LatentMDM outperforms all other models by a gap of 5\.5%\\%at a sampling batch size of11\. This efficiency arises because one LatentMDM inference pass takes roughly half the latency of an autoregressive model, even with KV caching, as LatentMDM first samples latent segments, which are then decoded by a lightweight decoder\. The gain further suggests that compact latent\-space search preserves useful exploration, in contrast to token\-level confidence decoding, where confidence over\-selects locally easy positions and compresses sample diversity\. We defer experimental details and additional results to Appendix[E](https://arxiv.org/html/2607.26504#A5)\.

Together, these results support our central claim: moving the any\-order interface from token positions to latent segmentsrelaxes the positional bottleneck, thus enabling semantic\-order search through a distinct and practically advantageous form of any\-order inference\.

Advances over prior work\.For FlexMDM, we advance both the theory and the empirical scope ofKimet al\.\([2025a](https://arxiv.org/html/2607.26504#bib.bib14)\)\. Theoretically, we show that the schedules can be changed at inference time without retraining via a time reparameterization trick, yielding further flexibility at inference time \(Appendix[D\.1](https://arxiv.org/html/2607.26504#A4.SS1)\)\. We further derive a count\-preserving*insertion temperature*— an insertion\-side analogue of the token temperature that gives flexible inference\-time control over the structural stochasticity of insertion \(Appendix[D\.5](https://arxiv.org/html/2607.26504#A4.SS5)\)\. We also reformulate insertion\-length prediction in log space, which is geometrically more natural and leads to more stable training\. Empirically, we move beyond the task\-specific LoRA fine\-tuning and fully fine\-tune Dream\-Coder\-7B into a general\-purpose Python FlexMDM\. Thus, in our work, FlexMDM serves both as a theoretically extended model and a repurposed empirical vehicle for investigating how insertion can elicit genuinely any\-order generation\. For LatentMDM, we note that prior work has not studied masked diffusion over semantic latent spaces; we provide a detailed discussion of related work in Appendix[A](https://arxiv.org/html/2607.26504#A1)\.

Discussion\.FlexMDM and LatentMDM relax the positional bottleneck in complementary ways rather than serving as competing designs\. FlexMDM stays in the token space, making it easier to*retrofit*from a pretrained MDM\. LatentMDM enables segment\-level order search but typically requires training from scratch\. Combining insertion\-based refinement with latent\-space order search may realize the benefits of both forms of any\-order inference; we leave this direction to future work\.

## 5Conclusion

Our two approaches, FlexMDM and LatentMDM, relax the fixed\-canvas, token\-level bottleneck of MDMs and thereby enable distinct forms of any\-order inference\. These approaches, however, also introduce trade\-offs\. More flexible inference procedures require additional design choices and may require extra care to ensure training and sampling stability\. Moreover, our empirical study focuses primarily on Python code generation, leaving open whether similar forms of any\-order inference transfer to broader domains, modalities, and scales\.

Beyond masked\-token diffusion, recent approaches based on continuous\-state modeling or uniform\-state diffusions\(von Rütteet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib79); Sahooet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib80); Leeet al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib77); Rooset al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib78); Potaptchiket al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib76)\)may also support non\-causal generation in principle\. However, they lack the explicit any\-order interface found in mask\-based models, leaving the elicitation of genuine any\-order inference in these alternative diffusion frameworks as an important open question\.222We note that very recently,Agarwalet al\.\([2026](https://arxiv.org/html/2607.26504#bib.bib4)\)proposed training an any\-order model in a continuous embedding space for discrete generation, which enables any\-order inference\.Another important direction is to investigate whether equipping masked diffusion models with self\-correction mechanisms\(Wanget al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib58); Kimet al\.,[2025b](https://arxiv.org/html/2607.26504#bib.bib56); Huanget al\.,[2025b](https://arxiv.org/html/2607.26504#bib.bib57); Schiffet al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib55)\)can elicit any\-order inference behavior\.

Outlook\.We, the authors, want the paper to be viewed as a*foundational work*for generative modeling on discrete spaces, asking whether a model actually realizes the capabilities suggested by its modeling design\. Recent progress in generative modeling has largely been driven by architectures and training procedures that scale effectively\. Our work asks whether scalability must be the sole organizing principle for generative model design, or whether models can be designed so that their inference procedures better reflect the structure of the objects they generate\. This perspective treats the structure of inference as a central modeling choice rather than a fixed sampling procedure, opening up a broader design space for models that more naturally support planning, refinement, and reasoning\.

## Acknowledgements

JK thanks Jiaxin Shi for insightful discussions that inspired this work\. JK is also grateful to Kiwhan Song for sharing the idea of any\-order models in continuous spaces\. SK thanks Minkai Xu for insightful discussions and valuable feedback on this work\. JK and YC thank Brian Lee and Michael Albergo for fruitful discussions on FlexMDM and parameterization via insertion schedules\. JK acknowledges support from the Kempner Institute\. SK and TL acknowledge support from the Texas Advanced Computing Center\. JK and YC are grateful to the Kempner Institute for providing the compute\. SC is supported in part by NSF CAREER award CCF2441635\.

## Contribution statement

SK, JK, TL, and YC are co\-first authors, and we include this contribution statement solely for clarity\. While all authors contributed to every aspect of the project, the primary leads for each component are as follows: Preliminary idea and paper presentation: JK, Similarity analysis: SK, TL, LatentMDM: SK, FlexMDM: JK, YC\.

## References

- M\. Agarwal, S\. Shah, C\. Lee, J\. Yoo, J\. Huang, S\. Hong, A\. Raghunathan, J\. Kim, and N\. M\. Boffi \(2026\)Posterior refinement: fast language generation via any\-order flow maps\.arXiv preprint arXiv:2606\.24773\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p9.1),[footnote 2](https://arxiv.org/html/2607.26504#footnote2)\.
- OpenCodeInstruct: a large\-scale instruction tuning dataset for code LLMs\.arXiv preprint arXiv:2504\.04030\.Cited by:[§D\.1](https://arxiv.org/html/2607.26504#A4.SS1.SSS0.Px2.p5.1),[§4\.1](https://arxiv.org/html/2607.26504#S4.SS1.p4.2)\.
- J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le,et al\.\(2021\)Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[Appendix C](https://arxiv.org/html/2607.26504#A3.p1.1),[§1](https://arxiv.org/html/2607.26504#S1.p4.3),[§3\.1](https://arxiv.org/html/2607.26504#S3.SS1.p3.6)\.
- L\. Barrault, P\. Duquenne, M\. Elbayad, A\. Kozhevnikov, B\. Alastruey, P\. Andrews, M\. Coria, G\. Couairon, M\. R\. Costa\-jussà, D\. Dale,et al\.\(2024\)Large concept models: language modeling in a sentence representation space\.arXiv preprint arXiv:2412\.08821\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p10.1)\.
- G\. Batzolis, M\. Girolami, and L\. Ambrogioni \(2026\)CoBit: language modeling with bitstream diffusion\.arXiv preprint arXiv:2605\.07013\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p9.1)\.
- M\. Bavarian, H\. Jun, N\. Tezak, J\. Schulman, C\. McLeavey, J\. Tworek, and M\. Chen \(2022\)Efficient training of language models to fill in the middle\.arXiv preprint arXiv:2207\.14255\.Cited by:[§1](https://arxiv.org/html/2607.26504#S1.p1.1)\.
- H\. Ben\-Hamu, I\. Gat, D\. Severo, N\. Nolte, and B\. Karrer \(2025\)Accelerated sampling from masked diffusion models via entropy bounded unmasking\.arXiv preprint arXiv:2505\.24857\.Cited by:[§E\.4](https://arxiv.org/html/2607.26504#A5.SS4.p1.1),[§2](https://arxiv.org/html/2607.26504#S2.p8.4)\.
- T\. Bie, M\. Cao, X\. Cao, B\. Chen, F\. Chen, K\. Chen, L\. Du, D\. Feng, H\. Feng, M\. Gong,et al\.\(2026\)LLaDA2\.1: speeding up text diffusion via token editing\.arXiv preprint arXiv:2602\.08676\.Cited by:[§2](https://arxiv.org/html/2607.26504#S2.p10.1)\.
- T\. Bie, M\. Cao, K\. Chen, L\. Du, M\. Gong, Z\. Gong, Y\. Gu, J\. Hu, Z\. Huang, Z\. Lan, C\. Li, C\. Li, J\. Li, Z\. Li, H\. Liu, L\. Liu, G\. Lu, X\. Lu, Y\. Ma, J\. Tan, L\. Wei, J\. Wen, Y\. Xing, X\. Zhang, J\. Zhao, D\. Zheng, J\. Zhou, J\. Zhou, Z\. Zhou, L\. Zhu, and Y\. Zhuang \(2025\)LLaDA2\.0: scaling up diffusion language models to 100B\.External Links:2512\.15745,[Link](https://arxiv.org/abs/2512.15745)Cited by:[§1](https://arxiv.org/html/2607.26504#S1.p2.1),[§2](https://arxiv.org/html/2607.26504#S2.p10.1)\.
- B\. Chen, D\. Martí Monsó, Y\. Du, M\. Simchowitz, R\. Tedrake, and V\. Sitzmann \(2024\)Diffusion forcing: next\-token prediction meets full\-sequence diffusion\.Advances in Neural Information Processing Systems37,pp\. 24081–24125\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p14.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. Zaremba \(2021\)Evaluating large language models trained on code\.External Links:2107\.03374Cited by:[Appendix C](https://arxiv.org/html/2607.26504#A3.p1.1),[§1](https://arxiv.org/html/2607.26504#S1.p4.3),[§3\.1](https://arxiv.org/html/2607.26504#S3.SS1.p3.6)\.
- S\. Chen, J\. Jiao, L\. J\. Ratliff, and B\. Zhu \(2025\)dUltra: ultra\-fast diffusion language models via reinforcement learning\.arXiv preprint arXiv:2512\.21446\.Cited by:[§B\.2](https://arxiv.org/html/2607.26504#A2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2607.26504#S3.SS2.p6.1)\.
- Y\. Chen, C\. Liang, H\. Sui, R\. Guo, C\. Cheng, J\. You, and G\. Liu \(2026\)LangFlow: continuous diffusion rivals discrete in language modeling\.arXiv preprint arXiv:2604\.11748\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p8.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4\.2](https://arxiv.org/html/2607.26504#S4.SS2.p8.1)\.
- B\. Dai, Y\. Liu, D\. Xue, Y\. Song, Q\. Guo, K\. Chen, X\. Wang, B\. Zhou, and Z\. Lin \(2025\)Context\-level language modeling by learning predictive context embeddings\.arXiv preprint arXiv:2510\.20280\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p6.1)\.
- G\. DeepMind \(2025\)Gemini diffusion\.External Links:[Link](https://blog.google/technology/google-deepmind/gemini-diffusion/)Cited by:[§2](https://arxiv.org/html/2607.26504#S2.p10.1),[§3](https://arxiv.org/html/2607.26504#S3.p1.1)\.
- J\. Deschenaux and C\. Gulcehre \(2026\)Language modeling with hyperspherical flows\.arXiv preprint arXiv:2605\.11125\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p9.1),[§1](https://arxiv.org/html/2607.26504#S1.p5.1),[§4\.2](https://arxiv.org/html/2607.26504#S4.SS2.p10.1),[§4\.2](https://arxiv.org/html/2607.26504#S4.SS2.p11.1),[footnote 1](https://arxiv.org/html/2607.26504#footnote1)\.
- J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova \(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 \(long and short papers\),pp\. 4171–4186\.Cited by:[§2](https://arxiv.org/html/2607.26504#S2.p6.1)\.
- L\. Fang, A\. Liu, H\. P\. Zou, Y\. Chen, E\. Ma, L\. Pan, C\. Miao, W\. Huang, X\. Liu, and P\. S\. Yu \(2026\)Locally confident, globally stuck: the quality\-exploration dilemma in diffusion language models\.arXiv preprint arXiv:2604\.00375\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p1.1),[Appendix A](https://arxiv.org/html/2607.26504#A1.p2.1),[§3](https://arxiv.org/html/2607.26504#S3.p3.1)\.
- M\. Ghazvininejad, O\. Levy, Y\. Liu, and L\. Zettlemoyer \(2019\)Mask\-predict: parallel decoding of conditional masked language models\.InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing \(EMNLP\-IJCNLP\),pp\. 6112–6121\.Cited by:[§2](https://arxiv.org/html/2607.26504#S2.p6.1)\.
- S\. Gong, R\. Zhang, H\. Zheng, J\. Gu, N\. Jaitly, L\. Kong, and Y\. Zhang \(2025\)DiffuCoder: understanding and improving masked diffusion models for code generation\.arXiv preprint arXiv:2506\.20639\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p1.1),[Appendix A](https://arxiv.org/html/2607.26504#A1.p4.1),[Appendix C](https://arxiv.org/html/2607.26504#A3.p1.1),[§1](https://arxiv.org/html/2607.26504#S1.p2.1),[§1](https://arxiv.org/html/2607.26504#S1.p3.1),[§2](https://arxiv.org/html/2607.26504#S2.p10.1),[§3\.1](https://arxiv.org/html/2607.26504#S3.SS1.p3.6),[§3](https://arxiv.org/html/2607.26504#S3.p5.1)\.
- D\. Gwak, M\. Jung, J\. Park, M\. Park, C\. Park, J\. Hyung, and J\. Choo \(2025\)Reward\-weighted sampling: enhancing non\-autoregressive characteristics in masked diffusion LLMs\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 34562–34582\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p4.1)\.
- S\. Hao, S\. Sukhbaatar, D\. Su, X\. Li, Z\. Hu, J\. Weston, and Y\. Tian \(2024\)Training large language models to reason in a continuous latent space\.arXiv preprint arXiv:2412\.06769\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p6.1)\.
- S\. Hayakawa, Y\. Takida, M\. Imaizumi, H\. Wakaki, and Y\. Mitsufuji \(2025\)Demystifying maskgit sampler and beyond: adaptive order selection in masked diffusion\.arXiv preprint arXiv:2510\.04525\.Cited by:[§E\.4](https://arxiv.org/html/2607.26504#A5.SS4.p1.1),[§2](https://arxiv.org/html/2607.26504#S2.p8.4)\.
- M\. Hersche, N\. Menet, R\. Tanios, and A\. Rahimi \(2026\)Locally coherent parallel decoding in diffusion language models\.arXiv preprint arXiv:2603\.20216\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p13.1)\.
- C\. Hong, S\. An, M\. Kim, and J\. C\. Ye \(2025\)Improving discrete diffusion unmasking policies beyond explicit reference policies\.arXiv preprint arXiv:2510\.05725\.Cited by:[§3\.2](https://arxiv.org/html/2607.26504#S3.SS2.p6.1)\.
- K\. Hu, L\. Qiu, Y\. Lu, H\. Zhao, T\. Li, Y\. Kim, J\. Andreas, and K\. He \(2026\)Elf: embedded language flows\.arXiv preprint arXiv:2605\.10938\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p9.1)\.
- S\. Huang, T\. Cheng, J\. K\. Liu, W\. Xu, J\. Hao, L\. Song, Y\. Xu, J\. Yang, J\. Liu, C\. Zhang,et al\.\(2025a\)OpenCoder: the open cookbook for top\-tier code large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 33167–33193\.Cited by:[§D\.1](https://arxiv.org/html/2607.26504#A4.SS1.SSS0.Px2.p5.1),[§4\.1](https://arxiv.org/html/2607.26504#S4.SS1.p4.2)\.
- Z\. Huang, Y\. Wang, Z\. Chen, and G\. Qi \(2025b\)Don’t settle too early: self\-reflective remasking for diffusion language models\.arXiv preprint arXiv:2509\.23653\.Cited by:[§5](https://arxiv.org/html/2607.26504#S5.p2.1)\.
- Inception–Lab, S\. Khanna, S\. Kharbanda, S\. Li, H\. Varma, E\. Wang, S\. Birnbaum, Z\. Luo, Y\. Miraoui, A\. Palrecha,et al\.\(2025\)Mercury: ultra\-fast language models based on diffusion\.arXiv preprint arXiv:2506\.17298\.Cited by:[§2](https://arxiv.org/html/2607.26504#S2.p10.1),[§3](https://arxiv.org/html/2607.26504#S3.p1.1)\.
- N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica \(2024\)Livecodebench: holistic and contamination free evaluation of large language models for code\.arXiv preprint arXiv:2403\.07974\.Cited by:[§3\.1](https://arxiv.org/html/2607.26504#S3.SS1.p3.6)\.
- M\. Jazbec, T\. X\. Olausson, L\. Béthune, P\. Ablin, M\. Kirchhof, J\. Monterio, V\. Turrisi, J\. Ramapuram, and M\. Cuturi \(2025\)Learning unmasking policies for diffusion language models\.arXiv preprint arXiv:2512\.09106\.Cited by:[§3\.2](https://arxiv.org/html/2607.26504#S3.SS2.p6.1)\.
- H\. Kang, Y\. Zhang, N\. L\. Kuang, N\. Majamaki, N\. Jaitly, Y\. Ma, and L\. Qin \(2025\)LaDiR: latent diffusion enhances LLMs for text reasoning\.arXiv preprint arXiv:2510\.04573\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p10.1),[§E\.2](https://arxiv.org/html/2607.26504#A5.SS2.p2.9)\.
- J\. Kim, L\. Cheuk\-Kit, C\. Domingo\-Enrich, Y\. Du, S\. Kakade, T\. Ngotiaoco, S\. Chen, and M\. Albergo \(2025a\)Any\-order flexible length masked diffusion\.arXiv preprint arXiv:2509\.01025\.Cited by:[Figure 9](https://arxiv.org/html/2607.26504#A4.F9),[Figure 9](https://arxiv.org/html/2607.26504#A4.F9.14.6.6),[§D\.1](https://arxiv.org/html/2607.26504#A4.SS1.SSS0.Px1.p2.1),[§D\.1](https://arxiv.org/html/2607.26504#A4.SS1.SSS0.Px2.p2.2),[§1](https://arxiv.org/html/2607.26504#S1.p4.3),[§4\.2](https://arxiv.org/html/2607.26504#S4.SS2.p15.1),[§4](https://arxiv.org/html/2607.26504#S4.p2.1)\.
- J\. Kim, J\. Geuter, D\. Alvarez\-Melis, S\. Kakade, and S\. Chen \(2026\)Stop training for the worst: progressive unmasking accelerates masked diffusion training\.arXiv preprint arXiv:2602\.10314\.Cited by:[§E\.2](https://arxiv.org/html/2607.26504#A5.SS2.p1.1)\.
- J\. Kim, S\. Kim, T\. Lee, D\. Z\. Pan, H\. Kim, S\. Kakade, and S\. Chen \(2025b\)Fine\-tuning masked diffusion for provable self\-correction\.arXiv preprint arXiv:2510\.01384\.Cited by:[§5](https://arxiv.org/html/2607.26504#S5.p2.1)\.
- J\. Kim, K\. Shah, V\. Kontonis, S\. Kakade, and S\. Chen \(2025c\)Train for the worst, plan for the best: understanding token ordering in masked diffusions\.arXiv preprint arXiv:2502\.06768\.Cited by:[§E\.4](https://arxiv.org/html/2607.26504#A5.SS4.p1.1),[§1](https://arxiv.org/html/2607.26504#S1.p2.1),[§2](https://arxiv.org/html/2607.26504#S2.p8.4),[§3](https://arxiv.org/html/2607.26504#S3.p2.1),[§4\.2](https://arxiv.org/html/2607.26504#S4.SS2.p5.3)\.
- S\. Lamont, C\. Walder, P\. Montague, A\. Dezfouli, and M\. Norrish \(2026\)Free lunch for pass@kk? low cost diverse sampling for diffusion language models\.arXiv preprint arXiv:2603\.04893\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p1.1),[Appendix A](https://arxiv.org/html/2607.26504#A1.p3.1),[§3](https://arxiv.org/html/2607.26504#S3.p3.1)\.
- C\. Lee, J\. Yoo, M\. Agarwal, S\. Shah, J\. Huang, A\. Raghunathan, S\. Hong, N\. M\. Boffi, and J\. Kim \(2026\)Flow map language models: one\-step language modeling via continuous denoising\.arXiv preprint arXiv:2602\.16813\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p8.1),[§1](https://arxiv.org/html/2607.26504#S1.p5.1),[§5](https://arxiv.org/html/2607.26504#S5.p2.1),[footnote 1](https://arxiv.org/html/2607.26504#footnote1)\.
- J\. Li, J\. Guan, W\. Wu, and C\. Li \(2025\)ReFusion: a diffusion large language model with parallel autoregressive decoding\.arXiv preprint arXiv:2512\.13586\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p13.1),[§E\.5](https://arxiv.org/html/2607.26504#A5.SS5.p1.1)\.
- P\. Li, D\. Muhtar, L\. Yin, T\. Chen, and S\. Liu \(2026\)Why diffusion language models struggle with truly parallel \(non\-autoregressive\) decoding?\.arXiv preprint arXiv:2602\.23225\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p1.1),[Appendix A](https://arxiv.org/html/2607.26504#A1.p4.1),[§1](https://arxiv.org/html/2607.26504#S1.p3.1),[§3](https://arxiv.org/html/2607.26504#S3.p5.1)\.
- T\. Li, Y\. Tian, H\. Li, M\. Deng, and K\. He \(2024\)Autoregressive image generation without vector quantization\.Advances in Neural Information Processing Systems37,pp\. 56424–56445\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p14.1)\.
- X\. Li, J\. Thickstun, I\. Gulrajani, P\. S\. Liang, and T\. B\. Hashimoto \(2022\)Diffusion\-lm improves controllable text generation\.Advances in neural information processing systems35,pp\. 4328–4343\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p7.1)\.
- Z\. Lin, Y\. Gong, Y\. Shen, T\. Wu, Z\. Fan, C\. Lin, N\. Duan, and W\. Chen \(2023\)Text generation with diffusion language models: a pre\-training approach with continuous paragraph denoise\.InInternational Conference on Machine Learning,pp\. 21051–21064\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p7.1)\.
- B\. Liu, S\. Bubeck, R\. Eldan, J\. Kulkarni, Y\. Li, A\. Nguyen, R\. Ward, and Y\. Zhang \(2023\)TinyGSM: achieving \>80% on GSM8k with small language models\.arXiv preprint arXiv:2312\.09241\.Cited by:[§E\.1](https://arxiv.org/html/2607.26504#A5.SS1.p1.4),[§1](https://arxiv.org/html/2607.26504#S1.p5.1),[§4\.2](https://arxiv.org/html/2607.26504#S4.SS2.p1.1),[§4\.2](https://arxiv.org/html/2607.26504#S4.SS2.p8.1)\.
- J\. Liu, X\. Dong, Z\. Ye, R\. Mehta, Y\. Fu, V\. Singh, J\. Kautz, C\. Zhang, and P\. Molchanov \(2025a\)Tidar: think in diffusion, talk in autoregression\.arXiv preprint arXiv:2511\.08923\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p13.1)\.
- Y\. Liu, L\. L\. Zhang, Y\. Zhu, B\. Dong, X\. Zhou, N\. Shang, F\. Yang, and M\. Yang \(2025b\)rStar\-Coder: scaling competitive code reasoning with a large\-scale verified dataset\.arXiv preprint arXiv:2505\.21297\.Cited by:[§D\.1](https://arxiv.org/html/2607.26504#A4.SS1.SSS0.Px2.p5.1),[§4\.1](https://arxiv.org/html/2607.26504#S4.SS1.p4.2)\.
- I\. Loshchilov and F\. Hutter \(2019\)Decoupled weight decay regularization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[§D\.1](https://arxiv.org/html/2607.26504#A4.SS1.SSS0.Px2.p6.4)\.
- J\. Lovelace, V\. Kishore, C\. Wan, E\. Shekhtman, and K\. Q\. Weinberger \(2023\)Latent diffusion for language generation\.Advances in Neural Information Processing Systems36,pp\. 56998–57025\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p10.1),[§E\.2](https://arxiv.org/html/2607.26504#A5.SS2.p2.9)\.
- R\. Lu, Y\. Huo, M\. Zhang, Y\. Li, and M\. R\. Lyu \(2025\)Next edit prediction: learning to predict code edits from context and interaction history\.arXiv preprint arXiv:2508\.10074\.Cited by:[§1](https://arxiv.org/html/2607.26504#S1.p1.1)\.
- V\. Meshchaninov, E\. Chimbulatov, A\. Shabalin, A\. Abramov, and D\. Vetrov \(2025\)Cosmos: compressed and smooth latent space for text diffusion modeling\.arXiv preprint arXiv:2506\.21170\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p10.1)\.
- Z\. Ni, S\. Wang, Y\. Yue, T\. Yu, W\. Zhao, Y\. Hua, T\. Chen, J\. Song, C\. Yu, B\. Zheng,et al\.\(2026\)The flexibility trap: why arbitrary order limits reasoning potential in diffusion language models\.arXiv preprint arXiv:2601\.15165\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p1.1),[Appendix A](https://arxiv.org/html/2607.26504#A1.p2.1),[§1](https://arxiv.org/html/2607.26504#S1.p3.1),[§3](https://arxiv.org/html/2607.26504#S3.p3.1),[§3](https://arxiv.org/html/2607.26504#S3.p4.1)\.
- S\. Nie, F\. Zhu, Z\. You, X\. Zhang, J\. Ou, J\. Hu, J\. Zhou, Y\. Lin, J\. Wen, and C\. Li \(2025\)Large language diffusion models\.arXiv preprint arXiv:2502\.09992\.Cited by:[Appendix C](https://arxiv.org/html/2607.26504#A3.p1.1),[§E\.4](https://arxiv.org/html/2607.26504#A5.SS4.p1.1),[§1](https://arxiv.org/html/2607.26504#S1.p2.1),[§2](https://arxiv.org/html/2607.26504#S2.p10.1),[§2](https://arxiv.org/html/2607.26504#S2.p8.4)\.
- T\. X\. Olausson, M\. Jazbec, X\. Wang, A\. Solar\-Lezama, C\. A\. Naesseth, S\. Mandt, and E\. Nalisnick \(2026\)A tale of two temperatures: simple, efficient, and diverse sampling from diffusion language models\.arXiv preprint arXiv:2604\.09921\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p1.1),[Appendix A](https://arxiv.org/html/2607.26504#A1.p3.1),[§3](https://arxiv.org/html/2607.26504#S3.p3.1)\.
- J\. Ou, S\. Nie, K\. Xue, F\. Zhu, J\. Sun, Z\. Li, and C\. Li \(2024\)Your absorbing discrete diffusion secretly models the conditional distributions of clean data\.arXiv preprint arXiv:2406\.03736\.Cited by:[§1](https://arxiv.org/html/2607.26504#S1.p2.1),[§2](https://arxiv.org/html/2607.26504#S2.p3.9)\.
- W\. Peebles and S\. Xie \(2023\)Scalable diffusion models with transformers\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 4195–4205\.Cited by:[§D\.1](https://arxiv.org/html/2607.26504#A4.SS1.SSS0.Px2.p6.4),[§4\.1](https://arxiv.org/html/2607.26504#S4.SS1.p4.2)\.
- F\. Z\. Peng, Z\. Bezemek, S\. Patel, J\. Rector\-Brooks, S\. Yao, A\. J\. Bose, A\. Tong, and P\. Chatterjee \(2025\)Path planning for masked diffusion model sampling\.External Links:2502\.03540,[Link](https://arxiv.org/abs/2502.03540)Cited by:[§E\.4](https://arxiv.org/html/2607.26504#A5.SS4.p1.1),[§2](https://arxiv.org/html/2607.26504#S2.p8.4)\.
- P\. Potaptchik, J\. Yim, A\. Saravanan, P\. Holderrieth, E\. Vanden\-Eijnden, and M\. S\. Albergo \(2026\)Discrete flow maps\.arXiv preprint arXiv:2604\.09784\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p9.1),[§5](https://arxiv.org/html/2607.26504#S5.p2.1)\.
- P\. Pynadath, J\. Shi, and R\. Zhang \(2025\)CANDI: hybrid discrete\-continuous diffusion models\.arXiv preprint arXiv:2510\.22510\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p11.1)\.
- S\. Qu \(2026\)VDLM: variable diffusion LMs via robust latent\-to\-text rendering\.arXiv preprint arXiv:2602\.15870\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p10.1)\.
- X\. Qu, S\. Wang, Z\. Huang, K\. Hua, F\. Yin, R\. Zhu, J\. Zhou, Q\. Min, Z\. Wang, Y\. Li,et al\.\(2025\)Dynamic large concept models: latent reasoning in an adaptive semantic space\.arXiv preprint arXiv:2512\.24617\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p6.1)\.
- D\. Roos, O\. Davis, F\. Eijkelboom, M\. Bronstein, M\. Welling, İ\. İ\. Ceylan, L\. Ambrogioni, and J\. van de Meent \(2026\)Categorical flow maps\.arXiv preprint arXiv:2602\.12233\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p9.1),[§5](https://arxiv.org/html/2607.26504#S5.p2.1)\.
- S\. Sahoo, M\. Arriola, Y\. Schiff, A\. Gokaslan, E\. Marroquin, J\. Chiu, A\. Rush, and V\. Kuleshov \(2024\)Simple and effective masked diffusion language models\.Advances in Neural Information Processing Systems37,pp\. 130136–130184\.Cited by:[§1](https://arxiv.org/html/2607.26504#S1.p2.1),[§2](https://arxiv.org/html/2607.26504#S2.p1.1),[§2](https://arxiv.org/html/2607.26504#S2.p3.9)\.
- S\. S\. Sahoo, J\. Deschenaux, A\. Gokaslan, G\. Wang, J\. Chiu, and V\. Kuleshov \(2025\)The diffusion duality\.arXiv preprint arXiv:2506\.10892\.Cited by:[§1](https://arxiv.org/html/2607.26504#S1.p5.1),[§4\.2](https://arxiv.org/html/2607.26504#S4.SS2.p10.1),[§5](https://arxiv.org/html/2607.26504#S5.p2.1)\.
- Y\. Schiff, O\. Belhasin, R\. Uziel, G\. Wang, M\. Arriola, G\. Turok, R\. Zilberstein, M\. Elad, and V\. Kuleshov \(2026\)Learn from your mistakes: self\-correcting masked diffusion models\.arXiv preprint arXiv:2602\.11590\.Cited by:[§5](https://arxiv.org/html/2607.26504#S5.p2.1)\.
- J\. Schusterbauer, M\. Gui, Y\. Li, P\. Ma, F\. Krause, and B\. Ommer \(2026\)Denoising, fast and slow: difficulty\-aware adaptive sampling for image generation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 43260–43270\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p15.1)\.
- K\. Shah, N\. Dikkala, X\. Wang, and R\. Panigrahy \(2024\)Causal language modeling can elicit search and reasoning capabilities on logic puzzles\.Advances in Neural Information Processing Systems37,pp\. 56674–56702\.Cited by:[§1](https://arxiv.org/html/2607.26504#S1.p1.1)\.
- J\. Shen, J\. Zhao, Z\. He, and Z\. Lin \(2026a\)CoDAR: continuous diffusion language models are more powerful than you think\.arXiv preprint arXiv:2603\.02547\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p10.1)\.
- Y\. Shen, T\. Feng, J\. Han, W\. Wang, T\. Chen, C\. Shen, J\. Leskovec, and S\. Ermon \(2026b\)Improving diffusion language model decoding through joint search in generation order and token space\.arXiv preprint arXiv:2601\.20339\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p1.1),[Appendix A](https://arxiv.org/html/2607.26504#A1.p4.1),[§3](https://arxiv.org/html/2607.26504#S3.p3.1)\.
- J\. Shi, K\. Han, Z\. Wang, A\. Doucet, and M\. Titsias \(2024\)Simplified and generalized masked diffusion for discrete data\.Advances in neural information processing systems37,pp\. 103131–103167\.Cited by:[§1](https://arxiv.org/html/2607.26504#S1.p2.1),[§2](https://arxiv.org/html/2607.26504#S2.p1.1),[§2](https://arxiv.org/html/2607.26504#S2.p3.9)\.
- K\. Song, B\. Chen, M\. Simchowitz, Y\. Du, R\. Tedrake, and V\. Sitzmann \(2025a\)History\-guided video diffusion\.arXiv preprint arXiv:2502\.06764\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p14.1)\.
- Y\. Song, Z\. Zhang, C\. Luo, P\. Gao, F\. Xia, H\. Luo, Z\. Li, Y\. Yang, H\. Yu, X\. Qu,et al\.\(2025b\)Seed diffusion: a large\-scale diffusion language model with high\-speed inference\.arXiv preprint arXiv:2508\.02193\.Cited by:[§2](https://arxiv.org/html/2607.26504#S2.p10.1),[§3](https://arxiv.org/html/2607.26504#S3.p1.1)\.
- I\. Trainin, S\. Ravfogel, O\. Abend, and A\. Feder \(2026\)Discrete diffusion models exploit asymmetry to solve lookahead planning tasks\.arXiv preprint arXiv:2602\.19980\.Cited by:[§3](https://arxiv.org/html/2607.26504#S3.p2.1)\.
- D\. von Rütte, J\. Fluri, O\. Pooladzandi, B\. Schölkopf, T\. Hofmann, and A\. Orvieto \(2025\)Scaling behavior of discrete diffusion language models\.arXiv preprint arXiv:2512\.10858\.Cited by:[§5](https://arxiv.org/html/2607.26504#S5.p2.1)\.
- A\. Wang and K\. Cho \(2019\)BERT has a mouth, and it must speak: bert as a markov random field language model\.InProceedings of the workshop on methods for optimizing and evaluating neural language generation,pp\. 30–36\.Cited by:[§2](https://arxiv.org/html/2607.26504#S2.p6.1)\.
- G\. Wang, Y\. Schiff, S\. Sahoo, and V\. Kuleshov \(2026\)Remasking discrete diffusion models with inference\-time scaling\.Advances in Neural Information Processing Systems38,pp\. 147282–147339\.Cited by:[§5](https://arxiv.org/html/2607.26504#S5.p2.1)\.
- C\. Wewer, B\. Pogodzinski, B\. Schiele, and J\. E\. Lenssen \(2025\)Spatial reasoning with denoising models\.arXiv preprint arXiv:2502\.21075\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p15.1)\.
- C\. Wu, H\. Zhang, S\. Xue, S\. Diao, Y\. Fu, Z\. Liu, P\. Molchanov, P\. Luo, S\. Han, and E\. Xie \(2025a\)Fast\-dLLM v2: efficient block\-diffusion LLM\.arXiv preprint arXiv:2509\.26328\.Cited by:[§3](https://arxiv.org/html/2607.26504#S3.p1.1)\.
- C\. Wu, H\. Zhang, S\. Xue, Z\. Liu, S\. Diao, L\. Zhu, P\. Luo, S\. Han, and E\. Xie \(2025b\)Fast\-dLLM: training\-free acceleration of diffusion LLM by enabling KV cache and parallel decoding\.arXiv preprint arXiv:2505\.22618\.Cited by:[§E\.4](https://arxiv.org/html/2607.26504#A5.SS4.p1.1),[§2](https://arxiv.org/html/2607.26504#S2.p8.4),[§3](https://arxiv.org/html/2607.26504#S3.p1.1)\.
- Z\. Xie, J\. Ye, L\. Zheng, J\. Gao, J\. Dong, Z\. Wu, X\. Zhao, S\. Gong, X\. Jiang, Z\. Li,et al\.\(2025\)Dream\-Coder 7B: an open diffusion language model for code\.arXiv preprint arXiv:2509\.01142\.Cited by:[Appendix C](https://arxiv.org/html/2607.26504#A3.p1.1),[§D\.1](https://arxiv.org/html/2607.26504#A4.SS1.SSS0.Px2.p6.4),[§D\.3](https://arxiv.org/html/2607.26504#A4.SS3.SSS0.Px5.p1.1),[§D\.4](https://arxiv.org/html/2607.26504#A4.SS4.p2.5),[§D\.5](https://arxiv.org/html/2607.26504#A4.SS5.p2.12),[§1](https://arxiv.org/html/2607.26504#S1.p2.1),[§1](https://arxiv.org/html/2607.26504#S1.p4.3),[§2](https://arxiv.org/html/2607.26504#S2.p10.1),[§3\.1](https://arxiv.org/html/2607.26504#S3.SS1.p3.6),[§4\.1](https://arxiv.org/html/2607.26504#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.26504#S4.SS1.p4.2)\.
- Z\. Xu, Y\. Liu, Y\. Yin, M\. Zhou, and R\. Poovendran \(2025\)KodCode: a diverse, challenging, and verifiable synthetic dataset for coding\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 6980–7008\.Cited by:[§D\.1](https://arxiv.org/html/2607.26504#A4.SS1.SSS0.Px2.p5.1),[§4\.1](https://arxiv.org/html/2607.26504#S4.SS1.p4.2)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§E\.1](https://arxiv.org/html/2607.26504#A5.SS1.p3.1),[§E\.2](https://arxiv.org/html/2607.26504#A5.SS2.p1.1)\.
- J\. Ye, J\. Gao, S\. Gong, L\. Zheng, X\. Jiang, Z\. Li, and L\. Kong \(2024a\)Beyond autoregression: discrete diffusion for complex reasoning and planning\.arXiv preprint arXiv:2410\.14157\.Cited by:[§3](https://arxiv.org/html/2607.26504#S3.p2.1)\.
- J\. Ye, S\. Gong, L\. Chen, L\. Zheng, J\. Gao, H\. Shi, C\. Wu, X\. Jiang, Z\. Li, W\. Bi,et al\.\(2024b\)Diffusion of thought: chain\-of\-thought reasoning in diffusion language models\.Advances in Neural Information Processing Systems37,pp\. 105345–105374\.Cited by:[§3\.2](https://arxiv.org/html/2607.26504#S3.SS2.p6.1)\.
- J\. Ye, Z\. Xie, L\. Zheng, J\. Gao, Z\. Wu, X\. Jiang, Z\. Li, and L\. Kong \(2025\)Dream 7B: diffusion large language models\.arXiv preprint arXiv:2508\.15487\.Cited by:[§2](https://arxiv.org/html/2607.26504#S2.p10.1)\.
- J\. Yoo, W\. Kim, F\. Eijkelboom, C\. Lee, N\. M\. Boffi, S\. Hong, and J\. Kim \(2026\)Self\-conditioned flow map language models via fixed\-point flows\.arXiv preprint arXiv:2607\.00714\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p9.1)\.
- R\. Zhang, S\. Zhai, J\. Gu, Y\. Zhang, H\. Zheng, T\. Chen, M\. A\. Bautista, J\. Susskind, and N\. Jaitly \(2025\)Flexible language modeling in continuous space with transformer\-based autoregressive flows\.arXiv preprint arXiv:2507\.00425\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p7.1)\.
- Y\. Zhang, J\. Gu, Z\. Wu, S\. Zhai, J\. Susskind, and N\. Jaitly \(2023\)Planner: generating diversified paragraph via latent language diffusion model\.Advances in Neural Information Processing Systems36,pp\. 80178–80190\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p10.1)\.
- H\. Zheng, S\. Gong, R\. Zhang, T\. Chen, J\. Gu, M\. Zhou, N\. Jaitly, and Y\. Zhang \(2025\)Continuously augmented discrete diffusion model for categorical generative modeling\.arXiv preprint arXiv:2510\.01329\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p11.1)\.
- K\. Zheng, Y\. Chen, H\. Mao, M\. Liu, J\. Zhu, and Q\. Zhang \(2024\)Masked diffusion models are secretly time\-agnostic masked models and exploit inaccurate categorical sampling\.arXiv preprint arXiv:2409\.02908\.Cited by:[§E\.4](https://arxiv.org/html/2607.26504#A5.SS4.p1.1),[§1](https://arxiv.org/html/2607.26504#S1.p2.1),[§2](https://arxiv.org/html/2607.26504#S2.p3.9),[§2](https://arxiv.org/html/2607.26504#S2.p8.4)\.
- C\. Zhou, C\. Yang, Y\. Hu, C\. Wang, C\. Zhang, M\. Zhang, L\. Mackey, T\. Jaakkola, S\. Bates, and D\. Zhang \(2025\)Coevolutionary continuous discrete diffusion: make your diffusion language model a latent reasoner\.arXiv preprint arXiv:2510\.03206\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p11.1)\.
- X\. Zhu, G\. Karadzhov, C\. Whitehouse, and A\. Vlachos \(2025\)Segment\-level diffusion: a framework for controllable long\-form generation with diffusion language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 4163–4183\.Cited by:[Appendix A](https://arxiv.org/html/2607.26504#A1.p10.1)\.

###### Contents

1. [1Introduction](https://arxiv.org/html/2607.26504#S1)
2. [2Preliminaries](https://arxiv.org/html/2607.26504#S2)
3. [3The Interface\-to\-Inference Gap](https://arxiv.org/html/2607.26504#S3)1. [3\.1Semantic collapse in Python code generation](https://arxiv.org/html/2607.26504#S3.SS1) 2. [3\.2Diagnosing Positional Uncertainty in MDMs](https://arxiv.org/html/2607.26504#S3.SS2)
4. [4Circumventing Positional Uncertainty](https://arxiv.org/html/2607.26504#S4)1. [4\.1FlexMDM](https://arxiv.org/html/2607.26504#S4.SS1) 2. [4\.2LatentMDM](https://arxiv.org/html/2607.26504#S4.SS2)
5. [5Conclusion](https://arxiv.org/html/2607.26504#S5)
6. [References](https://arxiv.org/html/2607.26504#bib)
7. [ARelated Works](https://arxiv.org/html/2607.26504#A1)
8. [BDetails of Section3\.1](https://arxiv.org/html/2607.26504#A2)1. [B\.1Additional similarity results](https://arxiv.org/html/2607.26504#A2.SS1) 2. [B\.2Semantic collapse in dUltra planner](https://arxiv.org/html/2607.26504#A2.SS2)
9. [CDetails of Section3\.2](https://arxiv.org/html/2607.26504#A3)
10. [DDetails of Section4\.1](https://arxiv.org/html/2607.26504#A4)1. [D\.1FlexMDM Training](https://arxiv.org/html/2607.26504#A4.SS1) 2. [D\.2Training FlexMDM in Insertion\-Progress Coordinates](https://arxiv.org/html/2607.26504#A4.SS2) 3. [D\.3Any\-Order Metrics](https://arxiv.org/html/2607.26504#A4.SS3) 4. [D\.4FlexMDM Inference and Details on Evaluation Setup](https://arxiv.org/html/2607.26504#A4.SS4) 5. [D\.5Insertion Temperature](https://arxiv.org/html/2607.26504#A4.SS5)
11. [EDetails of Section4\.2](https://arxiv.org/html/2607.26504#A5)1. [E\.1Data Pre\-processing](https://arxiv.org/html/2607.26504#A5.SS1) 2. [E\.2Architecture](https://arxiv.org/html/2607.26504#A5.SS2) 3. [E\.3Training Details](https://arxiv.org/html/2607.26504#A5.SS3) 4. [E\.4Inference Details](https://arxiv.org/html/2607.26504#A5.SS4) 5. [E\.5Additional Experiments](https://arxiv.org/html/2607.26504#A5.SS5) 6. [E\.6Generation Trace](https://arxiv.org/html/2607.26504#A5.SS6)

## Appendix ARelated Works

Causal collapse in MDMs\.Recent work has identified two related but distinct limitations of confidence\-based decoding in masked diffusion models: reduced sample diversity and the collapse of flexible generation orders toward AR\-like trajectories\[Niet al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib43), Olaussonet al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib42), Shenet al\.,[2026b](https://arxiv.org/html/2607.26504#bib.bib49), Lamontet al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib44), Fanget al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib46), Gonget al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib3), Liet al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib41)\]\. As we make it clear in Section[3\.1](https://arxiv.org/html/2607.26504#S3.SS1), the former concerns the entropy and coverage of the resulting samples, whereas the latter concerns the structure of the generation process itself\.

A first line of work studies why confidence\-based any\-order decoding produces less diverse outputs than fixed left\-to\-right sampling with a nonzero token\-sampling temperature\.Niet al\.\[[2026](https://arxiv.org/html/2607.26504#bib.bib43)\]characterize this phenomenon as the*flexibility trap*\. Since confidence\-based decoding can postpone uncertain tokens and instead resolve low\-entropy positions, it may bypass the branching points at which autoregressive trajectories would emerge\. Motivated by a similar observation,Fanget al\.\[[2026](https://arxiv.org/html/2607.26504#bib.bib46)\]describe a*quality–exploration dilemma*: greedily committing high\-confidence tokens can improve the quality of an individual trajectory while suppressing sequence\-level entropy and harming multi\-sample exploration at Pass@K\.

Several works develop inference\-time methods to mitigate this diversity degradation\.Olaussonet al\.\[[2026](https://arxiv.org/html/2607.26504#bib.bib42)\]distinguish the temperature used to sample token values from the temperature used to select or remask token positions\.Lamontet al\.\[[2026](https://arxiv.org/html/2607.26504#bib.bib44)\]instead coordinate multiple samples within a batch, discouraging each new trajectory from reproducing feature\-space directions already explored by previous samples\.

A second line of work more directly examines the generation trajectory itself\.Gonget al\.\[[2025](https://arxiv.org/html/2607.26504#bib.bib3)\]introduce local and global measures of*AR\-ness*, quantifying how closely the order in which an MDM finalizes tokens follows a left\-to\-right trajectory\. They show that this behavior depends on training and inference choices and that increasing sampling temperature can diversify not only token values but also generation orders\.Gwaket al\.\[[2025](https://arxiv.org/html/2607.26504#bib.bib65)\]similarly observe that independently selecting positions according to token\-level confidence often yields sequential, AR\-like trajectories, and propose reward\-weighted sampling to favor groups of tokens that better preserve non\-autoregressive generation\.Liet al\.\[[2026](https://arxiv.org/html/2607.26504#bib.bib41)\]attribute the tendency toward causal generation to biases inherited from the training data and propose a parallel\-inference design intended to reduce this bias\. Other methods treat generation order as an inference\-time search problem\. Rather than accepting the single trajectory induced by greedy confidence ranking,Shenet al\.\[[2026b](https://arxiv.org/html/2607.26504#bib.bib49)\]jointly search over both which positions to decode and which token values to assign\.

Our work focuses on this remaining mechanistic question: we ask why standard MDM inference, despite its*any\-order interface*, fails to go meaningfully beyond AR\-like reasoning in the first place\. We quantify how closely confidence\-based inference structurally approximates causal decoding on Python programs \(Section[3\.1](https://arxiv.org/html/2607.26504#S3.SS1)\) and attribute this collapse to a more fundamental bottleneck of token\-level, fixed\-canvas inference \(Section[3\.2](https://arxiv.org/html/2607.26504#S3.SS2)\)\. This mechanism also reveals broader limitations for tasks requiring globally structured generation\.

Continuous latent reasoning in autoregressive LLMs\.Reasoning in continuous representation space has recently gained attention as a way to reduce the cost of explicit chain\-of\-thought generation while preserving intermediate computation\. Since such methods avoid verbalizing every reasoning step, they can improve efficiency, though at the cost of reduced interpretability\. Coconut\[Haoet al\.,[2024](https://arxiv.org/html/2607.26504#bib.bib83)\]enables LLMs to perform intermediate reasoning in a continuous hidden\-state space by feeding the last hidden state back as the next input embedding, rather than decoding each reasoning step into language tokens\. ContextLM\[Daiet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib86)\]augments autoregressive LMs with latent context prediction, where fixed\-size context embeddings guide subsequent token generation\. DLCM\[Quet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib93)\]learns to parse sequences into variable\-length concept segments, runs a causal Transformer over the compressed concept sequence, and decodes the resulting representations back to tokens, thereby reallocating computation around semantic boundaries\. These works demonstrate the usefulness of continuous representations for efficient reasoning, but remain fundamentally autoregressive at the high\-level inference interface\.

Continuous generative models for language\.Another line of work develops continuous generative models for text, often avoiding discrete\-token realization during intermediate generation\. Early approaches\[Liet al\.,[2022](https://arxiv.org/html/2607.26504#bib.bib87), Linet al\.,[2023](https://arxiv.org/html/2607.26504#bib.bib88)\]apply diffusion models to token\-level continuous embeddings, mainly for controllable generation or sequence\-to\-sequence generation\. TarflowLM\[Zhanget al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib94)\]models token\-level Gaussian embeddings using Transformer\-based autoregressive normalizing flows\. A common issue in direct embedding\-space generation is the rounding problem: predicted continuous embeddings must eventually be mapped back to discrete tokens, and this projection can introduce rounding errors\.

More recent approaches\[Chenet al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib102), Leeet al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib77)\]indirectly mitigate this issue by matching token posteriors rather than directly regressing word embeddings\. This is especially explicit in FMLM\[Leeet al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib77)\], which operates in one\-hot token space and decodes by taking the argmax over predicted token probabilities\.

This flow\-map approach to few\-step generation of discrete data has advanced rapidly\. Categorical Flow Maps\[Rooset al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib78)\]learn maps toward the probability simplex using distillation and endpoint\-consistency objectives, while Discrete Flow Maps\[Potaptchiket al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib76)\]modify flow\-map training to better reflect the discrete structure of language and support full\-sequence generation in a single model evaluation\. Building on FMLM,Yooet al\.\[[2026](https://arxiv.org/html/2607.26504#bib.bib110)\]interpret self\-conditioning in flow\-based language models as a fixed\-point iteration and use this perspective to improve few\-step generation\. Other works replace one\-hot token representations with more compact continuous parameterizations: the hyperspherical flow language model \(𝕊\\mathbb\{S\}\-FLM\)\[Deschenaux and Gulcehre,[2026](https://arxiv.org/html/2607.26504#bib.bib59)\]runs latent flows on the unit hypersphere and narrows the gap to MDMs on mathematics and code \(we adopt it as a representative continuous\-flow baseline in Section[4\.2](https://arxiv.org/html/2607.26504#S4.SS2)\); ELF\[Huet al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib111)\]performs flow matching in word\-embedding space and converts the resulting continuous states to token predictions through a learned output map; and CoBit\[Batzoliset al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib112)\], applies diffusion to analog bitstream representations of tokens, reducing the dimensional dependence of each token prediction from linear to logarithmic in the vocabulary size\. Closest to our focus,Agarwalet al\.\[[2026](https://arxiv.org/html/2607.26504#bib.bib4)\]equip flow\-map language models with masking\-style noise schedules, yielding an any\-order continuous\-space sampler with an inference\-time self\-correction mechanism \(posterior refinement\); their goal, however, is to improve the speed–quality trade\-off of parallel generation rather than to analyze the structure of the generation orders that the any\-order interface induces\.

A complementary way to avoid token\-level rounding is to separate high\-level continuous planning from discrete token realization\. LD4PG\[Lovelaceet al\.,[2023](https://arxiv.org/html/2607.26504#bib.bib89)\], PLANNER\[Zhanget al\.,[2023](https://arxiv.org/html/2607.26504#bib.bib90)\], Segment\-Level Diffusion\[Zhuet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib91)\], VDLM\[Qu,[2026](https://arxiv.org/html/2607.26504#bib.bib95)\], and CoDAR\[Shenet al\.,[2026a](https://arxiv.org/html/2607.26504#bib.bib103)\]use continuous diffusion models over higher\-level semantic latents \(i\.e\., sentence\-, paragraph\-, or whole\-sequence\-level representations\), and rely on an autoregressive token\-level decoder or renderer to iteratively decode each denoised latent back into discrete text in the final stage of generation\. COSMOS\[Meshchaninovet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib101)\]similarly constructs a compressed continuous latent space for text diffusion, but decodes tokens in parallel\. LCM\[Barraultet al\.,[2024](https://arxiv.org/html/2607.26504#bib.bib92)\]and LaDiR\[Kanget al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib84)\]both use diffusion over continuous high\-level semantic representations beyond tokens while performing autoregressive generation at the semantic level: LCM denoises sentence\-level concept embeddings for semantic continuation, whereas LaDiR denoises structured latent chain\-of\-thought blocks for reasoning\.

Related hybrid continuous–discrete diffusion methods, including CADD\[Zhenget al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib81)\], CANDI\[Pynadathet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib104)\], and CCDD\[Zhouet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib105)\], combine continuous and discrete diffusion to reduce rounding mismatch, though the continuous variables differ across methods, including word embeddings, one\-hot vectors, and LLM\-contextualized latents\.

Our LatentMDM shares the broad motivation of using continuous latent variables to represent higher\-level semantic units\. However, our focus is different: we study whether moving the masked\-diffusion interface from token positions to segment\-level latents can elicit genuinely any\-order inference\. Most prior continuous or latent\-space language models primarily target likelihood modeling, controllability, or fast generation, and typically do not analyze whether their generation traces differ structurally from ARM or token\-level MDM decoding\.

Hybrid autoregressive and masked\-diffusion\.Our LatentMDM also relates to hybrid architectures that combine non\-autoregressive \(NAR\) prediction with autoregressive token generation\. TiDAR\[Liuet al\.,[2025a](https://arxiv.org/html/2607.26504#bib.bib106)\]uses a token\-level MDM as a parallel drafter and an ARM as a verifier or sampler within a unified module\. ReFusion\[Liet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib107)\]fine\-tunes an ARM for block\-level any\-order generation while preserving token\-level autoregressivity, enabling efficient inference with KV caching\. CoDiLA\[Herscheet al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib108)\]uses a token\-level MDM as a parallel drafter and a smaller local ARM as a conditional sampler given soft draft conditions\. These approaches are promising for improving sampling efficiency, but their any\-order structure is generally defined over fixed token blocks while preserving substantial token\-level autoregressive\-ness\. In contrast, our LatentMDM moves the reveal decision to variable\-length semantic segments and asks whether segment\-level confidence can induce generation orders that escape the left\-to\-right bias observed in token\-level MDMs\.

Any\-order models in continuous latent spaces\.MAR\[Liet al\.,[2024](https://arxiv.org/html/2607.26504#bib.bib109)\]and Diffusion Forcing\[Songet al\.,[2025a](https://arxiv.org/html/2607.26504#bib.bib18), Chenet al\.,[2024](https://arxiv.org/html/2607.26504#bib.bib19)\]propose any\-order interfaces for denoising\-based diffusion models in continuous latent spaces\. Their use of this flexibility, however, is mainly aimed at improving generation efficiency\[Liet al\.,[2024](https://arxiv.org/html/2607.26504#bib.bib109)\]or enabling conditional and long\-horizon video generation\[Songet al\.,[2025a](https://arxiv.org/html/2607.26504#bib.bib18), Chenet al\.,[2024](https://arxiv.org/html/2607.26504#bib.bib19)\], rather than searching over semantic generation orders\.

The works with the greatest conceptual overlap with ours are those of\[Weweret al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib1), Schusterbaueret al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib54)\]\. They train diffusion\-forcing\-style, any\-order continuous diffusion models in which each patch or frame can be assigned a separate time step, enabling flexible generation orders\. Although our work shares this high\-level conceptual connection, it differs substantially in formulation: these works focus on continuous\-space diffusion modeling, whereas we adopt a masked\-modeling formulation\.

## Appendix BDetails of Section[3\.1](https://arxiv.org/html/2607.26504#S3.SS1)

This appendix describes the code\-similarity metrics used in Section[3\.1](https://arxiv.org/html/2607.26504#S3.SS1)\. Each metric converts a generated Python code into a rooted ordered tree and then compares two such trees by tree edit distance\. A tree node represents a syntactic component of the program, and an edge represents immediate containment: a parent node is a larger construct, and its children are the constructs that appear inside it\. Children are ordered by their order in the source code\.

For two rooted ordered labeled trees𝒯\\mathcal\{T\}and𝒯′\\mathcal\{T\}^\{\\prime\}, letTEDρ​\(𝒯,𝒯′\)\\mathrm\{TED\}\_\{\\rho\}\(\\mathcal\{T\},\\mathcal\{T\}^\{\\prime\}\)denote the tree edit distance between them, computed with unit insertion cost, unit deletion cost, and rename costρ\\rho\. We normalize the distance into a similarity score:

Simρ​\(𝒯,𝒯′\)=1−TEDρ​\(𝒯,𝒯′\)max⁡\{\|𝒯\|,\|𝒯′\|\},\\mathrm\{Sim\}\_\{\\rho\}\(\\mathcal\{T\},\\mathcal\{T\}^\{\\prime\}\)=1\-\\frac\{\\mathrm\{TED\}\_\{\\rho\}\(\\mathcal\{T\},\\mathcal\{T\}^\{\\prime\}\)\}\{\\max\\\{\|\\mathcal\{T\}\|,\|\\mathcal\{T\}^\{\\prime\}\|\\\}\},where\|𝒯\|\|\\mathcal\{T\}\|is the number of nodes in𝒯\\mathcal\{T\}\. Scores are clipped to\[0,1\]\[0,1\]when necessary\. Larger values indicate a more similar tree structure\. The three metrics below differ only in how they construct the tree before applying this normalized tree\-edit similarity\.

#### Abstract Syntax Tree Distance \(ASTD\)\.

Our main metric, ASTD, uses Python’s built\-in parser to convert each program into an abstract syntax tree\. In this tree, nodes correspond to semantic syntactic units used by the Python interpreter, such asModule,FunctionDef,For,If,Assign,Return,Name,Constant, andAdd\. The parent\-child relation is given by Python’s AST fields: for example, aFunctionDefcontains its arguments and body statements, anIfcontains its condition and branches, and anAssigncontains its target and value\.

For example, for the code

def f\(x\):y = x \+ 1return y\\begin\{array\}\[\]\{l\}\\texttt\{def f\(x\):\}\\\\ \\quad\\texttt\{y = x \+ 1\}\\\\ \\quad\\texttt\{return y\}\\end\{array\}
the AST\-based representation contains nodes such asModule,FunctionDef,arguments,arg,Assign,Name,BinOp,Add,Constant, andReturn\. TheFunctionDefnode contains the argument list and the two body statements; theAssignnode contains the targetNameand the valueBinOp; and theBinOpnode contains the left operand, operator, and right operand\.

Before comparing ASTs, we anonymize identifiers so that arbitrary names do not dominate the score\. Variable and argument names are replaced by canonical placeholders, function names are canonicalized, and top\-level docstrings are ignored\. The resulting tree still preserves the program’s control flow, statement structure, operators, literals, and expression nesting\.

Let𝒜​\(c\)\\mathcal\{A\}\(c\)be this anonymized Python AST for codecc\. We define

ASTD​\(c,c′\)=Sim1​\(𝒜​\(c\),𝒜​\(c′\)\)\.\\mathrm\{ASTD\}\(c,c^\{\\prime\}\)=\\mathrm\{Sim\}\_\{1\}\\bigl\(\\mathcal\{A\}\(c\),\\mathcal\{A\}\(c^\{\\prime\}\)\\bigr\)\.The rename cost isρ=1\\rho=1, so changing a node label, such as replacingAddwithSuborConstant\(1\)withConstant\(2\), contributes edit cost\. Therefore, ASTD measures both the global program skeleton and local syntactic choices inside that skeleton\.

#### Tree Similarity of Edit Distance \(TSED\)\.

TSED uses the same tree\-edit\-distance normalization, but builds the tree withtree\-sitterrather than Python’s AST parser\. Python’s AST is an interpreter\-level abstraction, whereas tree\-sitter follows the surface grammar of the written program\. Its nodes correspond to grammar constructs such asfunction\_definition,parameters,block,assignment,return\_statement, andcall\. Edges again mean immediate containment in the parsed program, and children are ordered by source order\.

For the same example above, the tree\-sitter representation contains grammar nodes such asmodule,function\_definition,identifier,parameters,block,expression\_statement,assignment,binary\_operator, andreturn\_statement\. Compared with the Python AST, this tree follows the written grammar more directly: for instance, the function body appears under ablocknode, and the assignment may appear under anexpression\_statementnode\.

Let𝒢​\(c\)\\mathcal\{G\}\(c\)be the tree\-sitter parse tree of codecc\. We define

TSED​\(c,c′\)=Sim0​\(𝒢​\(c\),𝒢​\(c′\)\)\.\\mathrm\{TSED\}\(c,c^\{\\prime\}\)=\\mathrm\{Sim\}\_\{0\}\\bigl\(\\mathcal\{G\}\(c\),\\mathcal\{G\}\(c^\{\\prime\}\)\\bigr\)\.Here the rename cost isρ=0\\rho=0\. Thus, TSED does not charge for changing a node label; it mainly charges for inserting or deleting nodes to make the two parse trees align\. In other words, ASTD asks whether the two ASTs have matching node types and values, while TSED asks a weaker question: whether the two programs have similarly shaped parse trees\. This makes TSED a more permissive structural metric than ASTD\.

#### Coarse ASTD \(Coarse\)\.

Coarse ASTD is a lower\-resolution variant of ASTD\. It starts from the same anonymized Python AST, but keeps only statement\-level structure\. Nodes such asFunctionDef,For,While,If,Assign, andReturnare retained, while fine\-grained expression subtrees are collapsed into the label of the nearest statement node\.

For the same example, Coarse ASTD keeps a much smaller tree: a rootModulenode, aFunctionDefnode, and statement\-level children such asAssignandReturn\. The internal expressionx \+ 1is not expanded into separateName,Add, andConstantnodes\. Instead, it is treated as part of the assignment’s statement\-level label\. Thus, Coarse ASTD ignores many expression\-level differences and focuses on the high\-level statement layout\.

Let𝒞​\(c\)\\mathcal\{C\}\(c\)be this coarsened statement\-level tree\. We define

Coarse​\(c,c′\)=Sim1​\(𝒞​\(c\),𝒞​\(c′\)\)\.\\mathrm\{Coarse\}\(c,c^\{\\prime\}\)=\\mathrm\{Sim\}\_\{1\}\\bigl\(\\mathcal\{C\}\(c\),\\mathcal\{C\}\(c^\{\\prime\}\)\\bigr\)\.The rename cost is againρ=1\\rho=1, so mismatched statement labels contribute to the edit cost\. However, because expression internals are not expanded into separate nodes, Coarse ASTD focuses more directly on high\-level control flow and statement layout\. It tests whether two decoding strategies produce programs with similar global skeletons even after ignoring much of the expression\-level detail\.

#### Sampling protocol\.

For each model and prompt, all decoding policies use temperature 0\.2 and nucleus sampling top\-p=0\.95p=0\.95\. We generate one sample for each confidence\-based policy, since confidence\-based decoding at this temperature produces little diversity across repeated samples\. We generate 32 samples for strict L2R and random decoding\. The 32 L2R samples form the reference set used in best\-match aggregation, while the 32 random samples provide a calibration baseline and help obtain enough valid or correct programs despite the lower success rates of random and L2R decoding\.

#### Best\-match aggregation over samples\.

The metrics above define the similarity between a pair of code snippets\. In our experiments, however, each prompt has multiple generated samples from each decoding strategy\. We therefore aggregate pairwise similarities using a best\-match rule\. For a promptqq, let𝒟qπ\\mathcal\{D\}^\{\\pi\}\_\{q\}be the samples generated by the decoding strategy being evaluated, and let𝒟qL2R\\mathcal\{D\}^\{\\mathrm\{L2R\}\}\_\{q\}be the reference L2R samples\. For a chosen similarity metrics​\(⋅,⋅\)s\(\\cdot,\\cdot\), we compute

BMq​\(s\)=1\|𝒟qπ\|​∑y∈𝒟qπmaxx∈𝒟qL2R⁡s​\(y,x\)\.\\mathrm\{BM\}\_\{q\}\(s\)=\\frac\{1\}\{\|\\mathcal\{D\}^\{\\pi\}\_\{q\}\|\}\\sum\_\{y\\in\\mathcal\{D\}^\{\\pi\}\_\{q\}\}\\max\_\{x\\in\\mathcal\{D\}^\{\\mathrm\{L2R\}\}\_\{q\}\}s\(y,x\)\.That is, each sample from the evaluated decoding strategy is matched to its most similar L2R sample, and we average these best\-match scores\. We then report the mean ofBMq​\(s\)\\mathrm\{BM\}\_\{q\}\(s\)across prompts\.

We apply the same aggregation separately under the valid–valid and success–success filters\. For valid–valid, bothyyand its candidate matchesxxmust be syntactically valid\. For success–success, both must pass the unit tests\. This aggregation asks whether each generated program has a close L2R counterpart\.

### B\.1Additional similarity results

Tables[1](https://arxiv.org/html/2607.26504#A2.T1)and[2](https://arxiv.org/html/2607.26504#A2.T2)report the L2R comparison results for DiffuCoder\-base, DiffuCoder\-instruct, DiffuCoder\-cpGRPO, Dream\-Base, Dream\-instruct, LLaDA\-instruct, LLaDA\-cpGRPO, anddUltra\-coding\-b128on HumanEval and MBPP\. Table[3](https://arxiv.org/html/2607.26504#A2.T3)reports the comparison results for Dream\-Coder\-instruct on LiveCodeBench\. We report TSED, ASTD, and Coarse ASTD under two filtering regimes: valid–valid pairs, where both codes are syntactically valid, and correct–correct pairs, where both codes pass the unit tests\.

Across all models and both datasets, Random vs L2R consistently yields much lower similarity than Entropy, Top\-KKMargin, and Top\-KKvs L2R\. This trend holds not only for syntactically valid codes, but also for correct codes\. In other words, random decoding does not merely produce structurally different incorrect code\. Even among successful generations, structures are meaningfully less similar to L2R outputs than those produced by confidence\-based decoding\. This supports our claim that confidence\-based decoding remains structurally close to L2R, whereas random reveal orders provide a substantially more non\-left\-to\-right reference\.

Table 1:HumanEval code\-similarity to strict L2R baselines using best\-match aggregation\. Dream, DiffuCoder, LLaDA\-instruct, and LLaDA\-cpGRPO results are compared against L2R under the Pass@32 setup, whiledUltra\-coding\-b128is evaluated under its Pass@64 setup\. Valid pairs require both samples to parse; correct pairs require both samples to pass unit tests\.Table 2:MBPP code\-similarity to strict L2R baselines using best\-match aggregation\. Dream, DiffuCoder, LLaDA\-instruct, and LLaDA\-cpGRPO results are compared against L2R under the Pass@32 setup, whiledUltra\-coding\-b128is evaluated under its Pass@64 setup\. Valid pairs require both samples to parse; correct pairs require both samples to pass unit tests\.Table 3:Dream\-Coder\-instruct’s code\-similarity to the L2R baseline on LiveCodeBench v5 and v6, using best\-match aggregation\. For Dream\-Coder, the entropy, top\-KKmargin, and top\-KKgenerations on LCB are identical to each other under temperature 0\.2; we thus report all three as “confidence vs L2R” instead of repeating three times\.
### B\.2Semantic collapse in dUltra planner

Experimental setup\.dUltra\-coding\-b128\[Chenet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib28)\]uses LLaDA as the base masked diffusion language model and augments it with an unmasking planner head\. The base LLaDA model predicts the token values, whereas the planner head predicts which masked positions to reveal\. The planner is first pretrained with the frozen LLaDA backbone, and then the base model and the planner are jointly optimized with GRPO\. The goal is to learn an efficient reveal policy that selects weakly dependent masked positions for parallel unmasking while preserving the token\-prediction quality\.

Our goal is to test whether the learned dUltra unmasking planner produces structurally different outputs from standard decoding policies\. We therefore apply the same similarity analysis as in Section[3\.1](https://arxiv.org/html/2607.26504#S3.SS1)and compare each policy against strict L2R decoding under a Pass@64 evaluation setup\. For a controlled comparison, all policies follow the inference configuration ofdUltra\-coding\-b128, using the same LLaDA backbone, prompt formatting, maximum generation length, diffusion step schedule, block size, temperature, candidate sampling budget, and evaluation pipeline\. For the planner\-based policy, we use the trained dUltra unmasking head to select masked positions for reveal and keep the planner decoding temperature at its default value,0\. The planner\-based sampling scheme still introduces stochasticity through its learned reveal\-position selection procedure\. For planner\-free baselines, we disable the planner head and decode the same LLaDA token predictor with hand\-designed reveal policies: Top\-KK, Top\-KKmargin, entropy\-based selection, random selection, and strict L2R\. In these backbone\-only baselines, we use a temperature0\.20\.2following the standard sampling setup\. Thus, apart from this policy\-specific temperature convention, the comparison controls for the backbone, prompts, decoding budget, and evaluation pipeline, and isolates how the reveal\-position selection rule affects the resulting code structure\. The dUltra experiments follow a Pass@64 protocol and compare all policies against the L2R baseline\.

Results\.Tables[1](https://arxiv.org/html/2607.26504#A2.T1)and[2](https://arxiv.org/html/2607.26504#A2.T2)show that the learned dUltra planner produces outputs that remain highly similar to strict L2R decoding\. On both HumanEval and MBPP, planner decoding has substantially higher valid\-pair similarity to L2R than the original random\-style decoding policy, indicating that the learned planner does not induce a structurally diverse any\-order generation pattern\. Moreover, the planner is more L2R\-like than Top\-KKand Top\-KKmargin on most similarity metrics\. This suggests that explicitly learning an unmasking schedule is not sufficient to avoid semantic collapse, and the planner can still converge to reveal orders whose final code structures closely match those produced by L2R\-style decoding\.

## Appendix CDetails of Section[3\.2](https://arxiv.org/html/2607.26504#S3.SS2)

Experimental setup\.We evaluate positional uncertainty on large\-scale masked diffusion language models for code generation: Dream\-7B\-Base\[Xieet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib11)\], and DiffuCoder\-7B\-Base, DiffuCoder\-7B\-Instruct, DiffuCoder\-7B\-cpGRPO\[Gonget al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib3)\], and LLaDA\-8B\-instruct\[Nieet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib12)\]\. We use problem prompts from HumanEval\[Chenet al\.,[2021](https://arxiv.org/html/2607.26504#bib.bib23)\]and MBPP\[Austinet al\.,[2021](https://arxiv.org/html/2607.26504#bib.bib24)\]\.

For Dream and DiffuCoder, the logged runs use a 256\-token canvas and 256 diffusion steps with temperature 0\.2 and top\-p=0\.95p=0\.95\. LLaDA runs use a 512\-token canvas and 512 denoising steps\. For the released bundle visualizations, we export point data on the full HumanEval\-164 and MBPP\-500 task sets and render standalone plots at a step stride of 16\.

All generated sequences include non\-code text around them, including an instruction/prompt prefix before the code region and a natural language explanation after the code\. We therefore extract the code span before computing positional\-uncertainty statistics\. Since generation is performed at the token level, the generated sequence can be truncated in the middle of a code region or continue beyond the end of the program into natural language text\. In such cases, we discard tokens outside the extracted code span and compute all quantities only on the retained in\-code token positions\. Thus, the masked positions appearing in the definitions ofm​\(v;𝐱t\)m\(v;\{\\mathbf\{x\}\}\_\{t\}\)andLOC​\(v;𝐱t\)\\mathrm\{LOC\}\(v;\{\\mathbf\{x\}\}\_\{t\}\)are restricted to this trimmed code domain\.

Because storing the full vocabulary distribution at every step is too expensive, we do not log the full per\-position vocabulary posterior\. Instead, at each analyzed step, we retain up to 200 selected token IDs, chosen primarily by aggregate token mass, and we store their positional mass profiles over the trimmed code domain\. Committed token IDs are additionally forced into the logged set when necessary, and committed\-local statistics are recorded separately so that committed\-position uncertainty can still be computed even when a committed token would otherwise fall outside the generic 200\-token selection\. Unless otherwise specified, the positional\-uncertainty results in this subsection are reported under a Pass@32 setup against the strict L2R baseline\.

Additional Results\.Figure[6](https://arxiv.org/html/2607.26504#A3.F6)shows representative positional\-uncertainty plots across model families and benchmarks\. Since Dream and DiffuCoder use a 256\-step denoising schedule, whereas LLaDA uses a 512\-step schedule, we visualize step 16 for Dream and DiffuCoder and step 32 for LLaDA\. These choices correspond to the same relative denoising progress, i\.e\.,16/256=32/51216/256=32/512\. Across both HumanEval and MBPP, we observe the same qualitative pattern as in Section[3\.2](https://arxiv.org/html/2607.26504#S3.SS2): many high\-mass tokens remain poorly localized, while committed tokens are biased toward higher localization\. This supports that positional uncertainty is not specific to a single model family or benchmark\.

![Refer to caption](https://arxiv.org/html/2607.26504v1/x7.png)\(a\)Dream, HumanEval, step 16
![Refer to caption](https://arxiv.org/html/2607.26504v1/x8.png)\(b\)DiffuCoder, HumanEval, step 16
![Refer to caption](https://arxiv.org/html/2607.26504v1/x9.png)\(c\)LLaDA, HumanEval, step 32
![Refer to caption](https://arxiv.org/html/2607.26504v1/x10.png)\(d\)Dream, MBPP, step 16
![Refer to caption](https://arxiv.org/html/2607.26504v1/x11.png)\(e\)DiffuCoder, MBPP, step 16
![Refer to caption](https://arxiv.org/html/2607.26504v1/x12.png)\(f\)LLaDA, MBPP, step 32

Figure 6:Additional positional\-uncertainty visualizations across model families and benchmarks\. Dream and DiffuCoder are visualized at step 16 out of 256 denoising steps, while LLaDA is visualized at step 32 out of 512 denoising steps\. Across settings, high aggregate token mass does not necessarily imply high localization, and committed tokens tend to be more localized\.We further visualize how positional uncertainty evolves across denoising steps for DiffuCoder on MBPP\. Figure[7](https://arxiv.org/html/2607.26504#A3.F7)shows the mass\-localization plots at steps0,16,32,48,64,0,16,32,48,64,and8080\. Across the trajectory, high\-mass tokens can remain broadly dispersed over candidate positions, while committed tokens tend to concentrate in regions with higher localization\. This step\-wise view illustrates that positional uncertainty persists over a substantial portion of the inference trajectory\.

![Refer to caption](https://arxiv.org/html/2607.26504v1/x13.png)\(a\)Step 0
![Refer to caption](https://arxiv.org/html/2607.26504v1/x14.png)\(b\)Step 16
![Refer to caption](https://arxiv.org/html/2607.26504v1/x15.png)\(c\)Step 32
![Refer to caption](https://arxiv.org/html/2607.26504v1/x16.png)\(d\)Step 48
![Refer to caption](https://arxiv.org/html/2607.26504v1/x17.png)\(e\)Step 64
![Refer to caption](https://arxiv.org/html/2607.26504v1/x18.png)\(f\)Step 80

Figure 7:Step\-wise positional\-uncertainty visualization for DiffuCoder on MBPP\. Each panel plots aggregate token mass against localization at a different denoising step\. Across the trajectory, high\-mass tokens are not necessarily localized, while committed tokens tend to have higher localization\.
## Appendix DDetails of Section[4\.1](https://arxiv.org/html/2607.26504#S4.SS1)

In this appendix, we provide the theoretical foundation of FlexMDM, details on tree metrics for quantifying any\-order inference, and an evaluation pipeline\.

### D\.1FlexMDM Training

#### Preliminary theory\.

Letαt,βt\\alpha\_\{t\},\\beta\_\{t\}be smooth and monotone schedules withα0=β0=0\\alpha\_\{0\}=\\beta\_\{0\}=0andα1=β1=1\\alpha\_\{1\}=\\beta\_\{1\}=1\. They will be the insertion and unmasking schedules, respectively\. Given a clean sequence𝐱1=\(𝐱11,…,𝐱1L\)∼pdata\{\\mathbf\{x\}\}\_\{1\}=\(\{\\mathbf\{x\}\}\_\{1\}^\{1\},\.\.\.,\{\\mathbf\{x\}\}\_\{1\}^\{L\}\)\\sim p\_\{\\mathrm\{data\}\}of lengthLL, for each positioni∈\[L\]i\\in\[L\], define its insertion and unmasking timesT1iT\_\{1\}^\{i\},T2iT\_\{2\}^\{i\}as

T1i∼α˙t​d​t,T2i∼1t≥T1i⋅β˙t1−βT1i​d​t\.T\_\{1\}^\{i\}\\sim\\dot\{\\alpha\}\_\{t\}dt,\\quad T\_\{2\}^\{i\}\\sim\\mathrm\{1\}\_\{t\\geq T\_\{1\}^\{i\}\}\\cdot\\frac\{\\dot\{\\beta\}\_\{t\}\}\{1\-\\beta\_\{T\_\{1\}^\{i\}\}\}dt\.Fort∈\[0,1\]t\\in\[0,1\], define the \(sorted\) setsts\_\{t\}of inserted indices as

st=\{i∈\[L\]∣t≥T1i\}s\_\{t\}=\\\{i\\in\[L\]\\mid t\\geq T^\{i\}\_\{1\}\\\}withst​\[1\]<st​\[2\]<⋯<st​\[len​\(st\)\]s\_\{t\}\[1\]<s\_\{t\}\[2\]<\\cdots<s\_\{t\}\[\\mathrm\{len\}\(s\_\{t\}\)\], and with the boundary convention thatst​\[0\]=0s\_\{t\}\[0\]=0andst​\[len​\(st\)\+1\]=L\+1s\_\{t\}\[\\mathrm\{len\}\(s\_\{t\}\)\+1\]=L\+1\. Intuitively, this is the set of indices that have been inserted by timett\. The stochastic interpolant𝐱t\{\\mathbf\{x\}\}\_\{t\}, a sequence of length\|st\|\|s\_\{t\}\|, is then constructed by the following masking and deletion procedure:

𝐱ti=\{𝐦if​t<T2st​\[i\],𝐱1st​\[i\]if​t≥T2st​\[i\]\.\{\\mathbf\{x\}\}\_\{t\}^\{i\}=\\begin\{cases\}\\mathbf\{m\}&\\text\{if \}t<T\_\{2\}^\{s\_\{t\}\[i\]\},\\\\ \{\\mathbf\{x\}\}\_\{1\}^\{s\_\{t\}\[i\]\}&\\text\{if \}t\\geq T\_\{2\}^\{s\_\{t\}\[i\]\}\.\\end\{cases\}Similar to a standard MDM, a FlexMDM parametrizes an unmasking posteriorfθ​\(𝐱,t\)​\[i,v\]≈ℙ​\[𝐱1st​\[i\]=v∣𝐱t=𝐱\]f\_\{\\theta\}\(\{\\mathbf\{x\}\},t\)\[i,v\]\\approx\\mathbb\{P\}\[\{\\mathbf\{x\}\}\_\{1\}^\{s\_\{t\}\[i\]\}=v\\mid\{\\mathbf\{x\}\}\_\{t\}=\{\\mathbf\{x\}\}\]; however, it also parametrizes an insertion prediction

gθ​\(𝐱,t\)​\[i\]≈log⁡𝔼​\[st​\[i\]−st​\[i−1\]−1∣𝐱t=𝐱\]\.g\_\{\\theta\}\(\{\\mathbf\{x\}\},t\)\[i\]\\approx\\log\\mathbb\{E\}\[s\_\{t\}\[i\]\-s\_\{t\}\[i\-1\]\-1\\mid\{\\mathbf\{x\}\}\_\{t\}=\{\\mathbf\{x\}\}\]\.Intuitively, this is the log of the expected number of tokens to be inserted between theii\-th and\(i−1\)\(i\-1\)\-th positions in the current𝐱t\{\\mathbf\{x\}\}\_\{t\}\. At training time, we minimize the following loss function:

ℒ​\(θ\)\\displaystyle\\mathcal\{L\}\(\\theta\)=∫01𝔼𝐱1,st,𝐱t​\[−β˙t1−βt​∑i=1len⁡\(𝐱t\)\+1𝟏𝐱ti=𝐦​log⁡fθ​\(𝐱t,t\)​\[i,𝐱1st​\[i\]\]\]​𝑑t⏟unmasking loss\\displaystyle=\\underbrace\{\\int\_\{0\}^\{1\}\\mathbb\{E\}\_\{\{\\mathbf\{x\}\}\_\{1\},s\_\{t\},\{\\mathbf\{x\}\}\_\{t\}\}\\left\[\-\\frac\{\\dot\{\\beta\}\_\{t\}\}\{1\-\\beta\_\{t\}\}\\sum\_\{i=1\}^\{\\operatorname\{len\}\(\{\\mathbf\{x\}\}\_\{t\}\)\+1\}\\mathbf\{1\}\_\{\{\\mathbf\{x\}\}\_\{t\}^\{i\}=\\mathbf\{m\}\}\\log f\_\{\\theta\}\(\{\\mathbf\{x\}\}\_\{t\},t\)\[i,\{\\mathbf\{x\}\}\_\{1\}^\{s\_\{t\}\[i\]\}\]\\right\]\\,dt\}\_\{\\text\{unmasking loss\}\}\(1\)\+∫01𝔼𝐱1,st,𝐱t​\[−α˙t1−αt​∑i=1len​\(𝐱1\)\+1ϕ​\(st​\[i\]−st​\[i−1\]−1,gθ​\(𝐱t,t\)​\[i\]\)\]​𝑑t⏟insertion loss,\\displaystyle\\quad\+\\underbrace\{\\int\_\{0\}^\{1\}\\mathbb\{E\}\_\{\{\\mathbf\{x\}\}\_\{1\},s\_\{t\},\{\\mathbf\{x\}\}\_\{t\}\}\\left\[\-\\frac\{\\dot\{\\alpha\}\_\{t\}\}\{1\-\\alpha\_\{t\}\}\\sum\_\{i=1\}^\{\\mathrm\{len\}\(\{\\mathbf\{x\}\}\_\{1\}\)\+1\}\\phi\\\!\\left\(s\_\{t\}\[i\]\-s\_\{t\}\[i\-1\]\-1,\\,g\_\{\\theta\}\(\{\\mathbf\{x\}\}\_\{t\},t\)\[i\]\\right\)\\right\]\\,dt\}\_\{\\text\{insertion loss\}\},whereϕ​\(x,y\)=ey−x​y\\phi\(x,y\)=e^\{y\}\-xy\.

Notably, this differs from the original setup inKimet al\.\[[2025a](https://arxiv.org/html/2607.26504#bib.bib14)\], where the insertion head directly predicts the expected insertion length, whereas we found that empirically predicting in log space yields stabler training\. Geometrically, insertion counts live on a positive, multiplicative scale and enter inference as Poisson intensities, so the log map converts variation in expected lengths into an unconstrained additive coordinate\. This makes the insertion head predict the natural parameter of the Poisson rate, which better matches a neural network’s real\-valued outputs and avoids the boundary/scale issues of directly regressing a nonnegative expectation\.

#### Implementation details\.

We choose the following insertion and unmasking schedules:

αt=1−\(1−t\)a,βt=1−\(1−t\)a​b,\\alpha\_\{t\}=1\-\(1\-t\)^\{a\},\\qquad\\beta\_\{t\}=1\-\(1\-t\)^\{ab\},\(2\)wherea=b=1\.7a=b=1\.7\.

![Refer to caption](https://arxiv.org/html/2607.26504v1/figures/power_schedules.png)Figure 8:\(Left\)graphs of the insertion and unmasking schedules in \([2](https://arxiv.org/html/2607.26504#A4.E2)\)\.\(Center\)the number of inserted, mask, and clean tokens as a function of time during training, normalized by sequence length\.\(Right\)the number of inserted, mask, and clean tokens as a function of time during inference, normalized by sequence length\.An important practical observation beyond the original FlexMDM formulation inKimet al\.\[[2025a](https://arxiv.org/html/2607.26504#bib.bib14)\]is that, at inference time, we are not locked to the insertion schedule used during training\. Letαt\\alpha\_\{t\}denote the insertion schedule used during training, and letα~t\\tilde\{\\alpha\}\_\{t\}denote a desired insertion schedule at inference time\. We define the monotone time reparameterization

τ​\(t\):=α−1​\(α~t\)\.\\tau\(t\):=\\alpha^\{\-1\}\(\\tilde\{\\alpha\}\_\{t\}\)\.At inference steptt, we query the model at the reparameterized timeτ​\(t\)\\tau\(t\), i\.e\., we use the predictionsfθ​\(x,τ​\(t\)\)f\_\{\\theta\}\(x,\\tau\(t\)\)andgθ​\(x,τ​\(t\)\)g\_\{\\theta\}\(x,\\tau\(t\)\), while evolving the insertion process with the inference\-time scheduleα~t\\tilde\{\\alpha\}\_\{t\}\. In this way, the model is always evaluated at the training\-time corruption level whose insertion fraction matches the desired inference\-time insertion fraction, since

ατ​\(t\)=α~t\.\\alpha\_\{\\tau\(t\)\}=\\tilde\{\\alpha\}\_\{t\}\.Equivalently, the insertion Poisson intensity at gapiiis taken to be

α~˙t1−α~t​exp⁡\(gθ​\(x,τ​\(t\)\)​\[i\]\),\\frac\{\\dot\{\\tilde\{\\alpha\}\}\_\{t\}\}\{1\-\\tilde\{\\alpha\}\_\{t\}\}\\exp\\bigl\(g\_\{\\theta\}\(x,\\tau\(t\)\)\[i\]\\bigr\),while the unmasking process continues to use the original unmasking scheduleβt\\beta\_\{t\}\.

For the power\-family schedules in \([2](https://arxiv.org/html/2607.26504#A4.E2)\), this reparameterization is even more explicit: if the model is trained withαt=1−\(1−t\)a\\alpha\_\{t\}=1\-\(1\-t\)^\{a\}but we want to sample withα~t=1−\(1−t\)a~\\tilde\{\\alpha\}\_\{t\}=1\-\(1\-t\)^\{\\tilde\{a\}\}, then

τ​\(t\)=α−1​\(α~t\)=1−\(1−t\)a~/a\.\\tau\(t\)=\\alpha^\{\-1\}\(\\tilde\{\\alpha\}\_\{t\}\)=1\-\(1\-t\)^\{\\tilde\{a\}/a\}\.This flexibility is practically important\. We found that any\-order inference requires a sufficiently strong scaffold to be inserted early in the trajectory; otherwise, each currently visible position must summarize too broad a region of the eventual sequence, and token predictions become clogged amalgamations of several plausible future locations\. In our experiments, we therefore use a more aggressive insertion schedule at inference time witha~=2\.9\\tilde\{a\}=2\.9, while keeping the same unmasking schedule\. This front\-loads scaffold construction*without*retraining the model parameters\.

The unmasking and insertion dynamics are visualized in Figure[8](https://arxiv.org/html/2607.26504#A4.F8)\. While we insert tokens more aggressively at the early stage of inference, the number of clean tokens remains sub\-linear in this early phase, avoiding premature token commitments while the layout is still uncertain; we also retain a long tail of later insertions so the model can still add missing local details after partial content has been revealed, preserving the main flexibility that mitigates positional uncertainty\.

We obtain the training dataset by processing open\-source Python data from OpenCodeInstruct\[Ahmadet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib71)\], rStar\-Coder\[Liuet al\.,[2025b](https://arxiv.org/html/2607.26504#bib.bib72)\], KodCode\-V1\-SFT\-4o\[Xuet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib73)\], and opc\-sft\-stage2\-educational\[Huanget al\.,[2025a](https://arxiv.org/html/2607.26504#bib.bib74)\], resulting in≈\\approx2\.6M training sequences\. For OpenCodeInstruct, we keep only those samples whoseaverage\_test\_scoreis at least 0\.9 and from which valid Python code is extracted, resulting in 1\.85M samples out of the total 5M\. From rStar\-Coder, we use theseed\_sftandsynthetic\_sftsplits, keeping only the samples verified as passing their tests, which yields 81K and 369K samples respectively\. We use the entirety of KodCode\-V1\-SFT\-4o \(209K samples\) and opc\-sft\-stage2\-educational \(118K samples\) without additional filtering\. During training, we up\-sample rStar\-Coder, KodCode\-V1\-SFT\-4o, and opc\-sft\-stage2\-educational by a factor of two relative to OpenCodeInstruct\.

We initialize from Dream\-Coder 7B\[Xieet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib11)\], while attaching an auxiliary head and an AdaLN time embedding\[Peebles and Xie,[2023](https://arxiv.org/html/2607.26504#bib.bib70)\]for insertion prediction\. We use the AdamW\[Loshchilov and Hutter,[2019](https://arxiv.org/html/2607.26504#bib.bib2)\]optimizer withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95, and weight decay 0\.01\. We use a learning rate schedule with linear warmup and cosine decay, with warmup ratio 0\.1 and peak learning rate of10−510^\{\-5\}for the backbone and2×10−52\\times 10^\{\-5\}for the insertion head\. We train with the loss in \([1](https://arxiv.org/html/2607.26504#A4.E1)\) for 50000 optimizer steps, with global batch size 576, which took 3 days on 16 H100’s\.

### D\.2Training FlexMDM in Insertion\-Progress Coordinates

The standard FlexMDM construction conditions the model on a raw time variablet∈\[0,1\]t\\in\[0,1\], together with an insertion scheduleαt\\alpha\_\{t\}and an unmasking scheduleβt\\beta\_\{t\}\. We instead propose to condition the model directly on the*insertion progress*

τ:=αt∈\[0,1\]\.\\tau:=\\alpha\_\{t\}\\in\[0,1\]\.This removes the dependence of the model interface on the particular parameterization of time\. Raw timettis then used only as an external sampler clock, while the network always receives the schedule\-invariant coordinateτ\\tau\.

Letα:\[0,1\]→\[0,1\]\\alpha:\[0,1\]\\to\[0,1\]be a strictly increasing insertion schedule and letβ:\[0,1\]→\[0,1\]\\beta:\[0,1\]\\to\[0,1\]be the unmasking schedule used in the original raw\-time formulation\. Define the corresponding unmasking schedule in insertion\-progress coordinates by

B​\(τ\):=β​\(α−1​\(τ\)\)\.B\(\\tau\):=\\beta\\left\(\\alpha^\{\-1\}\(\\tau\)\\right\)\.Thus, after changing variables fromtttoτ\\tau, insertion progress is the identity schedule, while unmasking is governed byBB\.

For a clean sequencex1x\_\{1\}, letxτx\_\{\\tau\}denote the partially corrupted sequence at insertion progressτ\\tau, and letsτs\_\{\\tau\}be the alignment map from positions inxτx\_\{\\tau\}to positions inx1x\_\{1\}\. The model is trained as

fθ​\(xτ,τ\)​\[i,v\]≈ℙ​\(xsτ​\[i\]1=v∣xτ\),f\_\{\\theta\}\(x\_\{\\tau\},\\tau\)\[i,v\]\\approx\\mathbb\{P\}\\left\(x^\{1\}\_\{s\_\{\\tau\}\[i\]\}=v\\mid x\_\{\\tau\}\\right\),and

gθ​\(xτ,τ\)​\[i\]≈log⁡𝔼​\[sτ​\[i\]−sτ​\[i−1\]−1∣xτ\],g\_\{\\theta\}\(x\_\{\\tau\},\\tau\)\[i\]\\approx\\log\\mathbb\{E\}\\left\[s\_\{\\tau\}\[i\]\-s\_\{\\tau\}\[i\-1\]\-1\\mid x\_\{\\tau\}\\right\],wherefθf\_\{\\theta\}predicts clean token identities andgθg\_\{\\theta\}predicts the log of the expected number of missing tokens in each gap\.

In theτ\\tau\-coordinate, the FlexMDM training objective becomes

ℒ\(θ\)=∫01𝔼\[\\displaystyle\\mathcal\{L\}\(\\theta\)=\\int\_\{0\}^\{1\}\\mathbb\{E\}\\Bigg\[−B′​\(τ\)1−B​\(τ\)​∑i:xτi=𝐦log⁡fθ​\(xτ,τ\)​\[i,x1sτ​\[i\]\]\\displaystyle\-\\frac\{B^\{\\prime\}\(\\tau\)\}\{1\-B\(\\tau\)\}\\sum\_\{i:x\_\{\\tau\}^\{i\}=\\mathbf\{m\}\}\\log f\_\{\\theta\}\(x\_\{\\tau\},\\tau\)\\left\[i,x\_\{1\}^\{s\_\{\\tau\}\[i\]\}\\right\]\+11−τ∑iϕ\(sτ\[i\]−sτ\[i−1\]−1,gθ\(xτ,τ\)\[i\]\)\]dτ\.\\displaystyle\+\\frac\{1\}\{1\-\\tau\}\\sum\_\{i\}\\phi\\left\(s\_\{\\tau\}\[i\]\-s\_\{\\tau\}\[i\-1\]\-1,\\,g\_\{\\theta\}\(x\_\{\\tau\},\\tau\)\[i\]\\right\)\\Bigg\]\\,d\\tau\.whereϕ​\(x,y\)=ey−x​y\\phi\(x,y\)=e^\{y\}\-xy\. The insertion weight is1/\(1−τ\)1/\(1\-\\tau\)because

α˙t1−αt​d​t=11−τ​d​τ,\\frac\{\\dot\{\\alpha\}\_\{t\}\}\{1\-\\alpha\_\{t\}\}\\,dt=\\frac\{1\}\{1\-\\tau\}\\,d\\tau,and the unmasking weight isB′​\(τ\)/\(1−B​\(τ\)\)B^\{\\prime\}\(\\tau\)/\(1\-B\(\\tau\)\)because

β˙t1−βt​d​t=B′​\(τ\)1−B​\(τ\)​d​τ\.\\frac\{\\dot\{\\beta\}\_\{t\}\}\{1\-\\beta\_\{t\}\}\\,dt=\\frac\{B^\{\\prime\}\(\\tau\)\}\{1\-B\(\\tau\)\}\\,d\\tau\.Thus this objective is simply the original FlexMDM objective written in the coordinateτ\\tau\.

Equivalently, training data can be generated directly in theτ\\tau\-coordinate\. For each clean tokenjj, sample an insertion coordinate

Aj∼Unif​\(0,1\)\.A\_\{j\}\\sim\\mathrm\{Unif\}\(0,1\)\.Conditional onAjA\_\{j\}, sample an unmasking coordinateUj≥AjU\_\{j\}\\geq A\_\{j\}by drawing

Zj∼Unif​\(B​\(Aj\),1\),Uj=B−1​\(Zj\)\.Z\_\{j\}\\sim\\mathrm\{Unif\}\(B\(A\_\{j\}\),1\),\\qquad U\_\{j\}=B^\{\-1\}\(Z\_\{j\}\)\.At progress levelτ\\tau, tokenjjis deleted ifτ<Aj\\tau<A\_\{j\}, masked ifAj≤τ<UjA\_\{j\}\\leq\\tau<U\_\{j\}, and revealed ifUj≤τU\_\{j\}\\leq\\tau\. The model is then given\(xτ,τ\)\(x\_\{\\tau\},\\tau\)rather than\(xt,t\)\(x\_\{t\},t\)\.

At inference time, choose any desired event\-time insertion schedule

Γ:\[0,1\]→\[0,1\]\.\\Gamma:\[0,1\]\\to\[0,1\]\.At sampler timett, setτt:=Γ​\(t\)\\tau\_\{t\}:=\\Gamma\(t\)and query the model as

fθ​\(xt,τt\),gθ​\(xt,τt\)\.f\_\{\\theta\}\(x\_\{t\},\\tau\_\{t\}\),\\qquad g\_\{\\theta\}\(x\_\{t\},\\tau\_\{t\}\)\.The insertion CTMC in event time is then

Rtins​\(x,x⊲im\)=Γ˙​\(t\)1−Γ​\(t\)​exp⁡\(gθ​\(x,Γ​\(t\)\)​\[i\]\)\.R\_\{t\}^\{\\mathrm\{ins\}\}\(x,x\\triangleleft\_\{i\}m\)=\\frac\{\\dot\{\\Gamma\}\(t\)\}\{1\-\\Gamma\(t\)\}\\exp\\bigl\(g\_\{\\theta\}\(x,\\Gamma\(t\)\)\[i\]\\bigr\)\.Thus the sampler uses the event\-time hazard of the chosen inference schedule, but the model itself is always conditioned on the insertion\-progress valueτt=Γ​\(t\)\\tau\_\{t\}=\\Gamma\(t\)\.

With an event\-time unmasking scheduleΔ​\(t\)\\Delta\(t\), the unmasking transition rate is

Rtunmask​\(x,x​\[i←v\]\)=Δ˙​\(t\)1−Δ​\(t\)​fθ​\(x,Γ​\(t\)\)​\[i,v\]\.R\_\{t\}^\{\\mathrm\{unmask\}\}\(x,x\[i\\leftarrow v\]\)=\\frac\{\\dot\{\\Delta\}\(t\)\}\{1\-\\Delta\(t\)\}f\_\{\\theta\}\(x,\\Gamma\(t\)\)\[i,v\]\.If one wants to preserve the original training unmasking dynamics inτ\\tau\-coordinates, then the event\-time unmasking schedule should just beΔ​\(t\)=B​\(Γ​\(t\)\)\\Delta\(t\)=B\(\\Gamma\(t\)\)\.

The key distinction from a raw\-time\-conditioned model is that no inverse\-time correction is required at inference\. A model trained on raw time underαtrain\\alpha\_\{\\mathrm\{train\}\}must be queried at

αtrain−1​\(Γ​\(t\)\)\\alpha\_\{\\mathrm\{train\}\}^\{\-1\}\(\\Gamma\(t\)\)when using a new inference scheduleΓ\\Gamma\. In contrast, an insertion\-progress\-conditioned model is queried directly at

τt=Γ​\(t\)\.\\tau\_\{t\}=\\Gamma\(t\)\.Therefore changing the insertion schedule at inference only changes how quickly the sampler moves through insertion\-progress space; it does not change the semantic meaning of the model’s conditioning variable\.

### D\.3Any\-Order Metrics

To study the any\-order inference ability of FlexMDM, we evaluate both FlexMDM and Dream\-Coder on the three tree\-based metrics, each measuring the any\-order performance from a different perspective\. Below, we first describe how we parse the extracted Python code into a tree, and then define the metrics precisely\.

#### Parsing into trees\.

We base our metrics on the natural syntactic structure already exposed by the Python AST \(Abstract Syntax Tree\) parser\. For each generated sample, we first extract the Python code span and parse it into anast\.Module, using standard fallbacks for fenced code blocks, closing fences, and longest valid prefixes when direct parsing fails\. We then coarsen the full Python AST into a simplified statement\-level tree: expression\-level nodes are collapsed into their nearest containing statement, docstrings are collapsed into their owner node, and trailing generated tests or examples are optionally grouped into a synthetic reference node\. Each Python token is assigned to the deepest compatible AST node, so every tree node is associated with the set of generated tokens in its subtree\.

#### Coverage Before Commitment \(CBC\)\.

CBC measures whether the model previews multiple sibling subtrees before committing to finishing one of them\. This distinguishes genuinely any\-order behavior from a merely serial traversal: at a branching point in the program tree, a model with higher CBC begins work on several child blocks before completing any one\.

More precisely, letvvbe a non\-leaf node in the syntax tree withkkchildren \(k≥1k\\geq 1\)\. We sayvvissplitifk≥2k\\geq 2, andnon\-splitifk=1k=1\. Writechild​\(v\)=\{c1,…,ck\}\\mathrm\{child\}\(v\)=\\\{c\_\{1\},\\ldots,c\_\{k\}\\\}, where eachcic\_\{i\}represents the subtree rooted at theii\-th child node ofvv\. Lettcommit​\(v\)t\_\{\\mathrm\{commit\}\}\(v\)be the first generation step at which any child blockci∈child​\(v\)c\_\{i\}\\in\\mathrm\{child\}\(v\)becomes complete, and letNstarted​\(v\)N\_\{\\mathrm\{started\}\}\(v\)be the number of child blocks ofvvthat has at least one generated token by that time\. We define

CBC​\(v\)=Nstarted​\(v\)k\.\\mathrm\{CBC\}\(v\)=\\frac\{N\_\{\\mathrm\{started\}\}\(v\)\}\{k\}\.
The normalization bykkmakes CBC comparable across different branching factors and makes the score measure how much of the*available*sibling structure was explored before commitment\. At a split node,CBC​\(v\)=1/k\\mathrm\{CBC\}\(v\)=1/kis the strictly sequential baseline, while values closer to11indicate that the model opened most sibling blocks before completing the first one\. For a non\-split node, there is no branching opportunity in the inherent code structure, so sequential generation is the natural behavior; settingCBC​\(v\)=1\\mathrm\{CBC\}\(v\)=1ensures that the metric does not penalize the model for being sequential when the program structure itself is sequential\.

Lastly, after computingCBC​\(v\)\\mathrm\{CBC\}\(v\)for all non\-leaf nodesvv, we aggregate by reporting both the overall average over all non\-leaf nodes and the split\-only average over nodes withk≥2k\\geq 2; the former measures alignment with the full tree structure, while the latter isolates behavior at genuine branching points\.

#### Return to Unfinished Blocks \(RUB\)\.

RUB measures whether the model returns to a partially generated child block after moving away from it\. This captures a stronger form of any\-order inference than CBC: rather than only checking whether several sibling blocks were previewed before commitment, RUB asks whether generation actually moves back and forth across unfinished parts of the same local tree region\.

More precisely, letvvbe a non\-leaf split node in the syntax tree\. For a child blockci∈child​\(v\)c\_\{i\}\\in\\mathrm\{child\}\(v\), lettopen​\(ci\)t\_\{\\mathrm\{open\}\}\(c\_\{i\}\)be the first generation step at which any token incic\_\{i\}is generated, and lettfinish​\(ci\)t\_\{\\mathrm\{finish\}\}\(c\_\{i\}\)be the first generation step at which this subtree becomes complete\. We say thatcic\_\{i\}isreturned toif, aftercic\_\{i\}is opened and before it is finished, the model generates at least one token belonging to another sibling blockcj∈child​\(v\)c\_\{j\}\\in\\mathrm\{child\}\(v\),j≠ij\\neq i\. LetNreturned​\(v\)N\_\{\\mathrm\{returned\}\}\(v\)be the number of children ofvvthat are returned to\. We define

RUB​\(v\)=Nreturned​\(v\)k\.\\mathrm\{RUB\}\(v\)=\\frac\{N\_\{\\mathrm\{returned\}\}\(v\)\}\{k\}\.
The normalization bykkagain makes RUB comparable across branching factors and measures what fraction of the available sibling blocks are revisited after being left unfinished\. For a split node,RUB​\(v\)=0\\mathrm\{RUB\}\(v\)=0indicates no back\-and\-forth behavior across unfinished sibling blocks, while values closer to11indicate that many child blocks were revisited during generation\. For a non\-split node, there is no alternative sibling block to move to, so we setRUB​\(v\)=1\\mathrm\{RUB\}\(v\)=1to avoid penalizing sequential generation when the underlying code structure offers no branching opportunity\.

Graded Return to Unfinished Blocks \(RUB\+\)\.RUB is binary at each child: it saturates after a single return, so it cannot distinguish a trace that opens every sibling once and then closes them in turn from one that alternates between siblings many times\. RUB\+refines RUB into a three\-level graded score, while capping the credit at two returns so that it does not reward pathological alternation\.

More precisely, letvvbe a non\-leaf split node and letci∈child​\(v\)c\_\{i\}\\in\\mathrm\{child\}\(v\)\. Restricting attention to generation steps that reveal a token in some child ofvv, definevisits​\(ci\)\\mathrm\{visits\}\(c\_\{i\}\)to be the number of maximal contiguous runs of steps during which the model unmasks tokens belonging tocic\_\{i\}’s subtree; equivalently,visits​\(ci\)−1\\mathrm\{visits\}\(c\_\{i\}\)\-1counts the number of times the model leavescic\_\{i\}, works on a sibling, and returns\. We define the per\-child score

s​\(ci\)=min⁡\(visits​\(ci\)−1,2\)2∈\{0,0\.5,1\},s\(c\_\{i\}\)=\\frac\{\\min\\bigl\(\\mathrm\{visits\}\(c\_\{i\}\)\-1,2\\bigr\)\}\{2\}\\in\\\{0,0\.5,1\\\},so thats​\(ci\)s\(c\_\{i\}\)takes value0,0\.50\.5, or11according to whether the model never returns tocic\_\{i\}, returns exactly once, or returns at least twice\. We then set

RUB\+​\(v\)=1k​∑i=1ks​\(ci\)\.\\mathrm\{RUB\}^\{\+\}\(v\)=\\frac\{1\}\{k\}\\sum\_\{i=1\}^\{k\}s\(c\_\{i\}\)\.The cap at two returns reflects the view that genuine any\-order generation should exhibit a small number of revisits per branch — the kind a human writing code might exhibit — rather than frantic alternation; replacing the binary “did the model ever return” signal with this three\-level score lets us separate single\-return traces from richer back\-and\-forth without rewarding pathological interleaving\. As with RUB,RUB\+​\(v\)=0\\mathrm\{RUB\}^\{\+\}\(v\)=0indicates strictly serial generation atvv, and we setRUB\+​\(v\)=1\\mathrm\{RUB\}^\{\+\}\(v\)=1for non\-split nodes by convention\.

Lastly, same as above, we aggregateRUB​\(v\)\\mathrm\{RUB\}\(v\)andRUB\+​\(v\)\\mathrm\{RUB\}^\{\+\}\(v\)by reporting both the overall average over all non\-leaf nodes and the split\-only average over nodes withk≥2k\\geq 2\.

#### Open\-Block Width \(OBW\)\.

OBW measures how many sibling blocks the model keeps simultaneously unfinished during generation\. This captures the local breadth of any\-order inference: at a branching point in the program tree, a model with higher OBW maintains progress on several child blocks at once rather than completing one child block before opening the next\.

More precisely, letvvbe a non\-leaf node in the syntax tree\. For each generation steptt, letNopen​\(v,t\)N\_\{\\mathrm\{open\}\}\(v,t\)be the number of children blocksci∈child​\(v\)c\_\{i\}\\in\\mathrm\{child\}\(v\)that has at least one generated token but is not yet complete at timett\. We define

OBW​\(v\)=maxt⁡Nopen​\(v,t\)k\.\\mathrm\{OBW\}\(v\)=\\frac\{\\max\_\{t\}N\_\{\\mathrm\{open\}\}\(v,t\)\}\{k\}\.
Again, the normalization bykkmakes OBW comparable across branching factors and measures the largest fraction of available sibling blocks that are simultaneously active\. For a non\-split node, there is only one child block and we automatically getOBW​\(v\)=1\\mathrm\{OBW\}\(v\)=1, which also avoids penalizing sequential generation when the code structure itself is sequential\.

Same as the above two metrics, we report both overall and split\-only average\.

Table 4:Tree\-based any\-order metrics for Dream\-Coder\-7B\-Base and FlexMDM \(ours\) under bothfull\_outputevaluation, without sanitizing, andcode\_onlyevaluation, after discarding generated test\-case nodes\. Overall scores average over all non\-leaf nodes, while split\-only scores average only over branching nodes\.
#### Complete any\-order metrics results\.

We use the default inference setup in the original Dream\-Coder paper\[Xieet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib11)\], with temperature 0\.2 on HumanEval and 0\.1 on MBPP, and with confidence based on negative entropy on both benchmarks\.

We observed that the models sometimes generate some test cases after the function body itself, such as standaloneprint,assert, or\_\_main\_\_statements, which may confound the evaluation on the actual function body\. Thus, we sanitize the code and discard all nodes corresponding to the test cases\. We report the any\-order metrics both before and after the sanitizing, labeled asfull\_outputandcode\_only, respectively\.

As shown in Table[4](https://arxiv.org/html/2607.26504#A4.T4), with both variants, our FlexMDM significantly outperforms Dream\-Coder\. Notably, the gap on split\-only RUB shows that at branching points in the program tree, Dream\-Coder almost never revisits a sibling block once it has moved away \(split RUB≈0\.07\\approx 0\.07on HumanEval and≈0\.01\\approx 0\.01on MBPP\), generating in a near\-autoregressive manner, whereas FlexMDM revisits roughly half of all sibling branches on HumanEval \(split RUB0\.620\.62and0\.470\.47underfull\_outputandcode\_onlyrespectively\)\. The ratio between split\-only RUB\+and split\-only RUB further sharpens this picture: a child returned to exactly once contributes11to RUB but only12\\tfrac\{1\}\{2\}to RUB\+, while a child returned to at least twice contributes11to both, so this ratio reads off the fraction of returned children that were revisited at least twice\. For Dream\-Coder the ratio is≈0\.50\\approx 0\.50across both benchmarks, meaning that on the rare occasions it does return to an unfinished sibling, it does so exactly once; for FlexMDM the ratio rises to≈0\.70\\approx 0\.70–0\.820\.82, meaning that a substantial share of revisits involve two or more genuine returns rather than a single one\-pass ABA excursion\.

### D\.4FlexMDM Inference and Details on Evaluation Setup

At inference time, we follow the standard FlexMDM inference algorithm with top\-K for unmasking, as detailed in Alg\.[9](https://arxiv.org/html/2607.26504#A4.F9)\. For all models, we use 512 sampling steps; Dream\-Coder generates up to 512 new tokens, while FlexMDM caps the total sequence length \(prompt plus generation\) at 768 for HumanEval and 1100 for MBPP\. For Dream\-Coder, since using low temperature with the confidence\-based sampling strategy collapses to one single generation, we use temperature 1\.0 for all baseline models for diversity at pass@kk\. For FlexMDM, due to the inherent stochasticity in the insertion, we use a token temperature of 0\.1; the insertion temperature \(Appendix[D\.5](https://arxiv.org/html/2607.26504#A4.SS5)\) isTins=0\.6T\_\{\\mathrm\{ins\}\}=0\.6on MBPP/MBPP\+ and the neutralTins=1T\_\{\\mathrm\{ins\}\}=1on HumanEval/HumanEval\+\. We use the same code extraction and grading criteria from Dream\-Coder’s official codebase across all models and benchmarks, executing each sample against the benchmark’s full test suite under a 30\-second wall\-clock limit\. Results are summarized in Table[5](https://arxiv.org/html/2607.26504#A4.T5)\.

Discussion on FlexMDM’s downstream performance\.FlexMDM does not dominate Dream\-Coder\-7B uniformly: it trails at lowkkon HumanEval and by≤1\.6\{\\leq\}1\.6points atk≥2k\\geq 2on MBPP\. Two factors are at play\. First, Dream\-Coder\-7B\-Base is already a strong model — the best or second\-best on HumanEval and MBPP among comparable baselines\[Xieet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib11)\]— so the headroom left to fine\-tuning is limited\. Second, the token temperature never touches the insertion process, whose stochasticity perturbs*where*code is laid out and therefore matters most at smallkk\. This axis has its own dial: as Appendix[D\.5](https://arxiv.org/html/2607.26504#A4.SS5)shows, sharpening insertion placement improves every Pass@kkon MBPP/MBPP\+, whereas on HumanEval we keep the neutral setting, trading Pass@1 for the placement diversity that drives the Pass@16 gains\.

Subroutine 1:FlexMDM inference1:Learned functions

\(fθ,gθ\)\(f\_\{\\theta\},g\_\{\\theta\}\)
2:Discretization

0=t1<⋯<tN=10=t\_\{1\}<\\dots<t\_\{N\}=1
3:Insertion, Unmasking schedule

αt,βt\\alpha\_\{t\},\\beta\_\{t\}
4:Initialize

Xt1←empty\_sequenceX\_\{t\_\{1\}\}\\leftarrow\\texttt\{empty\\\_sequence\}
5:for

j=1j=1to

N−1N\-1do

6:

τ←tj\+1−tj\\tau\\leftarrow t\_\{j\+1\}\-t\_\{j\}
7:Invoke Subroutine 2 for unmasking

8:for

iiin\[len​\(Xtj\)\]\+1\[\\mathrm\{len\}\(X\_\{t\_\{j\}\}\)\]\+1do

9:Set insertion rate

rα←α˙tj1−αtj⋅τr\_\{\\alpha\}\\leftarrow\\tfrac\{\\dot\{\\alpha\}\_\{t\_\{j\}\}\}\{1\-\\alpha\_\{t\_\{j\}\}\}\\cdot\\tau
10:Sample

ℓ∼Poi​\(rα⋅exp⁡\(gθ​\(Xtj,tj\)​\[i\]\)\)\\ell\\sim\\mathrm\{Poi\}\\big\(r\_\{\\alpha\}\\cdot\\exp\(g\_\{\\theta\}\(X\_\{t\_\{j\}\},t\_\{j\}\)\[i\]\)\\big\)
11:Insertℓ\\ellmasks betweenXtji−1X\_\{t\_\{j\}\}^\{i\-1\}andXtjiX\_\{t\_\{j\}\}^\{i\}

12:endfor

13:endfor

14:return

XtNX\_\{t\_\{N\}\}

Subroutine 2:Unmasking step \(top\-kk\)1:Set unmasking rate

rβ←β˙tj1−βtj⋅τr\_\{\\beta\}\\leftarrow\\tfrac\{\\dot\{\\beta\}\_\{t\_\{j\}\}\}\{1\-\\beta\_\{t\_\{j\}\}\}\\cdot\\tau
2:Sample

k∼Poi​\(rβ⋅\|\{i∣Xtji=𝐦\}\|\)k\\sim\\mathrm\{Poi\}\(r\_\{\\beta\}\\cdot\|\\\{i\\mid X\_\{t\_\{j\}\}^\{i\}=\\mathbf\{m\}\\\}\|\)
3:for

i∈\{i∣Xtji=𝐦\}i\\in\\\{i\\mid X\_\{t\_\{j\}\}^\{i\}=\\mathbf\{m\}\\\}do

4:Sample

vi∼Cat​\(fθ​\(Xtj,tj\)​\[i\]\)v^\{i\}\\sim\\mathrm\{Cat\}\(f\_\{\\theta\}\(X\_\{t\_\{j\}\},t\_\{j\}\)\[i\]\)
5:Compute

𝒞i=fθ​\(Xtj,tj\)​\[i,vi\]\\mathcal\{C\}^\{i\}=f\_\{\\theta\}\(X\_\{t\_\{j\}\},t\_\{j\}\)\[i,v^\{i\}\]
6:endfor

7:for

iiin

argmaxk​\(𝒞\)\\mathrm\{argmaxk\}\(\\mathcal\{C\}\)do

8:Commit

Xtji←viX\_\{t\_\{j\}\}^\{i\}\\leftarrow v^\{i\}
9:endfor

Figure 9:FlexMDM inference\.At each step we performunmaskingandinsertion\. Forunmasking, we use confidence\-based top\-kkselection\. The number of mask tokens toinsertand the number of tokens tounmaskare drawn from a Poisson distribution\.Notation:Cat\\mathrm\{Cat\},Poi\\mathrm\{Poi\}denote the categorical and Poisson distribution, respectively\.argmaxk​\(𝒞\)\\mathrm\{argmaxk\}\(\\mathcal\{C\}\)is the indices set of thekklargest components of𝒞\\mathcal\{C\}\. Adapted fromKimet al\.\[[2025a](https://arxiv.org/html/2607.26504#bib.bib14)\]\.Table 5:Full pass@k results for FlexMDM and Dream\-Coder\-7B\-Base on HumanEval, HumanEval\+, MBPP, and MBPP\+\. FlexMDM decodes with insertion temperatureTins=0\.6T\_\{\\mathrm\{ins\}\}=0\.6on MBPP/MBPP\+ andTins=1T\_\{\\mathrm\{ins\}\}=1on HumanEval/HumanEval\+ \(Appendix[D\.5](https://arxiv.org/html/2607.26504#A4.SS5)\)\.
### D\.5Insertion Temperature

Just as the token temperature shapes the unmasking posterior, the insertion process carries its own natural temperature\. At each step, the insertion step of Alg\.[9](https://arxiv.org/html/2607.26504#A4.F9)\(Subroutine 1\) draws an independent Poisson count in every gapiiwith rateλi=r​egi\\lambda\_\{i\}=r\\,e^\{g\_\{i\}\}, whererris the scalar schedule hazard andgi=gθ​\(𝐱t,t\)​\[i\]g\_\{i\}=g\_\{\\theta\}\(\{\\mathbf\{x\}\}\_\{t\},t\)\[i\]\. By Poisson superposition and thinning, this is exactly equivalent to first drawing the*total*number of insertionsN∼Poisson​\(Λ\)N\\sim\\mathrm\{Poisson\}\(\\Lambda\),Λ=r​∑iegi\\Lambda=r\\sum\_\{i\}e^\{g\_\{i\}\}, and then*placing*theNNmasks multinomially with probabilitiesp=softmax​\(g\)p=\\mathrm\{softmax\}\(g\)\. The single headgθg\_\{\\theta\}thus encodes two separately tunable quantities —*how many*tokens to insert and*where*— and we define the*insertion temperature*TinsT\_\{\\mathrm\{ins\}\}by tempering the placement while holding the total fixed:

λi​\(Tins\)=Λ⋅softmax​\(g/Tins\)i,\\lambda\_\{i\}\(T\_\{\\mathrm\{ins\}\}\)\\;=\\;\\Lambda\\cdot\\mathrm\{softmax\}\(g/T\_\{\\mathrm\{ins\}\}\)\_\{i\},which reduces toλi\\lambda\_\{i\}atTins=1T\_\{\\mathrm\{ins\}\}=1\. Because the placement distribution is normalized for everyTinsT\_\{\\mathrm\{ins\}\}, the expected number of insertions — and hence the length statistics the model was trained on — is invariant to the knob\. LoweringTinsT\_\{\\mathrm\{ins\}\}makes structural commitments more decisive by concentrating insertions in the gaps the model is most confident about; raising it diversifies placement\. Count preservation is what makes placement the right axis to temper: naively rescaling the rates asegi/Te^\{g\_\{i\}/T\}changes the expected length \(the softmax denominator is exactly the missing normalization\), while suppressing the stochasticity of the count channel instead drives the sampler off the stochastic\-insertion process it was trained on — in our experiments collapsing generation length\. The knob is inference\-only and adds no compute\.

Table[6](https://arxiv.org/html/2607.26504#A4.T6)sweepsTinsT\_\{\\mathrm\{ins\}\}under the identical protocol as Table[5](https://arxiv.org/html/2607.26504#A4.T5), and the two benchmark families respond in opposite ways, tracking their prompt structure\. MBPP prompts give a one\-line description with no code scaffold, so the model must lay out the program structure itself, and placement noise perturbs exactly these least\-reversible early commitments\. Sharpening toTins=0\.6T\_\{\\mathrm\{ins\}\}=0\.6accordingly improves on the neutral setting at*every*kk: it is best or tied at everykkon MBPP, within half a point of the sweep\-best on MBPP\+, and lifts Pass@1 by\+2\.5\+2\.5\(MBPP\) and\+2\.1\+2\.1\(MBPP\+\) — past Dream\-Coder\-7B\-Base\. HumanEval prompts instead pin the scaffold in advance \(signature, docstring, worked examples\), so placement stochasticity acts as useful exploration over valid realizations — the source of the Pass@16 advantage; sharpening recovers less than a point of Pass@1 while giving up performance at largerkk, so the neutralTins=1T\_\{\\mathrm\{ins\}\}=1is preferable\. We therefore selectTinsT\_\{\\mathrm\{ins\}\}per benchmark in Table[5](https://arxiv.org/html/2607.26504#A4.T5)\(0\.60\.6on MBPP/MBPP\+,11on HumanEval/HumanEval\+\), just as token temperatures are routinely chosen per benchmark\[Xieet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib11)\]\. In the language of Section[3\.2](https://arxiv.org/html/2607.26504#S3.SS2),TinsT\_\{\\mathrm\{ins\}\}is a dial on the exploration–commitment trade\-off of any\-order inference: it sharpens placement precisely in the high\-positional\-uncertainty regime where the prompt does not anchor it\.

Table 6:Insertion\-temperature sweep for FlexMDM \(16 samples/task, identical protocol to Table[5](https://arxiv.org/html/2607.26504#A4.T5)\); Dream\-Coder\-7B\-Base in italics for reference\.Boldmarks the per\-benchmark setting adopted in Table[5](https://arxiv.org/html/2607.26504#A4.T5)\. Sharpening to0\.60\.6helps at everykkon MBPP/MBPP\+; on HumanEval/HumanEval\+, where prompts already fix the scaffold, sharpening recovers little Pass@1 and costs performance at largerkk\.

## Appendix EDetails of Section[4\.2](https://arxiv.org/html/2607.26504#S4.SS2)

### E\.1Data Pre\-processing

We use TinyGSM\[Liuet al\.,[2023](https://arxiv.org/html/2607.26504#bib.bib64)\]as the training corpus for all models in Section[4\.2](https://arxiv.org/html/2607.26504#S4.SS2)\. Each example consists of a promptPPand a target Python solution𝐱\{\\mathbf\{x\}\}\. For LatentMDM, we segment each target solution using\\newlineas a delimiter, treating each line of code as a semantic segment\. We fix the maximum number of segments toLs=16L\_\{s\}=16during training; examples with more thanLsL\_\{s\}segments are truncated, while examples with fewer segments are padded with empty all\-<EOS\>segments\.

Each segment is tokenized and padded to a maximum length ofLseg=32L\_\{\\mathrm\{seg\}\}=32\. If a segment exceeds this length, we truncate it and replace the final token with<EOS\>\. We definelen⁡\(𝐲i\)\\operatorname\{len\}\(\{\\mathbf\{y\}\}^\{i\}\)as the position of the first<EOS\>token in segment𝐲i\{\\mathbf\{y\}\}^\{i\}, which determines the effective length used in the loss and segment\-level scoring during inference\.

For the autoregressive and token\-level MDM baselines, we do not apply segmentation\. Instead, we concatenate the prompt and target solution, truncate the resulting sequence to 512 tokens, and pad shorter sequences with<EOS\>\. All models use the Qwen tokenizer\[Yanget al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib100)\]with vocabulary size 151,645\.

### E\.2Architecture

![Refer to caption](https://arxiv.org/html/2607.26504v1/x19.png)Figure 10:LatentMDM training pipeline\.We use a 125M\-parameter architecture for LatentMDM and all token\-space baselines, following the setup of\[Kimet al\.,[2026](https://arxiv.org/html/2607.26504#bib.bib69)\]\. All Transformer blocks follow the Qwen2\-style design\[Yanget al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib100)\]\. We modify the attention mask according to each module: the segment encoder, LatentMDM, and token\-level MDM baseline use bidirectional attention, while the autoregressive decoder and AR baseline use causal attention\. Unless otherwise specified, all modules use a hidden dimensionH=512H=512and rotary positional embeddings \(RoPE\)\.

LatentMDM\.The LatentMDM consists of a segment encoderEϕE\_\{\\phi\}, a bidirectional latent Transformerfθf\_\{\\theta\}, and an autoregressive segment decoderDψD\_\{\\psi\}\. The segment encoderEϕE\_\{\\phi\}is a bidirectional Transformer followed by mean pooling over non\-<EOS\>tokens, without an additional projection layer\. Unlike LaDiR\[Kanget al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib84)\]and LD4PG\[Lovelaceet al\.,[2023](https://arxiv.org/html/2607.26504#bib.bib89)\], which use learnable query embeddings to compress variable\-length sequences, we use mean pooling for simplicity\. The LatentMDMfθf\_\{\\theta\}takes the encoded segment sequence as input, replaces masked segments with a learned mask embeddingem∈ℝHe\_\{m\}\\in\\mathbb\{R\}^\{H\}, and uses in\-context conditioning by concatenating prompt token embeddings as a prefix to the segment\-latent sequence\. The autoregressive decoderDψD\_\{\\psi\}is a causal Transformer that reconstructs each segment autoregressively, using in\-context conditioning on the corresponding LatentMDM output𝐡θ,ϕi​\(𝐳\)∈ℝH′\\mathbf\{h\}\_\{\\theta,\\phi\}^\{i\}\(\{\\mathbf\{z\}\}\)\\in\\mathbb\{R\}^\{H^\{\\prime\}\}, which is prepended as a one\-token prefix\. We setH′=H=512H^\{\\prime\}=H=512for all three modules, so no projection layer is required between the encoder, LatentMDM, and decoder\.

For LatentMDM and all baselines, we tie the token embedding matrix to the output prediction head\. The same embedding matrix is shared across the segment encoder input embedding, the LatentMDM prompt embedding, the decoder input embedding, and the decoder output head\. Since the Qwen tokenizer has a vocabulary size of 151,645, embedding tying substantially reduces the parameter count and makes the comparison across models more controlled\.

Baselines\.The autoregressive baseline is a 14\-layer Qwen2\-style causal Transformer, and the token\-level MDM baseline is a 14\-layer Qwen2\-style bidirectional Transformer\. Both baselines use the same tokenizer, hidden dimension, RoPE positional encoding, and tied input/output embeddings as the LatentMDM\.

### E\.3Training Details

Figure[10](https://arxiv.org/html/2607.26504#A5.F10)illustrates the resulting training pipeline\. For each training example, the target solution is split into variable\-length code segments, and a random subset of segment positions is masked\. Unmasked segments are passed through the segment encoderEϕE\_\{\\phi\}, whereas masked positions are represented by the learned mask embedding𝐞𝐦\\mathbf\{e\}\_\{\\mathbf\{m\}\}\. The LatentMDMfθf\_\{\\theta\}then performs bidirectional contextual prediction over the prompt\-conditioned segment\-latent sequence, producing one latent conditioning vector for each masked segment\. Finally, the autoregressive decoderDψD\_\{\\psi\}reconstructs each masked segment from its predicted latent representation using teacher forcing\. Thus, the model learns latent\-segment\-level masked prediction while retaining token\-level autoregressive modeling within each segment\.

In the training setup, we never mask the prompt𝐏\\mathbf\{P\}for either LatentMDM or the token\-level MDM baseline\. For LatentMDM, masking is applied only at the segment level over the target solution\. Namely, we samplen∼Uniform​\{1,…,Ls\}n\\sim\\mathrm\{Uniform\}\\\{1,\\ldots,L\_\{s\}\\\}and uniformly choosennsegment positions to mask among theLsL\_\{s\}target segments\. If a segment is selected for masking, it is not passed through the segment encoderEϕE\_\{\\phi\}; instead, its latent representation is directly replaced by the learned mask embedding𝐞𝐦\\mathbf\{e\}\_\{\\mathbf\{m\}\}\. Unmasked segments are encoded byEϕE\_\{\\phi\}and provided to the LatentMDM together with the prompt prefix\. Training LatentMDM took approximately three days on 4 80GB NVIDIA A100 GPUs\. We defer the remaining optimization hyperparameters and training configurations to Table[8](https://arxiv.org/html/2607.26504#A5.T8)\.

For the token\-level MDM baseline, masking is applied only to ground\-truth response tokens, including<EOS\>tokens, while prompt tokens remain unmasked\. Concretely, we sampler∼Uniform​\(0,1\)r\\sim\\mathrm\{Uniform\}\(0,1\)and mask⌈r⋅\(512−len⁡\(𝐏\)\)⌉\\lceil r\\cdot\(512\-\\operatorname\{len\}\(\\mathbf\{P\}\)\)\\rceiltokens uniformly among the response\-token positions\. This matches the LatentMDM setup in that the conditional prompt is always visible, and the model is trained to reconstruct only the target solution\.

### E\.4Inference Details

Subroutine 1:LatentMDM parallel decoding1:Require:LatentMDM

fθf\_\{\\theta\}, segment encoder

EϕE\_\{\\phi\}, autoregressive decoder

DψD\_\{\\psi\}, prompt

𝐏\\mathbf\{P\}, number of segments

LsL\_\{s\}, reveal size

kk
2:Initialize latent sequence

𝐳←\(𝐞𝐦,…,𝐞𝐦\)\{\\mathbf\{z\}\}\\leftarrow\(\\mathbf\{e\}\_\{\\mathbf\{m\}\},\\ldots,\\mathbf\{e\}\_\{\\mathbf\{m\}\}\)
3:Initialize decoded segments

𝐲^←\(∅,…,∅\)\\hat\{\{\\mathbf\{y\}\}\}\\leftarrow\(\\varnothing,\\ldots,\\varnothing\)
4:Set

Ns←Ls/kN\_\{s\}\\leftarrow L\_\{s\}/k
5:for

n=1n=1to

NsN\_\{s\}do

6:

ℳ←\{i∣𝐳i=𝐞𝐦\}\\mathcal\{M\}\\leftarrow\\\{i\\mid\{\\mathbf\{z\}\}^\{i\}=\\mathbf\{e\}\_\{\\mathbf\{m\}\}\\\}
7:

𝐡←fθ​\(𝐳;𝐏\)\\mathbf\{h\}\\leftarrow\{\\color\[rgb\]\{0\.25390625,0\.41015625,0\.8828125\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.25390625,0\.41015625,0\.8828125\}f\_\{\\theta\}\(\{\\mathbf\{z\}\};\\mathbf\{P\}\)\}
8:for

i∈ℳi\\in\\mathcal\{M\}do⊳\\trianglerightParallel over masked segments

9:\# Sample tokens autoregressively

10:\# and compute its score

11:

𝐲^i,si←SampleSegment​\(Dψ,𝐡i\)\\hat\{\{\\mathbf\{y\}\}\}^\{i\},s^\{i\}\\leftarrow\\textsc\{SampleSegment\}\(\{\\color\[rgb\]\{1,0,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{1,0,0\}D\_\{\\psi\}\},\\mathbf\{h\}^\{i\}\)
12:endfor

13:Sample top\-

kkindices

𝒮⊆ℳ\\mathcal\{S\}\\subseteq\\mathcal\{M\}according to

sis^\{i\}
14:for

i∈𝒮i\\in\\mathcal\{S\}do⊳\\trianglerightParallel over selected segments

15:

𝐳i←Eϕ​\(𝐲^i\)\{\\mathbf\{z\}\}^\{i\}\\leftarrow\{\\color\[rgb\]\{0,0\.625,0\.52734375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0\.625,0\.52734375\}E\_\{\\phi\}\(\\hat\{\{\\mathbf\{y\}\}\}^\{i\}\)\}
16:endfor

17:endfor

18:return

𝐲^=\(𝐲^1,…,𝐲^Ls\)\\hat\{\{\\mathbf\{y\}\}\}=\(\\hat\{\{\\mathbf\{y\}\}\}^\{1\},\\ldots,\\hat\{\{\\mathbf\{y\}\}\}^\{L\_\{s\}\}\)

Subroutine 2:SampleSegment1:functionSampleSegment\(

Dψ,𝐡iD\_\{\\psi\},\\mathbf\{h\}^\{i\}\)

2:Initialize:

𝐲^0i←∅\\hat\{\{\\mathbf\{y\}\}\}^\{i\}\_\{0\}\\leftarrow\\varnothing,

si←0s^\{i\}\\leftarrow 0,

l=0l=0
3:for

t=1t=1to

LsegL\_\{\\mathrm\{seg\}\}do

4:

l←l\+1l\\leftarrow l\+1
5:

𝐩t←Dψ\(⋅∣𝐲^<ti,𝐡i\)\\mathbf\{p\}\_\{t\}\\leftarrow\{\\color\[rgb\]\{1,0,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{1,0,0\}D\_\{\\psi\}\(\\cdot\\mid\\hat\{\{\\mathbf\{y\}\}\}^\{i\}\_\{<t\},\\mathbf\{h\}^\{i\}\)\}
6:Sample

𝐲^ti∼Cat​\(𝐩t\)\\hat\{\{\\mathbf\{y\}\}\}^\{i\}\_\{t\}\\sim\\mathrm\{Cat\}\(\\mathbf\{p\}\_\{t\}\)
7:

si←si\+log⁡𝐩t​\[𝐲^ti\]s^\{i\}\\leftarrow s^\{i\}\+\\log\\mathbf\{p\}\_\{t\}\[\\hat\{\{\\mathbf\{y\}\}\}^\{i\}\_\{t\}\]
8:if

𝐲^ti=<EOS\>\\hat\{\{\\mathbf\{y\}\}\}^\{i\}\_\{t\}=\\texttt\{<EOS\>\}then

9:break

10:endif

11:endfor

12:return

𝐲^i,si/l\\hat\{\{\\mathbf\{y\}\}\}^\{i\},s^\{i\}/l
13:endfunction

Figure 11:LatentMDM inference\.Subroutine 1 performs parallel segment\-level decoding\.We further describe the LatentMDM inference procedure in a more general setting\. In Section[4\.2](https://arxiv.org/html/2607.26504#S4.SS2), we present the case where segments are revealed one at a time\. More generally, the same procedure can reveal multiple masked segments per iteration, analogous to multi\-token decoding in token\-level MDMs\. Decoding more than one segment per step can improve sampling speed, but introduces the usual speed–quality trade\-off in diffusion language model sampling\[Zhenget al\.,[2024](https://arxiv.org/html/2607.26504#bib.bib10), Kimet al\.,[2025c](https://arxiv.org/html/2607.26504#bib.bib15), Penget al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib6), Ben\-Hamuet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib21), Nieet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib12), Wuet al\.,[2025b](https://arxiv.org/html/2607.26504#bib.bib29), Hayakawaet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib31)\]\.

Algorithm[11](https://arxiv.org/html/2607.26504#A5.F11)gives the resulting inference procedure withkk\-segment parallel decoding\. For simplicity, we assume thatkkdividesLsL\_\{s\}and defineNs=Ls/kN\_\{s\}=L\_\{s\}/k\. At each iteration, the LatentMDM predicts contextual latent representations for all currently masked segment positions\. The autoregressive decoder then tentatively decodes a candidate segment for each masked position, in parallel across segment positions\. We score each candidate by its length\-normalized token log\-likelihood:

si≔1len​\(𝐲^i\)​∑j=1len​\(𝐲^i\)log⁡Dψ​\(𝐲^ji∣𝐲^<ji,𝐡θ,ϕi​\(𝐳\)\)\.s^\{i\}\\coloneqq\\frac\{1\}\{\\mathrm\{len\}\(\\hat\{\{\\mathbf\{y\}\}\}^\{i\}\)\}\\sum\_\{j=1\}^\{\\mathrm\{len\}\(\\hat\{\{\\mathbf\{y\}\}\}^\{i\}\)\}\\log D\_\{\\psi\}\\\!\\left\(\\hat\{\{\\mathbf\{y\}\}\}^\{i\}\_\{j\}\\mid\\hat\{\{\\mathbf\{y\}\}\}^\{i\}\_\{<j\},\\mathbf\{h\}\_\{\\theta,\\phi\}^\{i\}\(\{\\mathbf\{z\}\}\)\\right\)\.We then commit the top\-kkmasked segments according to this score, re\-encode the committed segments withEϕE\_\{\\phi\}, and update the corresponding positions in the latent sequence\. We use the length\-normalized token log\-likelihood as the default segment\-selection score, and ablate both alternative scoring rules andkkin Appendix[E\.5](https://arxiv.org/html/2607.26504#A5.SS5)\. KV caching is enabled for the autoregressive decoder and for the autoregressive baseline during inference\. We useLs=8L\_\{s\}=8segment positions at inference time, while keeping the remaining settings consistent with training\. For wall\-clock measurements, we use a single 80GB NVIDIA A100 GPU and a sampling batch size 1\.

### E\.5Additional Experiments

Table 7:Zero\-shot GSM8K performance of LatentMDM under different segment\-selection policies and parallel decoding sizes \(kk\)\. Token sampling within each segment is performed greedily\(T=0\)\(T=0\)\.Ablation on Segment Scoring\.We evaluate LatentMDM under several segment\-selection policies in a zero\-shot setting using greedy token sampling\. Avg\. log\-likelihood is our default scoring rule, which ranks each candidate segment by its length\-normalized token log\-likelihood\. First log\-likelihood scores a segment using only the log\-likelihood of its first generated token, following the primary slot\-scoring strategy used in ReFusion\[Liet al\.,[2025](https://arxiv.org/html/2607.26504#bib.bib107)\]\. Min\. log\-likelihood assigns each segment the minimum token log\-likelihood within the generated segment, thereby emphasizing the least confident local decision\. L2R reveals segments strictly from left to right, removing the any\-order segment\-selection mechanism, while Random selects masked segments uniformly at random\. Table[7](https://arxiv.org/html/2607.26504#A5.T7)shows that average log\-likelihood performs best, suggesting that reliable segment\-level confidence requires aggregating information across the whole generated segment\.

Parallel Decoding\.We further evaluate LatentMDM under parallel segment decoding, wherek∈\{1,2,4\}k\\in\\\{1,2,4\\\}denotes the number of masked segments committed at each decoding iteration\. As shown in Table[7](https://arxiv.org/html/2607.26504#A5.T7), the average log\-likelihood score achieves the best performance across all values ofkk\. In contrast, the alternative scoring variants degrade more sharply askkincreases, and often fall behind random segment selection in the more parallel regimes\. This suggests that effective parallel segment decoding requires a robust segment\-level confidence estimate; otherwise, committing multiple segments per iteration can amplify early selection errors\.

Table 8:Training hyperparameters and architecture details for LatentMDM on TinyGSM\. The reported LatentMDM model size separates the Transformer parameters from the tied token\-embedding matrix; the additional 77\.6M parameters correspond to this shared embedding matrix\.
### E\.6Generation Trace

![Refer to caption](https://arxiv.org/html/2607.26504v1/figures/LatentMDM_example.png)Figure 12:Generation trace\.At each iteration, the model tentatively decodes candidate segments for all masked positions, scores them using the segment\-selection criterion, and commits the highest\-scoring segment\. The highlighted row indicates the segment committed at the current iteration, while previously revealed segments are shown in gray\. The row color reflects the segment score, with stronger colors corresponding to higher\-scoring candidates\. This example illustrates that LatentMDM generates at the level of semantic code segments rather than individual token positions, enabling a non\-left\-to\-right segment reveal order\.We further provide a generation trace of LatentMDM in Fig\.[12](https://arxiv.org/html/2607.26504#A5.F12)\.