Representation-based Masked Diffusion Model

# Representation-based Masked Diffusion Model
Source: [https://arxiv.org/html/2609.12382](https://arxiv.org/html/2609.12382)
Yangrong HuAffiliation:Department of Data Science and Artificial IntelligenceEmail:[yangrong\.hu@connect\.polyu\.hk](mailto:[email protected])Ding HuangAffiliation:Department of Applied MathematicsThe Hong Kong Polytechnic University, Hong Kong SAR, ChinaEmail:[ding\.huang@connect\.polyu\.hk](mailto:[email protected])Xueyu ZhouAffiliation:Department of Data Science and Artificial IntelligenceEmail:[xueyu\.zhou@connect\.polyu\.hk](mailto:[email protected])Jian HuangAffiliation:Department of Data Science and Artificial IntelligenceAffiliation:Department of Applied MathematicsThe Hong Kong Polytechnic University, Hong Kong SAR, ChinaEmail:[j\.huang@polyu\.edu\.hk](mailto:[email protected])

###### Abstract

Masked Diffusion Models \(MDMs\) have emerged as a compelling paradigm for language modeling, offering the capability for efficient parallel text generation\. However, existing parallel sampling methods typically update multiple masked tokens independently and ignore the complex mutual dependencies among the masked tokens\. This independent updating mechanism lacks global coordination and might lead to incoherent outputs\. To address this limitation, we propose Representation\-based Masked Diffusion Model \(RMDM\), a framework that leverages the text representation to explicitly encode global semantics and help to parallel update tokens more precisely\. Specifically, we first encode text into a continuous semantic space using a pretrained encoder and learn an invertible transformation that normalizes the representation distribution to a Gaussian prior, facilitating efficient sampling during generation\. Conditioned on this latent semantic representation, we train a masked diffusion model to learn the conditional text distribution, where the representation serves as global semantic guidance to coordinate parallel token updates and faithfully approximate the target distribution\. Empirical results demonstrate that RMDM significantly improves generation quality, particularly in aggressive few\-step sampling regimes\.

††footnotetext:Corresponding author: Jian Huang\.‡‡footnotetext:Present address: ByteDance Seed, Beijing, China\. Email:huanhgding@bytedance\.com\.Stage I: Representation MappingStage II: Generative ModelingEncodergψg\_\{\\psi\}𝒛1\\bm\{z\}\_\{1\}MeanFlowuϕu\_\{\\phi\}𝒛0\\bm\{z\}\_\{0\}∼𝒩⁡\(𝟎,𝑰\)\\sim\\mathcal\{N\}\(\\bm\{0\},\\bm\{I\}\)𝒙0\\bm\{x\}\_\{0\}Mask𝒙t\\bm\{x\}\_\{t\}RMDMpθ​\(𝒙0\|𝒙t,𝒛0\)p\_\{\\theta\}\(\\bm\{x\}\_\{0\}\|\\bm\{x\}\_\{t\},\\bm\{z\}\_\{0\}\)𝒙^0\\hat\{\\bm\{x\}\}\_\{0\}

Figure 1:Stage I aligns the latent space𝒛0\\bm\{z\}\_\{0\}, derived from a pretrained encodergψ​\(𝒙0\)g\_\{\\psi\}\(\\bm\{x\}\_\{0\}\), with a Gaussian prior via MeanFlowuϕu\_\{\\phi\}\. Stage II trains the RMDMpθp\_\{\\theta\}to reconstruct original data𝒙0\\bm\{x\}\_\{0\}from corrupted data𝒙t\\bm\{x\}\_\{t\}guided by the latent representation𝒛0\\bm\{z\}\_\{0\}\.## 1Introduction

Large Language Models \(LLMs\) have achieved huge success for modeling text data\([Brown et al\., 2020](https://arxiv.org/html/2609.12382#bib.bib1);[Radford et al\., 2019](https://arxiv.org/html/2609.12382#bib.bib35);[Achiam et al\., 2023](https://arxiv.org/html/2609.12382#bib.bib24);[Liu et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib25)\), primarily through the autoregressive \(AR\) paradigm\. While AR models excel at capturing complex dependencies by generating tokens sequentially, their inference speed is fundamentally constrained by theirO⁡\(L\)O\(L\)sequential complexity for a sequence of lengthLL\. This latency bottleneck has spurred significant interest in non\-autoregressive or parallel generation frameworks\([Gu et al\., 2018](https://arxiv.org/html/2609.12382#bib.bib45)\)\. Masked Diffusion Models \(MDMs\)\([Austin et al\., 2021](https://arxiv.org/html/2609.12382#bib.bib5);[Sahoo et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib13);[Lou et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib40);[Ou et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib39)\)have recently emerged as a compelling alternative that enables parallel token generation via iterative denoising\. Compared to continuous diffusion models, MDMs utilize a*discrete diffusion*process, where the forward process progressively adds noise to a clean sequence by replacing its subset with special\[MASK\]tokens instead of gaussian noise and yields a partially observed context\. A denoiser is then trained to invert this corruption by predicting the original tokens from the context\. In the backward process, MDMs utilize the denoiser to generate the whole sequence by repeatedly refining a sentence that consists entirely of masked tokens\.

In practical deployment, MDMs are required to efficiently update multiple masked tokens in a single iteration\. Most parallel methods\([Nie et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib11);[Zhu et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib10);[Ye et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib12);[Chang et al\., 2022](https://arxiv.org/html/2609.12382#bib.bib49);[Wu et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib17)\)sample each masked position independently conditioned on the currently observed context, while ignoring the dependency between masked tokens\. Such independent sampling approximates the joint distribution over the masked positions with a product of token\-wise marginal distributions\. However, the factorized approximation is generally misaligned with natural language, where masked tokens often exhibit strong residual dependencies induced by syntactic agreement, long\-range constraints, multi\-word expressions, and entity consistency\. As a result, aggressive parallelization can produce sequences that are locally plausible at each position yet globally inconsistent under the true joint distribution\. We formalize and quantify thisConditional Dependency Gapin Section[3\.1](https://arxiv.org/html/2609.12382#S3.SS1), and build our method around mitigating it without sacrificing parallelism and efficiency\.

##### Our approach\.

To mitigate this inconsistency, we introduce a global latent representationzzthat encodes the semantic information of the whole sequence\. This representation captures shared global factors, such as topic and intent, which largely explain the remaining dependence among masked tokens\. Therefore, conditioning onzzreduces residual cross\-token dependence and makes parallel prediction closer to the true joint conditional distribution\. We formalize this intuition as a reduction of the*Conditional Dependency Gap*in Section[3\.1](https://arxiv.org/html/2609.12382#S3.SS1)\. To make the latent variablezzoperational, we use pretrained encoders, such as BERT models\([Devlin et al\., 2019](https://arxiv.org/html/2609.12382#bib.bib36);[Warner et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib51)\)and modern large\-scale embedding models\([Zhang et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib57)\), to extract contextual representations of the input text\. The representation encodes global attributes of the underlying sequence and captures high\-level semantic factors that are difficult to infer from a partially revealed context\. By providing this missing global information, the representation serves as a natural candidate forzzand helps reduce the Conditional Dependency Gap during parallel updates\. A key complication is that these representations are readily available from the ground\-truth text during training but are unobserved at inference time\. Directly conditioning on such embeddings would therefore introduce a train–test mismatch unlesszzcan be sampled from a well\-defined prior during generation\. To resolve the training–inference mismatch, we employ MeanFlow\([Geng et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib3)\)to map the empirical distribution of pretrained sequence representations to a Gaussian prior\. This transport allows us to directly sample a global latentz∼𝒩⁡\(0,I\)z\\sim\\mathcal\{N\}\(0,I\)at generation time, which serves as a global condition for the RMDM to ensure coherent parallel updates\. Although variational inference\([Kingma and Welling, 2014](https://arxiv.org/html/2609.12382#bib.bib38);[Xie et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib6)\)is a classical approach to Gaussian latents, its reconstruction and KL objectives often conflict in high\-capacity language models, leading to unstable optimization or posterior collapse\. Approaches such as CCDD\([Zhou et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib7)\)co\-evolve continuous and discrete diffusions, whereas RMDM samples a fixed latent once and reuses it throughout discrete denoising\. The pretrained encoder and MeanFlow network are used to construct training latents but are not invoked during generation\. Experimental results demonstrate that RMDM consistently outperforms standard MDMs in sample quality for a fixed number of sampling steps and achieves approximately3\.6×3\.6\\timeshigher measured throughput at matched generation quality in our A800 evaluation\. We provide a more detailed discussion of related discrete diffusion language models, continuous diffusion language models, and latent\-augmented masked diffusion methods in Appendix[A](https://arxiv.org/html/2609.12382#A1)\.

## 2Preliminaries

### 2\.1Continuous Diffusion and MeanFlow

Continuous diffusion models\([Ho et al\., 2020](https://arxiv.org/html/2609.12382#bib.bib4);[Song et al\., 2021](https://arxiv.org/html/2609.12382#bib.bib2)\)are built on the principle of a forward process that gradually corrupts data𝒛0∈ℝd\\bm\{z\}\_\{0\}\\in\\mathbb\{R\}^\{d\}into Gaussian noise𝒛1∼𝒩⁡\(𝟎,𝑰\)\\bm\{z\}\_\{1\}\\sim\\mathcal\{N\}\(\\bm\{0\},\\bm\{I\}\)over a continuous timet∈\[0,1\]t\\in\[0,1\]\. The generative process is then defined by learning to reverse this corruption—essentially "denoising" the latent—by following a velocity field that guides the noise back to the data manifold\. Flow matching\([Lipman et al\., 2023](https://arxiv.org/html/2609.12382#bib.bib8)\)simplifies this by training a model𝒗θ\\bm\{v\}\_\{\\theta\}to match the conditional velocity field:

ℒCFM​\(θ\)=𝔼t,𝒛0,𝒛1​\[‖𝒗θ​\(𝒛t,t\)−\(𝒛1−𝒛0\)‖2\],\\displaystyle\\mathcal\{L\}\_\{\\text\{CFM\}\}\(\\theta\)=\\mathbb\{E\}\_\{t,\\bm\{z\}\_\{0\},\\bm\{z\}\_\{1\}\}\\left\[\\\|\\bm\{v\}\_\{\\theta\}\(\\bm\{z\}\_\{t\},t\)\-\(\\bm\{z\}\_\{1\}\-\\bm\{z\}\_\{0\}\)\\\|^\{2\}\\right\],where𝒛t=\(1−t\)​𝒛0\+t​𝒛1\\bm\{z\}\_\{t\}=\(1\-t\)\\bm\{z\}\_\{0\}\+t\\bm\{z\}\_\{1\}represents the linear interpolation between data and noise\. While standard flow matching requires numerical ODE integration over multiple steps, MeanFlow\([Geng et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib3)\)enables efficient one\-step generation by modeling theaverage velocityu⁡\(𝒛t,r,t\)≜1t−r​∫rt𝒗⁡\(𝒛τ,τ\)​𝑑τu\(\\bm\{z\}\_\{t\},r,t\)\\triangleq\\frac\{1\}\{t\-r\}\\int\_\{r\}^\{t\}\\bm\{v\}\(\\bm\{z\}\_\{\\tau\},\\tau\)d\\tau\. The modeluθu\_\{\\theta\}is trained via the MeanFlow identity:

ℒMF​\(θ\)=𝔼​‖uθ​\(𝒛t,r,t\)−sg​\[𝒗t−\(t−r\)​dd​t​uθ​\(𝒛t,r,t\)\]‖2\\mathcal\{L\}\_\{\\text\{MF\}\}\(\\theta\)=\\mathbb\{E\}\\left\\\|u\_\{\\theta\}\(\\bm\{z\}\_\{t\},r,t\)\-\\text\{sg\}\\left\[\\bm\{v\}\_\{t\}\-\(t\-r\)\\frac\{d\}\{dt\}u\_\{\\theta\}\(\\bm\{z\}\_\{t\},r,t\)\\right\]\\right\\\|^\{2\}

Wheres​gsgis the stop gradient operator\. Once optimized,𝒛0\\bm\{z\}\_\{0\}can be recovered from noise𝒛1\\bm\{z\}\_\{1\}in a single step as𝒛0=𝒛1−uθ​\(𝒛1,0,1\)\\bm\{z\}\_\{0\}=\\bm\{z\}\_\{1\}\-u\_\{\\theta\}\(\\bm\{z\}\_\{1\},0,1\)\.

### 2\.2Masked Diffusion Models

While continuous diffusion is well\-suited for continuous distributions, language modeling requires a principled approach to handle discrete categorical variables\. Masked diffusion models \(MDMs\)\([Austin et al\., 2021](https://arxiv.org/html/2609.12382#bib.bib5);[Lou et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib40)\)extend the diffusion framework to discrete state spaces by defining a corruption process that gradually replaces tokens𝒙0\\bm\{x\}\_\{0\}with an absorbing\[MASK\]while a sequence with full\[MASK\]is denoted as state𝒎\\bm\{m\}\.

Similar to the continuous case, the forward processq⁡\(𝒙t\|𝒙0\)=Cat​\(𝒙t,αt​𝒙0\+\(1−αt\)​𝒎\)q\(\\bm\{x\}\_\{t\}\|\\bm\{x\}\_\{0\}\)=\\text\{Cat\}\(\\bm\{x\}\_\{t\};\\alpha\_\{t\}\\bm\{x\}\_\{0\}\+\(1\-\\alpha\_\{t\}\)\\bm\{m\}\)interpolates between clean data and the masked prior, whereαt∈\[0,1\]\\alpha\_\{t\}\\in\[0,1\]is a decreasing noise schedule\. The generative modelpθ​\(𝒙s\|𝒙t\)p\_\{\\theta\}\(\\bm\{x\}\_\{s\}\|\\bm\{x\}\_\{t\}\)learns to invert this masking process by approximating the closed\-form posterior:

q⁡\(𝒙s\|𝒙t,𝒙0\)=\{δ𝒙tif​𝒙t≠𝒎,Cat​\(𝒙s,αs−αt1−αt​𝒙0\+1−αs1−αt​𝒎\)if​𝒙t=𝒎q\(\\bm\{x\}\_\{s\}\|\\bm\{x\}\_\{t\},\\bm\{x\}\_\{0\}\)=\\begin\{cases\}\\delta\_\{\\bm\{x\}\_\{t\}\}&\\text\{if \}\\bm\{x\}\_\{t\}\\neq\\bm\{m\},\\\\ \\text\{Cat\}\\left\(\\bm\{x\}\_\{s\};\\frac\{\\alpha\_\{s\}\-\\alpha\_\{t\}\}\{1\-\\alpha\_\{t\}\}\\bm\{x\}\_\{0\}\+\\frac\{1\-\\alpha\_\{s\}\}\{1\-\\alpha\_\{t\}\}\\bm\{m\}\\right\)&\\text\{if \}\\bm\{x\}\_\{t\}=\\bm\{m\}\\end\{cases\}

By training a denoising modelpθ​\(𝒙0\|𝒙t\)p\_\{\\theta\}\(\\bm\{x\}\_\{0\}\|\\bm\{x\}\_\{t\}\), MDMs can be optimized via an objective derived from the data log\-likelihoodlog⁡pθ​\(𝒙0\)\\log p\_\{\\theta\}\(\\bm\{x\}\_\{0\}\):

ℒ⁡\(θ\)\\displaystyle\\mathcal\{L\}\(\\theta\)=∫011t𝔼q⁡\(𝒙t\|𝒙0\)\[∑i∈ℐt−logpθ\(𝒙0i\|𝒙t\)\]dt\\displaystyle=\\int\_\{0\}^\{1\}\\frac\{1\}\{t\}\\mathbb\{E\}\_\{q\(\\bm\{x\}\_\{t\}\|\\bm\{x\}\_\{0\}\)\}\\left\[\\sum\_\{i\\in\\mathcal\{I\}\_\{t\}\}\-\\log p\_\{\\theta\}\(\\bm\{x\}\_\{0\}^\{i\}\|\\bm\{x\}\_\{t\}\)\\right\]dt\(1\)whereℐt=\{i:𝒙ti=\[MASK\]\}\\mathcal\{I\}\_\{t\}=\\\{i:\\bm\{x\}\_\{t\}^\{i\}=\\texttt\{\[MASK\]\}\\\}\. This loss allows the model to predict the categorical distribution of original tokens at each masked position, providing a discrete analogue to the continuous denoising flow described above\.

## 3Methodology

### 3\.1The Conditional Dependency Gap

In this section, we denote𝒙M\\bm\{x\}\_\{M\}as the masked data, and𝒙U\\bm\{x\}\_\{U\}as the unmasked data in timettof the diffusion process\. A fundamental challenge in Masked Diffusion Models \(MDMs\) arises from the discrepancy between their training objective and the parallel sampling strategy required for efficiency\. While the standard objective Equation[1](https://arxiv.org/html/2609.12382#S2.E1)optimizes the reconstruction of individual tokenspθ​\(xi∣𝒙U\)p\_\{\\theta\}\(x\_\{i\}\\mid\\bm\{x\}\_\{U\}\)independently, fast generation typically necessitates sampling all masked tokens𝒙M\\bm\{x\}\_\{M\}simultaneously\. Existing parallel samplers implicitly approximate the joint posterior via factorization:

pθ​\(𝒙M∣𝒙U\)≜∏i∈Mpθ​\(xi∣𝒙U\)\\displaystyle p\_\{\\theta\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\}\)\\triangleq\\prod\_\{i\\in M\}p\_\{\\theta\}\(x\_\{i\}\\mid\\bm\{x\}\_\{U\}\)≈\\displaystyle\\approx∏i∈Mpdata​\(xi∣𝒙U\)≠pdata​\(𝒙M∣𝒙U\),\\displaystyle\\prod\_\{i\\in M\}p\_\{\\text\{data\}\}\(x\_\{i\}\\mid\\bm\{x\}\_\{U\}\)\\neq p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\}\),whereMMandUUdenote the sets of masked and unmasked indices, respectively\. This factorization assumes that masked tokens are conditionally independent given the context𝒙U\\bm\{x\}\_\{U\}\. However, for structured data, this assumption is often violated, leading to aConditional Dependency Gap\. We quantify this gap using the conditional total correlation

𝒛\\bm\{z\}𝒙U\\bm\{x\}\_\{U\}xM1x\_\{M\}^\{1\}xM2x\_\{M\}^\{2\}

Figure 2:Causal graph\.The global latent𝒛\\bm\{z\}acts as a common cause for masked tokens\. Conditioning on𝒛\\bm\{z\}explains away the correlation betweenxM1x\_\{M\}^\{1\}andxM2x\_\{M\}^\{2\}\.𝒯⁡\(𝒙M∣𝒙U\)≜DKL​\(pdata​\(𝒙M∣𝒙U\)∥∏i∈Mpdata​\(xi∣𝒙U\)\)\\mathcal\{T\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\}\)\\triangleq D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\}\)\\,\\middle\\\|\\,\\prod\_\{i\\in M\}p\_\{\\text\{data\}\}\(x\_\{i\}\\mid\\bm\{x\}\_\{U\}\)\\right\)

which measures the intrinsic dependence among masked tokens that is not explained by the observed context\.

To bridge this gap, we introduce a latent variable𝒛\\bm\{z\}that encodes global information about𝒙\\bm\{x\}, thereby capturing shared factors that induce dependencies among the masked tokens\.

𝒯⁡\(𝒙M∣𝒙U,𝒛\)=DKL​\(pdata​\(𝒙M∣𝒙U,𝒛\)∥∏i∈Mpdata​\(𝒙i∣𝒙U,𝒛\)\)\\mathcal\{T\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)=D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\\,\\middle\\\|\\,\\prod\_\{i\\in M\}p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{i\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\\right\)

Conditioning on𝒛\\bm\{z\}is able to reduce the independence gap, as formalized below\.

###### Proposition 3\.1\(Relation between Conditional Independence and Conditional Dependency Gap\)\.

The residual dependency gap𝒯⁡\(𝐱M∣𝐱U,𝐳\)\\mathcal\{T\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)vanishes if and only if the masked tokens𝐱M\\bm\{x\}\_\{M\}are conditionally independent given the context𝐱U\\bm\{x\}\_\{U\}and the latent variable𝐳\\bm\{z\}:

𝒯⁡\(𝒙M∣𝒙U,𝒛\)=0\\displaystyle\\mathcal\{T\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)=0⇔\\displaystyle\\iffxm1⟂⟂xm2⟂⟂⋯⟂⟂xm\|M\|\|\(𝒙U,𝒛\)\\displaystyle x\_\{m\_\{1\}\}\\perp\\\!\\\!\\\!\\perp x\_\{m\_\{2\}\}\\perp\\\!\\\!\\\!\\perp\\dots\\perp\\\!\\\!\\\!\\perp x\_\{m\_\{\|M\|\}\}\\mid\(\\bm\{x\}\_\{U\},\\bm\{z\}\)Furthermore, the introduction of𝐳\\bm\{z\}changes the dependency gap by:

Δ​𝒯\\displaystyle\\Delta\\mathcal\{T\}≜𝒯⁡\(𝒙M∣𝒙U\)−𝒯⁡\(𝒙M∣𝒙U,𝒛\)\\displaystyle\\triangleq\\mathcal\{T\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\}\)\-\\mathcal\{T\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)=∑i∈MI⁡\(xi;𝒛∣𝒙U\)−I⁡\(𝒙M;𝒛∣𝒙U\)\\displaystyle=\\sum\_\{i\\in M\}I\(x\_\{i\};\\bm\{z\}\\mid\\bm\{x\}\_\{U\}\)\-I\(\\bm\{x\}\_\{M\};\\bm\{z\}\\mid\\bm\{x\}\_\{U\}\)

We refer𝒯⁡\(𝒙M∣𝒙U,𝒛\)\\mathcal\{T\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)as the residual gap\. When𝒯⁡\(𝒙M∣𝒙U,𝒛\)=0\\mathcal\{T\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)=0, the conditional joint distribution factorizes, so sampling eachxi∼p⁡\(xi∣𝒙U,𝒛\)x\_\{i\}\\sim p\(x\_\{i\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)independently is equivalent to sampling the entire block𝒙M∼p⁡\(𝒙M∣𝒙U,𝒛\)\\bm\{x\}\_\{M\}\\sim p\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\), enabling exact parallel sampling\. The proof is provided in Appendix[B\.1](https://arxiv.org/html/2609.12382#A2.SS1)\. Figure[2](https://arxiv.org/html/2609.12382#S3.F2.fig1)shows an example of the casual graph forxMx\_\{M\},𝒙U\\bm\{x\}\_\{U\}and𝒛\\bm\{z\}\. More detailed numerical calculation examples are shown in Appendix[G\.1](https://arxiv.org/html/2609.12382#A7.SS1)\.

### 3\.2Representation\-based Masked Diffusion Model

![Refer to caption](https://arxiv.org/html/2609.12382v1/figs/e_umap_3d.png)\(a\)Contextualized embeddings𝒛1\\bm\{z\}\_\{1\}
![Refer to caption](https://arxiv.org/html/2609.12382v1/figs/z_umap_3d.png)\(b\)Gaussian representations𝒛0\\bm\{z\}\_\{0\}

Figure 3:MeanFlow\-based Representation Alignment\.We visualize the 3D UMAP projections of the contextualized embeddings𝒆\\bm\{e\}from the pretrained encoder, which exhibit complex structured manifolds, and the aligned representations𝒛0\\bm\{z\}\_\{0\}obtained via MeanFlow, which follow a standard Gaussian distribution while preserving semantic relations\.Algorithm 1Two\-stage training of RMDM0:Dataset

𝒟\\mathcal\{D\}, pretrained encoder

gψg\_\{\\psi\}, RMDM

pθp\_\{\\theta\}, MeanFlow network

uϕu\_\{\\phi\}
1:Stage I: Train MeanFlow in representation space\.

2:foreach minibatch

𝒙0∼𝒟\\bm\{x\}\_\{0\}\\sim\\mathcal\{D\}do

3:

𝒛1←gψ​\(𝒙0\)\\bm\{z\}\_\{1\}\\leftarrow g\_\{\\psi\}\(\\bm\{x\}\_\{0\}\)\{contextualized embeddings as target\}

4:Sample

𝒛0∼𝒩⁡\(𝟎,𝑰\)\\bm\{z\}\_\{0\}\\sim\\mathcal\{N\}\(\\bm\{0\},\\bm\{I\}\)and

\(r,t\)\(r,t\)\{noise as prior\}

5:

𝒛t←\(1−t\)​𝒛0\+t​𝒛1\\bm\{z\}\_\{t\}\\leftarrow\(1\-t\)\\bm\{z\}\_\{0\}\+t\\bm\{z\}\_\{1\},

vt←𝒛1−𝒛0v\_\{t\}\\leftarrow\\bm\{z\}\_\{1\}\-\\bm\{z\}\_\{0\}
6:Compute

\(uϕ​\(𝒛t,r,t\),dd​t​uϕ​\(𝒛t,r,t\)\)\(u\_\{\\phi\}\(\\bm\{z\}\_\{t\},r,t\),\\frac\{\\mathrm\{d\}\}\{\\mathrm\{d\}t\}u\_\{\\phi\}\(\\bm\{z\}\_\{t\},r,t\)\)via JVP

7:Update

ϕ\\phiby minimizing

ℒ⁡\(ϕ\)\\mathcal\{L\}\(\\phi\)in Equation[2](https://arxiv.org/html/2609.12382#S3.E2)

8:endfor

9:Freeze

ϕ\\phi\(and freeze

gg\)\.

10:Stage II: Train RMDM conditioned on representations\.

11:foreach minibatch

𝒙0∼𝒟\\bm\{x\}\_\{0\}\\sim\\mathcal\{D\}do

12:

𝒆←gψ​\(𝒙0\)\\bm\{e\}\\leftarrow g\_\{\\psi\}\(\\bm\{x\}\_\{0\}\)
13:

𝒛0←𝒆−uϕ​\(𝒆,0,1\)\\bm\{z\}\_\{0\}\\leftarrow\\bm\{e\}\-u\_\{\\phi\}\(\\bm\{e\},0,1\)\{map embedding to noise space\}

14:Sample

ttand construct a masked input

𝒙t\\bm\{x\}\_\{t\}by the MDM corruption process

15:Update

θ\\thetaby minimizing

ℒ⁡\(θ\)\\mathcal\{L\}\(\\theta\)in Equation[3](https://arxiv.org/html/2609.12382#S3.E3)

16:endfor

In this subsection, we introduce the Representation\-based Masked Diffusion Model \(RMDM\)\. The training process includes two stages\. First, we obtain the representation𝒛\\bm\{z\}from data𝒙\\bm\{x\}through a pretrained encoder and transform its distribution to a standard Gaussian distribution by training a MeanFlow model\. Then, we use this representation𝒛\\bm\{z\}as a condition to train the MDM\. To operationalize the latent variable𝒛\\bm\{z\}and bridge the conditional dependency gap discussed in Section[3\.1](https://arxiv.org/html/2609.12382#S3.SS1), we leverage the rich semantic representations provided by pretrained encoder models\. Encoder\-only architectures, ranging from the seminal BERT\([Devlin et al\., 2019](https://arxiv.org/html/2609.12382#bib.bib36)\)to modern large\-scale embedding models such as Qwen3\([Zhang et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib57)\), map a discrete sequence𝒙\\bm\{x\}to a contextualized continuous embedding𝒛1=gψ​\(x\)∈ℝL×Dz\\bm\{z\}\_\{1\}=g\_\{\\psi\}\(x\)\\in\\mathbb\{R\}^\{L\\times D\_\{z\}\}, wheregψg\_\{\\psi\}is the encoder model\. These contextualized embeddings capture high\-level semantic structures and provide a smoother, more informative representation, making them ideal candidates for the global latent\.

However, a direct utilization of this embedding introduces a fundamental discrepancy between training and inference: while𝒛1\\bm\{z\}\_\{1\}is readily computable from the ground\-truth sequence𝒙\\bm\{x\}during training, it remains unobserved during inference\. To resolve this, in stage I, we push the embedding distribution to a prior distribution which is easier to sample from while keeping the semantic structure of the embedding\. In particular, we adopt the MeanFlow framework\([Geng et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib3)\)to learn a mapping between the data embedding𝒛1\\bm\{z\}\_\{1\}and a standard Gaussian prior𝒛0∼𝒩⁡\(𝟎,𝑰\)\\bm\{z\}\_\{0\}\\sim\\mathcal\{N\}\(\\bm\{0\},\\bm\{I\}\)\. MeanFlow builds on the linear interpolation path𝒛t=\(1−t\)​𝒛0\+t​𝒛1\\bm\{z\}\_\{t\}=\(1\-t\)\\bm\{z\}\_\{0\}\+t\\bm\{z\}\_\{1\}like flow matching\([Lipman et al\., 2023](https://arxiv.org/html/2609.12382#bib.bib8)\), but replaces the instantaneous velocity field with anaverage velocityfieldu⁡\(𝒛t,r,t\)u\(\\bm\{z\}\_\{t\},r,t\)defined over an interval\[r,t\]\[r,t\]:

u⁡\(𝒛t,r,t\)≜1t−r​∫rtv⁡\(𝒛τ,τ\)​𝑑τ\.\\displaystyle u\(\\bm\{z\}\_\{t\},r,t\)\\triangleq\\frac\{1\}\{t\-r\}\\int\_\{r\}^\{t\}v\(\\bm\{z\}\_\{\\tau\},\\tau\)\\mathrm\{d\}\\tau\.We parameterizeuϕ​\(𝒛t,r,t\)u\_\{\\phi\}\(\\bm\{z\}\_\{t\},r,t\)with a neural network and train it using the MeanFlow identity, which relates average velocity and instantaneous velocity\. Concretely, lettingvt≜𝒛1−𝒛0v\_\{t\}\\triangleq\\bm\{z\}\_\{1\}\-\\bm\{z\}\_\{0\}denote the conditional velocity associated with the linear path, the MeanFlow objective regressesuϕu\_\{\\phi\}to the effective target induced by the identity:

ℒ⁡\(ϕ\)≜𝔼⁡\[‖uϕ​\(𝒛t,r,t\)−sg​\(vt−\(t−r\)​dd​t​uϕ​\(𝒛t,r,t\)\)‖2\]\\mathcal\{L\}\(\\phi\)\\triangleq\\mathbb\{E\}\\left\[\\left\\lVert u\_\{\\phi\}\(\\bm\{z\}\_\{t\},r,t\)\-\\text\{sg\}\\left\(v\_\{t\}\-\(t\-r\)\\frac\{\\mathrm\{d\}\}\{\\mathrm\{d\}t\}u\_\{\\phi\}\(\\bm\{z\}\_\{t\},r,t\)\\right\)\\right\\rVert^\{2\}\\right\]

\(2\)wheresgdenotes the stop\-gradient operator and the total derivativedd​t\\frac\{\\mathrm\{d\}\}\{\\mathrm\{d\}t\}is computed efficiently via Jacobian\-vector products \(as summarized in Section[2\.1](https://arxiv.org/html/2609.12382#S2.SS1)\)\.

Once trained, MeanFlow enables efficient one\-step transport: given a data embedding𝒛1\\bm\{z\}\_\{1\}, we recover its corresponding representation𝒛0\\bm\{z\}\_\{0\}in the Gaussian space by inverting the flow over the full interval\[0,1\]\[0,1\]:

𝒛0=𝒛1−uϕ​\(𝒛1,0,1\)\.\\displaystyle\\bm\{z\}\_\{0\}=\\bm\{z\}\_\{1\}\-u\_\{\\phi\}\(\\bm\{z\}\_\{1\},0,1\)\.Figure[3](https://arxiv.org/html/2609.12382#S3.F3)shows the UMAP\([McInnes et al\., 2018](https://arxiv.org/html/2609.12382#bib.bib58)\)visualization of embeddings𝒛1≜𝒆\\bm\{z\}\_\{1\}\\triangleq\\bm\{e\}and𝒛0\\bm\{z\}\_\{0\}obtained via MeanFlow, where𝒆\\bm\{e\}has its complex intrinsic geometric structure while the distribution of𝒛0\\bm\{z\}\_\{0\}is stable and easy to sample from\. We then use this Gaussian representation to condition the masked diffusion language model\. Specifically, during Stage II training we compute the data\-dependent latent𝒛0=gψ​\(𝒙0\)−uϕ​\(gψ​\(𝒙0\),0,1\)\\bm\{z\}\_\{0\}=g\_\{\\psi\}\(\\bm\{x\}\_\{0\}\)\-u\_\{\\phi\}\(g\_\{\\psi\}\(\\bm\{x\}\_\{0\}\),0,1\)using the frozen pretrained encodergψg\_\{\\psi\}and the trained MeanFlow networkuϕu\_\{\\phi\}\. We then train the RMDMpθ​\(𝒙0i∣𝒙t,𝒛0\)p\_\{\\theta\}\(\\bm\{x\}\_\{0\}^\{i\}\\mid\\bm\{x\}\_\{t\},\\bm\{z\}\_\{0\}\)to reconstruct masked tokens\. Using the standard MDM training objective in Equation[1](https://arxiv.org/html/2609.12382#S2.E1), we optimize:

ℒ\(θ\)≜𝔼t,𝒙0,𝒙t,𝒛0\[1t∑i∈ℐt−logpθ\(𝒙0i∣𝒙t,𝒛0\)\]\.\\mathcal\{L\}\(\\theta\)\\triangleq\\mathbb\{E\}\_\{t,\\bm\{x\}\_\{0\},\\bm\{x\}\_\{t\},\\bm\{z\}\_\{0\}\}\\left\[\\frac\{1\}\{t\}\\sum\_\{i\\in\\mathcal\{I\}\_\{t\}\}\-\\log p\_\{\\theta\}\(\\bm\{x\}\_\{0\}^\{i\}\\mid\\bm\{x\}\_\{t\},\\bm\{z\}\_\{0\}\)\\right\]\.\(3\)whereℐt=\{i:𝒙ti=\[MASK\]\}\\mathcal\{I\}\_\{t\}=\\\{i:\\bm\{x\}\_\{t\}^\{i\}=\\texttt\{\[MASK\]\}\\\}\. At inference time, since we have aligned the representation space with a Gaussian prior, we simply sample𝒛\\bm\{z\}directly from the prior𝒩⁡\(𝟎,𝑰\)\\mathcal\{N\}\(\\bm\{0\},\\bm\{I\}\), which yields a global representation used to guide parallel token generation\. In addition, during the training in stage two and inference, the integration ofz0z\_\{0\}will not lead to additional huge computation costs\.

Algorithm 2Sampling with RMDM0:RMDM

pθp\_\{\\theta\}, number of denoising steps

KK
1:Sample

𝒛∼𝒩⁡\(𝟎,𝑰\)\\bm\{z\}\\sim\\mathcal\{N\}\(\\bm\{0\},\\bm\{I\}\)
2:Initialize

𝒙\(0\)\\bm\{x\}^\{\(0\)\}as all\[MASK\]

3:for

k=0,1,…,K−1k=0,1,\\dots,K\-1do

4:Select a subset of masked positions to update \(parallel schedule\)

5:Sample tokens in parallel:

𝒙\(k\+1\)∼pθ\(⋅∣𝒙\(k\),𝒛\)\\bm\{x\}^\{\(k\+1\)\}\\sim p\_\{\\theta\}\(\\cdot\\mid\\bm\{x\}^\{\(k\)\},\\bm\{z\}\)on selected positions

6:endfor

7:return

𝒙\(K\)\\bm\{x\}^\{\(K\)\}

### 3\.3Theoretical Analysis: Decomposing the Generation Gap

In this section, we provide a rigorous decomposition of the error associated with parallel decoding in RMDM\.

###### Proposition 3\.2\(Tri\-part Decomposition of the Joint KL\)\.

Letpϕ​\(𝐳\)p\_\{\\phi\}\(\\bm\{z\}\)be the distribution of latents derived from the MeanFlow transport \(approximating the posterior\), and define the induced joint

pdata​\(𝒙M,𝒙U,𝒛\)≜pϕ​\(𝒛\)​pdata​\(𝒙U∣𝒛\)​pdata​\(𝒙M∣𝒙U,𝒛\),p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{M\},\\bm\{x\}\_\{U\},\\bm\{z\}\)\\triangleq p\_\{\\phi\}\(\\bm\{z\}\)\\,p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{U\}\\mid\\bm\{z\}\)\\,p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\),

RMDM generative samples factorized as

pθ​\(𝒙M,𝒙U,𝒛\)=p⁡\(𝒛\)​∏i∈Upθ​\(xi∣𝒛\)​∏i∈Mpθ​\(xi∣𝒙U,𝒛\)\.p\_\{\\theta\}\(\\bm\{x\}\_\{M\},\\bm\{x\}\_\{U\},\\bm\{z\}\)=p\(\\bm\{z\}\)\\,\\prod\_\{i\\in U\}p\_\{\\theta\}\(x\_\{i\}\\mid\\bm\{z\}\)\\,\\prod\_\{i\\in M\}p\_\{\\theta\}\(x\_\{i\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\.

wherep⁡\(z\)p\(z\)is the prior distribution\. Then the joint divergence between the data distribution and the model admits the following tri\-part decomposition\. Throughout the statement, a conditional KL such asDKL\(p\(y∣x\)∥q\(y∣x\)\)D\_\{\\mathrm\{KL\}\}\(p\(y\\mid x\)\\\|q\(y\\mid x\)\)denotes the averaged quantity𝔼p⁡\(x\)DKL\(p\(y∣x\)∥q\(y∣x\)\)\\mathbb\{E\}\_\{p\(x\)\}D\_\{\\mathrm\{KL\}\}\(p\(y\\mid x\)\\\|q\(y\\mid x\)\); total\-correlation terms are averaged over their conditioning variables in the same way:

DKL​\(pdata​\(𝒙M,𝒙U,𝒛\)∥pθ​\(𝒙M,𝒙U,𝒛\)\)\\displaystyle D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{M\},\\bm\{x\}\_\{U\},\\bm\{z\}\)\\,\\middle\\\|\\,p\_\{\\theta\}\(\\bm\{x\}\_\{M\},\\bm\{x\}\_\{U\},\\bm\{z\}\)\\right\)=\\displaystyle=\\;DKL​\(pϕ​\(𝒛\)∥p⁡\(𝒛\)\)\\displaystyle D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\phi\}\(\\bm\{z\}\)\\,\\middle\\\|\\,p\(\\bm\{z\}\)\\right\)\+\\displaystyle\+𝒯⁡\(𝒙U∣𝒛\)\+𝒯⁡\(𝒙M∣𝒙U,𝒛\)\\displaystyle\\mathcal\{T\}\(\\bm\{x\}\_\{U\}\\mid\\bm\{z\}\)\+\\mathcal\{T\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\+\\displaystyle\+∑i∈UDKL​\(pdata​\(xi∣𝒛\)∥pθ​\(xi∣𝒛\)\)\\displaystyle\\sum\_\{i\\in U\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(x\_\{i\}\\mid\\bm\{z\}\)\\,\\middle\\\|\\,p\_\{\\theta\}\(x\_\{i\}\\mid\\bm\{z\}\)\\right\)\+\\displaystyle\+∑i∈MDKL​\(pdata​\(xi∣𝒙U,𝒛\)∥pθ​\(xi∣𝒙U,𝒛\)\)\.\\displaystyle\\sum\_\{i\\in M\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(x\_\{i\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\\,\\middle\\\|\\,p\_\{\\theta\}\(x\_\{i\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\\right\)\.\(4\)The four lines after the equality correspond to \(I\) MeanFlow–prior mismatch, \(II\) remaining dependency, and the two parts of \(III\) RMDM prediction error\.

The proof is provided in Appendix[B\.2](https://arxiv.org/html/2609.12382#A2.SS2)\.

##### Derivation and Analysis\.

We analyze each term in the decomposition to demonstrate how our framework systematically addresses the challenges of parallel generation in masked diffusion models\.

\(I\) MeanFlow–Prior Mismatch\.The first termDKL\(pϕ\(𝒛\)∥p\(𝒛\)\)D\_\{\\mathrm\{KL\}\}\(p\_\{\\phi\}\(\\bm\{z\}\)\\\|p\(\\bm\{z\}\)\)quantifies the discrepancy between the aggregated posteriorpϕ​\(𝒛\)p\_\{\\phi\}\(\\bm\{z\}\)\(induced by MeanFlow transport from encoder embeddings\) and the priorp⁡\(𝒛\)=𝒩⁡\(𝟎,𝑰\)p\(\\bm\{z\}\)=\\mathcal\{N\}\(\\bm\{0\},\\bm\{I\}\)used during generation\. This mismatch is critical because the decomposition in Proposition[3\.2](https://arxiv.org/html/2609.12382#S3.Thmtheorem2)assumes that the ground truth statistical modeling is based on the posteriorpϕ​\(𝒛\)p\_\{\\phi\}\(\\bm\{z\}\), while inference samples frompθ​\(𝒛\)p\_\{\\theta\}\(\\bm\{z\}\)\. MeanFlow implicitly minimizes this term by learning a transport map that aligns the encoder\-derived latent distribution with the Gaussian prior at the population level\. Unlike standard variational autoencoders that enforce per\-sample posterior constraints, MeanFlow’s population\-level alignment ensures that latents sampled from the prior during inference closely match the distribution of latents seen during training, thereby preserving the validity of the decomposition\.

\(II\) Remaining Dependency\.The second term𝒯⁡\(𝒙U∣𝒛\)\+𝒯⁡\(𝒙M∣𝒙U,𝒛\)\\mathcal\{T\}\(\\bm\{x\}\_\{U\}\\mid\\bm\{z\}\)\+\\mathcal\{T\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)captures the conditional dependencies among tokens that remain after conditioning on the latent𝒛\\bm\{z\}\. These terms measure the irreducible error of the factorization model: even with perfect token\-wise modeling, dependencies not explained by𝒛\\bm\{z\}will cause the factorized approximation to deviate from the true joint distribution\. By leveraging a powerful pretrained encoder like Qwen3\([Zhang et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib57)\)to extract rich global semantics𝒛\\bm\{z\}, we minimize these dependency terms \(Proposition[3\.1](https://arxiv.org/html/2609.12382#S3.Thmtheorem1)\), making the factorized assumptionp⁡\(𝒙M∣𝒙U,𝒛\)=∏i∈Mp⁡\(xi∣𝒙U,𝒛\)p\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)=\\prod\_\{i\\in M\}p\(x\_\{i\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)approximately valid\. The quality of this term is thus determined by the representational capacity of the pretrained encodergψg\_\{\\psi\}\.

\(III\) RMDM Prediction Error\.The third term∑i∈U∪MDKL\(pdata\(xi∣⋅\)∥pθ\(xi∣⋅\)\)\\sum\_\{i\\in U\\cup M\}D\_\{\\mathrm\{KL\}\}\(p\_\{\\text\{data\}\}\(x\_\{i\}\\mid\\cdot\)\\\|p\_\{\\theta\}\(x\_\{i\}\\mid\\cdot\)\)measures the token\-wise prediction error ofpθp\_\{\\theta\}given the latent𝒛\\bm\{z\}\. This term corresponds directly to the RMDM training objective Equation[3](https://arxiv.org/html/2609.12382#S3.E3), which optimizes the model to reconstruct individual tokens conditioned on the masked context𝒙U\\bm\{x\}\_\{U\}and the global latent𝒛\\bm\{z\}\. By training a powerful diffusion modelpθp\_\{\\theta\}to minimize the ELBO, we implicitly reduce the prediction error\. The quality of this term depends on the capacity and training of the RMDMpθp\_\{\\theta\}\.

##### Summary\.

Proposition[3\.2](https://arxiv.org/html/2609.12382#S3.Thmtheorem2)provides a unified theoretical justification for our two\-stage framework:

- •We minimize\(I\)by aligning the latent space via MeanFlow transport, ensuring inference\-time latents match training\-time distributions\.
- •We minimize\(II\)by extracting rich global semantics𝒛\\bm\{z\}from a powerful pretrained encoder, reducing residual dependencies among tokens\.
- •We minimize\(III\)by training a powerful masked diffusion modelpθp\_\{\\theta\}to accurately model token distributions conditioned on𝒛\\bm\{z\}\.

### 3\.4Model Architecture

Table 1:Test perplexities \(PPL↓\\downarrow\) on LM1B\.†\\daggerdenotes our retrained models, other results are reported in[He et al\. \(2023\)](https://arxiv.org/html/2609.12382#bib.bib48)and[Sahoo et al\. \(2024\)](https://arxiv.org/html/2609.12382#bib.bib13)\. Best value is bolded\.Table 2:Generation quality on OpenWebText\. GenPPL is evaluated by GPT\-2; Judge is the average LLM\-judge overall\-quality score\.##### Backbone and Conditioning Mechanism\.

We employ a Diffusion Transformer \(DiT\) architecture\([Vaswani et al\., 2017](https://arxiv.org/html/2609.12382#bib.bib23);[Peebles and Xie, 2023](https://arxiv.org/html/2609.12382#bib.bib50)\)for both the MeanFlow networkuϕu\_\{\\phi\}and the RMDMpθp\_\{\\theta\}\. The model operates on a sequence of hidden states𝒉∈ℝL×D\\bm\{h\}\\in\\mathbb\{R\}^\{L\\times D\}, derived from discrete token inputs𝒙∈\{1,…,V\}L\\bm\{x\}\\in\\\{1,\\dots,V\\\}^\{L\}via a learned embedding table\. To incorporate the global semantic context, we inject the latent variable𝒛∈ℝL×Dz\\bm\{z\}\\in\\mathbb\{R\}^\{L\\times D\_\{z\}\}into every transformer block using Adaptive Layer Normalization \(AdaLN\)\([Xu et al\., 2019](https://arxiv.org/html/2609.12382#bib.bib52);[Peebles and Xie, 2023](https://arxiv.org/html/2609.12382#bib.bib50)\)\. Specifically, the layer input𝒉\\bm\{h\}is modulated as:

fAdaLN​\(𝒉,𝒛\)≜f⁡\(𝜸⁡\(𝒛\)⊙LN⁡\(𝒉\)\+𝜷⁡\(𝒛\)\)⊙α⁡\(𝒛\)\+𝒉\.f\_\{\\text\{AdaLN\}\}\(\\bm\{h\};\\bm\{z\}\)\\triangleq f\\\!\\left\(\\bm\{\\gamma\}\(\\bm\{z\}\)\\odot\\mathrm\{LN\}\(\\bm\{h\}\)\+\\bm\{\\beta\}\(\\bm\{z\}\)\\right\)\\odot\\alpha\(\\bm\{z\}\)\+\\bm\{h\}\.

where𝜸⁡\(⋅\),𝜷⁡\(⋅\),α⁡\(⋅\)\\bm\{\\gamma\}\(\\cdot\),\\bm\{\\beta\}\(\\cdot\),\\alpha\(\\cdot\)are parameters predicted from𝒛\\bm\{z\}by a shallow MLP,f⁡\(⋅\)f\(\\cdot\)is a multiplication\-like operation that can be implemented as an attention or MLP\. Unlike standard diffusion models that primarily condition on the timesteptt, we ensure the denoising process is consistently guided by the global semantic representation𝒛\\bm\{z\}across all layers\.

##### Latent Stabilization\.

Conditioning on high\-dimensional latents from large pretrained encoders presents a challenge: the model may overfit to the training latents, leading to poor generalization on samples drawn from the prior𝒩⁡\(𝟎,𝑰\)\\mathcal\{N\}\(\\bm\{0\},\\bm\{I\}\)during inference\. We mitigate this via a two\-step stabilization strategy\. First, we project𝒛\\bm\{z\}channel\-wise to a lower\-dimensional spaceℝL×dz\\mathbb\{R\}^\{L\\times d\_\{z\}\}\(wheredz<Dzd\_\{z\}<D\_\{z\}\) via an orthogonal linear map and then upsample it to the original channel dimension\. This bottleneck keeps the model from focusing on robust semantic features and reduces the complexity of the conditioning signal, while the simplicity of the prior is well preserved\. Second, during training, we inject isotropic Gaussian noiseϵ∼𝒩⁡\(𝟎,σ2​𝑰\)\\bm\{\\epsilon\}\\sim\\mathcal\{N\}\(\\bm\{0\},\\sigma^\{2\}\\bm\{I\}\)into the projected latent, which acts as a smoothing regularizer and encourages the model to be robust to local perturbations in the latent space\. Empirically, these strategies improve the generalizability\.

## 4Experiments

Table 3:Zero\-shot perplexities \(↓\\downarrow\) of models trained on OpenWebText \(OWT\)\. All perplexities for diffusion models are upper bounds\. Best is in bold and the second best is underlined\.### 4\.1Experimental Setup

##### Tasks and Datasets\.

We study unconditional text generation following the standard evaluation protocol for diffusion language models\([Lou et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib40);[Sahoo et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib13)\)\. We evaluate on One Billion Word \(LM1B\)\([Chelba et al\., 2013](https://arxiv.org/html/2609.12382#bib.bib43)\)and OpenWebText \(OWT\)\([Gokaslan and Cohen, 2019](https://arxiv.org/html/2609.12382#bib.bib33)\)\. For LM1B, we report test perplexity derived from Equation[3](https://arxiv.org/html/2609.12382#S3.E3)\. For OWT, we report zero\-shot perplexity on WikiText\([Merity et al\., 2017](https://arxiv.org/html/2609.12382#bib.bib53)\), LM1B, LAMBADA\([Paperno et al\., 2016](https://arxiv.org/html/2609.12382#bib.bib54)\), AG News\([Zhang et al\., 2015](https://arxiv.org/html/2609.12382#bib.bib55)\), and Scientific Papers\([Cohan et al\., 2018](https://arxiv.org/html/2609.12382#bib.bib56)\)\. For generation quality, we evaluate 256 generated samples with GPT\-2 GenPPL\([Radford et al\., 2019](https://arxiv.org/html/2609.12382#bib.bib35)\)\. Since GenPPL is an evaluator\-perplexity proxy and may not fully reflect human\-perceived generation quality\([Shi et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib21)\), we additionally evaluate overall quality with two independent LLM judges\. Implementation details, including tokenization, latent encoders, model sizes, and optimization settings, are provided in Appendix[C](https://arxiv.org/html/2609.12382#A3); training and inference costs are reported in Appendices[F](https://arxiv.org/html/2609.12382#A6)and[E](https://arxiv.org/html/2609.12382#A5); and the LLM\-judge protocol and confidence intervals are provided in Appendix[D](https://arxiv.org/html/2609.12382#A4)\. RMDM and our retrained baselines use the same tokenizer, DiT backbone, data preprocessing, and optimization schedule\. RMDM additionally uses representations from a pretrained encoder during training; the baselines do not receive equivalent external representation knowledge\.

### 4\.2Main Results

##### GenPPL improvements are the strongest when steps are scarce\.

On OpenWebText, Table[2](https://arxiv.org/html/2609.12382#S3.T2)shows that RMDM consistently improves GenPPL over MDM\-style baselines like MDLM\([Sahoo et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib13)\)and SEDD\([Lou et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib40)\), and the gap widens as the sampling budget decreases\. With only a few parallel refinement steps, masked diffusion often loses global coherence due to limited shared context; in contrast, RMDM injects a sequence\-level representation at each denoising step to coordinate simultaneous token updates\. RMDM also obtains the strongest primary\-judge scores across all sampling budgets, and an independent second judge yields the same overall ranking trend\. This agreement suggests that the GenPPL gains are accompanied by improvements in the qualities targeted by the evaluation rubric rather than being specific to one evaluator\.

Table 4:Matched\-quality throughput on one A800 GPU \(batch size 8,L=1024L=1024\)\.
##### Inference efficiency\.

Table[4](https://arxiv.org/html/2609.12382#S4.T4)reports measured throughput at representative matched\-quality operating points\. Although latent conditioning makes RMDM about 10% slower than MDLM at the same number of steps, RMDM reaches the same GenPPL with substantially fewer steps, yielding approximately3\.6×3\.6\\timeshigher throughput\. Peak sampling memory increases only from 18\.64 GB to 18\.93 GB\. The pretrained encoder and MeanFlow network are not run at inference; generation draws the latent once from the prior\. Full measurements are reported in Appendix[E](https://arxiv.org/html/2609.12382#A5)\.

##### Validation and zero\-shot perplexity\.

Beyond sample quality, RMDM does not degrade likelihood\-based evaluation on LM1B \(Table[1](https://arxiv.org/html/2609.12382#S3.T1)\)\. We speculate that its strong LM1B performance even against AR baselines is partly due to the large fraction of\[PAD\]tokens in LM1B sequences: the sequence\-level representation can identify padding structure and provide a shared global signal during denoising\. After scaling on OpenWebText, it also improves zero\-shot perplexity over diffusion baselines across diverse test sets \(Table[3](https://arxiv.org/html/2609.12382#S4.T3)\), suggesting that the learned representation distribution generalizes beyond training data\.

### 4\.3Ablation Study

Table 5:Ablation study on OpenWebText\. Indep\.zzdenotes a variant trained with Gaussian conditioning latents sampled independently of the target sequencexx\.##### Source of the continuous latentzz\.

Table[5](https://arxiv.org/html/2609.12382#S4.T5)isolates whether the gain comes from the data\-dependent representation or merely from adding a Gaussian conditioning variable\. We replace the encoded latent with an independent samplez∼𝒩⁡\(0,I\)z\\sim\\mathcal\{N\}\(0,I\)during the training process, while keeping RMDM interface, conditioning mechanism, and sampling schedule unchanged\. This control has a similar marginal distribution to the aligned latent prior, but removes the sequence\-level information that RMDM is designed to provide\. It is consistently worse than RMDM across all sampling budgets, with particularly clear degradation under few\-step sampling\. This pattern shows that the extra conditioning pathway alone is not sufficient: a random Gaussian vector may match the prior distribution, but it cannot explain residual dependencies among simultaneously masked tokens\. The useful signal is instead the data\-dependent representation carried byzz, which acts as a global plan and gives masked positions shared information for coordinating parallel token updates\.

## 5Conclusion

In this paper, we present Representation\-based Masked Diffusion Model \(RMDM\), which extends masked diffusion language models with a global continuous latent variable to explicitly model sequence\-level semantics and restore inter\-token dependencies\. This advantage of RMDM makes it possible to correctly generate multiple masked tokens simultaneously and is especially powerful with a small number of sampling steps, where standard parallel sampling methods are most prone to losing global coherence\. The MeanFlow\-based representation learning mechanism aligns pretrained encoder embeddings with a Gaussian prior, allowing the latent to be sampled once at inference without running the encoder or MeanFlow network\. By conditioning the masked diffusion model on these latent representations, we couple the updating of all masked tokens through shared global semantic information, reducing the Conditional Dependency Gap in parallel generation\. Experiments on LM1B and OpenWebText demonstrate that RMDM consistently improves generation quality, especially in aggressive few\-step regimes, and achieves approximately3\.6×3\.6\\timeshigher measured throughput at matched quality in our A800 evaluation\. Our approach demonstrates the importance of global semantic coordination in non\-autoregressive generation and provides a principled framework for addressing the conditional dependency gap\.

## Limitations

The experiments mainly focus on unconditional generation at model scales and computational budgets comparable to GPT\-2\-level\. While the results provide evidence for the effectiveness of the approach in this setting, its transferability to substantially larger\-scale models and conditional generation tasks remains to be examined\. We leave these extensions for future work\.

## Ethics Statement

This work focuses on the statistical modeling problem of masked diffusion language models, specifically the conditional independence assumption used in parallel masked\-token updates\. It does not involve human subjects, new data collection, or additional model training beyond the experimental setup described in the paper\. All experiments are conducted on publicly available benchmarks\. The proposed method does not introduce new ethical risks beyond those inherent to the underlying language models and the benchmark data on which they are evaluated\.

## Acknowledgments

The authors acknowledge support from The Hong Kong Polytechnic University \(Research Grant P0046811\)\.

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.GPT\-4 technical report\.arXiv preprint arXiv:2303\.08774\.External Links:[Link](https://arxiv.org/abs/2303.08774)Cited by:[§1](https://arxiv.org/html/2609.12382#S1.p1.1)\.
- Austinet al\.\(2021\)J\. Austin, D\. D\. Johnson, J\. Ho, D\. Tarlow, and R\. Van Den BergStructured denoising diffusion models in discrete state\-spaces\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 17981–17993\.Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.12382#S1.p1.1),[§2\.2](https://arxiv.org/html/2609.12382#S2.SS2.p1.1)\.
- Bowmanet al\.\(2016\)S\. R\. Bowman, L\. Vilnis, O\. Vinyals, A\. M\. Dai, R\. Jozefowicz, and S\. BengioGenerating sentences from a continuous space\.InProceedings of the 20th SIGNLL Conference on Computational Natural Language Learning,pp\. 10–21\.External Links:[Document](https://dx.doi.org/10.18653/v1/K16-1002)Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px3.p1.1)\.
- Brownet al\.\(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, G\. Shyam, G\. Sastry, A\. Askell,et al\.Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2609.12382#S1.p1.1)\.
- Campbellet al\.\(2022\)A\. Campbell, J\. Benton, V\. De Bortoli, T\. Rainforth, G\. Deligiannidis, and A\. DoucetA continuous time framework for discrete denoising models\.Advances in Neural Information Processing Systems35,pp\. 28266–28279\.Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px1.p1.1)\.
- Campbellet al\.\(2024\)A\. Campbell, J\. Yim, R\. Barzilay, T\. Rainforth, and T\. JaakkolaGenerative flows on discrete state\-spaces: enabling multimodal flows with applications to protein co\-design\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 5453–5512\.External Links:[Link](https://proceedings.mlr.press/v235/campbell24a.html)Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px2.p1.1)\.
- Changet al\.\(2022\)H\. Chang, H\. Zhang, L\. Jiang, C\. Liu, and W\. T\. FreemanMaskgit: masked generative image transformer\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 11315–11325\.Cited by:[§1](https://arxiv.org/html/2609.12382#S1.p2.1)\.
- Chelbaet al\.\(2013\)C\. Chelba, T\. Mikolov, M\. Schuster, Q\. Ge, T\. Brants, P\. Koehn, and T\. RobinsonOne billion word benchmark for measuring progress in statistical language modeling\.arXiv preprint arXiv:1312\.3005\.Cited by:[§4\.1](https://arxiv.org/html/2609.12382#S4.SS1.SSS0.Px1.p1.1)\.
- Cohanet al\.\(2018\)A\. Cohan, F\. Dernoncourt, D\. S\. Kim, T\. Bui, S\. Kim, W\. Chang, and N\. GoharianA discourse\-aware attention model for abstractive summarization of long documents\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 \(Short Papers\),pp\. 615–621\.External Links:[Link](http://dx.doi.org/10.18653/v1/n18-2097),[Document](https://dx.doi.org/10.18653/v1/n18-2097)Cited by:[§4\.1](https://arxiv.org/html/2609.12382#S4.SS1.SSS0.Px1.p1.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-AIDeepSeek\-V4\-Flash\.Note:DeepSeek API documentationExternal Links:[Link](https://api-docs.deepseek.com/updates/)Cited by:[Appendix D](https://arxiv.org/html/2609.12382#A4.p1.1)\.
- Devlinet al\.\(2019\)J\. Devlin, M\. Chang, K\. Lee, and K\. ToutanovaBERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 4171–4186\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1423)Cited by:[Appendix C](https://arxiv.org/html/2609.12382#A3.SS0.SSS0.Px1.p1.1),[Appendix C](https://arxiv.org/html/2609.12382#A3.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2609.12382#S1.SS0.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2609.12382#S3.SS2.p1.1)\.
- Dielemanet al\.\(2022\)S\. Dieleman, L\. Sartran, A\. Roshannai, N\. Savinov, Y\. Ganin, P\. H\. Richemond, A\. Doucet, R\. Strudel, C\. Dyer, C\. Durkan,et al\.Continuous diffusion for categorical data\.arXiv preprint arXiv:2211\.15089\.Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px2.p1.1)\.
- Duet al\.\(2024\)Y\. Du, J\. Mao, and J\. B\. TenenbaumLearning iterative reasoning through energy diffusion\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 11764–11776\.External Links:[Link](https://proceedings.mlr.press/v235/du24f.html)Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px1.p1.1)\.
- Gatet al\.\(2024\)I\. Gat, T\. Remez, N\. Shaul, F\. Kreuk, R\. T\. Chen, G\. Synnaeve, Y\. Adi, and Y\. LipmanDiscrete flow matching\.Advances in Neural Information Processing Systems37,pp\. 133345–133385\.Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px1.p1.1)\.
- Genget al\.\(2025\)Z\. Geng, M\. Deng, X\. Bai, J\. Z\. Kolter, and K\. HeMean flows for one\-step generative modeling\.External Links:2505\.13447,[Link](https://arxiv.org/abs/2505.13447)Cited by:[§1](https://arxiv.org/html/2609.12382#S1.SS0.SSS0.Px1.p1.1),[§2\.1](https://arxiv.org/html/2609.12382#S2.SS1.p1.2),[§3\.2](https://arxiv.org/html/2609.12382#S3.SS2.p2.1)\.
- Gokaslan and Cohen \(2019\)A\. Gokaslan and V\. CohenOpenWebText corpus\.Note:GitHub repositoryExternal Links:[Link](https://github.com/Skylion007/openwebtext)Cited by:[§4\.1](https://arxiv.org/html/2609.12382#S4.SS1.SSS0.Px1.p1.1)\.
- Gonget al\.\(2023\)S\. Gong, M\. Li, J\. Feng, Z\. Wu, and L\. KongDiffuSeq: sequence to sequence text generation with diffusion models\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=jQj-_rLVXsj)Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px2.p1.1)\.
- Google \(2026\)GoogleGemini 3\.1 Flash\-Lite\.Note:Gemini API documentationExternal Links:[Link](https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-lite)Cited by:[Appendix D](https://arxiv.org/html/2609.12382#A4.p1.1)\.
- Guet al\.\(2018\)J\. Gu, J\. Bradbury, C\. Xiong, V\. O\. Li, and R\. SocherNon\-autoregressive neural machine translation\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.12382#S1.p1.1)\.
- Gulrajani and Hashimoto \(2023\)I\. Gulrajani and T\. B\. HashimotoLikelihood\-based diffusion language models\.Advances in Neural Information Processing Systems36,pp\. 16693–16715\.Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px2.p1.1)\.
- Hanet al\.\(2023\)X\. Han, S\. Kumar, and Y\. TsvetkovSSD\-LM: semi\-autoregressive simplex\-based diffusion language model for text generation and modular control\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 11575–11596\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.647),[Link](https://aclanthology.org/2023.acl-long.647/)Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px2.p1.1)\.
- Hayakawaet al\.\(2025\)S\. Hayakawa, Y\. Takida, M\. Imaizumi, H\. Wakaki, and Y\. MitsufujiDistillation of discrete diffusion through dimensional correlations\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 22259–22297\.External Links:[Link](https://proceedings.mlr.press/v267/hayakawa25a.html)Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px3.p1.1)\.
- Heet al\.\(2023\)Z\. He, T\. Sun, Q\. Tang, K\. Wang, X\. Huang, and X\. QiuDiffusionBERT: improving generative masked language models with diffusion models\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 4521–4534\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.248),[Link](https://aclanthology.org/2023.acl-long.248/)Cited by:[Appendix C](https://arxiv.org/html/2609.12382#A3.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2609.12382#S3.T1)\.
- Hoet al\.\(2020\)J\. Ho, A\. Jain, and P\. AbbeelDenoising diffusion probabilistic models\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 6840–6851\.Cited by:[§2\.1](https://arxiv.org/html/2609.12382#S2.SS1.p1.1)\.
- Kaiseret al\.\(2018\)L\. Kaiser, S\. Bengio, A\. Roy, A\. Vaswani, N\. Parmar, J\. Uszkoreit, and N\. ShazeerFast decoding in sequence models using discrete latent variables\.InInternational Conference on Machine Learning,pp\. 2390–2399\.Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px3.p1.1)\.
- Kingma and Welling \(2014\)D\. P\. Kingma and M\. WellingAuto\-encoding variational bayes\.InICLR,Cited by:[§1](https://arxiv.org/html/2609.12382#S1.SS0.SSS0.Px1.p1.1)\.
- Konget al\.\(2025\)D\. Kong, M\. Zhao, D\. Xu, B\. Pang, S\. Wang, E\. Honig, Z\. Si, C\. Li, J\. Xie, S\. Xie, and Y\. N\. WuLatent thought models with variational Bayes inference\-time computation\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 31419–31436\.External Links:[Link](https://proceedings.mlr.press/v267/kong25c.html)Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px3.p1.1)\.
- Liet al\.\(2022\)X\. Li, J\. Thickstun, I\. Gulrajani, P\. S\. Liang, and T\. B\. HashimotoDiffusion\-lm improves controllable text generation\.Advances in neural information processing systems35,pp\. 4328–4343\.Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px2.p1.1)\.
- Lipmanet al\.\(2023\)Y\. Lipman, R\. T\. Q\. Chen, H\. Ben\-Hamu, M\. Nickel, and M\. LeFlow matching for generative modeling\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=PqvMRDCJT9t)Cited by:[§2\.1](https://arxiv.org/html/2609.12382#S2.SS1.p1.1),[§3\.2](https://arxiv.org/html/2609.12382#S3.SS2.p2.1)\.
- Liuet al\.\(2024\)A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§1](https://arxiv.org/html/2609.12382#S1.p1.1)\.
- Louet al\.\(2024\)A\. Lou, C\. Meng, and S\. ErmonDiscrete diffusion modeling by estimating the ratios of the data distribution\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 32819–32848\.External Links:[Link](https://proceedings.mlr.press/v235/lou24a.html)Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px1.p1.1),[Appendix C](https://arxiv.org/html/2609.12382#A3.SS0.SSS0.Px1.p1.1),[Appendix C](https://arxiv.org/html/2609.12382#A3.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2609.12382#S1.p1.1),[§2\.2](https://arxiv.org/html/2609.12382#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2609.12382#S4.SS1.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2609.12382#S4.SS2.SSS0.Px1.p1.1)\.
- Mahabadiet al\.\(2024\)R\. K\. Mahabadi, H\. Ivison, J\. Tae, J\. Henderson, I\. Beltagy, M\. E\. Peters, and A\. CohanTESS: text\-to\-text self\-conditioned simplex diffusion\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2347–2361\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.eacl-long.144),[Link](https://aclanthology.org/2024.eacl-long.144/)Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px2.p1.1)\.
- McInneset al\.\(2018\)L\. McInnes, J\. Healy, and J\. MelvilleUmap: uniform manifold approximation and projection for dimension reduction\.arXiv preprint arXiv:1802\.03426\.Cited by:[§3\.2](https://arxiv.org/html/2609.12382#S3.SS2.p3.2)\.
- Menget al\.\(2022\)C\. Meng, K\. Choi, J\. Song, and S\. ErmonConcrete score matching: generalized score matching for discrete data\.Advances in Neural Information Processing Systems35,pp\. 34532–34545\.Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px1.p1.1)\.
- Merityet al\.\(2017\)S\. Merity, C\. Xiong, J\. Bradbury, and R\. SocherPointer sentinel mixture models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Byj72udxe)Cited by:[§4\.1](https://arxiv.org/html/2609.12382#S4.SS1.SSS0.Px1.p1.1)\.
- Nieet al\.\(2025\)S\. Nie, F\. Zhu, Z\. You, X\. Zhang, J\. Ou, J\. Hu, J\. Zhou, Y\. Lin, J\. Wen, and C\. LiLarge language diffusion models\.arXiv preprint arXiv:2502\.09992\.Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.12382#S1.p2.1)\.
- Ouet al\.\(2025\)J\. Ou, S\. Nie, K\. Xue, F\. Zhu, J\. Sun, Z\. Li, and C\. LiYour absorbing discrete diffusion secretly models the conditional distributions of clean data\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=sMyXP8Tanm)Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.12382#S1.p1.1)\.
- Papernoet al\.\(2016\)D\. Paperno, G\. Kruszewski, A\. Lazaridou, N\. Q\. Pham, R\. Bernardi, S\. Pezzelle, M\. Baroni, G\. Boleda, and R\. FernándezThe LAMBADA dataset: word prediction requiring a broad discourse context\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),K\. Erk and N\. A\. Smith \(Eds\.\),Berlin, Germany,pp\. 1525–1534\.External Links:[Link](https://aclanthology.org/P16-1144/),[Document](https://dx.doi.org/10.18653/v1/P16-1144)Cited by:[§4\.1](https://arxiv.org/html/2609.12382#S4.SS1.SSS0.Px1.p1.1)\.
- Peebles and Xie \(2023\)W\. Peebles and S\. XieScalable diffusion models with transformers\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 4195–4205\.Cited by:[Appendix C](https://arxiv.org/html/2609.12382#A3.SS0.SSS0.Px3.p1.1),[§3\.4](https://arxiv.org/html/2609.12382#S3.SS4.SSS0.Px1.p1.1)\.
- Radfordet al\.\(2019\)A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. SutskeverLanguage models are unsupervised multitask learners\.OpenAI technical report\.External Links:[Link](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)Cited by:[§1](https://arxiv.org/html/2609.12382#S1.p1.1),[§4\.1](https://arxiv.org/html/2609.12382#S4.SS1.SSS0.Px1.p1.1)\.
- Rojaset al\.\(2025\)K\. Rojas, Y\. Zhu, S\. Zhu, F\. X\. Ye, and M\. TaoDiffuse everything: multimodal diffusion models on arbitrary state spaces\.arXiv preprint arXiv:2506\.07903\.Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px2.p1.1)\.
- Sahooet al\.\(2024\)S\. Sahoo, M\. Arriola, Y\. Schiff, A\. Gokaslan, E\. Marroquin, J\. Chiu, A\. Rush, and V\. KuleshovSimple and effective masked diffusion language models\.Advances in Neural Information Processing Systems37,pp\. 130136–130184\.Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px1.p1.1),[Appendix C](https://arxiv.org/html/2609.12382#A3.SS0.SSS0.Px1.p1.1),[Appendix C](https://arxiv.org/html/2609.12382#A3.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2609.12382#S1.p1.1),[Table 1](https://arxiv.org/html/2609.12382#S3.T1),[§4\.1](https://arxiv.org/html/2609.12382#S4.SS1.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2609.12382#S4.SS2.SSS0.Px1.p1.1)\.
- Sahooet al\.\(2025\)S\. S\. Sahoo, J\. Deschenaux, A\. Gokaslan, G\. Wang, J\. T\. Chiu, and V\. KuleshovThe diffusion duality\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 52584–52619\.External Links:[Link](https://proceedings.mlr.press/v267/sahoo25a.html)Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px2.p1.1)\.
- Shaulet al\.\(2024\)N\. Shaul, I\. Gat, M\. Havasi, D\. Severo, A\. Sriram, P\. Holderrieth, B\. Karrer, Y\. Lipman, and R\. T\. ChenFlow matching with general discrete paths: a kinetic\-optimal perspective\.arXiv preprint arXiv:2412\.03487\.Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px1.p1.1)\.
- Shiet al\.\(2024\)J\. Shi, K\. Han, Z\. Wang, A\. Doucet, and M\. TitsiasSimplified and generalized masked diffusion for discrete data\.Advances in neural information processing systems37,pp\. 103131–103167\.Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.12382#S4.SS1.SSS0.Px1.p1.1)\.
- Songet al\.\(2021\)Y\. Song, J\. Sohl\-Dickstein, D\. P\. Kingma, A\. Kumar, S\. Ermon, and B\. PooleScore\-based generative modeling through stochastic differential equations\.InInternational Conference on Learning Representations,Note:arXiv:2011\.13456Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px1.p1.1),[§2\.1](https://arxiv.org/html/2609.12382#S2.SS1.p1.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.Advances in neural information processing systems30\.Cited by:[§3\.4](https://arxiv.org/html/2609.12382#S3.SS4.SSS0.Px1.p1.1)\.
- von Rütteet al\.\(2025\)D\. von Rütte, J\. Fluri, Y\. Ding, A\. Orvieto, B\. Schölkopf, and T\. HofmannGeneralized interpolating discrete diffusion\.arXiv preprint arXiv:2503\.04482\.Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px1.p1.1)\.
- Warneret al\.\(2024\)B\. Warner, A\. Chaffin, B\. Clavié, O\. Weller, O\. Hallström, S\. Taghadouini, A\. Gallagher, R\. Biswas, F\. Ladhak, T\. Aarsen, N\. Cooper, G\. Adams, J\. Howard, and I\. PoliSmarter, better, faster, longer: a modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference\.External Links:2412\.13663,[Link](https://arxiv.org/abs/2412.13663)Cited by:[Appendix C](https://arxiv.org/html/2609.12382#A3.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2609.12382#S1.SS0.SSS0.Px1.p1.1)\.
- Wuet al\.\(2025\)C\. Wu, H\. Zhang, S\. Xue, Z\. Liu, S\. Diao, L\. Zhu, P\. Luo, S\. Han, and E\. XieFast\-dllm: training\-free acceleration of diffusion llm by enabling kv cache and parallel decoding\.arXiv preprint arXiv:2505\.22618\.Cited by:[§1](https://arxiv.org/html/2609.12382#S1.p2.1)\.
- Xieet al\.\(2025\)T\. Xie, S\. Xue, Z\. Feng, T\. Hu, J\. Sun, Z\. Li, and C\. ZhangVariational autoencoding discrete diffusion with enhanced dimensional correlations modeling\.External Links:2505\.17384,[Link](https://arxiv.org/abs/2505.17384)Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2609.12382#S1.SS0.SSS0.Px1.p1.1)\.
- Xuet al\.\(2019\)J\. Xu, X\. Sun, Z\. Zhang, G\. Zhao, and J\. LinUnderstanding and improving layer normalization\.Advances in neural information processing systems32\.Cited by:[§3\.4](https://arxiv.org/html/2609.12382#S3.SS4.SSS0.Px1.p1.1)\.
- Yeet al\.\(2024\)J\. Ye, S\. Gong, L\. Chen, L\. Zheng, J\. Gao, H\. Shi, C\. Wu, X\. Jiang, Z\. Li, W\. Bi,et al\.Diffusion of thought: chain\-of\-thought reasoning in diffusion language models\.Advances in Neural Information Processing Systems37,pp\. 105345–105374\.Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px1.p1.1),[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px2.p1.1)\.
- Yeet al\.\(2025\)J\. Ye, Z\. Xie, L\. Zheng, J\. Gao, Z\. Wu, X\. Jiang, Z\. Li, and L\. KongDream 7b: diffusion large language models\.arXiv preprint arXiv:2508\.15487\.Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.12382#S1.p2.1)\.
- Zhanget al\.\(2015\)X\. Zhang, J\. Zhao, and Y\. LeCunCharacter\-level convolutional networks for text classification\.Advances in neural information processing systems28\.Cited by:[§4\.1](https://arxiv.org/html/2609.12382#S4.SS1.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025\)Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin, F\. Huang, and J\. ZhouQwen3 embedding: advancing text embedding and reranking through foundation models\.arXiv preprint arXiv:2506\.05176\.External Links:[Link](https://arxiv.org/abs/2506.05176)Cited by:[Appendix C](https://arxiv.org/html/2609.12382#A3.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2609.12382#S1.SS0.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2609.12382#S3.SS2.p1.1),[§3\.3](https://arxiv.org/html/2609.12382#S3.SS3.SSS0.Px1.p3.1)\.
- Zhouet al\.\(2025\)C\. Zhou, C\. Yang, Y\. Hu, C\. Wang, C\. Zhang, M\. Zhang, L\. Mackey, T\. Jaakkola, S\. Bates, and D\. ZhangCoevolutionary continuous discrete diffusion: make your diffusion language model a latent reasoner\.External Links:2510\.03206,[Link](https://arxiv.org/abs/2510.03206)Cited by:[Appendix A](https://arxiv.org/html/2609.12382#A1.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2609.12382#S1.SS0.SSS0.Px1.p1.1)\.
- Zhuet al\.\(2025\)F\. Zhu, R\. Wang, S\. Nie, X\. Zhang, C\. Wu, J\. Hu, J\. Zhou, J\. Chen, Y\. Lin, J\. Wen,et al\.LLaDA 1\.5: variance\-reduced preference optimization for large language diffusion models\.arXiv preprint arXiv:2505\.19223\.Cited by:[§1](https://arxiv.org/html/2609.12382#S1.p2.1)\.

## Appendix ARelated Works

##### Discrete Diffusion Language Models\.

Discrete diffusion models generate text by reversing a corruption process defined directly on the discrete token space\. Early research generally falls into two paradigms: transition\-based frameworks and score\-based methods\. D3PM\([Austin et al\., 2021](https://arxiv.org/html/2609.12382#bib.bib5)\)pioneered the adaptation of continuous diffusion concepts to discrete spaces via transition matrices, which was later extended to continuous time using Markov chains\([Campbell et al\., 2022](https://arxiv.org/html/2609.12382#bib.bib46)\)\. Alternatively, inspired by score matching\([Song et al\., 2021](https://arxiv.org/html/2609.12382#bib.bib2)\), several works proposed discrete counterparts to the Stein score for modeling data distributions\([Meng et al\., 2022](https://arxiv.org/html/2609.12382#bib.bib47);[Lou et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib40)\)\. A dominant subclass within this domain is Masked Diffusion Models \(MDMs\), which treat the corruption process as token masking, starting from a masked sequence and refining tokens simultaneously using bidirectional context\. Subsequent studies\([Ou et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib39);[Sahoo et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib13);[Shi et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib21)\)demonstrated that simplified masking mechanisms can significantly enhance performance, effectively bridging the gap between diffusion and autoregressive models\. The iterative unmasking process inherent in MDMs supports sophisticated reasoning capabilities, such as iterative refinement\([Du et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib28)\)and reverse\-order reasoning\([Nie et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib11)\)\. The framework has also been integrated with chain\-of\-thought reasoning\([Ye et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib29)\), demonstrating strong performance in tasks requiring parallel context and systematic refinement\. Similar algorithms are proposed from the flow matching perspective\([Gat et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib15)\)\. In addition to masking noise, some works attempt to leverage uniform noise, though these tend to yield inferior performance\([von Rütte et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib34);[Shaul et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib16)\)\. Recently, MDMs have been scaled to large language models; for instance, LLaDA\([Nie et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib11)\)scales up to 8 billion parameters, showcasing reasoning capabilities previously unseen in non\-autoregressive models\. Furthermore, Dream\([Ye et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib12)\)introduces a training paradigm that initializes diffusion models with pretrained autoregressive weights, combining the strengths of both approaches\.

##### Continuous Diffusion Language Models\.

Continuous diffusion models \(CDMs\) reformulate text generation by performing diffusion in a continuous space, typically on word embeddings or logits\. Embedding\-based Diffusion Models \(EDMs\), such as Diffusion\-LM\([Li et al\., 2022](https://arxiv.org/html/2609.12382#bib.bib18)\), apply Gaussian diffusion to the continuous embeddings of discrete tokens\. This formulation naturally supports controllable generation and sequence\-to\-sequence tasks\([Dieleman et al\., 2022](https://arxiv.org/html/2609.12382#bib.bib30);[Mahabadi et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib31);[Gong et al\., 2023](https://arxiv.org/html/2609.12382#bib.bib19)\)\. Although early EDMs faced performance gaps compared to autoregressive models, Plaid\([Gulrajani and Hashimoto, 2023](https://arxiv.org/html/2609.12382#bib.bib32)\)established empirical scaling laws that significantly narrowed the efficiency gap with autoregressive models\. This framework was further extended by DoT\-Plaid\([Ye et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib29)\)to support chain\-of\-thought reasoning by leveraging iterative latent refinement\. Apart from embedding\-based methods, some approaches operate on the logit space\([Han et al\., 2023](https://arxiv.org/html/2609.12382#bib.bib20);[Sahoo et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib22)\)or explore multimodal integration\([Rojas et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib37);[Campbell et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib14)\)\. DUO\([Sahoo et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib22)\)attempts to connect two types of diffusion models via marginal matching and applies distillation tricks for continuous diffusion to discrete text diffusion\. While CDMs benefit from the well\-established theory of continuous diffusion, mapping the continuous latents back to discrete text remains a non\-trivial challenge that often requires specific regularization or rounding strategies\.

##### Latent\-Augmented Masked Diffusion\.

To enhance the expressivity of discrete diffusion, a growing body of work investigates integrating continuous latent variables or auxiliary processes into the discrete generation framework\. Several works have explored utilizing latent variable models to improve text modeling\([Bowman et al\., 2016](https://arxiv.org/html/2609.12382#bib.bib9);[Kaiser et al\., 2018](https://arxiv.org/html/2609.12382#bib.bib44)\)\. Recently,[Kong et al\. \(2025\)](https://arxiv.org/html/2609.12382#bib.bib41)used a latent variable structure for next\-token prediction in autoregressive models, optimized with variational Bayes\. In the context of diffusion,[Hayakawa et al\. \(2025\)](https://arxiv.org/html/2609.12382#bib.bib42)considered distilling pretrained MDMs with latent variables as the backward transition by optimizing consistency loss\. A complementary direction introduces continuous variables to capture joint structure along the diffusion path\. VADD\([Xie et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib6)\)associates a continuous latent with each reverse transition and trains a VAE\-style model to obtain non\-factorized posteriors\. Concurrent to our work, CCDD\([Zhou et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib7)\)jointly evolves continuous representations and discrete tokens during diffusion\. RMDM instead samples an aligned latent once and keeps it fixed while denoising the discrete sequence\. Thus, both methods use continuous representations to augment discrete diffusion, but assign them different roles in the sampling process\.

## Appendix BTheoretical Justifications

### B\.1Proof of Proposition[3\.1](https://arxiv.org/html/2609.12382#S3.Thmtheorem1)

For a set of indicesMM, write𝒙M=\{xi:i∈M\}\\bm\{x\}\_\{M\}=\\\{x\_\{i\}:i\\in M\\\}\. For any conditioning variableCC, define the conditional dependency gap

𝒯⁡\(𝒙M∣C\)=DKL​\(p⁡\(𝒙M∣C\)∥∏i∈Mp⁡\(xi∣C\)\),\\mathcal\{T\}\(\\bm\{x\}\_\{M\}\\mid C\)=D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\(\\bm\{x\}\_\{M\}\\mid C\)\\,\\middle\\\|\\,\\prod\_\{i\\in M\}p\(x\_\{i\}\\mid C\)\\right\),\(5\)where the conditional KL is averaged overCCwhenCCis random:

DKL​\(p⁡\(Y∣C\)∥q⁡\(Y∣C\)\)≜𝔼p⁡\(C\)​𝔼p⁡\(Y∣C\)​\[log⁡p⁡\(Y∣C\)q⁡\(Y∣C\)\]\.D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\(Y\\mid C\)\\,\\middle\\\|\\,q\(Y\\mid C\)\\right\)\\triangleq\\mathbb\{E\}\_\{p\(C\)\}\\mathbb\{E\}\_\{p\(Y\\mid C\)\}\\left\[\\log\\frac\{p\(Y\\mid C\)\}\{q\(Y\\mid C\)\}\\right\]\.\(6\)Equivalently,

𝒯⁡\(𝒙M∣C\)=∑i∈MH⁡\(xi∣C\)−H⁡\(𝒙M∣C\)\.\\mathcal\{T\}\(\\bm\{x\}\_\{M\}\\mid C\)=\\sum\_\{i\\in M\}H\(x\_\{i\}\\mid C\)\-H\(\\bm\{x\}\_\{M\}\\mid C\)\.\(7\)
##### Restatement of Proposition[3\.1](https://arxiv.org/html/2609.12382#S3.Thmtheorem1)\.

The residual gap𝒯⁡\(𝒙M∣𝒙U,𝒛\)\\mathcal\{T\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)is zero if and only if

p⁡\(𝒙M∣𝒙U,𝒛\)=∏i∈Mp⁡\(xi∣𝒙U,𝒛\)a\.s\.p\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)=\\prod\_\{i\\in M\}p\(x\_\{i\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\\quad\\text\{a\.s\.\}\(8\)Moreover,

Δ​𝒯≜𝒯⁡\(𝒙M∣𝒙U\)−𝒯⁡\(𝒙M∣𝒙U,𝒛\)=∑i∈MI⁡\(xi;𝒛∣𝒙U\)−I⁡\(𝒙M;𝒛∣𝒙U\)\.\\Delta\\mathcal\{T\}\\triangleq\\mathcal\{T\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\}\)\-\\mathcal\{T\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)=\\sum\_\{i\\in M\}I\(x\_\{i\};\\bm\{z\}\\mid\\bm\{x\}\_\{U\}\)\-I\(\\bm\{x\}\_\{M\};\\bm\{z\}\\mid\\bm\{x\}\_\{U\}\)\.\(9\)
###### Proof\.

The first claim follows directly from the non\-negativity of KL divergence:𝒯⁡\(𝒙M∣𝒙U,𝒛\)=0\\mathcal\{T\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)=0holds exactly when the two conditional distributions in its definition are equal almost surely, which is the factorization in \([8](https://arxiv.org/html/2609.12382#A2.E8)\)\.

For the second claim, apply \([7](https://arxiv.org/html/2609.12382#A2.E7)\) twice:

Δ​𝒯\\displaystyle\\Delta\\mathcal\{T\}=\[∑i∈MH⁡\(xi∣𝒙U\)−H⁡\(𝒙M∣𝒙U\)\]−\[∑i∈MH⁡\(xi∣𝒙U,𝒛\)−H⁡\(𝒙M∣𝒙U,𝒛\)\]\\displaystyle=\\left\[\\sum\_\{i\\in M\}H\(x\_\{i\}\\mid\\bm\{x\}\_\{U\}\)\-H\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\}\)\\right\]\-\\left\[\\sum\_\{i\\in M\}H\(x\_\{i\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\-H\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\\right\]=∑i∈M\[H⁡\(xi∣𝒙U\)−H⁡\(xi∣𝒙U,𝒛\)\]−\[H⁡\(𝒙M∣𝒙U\)−H⁡\(𝒙M∣𝒙U,𝒛\)\]\\displaystyle=\\sum\_\{i\\in M\}\\left\[H\(x\_\{i\}\\mid\\bm\{x\}\_\{U\}\)\-H\(x\_\{i\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\\right\]\-\\left\[H\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\}\)\-H\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\\right\]=∑i∈MI⁡\(xi;𝒛∣𝒙U\)−I⁡\(𝒙M;𝒛∣𝒙U\)\.\\displaystyle=\\sum\_\{i\\in M\}I\(x\_\{i\};\\bm\{z\}\\mid\\bm\{x\}\_\{U\}\)\-I\(\\bm\{x\}\_\{M\};\\bm\{z\}\\mid\\bm\{x\}\_\{U\}\)\.\(10\)∎

### B\.2Proof of Proposition[3\.2](https://arxiv.org/html/2609.12382#S3.Thmtheorem2)

We use the following factorization for the joint distribution induced by the encoder\-side latent variable:

pdata​\(𝒙M,𝒙U,𝒛\)\\displaystyle p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{M\},\\bm\{x\}\_\{U\},\\bm\{z\}\)=pϕ​\(𝒛\)​pdata​\(𝒙U∣𝒛\)​pdata​\(𝒙M∣𝒙U,𝒛\),\\displaystyle=p\_\{\\phi\}\(\\bm\{z\}\)\\,p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{U\}\\mid\\bm\{z\}\)\\,p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\),\(11\)pθ​\(𝒙M,𝒙U,𝒛\)\\displaystyle p\_\{\\theta\}\(\\bm\{x\}\_\{M\},\\bm\{x\}\_\{U\},\\bm\{z\}\)=p⁡\(𝒛\)​∏i∈Upθ​\(xi∣𝒛\)​∏i∈Mpθ​\(xi∣𝒙U,𝒛\)\.\\displaystyle=p\(\\bm\{z\}\)\\,\\prod\_\{i\\in U\}p\_\{\\theta\}\(x\_\{i\}\\mid\\bm\{z\}\)\\,\\prod\_\{i\\in M\}p\_\{\\theta\}\(x\_\{i\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\.\(12\)
##### Restatement of Proposition[3\.2](https://arxiv.org/html/2609.12382#S3.Thmtheorem2)\.

Under \([11](https://arxiv.org/html/2609.12382#A2.E11)\)–\([12](https://arxiv.org/html/2609.12382#A2.E12)\), the joint KL decomposes as

DKL​\(pdata​\(𝒙M,𝒙U,𝒛\)∥pθ​\(𝒙M,𝒙U,𝒛\)\)\\displaystyle D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{M\},\\bm\{x\}\_\{U\},\\bm\{z\}\)\\,\\middle\\\|\\,p\_\{\\theta\}\(\\bm\{x\}\_\{M\},\\bm\{x\}\_\{U\},\\bm\{z\}\)\\right\)\(13\)=\\displaystyle=DKL​\(pϕ​\(𝒛\)∥p⁡\(𝒛\)\)⏟\(I\) MeanFlow–prior mismatch\+𝒯⁡\(𝒙U∣𝒛\)\+𝒯⁡\(𝒙M∣𝒙U,𝒛\)⏟\(II\) residual conditional dependence\\displaystyle\\underbrace\{D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\phi\}\(\\bm\{z\}\)\\,\\middle\\\|\\,p\(\\bm\{z\}\)\\right\)\}\_\{\\text\{\(I\) MeanFlow\-\-prior mismatch\}\}\+\\underbrace\{\\mathcal\{T\}\(\\bm\{x\}\_\{U\}\\mid\\bm\{z\}\)\+\\mathcal\{T\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\}\_\{\\text\{\(II\) residual conditional dependence\}\}\+∑i∈UDKL​\(pdata​\(xi∣𝒛\)∥pθ​\(xi∣𝒛\)\)\+∑i∈MDKL​\(pdata​\(xi∣𝒙U,𝒛\)∥pθ​\(xi∣𝒙U,𝒛\)\)⏟\(III\) token\-wise prediction error,\\displaystyle\\quad\+\\underbrace\{\\sum\_\{i\\in U\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(x\_\{i\}\\mid\\bm\{z\}\)\\,\\middle\\\|\\,p\_\{\\theta\}\(x\_\{i\}\\mid\\bm\{z\}\)\\right\)\+\\sum\_\{i\\in M\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(x\_\{i\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\\,\\middle\\\|\\,p\_\{\\theta\}\(x\_\{i\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\\right\)\}\_\{\\text\{\(III\) token\-wise prediction error\}\},where the conditional KL terms are averaged over their conditioning variables\.

###### Proof\.

First, substituting the factorizations in \([11](https://arxiv.org/html/2609.12382#A2.E11)\)–\([12](https://arxiv.org/html/2609.12382#A2.E12)\) into the joint KL gives

DKL​\(pdata​\(𝒙M,𝒙U,𝒛\)∥pθ​\(𝒙M,𝒙U,𝒛\)\)\\displaystyle D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{M\},\\bm\{x\}\_\{U\},\\bm\{z\}\)\\,\\middle\\\|\\,p\_\{\\theta\}\(\\bm\{x\}\_\{M\},\\bm\{x\}\_\{U\},\\bm\{z\}\)\\right\)=𝔼pdata​\(𝒙M,𝒙U,𝒛\)​\[log⁡pϕ​\(𝒛\)​pdata​\(𝒙U∣𝒛\)​pdata​\(𝒙M∣𝒙U,𝒛\)p⁡\(𝒛\)​∏i∈Upθ​\(xi∣𝒛\)​∏i∈Mpθ​\(xi∣𝒙U,𝒛\)\]\.\\displaystyle=\\mathbb\{E\}\_\{p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{M\},\\bm\{x\}\_\{U\},\\bm\{z\}\)\}\\left\[\\log\\frac\{p\_\{\\phi\}\(\\bm\{z\}\)p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{U\}\\mid\\bm\{z\}\)p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\}\{p\(\\bm\{z\}\)\\prod\_\{i\\in U\}p\_\{\\theta\}\(x\_\{i\}\\mid\\bm\{z\}\)\\prod\_\{i\\in M\}p\_\{\\theta\}\(x\_\{i\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\}\\right\]\.\(14\)Separating the three log\-ratio terms yields

DKL​\(pdata​\(𝒙M,𝒙U,𝒛\)∥pθ​\(𝒙M,𝒙U,𝒛\)\)\\displaystyle D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{M\},\\bm\{x\}\_\{U\},\\bm\{z\}\)\\,\\middle\\\|\\,p\_\{\\theta\}\(\\bm\{x\}\_\{M\},\\bm\{x\}\_\{U\},\\bm\{z\}\)\\right\)=DKL​\(pϕ​\(𝒛\)∥p⁡\(𝒛\)\)\+DKL​\(pdata​\(𝒙U∣𝒛\)∥∏i∈Upθ​\(xi∣𝒛\)\)\\displaystyle=D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\phi\}\(\\bm\{z\}\)\\,\\middle\\\|\\,p\(\\bm\{z\}\)\\right\)\+D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{U\}\\mid\\bm\{z\}\)\\,\\middle\\\|\\,\\prod\_\{i\\in U\}p\_\{\\theta\}\(x\_\{i\}\\mid\\bm\{z\}\)\\right\)\+DKL​\(pdata​\(𝒙M∣𝒙U,𝒛\)∥∏i∈Mpθ​\(xi∣𝒙U,𝒛\)\)\.\\displaystyle\\quad\+D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\\,\\middle\\\|\\,\\prod\_\{i\\in M\}p\_\{\\theta\}\(x\_\{i\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\\right\)\.\(15\)
We next use the following identity to separate each conditional KL into a dependency term and token\-wise prediction terms\. For any index setAAand conditioning variableCC, all conditional KLs below follow the averaged definition in \([6](https://arxiv.org/html/2609.12382#A2.E6)\):

DKL​\(pdata​\(𝒙A∣C\)∥∏i∈Apθ​\(xi∣C\)\)\\displaystyle D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{A\}\\mid C\)\\,\\middle\\\|\\,\\prod\_\{i\\in A\}p\_\{\\theta\}\(x\_\{i\}\\mid C\)\\right\)=DKL​\(pdata​\(𝒙A∣C\)∥∏i∈Apdata​\(xi∣C\)\)\+∑i∈ADKL​\(pdata​\(xi∣C\)∥pθ​\(xi∣C\)\)\.\\displaystyle=D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{A\}\\mid C\)\\,\\middle\\\|\\,\\prod\_\{i\\in A\}p\_\{\\text\{data\}\}\(x\_\{i\}\\mid C\)\\right\)\+\\sum\_\{i\\in A\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(x\_\{i\}\\mid C\)\\,\\middle\\\|\\,p\_\{\\theta\}\(x\_\{i\}\\mid C\)\\right\)\.\(16\)To see this, start from the definition of KL and add and subtract∑i∈Alog⁡pdata​\(xi∣C\)\\sum\_\{i\\in A\}\\log p\_\{\\text\{data\}\}\(x\_\{i\}\\mid C\)inside the expectation:

𝔼p⁡\(C\)​𝔼pdata​\(𝒙A∣C\)​\[log⁡pdata​\(𝒙A∣C\)∏i∈Apθ​\(xi∣C\)\]\\displaystyle\\mathbb\{E\}\_\{p\(C\)\}\\mathbb\{E\}\_\{p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{A\}\\mid C\)\}\\left\[\\log\\frac\{p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{A\}\\mid C\)\}\{\\prod\_\{i\\in A\}p\_\{\\theta\}\(x\_\{i\}\\mid C\)\}\\right\]=𝔼p⁡\(C\)​𝔼pdata​\(𝒙A∣C\)​\[log⁡pdata​\(𝒙A∣C\)∏i∈Apdata​\(xi∣C\)\]\\displaystyle=\\mathbb\{E\}\_\{p\(C\)\}\\mathbb\{E\}\_\{p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{A\}\\mid C\)\}\\left\[\\log\\frac\{p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{A\}\\mid C\)\}\{\\prod\_\{i\\in A\}p\_\{\\text\{data\}\}\(x\_\{i\}\\mid C\)\}\\right\]\+𝔼p⁡\(C\)​𝔼pdata​\(𝒙A∣C\)​\[log⁡∏i∈Apdata​\(xi∣C\)∏i∈Apθ​\(xi∣C\)\]\\displaystyle\\quad\+\\mathbb\{E\}\_\{p\(C\)\}\\mathbb\{E\}\_\{p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{A\}\\mid C\)\}\\left\[\\log\\frac\{\\prod\_\{i\\in A\}p\_\{\\text\{data\}\}\(x\_\{i\}\\mid C\)\}\{\\prod\_\{i\\in A\}p\_\{\\theta\}\(x\_\{i\}\\mid C\)\}\\right\]=𝒯⁡\(𝒙A∣C\)\+∑i∈ADKL​\(pdata​\(xi∣C\)∥pθ​\(xi∣C\)\)\.\\displaystyle=\\mathcal\{T\}\(\\bm\{x\}\_\{A\}\\mid C\)\+\\sum\_\{i\\in A\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(x\_\{i\}\\mid C\)\\,\\middle\\\|\\,p\_\{\\theta\}\(x\_\{i\}\\mid C\)\\right\)\.\(17\)The last equality uses the definition of conditional total correlation for the first term and marginalizespdata​\(𝒙A∣C\)p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{A\}\\mid C\)topdata​\(xi∣C\)p\_\{\\text\{data\}\}\(x\_\{i\}\\mid C\)in each token\-wise term\.

Applying \([16](https://arxiv.org/html/2609.12382#A2.E16)\) with\(A,C\)=\(U,𝒛\)\(A,C\)=\(U,\\bm\{z\}\)gives

DKL​\(pdata​\(𝒙U∣𝒛\)∥∏i∈Upθ​\(xi∣𝒛\)\)\\displaystyle D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{U\}\\mid\\bm\{z\}\)\\,\\middle\\\|\\,\\prod\_\{i\\in U\}p\_\{\\theta\}\(x\_\{i\}\\mid\\bm\{z\}\)\\right\)=𝒯⁡\(𝒙U∣𝒛\)\+∑i∈UDKL​\(pdata​\(xi∣𝒛\)∥pθ​\(xi∣𝒛\)\),\\displaystyle=\\mathcal\{T\}\(\\bm\{x\}\_\{U\}\\mid\\bm\{z\}\)\+\\sum\_\{i\\in U\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(x\_\{i\}\\mid\\bm\{z\}\)\\,\\middle\\\|\\,p\_\{\\theta\}\(x\_\{i\}\\mid\\bm\{z\}\)\\right\),\(18\)where the right\-hand side is averaged over𝒛∼pϕ​\(𝒛\)\\bm\{z\}\\sim p\_\{\\phi\}\(\\bm\{z\}\)\. Applying the same identity with\(A,C\)=\(M,\(𝒙U,𝒛\)\)\(A,C\)=\(M,\(\\bm\{x\}\_\{U\},\\bm\{z\}\)\)gives

DKL​\(pdata​\(𝒙M∣𝒙U,𝒛\)∥∏i∈Mpθ​\(xi∣𝒙U,𝒛\)\)\\displaystyle D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\\,\\middle\\\|\\,\\prod\_\{i\\in M\}p\_\{\\theta\}\(x\_\{i\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\\right\)=𝒯⁡\(𝒙M∣𝒙U,𝒛\)\+∑i∈MDKL​\(pdata​\(xi∣𝒙U,𝒛\)∥pθ​\(xi∣𝒙U,𝒛\)\),\\displaystyle=\\mathcal\{T\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\+\\sum\_\{i\\in M\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(x\_\{i\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\\,\\middle\\\|\\,p\_\{\\theta\}\(x\_\{i\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\\right\),\(19\)where the right\-hand side is averaged over\(𝒙U,𝒛\)∼pdata​\(𝒙U,𝒛\)\(\\bm\{x\}\_\{U\},\\bm\{z\}\)\\sim p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{U\},\\bm\{z\}\)\.

Finally, substituting \([18](https://arxiv.org/html/2609.12382#A2.E18)\) and \([19](https://arxiv.org/html/2609.12382#A2.E19)\) into \([15](https://arxiv.org/html/2609.12382#A2.E15)\) yields

DKL​\(pdata​\(𝒙M,𝒙U,𝒛\)∥pθ​\(𝒙M,𝒙U,𝒛\)\)\\displaystyle D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(\\bm\{x\}\_\{M\},\\bm\{x\}\_\{U\},\\bm\{z\}\)\\,\\middle\\\|\\,p\_\{\\theta\}\(\\bm\{x\}\_\{M\},\\bm\{x\}\_\{U\},\\bm\{z\}\)\\right\)=\\displaystyle=DKL​\(pϕ​\(𝒛\)∥p⁡\(𝒛\)\)⏟\(I\) MeanFlow–prior mismatch\+𝒯⁡\(𝒙U∣𝒛\)\+𝒯⁡\(𝒙M∣𝒙U,𝒛\)⏟\(II\) residual conditional dependence\\displaystyle\\underbrace\{D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\phi\}\(\\bm\{z\}\)\\,\\middle\\\|\\,p\(\\bm\{z\}\)\\right\)\}\_\{\\text\{\(I\) MeanFlow\-\-prior mismatch\}\}\+\\underbrace\{\\mathcal\{T\}\(\\bm\{x\}\_\{U\}\\mid\\bm\{z\}\)\+\\mathcal\{T\}\(\\bm\{x\}\_\{M\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\}\_\{\\text\{\(II\) residual conditional dependence\}\}\+∑i∈UDKL​\(pdata​\(xi∣𝒛\)∥pθ​\(xi∣𝒛\)\)\+∑i∈MDKL​\(pdata​\(xi∣𝒙U,𝒛\)∥pθ​\(xi∣𝒙U,𝒛\)\)⏟\(III\) token\-wise prediction error,\\displaystyle\\quad\+\\underbrace\{\\sum\_\{i\\in U\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(x\_\{i\}\\mid\\bm\{z\}\)\\,\\middle\\\|\\,p\_\{\\theta\}\(x\_\{i\}\\mid\\bm\{z\}\)\\right\)\+\\sum\_\{i\\in M\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\text\{data\}\}\(x\_\{i\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\\,\\middle\\\|\\,p\_\{\\theta\}\(x\_\{i\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)\\right\)\}\_\{\\text\{\(III\) token\-wise prediction error\}\},which is Equation \([13](https://arxiv.org/html/2609.12382#A2.E13)\)\. ∎

## Appendix CExperimental Details

##### Data preprocessing\.

For LM1B, following[He et al\. \(2023\)](https://arxiv.org/html/2609.12382#bib.bib48);[Lou et al\. \(2024\)](https://arxiv.org/html/2609.12382#bib.bib40), we use the standard train/test split and tokenize the corpus with the BERT tokenizer\([Devlin et al\., 2019](https://arxiv.org/html/2609.12382#bib.bib36)\)\. We pad and truncate sequences to a fixed length ofN=128N=128\. For OWT, following[Lou et al\. \(2024\)](https://arxiv.org/html/2609.12382#bib.bib40);[Sahoo et al\. \(2024\)](https://arxiv.org/html/2609.12382#bib.bib13), we reserve the last 100K documents as a held\-out evaluation set\. To match the encoder used for OWT, we tokenize with the Qwen2 tokenizer and form sequences of lengthN=1024N=1024\.

##### Embedding models for the continuous latent\.

We instantiate the pretrained embedding modelgϕg\_\{\\phi\}differently across datasets to match their tokenization and domain\. On LM1B, we use BERT\-Base\([Devlin et al\., 2019](https://arxiv.org/html/2609.12382#bib.bib36)\)\. Because the BERT embedding is high\-dimensional, we split each 128\-token sequence into four groups, average embeddings within each group, and apply PCA whitening to obtain a compact 128\-dimensional representation from the original 768\-dimensional embeddings\. On OWT, we use Qwen3\-Embedding\-0\.6B\([Zhang et al\., 2025](https://arxiv.org/html/2609.12382#bib.bib57)\); because it supports flexible output dimensionality, we use the last\-layer embedding with dimension 32\.

##### Models and optimization\.

As described in Section[3\.4](https://arxiv.org/html/2609.12382#S3.SS4), both the discrete denoising backbone and the MeanFlow backbone are parameterized by a Diffusion Transformer \(DiT\)\([Peebles and Xie, 2023](https://arxiv.org/html/2609.12382#bib.bib50)\), following the architecture used in[Lou et al\. \(2024\)](https://arxiv.org/html/2609.12382#bib.bib40)\. We use 12 layers, hidden dimension 768, 12 attention heads, andσ=1\\sigma=1\. The latent dimensions areDz=128,dz=32D\_\{z\}=128,d\_\{z\}=32for LM1B andDz=32,dz=16D\_\{z\}=32,d\_\{z\}=16for OWT\. We follow the common diffusion\-LM training recipe\([Sahoo et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib13)\): AdamW with a constant learning rate of3×10−43\\times 10^\{\-4\}after 2\.5K warm\-up iterations, together with exponential moving average \(EMA\) decay 0\.9999\. For LM1B, we train for 1M iterations with batch size 512\. For OWT, we train for 50K iterations with batch size 512\. The MDLM and SEDD results reported on OWT are from our own retrained baselines, using the same Qwen2 tokenizer, data preprocessing, model size, batch size, optimizer, learning\-rate schedule, EMA setting, and 50K\-step training budget as RMDM\.

##### Encoder sensitivity\.

We replace Qwen3\-Embedding\-0\.6B with ModernBERT\-large\([Warner et al\., 2024](https://arxiv.org/html/2609.12382#bib.bib51)\)while keeping the downstream architecture and latent interface fixed\. ModernBERT supports the full OWT sequence length, and we reduce its 1024\-dimensional outputs to 32 dimensions with PCA\. Because this variant converges more slowly, we compare it with an MDLM baseline trained for the same approximately 60K\-step budget; the Qwen3 results use the 50K\-step models from the main experiment\. Tables[6](https://arxiv.org/html/2609.12382#A3.T6)and[7](https://arxiv.org/html/2609.12382#A3.T7)show that both encoder variants outperform their matched baselines across sampling budgets in terms of GenPPL and LLM\-judge scores, although the improvement is smaller with ModernBERT\. This result suggests that the benefit is not specific to one encoder family, while its magnitude depends on the representation source\.

Table 6:Encoder sensitivity on OWT measured by GenPPL \(↓\\downarrow\)\. Each RMDM variant is compared with the MDLM baseline at the corresponding training budget\.Table 7:Encoder sensitivity on OWT measured by LLM\-judge overall\-quality score \(↑\\uparrow; mean±\\pm95% confidence\-interval half\-width\)\. Each RMDM variant is compared with the MDLM baseline at the corresponding training budget\.

## Appendix DLLM\-judge Evaluation

We use an LLM judge to complement GenPPL because evaluator perplexity is only a proxy for generation quality and can miss aspects such as discourse coherence, degenerate repetition, and natural readability\. Following this motivation, each generated passage is judged as a standalone sample\. We use DeepSeek\-V4\-Flash\([DeepSeek\-AI, 2026](https://arxiv.org/html/2609.12382#bib.bib26)\)as the primary judge model and report the averageoverallscore in the main tables\. We additionally evaluate the same samples with Gemini\-3\.1\-Flash\-Lite\([Google, 2026](https://arxiv.org/html/2609.12382#bib.bib27)\)as an independent judge\. For each method and sampling budget, both judges evaluate all 256 generated passages\. Tables[8](https://arxiv.org/html/2609.12382#A4.T8)and[9](https://arxiv.org/html/2609.12382#A4.T9)report mean scores with 95% confidence intervals\. The agreement in relative rankings indicates that the main trend is not specific to a single judge\. The judge is asked to return minified JSON with five 1–10 scores:fluency,coherence,repetition,readability, andoverall\.

Table 8:DeepSeek\-V4\-Flash overall\-quality scores \(mean±\\pm95% confidence\-interval half\-width;n=256n=256per cell\)\.Table 9:Gemini\-3\.1\-Flash\-Lite overall\-quality scores \(mean±\\pm95% confidence\-interval half\-width;n=256n=256per cell\)\.##### Prompt\.

> Please act as an impartial judge and evaluate the quality of the generated text displayed below\.\.\. Scoring rubric, each from 1 to 10: \- fluency: grammar, phrasing, and local readability \- coherence: logical flow, topic consistency, and sentence\-to\-sentence continuity \- repetition: avoidance of loops, duplicate ideas, and degenerate repetition \- readability: how natural and human\-readable the passage feels overall \- overall: your final holistic judgment Instructions: \- Judge the text as a standalone passage\. \- Do not assume hidden context beyond the text itself\. \- Penalize malformed special tokens, obvious hallucinated structure, or text that reads like corrupted sampling\. \- Do not reward verbosity\. Return ONLY valid minified JSON\.\.\. Generated text: <<<TEXT \{text\} TEXT\>\>\>

## Appendix EInference Efficiency

We measure sampling efficiency on one NVIDIA A800 80GB GPU with the DDPM sampler, batch size 8, and sequence length 1024, after warm\-up\. RMDM and MDLM share the same DiT backbone; RMDM additionally conditions each block on a latent sampled once from the prior\. The pretrained encoder and MeanFlow network are not invoked during inference\. RMDM uses 2\.72 GB of weight memory and 18\.93 GB peak sampling memory, compared with 2\.43 GB and 18\.64 GB for MDLM\.

Table 10:Generation quality and measured throughput at matched sampling\-step budgets\.At the same step count, RMDM retains approximately 90% of MDLM’s throughput\. At matched quality, however, RMDM requires roughly one quarter as many steps: for example, GenPPL≤50\\leq 50is reached by MDLM at 1024 steps \(80 tok/s\) and by RMDM at 256 steps \(287 tok/s\), corresponding to a3\.6×3\.6\\timesthroughput improvement\. Similar comparisons give3\.53\.5–3\.6×3\.6\\timesimprovements across the quality range reported in Table[4](https://arxiv.org/html/2609.12382#S4.T4)\.

## Appendix FTraining Cost

##### Wall\-clock training time\.

We trained the models on 4 NVIDIA A800 GPUs\. In our current implementation, the MDLM baseline requires approximately 3 days and 5 hours for 50K training steps, while the RMDM decoder training stage requires approximately 5 days for the same number of steps\. This measured overhead mainly comes from our uncached implementation: at each training step, we compute the pretrained encoder representation online and then map it through the MeanFlow\-aligned latent module before conditioning the decoder\. Since the training corpus is fixed, these encoder representations can in principle be precomputed and cached, which would remove most of this repeated computation\. We did not use such caching in our reported runs due to storage constraints\.

##### MeanFlow alignment stage\.

The additional latent\-alignment stage is comparatively lightweight\. It is trained for 10K steps and takes approximately 1 day and 15 hours under the same hardware setting\. This number should also be interpreted as the cost of the uncached implementation, since online encoder representation extraction contributes to the measured time\. Overall, the wall\-clock times above report our implementation cost rather than an inherent training\-time lower bound of RMDM\.

## Appendix GExamples

### G\.1Detailed Numerical Example with Gaussian Distribution

We provide a concrete analytical example using Gaussian distributions that strictly follows the causal graph in Figure[2](https://arxiv.org/html/2609.12382#S3.F2.fig1)\. This example demonstrates how the latent variable𝒛\\bm\{z\}explains correlations between masked tokens that the partial context𝒙U\\bm\{x\}\_\{U\}cannot capture\.

##### Model Setup\.

Consider a linear Gaussian structural equation model where the global latent𝒛\\bm\{z\}, the observed context𝒙U\\bm\{x\}\_\{U\}, and the masked tokensxM1,xM2x\_\{M\}^\{1\},x\_\{M\}^\{2\}are scalar random variables:

𝒛\\displaystyle\\bm\{z\}∼𝒩⁡\(0,1\),\\displaystyle\\sim\\mathcal\{N\}\(0,1\),\(20\)𝒙U\\displaystyle\\bm\{x\}\_\{U\}=𝒛\+ϵU,\\displaystyle=\\bm\{z\}\+\\epsilon\_\{U\},\(21\)xM1\\displaystyle x\_\{M\}^\{1\}=𝒛\+α​𝒙U\+ϵ1,\\displaystyle=\\bm\{z\}\+\\alpha\\bm\{x\}\_\{U\}\+\\epsilon\_\{1\},\(22\)xM2\\displaystyle x\_\{M\}^\{2\}=𝒛\+β​𝒙U\+ϵ2,\\displaystyle=\\bm\{z\}\+\\beta\\bm\{x\}\_\{U\}\+\\epsilon\_\{2\},\(23\)whereϵU,ϵ1,ϵ2∼𝒩⁡\(0,σ2\)\\epsilon\_\{U\},\\epsilon\_\{1\},\\epsilon\_\{2\}\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}\)are independent noise terms\. This structure matches the causal graph where𝒛\\bm\{z\}acts as a confounder influencing all variables, while𝒙U\\bm\{x\}\_\{U\}has direct edges \(controlled byα\\alphaandβ\\beta\) to the masked tokens\.

##### Conditional Dependency Without Latent\.

When𝒛\\bm\{z\}is unobserved \(standard non\-latent modeling\), we condition only on𝒙U\\bm\{x\}\_\{U\}\. Although𝒙U\\bm\{x\}\_\{U\}provides partial information about𝒛\\bm\{z\}, it does not fully recover it\. The posterior distribution is:

p⁡\(𝒛∣𝒙U\)=𝒩⁡\(μz\|u,σz\|u2\),where​σz\|u2=Var​\(𝒛∣𝒙U\)=σ21\+σ2\.p\(\\bm\{z\}\\mid\\bm\{x\}\_\{U\}\)=\\mathcal\{N\}\(\\mu\_\{z\|u\},\\sigma^\{2\}\_\{z\|u\}\),\\quad\\text\{where \}\\sigma^\{2\}\_\{z\|u\}=\\text\{Var\}\(\\bm\{z\}\\mid\\bm\{x\}\_\{U\}\)=\\frac\{\\sigma^\{2\}\}\{1\+\\sigma^\{2\}\}\.\(24\)Since𝒛\\bm\{z\}remains uncertain \(varianceσz\|u2\>0\\sigma^\{2\}\_\{z\|u\}\>0\), it acts as a common noise source inducing correlation betweenxM1x\_\{M\}^\{1\}andxM2x\_\{M\}^\{2\}\. Given𝒙U\\bm\{x\}\_\{U\}, the termsα​𝒙U\\alpha\\bm\{x\}\_\{U\}andβ​𝒙U\\beta\\bm\{x\}\_\{U\}are constant, so the conditional covariance is:

Cov​\(xM1,xM2∣𝒙U\)=Var​\(𝒛∣𝒙U\)=σ21\+σ2\.\\text\{Cov\}\(x\_\{M\}^\{1\},x\_\{M\}^\{2\}\\mid\\bm\{x\}\_\{U\}\)=\\text\{Var\}\(\\bm\{z\}\\mid\\bm\{x\}\_\{U\}\)=\\frac\{\\sigma^\{2\}\}\{1\+\\sigma^\{2\}\}\.\(25\)
Settingσ=1\\sigma=1for concreteness, we compute:

Var​\(𝒛∣𝒙U\)\\displaystyle\\text\{Var\}\(\\bm\{z\}\\mid\\bm\{x\}\_\{U\}\)=12,\\displaystyle=\\frac\{1\}\{2\},\(26\)Cov​\(xM1,xM2∣𝒙U\)\\displaystyle\\text\{Cov\}\(x\_\{M\}^\{1\},x\_\{M\}^\{2\}\\mid\\bm\{x\}\_\{U\}\)=12,\\displaystyle=\\frac\{1\}\{2\},\(27\)Var​\(xMi∣𝒙U\)\\displaystyle\\text\{Var\}\(x\_\{M\}^\{i\}\\mid\\bm\{x\}\_\{U\}\)=Var​\(𝒛∣𝒙U\)\+Var​\(ϵi\)=12\+1=32,\\displaystyle=\\text\{Var\}\(\\bm\{z\}\\mid\\bm\{x\}\_\{U\}\)\+\\text\{Var\}\(\\epsilon\_\{i\}\)=\\frac\{1\}\{2\}\+1=\\frac\{3\}\{2\},\(28\)ρ\\displaystyle\\rho=Cov​\(xM1,xM2∣𝒙U\)Var​\(xM1∣𝒙U\)​Var​\(xM2∣𝒙U\)=1/23/2=13\.\\displaystyle=\\frac\{\\text\{Cov\}\(x\_\{M\}^\{1\},x\_\{M\}^\{2\}\\mid\\bm\{x\}\_\{U\}\)\}\{\\sqrt\{\\text\{Var\}\(x\_\{M\}^\{1\}\\mid\\bm\{x\}\_\{U\}\)\\text\{Var\}\(x\_\{M\}^\{2\}\\mid\\bm\{x\}\_\{U\}\)\}\}=\\frac\{1/2\}\{3/2\}=\\frac\{1\}\{3\}\.\(29\)For Gaussian variables, the dependency gap \(mutual information\) is:

𝒯⁡\(xM1,xM2∣𝒙U\)=−12​ln⁡\(1−ρ2\)=−12​ln⁡\(1−19\)\>0\.\\mathcal\{T\}\(x\_\{M\}^\{1\},x\_\{M\}^\{2\}\\mid\\bm\{x\}\_\{U\}\)=\-\\frac\{1\}\{2\}\\ln\(1\-\\rho^\{2\}\)=\-\\frac\{1\}\{2\}\\ln\\left\(1\-\\frac\{1\}\{9\}\\right\)\>0\.\(30\)This positive value quantifies the information loss incurred by assuming independence in parallel sampling\.

##### Conditional Independence With Latent\.

In our RMDM framework, we condition on both𝒙U\\bm\{x\}\_\{U\}and the sampled latent𝒛\\bm\{z\}\. When𝒛\\bm\{z\}is fixed, the only remaining randomness comes from the independent noise termsϵ1,ϵ2\\epsilon\_\{1\},\\epsilon\_\{2\}\. Therefore:

Cov\(xM1,xM2∣𝒙U,𝒛\)=𝔼\[ϵ1ϵ2\]=0\.\\text\{Cov\}\(x\_\{M\}^\{1\},x\_\{M\}^\{2\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)=\\mathbb\{E\}\[\\epsilon\_\{1\}\\epsilon\_\{2\}\]=0\.\(31\)The masked tokens become conditionally independent:p\(xM1,xM2∣𝒙U,𝒛\)=p\(xM1∣𝒙U,𝒛\)p\(xM2∣𝒙U,𝒛\)p\(x\_\{M\}^\{1\},x\_\{M\}^\{2\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)=p\(x\_\{M\}^\{1\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)p\(x\_\{M\}^\{2\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\), and the conditional dependency gap vanishes:𝒯\(xM1,xM2∣𝒙U,𝒛\)=0\\mathcal\{T\}\(x\_\{M\}^\{1\},x\_\{M\}^\{2\}\\mid\\bm\{x\}\_\{U\},\\bm\{z\}\)=0\.

##### Summary\.

By introducing𝒛\\bm\{z\}, we reduce the residual correlation fromσ2/\(1\+σ2\)\\sigma^\{2\}/\(1\+\\sigma^\{2\}\)to00\. The latent variable “explains away” the common fluctuation caused by the unobserved global context, making the independence assumption valid for parallel generation\.

### G\.2Generated Samples on OpenWebText

We provide generated samples on OpenWebText at four sampling budgets\.

#### G\.2\.11024 Sampling Steps

<\|im\_end\|\>interest rates\.

He said that overall, not including employment, they show lower interest rates than inflation in comparison to a higher level for older people\.

While many people are indebted, many have been given a credit line unsupported on home ownership as the UK rose\.

It is crucial to look at in real terms\. In 2008 the pound hit back on the economy’s growth, which was more than it was in 2007\. Back in 2008 was £204,000 with the average adjusted pay rate of 62%\. It is important to bear to recognise, that much of the rise in the debt level was mainly due to a tightness of inflation, which came through deflation\. However, as a result, we saw an increase in real value\.

Inflation dragged the pound to a broader level\.

He was forecasting the Federal Reserve Bank’s strength in April\. More than a month earlier, for the same period in April, the real debt level was about 20% — approaching 27% while the UK Government had experienced 28%\.

The implication, seems the government will raise debt levels less than a month earlier\. However, while rates were only 23\.04 in the UK on 19 July it reflects a reduction in mortgages for technical contracts, which it would expect between November and November\.

An additional element of confidence level will start to be added to in new contracts, including requiring more purchases of a property to sell as an asset\.

Mr Cherry said: "For the most recent year we experienced a rapid depreciation\."

Stability of interest

He said that the higher rates are likely due to a slightly higher interest level, but leaves the possibility of employment over a wider period of the government’s debt policy, without the potential for economic growth\.

"Given the urgency level of the UK Government to start action on the other side to affect economic activity, given this context, I think there is a temporary start of more widespread economic activity\."

The latest quarterly report suggests interest rates between 5\.3% and 4\.1%, though that the current "stable" funding status means, when interest rates continued to fall, the level of wages, which was large of the economy at this time and how much real pay kept falling\.

This sector is subject to a high level of joblessness, with real interest rates relating to real asset holders\.

Mr Cherry added that the UK economy has become almost certain, and with the coming boom with conditions hitting hard when it came to accelerating economic growth, the UK Government’s overall impact on employment continued to be constrained by the nominal interest rate of 5 per cent, in 2011 and an estimated rate of wage increases in 1978, 1991, 1904 and 1978, to the current January levels for February 2015\.

Real employment keeps falling

While the UK Government’s nominal interest rate of 12 per cent fell in two years at 5\.1 and down 4\.8%, there was a stand\-off in the late 2000s when the biggest drag on employment was lower employment worldwide\.

Yehy: "As compared with surveys of low employment, in large parts, that is understood that debt levels, or interest rates, steadily declined in 2007\."

He said from the figures "the average real interest rate in 2014 was 7\.4 and up 5 percent, indicating that most of us now have a higher level of employment\."

He also said the higher balance sheets may have helped force students into spending more on student loans, thus increasing their income\. Recent figures are particularly worrying as the UK ranks 36pc of students\.

ADAPT

"A lot of young people are under this state who are losing employment \- a number that is for the next parliament, I must say, yet\.

"It has been devastating \- who is living here with little hope or peace\."

Yehy said that despite the overall rise in employment activity, between the first nine months last year the UK government was by 11 per cent, compared with 42 among four months over in a decade, then up by three or four years\.

"This has been encouraging for lots of people in England and Ireland, in terms of employment, for the foreseeable future\."

Yehy is concerned because of 27 unemployment extensions under which led to further job increases and surveys showing UK government alone experienced rising unemployment\.

Unemployment is responsible for individual jobs\-related economic losses as employment is a small part of people losing employment," he said\.

"Sometimes when I hear people talk about unemployment extension cuts, they see \- I can say, ’The cuts worked\!"

’I don’t think they’re the best alternative to long\-term unemployment’

"So I think people are a little unhappy\. Some may sometimes have less savings than others\.

"I think in terms of benefits, maybe it doesn’t produce results, but I am told they will<\|im\_end\|\>

#### G\.2\.2512 Sampling Steps

<\|im\_end\|\>\. He had one of the last systematic trips to the Rocky Mountains and Hollywood\. Two years later, one of the first to be America’s reporter was to be on Arizona’s Ellulia Island, Hawaii\. He brings us three stories from Camp Everest, when The Post introduced him as a reporter in January\. He had no plans for hosting the program\.

Frank Marousy, we arrived in La Mesa, on the Rocky Mountain, from the Grand Canyon\. And I’m twenty years old and camped in La Mesa\. In high school, I started to have a child, and got a job\. And I thought I was fun\.

Authorities wanted me to know your story\. I wondered why you’re in the news when you had an interview with one former Indonesian journalist, the role of scientists on the changing carbon chain\.

Simon found it hard to dig up a series of problems that she needed us to continue her activities as she told The Guardian on where her job and priorities came up at the time\.

“Could you explain a little bit about the challenges that have been in the laboratory for years? Well, this is what the IPCC was focusing on\. Energy, and they had identified these problems, were the only challenges,” Mr\. Pauler asked two former editors of The Guardian and The Guardian\.

We detailed on discussion of the available water resources on climate change\. The Prime Minister reported to Malta and asked Mr\. Simon as he followed up on climate negotiations, and asked what information he learned on the issues of climate change, he did ask about broader insights into seeing press releases on sites: “I have seen your interviews\. has there was anything controversial?

“Mr\. Pauler said, “There was information about climate change\. Facebook, we were doing the research\. I’m an ecologist at Stanford University\. I’m on tour\. So I only got the information when I was overseas\. Catching news drew me on there; which, consequently, allowed me to learn about climate changes more slowly and more widely\. And I see the addresses of some very prominent working leaders and others about how much it has influenced her because around the globe it inspired me\. I had to support the prime minister, to support the national crisis in Cambodia, … to Beijing’s climate policies, Frank Marousy,” told The Guardian\. We also spoke with Mr\. Pauler about the new climate plan\. They both agreed last week that they would hold meetings with Mr\. Simon after a special session\. Mr\. Pauler taped a long video clip on Wednesday in the reporters’ office when he and fellow reporters discussed climate change on the scientists panel that is now organising his annual retreat\.

Mr\. Simon sat in a room with Mr\. Pauler and she talked about addressing the financial difficulties focused on reducing emissions or helping their farmers defend themselves to frictions higher than their European ones\. He also said, “It’s important to make sacrifices to be made from American food stocks, as well as our environment, from American food corporations — that they would protect themselves from food hazards and also have financial protection against food tariffs, because American farmers would be supported by their compensation for \[security\]\.”

Mr\. Simon said, “To me, something that you say is consistent with my understanding of going to the energy policy being on how we should be reducing carbon pollution, she said\. I think many of the kind of priorities that we need, there — we need to prioritize\.”

He also added, “I would say we’re going to be so vulnerable because of the fact that there are already climate change and economic impacts out there mentioned\. But that’s what my answer: Let’s make this country look better, so the health of our planet, not just the carbon\-neutral environment but the health of people, and where the climate change is, that we need to start that change of climate change for everybody,” Mr\. Simon told us\.

“At global levels, when we added carbon emissions in the way that we’re pushing for than climate change, the science and energy policy is being de\-recorded and used\.

We have released some numbers for the last two years, Mr\. Simon said\. And the results of the new MGH report show that Australia is planning to reduce greenhouse gas emissions by a global level of 22\.5 percent by 2050 from the 2010 current levels\. They are also trying to limit emissions from the order of two percent in 2060 to zero by 2070, according to this aim of the UN report\.

If the global economy was going to achieve the least 8\.2 percent of 2050 in the amount of CO2 emissions to nations, doing so instead is immoral and immoral\. I don’t think it should — even if it’s federal policy — ” he said\. “That is impossible\.” I like to think people say that we’re not getting to ourselves right now\.

“The success of this international study is that there are very specific, measurable differences between these countries,” Mr\. Simon said, referring to the provincial government carbon<\|im\_end\|\>

#### G\.2\.3256 Sampling Steps

<\|im\_end\|\>people who knew the TV footage provided by the deputies\. About 20 police officers took the show of a water truck photographed, and saw television crews right the other side of the fire and fire cars\.

One of it later came from Philadelphia fire departments\.

“That would be a deputy chief officer, and then he got to his car,” said Philadelphia Police Sarah at the television station\. Afterward, once he got closer, he looked at the “area of the Sheriff’s works, and it happened\.”

“He said, What do you want to happen, John?”

“We have four trucks\. We have five ones\. I’m concerned about that because it’s different\. And part of the plan was that they wanted the drivers to look at cars\.”

Now with most firefighters, deputies responded, “You can’t identify any anybody\.” Instead, local reporters tweeted out stories about a picture that sparked outrage\. With local reporters like CNN and CBS broadcast wires to display their images depicting a Philadelphia flag as their own trucks, social media consolidated\.

200 of the Philadelphia residents were angry that BPD officers were trying to plot the images into firefighters\.

“There have been a lot of reactions from those guys who know that in these things, it seems more sinister,” said Abby Yates from Berryessa\. Talking about it, “It’s convincing that they have something that they want because you’re there to see what’s being captured and that you will be right to go\.”

On Facebook, Yates observed that there was a big quick reaction from the Philadelphia fire\. It wasn’t a typical Philadelphia response\.

The Telegraph reported some local concussion video snippets from the Harrisburg interview of the fire cars\. The following link came from AT&T:

“We you saw is that fire car customers have to pay for rent for immediate repairs,” Yates said\.

Here is what the statement was: “If you see the same thing in a lot of a firefighter’s cars, would be some of them will have a life” to recease\.

Cornell 71’s Thomas Miller said that he was one of the journalists in Philadelphia to work the panoramic video\.

“All reporters are also interviewed by the Hall of Justice,” he said\. “Look at technology and better, look at the cameras and then it’s different\.” Miller also said that he was willing and determined to make such a video\.

Jokes describing a toic reporter as well as a YouTube climber, Miller described to CNN, “it was really cool, this is really a way to get this video to happen to first responders of Division 5/6 of Philadelphia\.”

"How did you make the video happen? Why this stuff?" Phillips asked\. “Do you think there was responsible equipment with equipment and people,” he answered\. "Do you support the America Foundation for Change and donate so much and have your support and help?"<\|im\_end\|\>Most of the races got started in Emanuel’s campaign for assistant Mayor Ped Pederson \(where here are Philadelphia’s 18 editors\) and last year was Eric Leverdehan, the owner of the newspaper, the owner of the Press and the owner of Wilson, who paid him huge amounts of money to help fund\.

But in November, it started in earnest and went ahead\. And it wasn’t for single parent’s children where the figure started growing in in the midly\-1950s and ’70s suburban side; it was the four other children of Debbie Leverdehan, who allegedly expressed a love for Ryan Schultz, in much of Sandy’s warding\.

The story happened shortly Debbie was 69 when her mother and her brother died recently in battling cancer from her family at the Center section of events\. Sandy’s aunt Kyle Schultz was visiting Rhode Island for a weekend before Sandy’s dad died aged 61\. Two of his sisters, Julia Owens, and Wayne Owens, had passed years from, and Lee, and Christine, Kelly\. They first came here last Saturday from First Avenue in the Mapleland, and now four others in Cyslewood, Long Beach, when she was young five\. And as soon as last was Debbie’s trip, Debbie said of her son Sandy’s getting to know she loved having Tim, Lee Schultz\. She was a little beautiful\. How was it? Debbie didn’t even know how much\. It was a heroic move because it was a beautiful moment\.

"A big family had really cried for the mayor and Christine when she came on going\. And this has happened from Kyle’s dad to come the whole other way\. In fact, it was actually on Debbie’s end\." Sandy’s daughter, Johanna Schultz and the Julia Owens family, was again turned blue Friday by her father Dick, who predicted won again be running like her mother thought they were\.

"I’ve got situations like this on here\. There’s a candidate that doesn’t allow racial discrimination, I wouldn’t say they’ll get to this point," Debbie\.

Other kids will be rallying against the plan, for the night of Sunday’s quarter and a half season\.

Photo: AP

Sunday 17\-28<\|im\_end\|\>

#### G\.2\.4128 Sampling Steps

<\|im\_end\|\>We also expect about two or 10 volunteers who will introduce Bitcoin to our community\.

As the first of those who write about Bitcoin there, the Canadian Bitcoin Foundation is an example — it is actually a bit of a silver roof to keep the money flowing in…

Bitcoin “root” represents an army of households that will feed them electricity and help us convert their income into urban and military — we will spread wealth all over the network\.

Ensure that, check out a way to use Bitcoin\.info and a blog post\.

Create a Bitcoin cant repository and have a regular Bitcoin account\.

Morgan’s Bricket Update

Before any new Bitcoin software is built in accordance with the initial materials we are going to start our for this purpose\. Every Bitcoin node is measured on a character propagation and, eventually, we will happen a few things before it is all rolled on to the user\. Bitcoin blocks is a fundamental specification with many types and multiple days\., when you design the protocol, with this example, we would recommend that the Bitcoin block finish over time to do so\. Then, that at the beginning of your BTC block development from Bitcoin Blocks is not interrupted by a 3rd fork\. Bitcoin blocks can then be used as interrupters, even if the type has a format or to develop them as a way to …read; the propagation of the Bitcoin blocks was not the way to handle transactions, and would be used by a miner\. Based on what last in the above description, we still also recommend putting one or two spikes of a few thousand times throughout all of your BTC block\. Try timed the “bittleneck” of the following block before the next transaction and change to 30 minutes a time if you want to get down to these transactions – 10 minutes – 30 as a BTC block\. Design of these updates will be presented in the ends – use of the timestamp for Bitcoin Blocks for a broadcast at the end of the/block, resulting in a reverting process until the block is to complete in the preceding months\.

We argue a lot against this structure, Bitcoin Bitcoinism, which mentioned that a certain type of Bitcoin enthusiast is likely to actively participate in Bitcoin developments\. However, we believe these types of projects should be more of the same as before: to allow the community to understand the hard forks that may occur in the current Bitcoin code, and expect to be frequently replaced with more forks in the future\.

It makes sense that the Bitcoin developers and the Bitcoin community would like to make our development easier, and we will try and not released them just so that it gets more sense\. Before proceeding by saying that without a hard fork you may want to add another version or another version of Bitcoin, we caution you not to recognize the prominence of changes, it can take some time\. The Bitcoin patchending has been updated and in the last 6 days, and our community will be working on this extension closely with the Bitcoin project\. But as the fact of this, note of the state of the Bitcoin concept will help to change as to how Bitcoin should go about functions\. For those who reach the Bitcoin community, this extension will help the Bitcoin community understand the future of Bitcoin, and their own future\.

Although it will take into account details of the current standard of development Bitcoin Vision\. In the Bitcoin community, as well as the Light Architecture’s development and the community expertise in Bitcoin Bitcoin development, we recognize that this is a high priority for the Bitcoin community, and hence, it is more important to provide support for others in our community\. Therefore, as a result of the Light Architecture’s development, we organize the Bitcoin Community workshop for conversations about the standard of development Bitcoin Vision\.

By the way,

The early developers of the Bitcoin Society also received informal feedback from a moderator on our subreddit, called btc, that preceded the discussions\. Participants received various changes in their project, some changes were random, and so often came from the design and implementation of the current Bitcoin development software, the Bitcoin Developers posted a thread to explain how they proposal it rather than tell us that there are currently no added versions\.

Coin Editions on the Internet

You are select the first set from the basic process to use the Bitcoin project: on the page of the Bitcoin development website, you will select the version to send back to you on the Bitcoin publication\. This page contains relevant information from the Bitcoin Republic, with links to draft papers, papers which will be gathered in the format below: Select the current version of their foundation; for those who submitted their project, they can click you to download a PDF copy, and they will select the complete version of the foundation\. If you downloading a copy agreed to pay a fee, strip the rest with it and then pay for their work\.

Once the Bitcoin development is in the run, you will click on our website to download a stock picture, you will find one of the four sets: either one will the development project use on hand for the infrastructure, the other set for the distribution, you will also know a digital supply unit and you will add one other code to the<\|im\_end\|\>