Support Before Frequency in Discrete Diffusion

# Support Before Frequency in Discrete Diffusion
Source: [https://arxiv.org/html/2605.13999](https://arxiv.org/html/2605.13999)
1Department of Computer ScienceETH ZürichSwitzerland2Institute of MathematicsEPFLSwitzerland3Institute of Statistical ScienceAcademia SinicaTaiwan

###### Abstract

Discrete diffusion models are increasingly competitive for language modeling, yet it remains unclear how their denoising objectives organize learning\. Although these objectives target the full data distribution, we show that the exact reverse process induces a hierarchy between coarse*support*information and finer*frequency*information\. For uniform and absorbing \(a\.k\.a\. masking\) diffusion, we prove that, in the small\-noise regime of the final denoising steps, each single\-token reverse edit decomposes into a leading scale, determined by whether it moves toward the data*support*\(e\.g\., grammatically valid sentences\), and a finer coefficient, determining*relative probabilities*within the same scale\. Thus, recovering validity structure only requires learning the correct order of magnitude of reverse probabilities, whereas recovering data frequencies requires coefficient\-level estimation\. The separation is mechanism\-dependent: uniform diffusion exhibits a trichotomy into validity\-improving, validity\-preserving, and validity\-worsening edits, while absorbing diffusion places its leading\-order mass on validity\-improving moves\. Experiments on a masked language diffusion model and synthetic regular\-language tasks support these predictions: support\-localization emerges earlier than within\-support frequency ranking, and the contrast between uniform and absorbing diffusion matches the predicted rate separation\. Together, our results suggest that discrete diffusion models learn data support before data frequencies\.

### 1Introduction

Since the influential work ofaustin2021structured, discrete Diffusion Language Models \(DLMs\) have gained significant traction as a promising alternative to auto\-regressive models due to their potential for faster inference through parallelization\(austin2021structured;wu2025fast;arriola2025block;israel2025accelerating\)and improved controllability of generation\(li2022diffusion\)\. These advantages have driven rapid progress in DLM design and scaling, with recent models beginning to close the gap with state\-of\-the\-art autoregressive language models\(googledeepmind2025geminidiffusion;labs2025mercury;song2025seed\); seeli2025surveyfor a survey\.

##### Central hypothesis\.

Most successful DLMs rely on reversing either*uniform*or*absorbing*\(masking\) diffusion processes\(austin2021structured\)\. While a large body of work studies how to parameterize, train, and sample from these models, our focus is different: we askwhat structure of the data distribution is exposed first by the reverse denoising problem\. Our central thesis is the following\.

Support\-before\-Frequency Hypothesis:DLMs first learn where admissible sequences are, and only later refine the relative probabilities among admissible sequences\.

Concretely, writeD≔supp​\(pdata\)D\\coloneqq\\text\{supp\}\(p\_\{\\text\{data\}\}\)for the support of the population data distribution, i\.e\., the set of sequences with nonzero probability underpdatap\_\{\\text\{data\}\}\. Intuitively,DDis the set of admissible sequences, such as context\-free languages for programming code generation, while the frequencies are the probabilitiespdata​\(x\)p\_\{\\text\{data\}\}\(x\)assigned to stringsx∈Dx\\in D\. In these terms, our hypothesis says that DLMs first recover support information aboutDD, in a sense made precise below, before they accurately calibrate the probabilitiespdata​\(x\)p\_\{\\text\{data\}\}\(x\)withinDD\.

##### Small\-noise expansion of DLMs\.

Our hypothesis is motivated by a small\-noise analysis of the*exact*reverse kernel, corresponding to the final denoising steps of the diffusion process\. This kernel is the population object targeted by standard discrete diffusion training objectives\(austin2021structured\)\. Thus, its low\-noise structure may expose an information hierarchy shared across training paradigms that approximate the true reverse process\.

For a stringxx, letddbe the Hamming distanced​\(x,z\)≔\|\{i:xi≠zi\}\|d\(x,z\)\\coloneqq\|\\\{i:x\_\{i\}\\neq z\_\{i\}\\\}\|, and letprojD​\(x\)≔\{z∈D:d​\(x,z\)=d​\(x,D\)\}\\text\{proj\}\_\{D\}\(x\)\\coloneqq\\\{z\\in D:d\(x,z\)=d\(x,D\)\\\}denote the set of nearest in\-support strings toxx\. Letσ\\sigmabe the noise level, and consider the low\-noise limitσ→0\+\\sigma\\to 0^\{\+\}\. For a one\-token reverse edit from a current string to a candidate string, our main theorems \([Theorems˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmtheorem1)and[2\.2](https://arxiv.org/html/2605.13999#S2.Thmtheorem2)\) show that the reverse edit probability has the schematic form \(up to normalization\)

reverse edit probability≈Γ𝖼𝗈𝗋𝗋​\(Δ​d\)⏟corruption gate⋅σΔ​d⏟scale: support signal⋅pdata​\(projD​\(candidate\)\)pdata​\(projD​\(current\)\)⏟coefficient: frequency signal,\\text\{reverse edit probability\}\\;\\approx\\;\\underbrace\{\\phantom\{\\frac\{j\_\{D\}\}\{j\_\{D\}\}\}\\Gamma\_\{\\mathsf\{corr\}\}\(\\Delta d\)\\phantom\{\\frac\{1\}\{1\}\}\}\_\{\\text\{corruption gate\}\}\\cdot\\underbrace\{\\phantom\{\\frac\{j\_\{D\}\}\{j\_\{D\}\}\}\\sigma^\{\\Delta d\}\\phantom\{\\frac\{1\}\{1\}\}\}\_\{\\text\{\{scale\}: support signal\}\}\\cdot\\underbrace\{\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(\\text\{candidate\}\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(\\text\{current\}\)\)\}\}\_\{\\text\{\{coefficient\}: frequency signal\}\},\(⋆\\star\)whereΔ​d=d​\(candidate,D\)−d​\(current,D\)\\Delta d=d\(\\text\{candidate\},D\)\-d\(\\text\{current\},D\), and the factorΓ𝖼𝗈𝗋𝗋\\Gamma\_\{\\mathsf\{corr\}\}records the effect of the corruption mechanism:

Γ𝖼𝗈𝗋𝗋​\(Δ​d\)≔\{1,𝖼𝗈𝗋𝗋=𝗎𝗇𝗂𝖿,𝟏​\{Δ​d=−1\},𝖼𝗈𝗋𝗋=𝗆𝖺𝗌𝗄\.\\Gamma\_\{\\mathsf\{corr\}\}\(\\Delta d\)\\coloneqq\\begin\{cases\}1,&\\mathsf\{corr\}=\\mathsf\{unif\},\\\\\[2\.84526pt\] \\mathbf\{1\}\\\{\\Delta d=\-1\\\},&\\mathsf\{corr\}=\\mathsf\{mask\}\.\\end\{cases\}
At first sight, \([⋆\\star‣1](https://arxiv.org/html/2605.13999#S1.Ex1)\) appears to mix support and frequency information in a single leading expression\. The key point, however, is that*these two pieces enter at different resolutions, and that the corruption mechanism determines which scales are active through the gate functionΓcorr\\Gamma\_\{\\mathrm\{corr\}\}\.*This leads to the two concrete predictions about practical DLMs, which we test in both synthetic and real\-data experiments\.

1. 1\.Prediction 1: support before frequency\.At low noise, recovering*support\-improving directions*only requires the model to learn the roughorder\-of\-magnitudeinσ\\sigmaof the corresponding denoising proposal, to determine whether it moves towards the support\. Asσ→0\\sigma\\to 0, these different orders become increasingly separated, making the support structure easier to detect from coarse scaling information alone\. By contrast, recovering the*frequencies*of the data distribution requires accurate estimation of thecoefficientswithin a fixed order of magnitude\. Thus, a trained model may acquire a support\-like denoising field before it matches fine within\-support frequencies\. We make this intuition precise in[Corollary˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmcorollary1): if the learned reverse probabilities approximate the true reverse probabilities up to multiplicative erroro​\(σ−1/2\),o\(\\sigma^\{\-1/2\}\),then thresholding the scaled reverse probabilities exactly recovers the support\-improving directions\. Importantly, this condition can hold even when the learned transition kernel remains far from the true kernel in additive metrics such as total variation or KL; see[Example˜B\.1](https://arxiv.org/html/2605.13999#A2.Thmexample1)for an illustration\.
2. 2\.Prediction 2: masking is closer to a support projector\.The same expansion predicts a qualitative contrast between corruption mechanisms\. For*uniform diffusion*,support\-improving,support\-preserving, andsupport\-worseningedits appear at three distinct scales: σ−𝟏,𝟏,σ\.\{\\mathbf\{\\color\[rgb\]\{0\.078125,0\.48828125,0\.2734375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.078125,0\.48828125,0\.2734375\}\\sigma^\{\-1\}\}\},\\mathbf\{\{\\color\[rgb\]\{0\.37109375,0\.37109375,0\.37109375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.37109375,0\.37109375,0\.37109375\}\\hskip 22\.99988pt1\}\},\\qquad\{\\mathbf\{\\color\[rgb\]\{0\.70703125,0\.21484375,0\.17578125\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.70703125,0\.21484375,0\.17578125\}\\sigma\}\}\.For*absorbing*, or*masked*, diffusion, the structure is sharper: only support\-improving unmasking moves contribute at leading order, while non\-improving moves vanish in the small\-noise limit\.

![Refer to caption](https://arxiv.org/html/2605.13999v1/x1.png)Figure 1:Support before frequencies in a web\-trained masked DLM\.We train a masked DLM on FineWeb and evaluate support and frequency proxies inspired by the separation in[Theorem˜2\.2](https://arxiv.org/html/2605.13999#S2.Thmtheorem2)\([Section˜3\.1](https://arxiv.org/html/2605.13999#S3.SS1)\)\. The support\-localization proxy reaches its peak gain earlier than the frequency\-ranking proxies\. Curves show means over three seeds with±1\\pm 1standard\-deviation bands; transition markers use the first checkpoint reaching90%90\\%of each seed’s peak gain\.[Section˜3\.1](https://arxiv.org/html/2605.13999#S3.SS1)testsPrediction 1by designing probes that disentangle*support localization*from*frequency ranking*for a masked DLM trained on FineWeb; see[Figure˜1](https://arxiv.org/html/2605.13999#S1.F1)\. This setting captures a widely used DLM family: an absorbing\-mask process trained with a weighted cross\-entropy objective, which coincides with several common formulations, including D3PM\-style posterior prediction, SUBS/mean parameterizations, and the induced\-score view; see[Appendix˜A](https://arxiv.org/html/2605.13999#A1)\. Across seeds, the support\-localization probe reaches its peak gain substantially earlier than the frequency\-ranking probes\. This temporal separation supports our hypothesis: although the training objective can in principle learn support identification and frequency refinement simultaneously, support identification emerges and stabilizes before the model refines relative probabilities within the support\.

[Section˜3\.2](https://arxiv.org/html/2605.13999#S3.SS2)testsPrediction 2, namely the predicted contrast between*uniform*and*absorbing*diffusion\. To isolate the projection effect, we use controlled synthetic experiments on*regular languages*, where membership in the data support can be evaluated exactly\. We then apply a theory\-guided*inference\-time thresholding*procedure, requiring no additional training, to isolate the leadingσ−1\\sigma^\{\-1\}\-scale component of the learned reverse scores\. As predicted, this intervention improves support recovery for uniform diffusion, but gives little additional benefit for absorbing diffusion\. This matches the exact reverse\-kernel expansion: absorbing diffusion already suppresses non\-projective moves at leading order, whereas uniform diffusion retains lower\-scale non\-projective components that can be removed by isolating the dominant scale\. Finally, we stress that our purpose is to highlight the distinct structural biases in the reverse dynamics, rather than endorsing one corruption process over the other\. While absorbing diffusion is closer to a support projector at small noise, masking also limits sampling flexibility by preventing incorrectly unmasked tokens from being revised\.

##### Contributions\.

To summarize, our main contributions are threefold\.

First, in[Theorems˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmtheorem1)and[2\.2](https://arxiv.org/html/2605.13999#S2.Thmtheorem2), we derive small\-noise expansions of the exact reverse kernels for both*uniform*and*absorbing*diffusion\. These expansions identify a scale separation between support and frequency information, and reveal a sharp contrast between the two corruption mechanisms\.

Second, in[Corollary˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmcorollary1), we formalize why coarse multiplicative accuracy of the reverse scores is sufficient to recover support\-improving directions, even when the learned reverse kernel remains far from the true kernel in additive metrics such as total variation\.

Third, in[Section˜3](https://arxiv.org/html/2605.13999#S3), we design probes for support\-localization and frequency\-ranking inspired by the theory, and show that trained DLMs exhibit the predicted separations in both real\-data and synthetic experiments\.

##### Relation to prior work\.

The conceptually closest perspective to our hypothesis is the recent observation that continuous score matching may recover geometric information about a data manifold before learning the full density on that manifold\(li2025scores\)\. Our work provides a discrete analogue for discrete spaces, without assuming any latent manifold structure: the ambient geometry is instead the Hamming graph, and the analogue of the manifold is the arbitrary support setsupp​\(pdata\)\\text\{supp\}\(p\_\{\\text\{data\}\}\)\.

### 2Small\-Noise Expansion of the Reverse Process Generator

This section formalizes the aforementioned small\-noise expansion \([⋆\\star‣1](https://arxiv.org/html/2605.13999#S1.Ex1)\) of the exact reverse kernel\.

#### 2\.1Recap of Uniform and Masking Diffusions

We begin by recalling the standard discrete\-time formulation of diffusion models on finite state spaces, followingaustin2021structured\. Our focus is on the two corruption mechanisms most relevant to diffusion language modeling:*uniform diffusion*and*absorbing*\(a\.k\.a\.*masking*\) diffusion\.111Similar small\-noise expansions can be derived for other discrete diffusions, with the Hamming distance replaced by the appropriate shortest\-path distance on the underlying diffusion graph\. We focus on uniform and absorbing diffusion because they capture the main design choices used in current text diffusion models\.

##### General setup\.

We consider a finite token space\[K\]≔\{1,…,K\}\[K\]\\coloneqq\\\{1,\\dots,K\\\}, a fixed sequence length \(dimension\)HH, and the corresponding set of all possible sequences𝒳≔\[K\]H\\mathcal\{X\}\\coloneqq\[K\]^\{H\}\. We denote the time horizon of the diffusion process byTT, meaning that generation is done in aTT\-step denoising procedure with time discretization1/T1/T\. We consider forward diffusion processes that are independent across token positions, given via a starting discrete distributionq0≔pdata∈Δ𝒳q\_\{0\}\\coloneqq p\_\{\\text\{data\}\}\\in\\Delta\_\{\\mathcal\{X\}\}on the simplex and the Markov kernelsqt\|t−1​\(xt\|xt−1\)=∏i=1Hqt\|t−1i​\(xti\|xt−1i\)=∏i=1H\[Qt\]xt−1i,xtiq\_\{t\|t\-1\}\(x\_\{t\}\|x\_\{t\-1\}\)=\\prod\_\{i=1\}^\{H\}q\_\{t\|t\-1\}^\{i\}\(x\_\{t\}^\{i\}\|x\_\{t\-1\}^\{i\}\)=\\prod\_\{i=1\}^\{H\}\[Q^\{t\}\]\_\{x\_\{t\-1\}^\{i\},x\_\{t\}^\{i\}\}\(t∈\[T\]t\\in\[T\]\) for fixed transition matricesQt∈ℝK×ℝKQ^\{t\}\\in\\mathbb\{R\}^\{K\}\\times\\mathbb\{R\}^\{K\}\. Fort∈\[T\]t\\in\[T\], we letXt≔\(Xt1,…,XtH\)∈𝒳X\_\{t\}\\coloneqq\(X\_\{t\}^\{1\},\\dots,X\_\{t\}^\{H\}\)\\in\\mathcal\{X\}denote the random variable this process determines\. We abbreviateqt​\(xt\)≔ℙ​\[Xt=xt\]q\_\{t\}\(x\_\{t\}\)\\coloneqq\\mathbb\{P\}\[X\_\{t\}=x\_\{t\}\]andqt\|s​\(xt\|xs\)≔ℙ​\[Xt=xt\|Xs=xs\]q\_\{t\|s\}\(x\_\{t\}\|x\_\{s\}\)\\coloneqq\\mathbb\{P\}\[X\_\{t\}=x\_\{t\}\|X\_\{s\}=x\_\{s\}\]for allxt,xs∈\[K\]x\_\{t\},x\_\{s\}\\in\[K\]such thatℙ​\[Xs=xs\]\>0\\mathbb\{P\}\[X\_\{s\}=x\_\{s\}\]\>0\.

##### Uniform diffusion\.

For a fixedq0=pdataq\_\{0\}=p\_\{\\text\{data\}\}, consider the uniform forward diffusion process given by

qt\|t−1i​\(y\|x\)≔\(1−βt\)​𝟙​\{y=x\}\+βtK\\displaystyle q\_\{t\|t\-1\}^\{i\}\(y\|x\)\\coloneqq\(1\-\\beta\_\{t\}\)\\mathbbm\{1\}\\left\\\{y=x\\right\\\}\+\\frac\{\\beta\_\{t\}\}\{K\}\(uniform\)for alli∈\[H\]i\\in\[H\],x,y∈\[K\]x,y\\in\[K\]\.

##### Absorbing \(masking\) diffusion\.

We derive analogous results for the absorbing diffusion process, which is similarly given byq0=pdataq\_\{0\}=p\_\{\\text\{data\}\}and

qt\|t−1i​\(y\|x\)≔\(1−βt\)​𝟙​\{y=x\}\+βt​𝟙​\{y=m\}\\displaystyle q\_\{t\|t\-1\}^\{i\}\(y\|x\)\\coloneqq\(1\-\\beta\_\{t\}\)\\mathbbm\{1\}\\left\\\{y=x\\right\\\}\+\\beta\_\{t\}\\mathbbm\{1\}\\left\\\{y=m\\right\\\}\(absorbing\)for alli∈\[H\]i\\in\[H\],x,y∈\[K\]x,y\\in\[K\], wheremmis a special masking token not contained in\[K\]\[K\]\.

When setting the*cumulative noise level*at time steptt\(probability of being noised before or at timett\) to beσt≔1−∏s=1t\(1−βs\)\\sigma\_\{t\}\\coloneqq 1\-\\prod\_\{s=1\}^\{t\}\(1\-\\beta\_\{s\}\), we can easily view the induced marginals for token indexiias

uniform:qti=\\displaystyle\\text\{uniform:\}\\quad q^\{i\}\_\{t\}=\(1−σt\)​pdatai\+σt​punif,absorbing:qti=\\displaystyle\(1\-\\sigma\_\{t\}\)p\_\{\\text\{data\}\}^\{i\}\+\\sigma\_\{t\}p\_\{\\text\{unif\}\},\\qquad\\qquad\\text\{absorbing:\}\\quad q^\{i\}\_\{t\}=\(1−σt\)​pdatai\+σt​pmask,\\displaystyle\(1\-\\sigma\_\{t\}\)p\_\{\\text\{data\}\}^\{i\}\+\\sigma\_\{t\}p\_\{\\text\{mask\}\},

##### Fitting the reverse kernel\.

The way standard DLMs work is to fit the*reverse transition*qt−1\|tq\_\{t\-1\|t\}with a modelpt−1\|tp\_\{t\-1\|t\}by minimizing the classical variational bound on the data likelihood \(see[Appendix˜A](https://arxiv.org/html/2605.13999#A1)\)\. This model is then used to stepwise denoise, starting from the source distributionxT∼punif⊗Hx\_\{T\}\\sim p\_\{\\text\{unif\}\}^\{\\otimes H\}\(resp\.xT∼pmask⊗Hx\_\{T\}\\sim p\_\{\\text\{mask\}\}^\{\\otimes H\}\) viaxt−1∼pt−1\|t\(⋅\|xt\)x\_\{t\-1\}\\sim p\_\{t\-1\|t\}\(\\cdot\|x\_\{t\}\)\. In practice,pt−1\|tp\_\{t\-1\|t\}is virtually always modeled as independent across token positions, meaning thatpt−1\|t​\(y\|xt\)=∏h=1Hpt−1\|th​\(yh\|xt\)p\_\{t\-1\|t\}\(y\|x\_\{t\}\)=\\prod\_\{h=1\}^\{H\}p\_\{t\-1\|t\}^\{h\}\(y^\{h\}\|x\_\{t\}\)and each component on the right\-hand side is parameterized by a neural network\.

#### 2\.2Main Results: Small\-Noise Expansion of Reverse Kernels

We now derive the small\-noise expansions of the exact reverse kernels previewed in the introduction\. To make the results applicable across different discretization schemes and noise schedules, we formulate the analysis under a general condition that captures the*continuous\-noise limit*underlying discrete diffusion: any fixed target noise level can be approximated arbitrarily well, while each individual denoising step becomes infinitesimal\. This condition is mild and is satisfied by standard schedules used in practice, including linear cumulative\-noise and cosine schedules\.

###### Assumption 2\.1\(Noise schedule\)\.

Let\(σt\)t∈\[T\]\(\\sigma\_\{t\}\)\_\{t\\in\[T\]\}be the cumulative noise levels induced by\(βt\)t∈\[T\]\(\\beta\_\{t\}\)\_\{t\\in\[T\]\}\. For every fixedσ∈\(0,1\)\\sigma\\in\(0,1\), let

tσ​\(T\)∈arg​mins∈\[T\]⁡\|σs−σ\|t\_\{\\sigma\}\(T\)\\in\\operatorname\*\{arg\\,min\}\_\{s\\in\[T\]\}\|\\sigma\_\{s\}\-\\sigma\|\(tσt\_\{\\sigma\}\)be a time index whose noise level is closest toσ\\sigma\. We assume thatσtσ​\(T\)→σ,βtσ​\(T\)→0\\sigma\_\{t\_\{\\sigma\}\(T\)\}\\to\\sigma,\\beta\_\{t\_\{\\sigma\}\(T\)\}\\to 0asT→∞T\\to\\infty\.

Forx∈𝒳x\\in\\mathcal\{X\}, a coordinateh∈\[H\]h\\in\[H\], and a proposed tokenyh∈\[K\]y^\{h\}\\in\[K\], writexh→yh≔\(x−h,yh\)x^\{h\\to y^\{h\}\}\\coloneqq\(x^\{\-h\},y^\{h\}\)for the string obtained by replacing thehh\-th token ofxxbyyhy^\{h\}\. Recall thatdddenotes the Hamming distance and

projD​\(x\)≔\{z∈D:d​\(x,z\)=d​\(x,D\)\},pdata​\(projD​\(x\)\)≔∑z∈projD​\(x\)pdata​\(z\)\.\\text\{proj\}\_\{D\}\(x\)\\coloneqq\\\{z\\in D:d\(x,z\)=d\(x,D\)\\\},\\qquad p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\\coloneqq\\sum\_\{z\\in\\text\{proj\}\_\{D\}\(x\)\}p\_\{\\text\{data\}\}\(z\)\.
##### Small\-noise expansion: Uniform diffusion\.

We are now ready to state the main theoretical results\. The first result shows that, for uniform diffusion, one\-token reverse edits split into*three*scale classes\.

###### Theorem 2\.1\(Rate separation for uniform diffusion\)\.

Letqqdenote the law of the uniform diffusion process\. Fix a statext∈supp​\(qt\)=𝒳x\_\{t\}\\in\\text\{supp\}\(q\_\{t\}\)=\\mathcal\{X\}, a coordinateh∈\[H\]h\\in\[H\], and a proposed replacement tokenyh∈\[K\]∖\{xth\}y^\{h\}\\in\[K\]\\setminus\\\{x\_\{t\}^\{h\}\\\}\. Define the corresponding one\-token edit byxt−1≔xth→yh\.x\_\{t\-1\}\\coloneqq x\_\{t\}^\{h\\to y^\{h\}\}\.Letσ∈\(0,1\)\\sigma\\in\(0,1\), and lett=tσ​\(T\)t=t\_\{\\sigma\}\(T\)be the time index specified in \([tσt\_\{\\sigma\}](https://arxiv.org/html/2605.13999#S2.Ex7)\)\. Then, under[˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmassumption1), asσ→0\\sigma\\to 0,

limT→∞qt−1\|th​\(yh∣xt\)βt/K=\{\(σK\)−1​pdata​\(projD​\(xt−1\)\)pdata​\(projD​\(xt\)\)\+O​\(1\),toward supportd​\(xt−1,D\)<d​\(xt,D\),pdata​\(projD​\(xt−1\)\)pdata​\(projD​\(xt\)\)\+O​\(σ\),same distanced​\(xt−1,D\)=d​\(xt,D\),\(σK\)​pdata​\(projD​\(xt−1\)\)pdata​\(projD​\(xt\)\)\+O​\(σ2\),away from supportd​\(xt−1,D\)\>d​\(xt,D\)\.\\lim\_\{T\\to\\infty\}\\frac\{q^\{h\}\_\{t\-1\|t\}\(y^\{h\}\\mid x\_\{t\}\)\}\{\\beta\_\{t\}/K\}=\\begin\{cases\}\\displaystyle\\left\(\\frac\{\\sigma\}\{K\}\\right\)^\{\-1\}\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\_\{t\-1\}\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\_\{t\}\)\)\}\+O\(1\),&\\begin\{gathered\}\\text\{\{\\color\[rgb\]\{0\.078125,0\.48828125,0\.2734375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.078125,0\.48828125,0\.2734375\}\{toward support\}\}\}\\\\ d\(x\_\{t\-1\},D\)<d\(x\_\{t\},D\),\\end\{gathered\}\\\\\[15\.00002pt\] \\displaystyle\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\_\{t\-1\}\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\_\{t\}\)\)\}\+O\(\\sigma\),&\\begin\{gathered\}\\text\{\{\\color\[rgb\]\{0\.37109375,0\.37109375,0\.37109375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.37109375,0\.37109375,0\.37109375\}\{same distance\}\}\}\\\\ d\(x\_\{t\-1\},D\)=d\(x\_\{t\},D\),\\end\{gathered\}\\\\\[15\.00002pt\] \\displaystyle\\left\(\\frac\{\\sigma\}\{K\}\\right\)\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\_\{t\-1\}\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\_\{t\}\)\)\}\+O\(\\sigma^\{2\}\),&\\begin\{gathered\}\\text\{\{\\color\[rgb\]\{0\.70703125,0\.21484375,0\.17578125\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.70703125,0\.21484375,0\.17578125\}\{away from support\}\}\}\\\\ d\(x\_\{t\-1\},D\)\>d\(x\_\{t\},D\)\.\\end\{gathered\}\\end\{cases\}

Equivalently, up to fixed vocabulary factors and lower\-order terms, the normalized one\-token reverse probability satisfies

qt−1\|th​\(yh∣x\)βt/K=Θ​\(σd​\(xh→yh,D\)−d​\(x,D\)​pdata​\(projD​\(xh→yh\)\)pdata​\(projD​\(x\)\)\)\.\\frac\{q^\{h\}\_\{t\-1\|t\}\(y^\{h\}\\mid x\)\}\{\\beta\_\{t\}/K\}=\\Theta\\left\(\\sigma^\{d\(x^\{h\\to y^\{h\}\},D\)\-d\(x,D\)\}\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x^\{h\\to y^\{h\}\}\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\right\)\.This expression separates the reverse edit probability into two conceptually different components\. The exponent

Δ​d≔d​\(xh→yh,D\)−d​\(x,D\)\\Delta d\\coloneqq d\(x^\{h\\to y^\{h\}\},D\)\-d\(x,D\)is the*support signal*: it indicates whether the proposed edit moves the current string closer to the supportDD, keeps it at the same distance, or moves it further away\. By contrast, the ratio

pdata​\(projD​\(xh→yh\)\)pdata​\(projD​\(x\)\)\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x^\{h\\to y^\{h\}\}\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}is a*frequency\-sensitive coefficient*: once the noise scaleσ\\sigmaof the edit is fixed, this coefficient determines how probability is distributed among edits of the same scale\.

Consequently, in the small\-noise regime, the reverse kernel has a clear*three\-scale*structure: Support\-improving edits, for whichΔ​d=−1\\Delta d=\-1, appear at scaleσ−1\\sigma^\{\-1\}; distance\-preserving edits, for whichΔ​d=0\\Delta d=0, appear at scale11; and support\-worsening edits, for whichΔ​d=1\\Delta d=1, appear at scaleσ\\sigma\. Thus the dominant entries of the low\-noise reverse kernel are precisely the local edits that move the string toward the data support\.

##### Small\-noise expansion: Absorbing diffusion\.

Our next goal is to show that the same*support\-seeking*principle takes an even sharper form for absorbing, or masking, diffusion\. At a masked coordinate, a reverse move can only replace the mask by a clean token\. Since the mask token does not appear in any string in the data supportDD, such a move can never increase the Hamming distance toDD\. Thus the uniform\-diffusion trichotomy collapses: reverse moves are either support\-improving or support\-preserving\. The next result shows that, in the small\-noise limit, only the support\-improving moves survive at leading order\.

###### Theorem 2\.2\(Rate separation for absorbing diffusion\)\.

Letqqbe the law of the absorbing diffusion process\. In the same setting as[Theorem˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmtheorem1), but withxth=mx\_\{t\}^\{h\}=ma masked token,yh∈\[K\]y^\{h\}\\in\[K\], andxt∈supp​\(qt\)x\_\{t\}\\in\\text\{supp\}\(q\_\{t\}\), we have, asσ→0\\sigma\\to 0,

limT→∞qt−1\|th​\(yh∣xt\)βt=\{σ−1​pdata​\(projD​\(xt−1\)\)pdata​\(projD​\(xt\)\)\+O​\(1\),support\-improvingd​\(xt−1,D\)<d​\(xt,D\),0,non\-improvingd​\(xt−1,D\)=d​\(xt,D\)\.\\lim\_\{T\\to\\infty\}\\frac\{q^\{h\}\_\{t\-1\|t\}\(y^\{h\}\\mid x\_\{t\}\)\}\{\\beta\_\{t\}\}=\\begin\{cases\}\\displaystyle\\sigma^\{\-1\}\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\_\{t\-1\}\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\_\{t\}\)\)\}\+O\(1\),&\\begin\{gathered\}\\text\{\{\\color\[rgb\]\{0\.078125,0\.48828125,0\.2734375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.078125,0\.48828125,0\.2734375\}\{support\-improving\}\}\}\\\\ d\(x\_\{t\-1\},D\)<d\(x\_\{t\},D\),\\end\{gathered\}\\\\\[15\.00002pt\] 0,&\\begin\{gathered\}\\text\{\{\\color\[rgb\]\{0\.37109375,0\.37109375,0\.37109375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.37109375,0\.37109375,0\.37109375\}\{non\-improving\}\}\}\\\\ d\(x\_\{t\-1\},D\)=d\(x\_\{t\},D\)\.\\end\{gathered\}\\end\{cases\}

This gives the formal contrast with uniform diffusion\. In uniform diffusion, the low\-noise reverse kernel contains a dominant support\-projecting component, but also retains lower\-scale non\-projective components\. In absorbing diffusion, the reverse graph is sharper: after normalization, the leading\-order mass is carried*entirely by support\-improving unmasking moves*, while non\-projective moves are suppressed in the small\-noise limit\.

##### Connection with discrete scores\.

We remark that the normalizations in[Theorems˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmtheorem1)and[2\.2](https://arxiv.org/html/2605.13999#S2.Thmtheorem2)remove the baseline forward proposal rate for a single\-token edit:βt/K\\beta\_\{t\}/Kfor uniform diffusion andβt\\beta\_\{t\}for absorbing diffusion\. This is the natural scaling in the continuous\-time limit\. Indeed, for uniform diffusion,

qt−1\|th​\(yh∣x\)βt/K⟶qt​\(xh→yh\)qt​\(x\),\\frac\{q^\{h\}\_\{t\-1\|t\}\(y^\{h\}\\mid x\)\}\{\\beta\_\{t\}/K\}\\;\\longrightarrow\\;\\frac\{q\_\{t\}\(x^\{h\\to y^\{h\}\}\)\}\{q\_\{t\}\(x\)\},\(1\)and the absorbing case has the analogous limit withβt/K\\beta\_\{t\}/Kreplaced byβt\\beta\_\{t\}\.

The right\-hand side of \([1](https://arxiv.org/html/2605.13999#S2.E1)\) has a standard interpretation: it is the discrete analogue of a score, often called a*concrete score*\(meng2022concrete;sun2022score;lou2023discrete\)\. Thus,[Theorems˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmtheorem1)and[2\.2](https://arxiv.org/html/2605.13999#S2.Thmtheorem2)can also be read as small\-noise expansions of the true discrete score field\. In this view, the order of magnitude of the score identifies support\-improving directions, whereas the within\-order coefficients encode frequency information\.

#### 2\.3Implication: Support Recovery from Coarsely Trained DLMs

[Theorems˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmtheorem1)and[2\.2](https://arxiv.org/html/2605.13999#S2.Thmtheorem2)show that, after the continuous\-time limitT→∞T\\to\\infty, support information appears in the*order of magnitude*of the normalized reverse scores, whereas frequency information appears only in the coefficient within a fixed order\. We now formalize the consequence that coarse multiplicative accuracy is sufficient to recover support\-improving directions\. For uniform diffusion, define the limiting normalized scores222A finite\-TTanalogue of[Corollary2\.1](https://arxiv.org/html/2605.13999#S2.Thmcorollary1)below follows from the more general estimates in[TheoremB\.1](https://arxiv.org/html/2605.13999#A2.Thmtheorem1); we state the limiting case for simplicity\.

S¯qh​\(yh∣x;σ\)≔limT→∞qtσ​\(T\)−1∣tσ​\(T\)h​\(yh∣x\)βtσ​\(T\)/K,S¯ph​\(yh∣x;σ\)≔limT→∞ptσ​\(T\)−1∣tσ​\(T\)h​\(yh∣x\)βtσ​\(T\)/K,\\bar\{S\}\_\{q\}^\{h\}\(y^\{h\}\\mid x;\\sigma\)\\coloneqq\\lim\_\{T\\to\\infty\}\\frac\{q^\{h\}\_\{t\_\{\\sigma\}\(T\)\-1\\mid t\_\{\\sigma\}\(T\)\}\(y^\{h\}\\mid x\)\}\{\\beta\_\{t\_\{\\sigma\}\(T\)\}/K\},\\qquad\\bar\{S\}\_\{p\}^\{h\}\(y^\{h\}\\mid x;\\sigma\)\\coloneqq\\lim\_\{T\\to\\infty\}\\frac\{p^\{h\}\_\{t\_\{\\sigma\}\(T\)\-1\\mid t\_\{\\sigma\}\(T\)\}\(y^\{h\}\\mid x\)\}\{\\beta\_\{t\_\{\\sigma\}\(T\)\}/K\},whenever the latter limit exists\. We measure the local multiplicative discrepancy betweenS¯p\\bar\{S\}\_\{p\}andS¯q\\bar\{S\}\_\{q\}by the exponentiated*Thompson distance*\(thompson1963certain\), or equivalently by the distortion factor

C¯σ​\(x\)≔maxh∈\[H\]⁡maxyh∈\[K\]∖\{xh\}⁡max⁡\{S¯ph​\(yh∣x;σ\)S¯qh​\(yh∣x;σ\),S¯qh​\(yh∣x;σ\)S¯ph​\(yh∣x;σ\)\}\.\\bar\{C\}\_\{\\sigma\}\(x\)\\coloneqq\\max\_\{h\\in\[H\]\}\\max\_\{y^\{h\}\\in\[K\]\\setminus\\\{x^\{h\}\\\}\}\\max\\left\\\{\\frac\{\\bar\{S\}\_\{p\}^\{h\}\(y^\{h\}\\mid x;\\sigma\)\}\{\\bar\{S\}\_\{q\}^\{h\}\(y^\{h\}\\mid x;\\sigma\)\},\\frac\{\\bar\{S\}\_\{q\}^\{h\}\(y^\{h\}\\mid x;\\sigma\)\}\{\\bar\{S\}\_\{p\}^\{h\}\(y^\{h\}\\mid x;\\sigma\)\}\\right\\\}\.This is the natural notion of accuracy for our purpose: the small\-noise expansion separates edits by powers ofσ\\sigma, so preserving the correct order of magnitude is enough to identify support\-improving directions, even when the leading coefficients are not yet accurately calibrated\. The following result makes this implication precise\.

###### Corollary 2\.1\(Coarse score accuracy recovers support\-improving edits\)\.

Fixx∈𝒳x\\in\\mathcal\{X\}and consider uniform diffusion in the low\-noise regime of[Theorem˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmtheorem1)\. If

C¯σ​\(x\)=o​\(σ−1/2\)as​σ→0\+,\\bar\{C\}\_\{\\sigma\}\(x\)=o\(\\sigma^\{\-1/2\}\)\\qquad\\text\{as \}\\sigma\\to 0^\{\+\},then, for all sufficiently smallσ\\sigma,S¯ph​\(yh∣x;σ\)\>σ−1/2⟺d​\(xh→yh,D\)<d​\(x,D\)\.\\bar\{S\}\_\{p\}^\{h\}\(y^\{h\}\\mid x;\\sigma\)\>\\sigma^\{\-1/2\}\\Longleftrightarrow d\(x^\{h\\to y^\{h\}\},D\)<d\(x,D\)\.Thus thresholding the limiting normalized learned scores recovers exactly the support\-improving directions\.

###### Proof\.

By[Theorem˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmtheorem1), after takingT→∞T\\to\\infty, there exist constantsax,Ax\>0a\_\{x\},A\_\{x\}\>0such that, for all sufficiently smallσ\\sigma,

S¯qh​\(yh∣x;σ\)≥ax​σ−1if​d​\(xh→yh,D\)<d​\(x,D\),S¯qh​\(yh∣x;σ\)≤Axotherwise\.\\bar\{S\}\_\{q\}^\{h\}\(y^\{h\}\\mid x;\\sigma\)\\geq a\_\{x\}\\sigma^\{\-1\}\\quad\\text\{if \}d\(x^\{h\\to y^\{h\}\},D\)<d\(x,D\),\\qquad\\bar\{S\}\_\{q\}^\{h\}\(y^\{h\}\\mid x;\\sigma\)\\leq A\_\{x\}\\quad\\text\{otherwise\}\.Therefore the multiplicative error bound givesS¯ph​\(yh∣x;σ\)≥C¯σ​\(x\)−1​ax​σ−1=ω​\(σ−1/2\)\\bar\{S\}\_\{p\}^\{h\}\(y^\{h\}\\mid x;\\sigma\)\\geq\\bar\{C\}\_\{\\sigma\}\(x\)^\{\-1\}a\_\{x\}\\sigma^\{\-1\}=\\omega\(\\sigma^\{\-1/2\}\)for support\-improving edits, andS¯ph​\(yh∣x;σ\)≤C¯σ​\(x\)​Ax=o​\(σ−1/2\)\\bar\{S\}\_\{p\}^\{h\}\(y^\{h\}\\mid x;\\sigma\)\\leq\\bar\{C\}\_\{\\sigma\}\(x\)A\_\{x\}=o\(\\sigma^\{\-1/2\}\)for all non\-support\-improving edits\. Hence the thresholdσ−1/2\\sigma^\{\-1/2\}separates the two classes\. ∎

The conditionC¯σ​\(x\)=o​\(σ−1/2\)\\bar\{C\}\_\{\\sigma\}\(x\)=o\(\\sigma^\{\-1/2\}\)is deliberately coarse\. By[Theorem˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmtheorem1), after taking the continuous\-time limit, support\-improving edits have true normalized scores of orderσ−1\\sigma^\{\-1\}, whereas non\-improving edits have scores of order at most11\. Hence such a multiplicative distortion, though allowed to diverge asσ→0\+\\sigma\\to 0^\{\+\}, cannot close the scale gap: support\-improving edits remain above the thresholdσ−1/2\\sigma^\{\-1/2\}, while non\-improving edits remain below it\. This is much weaker than coefficient matching, which would requireS¯ph​\(yh∣x;σ\)S¯qh​\(yh∣x;σ\)→1\\frac\{\\bar\{S\}\_\{p\}^\{h\}\(y^\{h\}\\mid x;\\sigma\)\}\{\\bar\{S\}\_\{q\}^\{h\}\(y^\{h\}\\mid x;\\sigma\)\}\\to 1on the relevant directions\. The reason is that theσ\\sigma\-scale in the low\-noise expansion depends only on how the edit changes the distance toD=supp​\(pdata\)D=\\text\{supp\}\(p\_\{\\text\{data\}\}\), whereas the coefficient depends on the frequencies assigned bypdatap\_\{\\text\{data\}\}withinDD\. Thus*two data distributions with the same support but very different probabilities on that support induce the same scale hierarchy, but may have very different coefficients\.*Support recovery therefore only requires preserving the correct*order of magnitude*, while frequency calibration requires coefficient\-level accuracy\. As a simple illustration,[Section˜B\.1](https://arxiv.org/html/2605.13999#A2.SS1)gives a concrete example showing that this type of coarse multiplicative control can preserve scale information even when the corresponding probability vectors remain far apart in total variation and KL divergence\.

### 3Experiments

The purpose of this section is to test the two predictions suggested by the small\-noise theory: that support localization emerges before frequency ranking, and that uniform and absorbing diffusion exhibit distinct projector\-like behaviors\.

#### 3\.1Test of prediction 1: support precedes frequency

The first takeaway from[Section˜2](https://arxiv.org/html/2605.13999#S2)is a support\-then\-frequency separation: the reverse transitions expose information related to the support before distributional information\. We empirically test the effect of this observation in two complementary settings\. First, we design a synthetic*regular language*for which the support and the within\-support frequencies are known, and directly monitor the corresponding metrics across training\. Second, on FineWeb, the true linguistic support is not available in closed form, so we construct*indirect*held\-out probes that mirror the exact metrics\.

##### Synthetic direct probes\.

We first construct a regular language that serves as a controlled analogue of a bigram language model: the next token depends only on the previous token, but with token\-dependent transition probabilities\. We sample the first token uniformly from\[K\]\[K\]\. Givenxhx\_\{h\}, the next tokenxh\+1x\_\{h\+1\}is sampled from the local neighborhood

\{xh−1,xh,xh\+1\},\\\{x\_\{h\}\-1,x\_\{h\},x\_\{h\}\+1\\\},with probabilitiespdown​\(xh\)p\_\{\\mathrm\{down\}\}\(x\_\{h\}\),pstay​\(xh\)p\_\{\\mathrm\{stay\}\}\(x\_\{h\}\), andpup​\(xh\)p\_\{\\mathrm\{up\}\}\(x\_\{h\}\), respectively\. The upward and downward probabilities are specified by a deterministic oscillating transform ofxhx\_\{h\}, and the walk is clamped at the boundary tokens11andKK; the exact construction is given in[Section˜C\.1](https://arxiv.org/html/2605.13999#A3.SS1)\. Thus the data support is the set of strings whose successive tokens obey the allowed local\-transition rule\. This gives exact access to both support membership and within\-support transition probabilities, allowing us to define direct probes for support recovery and frequency learning\.

For the fixed\-context transformer trained in the standard D3PM pipeline, we probe the learned reverse conditionals at low noiseσ=1/H\\sigma=1/H\. For absorbing diffusion, we draw a validation string and replace one token by the mask token\. For uniform diffusion, we instead replace that token by1010tokens sampled uniformly at random from\[K\]\[K\], and average the resulting probe metrics over these corruptions\. In both cases, we score candidate clean\-token replacements at the corrupted position\. The support probe asks whether valid replacements receive larger conditional probability than invalid replacements\. The frequency probes restrict attention to valid replacements and test whether the model recovers the most likely replacement under the true transition law \(top\-11accuracy\) and whether it correctly ranks pairs of valid replacements\. Each metric is averaged over1,0001\{,\}000validation samples per checkpoint\.

The results are shown in[Figure˜2](https://arxiv.org/html/2605.13999#S3.F2)\. Both absorbing and uniform diffusion exhibit the predicted ordering:*support\-recovery metrics improve before frequency\-oriented metrics*\. These direct probes are the synthetic counterparts of the indirect FineWeb probes, although the metrics are not identical because the true support is available here but not for natural language\. Additional experimental details and ablations are given in[Appendix˜C](https://arxiv.org/html/2605.13999#A3), see in particular[Figures˜5](https://arxiv.org/html/2605.13999#A3.F5),[6](https://arxiv.org/html/2605.13999#A3.F6)and[7](https://arxiv.org/html/2605.13999#A3.F7)\.

![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L32K64T64/triplet_metrics/triplet_metrics.png)\(a\)Absorbing diffusion
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L32K64T64/triplet_metrics/triplet_metrics.png)\(b\)Uniform diffusion

Figure 2:Synthetic echo of support before frequency\.In the regular\-language setting of[Section˜3\.2](https://arxiv.org/html/2605.13999#S3.SS2), direct support metrics improve before frequency ones, paralleling the FineWeb trend in[Figure˜1](https://arxiv.org/html/2605.13999#S1.F1)\.
##### FineWeb indirect probes\.

We then ask whether the same separation appears in a masked DLM trained on FineWeb\(penedo2024fineweb\)\. We choose masked diffusion for the real\-data experiment since it is popular, empirically strong, and the backbone of all modern discrete diffusion language models at scale, e\.g\.,nie2025large;ye2025dream;bie2025llada20;bie2026llada21;song2025seed;liu2025tidar\. Moreover, the absorbing case is particularly convenient since several popular parameterizations all collapse to the*same*loss and parameterization, so that the same trained model probes many different masked\-DLM formulations and parameterizations at the same time, including D3PM\-style posterior prediction, Bayes clean\-token substitution, mean/SUBS reverse kernels, and the normalized induced\-score parameterization\. Thus the experiment probes a common object shared by many masked\-DLM formulations rather than a parameterization\-specific artifact;[Appendix˜A](https://arxiv.org/html/2605.13999#A1)reviews the equivalences and the cases where they instead differ\. Since the true support of natural language is unavailable, we use held\-out FineWeb as a proxy\. We scan validation text for repeated one\-token restoration contexts: for a left tokenℓ\\elland right tokenrr, we record which center tokens actually occurred in the slot\(ℓ,\[MASK\],r\)\(\\ell,\\texttt\{\[MASK\]\},r\)\. For instance, if held\-out FineWeb containsthe end of, thenendis an empirical candidate forthe \[MASK\] of\. A non\-candidate for this slot is a token observed elsewhere in the same context bank but not betweentheandof\.

The primary support curve in[Figure˜1](https://arxiv.org/html/2605.13999#S1.F1)is the pairwise accuracy with which empirical candidate tokens outrank frequency\-matched non\-candidate negatives\. The negatives are not necessarily ungrammatical; they simply are tokens that were observed elsewhere in the FineWeb context bank but not in this particular slot\. Matching negatives to positives by global frequency prevents the probe from rewarding a model merely for preferring common tokens;[Figure˜14](https://arxiv.org/html/2605.13999#A4.F14)shows that other choices of negatives yield the same transition timing\. We compare this support probe to two within\-candidate frequency probes: pairwise empirical ranking accuracy among candidates, and top\-1 empirical mode recovery\.[Section˜D\.11](https://arxiv.org/html/2605.13999#A4.SS11)checks that these indirect probes mirror the empirical evolution of their direct equivalent in a synthetic setup where the support is known\.

##### FineWeb result\.

Across three independent training seeds, the frequency\-matched support probe reaches90%90\\%of its peak gain at116\.3±5\.7116\.3\\pm 5\.7M training tokens \(range113\.0113\.0–122\.9122\.9M\)\. The top\-1 empirical\-mode and pairwise empirical ranking probes reach the same criterion at234\.3±22\.2234\.3\\pm 22\.2M and247\.4±7\.5247\.4\\pm 7\.5M tokens, respectively\. Thus support localization precedes frequency sharpening by roughly118118–131131M tokens on average, and the ordering holds in each seed\. The absolute token counts depend on the model, data window, and context bank; the key comparison is within the same matched probe\. We view the separation as the real\-data analogue of the rate separation identified in[Section˜2](https://arxiv.org/html/2605.13999#S2): the model learns a useful support\-like projector before it learns the precise empirical distribution over that support\.

##### Generalization vs\. memorization\.

We also ask where this separation sits relative to the usual generalization–memorization distinction in the FineWeb experiment\. We evaluate masked\-token reconstruction losses on fixed train and validation windows at fixed masking levels\. A growing validation\-minus\-train gap would indicate train\-specific memorization; a small gap while both losses improve is consistent with a generalization\-like phase\. In the measured early\-token window, this diagnostic shows a small gap while training and validation improve\. Full probe definitions, scoring details, training recipes, and supporting diagnostics are given in[Appendix˜D](https://arxiv.org/html/2605.13999#A4)\.

#### 3\.2Test of prediction 2: Comparing uniform and masked projectors in synthetic examples

![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L32K64T64/sum_n0p1_t6_thr0p05/summary_frac_support.png)\(a\)Absorbing diffusion
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L32K64T64/sum_n0p1_t6_thr0p05/summary_frac_support.png)\(b\)Uniform diffusion

Figure 3:Projection\-style edits help uniform diffusion more\.The second takeaway of[Section˜2](https://arxiv.org/html/2605.13999#S2)is that at low noise, the reverse dynamics can act like a local projector onto data support, and that absorbing diffusion more strictly separates projective from non\-projective single\-token edits\. We thus train uniform and absorbing D3PMs on the aforementioned synthetic distribution \([Section˜3\.1](https://arxiv.org/html/2605.13999#S3.SS1)\) and test the fraction of strings that are in the support of the synthetic data distribution\. Notice that unlike the previous probes, this is a test of validity at full*sequence level*rather than of local edits\.

We compare the success rate of standard ancestral sampling to that of two slight modifications which, according to our theory, should leverage the information about projective directions more explicitly to enhance landing in the support\. Both modified samplers, which we call “thresholding” and “hard\-max” samplers respectively, first follow the standard denoising trajectory up until a small pre\-defined noise levelσ\\sigma\. Then, they repeatedly make a single\-token edit among all single token edits for which the normalized reverse score \([Section˜2\.2](https://arxiv.org/html/2605.13999#S2.SS2)\) is above a certain pre\-defined thresholdτ\>0\\tau\>0\(see[Section˜C\.4](https://arxiv.org/html/2605.13999#A3.SS4)for the implementation details\)\. Clearly, by our analysis in[Corollary˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmcorollary1), this should explicitly single out directions along a projection onto the support if the threshold is well\-calibrated: If the magnitudes of the reverse kernels have been learned to sufficient accuracy, theory predicts that the two samplers filter out all edits that do not strictly decrease the distance to the support\.

[Figure˜3](https://arxiv.org/html/2605.13999#S3.F3)reports the fraction of generated sequences that land in the true support under the three samplers\. For absorbing diffusion, the modified samplers perform nearly identically to ancestral sampling\. For uniform diffusion, in contrast, isolating the dominant component of the learned reverse scores substantially improves sample validity\. This behavior matches the prediction of[Theorem˜2\.2](https://arxiv.org/html/2605.13999#S2.Thmtheorem2): absorbing diffusion has a sharper low\-noise separation than uniform diffusion \([Theorem˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmtheorem1)\), so its permitted reverse moves are already much less affected by lower\-order, non\-projective components\.

This comparison should not be read as saying that absorbing diffusion is a better global projector onto the support\. The absorbing reverse process is structurally restricted: it can only fill in currently masked tokens, and cannot re\-mask or revise tokens that have already been incorrectly decoded\. Thus it may identify the projective directions very accurately among the*permitted*denoising moves, while still lacking access to other correcting moves needed for global support projection\. This is consistent with the observation that the modified uniform samplers can ultimately achieve a higher support\-landing rate than absorbing diffusion\. Rather, the result shows that absorbing diffusion is already close to a projector within its restricted reverse graph, whereas uniform diffusion benefits from explicitly filtering its richer set of possible edits\.

### 4Limitations and Future Directions

Our analysis focuses on the exact reverse process in the small\-noise regime, corresponding to the final denoising steps\. The resulting scale separation is not merely infinitesimal: since the gap scales as1/σ1/\\sigma, evenσ≈0\.1\\sigma\\approx 0\.1gives an order\-of\-magnitude separation between support and finer frequency information\. Nevertheless, the extent to which trained models recover the different terms in this expansion may depend on the loss, architecture, parameterization, and optimization procedure\. A sharper training theory, perhaps in a stylized model or through statistical guarantees, is needed to explain when these terms are learned in the predicted order\.

Several directions remain open\. First, extending the experiments to broader DLM parameterizations and larger real\-world datasets would clarify the scope of the phenomenon\. Second, support can be brittle for natural language and other high\-dimensional discrete data, where rare but valid strings may have extremely small probability\. A more robust formulation could replace the hard supportD=supp​\(pdata\)D=\\text\{supp\}\(p\_\{\\text\{data\}\}\)with anϵ\\epsilon\-effective support\. Finally, the practical implications of support\-before\-frequency remain to be developed, including early stopping criteria, validity\-improving inference\-time samplers, and downstream applications that prioritize structural correctness over full distributional calibration\.

### Acknowledgements

This work was supported in part by the Swiss State Secretariat for Education, Research and Innovation \(SERI\) under contract number MB22\.00027\. We thank EPFL’s RCP cluster for computational resources, and the RCP team for their organization during periods of high demand on the cluster\. Part of this research was performed while Adrian Müller was visiting the Institute for Mathematical and Statistical Innovation \(IMSI\), which is supported by the National Science Foundation \(Grant No\. DMS\-2425650\)\.

### References

## Appendix

### Appendix ADiscrete\-diffusion parameterizations for masked language modeling

This appendix summarizes the modeling choices behind the masked diffusion language model used in the FineWeb experiment in[Section˜3\.1](https://arxiv.org/html/2605.13999#S3.SS1)\. The goal is not to survey all diffusion language models, but to make clear which common parameterizations are equivalent in the absorbing\-mask case, which ones are not, and which family our experiments instantiate\. An illustration is given in[Figure˜4](https://arxiv.org/html/2605.13999#A1.F4)\.

The clean data sequence is denoted byx0∈𝒱Hx\_\{0\}\\in\\mathcal\{V\}^\{H\}, where𝒱=\[K\]\\mathcal\{V\}=\[K\]is the vocabulary\. In absorbing diffusion the state space is enlarged by one mask tokenm∉𝒱m\\notin\\mathcal\{V\}\. A “clean token” simply means a token in𝒱\\mathcal\{V\}, as opposed to the mask tokenmm\.

The neural network outputs raw numbers\(nnθ​\(xt,t\)\)h,v\\big\(\\mathrm\{nn\}\_\{\\theta\}\(x\_\{t\},t\)\\big\)\_\{h,v\}for each positionh∈\[H\]h\\in\[H\]and candidate clean tokenv∈𝒱v\\in\\mathcal\{V\}\. When a parameterization needs a probability distribution on clean tokens, we write

nn^θ,h​\(v∣xt,t\):=exp\(nnθ\(xt,t\)\)h,v∑u∈𝒱exp\(nnθ\(xt,t\)\)h,u\.\\widehat\{\\mathrm\{nn\}\}\_\{\\theta,h\}\(v\\mid x\_\{t\},t\):=\\frac\{\\exp\\big\(\\mathrm\{nn\}\_\{\\theta\}\(x\_\{t\},t\)\\big\)\_\{h,v\}\}\{\\sum\_\{u\\in\\mathcal\{V\}\}\\exp\\big\(\\mathrm\{nn\}\_\{\\theta\}\(x\_\{t\},t\)\\big\)\_\{h,u\}\}\.The learned object is the raw network outputnnθ\\mathrm\{nn\}\_\{\\theta\}; the hat only means that we have normalized it for a clean\-token parameterization\. In particular,nn^θ,h\\widehat\{\\mathrm\{nn\}\}\_\{\\theta,h\}is a distribution on𝒱\\mathcal\{V\}, not on𝒱∪\{m\}\\mathcal\{V\}\\cup\\\{m\\\}\. If one starts from logits over the enlarged alphabet, this corresponds to the “zero mask probability” substitution used in SUBS\[sahoo2024simple\]\.

We focus on this absorbing\-mask SUBS/mean family for a conservative reason\. Absorbing or masked diffusion has become the dominant route for diffusion language modeling\. It connects D3PMs to masked language modeling\[austin2021structured\], admits simplified objectives and parameterizations\[shi2024simplified,sahoo2024simple,ou2024your\], and has led to increasingly competitive text models and samplers\[lou2023discrete,sahoo2024simple,arriola2025block,wu2025fast,israel2025accelerating,nie2025large\]\. This absorbing case is also convenient to consider as it sits at the intersection of several otherwise different views: D3PM\-style posterior parameterization, Bayes clean\-token substitution, mean/SUBS reverse kernels, and the normalized induced\-score parameterization all collapse to the same clean\-token predictor\. Thus the FineWeb experiment is not tied to an idiosyncratic parameterization; in the mask case it probes the common object shared by these formulations\. Away from absorbing masks, especially for uniform corruption or direct positive score models such as SEDD, these parameterizations separate again\.

\\lxSVG@picture

Discrete timegeneration uses one\-step kernelsq​\(xt−1∣xt\)q\(x\_\{t\-1\}\\mid x\_\{t\}\)model the backward step byqθ​\(xt−1∣xt\)q\_\{\\theta\}\(x\_\{t\-1\}\\mid x\_\{t\}\)ELBO comparesq​\(xt−1∣xt,x0\)q\(x\_\{t\-1\}\\mid x\_\{t\},x\_\{0\}\)andqθ​\(xt−1∣xt\)q\_\{\\theta\}\(x\_\{t\-1\}\\mid x\_\{t\}\)Continuous timegeneration uses reverse ratesR~t​\(y→y′\)=Rt​\(y′→y\)​st​\(y→y′\)\\widetilde\{R\}\_\{t\}\(y\\to y^\{\\prime\}\)=R\_\{t\}\(y^\{\\prime\}\\to y\)s\_\{t\}\(y\\to y^\{\\prime\}\)model the reverse rate througha score parameterizationsθs\_\{\\theta\}score objective comparesthe true scorests\_\{t\}and model scoresθs\_\{\\theta\}absorbing \+ clean\-token outputBayes / D3PM / mean\-SUBS coincideand inducesθinds\_\{\\theta\}^\{\\mathrm\{ind\}\}weighted masked clean\-token CE∑twt​𝔼\[−log⁡nn^θ,h​\(x0h∣xt,t\)​𝟙​\{xth=m\}\]\\displaystyle\\sum\_\{t\}w\_\{t\}\\,\\operatorname\*\{\\mathbb\{E\}\}\[\-\\log\\widehat\{\\mathrm\{nn\}\}\_\{\\theta,h\}\(x\_\{0\}^\{h\}\\mid x\_\{t\},t\)\\mathbbm\{1\}\\left\\\{x\_\{t\}^\{h\}=m\\right\\\}\]direct positive scoreparameterizesθs\_\{\\theta\}itselfas in SEDDscore\-entropy lossfor a general positive score field\\endlxSVG@picture

Figure 4:Two routes for training absorbing\-mask diffusion models\. In the clean\-token family used by D3PM\-style and mean/SUBS parameterizations, the reverse kernel and the induced score are determined by the same normalized network output and lead to the same weighted masked\-token cross\-entropy\. Direct score parameterizations such as SEDD remain a separate family\. We use the weighted masked clean\-token CE in our experiments, in the mask case, which is equivalent to the popular SUBS/mean and D3PM parameterizations, and which also coincides with a score objective for the induced score field\.#### A\.1The reverse step and where the neural network enters

Generation proceeds by starting from a fully noised sequence and applying reverse steps one at a time:

xT→xT−1→⋯→x0\.x\_\{T\}\\to x\_\{T\-1\}\\to\\cdots\\to x\_\{0\}\.The ideal reverse step is the true conditional distributionqt−1∣tq\_\{t\-1\\mid t\}\. This object depends on the data distribution and is unknown\. Standard parameterized reverse kernels for DLMs are token\-factorized:

q​\(xt−1∣xt\)=∏h∈\[H\]q​\(xt−1h∣xt\)q\(x\_\{t\-1\}\\mid x\_\{t\}\)=\\prod\_\{h\\in\[H\]\}q\(x\_\{t\-1\}^\{h\}\\mid x\_\{t\}\)so it is enough to describe the token\-level reverse stepsq​\(xt−1h∣xt\)q\(x\_\{t\-1\}^\{h\}\\mid x\_\{t\}\)\. For one positionhh, a useful Bayes decomposition is

q​\(xt−1h∣xt\)=∑v∈𝒱q​\(xt−1h∣xt,x0h=v\)​q​\(x0h=v∣xt\)\.\\displaystyle q\(x\_\{t\-1\}^\{h\}\\mid x\_\{t\}\)=\\sum\_\{v\\in\\mathcal\{V\}\}q\(x\_\{t\-1\}^\{h\}\\mid x\_\{t\},x^\{h\}\_\{0\}=v\)\\,q\(x\_\{0\}^\{h\}=v\\mid x\_\{t\}\)\.\(2\)The first factor of each summand is easy to compute in practice: once the original tokenvvand the current noised tokenxtx\_\{t\}are fixed, the probability of the previous noised tokenxt−1hx\_\{t\-1\}^\{h\}is determined by the known corruption kernel\. For instance, in absorbing diffusion, ifxthx\_\{t\}^\{h\}is not masked, thenxt−1hx\_\{t\-1\}^\{h\}must be the same clean token; ifxthx\_\{t\}^\{h\}is masked, thenxt−1hx\_\{t\-1\}^\{h\}is either the same mask or the original clean token, with probabilities determined by the noise schedule\.

The second factor is, just like the original quantityq​\(xt−1h∣xt\)q\(x\_\{t\-1\}^\{h\}\\mid x\_\{t\}\), unknown\. It is the posterior belief about the original token at positionhh, given the whole corrupted sequencextx\_\{t\}\. Several parameterizations in the literature can be read as replacing this unknown posteriorq​\(x0h=v∣xt\)q\(x\_\{0\}^\{h\}=v\\mid x\_\{t\}\)by a neural network output\. Once plugged in the above formula, we deduce a surrogateqθ​\(xt−1h∣xt\)q\_\{\\theta\}\(x\_\{t\-1\}^\{h\}\\mid x\_\{t\}\)to use for sampling in place of the true reverse stepq​\(xt−1h∣xt\)q\(x\_\{t\-1\}^\{h\}\\mid x\_\{t\}\)\.

##### Bayes substitution as a reference formula\.

The most direct substitution in \([2](https://arxiv.org/html/2605.13999#A1.E2)\) would be

qθBayes​\(xt−1h∣xt\)=∑v∈𝒱q​\(xt−1h∣xt,x0h=v\)​nn^θ,h​\(v∣xt,t\)\.\\displaystyle q^\{\\mathrm\{Bayes\}\}\_\{\\theta\}\(x\_\{t\-1\}^\{h\}\\mid x\_\{t\}\)=\\sum\_\{v\\in\\mathcal\{V\}\}q\(x\_\{t\-1\}^\{h\}\\mid x\_\{t\},x\_\{0\}^\{h\}=v\)\\,\\widehat\{\\mathrm\{nn\}\}\_\{\\theta,h\}\(v\\mid x\_\{t\},t\)\.\(3\)We review below the usual constructions in the literature and how they relate to this direct Bayes substitution\. The main takeaway is that for*absorbing*diffusion, they all coincide with this Bayes substitution\.

##### D3PM posterior parameterization\.

D3PMs\[austin2021structured\]use a closely related but not identical posterior parameterization\. Note that in \([2](https://arxiv.org/html/2605.13999#A1.E2)\), the known factorq​\(xt−1h∣xt,x0h\)q\(x\_\{t\-1\}^\{h\}\\mid x\_\{t\},x\_\{0\}^\{h\}\)can be written as

q​\(xt−1h∣xt,x0h\)=q​\(xt−1h,xth∣x0h\)q​\(xth∣x0h\)\.q\(x\_\{t\-1\}^\{h\}\\mid x\_\{t\},x\_\{0\}^\{h\}\)=\\frac\{q\(x\_\{t\-1\}^\{h\},x\_\{t\}^\{h\}\\mid x\_\{0\}^\{h\}\)\}\{q\(x\_\{t\}^\{h\}\\mid x\_\{0\}^\{h\}\)\}\.Thus, the Bayes decomposition can be equivalently written as

q​\(xt−1h∣xt\)=∑v∈𝒱q​\(xt−1h,xth∣x0h=v\)​q​\(x0h=v∣xt\)q​\(xth∣x0h=v\)\.q\(x\_\{t\-1\}^\{h\}\\mid x\_\{t\}\)=\\sum\_\{v\\in\\mathcal\{V\}\}q\(x\_\{t\-1\}^\{h\},x\_\{t\}^\{h\}\\mid x\_\{0\}^\{h\}=v\)\\frac\{q\(x\_\{0\}^\{h\}=v\\mid x\_\{t\}\)\}\{q\(x\_\{t\}^\{h\}\\mid x\_\{0\}^\{h\}=v\)\}\.The D3PM parameterization replaces the fractionq​\(x0h=v∣xt\)/q​\(xth∣x0h=v\)q\(x\_\{0\}^\{h\}=v\\mid x\_\{t\}\)/q\(x\_\{t\}^\{h\}\\mid x\_\{0\}^\{h\}=v\)by the network output and then renormalizes overxt−1hx\_\{t\-1\}^\{h\}:

qθD3PM​\(xt−1h∣xt\)∝∑v∈𝒱q​\(xt−1h,xth∣x0h=v\)​nn^θ,h​\(v∣xt,t\)\.\\displaystyle q^\{\\mathrm\{D3PM\}\}\_\{\\theta\}\(x\_\{t\-1\}^\{h\}\\mid x\_\{t\}\)\\propto\\sum\_\{v\\in\\mathcal\{V\}\}q\(x\_\{t\-1\}^\{h\},x\_\{t\}^\{h\}\\mid x\_\{0\}^\{h\}=v\)\\,\\widehat\{\\mathrm\{nn\}\}\_\{\\theta,h\}\(v\\mid x\_\{t\},t\)\.\(4\)This differs from the direct Bayes substitution by the extra factorq​\(xth∣x0h\)q\(x\_\{t\}^\{h\}\\mid x\_\{0\}^\{h\}\)inside the mixture weights\. For general corruption kernels that factor depends onx0hx\_\{0\}^\{h\}, so the two substitutions are not the same\. For absorbing diffusion, whenxth=mx\_\{t\}^\{h\}=m, the known factorq​\(xth=m∣x0h=v\)q\(x\_\{t\}^\{h\}=m\\mid x\_\{0\}^\{h\}=v\)is independent ofvv, so the extra factor cancels in the normalization\. Whenxth≠mx\_\{t\}^\{h\}\\neq m, the absorbing process forcesx0h=xthx\_\{0\}^\{h\}=x\_\{t\}^\{h\}and the reverse step copies the clean token\. Hence, under the usual copy and normalization constraints, Bayes and D3PM parameterizations are the same for absorbing masks\.

##### Mean/SUBS parameterization for absorbing masks\.

A popular parameterization is the mean/SUBS parameterization used byshi2024simplified,sahoo2024simple, andou2024your\. This parameterization is designed for absorbing masks and is a special case of the Bayes substitution for that corruption kernel\. In particular, it coincides with the Bayes, and therefore D3PM, parameterization for absorbing masks\. Starting again from the exact Bayes formula in \([2](https://arxiv.org/html/2605.13999#A1.E2)\), the mean/SUBS parameterization first replaces the known factorq​\(xt−1h∣xt,x0h\)q\(x\_\{t\-1\}^\{h\}\\mid x\_\{t\},x\_\{0\}^\{h\}\)by its exact closed\-form expression in the absorbing case:

q​\(xt−1h∣xt,x0h\)=\{δxth​\(xt−1h\),xth≠m,σt−1σt​δm​\(xt−1h\)\+σt−σt−1σt​δx0h​\(xt−1h\),xth=m\.\\displaystyle q\(x\_\{t\-1\}^\{h\}\\mid x\_\{t\},x\_\{0\}^\{h\}\)=\\begin\{cases\}\\delta\_\{x\_\{t\}^\{h\}\}\(x\_\{t\-1\}^\{h\}\),&x\_\{t\}^\{h\}\\neq m,\\\\\[3\.44444pt\] \\displaystyle\\frac\{\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\\,\\delta\_\{m\}\(x\_\{t\-1\}^\{h\}\)\+\\frac\{\\sigma\_\{t\}\-\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\\,\\delta\_\{x\_\{0\}^\{h\}\}\(x\_\{t\-1\}^\{h\}\),&x\_\{t\}^\{h\}=m\.\\end\{cases\}\(5\)The interpretation is literal\. If the token at timettis not masked, it must be carried over to timet−1t\-1, since the forward process never substitutes one clean token for another\. If the current token is masked, then one step earlier it was either already masked, or it was still equal to the original clean tokenx0hx\_\{0\}^\{h\}\.

Plugging this absorbing\-mask posterior into the Bayes substitution gives a copy term for the unmasked case, and a mixture of the mask and clean\-token posterior for the masked case:

q​\(xt−1h∣xt\)=\{δxth​\(xt−1h\),xth≠m,σt−1σt​δm​\(xt−1h\)\+σt−σt−1σt​∑v∈𝒱q​\(x0h=v∣xt\)​δv​\(xt−1h\),xth=m\.\\displaystyle q\(x\_\{t\-1\}^\{h\}\\mid x\_\{t\}\)=\\begin\{cases\}\\delta\_\{x\_\{t\}^\{h\}\}\(x\_\{t\-1\}^\{h\}\),&x\_\{t\}^\{h\}\\neq m,\\\\\[3\.44444pt\] \\displaystyle\\frac\{\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\\,\\delta\_\{m\}\(x\_\{t\-1\}^\{h\}\)\+\\frac\{\\sigma\_\{t\}\-\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\\sum\_\{v\\in\\mathcal\{V\}\}q\(x\_\{0\}^\{h\}=v\\mid x\_\{t\}\)\\,\\delta\_\{v\}\(x\_\{t\-1\}^\{h\}\),&x\_\{t\}^\{h\}=m\.\\end\{cases\}\(6\)The mean/SUBS parameterization replaces the remaining unknown clean\-token posterior by the normalized network output:

qθSUBS​\(xt−1h∣xt\)=\{δxth​\(xt−1h\),xth≠m,σt−1σt​δm​\(xt−1h\)\+σt−σt−1σt​∑v∈𝒱nn^θ,h​\(v∣xt,t\)​δv​\(xt−1h\),xth=m\.\\displaystyle q^\{\\mathrm\{SUBS\}\}\_\{\\theta\}\(x\_\{t\-1\}^\{h\}\\mid x\_\{t\}\)=\\begin\{cases\}\\delta\_\{x\_\{t\}^\{h\}\}\(x\_\{t\-1\}^\{h\}\),&x\_\{t\}^\{h\}\\neq m,\\\\\[3\.44444pt\] \\displaystyle\\frac\{\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\\,\\delta\_\{m\}\(x\_\{t\-1\}^\{h\}\)\+\\frac\{\\sigma\_\{t\}\-\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\\sum\_\{v\\in\\mathcal\{V\}\}\\widehat\{\\mathrm\{nn\}\}\_\{\\theta,h\}\(v\\mid x\_\{t\},t\)\\,\\delta\_\{v\}\(x\_\{t\-1\}^\{h\}\),&x\_\{t\}^\{h\}=m\.\\end\{cases\}\(7\)This is the parameterization used by the model in[Section˜3\.1](https://arxiv.org/html/2605.13999#S3.SS1)\.

#### A\.2Losses for absorbing masks

We assume that we approximate the reverse process, which is given byqTq\_\{T\}and\(qt−1\|t\)t∈\[T\]\(q\_\{t\-1\|t\}\)\_\{t\\in\[T\]\}, with a parameterized process given bypT∈Δ𝒳p\_\{T\}\\in\\Delta\_\{\\mathcal\{X\}\}and\(pt−1\|t\(⋅∣⋅\)\)t∈\[T\]\(p\_\{t\-1\|t\}\(\\cdot\\mid\\cdot\)\)\_\{t\\in\[T\]\}\. The usual variational upper bound on the negative log\-likelihood decomposes as

𝔼x0∼pdata\[−log⁡p0​\(x0\)\]≤BT​\(pT\)\+∑t=1TLt​\(p\),\\operatorname\*\{\\mathbb\{E\}\}\_\{x\_\{0\}\\sim p\_\{\\text\{data\}\}\}\\big\[\-\\log p\_\{0\}\(x\_\{0\}\)\\big\]\\leq B\_\{T\}\(p\_\{T\}\)\+\\sum\_\{t=1\}^\{T\}L\_\{t\}\(p\),where the terminal boundary term isBT\(pT\):=𝔼x0∼pdata\[DKL\(qT\|0\(⋅∣x0\)∥pT\)\]B\_\{T\}\(p\_\{T\}\):=\\operatorname\*\{\\mathbb\{E\}\}\_\{x\_\{0\}\\sim p\_\{\\text\{data\}\}\}\\Big\[D\_\{\\text\{KL\}\}\\big\(q\_\{T\|0\}\(\\cdot\\mid x\_\{0\}\)\\,\\\|\\,p\_\{T\}\\big\)\\Big\], and the denoising term at timettis

Lt\(p\):=𝔼x0∼pdataxt∼qt\|0\(⋅∣x0\)\[DKL\(qt−1\|t,0\(⋅∣xt,x0\)∥pt−1\|t\(⋅∣xt\)\)\]\.L\_\{t\}\(p\):=\\operatorname\*\{\\mathbb\{E\}\}\_\{\\begin\{subarray\}\{c\}x\_\{0\}\\sim p\_\{\\text\{data\}\}\\\\ x\_\{t\}\\sim q\_\{t\|0\}\(\\cdot\\mid x\_\{0\}\)\\end\{subarray\}\}\\Big\[D\_\{\\text\{KL\}\}\\big\(q\_\{t\-1\|t,0\}\(\\cdot\\mid x\_\{t\},x\_\{0\}\)\\,\\\|\\,p\_\{t\-1\|t\}\(\\cdot\\mid x\_\{t\}\)\\big\)\\Big\]\.Here the caset=1t=1includes the reconstruction term, sinceq0\|1,0\(⋅∣x1,x0\)=δx0q\_\{0\|1,0\}\(\\cdot\\mid x\_\{1\},x\_\{0\}\)=\\delta\_\{x\_\{0\}\}\. Since the terminal term either vanishes or is independent of the learned reverse kernels whenpTp\_\{T\}is fixed to the fully corrupted marginal, we focus on the normalized denoising objective

L​\(p\):=1T​∑t=1TLt​\(p\)\.L\(p\):=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}L\_\{t\}\(p\)\.Most standard discrete diffusion models, starting from D3PM, train the reverse kernels through the empirical variational bound above; in the absorbing/masked case, this bound reduces to equivalent weighted clean\-token prediction objectives used in modern masked DLMs\[austin2021structured,shi2024simplified,ou2024your,sahoo2024simple\], as we now detail, and whose formula is given in[Equation˜8](https://arxiv.org/html/2605.13999#A1.E8)\.

If we fix a timettand consider, as in the previous subsection,p=qθp=q\_\{\\theta\}an approximation given by a neural network with parametersθ\\theta, then the variational bound is expressed in terms of denoising terms

Lt\(qθ\)=𝔼x0∼pdataxt∼qt∣0\(⋅∣x0\)\[KL\(qt−1∣t,0\(⋅∣xt,x0\)∥qθ\(⋅∣xt\)\)\]\.L\_\{t\}\(q\_\{\\theta\}\)=\\operatorname\*\{\\mathbb\{E\}\}\_\{\\begin\{subarray\}\{c\}x\_\{0\}\\sim p\_\{\\text\{data\}\}\\\\ x\_\{t\}\\sim q\_\{t\\mid 0\}\(\\cdot\\mid x\_\{0\}\)\\end\{subarray\}\}\\left\[\\mathrm\{KL\}\\\!\\left\(q\_\{t\-1\\mid t,0\}\(\\cdot\\mid x\_\{t\},x\_\{0\}\)\\,\\middle\\\|\\,q\_\{\\theta\}\(\\cdot\\mid x\_\{t\}\)\\right\)\\right\]\.The terminal term is fixed once the terminal distribution is fixed, so the model\-dependent part is the sum of these denoising terms∑t=1TLt​\(qθ\)\\sum\_\{t=1\}^\{T\}L\_\{t\}\(q\_\{\\theta\}\)\. In the absorbing case, the simplification above makes the loss a weighted clean\-token prediction objective\. The KL is against the posterior conditioned on the clean samplex0x\_\{0\}, not directly against the marginal reverse kernelqt−1∣t\(⋅∣xt\)q\_\{t\-1\\mid t\}\(\\cdot\\mid x\_\{t\}\)\. After averaging overx0∣xtx\_\{0\}\\mid x\_\{t\}, however, the population minimizer over kernelsqθ\(⋅∣xt\)q\_\{\\theta\}\(\\cdot\\mid x\_\{t\}\)is exactly the true marginal reverse kernelqt−1∣t\(⋅∣xt\)q\_\{t\-1\\mid t\}\(\\cdot\\mid x\_\{t\}\)\.

Indeed, for an unmasked coordinatexth≠mx\_\{t\}^\{h\}\\neq m, both the trueqt−1∣t,0\(⋅∣xt,x0\)q\_\{t\-1\\mid t,0\}\(\\cdot\\mid x\_\{t\},x\_\{0\}\)and the SUBS/Bayes/D3PM modelqθ\(⋅∣xt\)q\_\{\\theta\}\(\\cdot\\mid x\_\{t\}\)copyxthx\_\{t\}^\{h\}one step back, regardless of the network output\. Hence the KL divergence for that coordinate is zero\.

For a masked coordinatexth=mx\_\{t\}^\{h\}=m, plugging \([5](https://arxiv.org/html/2605.13999#A1.E5)\) and \([7](https://arxiv.org/html/2605.13999#A1.E7)\) into the KL gives, up to constants independent ofθ\\theta,

σt−σt−1σt​\[−log⁡nn^θ,h​\(x0h∣xt,t\)\]\.\\frac\{\\sigma\_\{t\}\-\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\\big\[\-\\log\\widehat\{\\mathrm\{nn\}\}\_\{\\theta,h\}\(x\_\{0\}^\{h\}\\mid x\_\{t\},t\)\\big\]\.Therefore the finite\-TTobjective reduces to

ℒT​\(θ\)=∑t=1Tσt−σt−1σt​𝔼\[∑h=1H𝟙​\{xth=m\}​\(−log⁡nn^θ,h​\(x0h∣xt,t\)\)\]\.\\displaystyle\\mathcal\{L\}\_\{T\}\(\\theta\)=\\sum\_\{t=1\}^\{T\}\\frac\{\\sigma\_\{t\}\-\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\\,\\operatorname\*\{\\mathbb\{E\}\}\\\!\\left\[\\sum\_\{h=1\}^\{H\}\\mathbbm\{1\}\\left\\\{x\_\{t\}^\{h\}=m\\right\\\}\\big\(\-\\log\\widehat\{\\mathrm\{nn\}\}\_\{\\theta,h\}\(x\_\{0\}^\{h\}\\mid x\_\{t\},t\)\\big\)\\right\]\.\(8\)This is the discrete\-time version\. TakingT→∞T\\to\\inftyturns the sum into the continuous\-time weighted masked\-token objective used by modern masked DLMs, with weightσ˙t/σt\\dot\{\\sigma\}\_\{t\}/\\sigma\_\{t\}whenσt\\sigma\_\{t\}denotes the cumulative mask probability\. Thus D3PM\-style posterior training, Bayes substitution, and mean/SUBS training collapse to the same clean\-token prediction objective in the absorbing\-mask case\.

#### A\.3Scores: direct parameterization versus induced score

Some other parameterizations in the literature are based on a continuous time formulation of the discrete\-space diffusion process\. In continuous time, there is no one\-step objectqt−1∣tq\_\{t\-1\\mid t\}until a time discretization is chosen\. The primitive object is instead a rate matrixRtR\_\{t\}that defines the infinitesimal forward transition probabilities:

qt\+δ​t∣t​\(y′∣y\)=\{Rt​\(y→y′\)​δ​t\+o​\(δ​t\),y′≠y,1\+Rt​\(y→y\)​δ​t\+o​\(δ​t\),y′=y\.q\_\{t\+\\delta t\\mid t\}\(y^\{\\prime\}\\mid y\)=\\begin\{cases\}R\_\{t\}\(y\\to y^\{\\prime\}\)\\,\\delta t\+o\(\\delta t\),&y^\{\\prime\}\\neq y,\\\\ 1\+R\_\{t\}\(y\\to y\)\\,\\delta t\+o\(\\delta t\),&y^\{\\prime\}=y\.\\end\{cases\}If the forward process has rate matrixRtR\_\{t\}, then the off\-diagonal reverse rates have the form

R~t​\(y→y′\)=Rt​\(y′→y\)​qt​\(y′\)qt​\(y\)\(y′≠y\)\.\\widetilde\{R\}\_\{t\}\(y\\to y^\{\\prime\}\)=R\_\{t\}\(y^\{\\prime\}\\to y\)\\,\\frac\{q\_\{t\}\(y^\{\\prime\}\)\}\{q\_\{t\}\(y\)\}\\qquad\(y^\{\\prime\}\\neq y\)\.Here the arrow iny→y′y\\to y^\{\\prime\}always means “current tokenyy, proposed replacementy′y^\{\\prime\}” for the process being discussed\. The forward rate in the reverse formula points in the opposite direction,y′→yy^\{\\prime\}\\to y, because a reverse move fromyytoy′y^\{\\prime\}undoes a forward move fromy′y^\{\\prime\}toyy\. For a full sequence, the same convention applies to neighboring sequences that differ by one token replacement\. The ratio

st​\(y→y′\):=qt​\(y′\)qt​\(y\)s\_\{t\}\(y\\to y^\{\\prime\}\):=\\frac\{q\_\{t\}\(y^\{\\prime\}\)\}\{q\_\{t\}\(y\)\}is called the discrete or concrete score in this literature\.

SEDD\[lou2023discrete\]directly parameterizes a positive score field and uses score entropy rather than a squared loss analogous to continuous\-space score matching\. In our arrow convention, the denoising score\-entropy loss has the form

ℒtSE​\(θ\)\\displaystyle\\mathcal\{L\}\_\{t\}^\{\\mathrm\{SE\}\}\(\\theta\)=𝔼x0∼pdataxt∼qt∣0\(⋅∣x0\)\[∑x~≠xtRt\(x~→xt\)\(sθ\(xt→x~∣xt,t\)\\displaystyle=\\operatorname\*\{\\mathbb\{E\}\}\_\{\\begin\{subarray\}\{c\}x\_\{0\}\\sim p\_\{\\text\{data\}\}\\\\ x\_\{t\}\\sim q\_\{t\\mid 0\}\(\\cdot\\mid x\_\{0\}\)\\end\{subarray\}\}\\left\[\\sum\_\{\\tilde\{x\}\\neq x\_\{t\}\}R\_\{t\}\(\\tilde\{x\}\\to x\_\{t\}\)\\left\(s\_\{\\theta\}\(x\_\{t\}\\to\\tilde\{x\}\\mid x\_\{t\},t\)\\right\.\\right\.−stx0\(xt→x~\)logsθ\(xt→x~∣xt,t\)\+Φ\(stx0\(xt→x~\)\)\)\]\.\\displaystyle\\hskip 80\.00012pt\\left\.\\left\.\-s\_\{t\}^\{x\_\{0\}\}\(x\_\{t\}\\to\\tilde\{x\}\)\\log s\_\{\\theta\}\(x\_\{t\}\\to\\tilde\{x\}\\mid x\_\{t\},t\)\+\\Phi\\\!\\left\(s\_\{t\}^\{x\_\{0\}\}\(x\_\{t\}\\to\\tilde\{x\}\)\\right\)\\right\)\\right\]\.\(9\)Here

stx0​\(x→x~\):=qt​\(x~∣x0\)qt​\(x∣x0\)andΦ​\(a\):=a​log⁡a−as\_\{t\}^\{x\_\{0\}\}\(x\\to\\tilde\{x\}\):=\\frac\{q\_\{t\}\(\\tilde\{x\}\\mid x\_\{0\}\)\}\{q\_\{t\}\(x\\mid x\_\{0\}\)\}\\qquad\\text\{and\}\\qquad\\Phi\(a\):=a\\log a\-aand theΦ\\Phiterm is independent ofθ\\theta\. As in denoising score matching, the loss uses the conditional scorestx0s\_\{t\}^\{x\_\{0\}\}as a tractable training target, while its population optimum is the marginal scorest​\(x→x~\)=qt​\(x~\)/qt​\(x\)s\_\{t\}\(x\\to\\tilde\{x\}\)=q\_\{t\}\(\\tilde\{x\}\)/q\_\{t\}\(x\)\.

The parameterization ofsθs\_\{\\theta\}is a modeling choice\. A direct score parameterization replaces the unknown score by a general positive function of the proposed replacement state\. Another option is to keepnn^θ,h\\widehat\{\\mathrm\{nn\}\}\_\{\\theta,h\}as an approximation ofq​\(x0h∣xt\)q\(x\_\{0\}^\{h\}\\mid x\_\{t\}\), and then re\-express the score in terms of this clean\-token posterior\. This gives an induced score parameterization\. In the absorbing case, this induced score is a deterministic rescaling of the same clean\-token posterior that appears in the Bayes/D3PM/SUBS reverse kernels, and the score\-entropy loss reduces, up toθ\\theta\-independent terms, to the same weighted cross\-entropy as \([8](https://arxiv.org/html/2605.13999#A1.E8)\)\.

Indeed, for mask diffusion, conditioning on a clean tokenu∈𝒱u\\in\\mathcal\{V\},

qt​\(v∣x0h=u\)=\(1−σt\)​𝟙​\{v=u\}\+σt​𝟙​\{v=m\}\.q\_\{t\}\(v\\mid x\_\{0\}^\{h\}=u\)=\(1\-\\sigma\_\{t\}\)\\mathbbm\{1\}\\left\\\{v=u\\right\\\}\+\\sigma\_\{t\}\\mathbbm\{1\}\\left\\\{v=m\\right\\\}\.Hence, for a move from mask to clean tokenjj,

st​\(m→j∣x0h=u\)=1−σtσt​𝟙​\{j=u\}\.\\displaystyle s\_\{t\}\(m\\to j\\mid x\_\{0\}^\{h\}=u\)=\\frac\{1\-\\sigma\_\{t\}\}\{\\sigma\_\{t\}\}\\mathbbm\{1\}\\left\\\{j=u\\right\\\}\.\(10\)After conditioning on the whole corrupted sequencextx\_\{t\}, this becomes

st​\(m→j∣xt\)=1−σtσt​q​\(x0h=j∣xt\)\.\\displaystyle s\_\{t\}\(m\\to j\\mid x\_\{t\}\)=\\frac\{1\-\\sigma\_\{t\}\}\{\\sigma\_\{t\}\}\\,q\(x\_\{0\}^\{h\}=j\\mid x\_\{t\}\)\.\(11\)Replacing the clean\-token posterior by the network output gives the induced score

sθ,tind​\(m→j∣xt\)=1−σtσt​nn^θ,h​\(j∣xt,t\),j∈𝒱\.\\displaystyle s\_\{\\theta,t\}^\{\\mathrm\{ind\}\}\(m\\to j\\mid x\_\{t\}\)=\\frac\{1\-\\sigma\_\{t\}\}\{\\sigma\_\{t\}\}\\,\\widehat\{\\mathrm\{nn\}\}\_\{\\theta,h\}\(j\\mid x\_\{t\},t\),\\qquad j\\in\\mathcal\{V\}\.\(12\)Thus, in the masked case, the relevant unmasking score is a deterministic rescaling of the same clean\-token posterior that appeared in the reverse\-kernel parameterizations\. This induced score is normalized:

∑j∈𝒱sθ,tind​\(m→j∣xt\)=1−σtσt\.\\sum\_\{j\\in\\mathcal\{V\}\}s\_\{\\theta,t\}^\{\\mathrm\{ind\}\}\(m\\to j\\mid x\_\{t\}\)=\\frac\{1\-\\sigma\_\{t\}\}\{\\sigma\_\{t\}\}\.Plugging \([12](https://arxiv.org/html/2605.13999#A1.E12)\) into the score\-entropy loss \([9](https://arxiv.org/html/2605.13999#A1.E9)\) leaves, up to constants independent ofθ\\theta,

σ˙tσt​1​\{xth=m\}​\[−log⁡nn^θ,h​\(x0h∣xt,t\)\],\\frac\{\\dot\{\\sigma\}\_\{t\}\}\{\\sigma\_\{t\}\}\\,\\mathbbm\{1\}\\left\\\{x\_\{t\}^\{h\}=m\\right\\\}\\big\[\-\\log\\widehat\{\\mathrm\{nn\}\}\_\{\\theta,h\}\(x\_\{0\}^\{h\}\\mid x\_\{t\},t\)\\big\],the continuous\-time analogue of \([8](https://arxiv.org/html/2605.13999#A1.E8)\)\. In contrast, a direct score parameterization such as SEDD replaces the unknown score by a general positive field and optimizes score entropy directly; it is not constrained to lie on the normalized clean\-token manifold above, although the optimum recovers the same marginal score when the model class is rich enough\. This is summarized in[Figure˜4](https://arxiv.org/html/2605.13999#A1.F4)\.

#### A\.4Recipe used in the real\-data experiment

The FineWeb experiment in[Section˜3\.1](https://arxiv.org/html/2605.13999#S3.SS1)uses the standard absorbing\-mask, clean\-token\-prediction family summarized above: a DDiT\-style denoiser, continuous\-time log\-linear mask noise, no explicit time conditioning, and the mean/SUBS masked\-token loss\. Sampling is then done with a finite discretization of the same absorbing reverse kernel\.

### Appendix BDeferred Proofs for[Section˜2](https://arxiv.org/html/2605.13999#S2)

#### B\.1Coarse multiplicative accuracy does not imply distributional accuracy

The next example illustrates that the multiplicative control used in[Corollary˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmcorollary1)is genuinely weaker than distributional closeness\. In particular, a learned kernel may preserve the correctσ\\sigma\-scale hierarchy while remaining far from the true kernel in additive metrics such as total variation or KL divergence\.

###### Example B\.1\(Scale accuracy need not imply distributional accuracy\)\.

Fix anyα∈\(0,1/2\)\\alpha\\in\(0,1/2\)\. Forσ∈\(0,1/4\)\\sigma\\in\(0,1/4\), define probability vectors on two points by

qσ=\(12−σ,12\+σ\),pσ=\(σα​\(12−σ\),1−σα​\(12−σ\)\)\.q\_\{\\sigma\}=\\left\(\\frac\{1\}\{2\}\-\\sigma,\\,\\frac\{1\}\{2\}\+\\sigma\\right\),\\qquad p\_\{\\sigma\}=\\left\(\\sigma^\{\\alpha\}\\left\(\\frac\{1\}\{2\}\-\\sigma\\right\),\\,1\-\\sigma^\{\\alpha\}\\left\(\\frac\{1\}\{2\}\-\\sigma\\right\)\\right\)\.Both vectors have full support for everyσ\>0\\sigma\>0\. Moreover, their symmetric multiplicative distortion satisfies

maxi⁡max⁡\{pσ,iqσ,i,qσ,ipσ,i\}=Θ​\(σ−α\)=o​\(σ−1/2\),\\max\_\{i\}\\max\\left\\\{\\frac\{p\_\{\\sigma,i\}\}\{q\_\{\\sigma,i\}\},\\frac\{q\_\{\\sigma,i\}\}\{p\_\{\\sigma,i\}\}\\right\\\}=\\Theta\(\\sigma^\{\-\\alpha\}\)=o\(\\sigma^\{\-1/2\}\),sinceα<1/2\\alpha<1/2\. Thuspσp\_\{\\sigma\}andqσq\_\{\\sigma\}obey the same type of coarse multiplicative control required in[Corollary˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmcorollary1)\.

However, this control does not imply additive closeness\. Indeed,

‖pσ−qσ‖1=1\+o​\(1\),TV​\(pσ,qσ\)=12\+o​\(1\)\.\\\|p\_\{\\sigma\}\-q\_\{\\sigma\}\\\|\_\{1\}=1\+o\(1\),\\qquad\\mathrm\{TV\}\(p\_\{\\sigma\},q\_\{\\sigma\}\)=\\frac\{1\}\{2\}\+o\(1\)\.Consequently, by Pinsker’s inequality,

KL​\(pσ∥qσ\)≥2​TV​\(pσ,qσ\)2=12\+o​\(1\),\\mathrm\{KL\}\(p\_\{\\sigma\}\\\|q\_\{\\sigma\}\)\\geq 2\\,\\mathrm\{TV\}\(p\_\{\\sigma\},q\_\{\\sigma\}\)^\{2\}=\\frac\{1\}\{2\}\+o\(1\),and the same lower bound also holds forKL​\(qσ∥pσ\)\\mathrm\{KL\}\(q\_\{\\sigma\}\\\|p\_\{\\sigma\}\)\. Hence coarse multiplicative scale accuracy can be sufficient for recovering the order\-of\-magnitude structure of the scores, while still being far too weak to guarantee distributional calibration\.

#### B\.2Proof of[Theorem˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmtheorem1)

We first prove the following auxiliary lemma\.

###### Lemma B\.1\.

For the uniform diffusion processqq, fort∈\[T\]t\\in\[T\]andx∈𝒳x\\in\\mathcal\{X\}, we have

qt​\(x\)=\\displaystyle q\_\{t\}\(x\)=pdata​\(projD​\(x\)\)​\(σtK\)d​\(x,D\)\+O​\(\(σt/K\)d​\(x,D\)\+1\)\\displaystyle p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\\left\(\{\\frac\{\\sigma\_\{t\}\}\{K\}\}\\right\)^\{d\(x,D\)\}\+O\(\(\\sigma\_\{t\}/K\)^\{d\(x,D\)\+1\}\)asσt→0\\sigma\_\{t\}\\to 0\.

###### Proof\.

For anyxx, we have

qt​\(x\)=\\displaystyle q\_\{t\}\(x\)=∑x0∈Dℙ​\[X0=x0\]⋅ℙ​\[Xt=x\|X0=x0\],\\displaystyle\\sum\_\{x\_\{0\}\\in D\}\\mathbb\{P\}\[X\_\{0\}=x\_\{0\}\]\\cdot\\mathbb\{P\}\[X\_\{t\}=x\|X\_\{0\}=x\_\{0\}\],whereℙ​\[X0=x0\]=pdata​\(x0\)\\mathbb\{P\}\[X\_\{0\}=x\_\{0\}\]=p\_\{\\text\{data\}\}\(x\_\{0\}\)and, by definition of the forward process,

ℙ​\[Xt=x\|X0=x0\]=\\displaystyle\\mathbb\{P\}\[X\_\{t\}=x\|X\_\{0\}=x\_\{0\}\]=∏i=1H\(\(1−σt\)​𝟙​\{xi=x0i\}\+σtK\)\\displaystyle\\prod\_\{i=1\}^\{H\}\\left\(\{\(1\-\\sigma\_\{t\}\)\\mathbbm\{1\}\\left\\\{x^\{i\}=x\_\{0\}^\{i\}\\right\\\}\+\\frac\{\\sigma\_\{t\}\}\{K\}\}\\right\)=\\displaystyle=∏i=1H\(\(1−σt​K−1K\)​𝟙​\{xi=x0i\}\+σtK​𝟙​\{xi≠x0i\}\)\\displaystyle\\prod\_\{i=1\}^\{H\}\\left\(\{\\left\(\{1\-\\sigma\_\{t\}\\frac\{K\-1\}\{K\}\}\\right\)\\mathbbm\{1\}\\left\\\{x^\{i\}=x\_\{0\}^\{i\}\\right\\\}\+\\frac\{\\sigma\_\{t\}\}\{K\}\\mathbbm\{1\}\\left\\\{x^\{i\}\\neq x\_\{0\}^\{i\}\\right\\\}\}\\right\)=\\displaystyle=\(1−σt​K−1K\)H−d​\(x,x0\)​\(σtK\)d​\(x,x0\)\.\\displaystyle\\left\(\{1\-\\sigma\_\{t\}\\frac\{K\-1\}\{K\}\}\\right\)^\{H\-d\(x,x\_\{0\}\)\}\\left\(\{\\frac\{\\sigma\_\{t\}\}\{K\}\}\\right\)^\{d\(x,x\_\{0\}\)\}\.Hence rearranging the sum by the orderd​\(x,x0\)d\(x,x\_\{0\}\)ofσt/K\\sigma\_\{t\}/K,

qt​\(x\)=\\displaystyle q\_\{t\}\(x\)=∑x0∈Dpdata​\(x0\)​\(1−σt​K−1K\)H−d​\(x,x0\)​\(σtK\)d​\(x,x0\)\\displaystyle\\sum\_\{x\_\{0\}\\in D\}p\_\{\\text\{data\}\}\(x\_\{0\}\)\\left\(\{1\-\\sigma\_\{t\}\\frac\{K\-1\}\{K\}\}\\right\)^\{H\-d\(x,x\_\{0\}\)\}\\left\(\{\\frac\{\\sigma\_\{t\}\}\{K\}\}\\right\)^\{d\(x,x\_\{0\}\)\}=\\displaystyle=∑d=0H\(∑x0∈D:d​\(x,x0\)=dpdata​\(x0\)\)​\(1−σt​K−1K\)H−d​\(σtK\)d\\displaystyle\\sum\_\{d=0\}^\{H\}\\left\(\{\\sum\_\{\\begin\{subarray\}\{c\}x\_\{0\}\\in D\\colon\\\\ d\(x,x\_\{0\}\)=d\\end\{subarray\}\}p\_\{\\text\{data\}\}\(x\_\{0\}\)\}\\right\)\\left\(\{1\-\\sigma\_\{t\}\\frac\{K\-1\}\{K\}\}\\right\)^\{H\-d\}\\left\(\{\\frac\{\\sigma\_\{t\}\}\{K\}\}\\right\)^\{d\}=\\displaystyle=∑d=d​\(x,D\)H\(∑x0∈D:d​\(x,x0\)=dpdata​\(x0\)\)​\(1−σt​K−1K\)H−d​\(σtK\)d,\\displaystyle\\sum\_\{d=d\(x,D\)\}^\{H\}\\left\(\{\\sum\_\{\\begin\{subarray\}\{c\}x\_\{0\}\\in D\\colon\\\\ d\(x,x\_\{0\}\)=d\\end\{subarray\}\}p\_\{\\text\{data\}\}\(x\_\{0\}\)\}\\right\)\\left\(\{1\-\\sigma\_\{t\}\\frac\{K\-1\}\{K\}\}\\right\)^\{H\-d\}\\left\(\{\\frac\{\\sigma\_\{t\}\}\{K\}\}\\right\)^\{d\},where we used thatd​\(x,x0\)≥d​\(x,D\)d\(x,x\_\{0\}\)\\geq d\(x,D\)for allx0∈Dx\_\{0\}\\in Din the last line\. Now whenσt→0\\sigma\_\{t\}\\to 0, we find that the lowest order term is

\(∑x0∈D:d​\(x,x0\)=d​\(x,D\)pdata​\(x0\)\)​\(σtK\)d​\(x,D\)\\displaystyle\\bigg\(\\sum\_\{\\begin\{subarray\}\{c\}x\_\{0\}\\in D\\colon\\\\ d\(x,x\_\{0\}\)=d\(x,D\)\\end\{subarray\}\}p\_\{\\text\{data\}\}\(x\_\{0\}\)\\bigg\)\\left\(\{\\frac\{\\sigma\_\{t\}\}\{K\}\}\\right\)^\{d\(x,D\)\}and all other terms are of order at leastd​\(x,D\)\+1d\(x,D\)\+1and thus contribute at mostO​\(\(σt/K\)d​\(x,D\)\+1\)O\(\(\\sigma\_\{t\}/K\)^\{d\(x,D\)\+1\}\)\. ∎

We are now ready to proceed with the proof of[Theorem˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmtheorem1)\.

###### Proof\.

\([Theorem˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmtheorem1)\) For ease of notation, we setx:=xtx:=x\_\{t\}andz:=xt−1=\(xt−h,yh\)z:=x\_\{t\-1\}=\(x\_\{t\}^\{\-h\},y^\{h\}\)\. We have

qt−1\|th​\(yh\|x\)=∑y−h∈\[K\]H−1qt−1\|t​\(yh\|x\)=∑y−h∈\[K\]H−1qt−1​\(y\)qt​\(x\)​qt\|t−1​\(x\|y\)\\displaystyle q\_\{t\-1\|t\}^\{h\}\(y^\{h\}\|x\)=\\sum\_\{y^\{\-h\}\\in\[K\]^\{H\-1\}\}q\_\{t\-1\|t\}\(y^\{h\}\|x\)=\\sum\_\{y^\{\-h\}\\in\[K\]^\{H\-1\}\}\\frac\{q\_\{t\-1\}\(y\)\}\{q\_\{t\}\(x\)\}q\_\{t\|t\-1\}\(x\|y\)by the law of total probability and Bayes’ law\. For the conditional term on the RHS we know that \(sincexth≠yhx^\{h\}\_\{t\}\\neq y^\{h\}\)

qt\|t−1​\(x\|y\)=\(1−βt​K−1K\)H−1−d​\(x−h,y−h\)​\(βtK\)d​\(x−h,y−h\)⋅βtK\.\\displaystyle q\_\{t\|t\-1\}\(x\|y\)=\\left\(\{1\-\\beta\_\{t\}\\frac\{K\-1\}\{K\}\}\\right\)^\{H\-1\-d\(x^\{\-h\},y^\{\-h\}\)\}\\left\(\{\\frac\{\\beta\_\{t\}\}\{K\}\}\\right\)^\{d\(x^\{\-h\},y^\{\-h\}\)\}\\cdot\\frac\{\\beta\_\{t\}\}\{K\}\.Plugging this into the previous equation and rearranging by the order in terms ofβt\\beta\_\{t\}reveals

qt−1\|th​\(yh\|x\)=\\displaystyle q\_\{t\-1\|t\}^\{h\}\(y^\{h\}\|x\)=∑y−h∈\[K\]H−1qt−1​\(y\)qt​\(x\)⋅\(1−βt​K−1K\)H−1−d​\(x−h,y−h\)​\(βtK\)d​\(x−h,y−h\)\+1\\displaystyle\\sum\_\{y^\{\-h\}\\in\[K\]^\{H\-1\}\}\\frac\{q\_\{t\-1\}\(y\)\}\{q\_\{t\}\(x\)\}\\cdot\\left\(\{1\-\\beta\_\{t\}\\frac\{K\-1\}\{K\}\}\\right\)^\{H\-1\-d\(x^\{\-h\},y^\{\-h\}\)\}\\left\(\{\\frac\{\\beta\_\{t\}\}\{K\}\}\\right\)^\{d\(x^\{\-h\},y^\{\-h\}\)\+1\}=\\displaystyle=∑d=0H−1∑y−h∈\[K\]H−1d​\(x−h,y−h\)=dqt−1​\(y\)qt​\(x\)⋅\(1−βt​K−1K\)H−1−d​\(βtK\)d\+1⏟=⁣:Cd\.\\displaystyle\\sum\_\{d=0\}^\{H\-1\}\\underbrace\{\\sum\_\{\\begin\{subarray\}\{c\}y^\{\-h\}\\in\[K\]^\{H\-1\}\\\\ d\(x^\{\-h\},y^\{\-h\}\)=d\\end\{subarray\}\}\\frac\{q\_\{t\-1\}\(y\)\}\{q\_\{t\}\(x\)\}\\cdot\\left\(\{1\-\\beta\_\{t\}\\frac\{K\-1\}\{K\}\}\\right\)^\{H\-1\-d\}\\left\(\{\\frac\{\\beta\_\{t\}\}\{K\}\}\\right\)^\{d\+1\}\}\_\{=:C\_\{d\}\}\.\(13\)Now invoking[Lemma˜B\.1](https://arxiv.org/html/2605.13999#A2.Thmlemma1)and using thatprojD​\(x\)=\{y∈D∣d​\(x,y\)=d​\(x,D\)\}\\text\{proj\}\_\{D\}\(x\)=\\\{y\\in D\\mid d\(x,y\)=d\(x,D\)\\\}, we can approximate the ratioqt−1​\(y\)qt​\(x\)\\frac\{q\_\{t\-1\}\(y\)\}\{q\_\{t\}\(x\)\}as

qt−1​\(y\)qt​\(x\)=\\displaystyle\\frac\{q\_\{t\-1\}\(y\)\}\{q\_\{t\}\(x\)\}=pdata​\(projD​\(y\)\)​\(σt−1K\)d​\(y,D\)​\(1\+O​\(σt−1K\)\)pdata​\(projD​\(x\)\)​\(σtK\)d​\(x,D\)​\(1\+O​\(σtK\)\)\\displaystyle\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\\left\(\{\\frac\{\\sigma\_\{t\-1\}\}\{K\}\}\\right\)^\{d\(y,D\)\}\\left\(\{1\+O\(\\frac\{\\sigma\_\{t\-1\}\}\{K\}\)\}\\right\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\\left\(\{\\frac\{\\sigma\_\{t\}\}\{K\}\}\\right\)^\{d\(x,D\)\}\\left\(\{1\+O\(\\frac\{\\sigma\_\{t\}\}\{K\}\)\}\\right\)\}=\\displaystyle=pdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​\(σtK\)d​\(y,D\)−d​\(x,D\)⋅\(σt−1σt\)d​\(y,D\)​\(1\+O​\(σtK\)\)\.\\displaystyle\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\left\(\{\\frac\{\\sigma\_\{t\}\}\{K\}\}\\right\)^\{d\(y,D\)\-d\(x,D\)\}\\cdot\\left\(\{\\frac\{\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\}\\right\)^\{d\(y,D\)\}\\left\(\{1\+O\\left\(\{\\frac\{\\sigma\_\{t\}\}\{K\}\}\\right\)\}\\right\)\.\(14\)We now consider the different termsCdC\_\{d\}in the sum in[Equation˜13](https://arxiv.org/html/2605.13999#A2.E13)by plugging in the ratio from[Equation˜14](https://arxiv.org/html/2605.13999#A2.E14):

Lowest order:𝐝=𝟎\\mathbf\{d=0\}\.We haved​\(x−h,y−h\)=0d\(x^\{\-h\},y^\{\-h\}\)=0, soy=\(x−h,yh\)y=\(x^\{\-h\},y^\{h\}\)\. Hence, the only term here is

C0=pdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​\(σtK\)d​\(y,D\)−d​\(x,D\)⋅\(σt−1σt\)d​\(y,D\)​\(1\+O​\(σtK\)\)​\(1−βt​K−1K\)H−1​\(βtK\)\.\\displaystyle C\_\{0\}=\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\left\(\{\\frac\{\\sigma\_\{t\}\}\{K\}\}\\right\)^\{d\(y,D\)\-d\(x,D\)\}\\cdot\\left\(\{\\frac\{\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\}\\right\)^\{d\(y,D\)\}\\left\(\{1\+O\\left\(\{\\frac\{\\sigma\_\{t\}\}\{K\}\}\\right\)\}\\right\)\\left\(\{1\-\\beta\_\{t\}\\frac\{K\-1\}\{K\}\}\\right\)^\{H\-1\}\\left\(\{\\frac\{\\beta\_\{t\}\}\{K\}\}\\right\)\.By[˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmassumption1), we haveσt−1σt→1\\frac\{\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\\to 1and1−βt​K−1K→11\-\\beta\_\{t\}\\frac\{K\-1\}\{K\}\\to 1, hence

Cdβt/K→pdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​\(σK\)d​\(y,D\)−d​\(x,D\)\+O​\(\(σK\)d​\(y,D\)−d​\(x,D\)\+1\)\\displaystyle\\frac\{C\_\{d\}\}\{\\beta\_\{t\}/K\}\\to\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\left\(\{\\frac\{\\sigma\}\{K\}\}\\right\)^\{d\(y,D\)\-d\(x,D\)\}\+O\\left\(\{\\left\(\{\\frac\{\\sigma\}\{K\}\}\\right\)^\{d\(y,D\)\-d\(x,D\)\+1\}\}\\right\)asT→∞T\\to\\infty\.

Higher orders:𝐝\>𝟎\\mathbf\{d\>0\}\.In this case, we have

Cdβt/K=∑y−h∈\[K\]H−1d​\(x−h,y−h\)=dqt−1​\(y\)qt​\(x\)⋅\(1−βt​K−1K\)H−1−d​\(βtK\)d→0\\displaystyle\\frac\{C\_\{d\}\}\{\\beta\_\{t\}/K\}=\\sum\_\{\\begin\{subarray\}\{c\}y^\{\-h\}\\in\[K\]^\{H\-1\}\\\\ d\(x^\{\-h\},y^\{\-h\}\)=d\\end\{subarray\}\}\\frac\{q\_\{t\-1\}\(y\)\}\{q\_\{t\}\(x\)\}\\cdot\\left\(\{1\-\\beta\_\{t\}\\frac\{K\-1\}\{K\}\}\\right\)^\{H\-1\-d\}\\left\(\{\\frac\{\\beta\_\{t\}\}\{K\}\}\\right\)^\{d\}\\to 0since\(βt/K\)d→0\(\\beta\_\{t\}/K\)^\{d\}\\to 0ford≥1d\\geq 1, while1−βt​K−1K→11\-\\beta\_\{t\}\\frac\{K\-1\}\{K\}\\to 1andqt−1​\(y\)qt​\(x\)\\frac\{q\_\{t\-1\}\(y\)\}\{q\_\{t\}\(x\)\}allows for the analogous upper bound independent ofTTas in thed=0d=0case by using[Equation˜14](https://arxiv.org/html/2605.13999#A2.E14)\.

Hence, we conclude that

limT→∞qt−1\|th​\(yh\|x\)βt/K=pdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​\(σK\)d​\(y,D\)−d​\(x,D\)\+O​\(\(σK\)d​\(y,D\)−d​\(x,D\)\+1\),\\displaystyle\\lim\_\{T\\to\\infty\}\\frac\{q^\{h\}\_\{t\-1\|t\}\(y^\{h\}\|x\)\}\{\\beta\_\{t\}/K\}=\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\left\(\{\\frac\{\\sigma\}\{K\}\}\\right\)^\{d\(y,D\)\-d\(x,D\)\}\+O\\left\(\{\\left\(\{\\frac\{\\sigma\}\{K\}\}\\right\)^\{d\(y,D\)\-d\(x,D\)\+1\}\}\\right\),wherey=\(x−h,yh\)y=\(x^\{\-h\},y^\{h\}\)\. A case distinction with respect tod​\(y,D\)∈\{d​\(x,D\)−1,d​\(x,D\),d​\(x,D\)\+1\}d\(y,D\)\\in\\\{d\(x,D\)\-1,d\(x,D\),d\(x,D\)\+1\\\}now shows the claim\. ∎

#### B\.3Proof of[Theorem˜2\.2](https://arxiv.org/html/2605.13999#S2.Thmtheorem2)

We first prove the following auxiliary lemma, which transfers the expansion of the marginal at timettin[Lemma˜B\.1](https://arxiv.org/html/2605.13999#A2.Thmlemma1)to absorbing diffusion\.

###### Lemma B\.2\.

For the absorbing diffusion processqq, fort∈\[T\]t\\in\[T\]andx∈𝒳x\\in\\mathcal\{X\}, we have

qt​\(x\)=\\displaystyle q\_\{t\}\(x\)=\{pdata​\(projD​\(x\)\)​σtd​\(x,D\)\+O​\(σtd​\(x,D\)\+1\)if​∃x0∈D:∀h∈\[H\]:\(xh≠m⇒xh=x0h\)0else\\displaystyle\\begin\{dcases\}p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\\sigma\_\{t\}^\{d\(x,D\)\}\+O\(\\sigma\_\{t\}^\{d\(x,D\)\+1\}\)\\qquad&\\text\{if \}\\exists x\_\{0\}\\in D\\colon\\forall h\\in\[H\]\\colon\(x^\{h\}\\neq m\\Rightarrow x^\{h\}=x\_\{0\}^\{h\}\)\\\\\[5\.69054pt\] 0\\qquad&\\text\{else\}\\end\{dcases\}asσt→0\\sigma\_\{t\}\\to 0\.

Note that we are in the first of the two cases if and only ifx∈supp​\(qt\)x\\in\\text\{supp\}\(q\_\{t\}\)\.

###### Proof\.

For anyxx, we have

qt​\(x\)=\\displaystyle q\_\{t\}\(x\)=∑x0∈Dℙ​\[X0=x0\]⋅ℙ​\[Xt=x\|X0=x0\],\\displaystyle\\sum\_\{x\_\{0\}\\in D\}\\mathbb\{P\}\[X\_\{0\}=x\_\{0\}\]\\cdot\\mathbb\{P\}\[X\_\{t\}=x\|X\_\{0\}=x\_\{0\}\],whereℙ​\[X0=x0\]=pdata​\(x0\)\\mathbb\{P\}\[X\_\{0\}=x\_\{0\}\]=p\_\{\\text\{data\}\}\(x\_\{0\}\)and, by definition of the forward process,

ℙ​\[Xt=x\|X0=x0\]=\\displaystyle\\mathbb\{P\}\[X\_\{t\}=x\|X\_\{0\}=x\_\{0\}\]=∏i=1H\(\(1−σt\)​𝟙​\{xi=x0i\}\+σt​𝟙​\{xi=m\}\)\\displaystyle\\prod\_\{i=1\}^\{H\}\\left\(\{\(1\-\\sigma\_\{t\}\)\\mathbbm\{1\}\\left\\\{x^\{i\}=x\_\{0\}^\{i\}\\right\\\}\+\\sigma\_\{t\}\\mathbbm\{1\}\\left\\\{x^\{i\}=m\\right\\\}\}\\right\)=\\displaystyle=∏i=1H\(\(1−σt\)​𝟙​\{xi=x0i\}\+σt​𝟙​\{xi=m\}\)\\displaystyle\\prod\_\{i=1\}^\{H\}\\left\(\{\(1\-\\sigma\_\{t\}\)\\mathbbm\{1\}\\left\\\{x^\{i\}=x\_\{0\}^\{i\}\\right\\\}\+\\sigma\_\{t\}\\mathbbm\{1\}\\left\\\{x^\{i\}=m\\right\\\}\}\\right\)=\\displaystyle=\(1−σt\)H−d​\(x,x0\)​σtd​\(x,x0\)​𝟙​\{∀i:xi∈\{x0i,m\}\}\.\\displaystyle\\left\(\{1\-\\sigma\_\{t\}\}\\right\)^\{H\-d\(x,x\_\{0\}\)\}\\sigma\_\{t\}^\{d\(x,x\_\{0\}\)\}\\mathbbm\{1\}\\left\\\{\\forall i\\colon x^\{i\}\\in\\\{x\_\{0\}^\{i\},m\\\}\\right\\\}\.Hence rearranging the sum by the orderd​\(x,x0\)d\(x,x\_\{0\}\)ofσt\\sigma\_\{t\},

qt​\(x\)=\\displaystyle q\_\{t\}\(x\)=∑x0∈D:∀h:xh∈\{x0h,m\}pdata​\(x0\)​\(1−σt\)H−d​\(x,x0\)​σtd​\(x,x0\)\\displaystyle\\sum\_\{\\begin\{subarray\}\{c\}x\_\{0\}\\in D\\colon\\\\ \\forall h\\colon x^\{h\}\\in\\\{x\_\{0\}^\{h\},m\\\}\\end\{subarray\}\}p\_\{\\text\{data\}\}\(x\_\{0\}\)\\left\(\{1\-\\sigma\_\{t\}\}\\right\)^\{H\-d\(x,x\_\{0\}\)\}\\sigma\_\{t\}^\{d\(x,x\_\{0\}\)\}=\\displaystyle=∑d=0H\(∑x0∈D:∀h:xh∈\{x0h,m\},d​\(x,x0\)=dpdata​\(x0\)\)​\(1−σt\)H−d​σtd\\displaystyle\\sum\_\{d=0\}^\{H\}\\left\(\{\\sum\_\{\\begin\{subarray\}\{c\}x\_\{0\}\\in D\\colon\\\\ \\forall h\\colon x^\{h\}\\in\\\{x\_\{0\}^\{h\},m\\\},\\\\ d\(x,x\_\{0\}\)=d\\end\{subarray\}\}p\_\{\\text\{data\}\}\(x\_\{0\}\)\}\\right\)\\left\(\{1\-\\sigma\_\{t\}\}\\right\)^\{H\-d\}\\sigma\_\{t\}^\{d\}Now whenσt→0\\sigma\_\{t\}\\to 0, we find that the lowest order non\-vanishing term is of order

d0:=\\displaystyle d\_\{0\}:=min⁡\{d​\(x,x0\)∣x0∈D,∀h∈\[H\]:\(xh=m∨xh=x0h\)\}\\displaystyle\\min\\\{d\(x,x\_\{0\}\)\\mid x\_\{0\}\\in D,~\\forall h\\in\[H\]\\colon\(x^\{h\}=m\\vee x^\{h\}=x\_\{0\}^\{h\}\)\\\}=\\displaystyle=\{\|\{h∈\[H\]:xh=m\}\|=d​\(x,D\)if​∃x0∈D:∀i:\(xh≠m⇒xh=x0h\)\+∞else\\displaystyle\\begin\{cases\}\|\\\{h\\in\[H\]\\colon x\_\{h\}=m\\\}\|=d\(x,D\)\\qquad&\\text\{if \}\\exists x\_\{0\}\\in D\\colon\\forall i\\colon\(x^\{h\}\\neq m\\Rightarrow x^\{h\}=x\_\{0\}^\{h\}\)\\\\ \+\\infty\\qquad&\\text\{else\}\\end\{cases\}since anyx0∈Dx\_\{0\}\\in Dhas no masked tokens\. Henceqt​\(x\)=0q\_\{t\}\(x\)=0in the second of the above cases\. Otherwise, in the first case, the lowest order term is

\(∑x0∈D:d​\(x,x0\)=d​\(x,D\)pdata​\(x0\)\)​σtd​\(x,D\)\\displaystyle\\bigg\(\\sum\_\{\\begin\{subarray\}\{c\}x\_\{0\}\\in D\\colon\\\\ d\(x,x\_\{0\}\)=d\(x,D\)\\end\{subarray\}\}p\_\{\\text\{data\}\}\(x\_\{0\}\)\\bigg\)\\sigma\_\{t\}^\{d\(x,D\)\}and all other terms are of order at leastd​\(x,D\)\+1d\(x,D\)\+1and thus contribute at mostO​\(σtd​\(x,D\)\+1\)O\(\\sigma\_\{t\}^\{d\(x,D\)\+1\}\)\. ∎

##### Proof of[Theorem˜2\.2](https://arxiv.org/html/2605.13999#S2.Thmtheorem2)\.

###### Proof\.

For ease of notation, we setx:=xtx:=x\_\{t\}andz:=xt−1=\(xt−h,yh\)z:=x\_\{t\-1\}=\(x\_\{t\}^\{\-h\},y^\{h\}\)\. We have

qt−1\|th​\(yh\|x\)=∑y−h∈\[K\]H−1qt−1\|t​\(yh\|x\)=∑y−h∈\[K\]H−1:∀i:yi=xi∨xi=mqt−1​\(y\)qt​\(x\)​qt\|t−1​\(x\|y\)\\displaystyle q\_\{t\-1\|t\}^\{h\}\(y^\{h\}\|x\)=\\sum\_\{y^\{\-h\}\\in\[K\]^\{H\-1\}\}q\_\{t\-1\|t\}\(y^\{h\}\|x\)=\\sum\_\{\\begin\{subarray\}\{c\}y^\{\-h\}\\in\[K\]^\{H\-1\}\\colon\\\\ \\forall i\\colon y^\{i\}=x^\{i\}\\vee x^\{i\}=m\\end\{subarray\}\}\\frac\{q\_\{t\-1\}\(y\)\}\{q\_\{t\}\(x\)\}q\_\{t\|t\-1\}\(x\|y\)by the law of total probability and Bayes’ law\. For the conditional term on the RHS we know that \(sincexth=mx^\{h\}\_\{t\}=m,yh≠my^\{h\}\\neq mand∀i≠h:yi=xi∨xi=m\\forall i\\neq h\\colon y^\{i\}=x^\{i\}\\vee x^\{i\}=m\)

qt\|t−1​\(x\|y\)=\(1−βt\)H−1−d​\(x−h,y−h\)​βtd​\(x−h,y−h\)⋅βt\.\\displaystyle q\_\{t\|t\-1\}\(x\|y\)=\\left\(\{1\-\\beta\_\{t\}\}\\right\)^\{H\-1\-d\(x^\{\-h\},y^\{\-h\}\)\}\\beta\_\{t\}^\{d\(x^\{\-h\},y^\{\-h\}\)\}\\cdot\\beta\_\{t\}\.Plugging this into the previous equation and rearranging by the order in terms ofβt\\beta\_\{t\}reveals

qt−1\|th​\(yh\|x\)=\\displaystyle q\_\{t\-1\|t\}^\{h\}\(y^\{h\}\|x\)=∑y−h∈\[K\]H−1:∀i:yi=xi∨xi=mqt−1​\(y\)qt​\(x\)⋅\(1−βt\)H−1−d​\(x−h,y−h\)​βtd​\(x−h,y−h\)\+1\\displaystyle\\sum\_\{\\begin\{subarray\}\{c\}y^\{\-h\}\\in\[K\]^\{H\-1\}\\colon\\\\ \\forall i\\colon y^\{i\}=x^\{i\}\\vee x^\{i\}=m\\end\{subarray\}\}\\frac\{q\_\{t\-1\}\(y\)\}\{q\_\{t\}\(x\)\}\\cdot\\left\(\{1\-\\beta\_\{t\}\}\\right\)^\{H\-1\-d\(x^\{\-h\},y^\{\-h\}\)\}\\beta\_\{t\}^\{d\(x^\{\-h\},y^\{\-h\}\)\+1\}=\\displaystyle=∑d=0H−1∑y−h∈\[K\]H−1:∀i:yi=xi∨xi=m,d​\(x−h,y−h\)=dqt−1​\(y\)qt​\(x\)⋅\(1−βt\)H−1−d​βtd\+1⏟=⁣:Cd\.\\displaystyle\\sum\_\{d=0\}^\{H\-1\}\\underbrace\{\\sum\_\{\\begin\{subarray\}\{c\}y^\{\-h\}\\in\[K\]^\{H\-1\}\\colon\\\\ \\forall i\\colon y^\{i\}=x^\{i\}\\vee x^\{i\}=m,\\\\ d\(x^\{\-h\},y^\{\-h\}\)=d\\end\{subarray\}\}\\frac\{q\_\{t\-1\}\(y\)\}\{q\_\{t\}\(x\)\}\\cdot\\left\(\{1\-\\beta\_\{t\}\}\\right\)^\{H\-1\-d\}\\beta\_\{t\}^\{d\+1\}\}\_\{=:C\_\{d\}\}\.\(15\)Consider each term in the sum in[Equation˜15](https://arxiv.org/html/2605.13999#A2.E15)\. Invoking[Lemma˜B\.2](https://arxiv.org/html/2605.13999#A2.Thmlemma2), we can see thatqt−1​\(y\)qt​\(x\)=0\\frac\{q\_\{t\-1\}\(y\)\}\{q\_\{t\}\(x\)\}=0if there is nox0∈Dx\_\{0\}\\in Dsuch that∀i:yi=m∨yi=x0i\\forall i\\colon y^\{i\}=m\\vee y^\{i\}=x\_\{0\}^\{i\}\(second case in[Lemma˜B\.2](https://arxiv.org/html/2605.13999#A2.Thmlemma2)\)\. Else \(first case in[Lemma˜B\.2](https://arxiv.org/html/2605.13999#A2.Thmlemma2)\), we can approximate the ratioqt−1​\(y\)qt​\(x\)\\frac\{q\_\{t\-1\}\(y\)\}\{q\_\{t\}\(x\)\}as

qt−1​\(y\)qt​\(x\)=\\displaystyle\\frac\{q\_\{t\-1\}\(y\)\}\{q\_\{t\}\(x\)\}=pdata​\(projD​\(y\)\)​σt−1d​\(y,D\)​\(1\+O​\(σt−1\)\)pdata​\(projD​\(x\)\)​σtd​\(x,D\)​\(1\+O​\(σt\)\)\\displaystyle\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\\sigma\_\{t\-1\}^\{d\(y,D\)\}\\left\(\{1\+O\(\\sigma\_\{t\-1\}\)\}\\right\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\\sigma\_\{t\}^\{d\(x,D\)\}\\left\(\{1\+O\(\\sigma\_\{t\}\)\}\\right\)\}=\\displaystyle=pdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​σtd​\(y,D\)−d​\(x,D\)⋅\(σt−1σt\)d​\(y,D\)​\(1\+O​\(σt\)\)\.\\displaystyle\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\sigma\_\{t\}^\{d\(y,D\)\-d\(x,D\)\}\\cdot\\left\(\{\\frac\{\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\}\\right\)^\{d\(y,D\)\}\\left\(\{1\+O\(\\sigma\_\{t\}\)\}\\right\)\.\(16\)We now consider the different termsCdC\_\{d\}in the sum in[Equation˜15](https://arxiv.org/html/2605.13999#A2.E15)by plugging in the ratio from[Equation˜16](https://arxiv.org/html/2605.13999#A2.E16):

Lowest order:𝐝=𝟎\\mathbf\{d=0\}\.We haved​\(x−h,y−h\)=0d\(x^\{\-h\},y^\{\-h\}\)=0, so the sum in[Equation˜15](https://arxiv.org/html/2605.13999#A2.E15)is only overy=\(x−h,yh\)y=\(x^\{\-h\},y^\{h\}\)\.

Case 1:d​\(y,D\)=d​\(x,D\)−1d\(y,D\)=d\(x,D\)\-1 Sincex∈supp​\(qt\)x\\in\\text\{supp\}\(q\_\{t\}\), this impliesy∈supp​\(qt−1\)y\\in\\text\{supp\}\(q\_\{t\-1\}\)\(i\.e\.∃x0∈D:∀i:\(yi=m∨yi=x0i\)\\exists x\_\{0\}\\in D\\colon\\forall i\\colon\(y^\{i\}=m\\vee y^\{i\}=x\_\{0\}^\{i\}\)\)\. Thus, the only term here is

C0=pdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​σtd​\(y,D\)−d​\(x,D\)⋅\(σt−1σt\)d​\(y,D\)​\(1\+O​\(σt\)\)​\(1−βt\)H−1​βt\.\\displaystyle C\_\{0\}=\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\sigma\_\{t\}^\{d\(y,D\)\-d\(x,D\)\}\\cdot\\left\(\{\\frac\{\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\}\\right\)^\{d\(y,D\)\}\\left\(\{1\+O\(\\sigma\_\{t\}\)\}\\right\)\\left\(\{1\-\\beta\_\{t\}\}\\right\)^\{H\-1\}\\beta\_\{t\}\.By[˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmassumption1), we haveσt−1σt→1\\frac\{\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\\to 1and1−βt→11\-\\beta\_\{t\}\\to 1, hence

Cdβt→pdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​σd​\(y,D\)−d​\(x,D\)\+O​\(σd​\(y,D\)−d​\(x,D\)\+1\)=pdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​σ−1\+O​\(1\)\\displaystyle\\frac\{C\_\{d\}\}\{\\beta\_\{t\}\}\\to\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\sigma^\{d\(y,D\)\-d\(x,D\)\}\+O\\left\(\{\\sigma^\{d\(y,D\)\-d\(x,D\)\+1\}\}\\right\)=\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\sigma^\{\-1\}\+O\\left\(\{1\}\\right\)asT→∞T\\to\\infty\.

Case 2:d​\(y,D\)≥d​\(x,D\)d\(y,D\)\\geq d\(x,D\)Note that sincexh=mx^\{h\}=mand allx∈Dx\\in Ddo not contain the mask tokenmm, we must in fact haved​\(y,D\)=d​\(x,D\)d\(y,D\)=d\(x,D\)\. Now sincexhx^\{h\}has been unmasked toyhy^\{h\}but the distance toDDdoes not decrease, this means that there is nox0∈Dx\_\{0\}\\in Dsuch that∀i:\(yi=m∨yi=x0i\)\\forall i\\colon\(y^\{i\}=m\\vee y^\{i\}=x\_\{0\}^\{i\}\)\. Henceqt−1​\(y\)=0q\_\{t\-1\}\(y\)=0and thus

Cdβt=0\.\\displaystyle\\frac\{C\_\{d\}\}\{\\beta\_\{t\}\}=0\.
Higher orders:𝐝\>𝟎\\mathbf\{d\>0\}\.Note that by[Equation˜16](https://arxiv.org/html/2605.13999#A2.E16)

qt−1​\(y\)qt​\(x\)≤\\displaystyle\\frac\{q\_\{t\-1\}\(y\)\}\{q\_\{t\}\(x\)\}\\leqpdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​σtd​\(y,D\)−d​\(x,D\)⋅\(σt−1σt\)d​\(y,D\)​\(1\+O​\(σt\)\)\\displaystyle\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\sigma\_\{t\}^\{d\(y,D\)\-d\(x,D\)\}\\cdot\\left\(\{\\frac\{\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\}\\right\)^\{d\(y,D\)\}\\left\(\{1\+O\(\\sigma\_\{t\}\)\}\\right\)→\\displaystyle\\to~pdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​σd​\(y,D\)−d​\(x,D\)\+O​\(σd​\(y,D\)−d​\(x,D\)\+1\)\\displaystyle\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\sigma^\{d\(y,D\)\-d\(x,D\)\}\+O\\left\(\{\\sigma^\{d\(y,D\)\-d\(x,D\)\+1\}\}\\right\)for the fixedσ\\sigmaasT→∞T\\to\\inftylike in thed=0d=0case\. Hence, in this case,using thatβtd→0\\beta\_\{t\}^\{d\}\\to 0ford\>1d\>1and1−βt→11\-\\beta\_\{t\}\\to 1, we have

Cdβt=∑y−h∈\[K\]H−1:∀i:yi=xi∨xi=m,d​\(x−h,y−h\)=dqt−1​\(y\)qt​\(x\)⋅\(1−βt\)H−1−d​βtd→0\\displaystyle\\frac\{C\_\{d\}\}\{\\beta\_\{t\}\}=\\sum\_\{\\begin\{subarray\}\{c\}y^\{\-h\}\\in\[K\]^\{H\-1\}\\colon\\\\ \\forall i\\colon y^\{i\}=x^\{i\}\\vee x^\{i\}=m,\\\\ d\(x^\{\-h\},y^\{\-h\}\)=d\\end\{subarray\}\}\\frac\{q\_\{t\-1\}\(y\)\}\{q\_\{t\}\(x\)\}\\cdot\\left\(\{1\-\\beta\_\{t\}\}\\right\)^\{H\-1\-d\}\\beta\_\{t\}^\{d\}\\to 0
Hence, we conclude that

limT→∞qt−1\|th​\(yh\|x\)βt=\{pdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​σ−1\+O​\(1\)if​d​\(y,D\)=d​\(x,D\)−10else\.,\\displaystyle\\lim\_\{T\\to\\infty\}\\frac\{q^\{h\}\_\{t\-1\|t\}\(y^\{h\}\|x\)\}\{\\beta\_\{t\}\}=\\begin\{dcases\}\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\sigma^\{\-1\}\+O\\left\(\{1\}\\right\)\\qquad&\\text\{ if \}d\(y,D\)=d\(x,D\)\-1\\\\ 0\\qquad&\\text\{ else\.\}\\end\{dcases\},∎

#### B\.4General Versions of[Theorem˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmtheorem1)and[Theorem˜2\.2](https://arxiv.org/html/2605.13999#S2.Thmtheorem2)

We now prove generalizations of[Theorem˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmtheorem1)and[Theorem˜2\.2](https://arxiv.org/html/2605.13999#S2.Thmtheorem2)including non\-vanishing discretization errors, which is necessary to relate the error of the learned reversed transition to the separation without actually having to resort to the continuous time limit\.

For both uniform and absorbing diffusion, we make the following assumption, which is stricter that[˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmassumption1)but is still satisfied by the common noise schedules like the cosine or the “1/\(T−t\+1\)1/\(T\-t\+1\)”\-schedule\.

###### Assumption B\.1\(Noise Schedule; Discrete Time\)\.

The noise schedule\(σt\)t∈\[T\]\(\\sigma\_\{t\}\)\_\{t\\in\[T\]\}induced by\(βt\)t∈\[T\]\(\\beta\_\{t\}\)\_\{t\\in\[T\]\}is such that for any fixedσ∈\(0,1\)\\sigma\\in\(0,1\)and time stept=tσ​\(T\):=arg⁡mins∈\[T\]⁡\|σs−σ\|t=t\_\{\\sigma\}\(T\):=\\arg\\min\_\{s\\in\[T\]\}\|\\sigma\_\{s\}\-\\sigma\|\(approximately\) corresponding to this, we have

σt−σt−1=O​\(T−1\),βt=O​\(T−1\)\\displaystyle\\sigma\_\{t\}\-\\sigma\_\{t\-1\}=O\\left\(\{T^\{\-1\}\}\\right\),\\qquad\\qquad\\beta\_\{t\}=O\(T^\{\-1\}\)asT→∞T\\to\\infty\.

##### Uniform diffusion\.

The only difference to[Theorem˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmtheorem1)is that we do not argue aboutlimT→∞qt−1\|tβt/K\\lim\_\{T\\to\\infty\}\\frac\{q\_\{t\-1\|t\}\}\{\\beta\_\{t\}/K\}but directly aboutqt−1\|tβt/K\\frac\{q\_\{t\-1\|t\}\}\{\\beta\_\{t\}/K\}under the additional assumption that the order1/T1/Tof the discretization error is sufficiently small \(O​\(σ2\)O\(\\sigma^\{2\}\)\) so as to not influence the orders of the noise level\.

###### Theorem B\.1\(Rate Separation \(Uniform\); Discrete Time\)\.

Letqqbe the law of the uniform diffusion process\. Letxt∈𝒳x\_\{t\}\\in\\mathcal\{X\}andyh∈\[K\]∖\{xth\}y^\{h\}\\in\[K\]\\setminus\\\{x\_\{t\}^\{h\}\\\}and setxt−1:=\(xt−h,yh\)x\_\{t\-1\}:=\(x\_\{t\}^\{\-h\},y^\{h\}\)\. Consider any noise levelσ<1\\sigma<1and the corresponding time stept=tσ​\(T\)=arg⁡mins∈\[T\]⁡\|σs−σ\|t=t\_\{\\sigma\}\(T\)=\\arg\\min\_\{s\\in\[T\]\}\|\\sigma\_\{s\}\-\\sigma\|\. Under[˜B\.1](https://arxiv.org/html/2605.13999#A2.Thmassumption1), for a sufficiently fine\-grained discretization1T=O​\(σ2\)\\frac\{1\}\{T\}=O\(\\sigma^\{2\}\), we have

qt−1\|th​\(yh\|xt\)βt/K=\{\(σK\)−1​pdata​\(projD​\(xt−1\)\)pdata​\(projD​\(xt\)\)\+O​\(1\)if​d​\(xt−1,D\)<d​\(xt,D\),pdata​\(projD​\(xt−1\)\)pdata​\(projD​\(xt\)\)\+O​\(σ\)if​d​\(xt−1,D\)=d​\(xt,D\),\(σK\)​pdata​\(projD​\(xt−1\)\)pdata​\(projD​\(xt\)\)\+O​\(σ2\)if​d​\(xt−1,D\)\>d​\(xt,D\),\\displaystyle\\frac\{q\_\{t\-1\|t\}^\{h\}\(y^\{h\}\|x\_\{t\}\)\}\{\\beta\_\{t\}/K\}=\\begin\{dcases\}\\left\(\{\\frac\{\\sigma\}\{K\}\}\\right\)^\{\-1\}\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\_\{t\-1\}\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\_\{t\}\)\)\}\+O\(1\)&\\text\{if \}d\(x\_\{t\-1\},D\)<d\(x\_\{t\},D\),\\\\\[5\.69054pt\] \\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\_\{t\-1\}\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\_\{t\}\)\)\}\+O\(\\sigma\)&\\text\{if \}d\(x\_\{t\-1\},D\)=d\(x\_\{t\},D\),\\\\\[5\.69054pt\] \\left\(\{\\frac\{\\sigma\}\{K\}\}\\right\)\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\_\{t\-1\}\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\_\{t\}\)\)\}\+O\(\\sigma^\{2\}\)&\\text\{if \}d\(x\_\{t\-1\},D\)\>d\(x\_\{t\},D\),\\end\{dcases\}in terms ofσ→0\\sigma\\to 0\.

###### Proof\.

For ease of notation, we setx:=xtx:=x\_\{t\}andz:=xt−1=\(xt−h,yh\)z:=x\_\{t\-1\}=\(x\_\{t\}^\{\-h\},y^\{h\}\)\. Following first part of the proof of[Theorem˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmtheorem1), we have \(see[Equation˜13](https://arxiv.org/html/2605.13999#A2.E13)\)

qt−1\|th​\(yh\|x\)=\\displaystyle q\_\{t\-1\|t\}^\{h\}\(y^\{h\}\|x\)=∑d=0H−1∑y−h∈\[K\]H−1d​\(x−h,y−h\)=dqt−1​\(y\)qt​\(x\)⋅\(1−βt​K−1K\)H−1−d​\(βtK\)d\+1⏟=⁣:Cd,\\displaystyle\\sum\_\{d=0\}^\{H\-1\}\\underbrace\{\\sum\_\{\\begin\{subarray\}\{c\}y^\{\-h\}\\in\[K\]^\{H\-1\}\\\\ d\(x^\{\-h\},y^\{\-h\}\)=d\\end\{subarray\}\}\\frac\{q\_\{t\-1\}\(y\)\}\{q\_\{t\}\(x\)\}\\cdot\\left\(\{1\-\\beta\_\{t\}\\frac\{K\-1\}\{K\}\}\\right\)^\{H\-1\-d\}\\left\(\{\\frac\{\\beta\_\{t\}\}\{K\}\}\\right\)^\{d\+1\}\}\_\{=:C\_\{d\}\},\(17\)where \(see[Equation˜14](https://arxiv.org/html/2605.13999#A2.E14)\)

qt−1​\(y\)qt​\(x\)=\\displaystyle\\frac\{q\_\{t\-1\}\(y\)\}\{q\_\{t\}\(x\)\}=pdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​\(σtK\)d​\(y,D\)−d​\(x,D\)⋅\(σt−1σt\)d​\(y,D\)​\(1\+O​\(σtK\)\)\.\\displaystyle\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\left\(\{\\frac\{\\sigma\_\{t\}\}\{K\}\}\\right\)^\{d\(y,D\)\-d\(x,D\)\}\\cdot\\left\(\{\\frac\{\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\}\\right\)^\{d\(y,D\)\}\\left\(\{1\+O\\left\(\{\\frac\{\\sigma\_\{t\}\}\{K\}\}\\right\)\}\\right\)\.\(18\)We now consider the different termsCdC\_\{d\}in the sum in[Equation˜17](https://arxiv.org/html/2605.13999#A2.E17)by plugging in the ratio from[Equation˜18](https://arxiv.org/html/2605.13999#A2.E18):

Lowest order:𝐝=𝟎\\mathbf\{d=0\}\.We haved​\(x−h,y−h\)=0d\(x^\{\-h\},y^\{\-h\}\)=0, soy=\(x−h,yh\)y=\(x^\{\-h\},y^\{h\}\)\. Hence, the only term here is

C0=pdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​\(σtK\)d​\(y,D\)−d​\(x,D\)⋅\(σt−1σt\)d​\(y,D\)​\(1\+O​\(σtK\)\)​\(1−βt​K−1K\)H−1​\(βtK\)\.\\displaystyle C\_\{0\}=\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\left\(\{\\frac\{\\sigma\_\{t\}\}\{K\}\}\\right\)^\{d\(y,D\)\-d\(x,D\)\}\\cdot\\left\(\{\\frac\{\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\}\\right\)^\{d\(y,D\)\}\\left\(\{1\+O\\left\(\{\\frac\{\\sigma\_\{t\}\}\{K\}\}\\right\)\}\\right\)\\left\(\{1\-\\beta\_\{t\}\\frac\{K\-1\}\{K\}\}\\right\)^\{H\-1\}\\left\(\{\\frac\{\\beta\_\{t\}\}\{K\}\}\\right\)\.Now recall that we assumed1T=O​\(σ2\)\\frac\{1\}\{T\}=O\(\\sigma^\{2\}\)\. Thus by[˜B\.1](https://arxiv.org/html/2605.13999#A2.Thmassumption1), we have\|σt−σ\|≤max⁡\{\|σt−σt−1\|,\|σt−σt\+1\|\}=O​\(T−1\)\|\\sigma\_\{t\}\-\\sigma\|\\leq\\max\\\{\|\\sigma\_\{t\}\-\\sigma\_\{t\-1\}\|,\|\\sigma\_\{t\}\-\\sigma\_\{t\+1\}\|\\\}=O\(T^\{\-1\}\), as well asσt−1σt=1−σt−σt−1σt=1\+=O​\(T−1\)σ\+O​\(T−1\)=O​\(σ−1​T−1\)\\frac\{\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}=1\-\\frac\{\\sigma\_\{t\}\-\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}=1\+=\\frac\{O\(T^\{\-1\}\)\}\{\\sigma\+O\(T^\{\-1\}\)\}=O\(\\sigma^\{\-1\}T^\{\-1\}\)and1−βt​K−1K=1\+O​\(T−1\)1\-\\beta\_\{t\}\\frac\{K\-1\}\{K\}=1\+O\(T^\{\-1\}\), hence

C0βt/K=\\displaystyle\\frac\{C\_\{0\}\}\{\\beta\_\{t\}/K\}=pdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​\(σK\)d​\(y,D\)−d​\(x,D\)​\(1\+O​\(σ−1​T−1\)\)\\displaystyle\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\left\(\{\\frac\{\\sigma\}\{K\}\}\\right\)^\{d\(y,D\)\-d\(x,D\)\}\\left\(\{1\+O\(\\sigma^\{\-1\}T^\{\-1\}\)\}\\right\)=\\displaystyle=pdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​\(σK\)d​\(y,D\)−d​\(x,D\)​\(1\+O​\(σ\)\)\.\\displaystyle\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\left\(\{\\frac\{\\sigma\}\{K\}\}\\right\)^\{d\(y,D\)\-d\(x,D\)\}\\left\(\{1\+O\(\\sigma\)\}\\right\)\.
Higher orders:𝐝\>𝟎\\mathbf\{d\>0\}\.In this case, we have

Cdβt/K=∑y−h∈\[K\]H−1d​\(x−h,y−h\)=dpdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​\(σtK\)d​\(y,D\)−d​\(x,D\)⋅\(σt−1σt\)d​\(y,D\)⋅\(1−βt​K−1K\)H−1−d​\(βtK\)d,\\displaystyle\\frac\{C\_\{d\}\}\{\\beta\_\{t\}/K\}=\\sum\_\{\\begin\{subarray\}\{c\}y^\{\-h\}\\in\[K\]^\{H\-1\}\\\\ d\(x^\{\-h\},y^\{\-h\}\)=d\\end\{subarray\}\}\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\left\(\{\\frac\{\\sigma\_\{t\}\}\{K\}\}\\right\)^\{d\(y,D\)\-d\(x,D\)\}\\cdot\\left\(\{\\frac\{\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\}\\right\)^\{d\(y,D\)\}\\cdot\\left\(\{1\-\\beta\_\{t\}\\frac\{K\-1\}\{K\}\}\\right\)^\{H\-1\-d\}\\left\(\{\\frac\{\\beta\_\{t\}\}\{K\}\}\\right\)^\{d\},Once more,1−βt​K−1K=1\+O​\(T−1\)1\-\\beta\_\{t\}\\frac\{K\-1\}\{K\}=1\+O\(T^\{\-1\}\)andσt−1/σt=1\+O​\(σ−1​T−1\)=1\+O​\(σ\)\\sigma\_\{t\-1\}/\\sigma\_\{t\}=1\+O\(\\sigma^\{\-1\}T^\{\-1\}\)=1\+O\(\\sigma\)since we assumedT−1=O​\(σ2\)T^\{\-1\}=O\(\\sigma^\{2\}\)\. Now\(βt/K\)d=O​\(\(T​K\)−d\)\(\\beta\_\{t\}/K\)^\{d\}=O\(\(TK\)^\{\-d\}\)\. Hence, the above simplifies to

Cdβt/K=∑y−h∈\[K\]H−1d​\(x−h,y−h\)=dpdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​\(σK\)d​\(y,D\)−d​\(x,D\)⋅O​\(\(1T​K\)d\)​\(1\+O​\(σ\)\)\\displaystyle\\frac\{C\_\{d\}\}\{\\beta\_\{t\}/K\}=\\sum\_\{\\begin\{subarray\}\{c\}y^\{\-h\}\\in\[K\]^\{H\-1\}\\\\ d\(x^\{\-h\},y^\{\-h\}\)=d\\end\{subarray\}\}\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\left\(\{\\frac\{\\sigma\}\{K\}\}\\right\)^\{d\(y,D\)\-d\(x,D\)\}\\cdot O\\left\(\{\\left\(\{\\frac\{1\}\{TK\}\}\\right\)^\{d\}\}\\right\)\(1\+O\(\\sigma\)\)We consider each term separately and distinguish cases according tod​\(z,D\)d\(z,D\)wherez:=\(x−h,yh\)z:=\(x^\{\-h\},y^\{h\}\)\. In each case we will make use of the fact that by the triangle inequality,

d​\(z,D\)=d​\(\(x−h,yh\),D\)≤d​\(\(x−h,yh\),y\)\+d​\(y,D\)=d​\(x−h,y−h\)\+d​\(y,D\)=d\+d​\(y,D\)\.\\displaystyle d\(z,D\)=d\(\(x^\{\-h\},y^\{h\}\),D\)\\leq d\(\(x^\{\-h\},y^\{h\}\),y\)\+d\(y,D\)=d\(x^\{\-h\},y^\{\-h\}\)\+d\(y,D\)=d\+d\(y,D\)\.\(19\)
Case 1:d​\(z,D\)=d​\(x,D\)−1d\(z,D\)=d\(x,D\)\-1\. Then by[Equation˜19](https://arxiv.org/html/2605.13999#A2.E19),d​\(y,D\)−d​\(x,D\)=d​\(y,D\)−d​\(z,D\)−1≥−d−1d\(y,D\)\-d\(x,D\)=d\(y,D\)\-d\(z,D\)\-1\\geq\-d\-1, and thus\(σ/K\)d​\(y,D\)−d​\(x,D\)≤\(σ/K\)−\(d\+1\)\(\\sigma/K\)^\{d\(y,D\)\-d\(x,D\)\}\\leq\(\\sigma/K\)^\{\-\(d\+1\)\}\. Hence,

Cdβt/K≤\\displaystyle\\frac\{C\_\{d\}\}\{\\beta\_\{t\}/K\}\\leqO\(σ−\(d\+1\)T−dK\(1\+σ−1T−1\)\)≤O\(σ2​d−d−1\(1\+σ\)\)\)≤O\(1\),\\displaystyle O\(\\sigma^\{\-\(d\+1\)\}T^\{\-d\}K\(1\+\\sigma^\{\-1\}T^\{\-1\}\)\)\\leq O\(\\sigma^\{2d\-d\-1\}\(1\+\\sigma\)\)\)\\leq O\(1\),since we assumed1T=O​\(σ2\)\\frac\{1\}\{T\}=O\(\\sigma^\{2\}\)andd≥1d\\geq 1\.

Case 2:d​\(z,D\)=d​\(x,D\)d\(z,D\)=d\(x,D\)\. Then by[Equation˜19](https://arxiv.org/html/2605.13999#A2.E19),d​\(y,D\)−d​\(x,D\)=d​\(y,D\)−d​\(z,D\)≥−dd\(y,D\)\-d\(x,D\)=d\(y,D\)\-d\(z,D\)\\geq\-d, and thus\(σ/K\)d​\(y,D\)−d​\(x,D\)≤\(σ/K\)−d\(\\sigma/K\)^\{d\(y,D\)\-d\(x,D\)\}\\leq\(\\sigma/K\)^\{\-d\}\. Hence,

Cdβt/K≤\\displaystyle\\frac\{C\_\{d\}\}\{\\beta\_\{t\}/K\}\\leqO\(σ−dT−d\(1\+σ−1T−1\)\)≤O\(σ2​d−d\(1\+σ\)\)\)≤O\(σ\),\\displaystyle O\(\\sigma^\{\-d\}T^\{\-d\}\(1\+\\sigma^\{\-1\}T^\{\-1\}\)\)\\leq O\(\\sigma^\{2d\-d\}\(1\+\\sigma\)\)\)\\leq O\(\\sigma\),since we assumed1T=O​\(σ2\)\\frac\{1\}\{T\}=O\(\\sigma^\{2\}\)andd≥1d\\geq 1\.

Case 3:d​\(z,D\)=d​\(x,D\)\+1d\(z,D\)=d\(x,D\)\+1\. Then by[Equation˜19](https://arxiv.org/html/2605.13999#A2.E19),d​\(y,D\)−d​\(x,D\)=d​\(y,D\)−d​\(z,D\)\+1≥−d\+1d\(y,D\)\-d\(x,D\)=d\(y,D\)\-d\(z,D\)\+1\\geq\-d\+1, and thus\(σ/K\)d​\(y,D\)−d​\(x,D\)≤\(σ/K\)−d\+1\(\\sigma/K\)^\{d\(y,D\)\-d\(x,D\)\}\\leq\(\\sigma/K\)^\{\-d\+1\}\. Hence,

Cdβt/K≤\\displaystyle\\frac\{C\_\{d\}\}\{\\beta\_\{t\}/K\}\\leqO\(σ−d\+1T−d\(1\+σ−1T−1\)\)≤O\(σ2​d−d\+1\(1\+σ\)\)\)≤O\(σ2\),\\displaystyle O\(\\sigma^\{\-d\+1\}T^\{\-d\}\(1\+\\sigma^\{\-1\}T^\{\-1\}\)\)\\leq O\(\\sigma^\{2d\-d\+1\}\(1\+\\sigma\)\)\)\\leq O\(\\sigma^\{2\}\),since we assumed1T=O​\(σ2\)\\frac\{1\}\{T\}=O\(\\sigma^\{2\}\)andd≥1d\\geq 1\.

Hence, distinguishing these cases for the overall valueqt−1\|th​\(yh\|x\)=C0\+∑d=1HCdq^\{h\}\_\{t\-1\|t\}\(y^\{h\}\|x\)=C\_\{0\}\+\\sum\_\{d=1\}^\{H\}C\_\{d\}and summing up yields, forz=\(x−h,yh\)z=\(x^\{\-h\},y^\{h\}\)

qt−1\|th​\(yh\|x\)βt/K=\{\(σK\)−1​pdata​\(projD​\(z\)\)pdata​\(projD​\(x\)\)\+O​\(1\)if​d​\(z,D\)=d​\(x,D\)−1,pdata​\(projD​\(z\)\)pdata​\(projD​\(x\)\)\+O​\(σ\)if​d​\(z,D\)=d​\(x,D\),\(σK\)​pdata​\(projD​\(z\)\)pdata​\(projD​\(x\)\)\+O​\(σ2\)if​d​\(z,D\)\>d​\(x,D\)\.\\displaystyle\\frac\{q^\{h\}\_\{t\-1\|t\}\(y^\{h\}\|x\)\}\{\\beta\_\{t\}/K\}=\\begin\{dcases\}\\left\(\{\\frac\{\\sigma\}\{K\}\}\\right\)^\{\-1\}\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(z\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\+O\(1\)&\\text\{if \}d\(z,D\)=d\(x,D\)\-1,\\\\\[5\.69054pt\] \\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(z\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\+O\(\\sigma\)&\\text\{if \}d\(z,D\)=d\(x,D\),\\\\\[5\.69054pt\] \\left\(\{\\frac\{\\sigma\}\{K\}\}\\right\)\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(z\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\+O\(\\sigma^\{2\}\)&\\text\{if \}d\(z,D\)\>d\(x,D\)\.\\end\{dcases\}∎

##### Absorbing diffusion\.

The statement and proof for absorbing diffusion only differ slightly\.

###### Theorem B\.2\(Rate Separation \(Absorbing\); Discrete Time\)\.

Letqqbe the law of the absorbing diffusion process\. Letxt∈supp​\(qt\)x\_\{t\}\\in\\text\{supp\}\(q\_\{t\}\)withxh=mx^\{h\}=mandyh∈\[K\]y^\{h\}\\in\[K\]and setxt−1:=\(xt−h,yh\)x\_\{t\-1\}:=\(x\_\{t\}^\{\-h\},y^\{h\}\)\. Consider any noise levelσ<1\\sigma<1and the corresponding time stept=tσ​\(T\)=arg⁡mins∈\[T\]⁡\|σs−σ\|t=t\_\{\\sigma\}\(T\)=\\arg\\min\_\{s\\in\[T\]\}\|\\sigma\_\{s\}\-\\sigma\|\. Under[˜B\.1](https://arxiv.org/html/2605.13999#A2.Thmassumption1), for a sufficiently fine\-grained discretization1T=O​\(σ2\)\\frac\{1\}\{T\}=O\(\\sigma^\{2\}\), we have

qt−1\|th​\(yh\|xt\)βt=\{σ−1​pdata​\(projD​\(xt−1\)\)pdata​\(projD​\(xt\)\)\+O​\(1\)if​d​\(xt−1,D\)<d​\(xt,D\),O​\(σ\)if​d​\(xt−1,D\)=d​\(xt,D\),\\displaystyle\\frac\{q\_\{t\-1\|t\}^\{h\}\(y^\{h\}\|x\_\{t\}\)\}\{\\beta\_\{t\}\}=\\begin\{dcases\}\\sigma^\{\-1\}\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\_\{t\-1\}\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\_\{t\}\)\)\}\+O\(1\)&\\text\{if \}d\(x\_\{t\-1\},D\)<d\(x\_\{t\},D\),\\\\\[5\.69054pt\] O\(\\sigma\)&\\text\{if \}d\(x\_\{t\-1\},D\)=d\(x\_\{t\},D\),\\end\{dcases\}in terms ofσ→0\\sigma\\to 0\.

###### Proof\.

For ease of notation, we setx:=xtx:=x\_\{t\}andz:=xt−1=\(xt−h,yh\)z:=x\_\{t\-1\}=\(x\_\{t\}^\{\-h\},y^\{h\}\)\. Following first part of the proof of[Theorem˜2\.2](https://arxiv.org/html/2605.13999#S2.Thmtheorem2), we have \(see[Equation˜15](https://arxiv.org/html/2605.13999#A2.E15)\)

qt−1\|th​\(yh\|x\)=\\displaystyle q\_\{t\-1\|t\}^\{h\}\(y^\{h\}\|x\)=∑d=0H−1∑y−h∈\[K\]H−1:∀i:yi=xi∨xi=m,d​\(x−h,y−h\)=dqt−1​\(y\)qt​\(x\)⋅\(1−βt\)H−1−d​βtd\+1⏟=⁣:Cd\.\\displaystyle\\sum\_\{d=0\}^\{H\-1\}\\underbrace\{\\sum\_\{\\begin\{subarray\}\{c\}y^\{\-h\}\\in\[K\]^\{H\-1\}\\colon\\\\ \\forall i\\colon y^\{i\}=x^\{i\}\\vee x^\{i\}=m,\\\\ d\(x^\{\-h\},y^\{\-h\}\)=d\\end\{subarray\}\}\\frac\{q\_\{t\-1\}\(y\)\}\{q\_\{t\}\(x\)\}\\cdot\\left\(\{1\-\\beta\_\{t\}\}\\right\)^\{H\-1\-d\}\\beta\_\{t\}^\{d\+1\}\}\_\{=:C\_\{d\}\}\.\(20\)where \(see[Equation˜16](https://arxiv.org/html/2605.13999#A2.E16)\)

qt−1​\(y\)qt​\(x\)=\\displaystyle\\frac\{q\_\{t\-1\}\(y\)\}\{q\_\{t\}\(x\)\}=pdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​σtd​\(y,D\)−d​\(x,D\)⋅\(σt−1σt\)d​\(y,D\)​\(1\+O​\(σt\)\)\.\\displaystyle\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\sigma\_\{t\}^\{d\(y,D\)\-d\(x,D\)\}\\cdot\\left\(\{\\frac\{\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\}\\right\)^\{d\(y,D\)\}\\left\(\{1\+O\(\\sigma\_\{t\}\)\}\\right\)\.\(21\)ify∈supp​\(qt−1\)y\\in\\text\{supp\}\(q\_\{t\-1\}\)and0otherwise\.

We now consider the different termsCdC\_\{d\}in the sum in[Equation˜20](https://arxiv.org/html/2605.13999#A2.E20)by plugging in the ratio from[Equation˜21](https://arxiv.org/html/2605.13999#A2.E21):

Lowest order:𝐝=𝟎\\mathbf\{d=0\}\.

We haved​\(x−h,y−h\)=0d\(x^\{\-h\},y^\{\-h\}\)=0, so the sum in[Equation˜15](https://arxiv.org/html/2605.13999#A2.E15)is only overy=\(x−h,yh\)y=\(x^\{\-h\},y^\{h\}\)\.

Case 1:d​\(y,D\)=d​\(x,D\)−1d\(y,D\)=d\(x,D\)\-1Sincex∈supp​\(qt\)x\\in\\text\{supp\}\(q\_\{t\}\), this impliesy∈supp​\(qt−1\)y\\in\\text\{supp\}\(q\_\{t\-1\}\)\(i\.e\.∃x0∈D:∀i:\(yi=m∨yi=x0i\)\\exists x\_\{0\}\\in D\\colon\\forall i\\colon\(y^\{i\}=m\\vee y^\{i\}=x\_\{0\}^\{i\}\)\)\. Thus, the only term here is

C0=pdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​σtd​\(y,D\)−d​\(x,D\)⋅\(σt−1σt\)d​\(y,D\)​\(1\+O​\(σt\)\)​\(1−βt\)H−1​βt\.\\displaystyle C\_\{0\}=\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\sigma\_\{t\}^\{d\(y,D\)\-d\(x,D\)\}\\cdot\\left\(\{\\frac\{\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\}\\right\)^\{d\(y,D\)\}\\left\(\{1\+O\(\\sigma\_\{t\}\)\}\\right\)\\left\(\{1\-\\beta\_\{t\}\}\\right\)^\{H\-1\}\\beta\_\{t\}\.Now recall that we assumed1T=O​\(σ2\)\\frac\{1\}\{T\}=O\(\\sigma^\{2\}\)\. Thus by[˜B\.1](https://arxiv.org/html/2605.13999#A2.Thmassumption1), we have\|σt−σ\|≤max⁡\{\|σt−σt−1\|,\|σt−σt\+1\|\}=O​\(T−1\)\|\\sigma\_\{t\}\-\\sigma\|\\leq\\max\\\{\|\\sigma\_\{t\}\-\\sigma\_\{t\-1\}\|,\|\\sigma\_\{t\}\-\\sigma\_\{t\+1\}\|\\\}=O\(T^\{\-1\}\), as well asσt−1σt=1−σt−σt−1σt=1\+O​\(T−1\)σ\+O​\(T−1\)=O​\(σ−1​T−1\)\\frac\{\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}=1\-\\frac\{\\sigma\_\{t\}\-\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}=1\+\\frac\{O\(T^\{\-1\}\)\}\{\\sigma\+O\(T^\{\-1\}\)\}=O\(\\sigma^\{\-1\}T^\{\-1\}\)and1−βt=1\+O​\(T−1\)1\-\\beta\_\{t\}=1\+O\(T^\{\-1\}\), hence

C0βt/K=\\displaystyle\\frac\{C\_\{0\}\}\{\\beta\_\{t\}/K\}=pdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​σd​\(y,D\)−d​\(x,D\)​\(1\+O​\(σ−1​T−1\)\)\\displaystyle\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\sigma^\{d\(y,D\)\-d\(x,D\)\}\\left\(\{1\+O\(\\sigma^\{\-1\}T^\{\-1\}\)\}\\right\)=\\displaystyle=pdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​σd​\(y,D\)−d​\(x,D\)​\(1\+O​\(σ\)\)\\displaystyle\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\sigma^\{d\(y,D\)\-d\(x,D\)\}\\left\(\{1\+O\(\\sigma\)\}\\right\)=\\displaystyle=pdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​σ−1\+O​\(1\)\.\\displaystyle\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\sigma^\{\-1\}\+O\(1\)\.
Case 2:d​\(y,D\)≥d​\(x,D\)d\(y,D\)\\geq d\(x,D\) Note that sincexh=mx^\{h\}=mand allx∈Dx\\in Ddo not contain the mask tokenmm, we must in fact haved​\(y,D\)=d​\(x,D\)d\(y,D\)=d\(x,D\)\. Now sincexhx^\{h\}has been unmasked toyhy^\{h\}but the distance toDDdoes not decrease, this means that there is nox0∈Dx\_\{0\}\\in Dsuch that∀i:\(yi=m∨yi=x0i\)\\forall i\\colon\(y^\{i\}=m\\vee y^\{i\}=x\_\{0\}^\{i\}\)\. Henceqt−1​\(y\)=0q\_\{t\-1\}\(y\)=0and thus

Cdβt=0\.\\displaystyle\\frac\{C\_\{d\}\}\{\\beta\_\{t\}\}=0\.
Higher orders:𝐝\>𝟎\\mathbf\{d\>0\}\.In this case, we have

Cdβt≤∑y−h∈\[K\]H−1:∀i:yi=xi∨xi=m,d​\(x−h,y−h\)=dpdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​σtd​\(y,D\)−d​\(x,D\)⋅\(σt−1σt\)d​\(y,D\)⋅\(1−βt\)H−1−d​βtd,\\displaystyle\\frac\{C\_\{d\}\}\{\\beta\_\{t\}\}\\leq\\sum\_\{\\begin\{subarray\}\{c\}y^\{\-h\}\\in\[K\]^\{H\-1\}\\colon\\\\ \\forall i\\colon y^\{i\}=x^\{i\}\\vee x^\{i\}=m,\\\\ d\(x^\{\-h\},y^\{\-h\}\)=d\\end\{subarray\}\}\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\sigma\_\{t\}^\{d\(y,D\)\-d\(x,D\)\}\\cdot\\left\(\{\\frac\{\\sigma\_\{t\-1\}\}\{\\sigma\_\{t\}\}\}\\right\)^\{d\(y,D\)\}\\cdot\\left\(\{1\-\\beta\_\{t\}\}\\right\)^\{H\-1\-d\}\\beta\_\{t\}^\{d\},Once more,1−βt=1\+O​\(T−1\)1\-\\beta\_\{t\}=1\+O\(T^\{\-1\}\)andσt−1/σt=1\+O​\(σ−1​T−1\)=1\+O​\(σ\)\\sigma\_\{t\-1\}/\\sigma\_\{t\}=1\+O\(\\sigma^\{\-1\}T^\{\-1\}\)=1\+O\(\\sigma\)since we assumedT−1=O​\(σ2\)T^\{\-1\}=O\(\\sigma^\{2\}\)\. Nowβtd=O​\(T−d\)\\beta\_\{t\}^\{d\}=O\(T^\{\-d\}\)\. Hence, the above simplifies to

Cdβt≤∑y−h∈\[K\]H−1:∀i:yi=xi∨xi=m,d​\(x−h,y−h\)=dpdata​\(projD​\(y\)\)pdata​\(projD​\(x\)\)​σd​\(y,D\)−d​\(x,D\)⋅O​\(T−d\)​\(1\+O​\(σ\)\)\\displaystyle\\frac\{C\_\{d\}\}\{\\beta\_\{t\}\}\\leq\\sum\_\{\\begin\{subarray\}\{c\}y^\{\-h\}\\in\[K\]^\{H\-1\}\\colon\\\\ \\forall i\\colon y^\{i\}=x^\{i\}\\vee x^\{i\}=m,\\\\ d\(x^\{\-h\},y^\{\-h\}\)=d\\end\{subarray\}\}\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(y\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\\sigma^\{d\(y,D\)\-d\(x,D\)\}\\cdot O\(T^\{\-d\}\)\(1\+O\(\\sigma\)\)We consider each term separately and distinguish cases according tod​\(z,D\)d\(z,D\)wherez:=\(x−h,yh\)z:=\(x^\{\-h\},y^\{h\}\)\. In each case we will make use of the fact that by the triangle inequality,

d​\(z,D\)=d​\(\(x−h,yh\),D\)≤d​\(\(x−h,yh\),y\)\+d​\(y,D\)=d​\(x−h,y−h\)\+d​\(y,D\)=d\+d​\(y,D\)\.\\displaystyle d\(z,D\)=d\(\(x^\{\-h\},y^\{h\}\),D\)\\leq d\(\(x^\{\-h\},y^\{h\}\),y\)\+d\(y,D\)=d\(x^\{\-h\},y^\{\-h\}\)\+d\(y,D\)=d\+d\(y,D\)\.\(22\)Note that sinceyh≠my^\{h\}\\neq mandxh=mx^\{h\}=m, we can only haved​\(z,D\)=d​\(x,D\)−1d\(z,D\)=d\(x,D\)\-1ord​\(z,D\)=d​\(x,D\)d\(z,D\)=d\(x,D\)\.

Case 1:d​\(z,D\)=d​\(x,D\)−1d\(z,D\)=d\(x,D\)\-1\. Then by[Equation˜22](https://arxiv.org/html/2605.13999#A2.E22),d​\(y,D\)−d​\(x,D\)=d​\(y,D\)−d​\(z,D\)−1≥−d−1d\(y,D\)\-d\(x,D\)=d\(y,D\)\-d\(z,D\)\-1\\geq\-d\-1, and thusσd​\(y,D\)−d​\(x,D\)≤σ−\(d\+1\)\\sigma^\{d\(y,D\)\-d\(x,D\)\}\\leq\\sigma^\{\-\(d\+1\)\}\. Hence,

Cdβt≤\\displaystyle\\frac\{C\_\{d\}\}\{\\beta\_\{t\}\}\\leqO\(σ−\(d\+1\)T−d\(1\+σ−1T−1\)\)≤O\(σ2​d−d−1\(1\+σ\)\)\)≤O\(1\),\\displaystyle O\(\\sigma^\{\-\(d\+1\)\}T^\{\-d\}\(1\+\\sigma^\{\-1\}T^\{\-1\}\)\)\\leq O\(\\sigma^\{2d\-d\-1\}\(1\+\\sigma\)\)\)\\leq O\(1\),since we assumed1T=O​\(σ2\)\\frac\{1\}\{T\}=O\(\\sigma^\{2\}\)andd≥1d\\geq 1\.

Case 2:d​\(z,D\)=d​\(x,D\)d\(z,D\)=d\(x,D\)\. Then by[Equation˜22](https://arxiv.org/html/2605.13999#A2.E22),d​\(y,D\)−d​\(x,D\)=d​\(y,D\)−d​\(z,D\)≥−dd\(y,D\)\-d\(x,D\)=d\(y,D\)\-d\(z,D\)\\geq\-d, and thusσd​\(y,D\)−d​\(x,D\)≤σ−d\\sigma^\{d\(y,D\)\-d\(x,D\)\}\\leq\\sigma^\{\-d\}\. Hence,

Cdβt≤\\displaystyle\\frac\{C\_\{d\}\}\{\\beta\_\{t\}\}\\leqO\(σ−dT−d\(1\+σ−1T−1\)\)≤O\(σ2​d−d\(1\+σ\)\)\)≤O\(σ\),\\displaystyle O\(\\sigma^\{\-d\}T^\{\-d\}\(1\+\\sigma^\{\-1\}T^\{\-1\}\)\)\\leq O\(\\sigma^\{2d\-d\}\(1\+\\sigma\)\)\)\\leq O\(\\sigma\),since we assumed1T=O​\(σ2\)\\frac\{1\}\{T\}=O\(\\sigma^\{2\}\)andd≥1d\\geq 1\.

Hence, distinguishing these cases for the overall valueqt−1\|th​\(yh\|x\)=C0\+∑d=1HCdq^\{h\}\_\{t\-1\|t\}\(y^\{h\}\|x\)=C\_\{0\}\+\\sum\_\{d=1\}^\{H\}C\_\{d\}and summing up yields, forz=\(x−h,yh\)z=\(x^\{\-h\},y^\{h\}\)

qt−1\|th​\(yh\|x\)βt/K=\{σ−1​pdata​\(projD​\(z\)\)pdata​\(projD​\(x\)\)\+O​\(1\)if​d​\(z,D\)=d​\(x,D\)−1,O​\(σ\)if​d​\(z,D\)=d​\(x,D\)\.\\displaystyle\\frac\{q^\{h\}\_\{t\-1\|t\}\(y^\{h\}\|x\)\}\{\\beta\_\{t\}/K\}=\\begin\{dcases\}\\sigma^\{\-1\}\\frac\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(z\)\)\}\{p\_\{\\text\{data\}\}\(\\text\{proj\}\_\{D\}\(x\)\)\}\+O\(1\)&\\text\{if \}d\(z,D\)=d\(x,D\)\-1,\\\\\[5\.69054pt\] O\(\\sigma\)&\\text\{if \}d\(z,D\)=d\(x,D\)\.\\\\\[5\.69054pt\] \\end\{dcases\}∎

### Appendix CFurther Details on the Synthetic Experiments

#### C\.1A simple regular language baseline

We first describe the underlying synthetic data distribution\. We consider a vocabulary of sizeKKindexed by the natural numbers from11toKKand sequences of lengthHH\. We randomly sample a string of this length by rolling out the following random walk: The first tokenX1X^\{1\}is sampled uniformly at random from\[K\]\[K\]\. Next, for any1≤h≤H−11\\leq h\\leq H\-1, givenXh=xhX^\{h\}=x^\{h\}we sampleXh\+1X^\{h\+1\}according to anxhx\_\{h\}\-dependent probability distribution as

xh\+1=\{xh\+1w\.p\.​pup​\(xh\)xhw\.p\.​pstay​\(xh\)xh−1w\.p\.​pdown​\(xh\)\\displaystyle x\_\{h\+1\}=\\begin\{cases\}x\_\{h\}\+1\\quad&\\text\{w\.p\. \}p\_\{\\text\{up\}\}\(x\_\{h\}\)\\\\ x\_\{h\}&\\text\{w\.p\. \}p\_\{\\text\{stay\}\}\(x\_\{h\}\)\\\\ x\_\{h\}\-1&\\text\{w\.p\. \}p\_\{\\text\{down\}\}\(x\_\{h\}\)\\\\ \\end\{cases\}forxh∈\{2,…,K−1\}x\_\{h\}\\in\\\{2,\\dots,K\-1\\\}, where

\(pup​\(x\)pstay​\(x\)pdown​\(x\)\)\\displaystyle\\begin\{pmatrix\}p\_\{\\text\{up\}\}\(x\)\\\\ p\_\{\\text\{stay\}\}\(x\)\\\\ p\_\{\\text\{down\}\}\(x\)\\end\{pmatrix\}∝\(0\.45​exp⁡\(1\.0⋅sin⁡\(0\.61​\(x−1\)\+1\.37\)\)0\.2​exp⁡\(1\.0⋅sin⁡\(0\.37​\(x−1\)\+0\.11\)\)0\.35​exp⁡\(1\.0⋅cos⁡\(0\.53⋅\(x−1\)−0\.29\)\),\)\\displaystyle\\propto\\begin\{pmatrix\}0\.45\\exp\(1\.0\\cdot\\sin\(0\.61\(x\-1\)\+1\.37\)\)\\\\ 0\.2\\exp\(1\.0\\cdot\\sin\(0\.37\(x\-1\)\+0\.11\)\)\\\\ 0\.35\\exp\(1\.0\\cdot\\cos\(0\.53\\cdot\(x\-1\)\-0\.29\)\),\\end\{pmatrix\}simply meaning that the transitions vary in an oscillating way from the biased random walk\(pup,pstay,pdown\)=\(0\.45,0\.2,0\.35\)\(p\_\{\\text\{up\}\},p\_\{\\text\{stay\}\},p\_\{\\text\{down\}\}\)=\(0\.45,0\.2,0\.35\)\. Whenever the above random walk would leave\[K\]\[K\], we clamp it to11orKK, respectively\. Deciding whether a string is in the support of this distribution can trivially be done by checking the range of neighboring tokens for each token position\. Computing the exact Hamming distance can be done via a simple and standard dynamic programming subroutine\.

#### C\.2Further experimental details

As for[Figures˜2](https://arxiv.org/html/2605.13999#S3.F2)and[3](https://arxiv.org/html/2605.13999#S3.F3), we consider learning the distribution of the above section withn=100,000n=100,000samples by training standard discrete denoising diffusion models\[austin2021structured\]\. We equip this model with a conventional bidirectional transformer architecture with108,256108,256trainable parameters \(two layers, four heads, and feedforward dimension6464\) and train for50,00050,000steps with a batch size of512512\. As for the distribution, we vary sequence lengthH∈\{32,64\}H\\in\\\{32,64\\\}and vocabulary sizeK∈\{32,64\}K\\in\\\{32,64\\\}\. On the model side, we modify the number of diffusion steps and considerT∈\{H,2​H\}T\\in\\\{H,2H\\\}\. We use a “linear” noise scheduleσt=t/T\\sigma\_\{t\}=t/T\(corresponding toβt=\(T−t\+1\)−1\\beta\_\{t\}=\(T\-t\+1\)^\{\-1\}\) throughout\. We use the AdamW optimizer with learning rate0\.00030\.0003and set the hybrid loss coefficient in D3PM to0\.0010\.001\. The validation set size is1,0001,000for all runs and all listed metrics\. All runs are averaged across five independent training runs\.

#### C\.3Support and frequency probes

We report the additional support versus frequency results for the synthetic setup\.

![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L32K64T32/triplet_metrics/triplet_metrics.png)\(a\)Absorbing diffusion
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L32K64T32/triplet_metrics/triplet_metrics.png)\(b\)Uniform diffusion

Figure 5:Results forH=32H=32,K=64K=64,T=32T=32\.![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L64K32T64/triplet_metrics/triplet_metrics.png)\(a\)Absorbing diffusion
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L64K32T64/triplet_metrics/triplet_metrics.png)\(b\)Uniform diffusion

Figure 6:Results forH=64H=64,K=32K=32,T=64T=64\.![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L64K32T128/triplet_metrics/triplet_metrics.png)\(a\)Absorbing diffusion
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L64K32T128/triplet_metrics/triplet_metrics.png)\(b\)Uniform diffusion

Figure 7:Results forH=64H=64,K=32K=32,T=128T=128\.
#### C\.4Modified Inference\-Time Algorithms

As motivated by our theory, we split the inference\-time procedure into two phases:

- •Phase 1:Standard decoding up to a small noise levelσ\\sigma,
- •Phase 2:Modified sampling based on the derived rate separation\.

The role of the first phase is that of the standard sampler—it denoises the initial samplex0∼punifx\_\{0\}\\sim p\_\{\\text\{unif\}\}, but only up until a certain point\. Since our theory suggests thatpt−1\|thp^\{h\}\_\{t\-1\|t\}implicitly learns a classifier for the distance to the supportD=supp​\(pdata\)D=\\text\{supp\}\(p\_\{\\text\{data\}\}\), we explicitly make use of this fact by emulating such a projection\.

[Corollary˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmcorollary1)directly motivates the following algorithm, which assumes that there is a threshold separating the direction, but thepdatap\_\{\\text\{data\}\}\-dependent coefficient in the expansion in[Theorem˜2\.1](https://arxiv.org/html/2605.13999#S2.Thmtheorem1)is not well\-estimated\. \(We write the algorithms for the uniform case; in the absorbing case, the only difference is that we should replaceKKby11\)\.

Algorithm 1Threshold samplingInitial distribution

p^σ\\hat\{p\}\_\{\\sigma\}, normalized

s:=pt−1\|thβt/Ks:=\\frac\{p^\{h\}\_\{t\-1\|t\}\}\{\\beta\_\{t\}/K\}\(where

t=tσ​\(T\)t=t\_\{\\sigma\}\(T\)\), threshold

τ\\tau
Sample

X0∼p^σX\_\{0\}\\sim\\hat\{p\}\_\{\\sigma\}\.

for

t=1,2,…t=1,2,\\dotsdo

Set

Sproj:=\{\(h,yh\)∈\[H\]×\[K\]∣\(σ/K\)​s​\(Xt\)h,yh\>τ\}S\_\{\\text\{proj\}\}:=\\\{\(h,y^\{h\}\)\\in\[H\]\\times\[K\]\\mid\(\\sigma/K\)s\(X\_\{t\}\)\_\{h,y^\{h\}\}\>\\tau\\\}
if

Sproj≠∅S\_\{\\text\{proj\}\}\\neq\\emptysetthen

Sample

\(h,yh\)∈Sproj\(h,y^\{h\}\)\\in S\_\{\\text\{proj\}\}u\.a\.r\.

Xt\+1=\(Xt−h,yh\)X\_\{t\+1\}=\(X\_\{t\}^\{\-h\},y^\{h\}\)

else

return

XtX\_\{t\}

A simple variant that is more robust to the error assumption but produces samples with worse diversity is the following greedy algorithm\.

Algorithm 2Hard\-max samplingInitial distribution

p^σ\\hat\{p\}\_\{\\sigma\}, effective score

s:=pt−1\|thβt/Ks:=\\frac\{p^\{h\}\_\{t\-1\|t\}\}\{\\beta\_\{t\}/K\}\(where

t=tσ​\(T\)t=t\_\{\\sigma\}\(T\)\), threshold

τ\\tau
Sample

X0∼p^σX\_\{0\}\\sim\\hat\{p\}\_\{\\sigma\}\.

for

t=1,2,…t=1,2,\\dotsdo

Sample

\(h,yh\)∈arg⁡max⁡\{\(σ/K\)​s​\(Xt\)h,yh\}\(h,y^\{h\}\)\\in\\arg\\max\\left\\\{\{\(\\sigma/K\)s\(X\_\{t\}\)\_\{h,y^\{h\}\}\}\\right\\\}
if

\(σ/K\)​s​\(Xt\)h,yh\>τ\(\\sigma/K\)s\(X\_\{t\}\)\_\{h,y^\{h\}\}\>\\tauthen

Xt\+1=\(Xt−h,yh\)X\_\{t\+1\}=\(X\_\{t\}^\{\-h\},y^\{h\}\)

else

return

XtX\_\{t\}

For all our experiments, we choose a noise level ofσ=0\.1\\sigma=0\.1and threshold of0\.050\.05\. We have also observed the same effects robustly across a range of other noise levels and thresholds for both uniform and masked diffusion, which we omit here for the sake of clarity of presentation\.

#### C\.5Extended sampler comparison

We provide extended details about the evaluation of masked and uniform diffusion with the modified samplers\. Throughout, \(a\) and \(b\) measure the fraction of samples in the support and the average distance to the support of the samples across training checkpoints, respectively\. In \(c\), we evaluate the final checkpoint with and without the modified samplers, and measure the fraction of their samples landing in the support\. In \(d\), we directly compare the distance of these samples, displaying the fraction of samples in which the modified sampler wins \(green\), draws \(gray\), or loses against the standard sampler in terms of getting closer to the support in Hamming distance\.

Masked diffusion

![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L32K64T64/sum_n0p1_t6_thr0p05/summary_frac_support.png)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L32K64T64/sum_n0p1_t6_thr0p05/summary_mean_dist.png)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L32K64T64/ckpt19_n0p1_t6_thr0p05/ckpt19_frac_support.png)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L32K64T64/ckpt19_n0p1_t6_thr0p05/ckpt19_wdl.png)
Uniform diffusion

![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L32K64T64/sum_n0p1_t6_thr0p05/summary_frac_support.png)\(a\)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L32K64T64/sum_n0p1_t6_thr0p05/summary_mean_dist.png)\(b\)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L32K64T64/ckpt19_n0p1_t6_thr0p05/ckpt19_frac_support.png)\(c\)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L32K64T64/ckpt19_n0p1_t6_thr0p05/ckpt19_wdl.png)\(d\)

Figure 8:Results forH=32H=32,K=64K=64,T=64T=64\(see[Figure˜3](https://arxiv.org/html/2605.13999#S3.F3)\): \(a\) Fraction in support across training; \(b\) Distance to support across training; \(c\) Fraction in support \(final checkpoint\); \(d\) Win rate: Distance to support \(final checkpoint\)Masked diffusion

![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L32K64T32/sum_n0p1_t3_thr0p05/summary_frac_support.png)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L32K64T32/sum_n0p1_t3_thr0p05/summary_mean_dist.png)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L32K64T32/ckpt19_n0p1_t3_thr0p05/ckpt19_frac_support.png)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L32K64T32/ckpt19_n0p1_t3_thr0p05/ckpt19_wdl.png)
Uniform diffusion

![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L32K64T32/sum_n0p1_t3_thr0p05/summary_frac_support.png)\(a\)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L32K64T32/sum_n0p1_t3_thr0p05/summary_mean_dist.png)\(b\)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L32K64T32/ckpt19_n0p1_t3_thr0p05/ckpt19_frac_support.png)\(c\)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L32K64T32/ckpt19_n0p1_t3_thr0p05/ckpt19_wdl.png)\(d\)

Figure 9:Results forH=32H=32,K=64K=64,T=32T=32: \(a\) Fraction in support across training; \(b\) Distance to support across training; \(c\) Fraction in support \(final checkpoint\); \(d\) Win rate: Distance to support \(final checkpoint\)Masked diffusion

![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L64K32T64/sum_n0p1_t6_thr0p05/summary_frac_support.png)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L64K32T64/sum_n0p1_t6_thr0p05/summary_mean_dist.png)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L64K32T64/ckpt19_n0p1_t6_thr0p05/ckpt19_frac_support.png)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L64K32T64/ckpt19_n0p1_t6_thr0p05/ckpt19_wdl.png)
Uniform diffusion

![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L64K32T64/sum_n0p1_t6_thr0p05/summary_frac_support.png)\(a\)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L64K32T64/sum_n0p1_t6_thr0p05/summary_mean_dist.png)\(b\)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L64K32T64/ckpt19_n0p1_t6_thr0p05/ckpt19_frac_support.png)\(c\)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L64K32T64/ckpt19_n0p1_t6_thr0p05/ckpt19_wdl.png)\(d\)

Figure 10:Results forH=64H=64,K=32K=32,T=64T=64: \(a\) Fraction in support across training; \(b\) Distance to support across training; \(c\) Fraction in support \(final checkpoint\); \(d\) Win rate: Distance to support \(final checkpoint\)Masked diffusion

![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L64K32T128/sum_n0p1_t13_thr0p05/summary_frac_support.png)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L64K32T128/sum_n0p1_t13_thr0p05/summary_mean_dist.png)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L64K32T128/ckpt19_n0p1_t13_thr0p05/ckpt19_frac_support.png)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/absorb_L64K32T128/ckpt19_n0p1_t13_thr0p05/ckpt19_wdl.png)
Uniform diffusion

![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L64K32T128/sum_n0p1_t13_thr0p05/summary_frac_support.png)\(a\)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L64K32T128/sum_n0p1_t13_thr0p05/summary_mean_dist.png)\(b\)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L64K32T128/ckpt19_n0p1_t13_thr0p05/ckpt19_frac_support.png)\(c\)
![Refer to caption](https://arxiv.org/html/2605.13999v1/figures/app_synthetic_shortened/unif_L64K32T128/ckpt19_n0p1_t13_thr0p05/ckpt19_wdl.png)\(d\)

Figure 11:Results forH=64H=64,K=32K=32,T=128T=128: \(a\) Fraction in support across training; \(b\) Distance to support across training; \(c\) Fraction in support \(final checkpoint\); \(d\) Win rate: Distance to support \(final checkpoint\)
#### C\.6Computational resources

All runs in this section were performed on a single NVIDIA H100 Tensor Core GPU with 96GB available GPU memory \(which was never used to completion by our experiments\)\. Each run and evaluation was performed in under one hour on one such machine\. The total compute to reproduce the experiment is thus this, multiplied by the number of seeds \(five\) and the number of setups \(eight in total\)\. Any additional compute was spent on developing and testing these training runs\.

### Appendix DReal\-data masked\-DLM probes on FineWeb

This appendix makes the FineWeb experiment in[Section˜3\.1](https://arxiv.org/html/2605.13999#S3.SS1)self\-contained\. The experiment asks whether the support–frequency separation predicted by our small\-noise analysis is visible in a diffusion language model trained on ordinary web text\.

#### D\.1Why the real\-data probe is indirect

In the synthetic experiments, the supportD=supp​\(pdata\)D=\\text\{supp\}\(p\_\{\\text\{data\}\}\)is known: we can directly test whether a sample lies inDD, estimate its distance toDD, and identify local moves that decrease this distance\. For web text, no such oracle exists\. There is no tractable list of all valid strings\.

We therefore use a held\-out one\-token restoration proxy\. From held\-out FineWeb text, we collect local contextsc=\(ℓ,r\)c=\(\\ell,r\)and the set of center tokens that occur betweenℓ\\ellandrr\. For each context, this gives an empirical candidate setCcC\_\{c\}\. A token inCcC\_\{c\}is not guaranteed to be the full linguistic support for the context, but it is a concrete support\-like object measured from held\-out data\. The real\-data question is whether the model first learns to separateCcC\_\{c\}from non\-candidates, and only later learns the empirical frequency ordering insideCcC\_\{c\}\.

For instance, suppose the local context isthe \[MASK\] of\. We scan held\-out FineWeb and count every center token that appears between the left tokentheand the right tokenof\. If the held\-out text containsthe end of, thenendis an empirical candidate for this context\. A non\-candidate is a token that appears elsewhere in the same context bank but was not observed betweentheandof\. The support probe asks whether the model scores attested fillers above such slot\-specific non\-candidates\. The frequency probe asks a different question: among attested fillers such asend,rest, andstate, does the model rank the more frequent fillers above the rarer ones? The implementation uses GPT\-2 tokens rather than word\-level strings, but the logic is exactly this slot\-filling comparison\.

#### D\.2Model and training recipe

We train a masked diffusion language model on FineWeb\. The model belongs to the absorbing\-mask, clean\-token\-prediction family reviewed in[Appendix˜A](https://arxiv.org/html/2605.13999#A1)\.

The model uses the GPT\-2 tokenizer vocabulary of sizeK=50,257K=50\{,\}257and appends one absorbing mask token\. The context length isH=1024H=1024\. The denoiser is a DDiT transformer with 12 layers, 12 attention heads, hidden width 768, RoPE position embeddings, AdaLN conditioning layers with conditioning dimension 128, dropout0\.10\.1, no time conditioning, and untied input/output embeddings\. The resulting model has169\.6169\.6M trainable parameters, of which131\.0131\.0M are non\-embedding parameters\. In the pseudocode below we keepσ\\sigmain the notation to identify the standard MDLM mask probability, but this particular network’s logits depend only on the corrupted token sequence\.

Algorithm 3FineWeb masked\-DLM training update1:Clean token batch

x∈\[K\]B×Hx\\in\[K\]^\{B\\times H\}, mask token

mm, noise floor

εnoise=10−6\\varepsilon\_\{\\rm noise\}=10^\{\-6\}
2:Draw

ui∈\[0,1\]u\_\{i\}\\in\[0,1\]and set

ti=εnoise\+\(1−εnoise\)​uit\_\{i\}=\\varepsilon\_\{\\rm noise\}\+\(1\-\\varepsilon\_\{\\rm noise\}\)u\_\{i\}
3:Set mask probability

σi=\(1−εnoise\)​ti\\sigma\_\{i\}=\(1\-\\varepsilon\_\{\\rm noise\}\)t\_\{i\}and keep probability

αi=1−σi\\alpha\_\{i\}=1\-\\sigma\_\{i\}
4:Form

x~\\tilde\{x\}by replacing each token

xihx\_\{i\}^\{h\}by

mmindependently with probability

σi\\sigma\_\{i\}
5:Compute logits

nnθ​\(x~\)\\mathrm\{nn\}\_\{\\theta\}\(\\tilde\{x\}\)and clean\-token probabilities

nn^θ,h\(⋅∣x~i\)\\widehat\{\\mathrm\{nn\}\}\_\{\\theta,h\}\(\\cdot\\mid\\tilde\{x\}\_\{i\}\)for all positions

hh
6:Minimize the token\-mean SUBS loss

1B​H​∑i,h𝟏​\{x~ih=m\}​1−εnoiseσi​\[−log⁡nn^θ,h​\(xih∣x~i\)\]\.\\frac\{1\}\{BH\}\\sum\_\{i,h\}\\mathbf\{1\}\\\{\\tilde\{x\}\_\{i\}^\{h\}=m\\\}\\frac\{1\-\\varepsilon\_\{\\rm noise\}\}\{\\sigma\_\{i\}\}\\bigl\[\-\\log\\widehat\{\\mathrm\{nn\}\}\_\{\\theta,h\}\(x\_\{i\}^\{h\}\\mid\\tilde\{x\}\_\{i\}\)\\bigr\]\.
7:Apply one AdamW step with learning rate

3⋅10−43\\cdot 10^\{\-4\}, betas

\(0\.9,0\.999\)\(0\.9,0\.999\), and

εAdam=10−8\\varepsilon\_\{\\rm Adam\}=10^\{\-8\}

FineWeb training runs use 8 H100 80GB GPUs with a total batch size of409,600409\{,\}600tokens \(50 sequences of length 1024 per GPU\)\. Each run \(170M parameters, 354M tokens\) takes about 4 minutes of training time \(∼\\sim7 minutes wall\-clock\) per seed\. Probe jobs are run separately on a single GPU and take about 1h44 in total\.

Note that the support and frequency probes in[Figure˜1](https://arxiv.org/html/2605.13999#S1.F1)do not depend on a sampler choice: they score one\-token masked contexts with the clean\-token predictor described in[Appendix˜A](https://arxiv.org/html/2605.13999#A1)\. However, we had to choose a sampler to check the sampling capabilities of the trained models, as reported in[Section˜D\.3](https://arxiv.org/html/2605.13999#A4.SS3)\. The sampler chosen for this auxiliary diagnostic will be detailed then\.

For reference, we use the cached GPT\-2\-token shard stream fromkjj0/fineweb10B\-gpt2\. This is under MIT license, while the underlying text remains FineWeb and is covered by the Open Data Commons Attribution License \(ODC\-By\) v1\.0 and is also subject to the CommonCrawl terms of use\.

#### D\.3Training loss curves and samples

[Figure˜12](https://arxiv.org/html/2605.13999#A4.F12)shows the denoising loss logged during the FineWeb masked\-diffusion runs used in[Figure˜1](https://arxiv.org/html/2605.13999#S1.F1)\. Each run logs a stochastic training loss at every optimizer step\. To make the trend readable, we smooth each seed with a trailing 25\-step average, meaning that each plotted training point averages the current logged loss with the previous 24 logged losses when available, and then we average the resulting traces across seeds on the shared training window\. They show a smooth optimization trajectory over the 0\.354B\-token window used in[Figure˜1](https://arxiv.org/html/2605.13999#S1.F1)\. Diamonds mark the validation evaluations at the initial and final steps\.

![Refer to caption](https://arxiv.org/html/2605.13999v1/x2.png)Figure 12:Logged denoising losses for the FineWeb masked\-diffusion runs\. The training line is the seed mean after a trailing 25\-step moving average\. Diamonds mark the validation evaluations available in the logs\.We also extend training to3\.933\.93B\-token for one of the seeds to give an illustrative example of how losses continue to evolve past the window used in[Figure˜1](https://arxiv.org/html/2605.13999#S1.F1)\. For this extended run, we also logged validation every 120 optimization steps\. This gives a denser single\-seed view of the same quantity over the longer trajectory, see[Figure˜13](https://arxiv.org/html/2605.13999#A4.F13)\.

![Refer to caption](https://arxiv.org/html/2605.13999v1/x3.png)Figure 13:Logged denoising losses for the3\.933\.93B\-token training run\. The training curve uses the same trailing 25\-step moving average as[Figure˜12](https://arxiv.org/html/2605.13999#A4.F12); diamonds mark the validation evaluations from that continuation\.We also sampled at different stages during the extended run on3\.93\.9B tokens\. The sampler \([Algorithm˜4](https://arxiv.org/html/2605.13999#alg4)\) is basic ancestral sampling, as per Eq\. 7 insahoo2024simple, specialized to the log\-linear mask schedule\.

Algorithm 4Default absorbing\-mask ancestral sampler1:Number of reverse steps

TT, sampling endpoint

εsample=10−5\\varepsilon\_\{\\rm sample\}=10^\{\-5\}, target length

HH
2:Initialize

xxas the all\-mask sequence of length

HH
3:for

t=0,…,T−1t=0,\\ldots,T\-1do

4:Set mask probabilities

σt=1−t​\(1−εsample\)/T\\sigma\_\{t\}=1\-t\(1\-\\varepsilon\_\{\\rm sample\}\)/Tand

σs=σt−\(1−εsample\)/T\\sigma\_\{s\}=\\sigma\_\{t\}\-\(1\-\\varepsilon\_\{\\rm sample\}\)/T
5:Evaluate

nn^θ,h\(⋅∣x\)\\widehat\{\\mathrm\{nn\}\}\_\{\\theta,h\}\(\\cdot\\mid x\)for all positions

hh; positions already unmasked are forced to keep their current token

6:foreach still\-masked position

hhdo

7:Sample from the unnormalized distribution with mass

\(σt−σs\)​nn^θ,h​\(a∣x\)\(\\sigma\_\{t\}\-\\sigma\_\{s\}\)\\widehat\{\\mathrm\{nn\}\}\_\{\\theta,h\}\(a\\mid x\)on each vocabulary token

a∈\[K\]a\\in\[K\]
8:Assign mass

σs\\sigma\_\{s\}to staying masked

9:Fill any remaining mask at position

hhby

arg⁡maxa∈\[K\]⁡nn^θ,h​\(a∣x\)\\arg\\max\_\{a\\in\[K\]\}\\widehat\{\\mathrm\{nn\}\}\_\{\\theta,h\}\(a\\mid x\)

While strong generation quality is not expected—given the model’s small size \(under 200M parameters vs\. billions for models like LlaDA\[nie2025large\]\) and limited training data \(a few billion tokens vs\. trillions for LlaDA\)—our goal is simply to assess qualitative improvements over training\. We observe that samples do improve over time and report representative examples below\.

At4\.94\.9M tokens, outputs are mostly token soup\. By the support peak at265~265M tokens and the frequency peaks at334~334M and452~452M tokens, the model can already produce short spans with recognizable local syntax or domain templates, but many generations still collapse into repetitions, malformed entities, or abrupt topic shifts\. Extending the run to3\.933\.93B tokens, some samples start sustaining a longer topic for several sentences, yet the outputs remain clearly below high\-quality web text\.

##### Illustrative samples\.

4\.94\.9M tokens\.best\) arise classification companies him community first\.\.\. mechanism game P weuner didnall Amenweb years Justice emailf willing childen active run designedty\! York your options bit Gods allSized Matthew alerts’ institutions\.

265\.4265\.4M tokens \(support peak\)\.The Pulitzer award was the award for sponsoring Workstanding Department of Kids Walk for a free attendance performed in one of the upcoming family time\. She currently previously include the Basketball Line Scholarship and the National Theater Heritage Center of Year’s members of The International Park\.

334\.2334\.2M tokens \(pairwise\-frequency peak\)\.A reputable or company cannot claim protection in the neighbourhood that give your loan to a real estate\. Yes, you will currently re\-dating a reasonable e\-gor claim\.

452\.2452\.2M tokens \(top\-1\-frequency peak\)\.To locate that language corresponds very very quickly, so it is difficult to accomplish effective representations\. In fact, the analogy is simplified so allows the language in the subject\.

3\.933\.93B tokens\.As a freelance writing editor, it have been important to consider precisely what it may be if you’re working on writing for a New York agency\. You usually need someone as you might to manage this project in place and work to work with someone who will ultimately help your clients\.

#### D\.4How one\-token scores are read

The notation follows[Appendix˜A](https://arxiv.org/html/2605.13999#A1)\. The denoiser produces one logit for each vocabulary token at each position\. After a softmax, these logits define the clean\-token predictor

nn^θ,h​\(a∣x~\),\\widehat\{\\mathrm\{nn\}\}\_\{\\theta,h\}\(a\\mid\\tilde\{x\}\),wherex~\\tilde\{x\}is the corrupted sequence given to the model,hhis the queried position, anda∈\[K\]a\\in\[K\]is a candidate clean token\. In the real\-data model used here, the network is not explicitly time\-conditioned, so we suppress the time argument that appears in the more general notationnn^θ,h​\(a∣xt,t\)\\widehat\{\\mathrm\{nn\}\}\_\{\\theta,h\}\(a\\mid x\_\{t\},t\)\.

For a one\-token contextc=\(ℓ,r\)c=\(\\ell,r\), we form the corrupted sequence

x~\(c\)=\(ℓ,\[MASK\],r\)\\tilde\{x\}^\{\(c\)\}=\(\\ell,\\texttt\{\[MASK\]\},r\)and read the score at the masked center positionhch\_\{c\}:

sθ​\(a∣c\)=log⁡nn^θ,hc​\(a∣x~\(c\)\)s\_\{\\theta\}\(a\\mid c\)=\\log\\widehat\{\\mathrm\{nn\}\}\_\{\\theta,h\_\{c\}\}\(a\\mid\\tilde\{x\}^\{\(c\)\}\)for each possible center tokenaa\. Equivalently, one may use the raw logits, since softmax and logarithm preserve the ranking\. Candidates and negatives for the same context are always compared using the same corrupted inputx~\(c\)\\tilde\{x\}^\{\(c\)\}, so all probe metrics depend only on the relative denoiser scores of alternative fillers for that slot\.

#### D\.5Context bank

The held\-out context bank is built from FineWeb validation token shards\. We use one left token and one right token around the center token,

c=\(xh−1,xh\+1\)\.c=\(x^\{h\-1\},x^\{h\+1\}\)\.We keep contexts with at least 8 observed center\-token occurrences, evaluate up to 8192 contexts, and cap each empirical candidate set at 32 tokens\. The cap keeps evaluation tractable while retaining the repeated\-context structure needed to estimate empirical within\-context frequencies\.

In the context bank used for the FineWeb figures, the median context has 22 observed center\-token occurrences and 12 distinct observed center tokens\. The mean context has 54 observed occurrences and 23 distinct observed center tokens\. Because candidate sets are capped at 32 tokens, extremely broad contexts are truncated to their most frequent fillers; this cap retains96\.7%96\.7\\%of the empirical center\-token mass on average and100%100\\%at the median context\.

#### D\.6Support probe

For each contextcc, letCcC\_\{c\}be its empirical candidate set and letNcN\_\{c\}be a set of non\-candidate negatives\. A non\-candidate is not claimed to be ungrammatical in English\. It is simply a token that was not observed as a center filler for this particular held\-out context\. The support probe is the candidate\-vs\-negative pairwise accuracy

S​\(θ\)=1\|𝒞\|​∑c∈𝒞1\|Cc\|​\|Nc\|​∑a∈Cc∑b∈Nc𝟙​\{sθ​\(a∣c\)\>sθ​\(b∣c\)\}\.\\displaystyle S\(\\theta\)=\\frac\{1\}\{\|\\mathcal\{C\}\|\}\\sum\_\{c\\in\\mathcal\{C\}\}\\frac\{1\}\{\|C\_\{c\}\|\|N\_\{c\}\|\}\\sum\_\{a\\in C\_\{c\}\}\\sum\_\{b\\in N\_\{c\}\}\\mathbbm\{1\}\\left\\\{s\_\{\\theta\}\(a\\mid c\)\>s\_\{\\theta\}\(b\\mid c\)\\right\\\}\.\(23\)The primary version uses 128*bank frequency\-matched*negatives per context\. The matching is by global token frequency in the held\-out context bank, not by syntax or semantics\. Concretely, define

g​\(a\)=∑c∈𝒞nc​\(a\),g\(a\)=\\sum\_\{c\\in\\mathcal\{C\}\}n\_\{c\}\(a\),the total number of times tokenaaappears as an empirical center candidate anywhere in the context bank\. For a contextcc, we select negativesb∉Ccb\\notin C\_\{c\}from the same context bank withlog⁡\(1\+g​\(b\)\)\\log\(1\+g\(b\)\)close to thelog⁡\(1\+g​\(a\)\)\\log\(1\+g\(a\)\)values of the positive candidatesa∈Cca\\in C\_\{c\}\. The logarithm is intentional: context\-bank token counts are heavy\-tailed, and matching in log\-count space matches candidates and negatives by frequency scale rather than by raw absolute count\. In words, if a positive filler is a common token in the context bank, its negative competitor is also chosen to be common; if the positive is rare, its negative competitor is chosen to be rare\. This prevents the model from passing the support probe by simply assigning high scores to globally frequent words\. We also test other types of negative examples, such as random ones and very frequent tokens from the whole bank\. The support signal appears at the same stage under all choices, see[Figure˜14](https://arxiv.org/html/2605.13999#A4.F14)\. We therefore use frequency\-matched negatives in the main text, as a balanced choice: they are neither too hard \(like separating against the most frequent negative tokens\) nor too easy \(like separating against random negative tokens\)\.

![Refer to caption](https://arxiv.org/html/2605.13999v1/x4.png)Figure 14:Support probe under different choices of non\-candidate negatives\. Uniform negatives are easiest; negatives drawn from frequent tokens in the context bank are harder; frequency\-matched negatives are the primary curve used in the main text because they compare candidates against non\-candidates with similar global frequency\. The support signal rises at the same time under all choices\.
#### D\.7Frequency probes

The frequency probes are evaluated only inside the empirical candidate setCcC\_\{c\}\. Returning to the example above, after we know thatend,rest, andstateare attested fillers forthe \[MASK\] of, the frequency probes ignore all non\-candidates and ask whether the model orders those attested fillers according to their held\-out counts\. Letnc​\(a\)n\_\{c\}\(a\)be the held\-out count of candidateaain contextcc\. The pairwise frequency probe is

Fpair​\(θ\)=1\|𝒞\|​∑c∈𝒞1\|𝒫c\|​∑\(a,b\)∈𝒫c𝟙​\{sθ​\(a∣c\)\>sθ​\(b∣c\)\},𝒫c=\{\(a,b\):nc​\(a\)\>nc​\(b\)\}\.\\displaystyle F\_\{\\mathrm\{pair\}\}\(\\theta\)=\\frac\{1\}\{\|\\mathcal\{C\}\|\}\\sum\_\{c\\in\\mathcal\{C\}\}\\frac\{1\}\{\|\\mathcal\{P\}\_\{c\}\|\}\\sum\_\{\(a,b\)\\in\\mathcal\{P\}\_\{c\}\}\\mathbbm\{1\}\\left\\\{s\_\{\\theta\}\(a\\mid c\)\>s\_\{\\theta\}\(b\\mid c\)\\right\\\},\\quad\\mathcal\{P\}\_\{c\}=\\\{\(a,b\):n\_\{c\}\(a\)\>n\_\{c\}\(b\)\\\}\.\(24\)The top\-1 frequency probe asks whether the highest\-scored candidate is the empirical mode:

Ftop1​\(θ\)=1\|𝒞\|​∑c∈𝒞𝟙​\{arg⁡maxa∈Cc⁡sθ​\(a∣c\)=arg⁡maxa∈Cc⁡nc​\(a\)\}\.\\displaystyle F\_\{\\mathrm\{top1\}\}\(\\theta\)=\\frac\{1\}\{\|\\mathcal\{C\}\|\}\\sum\_\{c\\in\\mathcal\{C\}\}\\mathbbm\{1\}\\left\\\{\\arg\\max\_\{a\\in C\_\{c\}\}s\_\{\\theta\}\(a\\mid c\)=\\arg\\max\_\{a\\in C\_\{c\}\}n\_\{c\}\(a\)\\right\\\}\.\(25\)Thus the support probe asks whether the model has localized the right set of plausible center tokens, while the frequency probes ask whether it has learned their relative empirical probabilities\.

#### D\.8Transition criterion

For a trajectory metricMtM\_\{t\}, we report the first checkpoint reaching a fixed fraction of its peak gain:

τq​\(M\)=min⁡\{t:Mt≥M0\+q​\(maxt′⁡Mt′−M0\)\},q=0\.9\.\\displaystyle\\tau\_\{q\}\(M\)=\\min\\\{t:M\_\{t\}\\geq M\_\{0\}\+q\(\\max\_\{t^\{\\prime\}\}M\_\{t^\{\\prime\}\}\-M\_\{0\}\)\\\},\\qquad q=0\.9\.\(26\)This criterion compares the onset of support localization and frequency sharpening without requiring either curve to saturate at exactly one\.

For the main learning\-rate schedule, averaged over three independent seeds, the frequency\-matched support metric reachesτ0\.9\\tau\_\{0\.9\}at116\.3±5\.7116\.3\\pm 5\.7M tokens, while top\-1 and pairwise frequency reach it at234\.3±22\.2234\.3\\pm 22\.2M and247\.4±7\.5247\.4\\pm 7\.5M tokens\. The seed\-wise ranges are113\.0113\.0–122\.9122\.9M,211\.4211\.4–255\.6255\.6M, and240\.8240\.8–255\.6255\.6M, respectively\. The absolute token counts are specific to this model, data window, context bank, and transition criterion\. Their role is to compare support and frequency under matched conditions\.

![Refer to caption](https://arxiv.org/html/2605.13999v1/x5.png)Figure 15:Raw metric levels for the FineWeb masked\-DLM probes\. The support probe rises from0\.6240\.624to a peak of0\.8730\.873candidate\-vs\-negative accuracy, with chance at0\.50\.5\. Top\-1 frequency recovery rises from0\.3620\.362to0\.4640\.464, compared with a random\-candidate baseline of0\.1150\.115\. Pairwise frequency recovery rises from0\.6950\.695to0\.7560\.756, with chance at0\.50\.5\. Dotted vertical lines mark the90%90\\%peak\-gain transition times\.
#### D\.9Generalization and memorization diagnostic

Several recent works study whether diffusion models generalize or memorize training samples\. In particular,bonnaire2025whyidentify two training timescales in continuous diffusion models: an early time at which high\-quality samples appear, and a later time at which memorization can emerge\. Our real\-DLM experiment targets this earlier generalization window\. Within that window, we identify a substructure: support localization and within\-support frequency matching can be temporally decoupled\.

To see that, we include here a train/validation diagnostic to locate the observed support–frequency separation relative to train\-specific memorization\.

For each checkpoint, we evaluate the same clean\-token predictor on fixed train and validation windows using fixed masking levels\. Letxxbe a clean token window, letx~ρ\\tilde\{x\}\_\{\\rho\}be the same window after masking a fraction approximatelyρ\\rhoof positions, and letMρM\_\{\\rho\}be the masked positions\. We compute the masked\-token denoising cross\-entropy

ℒρ​\(θ;x\)=1\|Mρ\|​∑h∈Mρ−log⁡nn^θ,h​\(xh∣x~ρ\)\.\\mathcal\{L\}\_\{\\rho\}\(\\theta;x\)=\\frac\{1\}\{\|M\_\{\\rho\}\|\}\\sum\_\{h\\in M\_\{\\rho\}\}\-\\log\\widehat\{\\mathrm\{nn\}\}\_\{\\theta,h\}\(x^\{h\}\\mid\\tilde\{x\}\_\{\\rho\}\)\.This is the clean\-token prediction loss family used by the masked\-DLM objective, evaluated at fixed masking levels\. We use three masking levels, masking approximately15%15\\%,50%50\\%, or almost all tokens in each window\. For a masking levelρ\\rho, we track

Gρ​\(t\)=ℒval,ρ​\(t\)−ℒtrain,ρ​\(t\)\.\\displaystyle G\_\{\\rho\}\(t\)=\\mathcal\{L\}\_\{\\mathrm\{val\},\\rho\}\(t\)\-\\mathcal\{L\}\_\{\\mathrm\{train\},\\rho\}\(t\)\.\(27\)A growing positive gap would indicate train\-specific memorization\. In the early\-token window used for the support and frequency probes, both train and validation losses improve while the gap remains small\. We do not see the widening positive train advantage expected from train\-specific memorization in the window where support and frequency separate\. See[Figure˜16](https://arxiv.org/html/2605.13999#A4.F16)for this gap, and[Figure˜17](https://arxiv.org/html/2605.13999#A4.F17)for the corresponding absolute train and validation masked\-token denoising cross\-entropies, averaged over the three seeds\.

![Refer to caption](https://arxiv.org/html/2605.13999v1/x6.png)Figure 16:Train/validation diagnostic for the FineWeb masked\-DLM trajectory\. In the support–frequency transition window, train and validation masked\-token denoising losses improve together and the validation\-minus\-train gap remains small across masking levels\.![Refer to caption](https://arxiv.org/html/2605.13999v1/x7.png)Figure 17:Absolute train and validation masked\-token denoising cross\-entropies for the FineWeb masked\-DLM trajectory\. Curves are averages over the three seeds at three masking levels\.
#### D\.10Learning\-rate schedule robustness

Because these transitions occur early in training, we check that the conclusion is not tied to a single learning\-rate schedule\. We repeat the probe suite with the same data, context bank, model, and token window, but insert a 384\-step learning\-rate warmup\. The support transition moves later under this schedule, while the frequency transitions do not move by the same amount\. In this single\-seed control, the frequency\-matched support transition occurs at186\.8186\.8M tokens, top\-1 mode recovery at191\.7191\.7M tokens, and pairwise frequency ranking at255\.6255\.6M tokens\. Thus the exact margin between support and top\-1 frequency is schedule\-sensitive, but the ordering is not reversed and the pairwise frequency ranking remains later\.

![Refer to caption](https://arxiv.org/html/2605.13999v1/x8.png)Figure 18:Learning\-rate schedule robustness\. Labeled dotted lines show the90%90\\%peak\-gain transition for each curve\. Adding a 384\-step learning\-rate warmup delays the support transition and brings the top\-1 frequency transition close to it, while pairwise frequency ranking remains later\.
#### D\.11Synthetic counterpart

As a check on the interpretation of the probes used on FineWeb, we also run the same experiment in a synthetic setting where the true distributionpdatap\_\{\\text\{data\}\}is known\. In this setting, direct versions of the probes considered before are available, so we can check whether the indirect versions used on FineWeb track their exact oracle counterpart in this synthetic case\.[Figure˜19](https://arxiv.org/html/2605.13999#A4.F19)shows that they do\. We now detail the setup below\.

![Refer to caption](https://arxiv.org/html/2605.13999v1/x9.png)Figure 19:Synthetic counterpart to the FineWeb probes in[Equations˜23](https://arxiv.org/html/2605.13999#A4.E23)and[24](https://arxiv.org/html/2605.13999#A4.E24)\.Each panel uses length\-6464sequences over 32 symbols and changes either the number of training sequences relative to sequence length \(n/Ln/L\) or the diffusion horizon relative to sequence length \(T/LT/L\)\. Solid curves use held\-out empirical context counts, as in FineWeb, dashed curves use the exact random\-walk oracle and are almost everywhere coinciding with the indirect versions\.Instead of defining candidate center tokens by held\-out counts as in[Equation˜23](https://arxiv.org/html/2605.13999#A4.E23), we define the support probe from the exact conditional support\{a:p​\(a∣ℓ,r\)\>0\}\\\{a:p\(a\\mid\\ell,r\)\>0\\\}, and instead of ranking candidates by empirical held\-out frequencies as in[Equation˜24](https://arxiv.org/html/2605.13999#A4.E24), we rank them by the exact conditional probabilitiesp​\(a∣ℓ,r\)p\(a\\mid\\ell,r\)\.

For a local contextc=\(ℓ,r\)c=\(\\ell,r\), let

qc\(a\):=pdata\(Xh=a∣Xh−1=ℓ,Xh\+1=r\)=P​\(a∣ℓ\)​P​\(r∣a\)∑bP​\(b∣ℓ\)​P​\(r∣b\),\\displaystyle q\_\{c\}\(a\):=p\_\{\\text\{data\}\}\(X^\{h\}=a\\mid X^\{h\-1\}=\\ell,X^\{h\+1\}=r\)=\\frac\{P\(a\\mid\\ell\)P\(r\\mid a\)\}\{\\sum\_\{b\}P\(b\\mid\\ell\)P\(r\\mid b\)\},\(28\)whereP\(⋅∣⋅\)P\(\\cdot\\mid\\cdot\)is the known random\-walk transition matrix\. LetCc⋆=\{a:qc​\(a\)\>0\}C\_\{c\}^\{\\star\}=\\\{a:q\_\{c\}\(a\)\>0\\\}be the exact candidate set, and letNc⋆N\_\{c\}^\{\\star\}be the frequency\-matched non\-candidate set obtained by the same global\-frequency matching rule as in the FineWeb support probe, but excludingCc⋆C\_\{c\}^\{\\star\}\. We use the comparison \(with1/21/2in case of tie\)

ψθc​\(a,b\)=𝟙​\{sθ​\(a∣c\)\>sθ​\(b∣c\)\}\+12​𝟙​\{sθ​\(a∣c\)=sθ​\(b∣c\)\}\.\\psi\_\{\\theta\}^\{c\}\(a,b\)=\\mathbbm\{1\}\\left\\\{s\_\{\\theta\}\(a\\mid c\)\>s\_\{\\theta\}\(b\\mid c\)\\right\\\}\+\\frac\{1\}\{2\}\\mathbbm\{1\}\\left\\\{s\_\{\\theta\}\(a\\mid c\)=s\_\{\\theta\}\(b\\mid c\)\\right\\\}\.The direct support probe computed here is

Sdirect​\(θ\)=1\|𝒞\|​∑c∈𝒞1\|Cc⋆\|​\|Nc⋆\|​∑a∈Cc⋆∑b∈Nc⋆ψθc​\(a,b\)\.\\displaystyle S\_\{\\mathrm\{direct\}\}\(\\theta\)=\\frac\{1\}\{\|\\mathcal\{C\}\|\}\\sum\_\{c\\in\\mathcal\{C\}\}\\frac\{1\}\{\|C\_\{c\}^\{\\star\}\|\|N\_\{c\}^\{\\star\}\|\}\\sum\_\{a\\in C\_\{c\}^\{\\star\}\}\\sum\_\{b\\in N\_\{c\}^\{\\star\}\}\\psi\_\{\\theta\}^\{c\}\(a,b\)\.\(29\)The plotted direct frequency probe is the pairwise ranking accuracy

Fdirect​\(θ\)=1\|𝒞\|​∑c∈𝒞1\|𝒫c⋆\|​∑\(a,b\)∈𝒫c⋆ψθc​\(a,b\),𝒫c⋆=\{\(a,b\)∈Cc⋆×Cc⋆:qc​\(a\)\>qc​\(b\)\}\.\\displaystyle F\_\{\\mathrm\{direct\}\}\(\\theta\)=\\frac\{1\}\{\|\\mathcal\{C\}\|\}\\sum\_\{c\\in\\mathcal\{C\}\}\\frac\{1\}\{\|\\mathcal\{P\}\_\{c\}^\{\\star\}\|\}\\sum\_\{\(a,b\)\\in\\mathcal\{P\}\_\{c\}^\{\\star\}\}\\psi\_\{\\theta\}^\{c\}\(a,b\),\\quad\\mathcal\{P\}\_\{c\}^\{\\star\}=\\\{\(a,b\)\\in C\_\{c\}^\{\\star\}\\times C\_\{c\}^\{\\star\}:q\_\{c\}\(a\)\>q\_\{c\}\(b\)\\\}\.\(30\)
[Figure˜19](https://arxiv.org/html/2605.13999#A4.F19)shows the mean±\\pmone standard deviation over seeds\. The indirect probes based on held\-out counts and the direct oracle probes have the same qualitative behavior\. For the support metric, they coincide on these runs\. For the pairwise frequency metric, their mean absolute difference across checkpoints is about0\.020\.02in accuracy\. Support saturates early, and frequency ranking improves later and remains noisier\. In this synthetic case the support probe is particularly easy and saturates much earlier than frequency\.

This synthetic result is only a sanity check in a synthetic case: it shows that, in a case where the true distribution is known, the held\-out\-count probes do track the exact support and frequency versions of the metrics in[Equations˜23](https://arxiv.org/html/2605.13999#A4.E23)and[24](https://arxiv.org/html/2605.13999#A4.E24)\.

The data are sequences of lengthH=64H=64over the vocabulary\{0,…,K−1\}\\\{0,\\ldots,K\-1\\\}withK=32K=32\. The first token is uniform\. If the current token isvv, the next token is drawn from the three\-token set\{v−1,v,v\+1\}\\\{v\-1,v,v\+1\\\}, clipped at the vocabulary boundaries\. The baseline transition probabilities favor upward moves:p​\(v\+1∣v\)=0\.45p\(v\+1\\mid v\)=0\.45,p​\(v∣v\)=0\.35p\(v\\mid v\)=0\.35, andp​\(v−1∣v\)=0\.20p\(v\-1\\mid v\)=0\.20, with a deterministic token\-dependent renormalization that changes the probabilities across positions while leaving the three\-token support unchanged\. The baseline setting usesn=2048n=2048training sequences \(n/H=32n/H=32\) and diffusion horizonT=128T=128\(T/H=2T/H=2\)\. We also evaluaten/H=16n/H=16andn/H=64n/H=64at fixedT/H=2T/H=2, andT/H=1T/H=1andT/H=3T/H=3at fixedn/H=32n/H=32\. Lowern/Hn/Hmeans fewer distinct training sequences under the same optimizer\-step budget; changingT/HT/Hchanges the number of discrete mask\-noise levels\.