Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Design

# Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Design
Source: [https://arxiv.org/html/2606.15327](https://arxiv.org/html/2606.15327)
Keyue Jiang1 3, Yuxiang Wang1 2, Yanan Zhao4, Xiang Yu1 2, Qifang Zhao1, Bohan Tang5, Baojian Zhou2, Yanghua Xiao2, Lin Qu1, Xiaoxiao Xu1 1Alibaba Group2Fudan University3University College London 4Nanyang Technological University5University of Oxford

###### Abstract

Diffusion Language Models \(DLMs\) have demonstrated strong scaling capacity as alternatives to autoregressive language models\. However, their performance is highly sensitive to the choice of transition kernels, and poorly designed kernels can lead to issues like training instability, slow convergence, and biased sampling\. In this paper, we study this sensitivity through a principled analysis of generalization error and identify three critical factors: asymptotic bias \(difficulty in approximating the posterior distribution\), exposure bias \(error propagation during sampling\), and optimization variance induced by kernel dispersion\. We further compare different transition kernels: masking diffusion yields sparse and easier posterior\-approximation targets, while uniform diffusion provides stronger sampling\-side repair but induces harder approximation\. Motivated by this trade\-off, we revisit a previously overlooked variant, semantic DLM \(SemDLM\), where the transition kernel corrupts tokens to neighborhoods that are semantically similar\. Our theory suggests that SemDLM can serve as a plausible middle ground by reducing the posterior approximation difficulty of uniform diffusion while retaining repair ability\. However, we find that SemDLM suffers from a semantic basin problem, where sampling repeatedly stays within a semantic region and produces low\-diversity text\. To address this, we propose SemDLM\+, which adds a global transition and a semantic\-frequency penalty during sampling\. Experiments on LM1B and OpenWebText show that SemDLM\+ improves training dynamics and achieves competitive language modeling and generation quality with satisfactory diversity\.

## 1Introduction

Diffusion language models \(DLMs\)\(dream2025;nie2025llada\)have emerged as a compelling alternative to autoregressive language models \(ALMs\)Dubey2024TheL3;Yang2024Qwen25TR;DeepSeekAI2024DeepSeekV3TRdue to their parallelizable training and faster decoding speeds\. DLMs need to design a transition kernel to gradually corrupt the clean data to noise, and different kernels can lead to substantially different training and sampling dynamics\. The predominant archetype is absorbing kernel\(sahoo2024simple;DBLP:conf/nips/ShiHWDT24;ou2025your\), as prior work shows that masking DLMs\(austin2021structured;hoogeboom2021argmax\)can effectively alleviate slow convergence, training instability, and weak generalization\(wang202610openchallengessteering\)compared to others such as uniform, marginal, and semantic neighborhood diffusion\(austin2021structured;hoogeboom2021argmax;DBLP:conf/iclr/SchiffSPWBDARPK25;qin2025defog\)\. However, recent studies demonstrated that uniform DLMs can enjoy better scaling capacity when sufficient data and training budget are given\(vonruette2025scalingbehaviordiscretediffusion;DBLP:conf/icml/RutteFDOS025;sahoo2026scalingmaskeddiffusionlanguage;wang2026trainabilitymaskeddiffusionlanguage\)\. This reveals a gap that is not fully explained:

RQ1: Why does uniform diffusion scale well under large resources, while masking diffusion remains stronger in many practical regimes? Can a transition kernel combine the advantages of both?

In this paper, we answer this question by developing a principled error analysis framework for DLMs\. We first decompose the generation error intoApproximation Error,Sampling Error, andForward Kernel Mismatch, and then separate the first two through a bias\-variance lens\. This highlights three kernel\-dependent factors: asymptotic bias that reflects approximation difficulty, exposure bias that measures error accumulation during reverse sampling, and optimization variance that captures finite\-resource instability\. Our analysis uncovers a trade\-off in existing paradigms: masking diffusion has sparse posterior targets that are easier to fit, but provides limited intrinsic repair during sampling\. Uniform diffusion has denser posterior targets that are harder to optimize, but can be preferable for sampling because its reverse dynamics naturally preserve the ability to repair earlier errors\.

This trade\-off motivates us to revisit a previously overlooked variant, namely semantic diffusion language models \(SemDLMs\), where the forward kernel corrupts a token into semantically related tokens\. SemDLMs are theoretically attractive because they restrict the posterior to a meaningful neighborhood, reducing the approximation difficulty relative to uniform diffusion, while still allowing richer transitions than masking diffusion\. However, prior SemDLM designs have not consistently delivered strong generation performance\(austin2021structured;DBLP:journals/corr/abs\-2603\-21342\)\. This leads to our second question:

RQ2: While theoretically plausible, why does SemDLM still underperform existing methods? And how can we translate the theoretical advantages of SemDLM into practical gains?

In our experiments, we find a*semantic basin*issue: the reverse sampling may repeatedly generate semantically adjacent tokens, producing locally plausible but low\-diversity text\. This happens because the semantic likelihood term and the model’s rollout\-induced bias can reinforce sampling within a same semantic cluster\. To solve this issue, we propose SemDLM\+\. First, we add a global transition on top of the semantic transition kernel to prevent the sampling being trapped in some semantic neighborhoods\. Second, we introduce a semantic\-frequency penalty mechanism during sampling that counteracts the rollout\-induced tendency to overproduce tokens from the same semantic basin\. Together, these two mechanisms turn SemDLM into a powerful variants as DLMs: easier to train than fully uniform diffusion, but more repairable during sampling than purely masking\-based diffusion\.

In summary, our contributions are threefold\. 1\) We provide a principled error analysis for DLMs that explains how transition\-kernel design affects approximation difficulty, sampling dynamics, and finite\-resource optimization\. 2\) Guided by this analysis, we develop an improved SemDLM\+, which augments SemDLM with a global transition for sampling repair and semantic\-frequency penalty to avoid semantic\-basin collapse\. These designs make SemDLM practically successful and preserve its properties of efficient training and reliable sampling\. 3\) Experiments on LM1B and OpenWebText show that SemDLM\+ improves training dynamics and achieves strong language modeling and generation performance, highlighting SemDLM\+ as a promising direction for DLM kernel design\.

## 2Preliminaries

We denote byq0q\_\{0\}the data distribution over support𝒳\\mathcal\{X\}, and byq1q\_\{1\}a reference distribution that is easy to sample \(e\.g\., the absorbing or uniform distribution\)\. In language modeling,𝒳\{\\mathcal\{X\}\}represents the space of length\-LLsequences wherex=\(x\(1\),…,x\(L\)\)∈𝒳:=𝒱Lx=\\left\(x^\{\(1\)\},\\ldots,x^\{\(L\)\}\\right\)\\in\\mathcal\{X\}:=\\mathcal\{V\}^\{L\}with𝒱\{\\mathcal\{V\}\}being a vocabulary of size\|𝒱\|=V\|\{\\mathcal\{V\}\}\|=V\. Diffusion models aim to construct a probability pathqt,0≤t≤1q\_\{t\},0\\leq t\\leq 1such that one can sample fromq1q\_\{1\}and transform it through the learned reverse process to get samples that approximately followq0q\_\{0\}\.

Diffusion as Continuous\-time Markov Chains \(CTMC\)\.FollowingDBLP:conf/icml/LouME24, we view the forward noising process to construct the probability path as a CTMC withinfinitesimal generatorQtQ\_\{t\}, i\.e\.,d​qtd​t=Qt​qt,0≤t≤1\\frac\{dq\_\{t\}\}\{dt\}=Q\_\{t\}q\_\{t\},0\\leq t\\leq 1\. One can simulate the forward CTMC via:

Forward Process via Euler Sampling:​q​\(xt\+d​t=y∣xt=z\)=δz​y\+Qt​\(z,y\)​d​t\+O​\(d​t2\)\\text\{Forward Process via Euler Sampling: \}q\\left\(x\_\{t\+dt\}=y\\mid x\_\{t\}=z\\right\)=\\delta\_\{zy\}\+Q\_\{t\}\(z,y\)dt\+O\\left\(dt^\{2\}\\right\)\(1\)
Diffusion deploys a parameterized model to mimic the reverse process,pθ​\(xt−d​t∣xt\)≈q​\(xt−d​t∣xt\)p\_\{\\theta\}\(x\_\{t\-dt\}\\mid x\_\{t\}\)\\approx q\(x\_\{t\-dt\}\\mid x\_\{t\}\), such that one can iteratively sample a trajectory from the reference distribution to the data distribution throughpθ​\(xt−d​t∣xt\)p\_\{\\theta\}\(x\_\{t\-dt\}\\mid x\_\{t\}\)\. A predominantly used parameterization in DLMs isxx\-prediction\(nie2025llada;dream2025;cheng2025sdarsynergisticdiffusionautoregressionparadigm;liu2025wedlmreconcilingdiffusionlanguage\), which buildspθ​\(x0∣xt\)p\_\{\\theta\}\\left\(x\_\{0\}\\mid x\_\{t\}\\right\)instead of directly approximatingq​\(xt−d​t∣xt\)q\(x\_\{t\-dt\}\\mid x\_\{t\}\)as:

pθ​\(xt−d​t∣xt\):=∫x0∈𝒳q​\(xt−d​t∣xt,x0\)​pθ​\(x0∣xt\)​𝑑x0\.p\_\{\\theta\}\\left\(x\_\{t\-dt\}\\mid x\_\{t\}\\right\):=\\int\_\{x\_\{0\}\\in\\mathcal\{X\}\}q\\left\(x\_\{t\-dt\}\\mid x\_\{t\},x\_\{0\}\\right\)p\_\{\\theta\}\\left\(x\_\{0\}\\mid x\_\{t\}\\right\)dx\_\{0\}\.\(2\)The posterior isq​\(xt−d​t∣xt,x0\)∝q​\(xt∣xt−d​t\)​q​\(xt−d​t∣x0\)q\\left\(x\_\{t\-dt\}\\mid x\_\{t\},x\_\{0\}\\right\)\\propto q\\left\(x\_\{t\}\\mid x\_\{t\-dt\}\\right\)q\(x\_\{t\-dt\}\\mid x\_\{0\}\)\. We note that the local transitionqt∣t−d​tq\_\{t\\mid t\-dt\}, the cumulative forwardqt∣0q\_\{t\\mid 0\}, and the generatorQtQ\_\{t\}are equivalent representations of the same forward process\. As such, we describe the process in terms ofqt∣0q\_\{t\\mid 0\}andxx\-prediction in following\.

Training objective\.DLMs optimize over a variational upper bound of negative log\-likelihood\(ho2020ddpm\),−log⁡pθ​\(x0\)≤ℓ0\+ℓprior\+∑tℓt\-\\log p\_\{\\theta\}\(x\_\{0\}\)\\leq\\ell\_\{0\}\+\\ell\_\{\\text\{prior\}\}\+\\sum\_\{t\}\\ell\_\{t\}, withℓt=𝔼xt\[DK​L\(q\(xt−d​t\|xt,x0\)∥pθ\(xt−d​t\|xt\)\)\]\\ell\_\{t\}=\\mathbb\{E\}\_\{x\_\{t\}\}\[D\_\{KL\}\(q\(x\_\{t\-dt\}\|x\_\{t\},x\_\{0\}\)\\\|p\_\{\\theta\}\(x\_\{t\-dt\}\|x\_\{t\}\)\)\],ℓ0=𝔼q​\(x0:1\|x0\)​\[−log⁡pθ​\(x0:1\)\]\\ell\_\{0\}=\\mathbb\{E\}\_\{q\(x\_\{0:1\}\|x\_\{0\}\)\}\[\-\\log p\_\{\\theta\}\(x\_\{0:1\}\)\]andℓprior=DK​L​\(q​\(x1\|x0\)∥pθ​\(x1\)\)\\ell\_\{\\text\{prior\}\}=D\_\{KL\}\(q\(x\_\{1\}\|x\_\{0\}\)\\\|p\_\{\\theta\}\(x\_\{1\}\)\)\. Takingd​t→0dt\\to 0will make the first two terms negligible\. With[Eq\.˜2](https://arxiv.org/html/2606.15327#S2.E2), we can deriveDKL\(q\(xt−d​t∣xt\)∥pθ\(xt−d​t∣xt\)\)≤DKL\(q\(x0∣xt\)∥pθ\(x0∣xt\)\)D\_\{\\mathrm\{KL\}\}\\left\(q\\left\(x\_\{t\-dt\}\\mid x\_\{t\}\\right\)\\\|p\_\{\\theta\}\\left\(x\_\{t\-dt\}\\mid x\_\{t\}\\right\)\\right\)\\leq D\_\{\\mathrm\{KL\}\}\\left\(q\\left\(x\_\{0\}\\mid x\_\{t\}\\right\)\\\|p\_\{\\theta\}\\left\(x\_\{0\}\\mid x\_\{t\}\\right\)\\right\)\(li2023on\), which then yields the common training objective for DLMs:

ℒ\(θ\)=𝔼t,x0,xt∼q​\(xt∣x0\)DKL\(q\(x0∣xt\)∥pθ\(x0∣xt\)\)\.\{\\mathcal\{L\}\}\(\\theta\)=\\mathbb\{E\}\_\{t,x\_\{0\},x\_\{t\}\\sim q\(x\_\{t\}\\mid x\_\{0\}\)\}D\_\{\\mathrm\{KL\}\}\\left\(q\\left\(x\_\{0\}\\mid x\_\{t\}\\right\)\\\|p\_\{\\theta\}\\left\(x\_\{0\}\\mid x\_\{t\}\\right\)\\right\)\.\(3\)Sampling Objective\.After learningpθ​\(x0∣xt\)p\_\{\\theta\}\(x\_\{0\}\\mid x\_\{t\}\), the synthesized data pointsx0x\_\{0\}are generated by iteratively sampling from the induced reverse kernelxt−d​t∼pθ​\(xt−d​t∣xt\)x\_\{t\-dt\}\\sim p\_\{\\theta\}\(x\_\{t\-dt\}\\mid x\_\{t\}\)fromt=1t=1to 0\. The quality of synthesized samples is measured by the generation risk:

ℛ​\(θ\)=DKL​\(q​\(x0\)∥pθ​\(x0\)\)\.\{\\mathcal\{R\}\}\(\\theta\)=D\_\{\\mathrm\{KL\}\}\\left\(q\\left\(x\_\{0\}\\right\)\\\|p\_\{\\theta\}\\left\(x\_\{0\}\\right\)\\right\)\.\(4\)Transition Kernel Design\.With the CTMC framework, we can interpret DLM variants via their transition kernel design\. In this paper, we mainly consider the following variants ofq​\(xt∣x0\)q\(x\_\{t\}\\mid x\_\{0\}\)\.

Absorbing \(Masking\) Transitiondefines a special token\[𝙼𝙰𝚂𝙺\]\\mathtt\{\[MASK\]\}as the absorbing state such that,

q​\(xt=j∣x0=i\)=αt​δi​j\+\(1−αt\)​δj,\[𝙼𝙰𝚂𝙺\],q\(x\_\{t\}=j\\mid x\_\{0\}=i\)=\\alpha\_\{t\}\\delta\_\{ij\}\+\(1\-\\alpha\_\{t\}\)\\delta\_\{j,\\mathtt\{\[MASK\]\}\},\(5\)whereαt\\alpha\_\{t\}is the decay factor andδi​j\\delta\_\{ij\}is the Kronecker delta\.

Uniform transition\.Uniform diffusion spreads the corrupted mass over the full vocabulary:

q​\(xt=j∣x0=i\)=αt​δi​j\+V−1​\(1−αt\),i,j∈𝒱\.q\(x\_\{t\}=j\\mid x\_\{0\}=i\)=\\alpha\_\{t\}\\delta\_\{ij\}\+V^\{\-1\}\(1\-\\alpha\_\{t\}\),\\qquad i,j\\in\\mathcal\{V\}\.\(6\)Semantic Transitionis first introduced inaustin2021structured, which has its kernel defined as,

q​\(xt=j∣x0=i\)=αt​δi​j\+\(1−αt\)​stsem​\(j∣i\),q\(x\_\{t\}=j\\mid x\_\{0\}=i\)=\\alpha\_\{t\}\\delta\_\{ij\}\+\(1\-\\alpha\_\{t\}\)s\_\{t\}^\{\\mathrm\{sem\}\}\(j\\mid i\),\(7\)where the semantic kernel was initially designed as transitions over semantic clusters, such thatstsem​\(j∣i\)=\(kt\)−1​𝕀​\(j∈𝒩kt​\(i\)\)s\_\{t\}^\{\\mathrm\{sem\}\}\(j\\mid i\)=\(k\_\{t\}\)^\{\-1\}\\mathbb\{I\}\\bigl\(j\\in\\mathcal\{N\}\_\{k\_\{t\}\}\(i\)\\bigr\)where𝒩kt​\(i\)\\mathcal\{N\}\_\{k\_\{t\}\}\(i\)is the top\-ktk\_\{t\}semantic neighborhood of tokenii\. Unfortunately, this design suffers from strong training\-sampling mismatch\(DBLP:conf/iclr/NingLSSE24\)that leads to significant performance degeneration\. So we can lightly modify the kernel to make sure the reference distribution matches in forward and reverse process\. This gives the transition as

stsem​\(j∣i\)=exp⁡\(τt−1​sim​\(i,j\)\)/∑k∈𝒱exp⁡\(τt−1​sim​\(i,k\)\),s\_\{t\}^\{\\mathrm\{sem\}\}\(j\\mid i\)=\\exp\(\\tau\_\{t\}^\{\-1\}\\mathrm\{sim\}\(i,j\)\)/\\sum\_\{k\\in\\mathcal\{V\}\}\\exp\(\\tau\_\{t\}^\{\-1\}\\mathrm\{sim\}\(i,k\)\),\(8\)wheresim​\(i,j\)\\mathrm\{sim\}\(i,j\)is a similarity score between embeddings for wordiiandjj;τt\>0\\tau\_\{t\}\>0is a temperature parameter scheduled to monotonically increase fromτt→0,t→0\\tau\_\{t\}\\rightarrow 0,t\\rightarrow 0andτt→\+∞,t→1\\tau\_\{t\}\\rightarrow\+\\infty,t\\rightarrow 1\.

## 3Principled Error Analysis in DLMs through Bias\-Variance Trade\-off

In this section, we first build a principled error analysis for DLMs \([Sec\.˜3\.1](https://arxiv.org/html/2606.15327#S3.SS1)\), where we identify several sources that cause the generation error\. Then, we provide an interpretation of the transition kernel designs’ impact on algorithm behaviors in[sec\.˜3\.2](https://arxiv.org/html/2606.15327#S3.SS2)\.

### 3\.1Error Analysis and Bias\-variance Trade\-off in DLMs

We can decompose the generation error in[eq\.˜4](https://arxiv.org/html/2606.15327#S2.E4)as \(detailed proof in[sec\.˜B\.1](https://arxiv.org/html/2606.15327#A2.SS1)\):

DKL​\(q​\(x0\)∥pθ​\(x0\)\)=𝔼t​\[DKL​\(q​\(xt\)∥pθ​\(xt\)\)\]\+\\displaystyle D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\)\\\|p\_\{\\theta\}\(x\_\{0\}\)\\big\)=\\mathbb\{E\}\_\{t\}\\Big\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{t\}\)\\\|p\_\{\\theta\}\(x\_\{t\}\)\\big\)\\Big\]\+\(9\)𝔼t,xt∼qt\[DKL\(q\(x0∣xt\)∥pθ\(x0∣xt\)\)\]−𝔼t,x0∼q\[DKL\(q\(xt∣x0\)∥pθ\(xt∣x0\)\)\]\.\\displaystyle\\mathbb\{E\}\_\{t,x\_\{t\}\\sim q\_\{t\}\}\\left\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\\mid x\_\{t\}\)\\\|p\_\{\\theta\}\(x\_\{0\}\\mid x\_\{t\}\)\\big\)\\right\]\-\\mathbb\{E\}\_\{t,x\_\{0\}\\sim q\}\\left\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{t\}\\mid x\_\{0\}\)\\\|p\_\{\\theta\}\(x\_\{t\}\\mid x\_\{0\}\)\\big\)\\right\]\.Effectively, 1\)Approximation Error\.The second term is the approximation error under the true marginals\{qt\}\\\{q\_\{t\}\\\}, which is our training objective in[Eq\.˜3](https://arxiv.org/html/2606.15327#S2.E3)\. 2\)Sampling Error\.The first term is a time\-averaged marginal mismatch between the model’s roll\-out\{pt\}\\\{p\_\{t\}\\\}and the true marginals\{qt\}\\\{q\_\{t\}\\\}\. 3\)The forward path mismatch\.The last term is a correction term measuring the mismatch between the model’s forward conditionalsp​\(xt∣x0\)p\(x\_\{t\}\\mid x\_\{0\}\)and the true forward processq​\(xt∣x0\)q\(x\_\{t\}\\mid x\_\{0\}\)\.

###### Proposition 1\(Error Decomposition\)\. It is common to assume thatp​\(xt∣x0\)≡q​\(xt∣x0\)p\(x\_\{t\}\\mid x\_\{0\}\)\\equiv q\(x\_\{t\}\\mid x\_\{0\}\)for all\(x0,t\)\(x\_\{0\},t\)as the forward paths are manually designed\. Then the generation error becomes,DKL​\(q​\(x0\)∥p​\(x0\)\)=𝔼t​\[DKL​\(qt∥pt\)\]⏟Sampling Error\+𝔼t,xt\[DKL\(q\(x0∣xt\)∥p\(x0∣xt\)\)\]⏟Approximation Error\.D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\)\\\|p\(x\_\{0\}\)\\big\)=\\underbrace\{\\mathbb\{E\}\_\{t\}\\Big\[D\_\{\\mathrm\{KL\}\}\\big\(q\_\{t\}\\\|p\_\{t\}\\big\)\\Big\]\}\_\{\\textbf\{Sampling Error\}\}\+\\underbrace\{\\mathbb\{E\}\_\{t,x\_\{t\}\}\\left\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\\mid x\_\{t\}\)\\\|p\(x\_\{0\}\\mid x\_\{t\}\)\\big\)\\right\]\}\_\{\\textbf\{Approximation Error\}\}\.\(10\)

##### Bias\-Variance Decomposition\.

In general ML, the generalization error, a measurement that gives the prediction ability of an ML algorithm, can be decomposed into 3 meaningful terms111Irreducible risks are usually introduced by the noise, so we omit them in the following analysis\.:

Generalization Error=Bias\+Variance\+Irreducible Risk\\text\{Generalization Error\}=\\text\{Bias\}\+\\text\{Variance\}\+\\text\{Irreducible Risk\}*Bias*is the error between the model’s expected prediction and the ground truth, primarily stemming from limitations in the algorithm design or hypothesis space\.*Variance*quantifies the spread of the estimated model parameters around their expected value and its impact on inference, typically arising from sensitivity to finite data sampling and the stochasticity of the optimization process\.

We can utilize a similar framework to respectively decompose the approximation and sampling error from[Eq\.˜10](https://arxiv.org/html/2606.15327#S3.E10)\(details in Appendix[B\.3](https://arxiv.org/html/2606.15327#A2.SS3)\)\.

###### Proposition 2\(Bias\-Variance Trade\-off\)\. Letp¯​\(x0∣xt\):=𝔼S​\[pθ^S​\(x0∣xt\)\]=𝔼S​\[p^S​\(x0∣xt\)\]\\bar\{p\}\(x\_\{0\}\\mid x\_\{t\}\):=\\mathbb\{E\}\_\{S\}\\big\[p\_\{\\hat\{\\theta\}\_\{S\}\}\(x\_\{0\}\\mid x\_\{t\}\)\\big\]=\\mathbb\{E\}\_\{S\}\\big\[\\hat\{p\}\_\{S\}\(x\_\{0\}\\mid x\_\{t\}\)\\big\]denote the expected predictive distribution, whereθ^​\(S\)\\hat\{\\theta\}\(S\)is the parameter induced by a specific training set and optimization randomnessSS\(short asp^S:=pθ^S\\hat\{p\}\_\{S\}:=p\_\{\\hat\{\\theta\}\_\{S\}\}\)\. The approximation error in[Eq\.˜10](https://arxiv.org/html/2606.15327#S3.E10)can be decomposed as,𝔼t,xt𝔼S\[DKL\(q\(x0∣xt\)∥p^S\(x0∣xt\)\)\]=𝔼t,xt\[DKL\(q\(x0∣xt\)∥p¯\(x0∣xt\)\)\]⏟Asymptotic Bias​ℬasym\+𝔼t,xt​\[𝒱​\(xt\)\]⏟Variance​𝒱,\\mathbb\{E\}\_\{t,x\_\{t\}\}\\mathbb\{E\}\_\{S\}\\Big\[D\_\{\\mathrm\{KL\}\}\(q\(x\_\{0\}\\mid x\_\{t\}\)\\\|\\hat\{p\}\_\{S\}\(x\_\{0\}\\mid x\_\{t\}\)\)\\Big\]=\\underbrace\{\\mathbb\{E\}\_\{t,x\_\{t\}\}\\Big\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\\mid x\_\{t\}\)\\\|\\bar\{p\}\(x\_\{0\}\\mid x\_\{t\}\)\\big\)\\Big\]\}\_\{\\textbf\{Asymptotic Bias \}\{\\mathcal\{B\}\}\_\{\\text\{asym\}\}\}\+\\underbrace\{\\mathbb\{E\}\_\{t,x\_\{t\}\}\\big\[\\mathcal\{V\}\(x\_\{t\}\)\\big\]\}\_\{\\textbf\{Variance \}\{\\mathcal\{V\}\}\},\(11\)wherextx\_\{t\}is constructed over the forward process,q​\(xt\)=∫x0q​\(xt∣x0\)​p​\(x0\)​𝑑x0q\(x\_\{t\}\)=\\int\_\{x\_\{0\}\}q\(x\_\{t\}\\mid x\_\{0\}\)p\(x\_\{0\}\)dx\_\{0\}\. The posterior approximation variance term is𝒱​\(xt\):=𝔼q​\(x0∣xt\)​\[log⁡p¯​\(x0∣xt\)−𝔼S​log⁡\(p^S​\(x0∣xt\)\)\]≥0\{\\mathcal\{V\}\}\(x\_\{t\}\):=\\mathbb\{E\}\_\{q\(x\_\{0\}\\mid x\_\{t\}\)\}\\Big\[\\log\\bar\{p\}\(x\_\{0\}\\mid x\_\{t\}\)\-\\mathbb\{E\}\_\{S\}\\log\(\\hat\{p\}\_\{S\}\(x\_\{0\}\\mid x\_\{t\}\)\)\\Big\]\\geq 0\. And the step\-wise sampling error in[Eq\.˜10](https://arxiv.org/html/2606.15327#S3.E10)can be written as,DKL​\(qt​\(xt\)∥pt​\(xt\)\)=DKL​\(qt​\(xt\)∥p¯t​\(xt\)\)⏟Exposure Bias​ℬt\+𝔼xt​\[log⁡p¯​\(xt\)−𝔼S​\(log⁡p^S​\(xt\)\)\]⏟Sampling Roll\-out Variance​𝒱tD\_\{\\mathrm\{KL\}\}\\big\(q\_\{t\}\(x\_\{t\}\)\\\|p\_\{t\}\(x\_\{t\}\)\\big\)=\\underbrace\{D\_\{\\mathrm\{KL\}\}\\big\(q\_\{t\}\(x\_\{t\}\)\\\|\\bar\{p\}\_\{t\}\(x\_\{t\}\)\\big\)\}\_\{\\textbf\{Exposure Bias \}\{\\mathcal\{B\}\}\_\{t\}\}\+\\underbrace\{\\mathbb\{E\}\_\{x\_\{t\}\}\[\\log\\bar\{p\}\(x\_\{t\}\)\-\\mathbb\{E\}\_\{S\}\(\\log\\hat\{p\}\_\{S\}\(x\_\{t\}\)\)\]\}\_\{\\textbf\{Sampling Roll\-out Variance \}\{\\mathcal\{V\}\}\_\{t\}\}\(12\)

[Sec\.˜B\.6\.1](https://arxiv.org/html/2606.15327#A2.SS6.SSS1)shows that sampling variance in[Eq\.˜12](https://arxiv.org/html/2606.15327#S3.E12)can be controlled by approximation variance\. We therefore unify posterior and roll\-out variances in the following analysis\.

### 3\.2How Transition Kernel Design affects Generation Error?

From[Sec\.˜3\.1](https://arxiv.org/html/2606.15327#S3.SS1), we effectively identified three important sources of error: asymptotic bias in training, exposure bias in sampling, and the variance\. In this section, we will illustrate how different transition designs introduced in[Sec\.˜2](https://arxiv.org/html/2606.15327#S2)affect the model performance of DLMs through these terms\. We first provide an overview of the three sources\.

Asymptotic Bias\.ℬasym\{\\mathcal\{B\}\}\_\{\\text\{asym\}\}reflects the intrinsic difficulty of approximating the true denoising posteriorq​\(x0∣xt\)q\(x\_\{0\}\\mid x\_\{t\}\)with a parameterized predictorpθp\_\{\\theta\}even with infinite training data and optimal convergence\. It comes fromModel MisspecificationandCapacity Bottlenecks, especially when the architecture lacks the expressivity to represent the complexity of the true noise\-corrupted posterior\.Exposure Bias\.ℬexp:=∑t=0Tℬt\\mathcal\{B\}\_\{\\exp\}:=\\sum\_\{t=0\}^\{T\}\{\\mathcal\{B\}\}\_\{t\}\. While sampling, the model generates the sequence iteratively fromt=1t=1tot=0t=0\. The inputxtx\_\{t\}is not drawn from the true marginalqtq\_\{t\}, but from the model’s own previous generative distributionp¯t\\bar\{p\}\_\{t\}\. If the distributionp¯t\\bar\{p\}\_\{t\}deviates even slightly fromqtq\_\{t\}, then subsequent reverse steps are evaluated on shifted inputs\. This phenomenon is known as Exposure Bias, and it quantifies the error coming fromSampling DynamicsandError Accumulation\.Variance\.𝒱\{\\mathcal\{V\}\}measures the instability of the learned predictor under finite training resources, which captures the error caused byFinite Data,Optimization Stochasticity, andResource\-limited Training\. Different transition kernelsq​\(xt∣x0\)q\(x\_\{t\}\\mid x\_\{0\}\)can induce different levels of variance\.

#### 3\.2\.1Asymptotic Bias Analysis through Posterior Geometry\.

To describe the posterior approximation difficulty, we can characterize the*local geometry*of the target posterior\. Letfθ​\(xt\)i∈ℝ\|V\|f\_\{\\theta\}\(x\_\{t\}\)^\{i\}\\in\\mathbb\{R\}^\{\|V\|\}denote the predicted logits at positioni∈\[1:L\]i\\in\[1:L\]such thatpθi\(⋅∣xt\)=softmax\(fθ\(xt\)i\)p\_\{\\theta\}^\{i\}\(\\cdot\\mid x\_\{t\}\)=\\mathrm\{softmax\}\\bigl\(f\_\{\\theta\}\(x\_\{t\}\)^\{i\}\\bigr\)\. Assuming conditional factorization across positions givenxtx\_\{t\}, the asymptotic bias can be decomposed into local losses:

ℬasym=𝔼t,xt​\[∑i=1Lℓ​\(fθ​\(xt\)i;q​\(x0i∣xt\)\)\],ℓ​\(f;q\):=DKL​\(q​\(x0i∣xt\)∥softmax​\(f\)\),\{\\mathcal\{B\}\}\_\{\\text\{asym\}\}=\\mathbb\{E\}\_\{t,x\_\{t\}\}\\left\[\\sum\_\{i=1\}^\{L\}\\ell\\left\(f\_\{\\theta\}\(x\_\{t\}\)^\{i\};q\(x\_\{0\}^\{i\}\\mid x\_\{t\}\)\\right\)\\right\],\\quad\\ell\(f;q\):=D\_\{\\mathrm\{KL\}\}\\left\(q\(x\_\{0\}^\{i\}\\mid x\_\{t\}\)\\,\\\|\\,\\mathrm\{softmax\}\(f\)\\right\),\(13\)Applying the chain rule, the parameter gradient can be written as,

∇θℒ=𝔼t,xt​\[∑i=1n∑k∈V\(pθ​\(x0i=k∣xt\)−q​\(x0i=k∣xt\)\)⋅∇θfθ​\(xt\)ki\]\.\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}=\\mathbb\{E\}\_\{t,x\_\{t\}\}\\left\[\\sum\_\{i=1\}^\{n\}\\sum\_\{k\\in V\}\\left\(p\_\{\\theta\}\(x\_\{0\}^\{i\}=k\\mid x\_\{t\}\)\-q\(x\_\{0\}^\{i\}=k\\mid x\_\{t\}\)\\right\)\\cdot\\nabla\_\{\\theta\}f\_\{\\theta\}\(x\_\{t\}\)^\{i\}\_\{k\}\\right\]\.\(14\)
To formalize the local approximation difficulty, letf⋆​\(q\)f^\{\\star\}\(q\)denote optimal logit vector satisfyingsoftmax​\(f⋆​\(q\)\)=q\\mathrm\{softmax\}\\bigl\(f^\{\\star\}\(q\)\\bigr\)=q\. We apply a local Taylor expansion which yields

∇fℓ​\(f;q\)=Σ​\(q\)​\(f−f∗​\(q\)\)\+o​\(‖f−f∗​\(q\)‖\),with​Σ​\(q\):=∇f2ℓ​\(f;q\)=Diag​\(q\)−q​q⊤\.\\nabla\_\{f\}\\ell\(f;q\)=\\Sigma\(q\)\(f\-f^\{\*\}\(q\)\)\+o\\left\(\\\|f\-f^\{\*\}\(q\)\\\|\\right\),\\text\{with \}\\Sigma\(q\):=\\nabla^\{2\}\_\{f\}\\ell\(f;q\)=\\text\{Diag\}\(q\)\-qq^\{\\top\}\.\(15\)whereΣ​\(q\)\\Sigma\(q\)is the softmax Hessian\. Thus, the posterior approximation difficulty is fully determined by the following properties ofΣ​\(q\)\\Sigma\(q\)\.

##### Posterior approximation difficulty\. For a target posteriorq:=q​\(x0∣xt\)q:=q\(x\_\{0\}\\mid x\_\{t\}\), we quantify its local approximation difficulty via the following metrics ofΣ​\(q\)\\Sigma\(q\):Logits Active Direction:dact​\(q\):=rank​\(Σ​\(q\)\)=\|supp​\(q\)\|−1,\\displaystyle d\_\{\\mathrm\{act\}\}\(q\)=\\text\{rank\}\\bigl\(\\Sigma\(q\)\\bigr\)=\|\\text\{supp\}\(q\)\|\-1,\(16\)Logits Error Sensitivity:ℐ1​\(q\):=tr​\(Σ​\(q\)\)=1−‖q‖22,ℐ2​\(q\):=tr​\(Σ​\(q\)2\)\.\\displaystyle\{\\mathcal\{I\}\}\_\{1\}\(q\)=\\text\{tr\}\\bigl\(\\Sigma\(q\)\\bigr\)=1\-\\\|q\\\|\_\{2\}^\{2\},\\quad\{\\mathcal\{I\}\}\_\{2\}\(q\)=\\text\{tr\}\\bigl\(\\Sigma\(q\)^\{2\}\\bigr\)\.dactd\_\{\\mathrm\{act\}\}measures the number of independent logit directions that must be simultaneously fitted\.ℐ1\\mathcal\{I\}\_\{1\}quantifies how local logit mismatch is converted into error, andℐ2\\mathcal\{I\}\_\{2\}quantifies gradient energy\. Lower values mean easier approximation\. We provide more explanations in[sec\.˜B\.4](https://arxiv.org/html/2606.15327#A2.SS4)\.

Based on these metrics, we can now study how the shape of the posteriorq​\(x0∣xt\)q\(x\_\{0\}\\mid x\_\{t\}\)affects the difficulty of fitting\. By Bayes’ rule,q​\(x0∣xt\)∝pdata​\(x0\)​q​\(xt∣x0\)q\(x\_\{0\}\\mid x\_\{t\}\)\\propto p\_\{\\mathrm\{data\}\}\(x\_\{0\}\)\\,q\(x\_\{t\}\\mid x\_\{0\}\)\. Therefore, the geometry of the posterior is determined jointly bypdata​\(x0\)p\_\{\\mathrm\{data\}\}\(x\_\{0\}\)and the forward kernelq​\(xt∣x0\)q\(x\_\{t\}\\mid x\_\{0\}\)\. Substituting the kernel for masking \([Eq\.˜5](https://arxiv.org/html/2606.15327#S2.E5)\), uniform \([Eq\.˜6](https://arxiv.org/html/2606.15327#S2.E6)\), and semantic diffusion \([Eq\.˜7](https://arxiv.org/html/2606.15327#S2.E7)\) obtains:

###### Proposition 3\(Approximation difficulty across diffusion variants\.\)\. We derive in[Sec\.˜B\.4](https://arxiv.org/html/2606.15327#A2.SS4)that,𝔼​\[dactmask\]≤𝔼​\[dactsem\]​𝔼≤\[dactuni\],𝔼​\[ℐ1mask\]≤𝔼​\[ℐ1sem\]≤𝔼​\[ℐ1uni\]\\mathbb\{E\}\\left\[d\_\{\\mathrm\{act\}\}^\{\\mathrm\{mask\}\}\\right\]\\leq\\mathbb\{E\}\\left\[d\_\{\\mathrm\{act\}\}^\{\\mathrm\{sem\}\}\\right\]\\mathbb\{E\}\\leq\\left\[d\_\{\\mathrm\{act\}\}^\{\\mathrm\{uni\}\}\\right\],\\quad\\mathbb\{E\}\\left\[\{\\mathcal\{I\}\}\_\{1\}^\{\\mathrm\{mask\}\}\\right\]\\leq\\mathbb\{E\}\\left\[\{\\mathcal\{I\}\}\_\{1\}^\{\\mathrm\{sem\}\}\\right\]\\leq\\mathbb\{E\}\\left\[\{\\mathcal\{I\}\}\_\{1\}^\{\\mathrm\{uni\}\}\\right\]\(17\)where superscripts suggest diffusion variants\. This suggests two complementary regimes\. When compute or model capacity is limited, uniform diffusion is harder to optimize because it forces the model to fit dense posteriors at nearly every position \(largedactd\_\{\\text\{act\}\}\), and any local mismatch would lead to large generalization error \(largeℐ1\{\\mathcal\{I\}\}\_\{1\}\)\. Masking diffusion and semantic diffusion are easier to optimize as they have a more restricted posterior space and activate fewer logit directions\. In the data\-sufficient and compute\-abundant regime, however, these additional active directions can become useful supervision: masking diffusion receives little learning signal from visible tokens, whereas uniform diffusion keeps almost every position informative for training\. This partially explains why uniform diffusion scales better but underperforms other variants in most scenarios\.

#### 3\.2\.2The Impact of Generators on Exposure Bias Propagation

##### Exposure Bias\.

ℬexp:=∑t=0Tℬt\\mathcal\{B\}\_\{\\exp\}:=\\sum\_\{t=0\}^\{T\}\{\\mathcal\{B\}\}\_\{t\}measures the mismatch accumulated along the generation trajectory\. Unlike the asymptotic bias in the approximation stage, exposure bias propagates through the reverse generator itself\. We give the following proposition and leave the derivation in[Sec\.˜B\.5](https://arxiv.org/html/2606.15327#A2.SS5)\.

###### Proposition 4\(Exposure Bias Propagation\)\. With mild conditions, the propagation satisfies:ℬt−d​t≤ηt​ℬt\+ρt,\{\\mathcal\{B\}\}\_\{t\-dt\}\\leq\\eta\_\{t\}\\,\{\\mathcal\{B\}\}\_\{t\}\+\\rho\_\{t\},\(18\)whereρt\\rho\_\{t\}upper\-bounds the step\-wise error:ρt≥supxtDKL\(qt\(xt−d​t∣xt\)∥ptθ\(xt−d​t∣xt\)\)\\rho\_\{t\}\\geq\\sup\_\{x\_\{t\}\}D\_\{\\mathrm\{KL\}\}\\bigl\(q\_\{t\}\(x\_\{t\-dt\}\\mid x\_\{t\}\)\\\|p\_\{t\}^\{\\theta\}\(x\_\{t\-dt\}\\mid x\_\{t\}\)\\bigr\)andηtKL∈\[0,1\]\\eta\_\{t\}^\{\\mathrm\{KL\}\}\\in\[0,1\]is the error propagation coefficient for forwardingqtq\_\{t\}toqt−d​tq\_\{t\-dt\}\. In particular,ηt,mask≈1,ηt,uni≤1−λtuni<1,ηt,sem≤1−λtsem<1\.\\eta\_\{t,\\mathrm\{mask\}\}\\approx 1,\\quad\\eta\_\{t,\\mathrm\{uni\}\}\\leq 1\-\\lambda\_\{t\}^\{\\mathrm\{uni\}\}<1,\\quad\\eta\_\{t,\\mathrm\{sem\}\}\\leq 1\-\\lambda\_\{t\}^\{\\mathrm\{sem\}\}<1\.Consequently, forsemantic and uniform diffusion,ρt≤ρ\\rho\_\{t\}\\leq\\rhoandηt≤η<1\\eta\_\{t\}\\leq\\eta<1, thenℬexp=∑tℬt=𝒪​\(T​ρ1−η\)\\mathcal\{B\}\_\{\\exp\}=\\sum\_\{t\}\{\\mathcal\{B\}\}\_\{t\}=\\mathcal\{O\}\\\!\\left\(\\frac\{T\\,\\rho\}\{1\-\\eta\}\\right\); whereas formasking diffusionηt≈1\\eta\_\{t\}\\approx 1andℬexp=𝒪​\(T2​ρ\)\\mathcal\{B\}\_\{\\exp\}=\\mathcal\{O\}\\\!\\left\(T^\{2\}\\rho\\right\)with sampling stepTT\.

Implication\.The coefficientηt\\eta\_\{t\}measures how strongly sampling errors propagate across reverse steps\. For masking diffusion,ηt≈1\\eta\_\{t\}\\approx 1, so early errors will accumulate quadratically w\.r\.tTTand can become “early commitments” that later steps cannot easily repair\. Remasking can mitigate this but cannot completely fix the issue\. By contrast, uniform diffusion and semantic diffusion haveηt<1\\eta\_\{t\}<1, so the reverse dynamics can contract accumulated error and provide an intrinsic repair mechanism, which leads to linear error accumulation w\.r\.tTT\.

#### 3\.2\.3The Impact of Generators on Variance

Then we explain how the transition kernel affects optimization variance\. Letℒxt​\(θ\):=−log⁡pθ​\(x0∣xt\)\{\\mathcal\{L\}\}\_\{x\_\{t\}\}\(\\theta\):=\-\\log p\_\{\\theta\}\(x\_\{0\}\\mid x\_\{t\}\)denote the per\-sample denoising likelihood andℒt​\(θ\):=𝔼xt​\[ℒxt​\(θ\)\]\{\\mathcal\{L\}\}\_\{t\}\(\\theta\):=\\mathbb\{E\}\_\{x\_\{t\}\}\[\{\\mathcal\{L\}\}\_\{x\_\{t\}\}\(\\theta\)\]the per\-step loss\. Letθ⋆\\theta^\{\\star\}minimizeℒ​\(θ\)=𝔼t​\[ℒt​\(θ\)\]\{\\mathcal\{L\}\}\(\\theta\)=\\mathbb\{E\}\_\{t\}\[\{\\mathcal\{L\}\}\_\{t\}\(\\theta\)\]so that𝔼t​\[∇θℒt​\(θ⋆\)\]=0\\mathbb\{E\}\_\{t\}\[\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{t\}\(\\theta^\{\\star\}\)\]=0\. We look into the gradient variance as a surrogate for the total variance \(Details in Appendix[B\.6\.2](https://arxiv.org/html/2606.15327#A2.SS6.SSS2)\)\. Atθ⋆\\theta^\{\\star\}, the law of total variance yields

Vart,xt​\(∇θℒxt​\(θ⋆\)\)=𝔼t​\[Varxt​\(∇θℒxt​\(θ⋆\)\)\]⏟within\-step noise\+Vart​\(∇θℒt​\(θ⋆\)\)⏟between\-step heterogeneity\.\\displaystyle\\mathrm\{Var\}\_\{t,x\_\{t\}\}\\big\(\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{x\_\{t\}\}\(\\theta^\{\\star\}\)\\big\)=\\underbrace\{\\mathbb\{E\}\_\{t\}\\left\[\\mathrm\{Var\}\_\{x\_\{t\}\}\\big\(\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{x\_\{t\}\}\(\\theta^\{\\star\}\)\\big\)\\right\]\}\_\{\\text\{within\-step noise\}\}\+\\underbrace\{\\mathrm\{Var\}\_\{t\}\\left\(\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{t\}\(\\theta^\{\\star\}\)\\right\)\}\_\{\\text\{between\-step heterogeneity\}\}\.\(19\)The first term is the usual gradient noise within a fixed steptt\. The second term measures how the optimization tasks vary across diffusion times\. This term is shaped by the transition kernel because∇θℒt​\(θ⋆\)\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{t\}\(\\theta^\{\\star\}\)depends onqt​\(xt∣x0\)q\_\{t\}\(x\_\{t\}\\mid x\_\{0\}\)through the corrupted inputxtx\_\{t\}and the induced posterior targetq​\(x0∣xt\)q\(x\_\{0\}\\mid x\_\{t\}\)\. Therefore, kernels with more dispersed or more time\-varying corruption patterns can induce larger between\-t heterogeneity \(such as a uniform\)\. We prove this connection in Appendix[B\.6\.2](https://arxiv.org/html/2606.15327#A2.SS6.SSS2), and[sec\.˜5\.1](https://arxiv.org/html/2606.15327#S5.SS1.SSS0.Px1)empirically validates the correlation between gradient variance and the transition kernel dispersion across timett\. This suggests that a dispersed kernel will induce training instability\.

Take\-home Message for[Sec\.˜3](https://arxiv.org/html/2606.15327#S3)\.The analysis in this section reveals a fundamental trade\-off in the transition kernel design for DLMs\.Masking diffusionis easy to train because the absorbing kernel induces sparse denoising posteriors, but it lacks intrinsic repair and can suffer from sampling error propagation\.Uniform diffusionrepairs such errors through global transition, but its dense and dispersed posteriors make training harder and noisier\.Semantic diffusionis therefore a natural middle ground: semantic locality can reduce approximation difficulty, while global transition can preserve repair ability\.

## 4Semantic Diffusion: Towards Bias\-Variance Minimization in DLMs

### 4\.1Revisiting Semantic Diffusion

The analysis in Sec\.[3](https://arxiv.org/html/2606.15327#S3)suggests that semantic diffusion has the potential to mitigate the approximation difficulty of uniform diffusion, and can avoid error accumulation in sampling\. However, prior studies report that semantic diffusion often underperforms alternatives such as uniform or marginal diffusion\(austin2021structured\)\. Although a concurrent work\(DBLP:journals/corr/abs\-2603\-21342\)reports extremely low test perplexity with a semantic kernel, it does not evaluate unconditional generation\. In our experiments, when testing the generation ability of semantic diffusion, we consistently observe a semantic basin problem\.

###### Definition 4\.1\(Semantic Basin\)\. We define a semantic basin as a local semantic region in which the reverse chain repeatedly samples semantically adjacent tokens, producing locally plausible but low\-diversity text\. An example is shown in[Table˜1](https://arxiv.org/html/2606.15327#S4.T1)\.

Table 1:An example of text generation that suffers from semantic basin\.##### Why does semantic basin occur?

Consider a token positionii, and letx^t\\hat\{x\}\_\{t\}denote the current state during sampling\. Ideally, the denoiser should approximate the clean posteriorq​\(x0i=k∣x^t\)q\(x\_\{0\}^\{i\}=k\\mid\\hat\{x\}\_\{t\}\)\. By Bayes’ rule, the posterior logit can be decomposed as

li⋆​\(k;x^t\)=log⁡pdata​\(x0i=k∣x^t−i\)⏟contextual prior\+log⁡qt​\(x^ti∣x0i=k,x^t−i\)⏟local forward likelihood\+const\.l\_\{i\}^\{\\star\}\(k;\\hat\{x\}\_\{t\}\)=\\underbrace\{\\log p\_\{\\mathrm\{data\}\}\(x\_\{0\}^\{i\}=k\\mid\\hat\{x\}\_\{t\}^\{\-i\}\)\}\_\{\\text\{contextual prior\}\}\+\\underbrace\{\\log q\_\{t\}\(\\hat\{x\}\_\{t\}^\{i\}\\mid x\_\{0\}^\{i\}=k,\\hat\{x\}\_\{t\}^\{\-i\}\)\}\_\{\\text\{local forward likelihood\}\}\+\\mathrm\{const\}\.\(20\)During sampling, however,x^t\\hat\{x\}\_\{t\}is generated by the model itself rather than drawn from the true forward marginal\. Therefore, the denoiser is evaluated on rollout states that may already contain accumulated errors\. We write the actual sampling logit as

lθ,i​\(k;x^t\)=li⋆​\(k;x^t\)\+Δroll,i​\(k;x^t\)\+ϵi​\(k\)l\_\{\\theta,i\}\(k;\\hat\{x\}\_\{t\}\)=l\_\{i\}^\{\\star\}\(k;\\hat\{x\}\_\{t\}\)\+\\Delta\_\{\\mathrm\{roll\},i\}\(k;\\hat\{x\}\_\{t\}\)\+\\epsilon\_\{i\}\(k\)\(21\)whereΔroll,i\\Delta\_\{\\mathrm\{roll\},i\}denotes the logit bias induced by the current sampling trajectory\. Letni\(W\)​\(l\)n\_\{i\}^\{\(W\)\}\(l\)be the count of tokenllin a recent windowWW, and letA​\(k,l\)A\(k,l\)denote the contextual contribution of tokenllto the logit of tokenkk\. In[sec\.˜D\.1](https://arxiv.org/html/2606.15327#A4.SS1), we show that the rollout bias can be locally approximated asΔroll,i​\(k;x^t\)≈∑l∈VA​\(k,l\)​ni\(W\)​\(l\)\\Delta\_\{\\mathrm\{roll\},i\}\(k;\\hat\{x\}\_\{t\}\)\\approx\\sum\_\{l\\in V\}A\(k,l\)n\_\{i\}^\{\(W\)\}\(l\)\. IfA​\(k,l\)\>0A\(k,l\)\>0for semantically related tokensk,l∈Ck,l\\in C, then over\-producing tokens from clusterCCincreases the logits of other tokens in the same cluster:

ni\(W\)\(l∈C\)↑⇒Δroll,i\(k\)↑fork∈C⇒pθ\(x0i∈C∣x^t\)↑\.n\_\{i\}^\{\(W\)\}\(l\\in C\)\\uparrow\\Rightarrow\\Delta\_\{\\mathrm\{roll\},i\}\(k\)\\uparrow\\text\{ for \}k\\in C\\Rightarrow p\_\{\\theta\}\(x\_\{0\}^\{i\}\\in C\\mid\\hat\{x\}\_\{t\}\)\\uparrow\.Meanwhile, in a semantic diffusion kernel, the local likelihoodqt​\(j∣k\)q\_\{t\}\(j\\mid k\)is large whenkkis semantically close tojj\. Thus, the likelihood term in[Eq\.˜20](https://arxiv.org/html/2606.15327#S4.E20)also favors tokens within the same semantic cluster\. These two effects reinforce each other and create a positive feedback loop, which drives the reverse chain into a semantic basin\. We provide a formal analysis in[sec\.˜D\.1](https://arxiv.org/html/2606.15327#A4.SS1)\.

### 4\.2From SemDLM to SemDLM\+: Mitigating Semantic Basins\.

The above analysis suggests that semantic basins can be mitigated by weakening this positive feedback\. We use two complementary mechanisms to construct a negative feedback:

\(a\) Global jumping in the forward kernel\.Instead of using a purely local semantic transition, we add a global jumping component:

qt​\(j∣k,c\)=αt​δk​j\+βt​νt​\(j\)\+\(1−αt−βt\)​stsem​\(j∣k,c\)\.q\_\{t\}\(j\\mid k,c\)=\\alpha\_\{t\}\\delta\_\{kj\}\+\{\\color\[rgb\]\{0,0,1\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0,1\}\\beta\_\{t\}\\nu\_\{t\}\(j\)\}\+\(1\-\\alpha\_\{t\}\-\\beta\_\{t\}\)s\_\{t\}^\{\\mathrm\{sem\}\}\(j\\mid k,c\)\.\(22\)The global transitionβt​νt​\(j\)\\beta\_\{t\}\\nu\_\{t\}\(j\)provides a mixing channel, preventing the model from absorbing into the local semantic cluster\. The semantic kernel follows the KNN version as in[Eq\.˜7](https://arxiv.org/html/2606.15327#S2.E7), withstsem​\(j∣i\)=\(kt\)−1​𝕀​\(j∈𝒩kt​\(i\)\)s\_\{t\}^\{\\mathrm\{sem\}\}\(j\\mid i\)=\(k\_\{t\}\)^\{\-1\}\\mathbb\{I\}\\bigl\(j\\in\\mathcal\{N\}\_\{k\_\{t\}\}\(i\)\\bigr\)where the number of semantic neighborhoodsktk\_\{t\}increases over time\. Specifically, we setαt=1−t\\alpha\_\{t\}=1\-t,βt=t2\\beta\_\{t\}=t^\{2\}andkt=1\+\(kmax−1\)​tγk\_\{t\}=1\+\\left\(k\_\{\\max\}\-1\\right\)t^\{\\gamma\}with a hyperparameterγ\\gamma\.

\(b\) Semantic\-frequency Penalty\.We further counteract rollout\-induced positive feedback during sampling\. Ideally, one would subtract the positive feedback term∑l∈V\[A​\(k,l\)\]\+​ni\(W\)​\(l\)\\sum\_\{l\\in V\}\[A\(k,l\)\]\_\{\+\}n\_\{i\}^\{\(W\)\}\(l\)\. However,A​\(k,l\)A\(k,l\)is model\- and context\-dependent and is generally unavailable during sampling\. We therefore use semantic similarity scoresS\+​\(k,l\)≥0S\_\{\+\}\(k,l\)\\geq 0as a practical surrogate, and definemi\(W\)​\(k\)=∑l∈VS\+​\(k,l\)​ni\(W\)​\(l\)m\_\{i\}^\{\(W\)\}\(k\)=\\sum\_\{l\\in V\}S\_\{\+\}\(k,l\)n\_\{i\}^\{\(W\)\}\(l\)as a semantic\-frequency regularizer in sampling\.

Taken together with a frequency penalty, we then apply a correction on the logits:

l~θ,i​\(k\)=lθ,i​\(k\)−λfreq​log⁡\(1\+ni\(W\)​\(k\)\)−λsem​log⁡\(1\+mi\(W\)​\(k\)\)\.\\tilde\{l\}\_\{\\theta,i\}\(k\)=l\_\{\\theta,i\}\(k\)\-\{\\color\[rgb\]\{0,0,1\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0,1\}\\lambda\_\{\\mathrm\{freq\}\}\\log\\left\(1\+n\_\{i\}^\{\(W\)\}\(k\)\\right\)\}\-\{\\color\[rgb\]\{0,0,1\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0,1\}\\lambda\_\{\\mathrm\{sem\}\}\\log\\left\(1\+m\_\{i\}^\{\(W\)\}\(k\)\\right\)\}\.\(23\)The first penalty suppresses exact token over\-production, while the second suppresses over\-production of the semantic neighborhood ofkk\. This correction introduces a negative feedback mechanism that counteracts the rollout\-induced positive feedback which leads semantic basins\. Finally, we show in[Appendix˜B](https://arxiv.org/html/2606.15327#A2)that SemDLM\+ preserves the desirable properties of semantic diffusion: it reduces posterior approximation difficulty while retaining sampling\-side repair ability\.

## 5Experiments

We now evaluate SemDLM\+’s ability in both language modeling and text generation\. We first outline the experimental setup in[Sec\.˜5\.1](https://arxiv.org/html/2606.15327#S5.SS1), followed by the results on language modeling capacity comparison and generation quality comparison in[Sec\.˜5\.2](https://arxiv.org/html/2606.15327#S5.SS2)\. Next, we conduct behavior analysis in[Sec\.˜5\.3](https://arxiv.org/html/2606.15327#S5.SS3)to understand how the designs of SemDLM\+ affect training and sampling dynamics\.

### 5\.1Experiment settings

![Refer to caption](https://arxiv.org/html/2606.15327v1/figures/test_ppl.png)
Figure 1:Testing perplexity with varyingγ\\gammafor model training in LM1B \(15k steps\)\.![Refer to caption](https://arxiv.org/html/2606.15327v1/figures/gen_ppl.png)
Figure 2:Generation perplexity with varyingγ\\gammafor model training in LM1B \(15k steps\)\.
Table 2:Test perplexities \(PPL↓\\downarrow\) comparison for models trained on LM1B\.
Table 3:Generation perplexities \(PPL↓\\downarrow\) of models trained on LM1B and OWT\. The generation length for LM1B is 128 and OWT is 1024\. We believe scaling sampling steps infinitely is problematic thus constrained the sampling steps to \{128, 256, 512\} for LM1B and \{128, 256, 512, 1024\} for OWT\. We conducted 3 tests and results are upper bound, and those with†are reproduced by us and with‡are taken from the respective paper\.##### Training Setup\.

Followingarriola2025block, we train two variants of SemDLM\+ on the One Billion Words dataset \(LM1B\(chelba2014billion\)\) and OpenWebText \(OWT\(Gokaslan2019OpenWeb\)\) at a model scale of 0\.1B\. Models trained on both datasets use the*bert\-base\-uncased*tokenizer\. For both datasets, we set up the maximum training steps to be 200K global steps\. We fixed the context length to be 128 for LM1B and 1024 for OWT\. For the results that we reproduced, we utilize a change\-aware loss, which we found helpful in stabilizing training\. We refer to Appendix[E\.1](https://arxiv.org/html/2606.15327#A5.SS1)for a detailed model setup and algorithm implementation details\.

Generation Setup\.We follow semi\-autoregressive sampler\. Each stride is initialized from the global transition prior, and each denoising step applies Continuous\-time Markov Chain sampling \(CTMC\) sampler with tau\-leaping and predictor\-correctorDBLP:conf/nips/CampbellBBRDD22\. We vary the sampling steps ranging\{128,256,512\}\\\{128,256,512\\\}for LM1B and\{128,256,512,1024\}\\\{128,256,512,1024\\\}for OWT, and set the number of strides as 2\.

Evaluation Setup\.Given the issue of semantic basin, perplexity \(PPL\) and entropy together serve as the evaluation metric for language modelling and generation ability\. Testing PPL is measured on texts produced by the model\-native sampler and Generation PPL is measured by GPT\-2\-Large\.

![Refer to caption](https://arxiv.org/html/2606.15327v1/x1.png)Figure 3:The high correlation of transition kernel dispersion and gradient variance\.![Refer to caption](https://arxiv.org/html/2606.15327v1/figures/genppl_sampling.png)
Figure 4:Generation Perplexity with varyingλfreq\\lambda\_\{\\text\{freq\}\}andλsem\\lambda\_\{\\text\{sem\}\}\. SemDLM\+ trained on OWT\.![Refer to caption](https://arxiv.org/html/2606.15327v1/figures/gen_entropy_sampling.png)
Figure 5:Generation entropy with varyingλfreq\\lambda\_\{\\text\{freq\}\}andλsem\\lambda\_\{\\text\{sem\}\}\. SemDLM\+ trained on OWT\.

### 5\.2Main Results

##### Likelihood Evaluation\.

We first evaluate our model for language modelling based on likelihood metric by reporting perplexities on the test split of LM1B\. We only care about the choice of transition kernel, thus omitting the architecture designs like block diffusion\(arriola2025block\)\.[Table˜2](https://arxiv.org/html/2606.15327#S5.T2)summarizes the language modeling performance\. SemDLM\+ achieves a test perplexity that outperforms other competitors with different noising processes, including uniform, masking, and mixture \(GIDD\)\. This improvement empirically validates our theoretical analysis in[Sec\.˜3](https://arxiv.org/html/2606.15327#S3): by constraining the transition kernel via semantic neighborhoods, SemDLM\+ effectively reduces the asymptotic bias inherent in uniform diffusion while avoiding the high variance in optimization induced by large vocabulary spaces\. Furthermore,[Figure˜1](https://arxiv.org/html/2606.15327#S5.F1)demonstrates that SemDLM\+ converges significantly faster than Uniform Diffusion, confirming the variance reduction property of our noising kernel design\.

##### Generation Quality Evaluation\.

We evaluate generation quality on LM1B and OWT by varying the number of sampling steps in[Table˜3](https://arxiv.org/html/2606.15327#S5.T3)\. Since semantic basin can produce deceptively low PPL with low diversity, we set up a threshold for entropy \(≥4\.8\\geq 4\.8\) and only report results above\. On LM1B and OWT, SemDLM\+ consistently outperforms all baselines across sampling budgets, showing that semantic neighborhoods and global jumping kernel provide an effective transition kernel\.

### 5\.3Behavior Analysis

The Impacts of Generator Dispersion\. In our theoretical results in[Sec\.˜3\.2\.1](https://arxiv.org/html/2606.15327#S3.SS2.SSS1)and[3\.2\.3](https://arxiv.org/html/2606.15327#S3.SS2.SSS3), we show that reducing the dispersion of generatorqtq\_\{t\}can effectively reduce approximation difficulty and stabilize training\. In SemDLM\+, this is reflected in the choice ofγ\\gamma\. Largeγ\\gammainduces a more complex optimization landscape and lower dispersion\. Thus, we systematically analyze the impact ofγ\\gammain[Fig\.˜1](https://arxiv.org/html/2606.15327#S5.F1)and[2](https://arxiv.org/html/2606.15327#S5.F2)and validate this hypothesis\. With a largerγ\\gamma, the training convergence significantly improves\. However, withγ\\gammaoverly enlarged, the generation quality degrades, as the transition is constrained in the semantic basin without any escape, thus becoming harmful for the sampling\.

##### Sampling Error Accumulation\.

![Refer to caption](https://arxiv.org/html/2606.15327v1/x2.png)Figure 6:Gen\-PPL on LM1B as sampling steps increase\.The step\-scaling results in[Table˜3](https://arxiv.org/html/2606.15327#S5.T3)and[Fig\.˜6](https://arxiv.org/html/2606.15327#S5.F6)show how kernels handle accumulated sampling errors\. SemDLM\+ benefits more clearly from additional reverse steps than MDLM variants: its PPL drops from 74\.52 to 36\.60 on LM1B and from 64\.39 to 34\.37 in OWT\. This suggests that SemDLM\+ successfully preserves the repairing mechanism in uniform DLMs\.

##### The Impacts of Semantic Frequency Penalty\.

[Fig\.˜4](https://arxiv.org/html/2606.15327#S5.F4)and[5](https://arxiv.org/html/2606.15327#S5.F5)further illustrate the role of the semantic frequency penalties\. We can observe an extremely low generation ppl and entropy whenλfreq=0\\lambda\_\{\\mathrm\{freq\}\}=0, which coincides with the results inDBLP:journals/corr/abs\-2603\-21342but this actually indicates a collapsed low\-diversity regime rather than genuinely strong generation\. Increasingλfreq\\lambda\_\{\\mathrm\{freq\}\}raises entropy and prevents exact\-token repetition, whileλsem\\lambda\_\{\\mathrm\{sem\}\}further suppresses repetition within a semantic neighborhood\. However, overly strong penalties can over\-flatten the clean posterior and worsen PPL\. This shows that these penalties trade off likelihood and diversity by counteracting the semantic basin\.

## 6Discussions and Limitations

In this work, we establish an error analysis framework for understanding how transition kernel design affects DLMs and propose SemDLM\+, which balances training difficulty and sampling\-side repair\. We hereby list important limitations for the benefit of the community\. 1\) Our semantic graph is token\-level, so it cannot fully capture context\-dependent meanings\. 2\) The semantic penalty is still an approximation to the ideal transition, which makes generation sensitive to the hyperparameter design\. 3\) Semantic basin behavior shows that generation PPL alone can be misleading\. We believe broader diversity and quality evaluations are needed for the DLM community for a comprehensive analysis\.

## References

## Appendix ARelated Work

##### Discrete Diffusion Models\.

Diffusion language models \(DLMs\)\(austin2021structured;hoogeboom2021argmax\)have emerged as a promising alternative to autoregressive \(AR\) language models\. Unlike AR LLMs\(Dubey2024TheL3;Yang2024Qwen25TR;DeepSeekAI2024DeepSeekV3TR\)that decode tokens strictly left\-to\-right, DLMs generate text by iterative refinement, progressively denoising from completely random noise sampled from reference distribution \(e\.g\., all \[MASK\] tokens or samples from uniform distribution\) into a clean data points\(ho2020ddpm;li2022diffusion\)\. DLMs have achieved competitive language modeling performance\(nie2025llada;wang2026trainabilitymaskeddiffusionlanguage\), attracting interest for their potential to reduce decoding latency via parallel updates without sacrificing quality\. Representative formulations include D3PM\(austin2021structured\)and its language\-model instantiations, masking\-based diffusion \(MDLM\)\(sahoo2024simple;DBLP:conf/nips/ShiHWDT24;ou2025your\), as well as alternative discrete\-time or continuous\-time constructions such as SEDD\(DBLP:conf/icml/LouME24\), tau\-leaping for discrete\-state continuous\-time diffusion\(DBLP:conf/nips/CampbellBBRDD22\), and discrete flow matching\(DBLP:conf/icml/CampbellYBRJ24\)\. These models vary in their design of the*transition kernel*\. Masking DLMs are the predominant archetype, due to stable training and convenient likelihood objectives\(sahoo2024simple;DBLP:conf/nips/ShiHWDT24;ou2025your\)\. Beyond masking, prior work considers uniform diffusion\(austin2021structured;DBLP:conf/iclr/SchiffSPWBDARPK25\), marginal diffusion\(qin2025defog\), and semantic\-neighborhood diffusion\(austin2021structured\)to improve mixing and address exposure to hard corruptions\. Early studies argue that masking\-style diffusion can alleviate slow convergence, training instability, and weaker generalization observed in some discrete diffusion setups\(austin2021structured;hoogeboom2021argmax;wang202610openchallengessteering\)\. More recent scaling analyses, however, report that uniform diffusion can enjoy better scaling behavior given sufficient data and training budget\(vonruette2025scalingbehaviordiscretediffusion;DBLP:conf/icml/RutteFDOS025;sahoo2026scalingmaskeddiffusionlanguage\)\. To bridge these regimes,*GIDD*\(DBLP:conf/icml/RutteFDOS025\)interpolates between masking and uniform noising, offering a controlled mixture family for both training and inference\.

##### Large Diffusion Language Models\.

At scale,nie2025lladaintroduces LLaDA, the first 8B scale diffusion Large Language Models \(DLLM\) with masked diffusion trained*from scratch*with a standard pretraining \+ supervised fine\-tuning \(SFT\) pipeline\. LLaDA demonstrates strong scalability and competitive in\-context learning and instruction\-following ability at the 8B scale\. Beyond purely diffusion training from scratch, hybrid paradigms aim to combine AR coherence with diffusion parallelism:dream2025presents Dream 7B andcheng2025sdarsynergisticdiffusionautoregressionparadigmpropose SDAR\. They both convert a pretrained AR model into a block\-wise diffusion decoder\. Finally, Tencent\(liu2025wedlmreconcilingdiffusionlanguage\)target deployment efficiency by reconciling diffusion decoding with standard KV caching: they propose WeDLM, which realizes diffusion\-style parallel decoding under causal attention via reordering and streaming commitment, enabling substantial speedups over optimized AR serving \(e\.g\., vLLM\)\. Overall, these works indicate a rapidly maturing ecosystem of large diffusion LLMs\.

##### Comparing Diffusion and Autoregressive Models\.

A growing literature compares diffusion and autoregressive \(AR\) generation across empirical performance, architectural design, and learning/inference dynamics\. Empirically,DBLP:journals/corr/abs\-2411\-07873compares AR and diffusion models on downstream tasks and characterizes practical trade\-offs\. From an architectural viewpoint,DBLP:journals/corr/abs\-2505\-15045analyzes how causal \(AR\) versus bidirectional \(diffusion\-style\) attention affects modeling and decoding efficiency\. On the learning side,kim2025trainstudies diffusion trajectory construction \(token update ordering\) and shows that it can induce a harder optimization problem than AR’s fixed causal ordering;DBLP:conf/iclr/GongA0YZLAZB00K25demonstrates that diffusion and AR objectives can be cast in closely related forms, enabling transfer and distillation across paradigms\. Recent scaling evidence further suggests that diffusion models may generalize better as data and compute increase\(DBLP:journals/corr/abs\-2511\-03276\)\. Despite these advances, existing studies do not offer a unified framework that links training dynamics, generator transition dynamics\. We serve as the first to provide a unified framework that originated from generalization error analysis, and identify the bias\-variance trade\-offs in DLLMs

##### Kernel Design in Diffusion Language Models\.

Beyond model architectures, another line of work in discrete diffusion improves decoding quality by manipulating the generation path\.qin2025defogpropose target\-guided diffusion to steer generation toward desired outcomes during sampling\.wang2025learningorderlearns the token update order with reinforcement learning, adapting the path to the input and improving decoding efficiency\.wang2025remaskingintroduces remasking strategies that re\-corrupt tokens during inference to correct early mistakes and reduce degeneration\. From a geometric perspective,jiang2025bureswassersteinredesigns the diffusion path via optimal\-transport alignment to better match intermediate states with the target distribution\. More recently,peng2025path;liu2025thinkpropose trainable planners that predict which positions to decode or commit at each step, enabling selective refinement and faster convergence\. Collectively, these methods highlight the importance of the transition path as a controllable degree of freedom in diffusion\-based sequence generation\.

##### Theoretical Guarantees of DLMs\.

While diffusion language models \(DLMs\) are often motivated by the prospect of parallel decoding, recent theory makes precise when such acceleration is*provably*attainable and when it is fundamentally limited\.Feng2025TheoreticalBAprovide a rigorous analysis for masked diffusion models \(MDMs\) and show that the efficiency–accuracy trade\-off is highly metric\-dependent\. In particular, under mild regularity assumptions, they prove that MDMs can reach near\-optimal*perplexity*with a number of refinement steps that does not grow with the sequence length, supporting the intuition that parallel refinement can match AR\-level likelihood quality with bounded\-step sampling\.kim2025trainandtang2026gdsdreinforcementlearningguideddemonstrate that diffusion LLMs are difficult to train, with training instability, severe training\-inference mismatch, and a complex optimization landscape, but are more powerful in sampling once trained\. Later,li2025convergence\_dlmdevelop an information\-theoretic convergence theory for DLM sampling\. Their results offers a concrete theoretical lens: DLMs can admit provable acceleration for likelihood\-oriented metrics and weakly dependent sequences, yet face intrinsic limitations when the goal is exact\-sequence correctness or when token dependencies are strong\.

## Appendix BOmitted Proofs

### B\.1Error decomposition

###### Proposition 5\(Error Decomposition\)\. The exact KL divergence in the generation process can be decomposed into:DKL​\(q​\(x0\)∥pθ​\(x0\)\)\\displaystyle D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\)\\\|p\_\{\\theta\}\(x\_\{0\}\)\\big\)\(24\)=\\displaystyle=𝔼t​\[DKL​\(q​\(xt\)∥pθ​\(xt\)\)\]⏟Sampling Error\+𝔼t,xt∼qt\[DKL\(q\(x0∣xt\)∥pθ\(x0∣xt\)\)\]⏟Approximation Error\\displaystyle\\underbrace\{\\mathbb\{E\}\_\{t\}\\Big\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{t\}\)\\\|p\_\{\\theta\}\(x\_\{t\}\)\\big\)\\Big\]\}\_\{\\text\{Sampling Error\}\}\+\\underbrace\{\\mathbb\{E\}\_\{t,x\_\{t\}\\sim q\_\{t\}\}\\left\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\\mid x\_\{t\}\)\\\|p\_\{\\theta\}\(x\_\{0\}\\mid x\_\{t\}\)\\big\)\\right\]\}\_\{\\text\{Approximation Error\}\}−𝔼t,x0∼q\[DKL\(q\(xt∣x0\)∥pθ\(xt∣x0\)\)\]⏟Forward Process Mismatch\.\\displaystyle\-\\underbrace\{\\mathbb\{E\}\_\{t,x\_\{0\}\\sim q\}\\left\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{t\}\\mid x\_\{0\}\)\\\|p\_\{\\theta\}\(x\_\{t\}\\mid x\_\{0\}\)\\big\)\\right\]\}\_\{\\text\{Forward Process Mismatch\}\}\.1\)Approximation Error\.The middle term is the*approximation \(denoising\) error*under the true marginals\{qt\}\\\{q\_\{t\}\\\}, which has almost the same form as our training objective in[Eq\.˜3](https://arxiv.org/html/2606.15327#S2.E3)\. 2\)Sampling Error\.The first term is a time\-averaged*marginal mismatch*between the model’s rolled\-out marginals\{pt\}\\\{p\_\{t\}\\\}and the true marginals\{qt\}\\\{q\_\{t\}\\\}\. 3\)The forward path mismatchThe last term is a time\-averaged correction term depending on whether the model’s forward conditionalsp​\(xt∣x0\)p\(x\_\{t\}\\mid x\_\{0\}\)match the true noising processq​\(xt∣x0\)q\(x\_\{t\}\\mid x\_\{0\}\)\.

Proof\.For simplicity, in the following we omit the parameterθ\\thetabut just denotepθp\_\{\\theta\}aspp\. We first consider the KL divergence on the joint distribution of\(x0,xt\)\(x\_\{0\},x\_\{t\}\), where

DKL​\(q​\(x0,xt\)∥p​\(x0,xt\)\):=∫x0,xt𝑑x0​𝑑xt​q​\(x0,xt\)​log⁡q​\(x0,xt\)p​\(x0,xt\)\.D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\},x\_\{t\}\)\\\|p\(x\_\{0\},x\_\{t\}\)\\big\):=\\int\_\{x\_\{0\},x\_\{t\}\}dx\_\{0\}dx\_\{t\}q\(x\_\{0\},x\_\{t\}\)\\log\\frac\{q\(x\_\{0\},x\_\{t\}\)\}\{p\(x\_\{0\},x\_\{t\}\)\}\.\(25\)wherextx\_\{t\}andx0x\_\{0\}are sampled throughxt∼q​\(xt∣x0\)x\_\{t\}\\sim q\(x\_\{t\}\\mid x\_\{0\}\),x0∼q​\(x0\)x\_\{0\}\\sim q\(x\_\{0\}\)\. We useq​\(x0,xt\)=qt​\(xt\)​q​\(x0∣xt\)q\(x\_\{0\},x\_\{t\}\)=q\_\{t\}\(x\_\{t\}\)q\(x\_\{0\}\\mid x\_\{t\}\), andp​\(x0,xt\)=pt​\(xt\)​p​\(x0∣xt\)p\(x\_\{0\},x\_\{t\}\)=p\_\{t\}\(x\_\{t\}\)p\(x\_\{0\}\\mid x\_\{t\}\)to write,

log⁡q​\(x0,xt\)p​\(x0,xt\)=log⁡qt​\(xt\)​q​\(x0∣xt\)pt​\(xt\)​p​\(x0∣xt\)=log⁡qt​\(xt\)pt​\(xt\)\+log⁡q​\(x0∣xt\)p​\(x0∣xt\)\.\\log\\frac\{q\(x\_\{0\},x\_\{t\}\)\}\{p\(x\_\{0\},x\_\{t\}\)\}=\\log\\frac\{q\_\{t\}\(x\_\{t\}\)q\(x\_\{0\}\\mid x\_\{t\}\)\}\{p\_\{t\}\(x\_\{t\}\)p\(x\_\{0\}\\mid x\_\{t\}\)\}=\\log\\frac\{q\_\{t\}\(x\_\{t\}\)\}\{p\_\{t\}\(x\_\{t\}\)\}\+\\log\\frac\{q\(x\_\{0\}\\mid x\_\{t\}\)\}\{p\(x\_\{0\}\\mid x\_\{t\}\)\}\.\(26\)Substitute[Eq\.˜26](https://arxiv.org/html/2606.15327#A2.E26)into[Eq\.˜25](https://arxiv.org/html/2606.15327#A2.E25), we obtain,

DKL​\(q​\(x0,xt\)∥p​\(x0,xt\)\)\\displaystyle D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\},x\_\{t\}\)\\\|p\(x\_\{0\},x\_\{t\}\)\\big\)\(27\)=∫x0,xtq​\(x0,xt\)​log⁡qt​\(xt\)pt​\(xt\)​d​x0​d​xt⏟Term \(1\)\+∫x0,xtq​\(x0,xt\)​log⁡q​\(x0∣xt\)p​\(x0∣xt\)​d​x0​d​xt⏟Term \(2\)\.\\displaystyle=\\underbrace\{\\int\_\{x\_\{0\},x\_\{t\}\}q\(x\_\{0\},x\_\{t\}\)\\log\\frac\{q\_\{t\}\(x\_\{t\}\)\}\{p\_\{t\}\(x\_\{t\}\)\}dx\_\{0\}dx\_\{t\}\}\_\{\\text\{Term \(1\)\}\}\+\\underbrace\{\\int\_\{x\_\{0\},x\_\{t\}\}q\(x\_\{0\},x\_\{t\}\)\\log\\frac\{q\(x\_\{0\}\\mid x\_\{t\}\)\}\{p\(x\_\{0\}\\mid x\_\{t\}\)\}dx\_\{0\}dx\_\{t\}\}\_\{\\text\{Term \(2\)\}\}\.Given thatlog⁡qt​\(xt\)pt​\(xt\)\\log\\frac\{q\_\{t\}\(x\_\{t\}\)\}\{p\_\{t\}\(x\_\{t\}\)\}depends only onxtx\_\{t\}but not onx0x\_\{0\}, term \(1\) becomes

Term \(1\)=∫xt\(∫x0q​\(x0,xt\)​𝑑x0\)​log⁡qt​\(xt\)pt​\(xt\)​d​xt\\displaystyle=\\int\_\{x\_\{t\}\}\\left\(\\int\_\{x\_\{0\}\}q\(x\_\{0\},x\_\{t\}\)dx\_\{0\}\\right\)\\log\\frac\{q\_\{t\}\(x\_\{t\}\)\}\{p\_\{t\}\(x\_\{t\}\)\}dx\_\{t\}\(28\)=∫xtqt​\(xt\)​log⁡qt​\(xt\)pt​\(xt\)​d​xt=DKL​\(qt​\(xt\)∥pt​\(xt\)\)\.\\displaystyle=\\int\_\{x\_\{t\}\}q\_\{t\}\(x\_\{t\}\)\\log\\frac\{q\_\{t\}\(x\_\{t\}\)\}\{p\_\{t\}\(x\_\{t\}\)\}dx\_\{t\}=D\_\{\\mathrm\{KL\}\}\\big\(q\_\{t\}\(x\_\{t\}\)\\\|p\_\{t\}\(x\_\{t\}\)\\big\)\.Similarly, usingq​\(x0,xt\)=qt​\(xt\)​q​\(x0∣xt\)q\(x\_\{0\},x\_\{t\}\)=q\_\{t\}\(x\_\{t\}\)q\(x\_\{0\}\\mid x\_\{t\}\), Term \(2\) becomes,

Term \(2\)=∫xt,x0qt​\(xt\)​q​\(x0∣xt\)​log⁡q​\(x0∣xt\)p​\(x0∣xt\)​d​xt​d​x0\\displaystyle=\\int\_\{x\_\{t\},x\_\{0\}\}q\_\{t\}\(x\_\{t\}\)q\(x\_\{0\}\\mid x\_\{t\}\)\\log\\frac\{q\(x\_\{0\}\\mid x\_\{t\}\)\}\{p\(x\_\{0\}\\mid x\_\{t\}\)\}dx\_\{t\}dx\_\{0\}\(29\)=∫xt𝑑xt​qt​\(xt\)​\[∫x0q​\(x0∣xt\)​log⁡q​\(x0∣xt\)p​\(x0∣xt\)​d​x0\]\\displaystyle=\\int\_\{x\_\{t\}\}dx\_\{t\}q\_\{t\}\(x\_\{t\}\)\\left\[\\int\_\{x\_\{0\}\}q\(x\_\{0\}\\mid x\_\{t\}\)\\log\\frac\{q\(x\_\{0\}\\mid x\_\{t\}\)\}\{p\(x\_\{0\}\\mid x\_\{t\}\)\}dx\_\{0\}\\right\]=∫xtqt\(xt\)DKL\(q\(x0∣xt\)∥p\(x0∣xt\)\)dxt\\displaystyle=\\int\_\{x\_\{t\}\}q\_\{t\}\(x\_\{t\}\)D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\\mid x\_\{t\}\)\\\|p\(x\_\{0\}\\mid x\_\{t\}\)\\big\)dx\_\{t\}=𝔼xt∼qt\[DKL\(q\(x0∣xt\)∥p\(x0∣xt\)\)\]\.\\displaystyle=\\mathbb\{E\}\_\{x\_\{t\}\\sim q\_\{t\}\}\\Big\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\\mid x\_\{t\}\)\\\|p\(x\_\{0\}\\mid x\_\{t\}\)\\big\)\\Big\]\.Plugging[Eq\.˜28](https://arxiv.org/html/2606.15327#A2.E28)and[Eq\.˜29](https://arxiv.org/html/2606.15327#A2.E29)into[Eq\.˜27](https://arxiv.org/html/2606.15327#A2.E27)yields

DKL\(q\(x0,xt\)∥p\(x0,xt\)\)=DKL\(qt∥pt\)\+𝔼xt∼qt\[DKL\(q\(x0∣xt\)∥p\(x0∣xt\)\)\]\.D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\},x\_\{t\}\)\\\|p\(x\_\{0\},x\_\{t\}\)\\big\)=D\_\{\\mathrm\{KL\}\}\\big\(q\_\{t\}\\\|p\_\{t\}\\big\)\+\\mathbb\{E\}\_\{x\_\{t\}\\sim q\_\{t\}\}\\Big\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\\mid x\_\{t\}\)\\\|p\(x\_\{0\}\\mid x\_\{t\}\)\\big\)\\Big\]\.\(30\)Now considerq​\(x0,xt\)=q​\(x0\)​q​\(xt∣x0\)q\(x\_\{0\},x\_\{t\}\)=q\(x\_\{0\}\)q\(x\_\{t\}\\mid x\_\{0\}\)andp​\(x0,xt\)=p​\(x0\)​p​\(xt∣x0\)p\(x\_\{0\},x\_\{t\}\)=p\(x\_\{0\}\)p\(x\_\{t\}\\mid x\_\{0\}\), we get

log⁡q​\(x0,xt\)p​\(x0,xt\)=log⁡q​\(x0\)​q​\(xt∣x0\)p​\(x0\)​p​\(xt∣x0\)=log⁡q​\(x0\)p​\(x0\)\+log⁡q​\(xt∣x0\)p​\(xt∣x0\)\.\\log\\frac\{q\(x\_\{0\},x\_\{t\}\)\}\{p\(x\_\{0\},x\_\{t\}\)\}=\\log\\frac\{q\(x\_\{0\}\)q\(x\_\{t\}\\mid x\_\{0\}\)\}\{p\(x\_\{0\}\)p\(x\_\{t\}\\mid x\_\{0\}\)\}=\\log\\frac\{q\(x\_\{0\}\)\}\{p\(x\_\{0\}\)\}\+\\log\\frac\{q\(x\_\{t\}\\mid x\_\{0\}\)\}\{p\(x\_\{t\}\\mid x\_\{0\}\)\}\.\(31\)Substitute[Eq\.˜31](https://arxiv.org/html/2606.15327#A2.E31)into[Eq\.˜25](https://arxiv.org/html/2606.15327#A2.E25):

DKL​\(q​\(x0,xt\)∥p​\(x0,xt\)\)\\displaystyle D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\},x\_\{t\}\)\\\|p\(x\_\{0\},x\_\{t\}\)\\big\)=∫x0,xt𝑑x0​𝑑xt​q​\(x0,xt\)​\[log⁡q​\(x0\)p​\(x0\)\+log⁡q​\(xt∣x0\)p​\(xt∣x0\)\]\\displaystyle=\\int\_\{x\_\{0\},x\_\{t\}\}dx\_\{0\}dx\_\{t\}q\(x\_\{0\},x\_\{t\}\)\\left\[\\log\\frac\{q\(x\_\{0\}\)\}\{p\(x\_\{0\}\)\}\+\\log\\frac\{q\(x\_\{t\}\\mid x\_\{0\}\)\}\{p\(x\_\{t\}\\mid x\_\{0\}\)\}\\right\]=∫x0,xt𝑑x0​𝑑xt​q​\(x0,xt\)​log⁡q​\(x0\)p​\(x0\)⏟Term \(3\)\+∫x0,xt𝑑x0​𝑑xt​q​\(x0,xt\)​log⁡q​\(xt∣x0\)p​\(xt∣x0\)⏟Term \(4\)\.\\displaystyle=\\underbrace\{\\int\_\{x\_\{0\},x\_\{t\}\}dx\_\{0\}dx\_\{t\}q\(x\_\{0\},x\_\{t\}\)\\log\\frac\{q\(x\_\{0\}\)\}\{p\(x\_\{0\}\)\}\}\_\{\\text\{Term \(3\)\}\}\+\\underbrace\{\\int\_\{x\_\{0\},x\_\{t\}\}dx\_\{0\}dx\_\{t\}q\(x\_\{0\},x\_\{t\}\)\\log\\frac\{q\(x\_\{t\}\\mid x\_\{0\}\)\}\{p\(x\_\{t\}\\mid x\_\{0\}\)\}\}\_\{\\text\{Term \(4\)\}\}\.Then, using the same marginalization trick as[Eq\.˜28](https://arxiv.org/html/2606.15327#A2.E28),

Term \(3\)=∫x0\(∫xtq​\(x0,xt\)​𝑑xt\)​log⁡q​\(x0\)p​\(x0\)​d​x0\\displaystyle=\\int\_\{x\_\{0\}\}\\left\(\\int\_\{x\_\{t\}\}q\(x\_\{0\},x\_\{t\}\)dx\_\{t\}\\right\)\\log\\frac\{q\(x\_\{0\}\)\}\{p\(x\_\{0\}\)\}dx\_\{0\}\(32\)=∫x0q​\(x0\)​log⁡q​\(x0\)p​\(x0\)​d​x0=DKL​\(q​\(x0\)∥p​\(x0\)\)\.\\displaystyle=\\int\_\{x\_\{0\}\}q\(x\_\{0\}\)\\log\\frac\{q\(x\_\{0\}\)\}\{p\(x\_\{0\}\)\}dx\_\{0\}=D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\)\\\|p\(x\_\{0\}\)\\big\)\.And similarly for Term \(4\),

Term \(4\)=∫x0,xt𝑑xt​𝑑x0q​\(x0\)​q​\(xt∣x0\)​log⁡q​\(xt∣x0\)p​\(xt∣x0\)\\displaystyle=\\int\_\{x\_\{0\},x\_\{t\}\}dx\_\{t\}dx\_\{0\}\\quad q\(x\_\{0\}\)q\(x\_\{t\}\\mid x\_\{0\}\)\\log\\frac\{q\(x\_\{t\}\\mid x\_\{0\}\)\}\{p\(x\_\{t\}\\mid x\_\{0\}\)\}\(33\)=∫𝑑x0​q​\(x0\)​\[∫xtq​\(xt∣x0\)​log⁡q​\(xt∣x0\)p​\(xt∣x0\)​d​xt\]\\displaystyle=\\int\{dx\_\{0\}\}q\(x\_\{0\}\)\\left\[\\int\_\{x\_\{t\}\}q\(x\_\{t\}\\mid x\_\{0\}\)\\log\\frac\{q\(x\_\{t\}\\mid x\_\{0\}\)\}\{p\(x\_\{t\}\\mid x\_\{0\}\)\}dx\_\{t\}\\right\]=𝔼x0∼q\[DKL\(q\(xt∣x0\)∥p\(xt∣x0\)\)\]\.\\displaystyle=\\mathbb\{E\}\_\{x\_\{0\}\\sim q\}\\Big\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{t\}\\mid x\_\{0\}\)\\\|p\(x\_\{t\}\\mid x\_\{0\}\)\\big\)\\Big\]\.Plugging[Eq\.˜32](https://arxiv.org/html/2606.15327#A2.E32)and[Eq\.˜33](https://arxiv.org/html/2606.15327#A2.E33)into[Sec\.˜B\.1](https://arxiv.org/html/2606.15327#A2.Ex4)yields

DKL\(q\(x0,xt\)∥p\(x0,xt\)\)=DKL\(q\(x0\)∥p\(x0\)\)\+𝔼x0∼q\[DKL\(q\(xt∣x0\)∥p\(xt∣x0\)\)\]\.D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\},x\_\{t\}\)\\\|p\(x\_\{0\},x\_\{t\}\)\\big\)=D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\)\\\|p\(x\_\{0\}\)\\big\)\+\\mathbb\{E\}\_\{x\_\{0\}\\sim q\}\\Big\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{t\}\\mid x\_\{0\}\)\\\|p\(x\_\{t\}\\mid x\_\{0\}\)\\big\)\\Big\]\.\(34\)Both[Eq\.˜30](https://arxiv.org/html/2606.15327#A2.E30)and[Eq\.˜34](https://arxiv.org/html/2606.15327#A2.E34)equal the same quantityDKL​\(q​\(x0,xt\)∥p​\(x0,xt\)\)D\_\{\\mathrm\{KL\}\}\(q\(x\_\{0\},x\_\{t\}\)\\\|p\(x\_\{0\},x\_\{t\}\)\), then

DKL\(qt\(xt\)∥pt\(xt\)\)\+𝔼xt∼q​\(xt\)\[DKL\(q\(x0∣xt\)∥p\(x0∣xt\)\)\]\\displaystyle D\_\{\\mathrm\{KL\}\}\\big\(q\_\{t\}\(x\_\{t\}\)\\\|p\_\{t\}\(x\_\{t\}\)\\big\)\+\\mathbb\{E\}\_\{x\_\{t\}\\sim q\(x\_\{t\}\)\}\\Big\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\\mid x\_\{t\}\)\\\|p\(x\_\{0\}\\mid x\_\{t\}\)\\big\)\\Big\]=\\displaystyle=DKL\(q\(x0\)∥p\(x0\)\)\+𝔼x0∼q​\(x0\)\[DKL\(q\(xt∣x0\)∥p\(xt∣x0\)\)\]\.\\displaystyle D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\)\\\|p\(x\_\{0\}\)\\big\)\+\\mathbb\{E\}\_\{x\_\{0\}\\sim q\(x\_\{0\}\)\}\\Big\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{t\}\\mid x\_\{0\}\)\\\|p\(x\_\{t\}\\mid x\_\{0\}\)\\big\)\\Big\]\.Hence

DKL​\(q​\(x0\)∥p​\(x0\)\)\\displaystyle D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\)\\\|p\(x\_\{0\}\)\\big\)\(35\)=\\displaystyle=DKL\(qt∥pt\)\+𝔼xt∼qt\[DKL\(q\(x0∣xt\)∥p\(x0∣xt\)\)\]−𝔼x0∼q\[DKL\(q\(xt∣x0\)∥p\(xt∣x0\)\)\]\.\\displaystyle D\_\{\\mathrm\{KL\}\}\\big\(q\_\{t\}\\\|p\_\{t\}\\big\)\+\\mathbb\{E\}\_\{x\_\{t\}\\sim q\_\{t\}\}\\Big\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\\mid x\_\{t\}\)\\\|p\(x\_\{0\}\\mid x\_\{t\}\)\\big\)\\Big\]\-\\mathbb\{E\}\_\{x\_\{0\}\\sim q\}\\Big\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{t\}\\mid x\_\{0\}\)\\\|p\(x\_\{t\}\\mid x\_\{0\}\)\\big\)\\Big\]\.Taking expectation of[Eq\.˜35](https://arxiv.org/html/2606.15327#A2.E35)with respect tottyields,

DKL​\(q​\(x0\)∥p​\(x0\)\)=\\displaystyle D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\)\\\|p\(x\_\{0\}\)\\big\)=𝔼t\[DKL\(qt∥pt\)\]\+𝔼t,xt∼qt\[DKL\(q\(x0∣xt\)∥p\(x0∣xt\)\)\]\\displaystyle\\mathbb\{E\}\_\{t\}\\Big\[D\_\{\\mathrm\{KL\}\}\\big\(q\_\{t\}\\\|p\_\{t\}\\big\)\\Big\]\+\\mathbb\{E\}\_\{t,x\_\{t\}\\sim q\_\{t\}\}\\Big\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\\mid x\_\{t\}\)\\\|p\(x\_\{0\}\\mid x\_\{t\}\)\\big\)\\Big\]\(36\)−𝔼t,x0∼q\[DKL\(q\(xt∣x0\)∥p\(xt∣x0\)\)\]\.\\displaystyle\-\\mathbb\{E\}\_\{t,x\_\{0\}\\sim q\}\\Big\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{t\}\\mid x\_\{0\}\)\\\|p\(x\_\{t\}\\mid x\_\{0\}\)\\big\)\\Big\]\.

which is exactly the three\-term error decomposition that we are interested\.

### B\.2When the forward process matches

Assume that for all\(x0,t\)\(x\_\{0\},t\),p​\(xt∣x0\)≡q​\(xt∣x0\)\.p\(x\_\{t\}\\mid x\_\{0\}\)\\equiv q\(x\_\{t\}\\mid x\_\{0\}\)\.Then for every\(x0,t\)\(x\_\{0\},t\),DKL\(q\(xt∣x0\)∥p\(xt∣x0\)\)=0D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{t\}\\mid x\_\{0\}\)\\\|p\(x\_\{t\}\\mid x\_\{0\}\)\\big\)=0\. Hence the entire time\-averaged correction term in[Eq\.˜36](https://arxiv.org/html/2606.15327#A2.E36)is zero:

𝔼t,x0∼q\[DKL\(q\(xt∣x0\)∥p\(xt∣x0\)\)\]=0\.\\mathbb\{E\}\_\{t,x\_\{0\}\\sim q\}\\Big\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{t\}\\mid x\_\{0\}\)\\\|p\(x\_\{t\}\\mid x\_\{0\}\)\\big\)\\Big\]=0\.\(37\)Therefore[Eq\.˜36](https://arxiv.org/html/2606.15327#A2.E36)reduces to

DKL​\(q​\(x0\)∥p​\(x0\)\)=𝔼t​\[DKL​\(qt∥pt\)\]⏟Sampling Error\+𝔼t,xt∼qt\[DKL\(q\(x0∣xt\)∥p\(x0∣xt\)\)\]⏟Approximation Error\.D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\)\\\|p\(x\_\{0\}\)\\big\)=\\underbrace\{\\mathbb\{E\}\_\{t\}\\Big\[D\_\{\\mathrm\{KL\}\}\\big\(q\_\{t\}\\\|p\_\{t\}\\big\)\\Big\]\}\_\{\\text\{Sampling Error\}\}\+\\underbrace\{\\mathbb\{E\}\_\{t,x\_\{t\}\\sim q\_\{t\}\}\\Big\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\\mid x\_\{t\}\)\\\|p\(x\_\{0\}\\mid x\_\{t\}\)\\big\)\\Big\]\}\_\{\\text\{Approximation Error\}\}\.\(38\)This equation is importance, as it helps us to consider the error seperately from sampling and training\.

### B\.3Bias\-Variance Decomposition

#### B\.3\.1Training and sampling bias\-variance trade\-off

The bias\-variance decomposition holds for both training and sampling, so we utilizep​\(x0∣xt\)p\(x\_\{0\}\\mid x\_\{t\}\)in approximation error as an illustration, which can be easily generalized to sampling distributionp​\(xt\)p\(x\_\{t\}\)\.

With the expected predictor

p¯​\(x0∣xt\):=𝔼S​\[pθ^S​\(x0∣xt\)\]=𝔼S​\[p^S​\(x0∣xt\)\]\.\\bar\{p\}\(x\_\{0\}\\mid x\_\{t\}\):=\\mathbb\{E\}\_\{S\}\\big\[p\_\{\\hat\{\\theta\}\_\{S\}\}\(x\_\{0\}\\mid x\_\{t\}\)\\big\]=\\mathbb\{E\}\_\{S\}\\big\[\\hat\{p\}\_\{S\}\(x\_\{0\}\\mid x\_\{t\}\)\\big\]\.\(39\)whereθ^​\(S\)\\hat\{\\theta\}\(S\)denote parameter induced by specific training set and optimization randomnessSS\(short asp^S:=pθ^S\\hat\{p\}\_\{S\}:=p\_\{\\hat\{\\theta\}\_\{S\}\}\)\. Then[Eq\.˜3](https://arxiv.org/html/2606.15327#S2.E3)admits the following exact decomposition \(p​\(x0∣xt\)p\(x\_\{0\}\\mid x\_\{t\}\)simplified aspp\):

𝔼S​\[DKL​\(q∥p^S\)\]=DKL​\(q∥p¯\)\+𝒱​\(xt\),\\displaystyle\\mathbb\{E\}\_\{S\}\\Big\[D\_\{\\mathrm\{KL\}\}\(q\\\|\\hat\{p\}\_\{S\}\)\\Big\]=D\_\{\\mathrm\{KL\}\}\(q\\\|\\bar\{p\}\)\+\{\\mathcal\{V\}\}\(x\_\{t\}\),\(40\)where,​𝒱​\(xt\):=𝔼q​\(x0∣xt\)​\[log⁡p¯−𝔼S​log⁡\(p^S\)\]≥0\.\\displaystyle\\text\{where, \}\{\\mathcal\{V\}\}\(x\_\{t\}\)=\\mathbb\{E\}\_\{q\(x\_\{0\}\\mid x\_\{t\}\)\}\\Big\[\\log\\bar\{p\}\-\\mathbb\{E\}\_\{S\}\\log\(\\hat\{p\}\_\{S\}\)\\Big\]\\geq 0\.The nonnegativity of𝒱​\(xt\)\{\\mathcal\{V\}\}\(x\_\{t\}\)follows from Jensen’s inequality:𝔼S​\[log⁡p^S​\(x0∣xt\)\]≤log⁡𝔼S​\[p^S​\(x0∣xt\)\]=log⁡p¯​\(x0∣xt\)\\mathbb\{E\}\_\{S\}\[\\log\\hat\{p\}\_\{S\}\(x\_\{0\}\\mid x\_\{t\}\)\]\\leq\\log\\mathbb\{E\}\_\{S\}\[\\hat\{p\}\_\{S\}\(x\_\{0\}\\mid x\_\{t\}\)\]=\\log\\bar\{p\}\(x\_\{0\}\\mid x\_\{t\}\)\. Specifically,

𝔼S\[DKL\(q\(x0∣xt\)∥pθ^S\(x0∣xt\)\)\]=𝔼S\[∫q\(x0∣xt\)logq​\(x0∣xt\)pθ^S​\(x0∣xt\)dx0\]\\displaystyle\\mathbb\{E\}\_\{S\}\\Big\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\\mid x\_\{t\}\)\\\|p\_\{\\hat\{\\theta\}\_\{S\}\}\(x\_\{0\}\\mid x\_\{t\}\)\\big\)\\Big\]=\\mathbb\{E\}\_\{S\}\\left\[\\int q\(x\_\{0\}\\mid x\_\{t\}\)\\log\\frac\{q\(x\_\{0\}\\mid x\_\{t\}\)\}\{p\_\{\\hat\{\\theta\}\_\{S\}\}\(x\_\{0\}\\mid x\_\{t\}\)\}dx\_\{0\}\\right\]\(41\)=∫q​\(x0∣xt\)​log⁡q​\(x0∣xt\)​𝑑x0−∫q​\(x0∣xt\)​𝔼S​\[log⁡pθ^S​\(x0∣xt\)\]⏟log⁡p¯​\(x0\)​𝑑x0\\displaystyle=\\int q\(x\_\{0\}\\mid x\_\{t\}\)\\log q\(x\_\{0\}\\mid x\_\{t\}\)dx\_\{0\}\-\\int q\(x\_\{0\}\\mid x\_\{t\}\)\\underbrace\{\\mathbb\{E\}\_\{S\}\\left\[\\log p\_\{\\hat\{\\theta\}\_\{S\}\}\(x\_\{0\}\\mid x\_\{t\}\)\\right\]\}\_\{\\log\\bar\{p\}\(x\_\{0\}\)\}dx\_\{0\}=∫q​\(x0∣xt\)​\(log⁡q​\(x0∣xt\)−log⁡p¯​\(x0∣xt\)\)​𝑑x0\\displaystyle=\\int q\(x\_\{0\}\\mid x\_\{t\}\)\\left\(\\log q\(x\_\{0\}\\mid x\_\{t\}\)\-\\log\\bar\{p\}\(x\_\{0\}\\mid x\_\{t\}\)\\right\)dx\_\{0\}\+∫q​\(x0∣xt\)​\(log⁡p¯​\(x0∣xt\)−𝔼S​\[log⁡pθ^S​\(x0∣xt\)\]\)​𝑑x0\(Add & Sub\)\\displaystyle\\quad\+\\int q\(x\_\{0\}\\mid x\_\{t\}\)\\left\(\\log\\bar\{p\}\(x\_\{0\}\\mid x\_\{t\}\)\-\\mathbb\{E\}\_\{S\}\[\\log p\_\{\\hat\{\\theta\}\_\{S\}\}\(x\_\{0\}\\mid x\_\{t\}\)\]\\right\)dx\_\{0\}\\quad\\text\{\(Add \\& Sub\)\}=DKL​\(q∥p¯\)⏟Bias\+𝔼q​\(x0∣xt\)​\[log⁡p¯−𝔼S​log⁡\(p^S\)\]⏟Variance\.\\displaystyle=\\underbrace\{D\_\{\\mathrm\{KL\}\}\(q\\\|\\bar\{p\}\)\}\_\{\\textbf\{Bias\}\}\+\\underbrace\{\\mathbb\{E\}\_\{q\(x\_\{0\}\\mid x\_\{t\}\)\}\\left\[\\log\\bar\{p\}\-\\mathbb\{E\}\_\{S\}\\log\(\\hat\{p\}\_\{S\}\)\\right\]\}\_\{\\textbf\{Variance\}\}\.Averaging overxtx\_\{t\}yields an expected risk decomposition:

𝔼t,xt𝔼S\[DKL\(q\(x0∣xt\)∥p^S\(x0∣xt\)\)\]=𝔼t,xt\[DKL\(q\(x0∣xt\)∥p¯\(x0∣xt\)\)\]⏟Asymptotic Bias​ℬasym\+𝔼t,xt​\[𝒱​\(xt\)\]⏟Variance​𝒱,\\mathbb\{E\}\_\{t,x\_\{t\}\}\\mathbb\{E\}\_\{S\}\\Big\[D\_\{\\mathrm\{KL\}\}\(q\(x\_\{0\}\\mid x\_\{t\}\)\\\|\\hat\{p\}\_\{S\}\(x\_\{0\}\\mid x\_\{t\}\)\)\\Big\]=\\underbrace\{\\mathbb\{E\}\_\{t,x\_\{t\}\}\\Big\[D\_\{\\mathrm\{KL\}\}\\big\(q\(x\_\{0\}\\mid x\_\{t\}\)\\\|\\bar\{p\}\(x\_\{0\}\\mid x\_\{t\}\)\\big\)\\Big\]\}\_\{\\textbf\{Asymptotic Bias \}\{\\mathcal\{B\}\}\_\{\\text\{asym\}\}\}\+\\underbrace\{\\mathbb\{E\}\_\{t,x\_\{t\}\}\\big\[\\mathcal\{V\}\(x\_\{t\}\)\\big\]\}\_\{\\textbf\{Variance \}\{\\mathcal\{V\}\}\},\(42\)wherextx\_\{t\}are distributed according to the forward process,q​\(xt\)=∫x0q​\(xt∣x0\)​p​\(x0\)​𝑑x0q\(x\_\{t\}\)=\\int\_\{x\_\{0\}\}q\(x\_\{t\}\\mid x\_\{0\}\)p\(x\_\{0\}\)dx\_\{0\}\. The sampling bias\-variance trade\-off follows the exact same derivation, which only requires replacingp​\(x0∣xt\)p\(x\_\{0\}\\mid x\_\{t\}\)withp​\(xt\)p\(x\_\{t\}\), such that

DKL​\(qt​\(xt\)∥pt​\(xt\)\)=DKL​\(qt​\(xt\)∥p¯t​\(xt\)\)⏟Exposure Bias​ℬt\+𝔼xt​\[log⁡p¯​\(xt\)−𝔼S​\(log⁡p^S​\(xt\)\)\]⏟Sampling Roll\-out Variance​𝒱tD\_\{\\mathrm\{KL\}\}\\big\(q\_\{t\}\(x\_\{t\}\)\\\|p\_\{t\}\(x\_\{t\}\)\\big\)=\\underbrace\{D\_\{\\mathrm\{KL\}\}\\big\(q\_\{t\}\(x\_\{t\}\)\\\|\\bar\{p\}\_\{t\}\(x\_\{t\}\)\\big\)\}\_\{\\textbf\{Exposure Bias \}\{\\mathcal\{B\}\}\_\{t\}\}\+\\underbrace\{\\mathbb\{E\}\_\{x\_\{t\}\}\[\\log\\bar\{p\}\(x\_\{t\}\)\-\\mathbb\{E\}\_\{S\}\(\\log\\hat\{p\}\_\{S\}\(x\_\{t\}\)\)\]\}\_\{\\textbf\{Sampling Roll\-out Variance \}\{\\mathcal\{V\}\}\_\{t\}\}\(43\)

### B\.4Derivation for Asymptotic Bias: Local Geometry and Posterior Approximation Difficulty

This section, we prove Proposition[3](https://arxiv.org/html/2606.15327#Thmprop3), which we restate here

###### Proposition 6\(Approximation difficulty across diffusion variants\.\)\. We derive in[Sec\.˜B\.4](https://arxiv.org/html/2606.15327#A2.SS4)that,𝔼​\[dactmask\]≤𝔼​\[dactsem\]​𝔼≤\[dactuni\],𝔼​\[ℐ1mask\]≤𝔼​\[ℐ1sem\]≤𝔼​\[ℐ1uni\]\\mathbb\{E\}\\left\[d\_\{\\mathrm\{act\}\}^\{\\mathrm\{mask\}\}\\right\]\\leq\\mathbb\{E\}\\left\[d\_\{\\mathrm\{act\}\}^\{\\mathrm\{sem\}\}\\right\]\\mathbb\{E\}\\leq\\left\[d\_\{\\mathrm\{act\}\}^\{\\mathrm\{uni\}\}\\right\],\\quad\\mathbb\{E\}\\left\[\{\\mathcal\{I\}\}\_\{1\}^\{\\mathrm\{mask\}\}\\right\]\\leq\\mathbb\{E\}\\left\[\{\\mathcal\{I\}\}\_\{1\}^\{\\mathrm\{sem\}\}\\right\]\\leq\\mathbb\{E\}\\left\[\{\\mathcal\{I\}\}\_\{1\}^\{\\mathrm\{uni\}\}\\right\]\(44\)

#### B\.4\.1Characterizing the local geometry with Hessian matrix

To describe the posterior approximation difficulty, we characterize the*local geometry*of the target posterior\. Letfθ​\(xt\)i∈ℝ\|V\|f\_\{\\theta\}\(x\_\{t\}\)^\{i\}\\in\\mathbb\{R\}^\{\|V\|\}denote the predicted logits at positioni∈\[1:L\]i\\in\[1:L\]such thatpθi\(⋅∣xt\)=softmax\(fθ\(xt\)i\)p\_\{\\theta\}^\{i\}\(\\cdot\\mid x\_\{t\}\)=\\mathrm\{softmax\}\(f\_\{\\theta\}\(x\_\{t\}\)^\{i\}\)\. Assuming conditional factorization across positions givenxtx\_\{t\}, the asymptotic bias can be decomposed into local losses:

ℬasym=𝔼t,xt​\[∑i=1Lℓi​\(fθ​\(xt\)i;xt\)\],ℓi​\(fθ;xt\):=DKL​\(q​\(x0i∣xt\)∥softmax​\(fθ\)\)\.\{\\mathcal\{B\}\}\_\{\\text\{asym\}\}=\\mathbb\{E\}\_\{t,x\_\{t\}\}\\left\[\\sum\_\{i=1\}^\{L\}\\ell\_\{i\}\\left\(f\_\{\\theta\}\(x\_\{t\}\)^\{i\};x\_\{t\}\\right\)\\right\],\\qquad\\ell\_\{i\}\(f\_\{\\theta\};x\_\{t\}\):=D\_\{\\mathrm\{KL\}\}\\bigl\(q\(x\_\{0\}^\{i\}\\mid x\_\{t\}\)\\\|\\mathrm\{softmax\}\(f\_\{\\theta\}\)\\bigr\)\.\(45\)
Fix a positioniiand a corruptedxtx\_\{t\}, we have,

ℓi\(f;xt\)=DKL\(q\(x0i=⋅∣xt\)∥softmax\(fθ\)\)=∑k∈Vq\(x0i=k∣xt\)logq​\(x0i=k∣xt\)pθ​\(x0i=k∣xt\)\.\\ell\_\{i\}\(f;x\_\{t\}\)=D\_\{\\mathrm\{KL\}\}\\bigl\(q\(x\_\{0\}^\{i\}=\\cdot\\mid x\_\{t\}\)\\,\\\|\\,\\mathrm\{softmax\}\(f\_\{\\theta\}\)\\bigr\)=\\sum\_\{k\\in V\}q\(x\_\{0\}^\{i\}=k\\mid x\_\{t\}\)\\log\\frac\{q\(x\_\{0\}^\{i\}=k\\mid x\_\{t\}\)\}\{p\_\{\\theta\}\(x\_\{0\}^\{i\}=k\\mid x\_\{t\}\)\}\.\(46\)Since∑kq​\(x0i=k∣xt\)​log⁡q​\(x0i=k∣xt\)\\sum\_\{k\}q\(x\_\{0\}^\{i\}=k\\mid x\_\{t\}\)\\log q\(x\_\{0\}^\{i\}=k\\mid x\_\{t\}\)is constant w\.r\.tθ\\theta, minimizingℓi​\(f;xt\)\\ell\_\{i\}\(f;x\_\{t\}\)is equivalent to minimizing the cross\-entropy

−∑k∈Vq​\(x0i=k∣xt\)​log⁡pθ​\(x0i=k∣xt\)\.\-\\sum\_\{k\\in V\}q\(x\_\{0\}^\{i\}=k\\mid x\_\{t\}\)\\log p\_\{\\theta\}\(x\_\{0\}^\{i\}=k\\mid x\_\{t\}\)\.Recall thatlog⁡pθ​\(x0i=k∣xt\)=fθ​\(xt\)ki−log​∑m∈Vefθ​\(xt\)mi\\log p\_\{\\theta\}\(x\_\{0\}^\{i\}=k\\mid x\_\{t\}\)=f\_\{\\theta\}\(x\_\{t\}\)^\{i\}\_\{k\}\-\\log\\sum\_\{m\\in V\}e^\{f\_\{\\theta\}\(x\_\{t\}\)^\{i\}\_\{m\}\}, whenfθ​\(xt\)kif\_\{\\theta\}\(x\_\{t\}\)^\{i\}\_\{k\}denote thekk\-th logit at positionii\(we usefkf\_\{k\}when there is no ambuguity\), we obtain

∂log⁡pθ​\(x0i=k∣xt\)∂fr=δk​r−pθ​\(x0i=r∣xt\)\.\\frac\{\\partial\\log p\_\{\\theta\}\(x\_\{0\}^\{i\}=k\\mid x\_\{t\}\)\}\{\\partial f\_\{r\}\}=\\delta\_\{kr\}\-p\_\{\\theta\}\(x\_\{0\}^\{i\}=r\\mid x\_\{t\}\)\.Therefore, the gradient is

∂ℓi​\(f;xt\)∂fr\\displaystyle\\frac\{\\partial\\ell\_\{i\}\(f;x\_\{t\}\)\}\{\\partial f\_\{r\}\}=−∑k∈Vq​\(x0i=k∣xt\)​∂log⁡pθ​\(x0i=k∣xt\)∂fθ,r\\displaystyle=\-\\sum\_\{k\\in V\}q\(x\_\{0\}^\{i\}=k\\mid x\_\{t\}\)\\frac\{\\partial\\log p\_\{\\theta\}\(x\_\{0\}^\{i\}=k\\mid x\_\{t\}\)\}\{\\partial f\_\{\\theta,r\}\}=−∑k∈Vq​\(x0i=k∣xt\)​\(δk​r−pθ​\(x0i=r∣xt\)\)\\displaystyle=\-\\sum\_\{k\\in V\}q\(x\_\{0\}^\{i\}=k\\mid x\_\{t\}\)\(\\delta\_\{kr\}\-p\_\{\\theta\}\(x\_\{0\}^\{i\}=r\\mid x\_\{t\}\)\)=−q​\(x0i=r∣xt\)\+pθ​\(x0i=r∣xt\)​∑k∈Vq​\(x0i=k∣xt\)⏟=1\\displaystyle=\-q\(x\_\{0\}^\{i\}=r\\mid x\_\{t\}\)\+p\_\{\\theta\}\(x\_\{0\}^\{i\}=r\\mid x\_\{t\}\)\\underbrace\{\\sum\_\{k\\in V\}q\(x\_\{0\}^\{i\}=k\\mid x\_\{t\}\)\}\_\{=1\}=pθ​\(x0i=r∣xt\)−q​\(x0i=r∣xt\),\\displaystyle=p\_\{\\theta\}\(x\_\{0\}^\{i\}=r\\mid x\_\{t\}\)\-q\(x\_\{0\}^\{i\}=r\\mid x\_\{t\}\),
By the chain rule and the decomposition of the global loss,∇θℒ\\displaystyle\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}=∇θ𝔼t,xt​\[∑i=1Lℓi​\(fθ​\(xt\)i;xt\)\]\\displaystyle=\\nabla\_\{\\theta\}\\mathbb\{E\}\_\{t,x\_\{t\}\}\\\!\\left\[\\sum\_\{i=1\}^\{L\}\\ell\_\{i\}\\\!\\left\(f\_\{\\theta\}\(x\_\{t\}\)^\{i\};x\_\{t\}\\right\)\\right\]=𝔼t,xt​\[∑i=1L∑k∈V∂ℓi​\(fθ​\(xt\)i;xt\)∂fθ​\(xt\)ki⋅∇θfθ​\(xt\)ki\]\\displaystyle=\\mathbb\{E\}\_\{t,x\_\{t\}\}\\left\[\\sum\_\{i=1\}^\{L\}\\sum\_\{k\\in V\}\\frac\{\\partial\\ell\_\{i\}\(f\_\{\\theta\}\(x\_\{t\}\)^\{i\};x\_\{t\}\)\}\{\\partial f\_\{\\theta\}\(x\_\{t\}\)^\{i\}\_\{k\}\}\\cdot\\nabla\_\{\\theta\}f\_\{\\theta\}\(x\_\{t\}\)^\{i\}\_\{k\}\\right\]=𝔼t,xt​\[∑i=1L∑k∈V\(pθ​\(x0i=k∣xt\)−q​\(x0i=k∣xt\)\)⋅∇θfθ​\(xt\)ki\],\\displaystyle=\\mathbb\{E\}\_\{t,x\_\{t\}\}\\left\[\\sum\_\{i=1\}^\{L\}\\sum\_\{k\\in V\}\\Big\(p\_\{\\theta\}\(x\_\{0\}^\{i\}=k\\mid x\_\{t\}\)\-q\(x\_\{0\}^\{i\}=k\\mid x\_\{t\}\)\\Big\)\\cdot\\nabla\_\{\\theta\}f\_\{\\theta\}\(x\_\{t\}\)^\{i\}\_\{k\}\\right\],\(47\)which is exactly[Eq\.˜14](https://arxiv.org/html/2606.15327#S3.E14)\.

Sinceqqis irrelevant with f, differentiate[Sec\.˜B\.4\.1](https://arxiv.org/html/2606.15327#A2.Ex8)once more we obtain

∇f2ℓ​\(f;q\)=∂p∂f\.\\nabla\_\{f\}^\{2\}\\ell\(f;q\)=\\frac\{\\partial p\}\{\\partial f\}\.For softmax, the Jacobian is

∂pk∂fr=pk​\(δk​r−pr\),\\frac\{\\partial p\_\{k\}\}\{\\partial f\_\{r\}\}=p\_\{k\}\(\\delta\_\{kr\}\-p\_\{r\}\),hence

∇f2ℓ​\(f;q\)=Diag​\(p\)−p​p⊤\.\\nabla\_\{f\}^\{2\}\\ell\(f;q\)=\\text\{Diag\}\(p\)\-pp^\{\\top\}\.\(48\)At any optimumf⋆​\(q\)f^\{\\star\}\(q\)such thatsoftmax​\(f⋆​\(q\)\)=q\\mathrm\{softmax\}\(f^\{\\star\}\(q\)\)=q, we obtain

∇f2ℓ​\(f⋆​\(q\);q\)=Diag​\(q\)−q​q⊤=Σ​\(q\)\.\\nabla\_\{f\}^\{2\}\\ell\(f^\{\\star\}\(q\);q\)=\\text\{Diag\}\(q\)\-qq^\{\\top\}=\\Sigma\(q\)\.\(49\)
##### Local Taylor expansion\.

By standard second\-order expansion aroundf⋆​\(q\)f^\{\\star\}\(q\), and denoteδ​f≈f−f∗​\(q\)\\delta f\\approx f\-f^\{\*\}\(q\)a small variation aroundff, using taylor expansion aroundf∗f^\{\*\}we have,

ℓ​\(f⋆​\(q\)\+δ​f;q\)=ℓ​\(f⋆​\(q\);q\)\+∇fℓ​\(f⋆​\(q\);q\)⏟=0⊤​δ​f\+12​δ​f⊤​Σ​\(q\)​δ​f\+o​\(‖δ​f‖2\),\\ell\(f^\{\\star\}\(q\)\+\\delta f;q\)=\\ell\(f^\{\\star\}\(q\);q\)\+\\underbrace\{\\nabla\_\{f\}\\ell\(f^\{\\star\}\(q\);q\)\}\_\{=\\,0\}^\{\\top\}\\delta f\+\\frac\{1\}\{2\}\\delta f^\{\\top\}\\Sigma\(q\)\\delta f\+o\(\\\|\\delta f\\\|^\{2\}\),\(50\)which gives

ℓ​\(f;q\)−ℓ​\(f⋆​\(q\);q\)=12​δ​f⊤​Σ​\(q\)​δ​f\+o​\(‖δ​f‖2\)\.\\ell\(f;q\)\-\\ell\(f^\{\\star\}\(q\);q\)=\\frac\{1\}\{2\}\\delta f^\{\\top\}\\Sigma\(q\)\\delta f\+o\(\\\|\\delta f\\\|^\{2\}\)\.\(51\)hence, we obtain[Eq\.˜15](https://arxiv.org/html/2606.15327#S3.E15), where

∇fℓ​\(f;q\)=Σ​\(q\)​\(f−f∗​\(q\)\)\+o​\(‖f−f∗​\(q\)‖\),with​Σ​\(q\):=∇f2ℓ​\(f;q\)=Diag​\(q\)−q​q⊤\.\\nabla\_\{f\}\\ell\(f;q\)=\\Sigma\(q\)\(f\-f^\{\*\}\(q\)\)\+o\\left\(\\\|f\-f^\{\*\}\(q\)\\\|\\right\),\\text\{with \}\\Sigma\(q\):=\\nabla^\{2\}\_\{f\}\\ell\(f;q\)=\\text\{Diag\}\(q\)\-qq^\{\\top\}\.\(52\)This suggest that the shape of the second order hessian will dominant the optimization dynamics of our diffusion LLMs\.

#### B\.4\.2Metrics for posterior approximation difficulty\.

Through our Hessian matrixΣ​\(q\)\\Sigma\(q\), we can then define a few metrics that is helpful in analyzing the posterior approximation difficulty in DLMs\. Specifically,

dact​\(q\):=rank⁡\(Σ​\(q\)\),ℐ1​\(q\):=tr⁡\(Σ​\(q\)\),ℐ2​\(q\):=tr⁡\(Σ​\(q\)2\)\.d\_\{\\mathrm\{act\}\}\(q\):=\\operatorname\{rank\}\\bigl\(\\Sigma\(q\)\\bigr\),\\quad\{\\mathcal\{I\}\}\_\{1\}\(q\):=\\operatorname\{tr\}\\bigl\(\\Sigma\(q\)\\bigr\),\\quad\{\\mathcal\{I\}\}\_\{2\}\(q\):=\\operatorname\{tr\}\\bigl\(\\Sigma\(q\)^\{2\}\\bigr\)\.\(53\)Specifically,

1. 1\.dactd\_\{\\mathrm\{act\}\}counts the dimension of the tangent space on which the KL objective has non\-zero curvature\. Hence it is the number of independent logit directions that must be fitted simultaneously\.
2. 2\.ℐ1\{\\mathcal\{I\}\}\_\{1\}measures the average curvature and therefore how much an isotropic local logit error increases the KL loss\.
3. 3\.ℐ2\{\\mathcal\{I\}\}\_\{2\}measures the squared curvature and therefore the local gradient energy induced by the same logit error\.

We provide some justification on what they mean in the optimization problem\.

##### dact​\(q\)d\_\{\\mathrm\{act\}\}\(q\)measures the independent direction to be fitted\.

Given thatdact\(q\)=rank\(Σ\(q\)=supp\(q\)d\_\{\\mathrm\{act\}\}\(q\)=\\operatorname\{rank\}\(\\Sigma\(q\)=\\operatorname\{supp\}\(q\)\. It directly measure the independent directions that is required to be fitted\.

##### 2\)ℐ1​\(q\)\{\\mathcal\{I\}\}\_\{1\}\(q\)measures local error sensitivity\.

Usingtr⁡\(q​q⊤\)=‖q‖22\\operatorname\{tr\}\(qq^\{\\top\}\)=\\\|q\\\|\_\{2\}^\{2\},

ℐ1​\(q\)=tr⁡\(Σ​\(q\)\)=tr⁡\(Diag⁡\(q\)\)−tr⁡\(q​q⊤\)=1−‖q‖22\.\{\\mathcal\{I\}\}\_\{1\}\(q\)=\\operatorname\{tr\}\(\\Sigma\(q\)\)=\\operatorname\{tr\}\(\\operatorname\{Diag\}\(q\)\)\-\\operatorname\{tr\}\(qq^\{\\top\}\)=1\-\\\|q\\\|\_\{2\}^\{2\}\.\(54\)Around the optimal logitf⋆​\(q\)f^\{\\star\}\(q\),𝔼​\[δ​f\]=0\\mathbb\{E\}\[\\delta f\]=0and𝔼​\[δ​f​δ​f⊤\]=σ2​I\\mathbb\{E\}\[\\delta f\\delta f^\{\\top\}\]=\\sigma^\{2\}I, The Taylor expansion gives

𝔼​\[ℓ​\(f⋆​\(q\)\+δ​f;q\)−ℓ​\(f⋆​\(q\);q\)\]=σ22​tr⁡\(Σ​\(q\)\)\+o​\(σ2\)=σ22​ℐ1​\(q\)\+o​\(σ2\)\.\\mathbb\{E\}\\bigl\[\\ell\(f^\{\\star\}\(q\)\+\\delta f;q\)\-\\ell\(f^\{\\star\}\(q\);q\)\\bigr\]=\\frac\{\\sigma^\{2\}\}\{2\}\\operatorname\{tr\}\(\\Sigma\(q\)\)\+o\(\\sigma^\{2\}\)=\\frac\{\\sigma^\{2\}\}\{2\}\{\\mathcal\{I\}\}\_\{1\}\(q\)\+o\(\\sigma^\{2\}\)\.\(55\)Thereforeℐ1​\(q\)\{\\mathcal\{I\}\}\_\{1\}\(q\)is exactly the first\-order coefficient converting local logit mismatch into excess posterior KL\.

##### ℐ2​\(q\)\{\\mathcal\{I\}\}\_\{2\}\(q\)measures gradient energy\.

The local gradient linearization gives∇fℓ​\(f⋆​\(q\)\+δ​f;q\)=Σ​\(q\)​δ​f\+o​\(‖δ​f‖\)\\nabla\_\{f\}\\ell\(f^\{\\star\}\(q\)\+\\delta f;q\)=\\Sigma\(q\)\\delta f\+o\(\\\|\\delta f\\\|\)\. Hence

𝔼​‖∇fℓ​\(f⋆​\(q\)\+δ​f;q\)‖22=σ2​tr⁡\(Σ​\(q\)2\)\+o​\(σ2\)=σ2​ℐ2​\(q\)\+o​\(σ2\)\.\\mathbb\{E\}\\\|\\nabla\_\{f\}\\ell\(f^\{\\star\}\(q\)\+\\delta f;q\)\\\|\_\{2\}^\{2\}=\\sigma^\{2\}\\operatorname\{tr\}\(\\Sigma\(q\)^\{2\}\)\+o\(\\sigma^\{2\}\)=\\sigma^\{2\}\{\\mathcal\{I\}\}\_\{2\}\(q\)\+o\(\\sigma^\{2\}\)\.\(56\)Thusℐ2​\(q\)\{\\mathcal\{I\}\}\_\{2\}\(q\)measures the local gradient energy created by posterior fitting\. A largerℐ2\{\\mathcal\{I\}\}\_\{2\}means that small logit errors induce larger gradients and can contribute to noisier optimization\.

Together,dactd\_\{\\mathrm\{act\}\},ℐ1\{\\mathcal\{I\}\}\_\{1\}, andℐ2\{\\mathcal\{I\}\}\_\{2\}summarize three aspects of posterior fitting: how many logit directions are active, how strongly local mismatch becomes KL error, and how much gradient energy the mismatch creates\. Smaller values indicate an easier local posterior approximation problem\.

##### Approximation difficulty across diffusion variants\.

We next show how the transition kernel changes these quantities\. For simplicity, we fix one position and suppress the position index and the context\. Letqdata​\(x0=k\)q\_\{\\mathrm\{data\}\}\(x\_\{0\}=k\)be the clean\-token data prior, and define the support asS:=\{k:qdata​\(x0=k\)\>0\}S:=\\\{k:q\_\{\\mathrm\{data\}\}\(x\_\{0\}=k\)\>0\\\}\.We denote the forward kernel asqt​\(j∣k\):=q​\(xt=j∣x0=k\)q\_\{t\}\(j\\mid k\):=q\(x\_\{t\}=j\\mid x\_\{0\}=k\)and gives the posterior

q​\(x0=k∣xt=j\)=qdata​\(x0=k\)​qt​\(j∣k\)∑r∈Sqdata​\(x0=r\)​qt​\(j∣r\)\.q\(x\_\{0\}=k\\mid x\_\{t\}=j\)=\\frac\{q\_\{\\mathrm\{data\}\}\(x\_\{0\}=k\)q\_\{t\}\(j\\mid k\)\}\{\\sum\_\{r\\in S\}q\_\{\\mathrm\{data\}\}\(x\_\{0\}=r\)q\_\{t\}\(j\\mid r\)\}\.\(57\)Thus the support for the posterior is,

suppq\(x0=⋅∣xt=j\)=S∩\{k:qt\(j∣k\)\>0\}\.\\operatorname\{supp\}q\(x\_\{0\}=\\cdot\\mid x\_\{t\}=j\)=S\\cap\\\{k:q\_\{t\}\(j\\mid k\)\>0\\\}\.\(58\)This candidate set controlsdactd\_\{\\mathrm\{act\}\}, while its concentration controlsℐ1=1−‖q‖22\{\\mathcal\{I\}\}\_\{1\}=1\-\\\|q\\\|\_\{2\}^\{2\}\.

We then discuss different diffusion variants\. Please note that all the following discussion we useqt​\(j∣k\):=q​\(xt=j∣x0=k\)q\_\{t\}\(j\\mid k\):=q\(x\_\{t\}=j\\mid x\_\{0\}=k\)for simplicity\.

##### Masking diffusion\.

For the absorbing kernel,

qtmask​\(j∣k\)=αt​𝕀​\{j=k\}\+\(1−αt\)​𝕀​\{j=\[𝙼𝙰𝚂𝙺\]\}\.q\_\{t\}^\{\\mathrm\{mask\}\}\(j\\mid k\)=\\alpha\_\{t\}\\mathbb\{I\}\\\{j=k\\\}\+\(1\-\\alpha\_\{t\}\)\\mathbb\{I\}\\\{j=\\mathtt\{\[MASK\]\}\\\}\.Ifj≠\[𝙼𝙰𝚂𝙺\]j\\neq\\mathtt\{\[MASK\]\}, the clean token must bejj, so the posterior isδj\\delta\_\{j\}anddact=ℐ1=0d\_\{\\mathrm\{act\}\}=\{\\mathcal\{I\}\}\_\{1\}=0\. Ifj=\[𝙼𝙰𝚂𝙺\]j=\\mathtt\{\[MASK\]\}, the corrupted token gives no token\-level information and the posterior isqdata​\(x0=⋅\)q\_\{\\mathrm\{data\}\}\(x\_\{0\}=\\cdot\), sodact=\|S\|−1d\_\{\\mathrm\{act\}\}=\|S\|\-1andℐ1=1−∑k∈Sqdata​\(x0=k\)2\{\\mathcal\{I\}\}\_\{1\}=1\-\\sum\_\{k\\in S\}q\_\{\\mathrm\{data\}\}\(x\_\{0\}=k\)^\{2\}\. Since the mask event has probability1−αt1\-\\alpha\_\{t\}, we can take expectation and get

𝔼​\[dactmask\]=\(1−αt\)​\(\|S\|−1\),𝔼​\[ℐ1mask\]=\(1−αt\)​\(1−∑k∈Sqdata​\(x0=k\)2\)\.\\mathbb\{E\}\[d\_\{\\mathrm\{act\}\}^\{\\mathrm\{mask\}\}\]=\(1\-\\alpha\_\{t\}\)\(\|S\|\-1\),\\qquad\\mathbb\{E\}\[\{\\mathcal\{I\}\}\_\{1\}^\{\\mathrm\{mask\}\}\]=\(1\-\\alpha\_\{t\}\)\\left\(1\-\\sum\_\{k\\in S\}q\_\{\\mathrm\{data\}\}\(x\_\{0\}=k\)^\{2\}\\right\)\.\(59\)

##### Uniform diffusion\.

For the uniform kernel,

qtuni​\(j∣k\)=αt​𝕀​\{j=k\}\+\|𝒱\|−1​\(1−αt\)\.q\_\{t\}^\{\\mathrm\{uni\}\}\(j\\mid k\)=\\alpha\_\{t\}\\mathbb\{I\}\\\{j=k\\\}\+\|\\mathcal\{V\}\|^\{\-1\}\(1\-\\alpha\_\{t\}\)\.When0<αt<10<\\alpha\_\{t\}<1,qtuni​\(j∣k\)\>0q\_\{t\}^\{\\mathrm\{uni\}\}\(j\\mid k\)\>0for everyk∈Sk\\in S\. Hence every observedjjleaves all clean tokens inSSpossible:

dactuni​\(j\)=\|S\|−1\.d\_\{\\mathrm\{act\}\}^\{\\mathrm\{uni\}\}\(j\)=\|S\|\-1\.\(60\)Uniform corruption is therefore harder than masking with more active dimension\. It is also less informative such that

𝔼​\[ℐ1mask\]≤𝔼​\[ℐ1uni\]\.\\mathbb\{E\}\[\{\\mathcal\{I\}\}\_\{1\}^\{\\mathrm\{mask\}\}\]\\leq\\mathbb\{E\}\[\{\\mathcal\{I\}\}\_\{1\}^\{\\mathrm\{uni\}\}\]\.\(61\)

##### Semantic diffusion\.

For a localized semantic kernel,

qtsem​\(j∣k\)=αt​𝕀​\{j=k\}\+\(1−αt\)​stsem​\(j∣k\),q\_\{t\}^\{\\mathrm\{sem\}\}\(j\\mid k\)=\\alpha\_\{t\}\\mathbb\{I\}\\\{j=k\\\}\+\(1\-\\alpha\_\{t\}\)s\_\{t\}^\{\\mathrm\{sem\}\}\(j\\mid k\),the possible clean explanations ofjjare

Ssem​\(j\):=S∩\(\{j\}∪\{k:stsem​\(j∣k\)\>0\}\)\.S\_\{\\mathrm\{sem\}\}\(j\):=S\\cap\\left\(\\\{j\\\}\\cup\\\{k:s\_\{t\}^\{\\mathrm\{sem\}\}\(j\\mid k\)\>0\\\}\\right\)\.Then

dactsem​\(j\)=\|Ssem​\(j\)\|−1\.d\_\{\\mathrm\{act\}\}^\{\\mathrm\{sem\}\}\(j\)=\|S\_\{\\mathrm\{sem\}\}\(j\)\|\-1\.\(62\)If the semantic neighborhoods are genuinely local, then

\{j\}∩S⊆Ssem​\(j\)⊆S,\\\{j\\\}\\cap S\\subseteq S\_\{\\mathrm\{sem\}\}\(j\)\\subseteq S,so semantic diffusion activates a local set of clean explanations: larger than a copied masking token, but smaller than the full clean support used by uniform diffusion\. This gives

𝔼​\[dactmask\]≤𝔼​\[dactsem\]≤𝔼​\[dactuni\]\.\\mathbb\{E\}\[d\_\{\\mathrm\{act\}\}^\{\\mathrm\{mask\}\}\]\\leq\\mathbb\{E\}\[d\_\{\\mathrm\{act\}\}^\{\\mathrm\{sem\}\}\]\\leq\\mathbb\{E\}\[d\_\{\\mathrm\{act\}\}^\{\\mathrm\{uni\}\}\]\.\(63\)
Forℐ1\{\\mathcal\{I\}\}\_\{1\}, recall thatℐ1​\(q\)=1−‖q‖22\{\\mathcal\{I\}\}\_\{1\}\(q\)=1\-\\\|q\\\|\_\{2\}^\{2\}increases as the posterior becomes less concentrated\. Masking gives delta posteriors on visible tokens, uniform keeps the broadest set of explanations active, and semantic diffusion lies between them by restricting explanations to semantic neighborhoods\. Therefore, under the same intermediate\-kernel condition,

𝔼​\[ℐ1mask\]≤𝔼​\[ℐ1sem\]≤𝔼​\[ℐ1uni\]\.\\mathbb\{E\}\[\{\\mathcal\{I\}\}\_\{1\}^\{\\mathrm\{mask\}\}\]\\leq\\mathbb\{E\}\[\{\\mathcal\{I\}\}\_\{1\}^\{\\mathrm\{sem\}\}\]\\leq\\mathbb\{E\}\[\{\\mathcal\{I\}\}\_\{1\}^\{\\mathrm\{uni\}\}\]\.\(64\)A simple sufficient case is a locally flat posterior over each candidate set: ifqqis approximately uniform overmmcandidates, thenℐ1​\(q\)=1−1/m\{\\mathcal\{I\}\}\_\{1\}\(q\)=1\-1/m, which grows withmm\.

Finally, averaging equation[63](https://arxiv.org/html/2606.15327#A2.E63)and equation[64](https://arxiv.org/html/2606.15327#A2.E64)overttyields the main\-text comparison:

𝔼​\[dactmask\]≤𝔼​\[dactsem\]≤𝔼​\[dactuni\],𝔼​\[ℐ1mask\]≤𝔼​\[ℐ1sem\]≤𝔼​\[ℐ1uni\]\.\\mathbb\{E\}\\left\[d\_\{\\mathrm\{act\}\}^\{\\mathrm\{mask\}\}\\right\]\\leq\\mathbb\{E\}\\left\[d\_\{\\mathrm\{act\}\}^\{\\mathrm\{sem\}\}\\right\]\\leq\\mathbb\{E\}\\left\[d\_\{\\mathrm\{act\}\}^\{\\mathrm\{uni\}\}\\right\],\\qquad\\mathbb\{E\}\\left\[\{\\mathcal\{I\}\}\_\{1\}^\{\\mathrm\{mask\}\}\\right\]\\leq\\mathbb\{E\}\\left\[\{\\mathcal\{I\}\}\_\{1\}^\{\\mathrm\{sem\}\}\\right\]\\leq\\mathbb\{E\}\\left\[\{\\mathcal\{I\}\}\_\{1\}^\{\\mathrm\{uni\}\}\\right\]\.\(65\)

##### Dense semantic proposals\.

Ifstsems\_\{t\}^\{\\mathrm\{sem\}\}is implemented by a dense softmax over semantic scores, then the exact support can become the full vocabulary, making the exact rank equal to the uniform rank\. In that case the same argument should be read in terms of theϵ\\epsilon\-effective support

suppϵ⁡\(q\):=\{k:qk≥ϵ\},dact,ϵ​\(q\):=\|suppϵ⁡\(q\)\|−1,\\operatorname\{supp\}\_\{\\epsilon\}\(q\):=\\\{k:q\_\{k\}\\geq\\epsilon\\\},\\qquad d\_\{\\mathrm\{act\},\\epsilon\}\(q\):=\|\\operatorname\{supp\}\_\{\\epsilon\}\(q\)\|\-1,or equivalently in the low\-temperature regime where most posterior mass lies in the semantic neighborhood\. This is the practical sense in which semantic diffusion reduces the active posterior geometry relative to uniform diffusion while retaining more active directions than masking\.

### B\.5Proof for Exposure\-bias Propagation

We restate proposition[4](https://arxiv.org/html/2606.15327#Thmprop4)and give proof here\.

###### Proposition 7\(Exposure Bias Propagation\)\. With mild conditions, the propagation satisfies:ℬt−1≤ηt​ℬt\+ρt,\{\\mathcal\{B\}\}\_\{t\-1\}\\leq\\eta\_\{t\}\\,\{\\mathcal\{B\}\}\_\{t\}\+\\rho\_\{t\},whereρt\\rho\_\{t\}upper\-bounds the step\-wise error:ρt≥supxtDKL\(qt\(xt−1∣xt\)∥ptθ\(xt−1∣xt\)\)\\rho\_\{t\}\\geq\\sup\_\{x\_\{t\}\}D\_\{\\mathrm\{KL\}\}\\bigl\(q\_\{t\}\(x\_\{t\-1\}\\mid x\_\{t\}\)\\\|p\_\{t\}^\{\\theta\}\(x\_\{t\-1\}\\mid x\_\{t\}\)\\bigr\)andηtKL∈\[0,1\]\\eta\_\{t\}^\{\\mathrm\{KL\}\}\\in\[0,1\]is the step\-wise error propagation coefficient\. In particular,ηt,mask=1,ηt,rm≤1−rt​λtref,ηt,uni≤1−λtuni<1,ηt,sem≤1−λtsem<1\.\\eta\_\{t,\\mathrm\{mask\}\}=1,\\quad\\eta\_\{t,\\mathrm\{rm\}\}\\leq 1\-r\_\{t\}\\lambda\_\{t\}^\{\\mathrm\{ref\}\},\\quad\\eta\_\{t,\\mathrm\{uni\}\}\\leq 1\-\\lambda\_\{t\}^\{\\mathrm\{uni\}\}<1,\\quad\\eta\_\{t,\\mathrm\{sem\}\}\\leq 1\-\\lambda\_\{t\}^\{\\mathrm\{sem\}\}<1\.Consequently, forsemantic and uniform diffusion,ρt≤ρ\\rho\_\{t\}\\leq\\rhoandηt≤η<1\\eta\_\{t\}\\leq\\eta<1, thenℬexp=∑t=0Tℬt=𝒪​\(T​ρ1−η\);\\mathcal\{B\}\_\{\\exp\}=\\sum\_\{t=0\}^\{T\}\{\\mathcal\{B\}\}\_\{t\}=\\mathcal\{O\}\\\!\\left\(\\frac\{T\\,\\rho\}\{1\-\\eta\}\\right\);whereas formasking diffusionηtKL≈1\\eta\_\{t\}^\{\\mathrm\{KL\}\}\\approx 1andℬexp=𝒪​\(T2​ρ\)\.\\mathcal\{B\}\_\{\\exp\}=\\mathcal\{O\}\\\!\\left\(T^\{2\}\\rho\\right\)\.

Please note that, in the main text, we uset−d​tt\-dtinstead oft−1t\-1in the recursion\. This is just a re\-indexing of thettfrom exact continuous time to sampling stepTT, thus have no impact in our conclusion\. In this proof, we followt=\[0:T\]t=\[0:T\]notation as this can simplify the proof\.

Proof\.For one reverse stept→t−1t\\to t\-1, write

Qt​\(y∣x\):=q​\(xt−1=y∣xt=x\),Ptθ​\(y∣x\):=pθ​\(xt−1=y∣xt=x\)\.Q\_\{t\}\(y\\mid x\):=q\(x\_\{t\-1\}=y\\mid x\_\{t\}=x\),\\qquad P\_\{t\}^\{\\theta\}\(y\\mid x\):=p\_\{\\theta\}\(x\_\{t\-1\}=y\\mid x\_\{t\}=x\)\.\(66\)Thusqt−1=qt​Qtq\_\{t\-1\}=q\_\{t\}Q\_\{t\},pt−1=pt​Ptθp\_\{t\-1\}=p\_\{t\}P\_\{t\}^\{\\theta\}, andℬt=DKL​\(qt∥pt\)\{\\mathcal\{B\}\}\_\{t\}=D\_\{\\mathrm\{KL\}\}\(q\_\{t\}\\\|p\_\{t\}\)\. We measure the repair ability of the true reverse kernel by its KL contraction coefficient

ηt:=supμ,νDKL​\(μ​Qt∥π​Qt\)DKL​\(μ∥π\)∈\[0,1\],\\eta\_\{t\}:=\\sup\_\{\\mu,\\nu\}\\frac\{D\_\{\\mathrm\{KL\}\}\(\\mu Q\_\{t\}\\\|\\pi Q\_\{t\}\)\}\{D\_\{\\mathrm\{KL\}\}\(\\mu\\\|\\pi\)\}\\in\[0,1\],\(67\)where the supremum is over pairs with finite nonzero denominator\.ηt\\eta\_\{t\}measures under true transition kernelQtQ\_\{t\}, what will be the error propagated\.

We then useρt\\rho\_\{t\}to measure the error between true kernelQtQ\_\{t\}and learned kernelPtθP\_\{t\}^\{\\theta\},

ρt≥supx,ylog⁡Qt​\(y∣x\)Ptθ​\(y∣x\)\.\\rho\_\{t\}\\geq\\sup\_\{x,y\}\\log\\frac\{Q\_\{t\}\(y\\mid x\)\}\{P\_\{t\}^\{\\theta\}\(y\\mid x\)\}\.This gives

Qt​\(y∣x\)≤eρt​Ptθ​\(y∣x\),∀x,y\.Q\_\{t\}\(y\\mid x\)\\leq e^\{\\rho\_\{t\}\}P\_\{t\}^\{\\theta\}\(y\\mid x\),\\qquad\\forall x,y\.\(68\)With the definition ofηt\\eta\_\{t\}andρt\\rho\_\{t\}, we can derive the one\-step exposure bias recursion\.

###### Theorem B\.1\(One\-step exposure\-bias recursion\)\.

Under[Eq\.˜67](https://arxiv.org/html/2606.15327#A2.E67)and[Eq\.˜68](https://arxiv.org/html/2606.15327#A2.E68), we have

ℬt−1≤ηt​ℬt\+ρt\.\{\\mathcal\{B\}\}\_\{t\-1\}\\leq\\eta\_\{t\}\\,\{\\mathcal\{B\}\}\_\{t\}\+\\rho\_\{t\}\.\(69\)

###### Proof\.

We expand the per\-step exposure bias as

ℬt−1=DKL​\(qt−1∥pt−1θ\)​and​ℬt−1=DKL​\(qt∥ptθ\)\{\\mathcal\{B\}\}\_\{t\-1\}=D\_\{\\mathrm\{KL\}\}\(q\_\{t\-1\}\\\|p\_\{t\-1\}^\{\\theta\}\)\\text\{ and \}\{\\mathcal\{B\}\}\_\{t\-1\}=D\_\{\\mathrm\{KL\}\}\(q\_\{t\}\\\|p\_\{t\}^\{\\theta\}\)By[Eq\.˜68](https://arxiv.org/html/2606.15327#A2.E68), the mixed distributions also satisfy

\(pt​Qt\)​\(y\)=∑xpt​\(x\)​Qt​\(y∣x\)≤eρt​∑xpt​\(x\)​Ptθ​\(y∣x\)=eρt​\(pt​Ptθ\)​\(y\)\.\(p\_\{t\}Q\_\{t\}\)\(y\)=\\sum\_\{x\}p\_\{t\}\(x\)Q\_\{t\}\(y\\mid x\)\\leq e^\{\\rho\_\{t\}\}\\sum\_\{x\}p\_\{t\}\(x\)P\_\{t\}^\{\\theta\}\(y\\mid x\)=e^\{\\rho\_\{t\}\}\(p\_\{t\}P\_\{t\}^\{\\theta\}\)\(y\)\.Therefore, withqt−1=qt​Qtq\_\{t\-1\}=q\_\{t\}Q\_\{t\}andpt−1=pt​Ptθp\_\{t\-1\}=p\_\{t\}P\_\{t\}^\{\\theta\}

DKL​\(qt−1∥pt−1θ\)=DKL​\(qt​Qt∥pt​Ptθ\)\\displaystyle D\_\{\\mathrm\{KL\}\}\(q\_\{t\-1\}\\\|p\_\{t\-1\}^\{\\theta\}\)=D\_\{\\mathrm\{KL\}\}\(q\_\{t\}Q\_\{t\}\\\|p\_\{t\}P\_\{t\}^\{\\theta\}\)=∑y\(qt​Qt\)​\(y\)​log⁡\(qt​Qt\)​\(y\)\(pt​Ptθ\)​\(y\)\\displaystyle=\\sum\_\{y\}\(q\_\{t\}Q\_\{t\}\)\(y\)\\log\\frac\{\(q\_\{t\}Q\_\{t\}\)\(y\)\}\{\(p\_\{t\}P\_\{t\}^\{\\theta\}\)\(y\)\}≤∑y\(qt​Qt\)​\(y\)​log⁡\(qt​Qt\)​\(y\)\(pt​Qt\)​\(y\)\+ρt\\displaystyle\\leq\\sum\_\{y\}\(q\_\{t\}Q\_\{t\}\)\(y\)\\log\\frac\{\(q\_\{t\}Q\_\{t\}\)\(y\)\}\{\(p\_\{t\}Q\_\{t\}\)\(y\)\}\+\\rho\_\{t\}=DKL​\(qt​Qt∥pt​Qt\)\+ρt\\displaystyle=D\_\{\\mathrm\{KL\}\}\(q\_\{t\}Q\_\{t\}\\\|p\_\{t\}Q\_\{t\}\)\+\\rho\_\{t\}≤ηt​DKL​\(qt∥ptθ\)\+ρt\.\\displaystyle\\leq\\eta\_\{t\}D\_\{\\mathrm\{KL\}\}\(q\_\{t\}\\\|p\_\{t\}^\{\\theta\}\)\+\\rho\_\{t\}\.\(70\)This is exactly[Eq\.˜69](https://arxiv.org/html/2606.15327#A2.E69)\. ∎

The recursion shows that exposure bias has two sources\. The termρt\\rho\_\{t\}is the local denoising error of this step\. The coefficientηt\\eta\_\{t\}decides whether previous rollout mismatch is preserved or contracted\.

##### When isηt<1\\eta\_\{t\}<1?

The useful case is when the reverse kernel contains a shared component \(for instance, global jumping\)\. If there existλt\>0\\lambda\_\{t\}\>0and a distributionνt\\nu\_\{t\}such that

Qt\(⋅∣x\)≥λtνt\(⋅\)∀x,such thatηt≤1−λtQ\_\{t\}\(\\cdot\\mid x\)\\geq\\lambda\_\{t\}\\nu\_\{t\}\(\\cdot\)\\quad\\forall x,\\text\{ such that \}\\eta\_\{t\}\\leq 1\-\\lambda\_\{t\}\\quad\(71\)We can[Eq\.˜71](https://arxiv.org/html/2606.15327#A2.E71)allows us to rewrite the transition

Qt\(⋅∣x\)=λtνt\(⋅\)\+\(1−λt\)Q~t\(⋅∣x\)Q\_\{t\}\(\\cdot\\mid x\)=\\lambda\_\{t\}\\nu\_\{t\}\(\\cdot\)\+\(1\-\\lambda\_\{t\}\)\\widetilde\{Q\}\_\{t\}\(\\cdot\\mid x\)for another Markov kernelQ~t\\widetilde\{Q\}\_\{t\}\. This is because for anyμ,ν\\mu,\\nu, joint convexity of KL gives

DKL​\(μ​Qt∥ν​Qt\)≤\(1−λt\)​DKL​\(μ​Q~t∥ν​Q~t\),D\_\{\\mathrm\{KL\}\}\(\\mu Q\_\{t\}\\\|\\nu Q\_\{t\}\)\\leq\(1\-\\lambda\_\{t\}\)D\_\{\\mathrm\{KL\}\}\(\\mu\\widetilde\{Q\}\_\{t\}\\\|\\nu\\widetilde\{Q\}\_\{t\}\),and ordinary data processing gives

DKL​\(μ​Q~t∥ν​Q~t\)≤DKL​\(μ∥ν\)\.D\_\{\\mathrm\{KL\}\}\(\\mu\\widetilde\{Q\}\_\{t\}\\\|\\nu\\widetilde\{Q\}\_\{t\}\)\\leq D\_\{\\mathrm\{KL\}\}\(\\mu\\\|\\nu\)\.Thus, a source\-independent refresh mass \(for instance, the global jumping in uniform and semantic diffusion\) can make the reverse step reduce exposure bias\.

##### Instantiating the coefficient for diffusion variants\.

We now apply the above criterion to the transition kernels discussed in the main text\. Letat∈\(0,1\)a\_\{t\}\\in\(0,1\)denote the one\-step probability of retaining the current token in the forward kernel\.

Masking diffusion\.For the absorbing kernel, observing a visible tokenxt=j≠\[𝙼𝙰𝚂𝙺\]x\_\{t\}=j\\neq\\mathtt\{\[MASK\]\}forcesxt−1=jx\_\{t\-1\}=j\. Hence, on the visible\-token subspace,Qtmask\(⋅∣j\)=δjQ\_\{t\}^\{\\mathrm\{mask\}\}\(\\cdot\\mid j\)=\\delta\_\{j\}\. The reverse step acts as the identity on that subspace, so one can choose two visible\-token distributionsμ,ν\\mu,\\nufor which

DKL​\(μ​Qtmask∥ν​Qtmask\)=DKL​\(μ∥ν\)\.D\_\{\\mathrm\{KL\}\}\(\\mu Q\_\{t\}^\{\\mathrm\{mask\}\}\\\|\\nu Q\_\{t\}^\{\\mathrm\{mask\}\}\)=D\_\{\\mathrm\{KL\}\}\(\\mu\\\|\\nu\)\.Thereforeηt,mask=1\\eta\_\{t,\\mathrm\{mask\}\}=1\. This reflects the “early commitment” effect: once a wrong visible token appears, the standard masking reverse kernel has no mechanism to repair it\.

Remasking\.If a remasking sampler has the effective kernelQtrm=\(1−rt\)​Qtmask\+rt​QtrefQ\_\{t\}^\{\\mathrm\{rm\}\}=\(1\-r\_\{t\}\)Q\_\{t\}^\{\\mathrm\{mask\}\}\+r\_\{t\}Q\_\{t\}^\{\\mathrm\{ref\}\}, and the refresh part satisfies

Qtref\(⋅∣x\)≥λtrefνt\(⋅\),∀x,Q\_\{t\}^\{\\mathrm\{ref\}\}\(\\cdot\\mid x\)\\geq\\lambda\_\{t\}^\{\\mathrm\{ref\}\}\\nu\_\{t\}\(\\cdot\),\\qquad\\forall x,thenQtrm\(⋅∣x\)≥rtλtrefνt\(⋅\)Q\_\{t\}^\{\\mathrm\{rm\}\}\(\\cdot\\mid x\)\\geq r\_\{t\}\\lambda\_\{t\}^\{\\mathrm\{ref\}\}\\nu\_\{t\}\(\\cdot\)\. Henceηt,rm≤1−rt​λtref\\eta\_\{t,\\mathrm\{rm\}\}\\leq 1\-r\_\{t\}\\lambda\_\{t\}^\{\\mathrm\{ref\}\}\. Remasking therefore helps precisely through the amount of shared refresh it injects\.

Uniform diffusion\.For a one\-step uniform forward kernel on a vocabulary of sizeVV,q​\(xt=j∣xt−1=y\)=at​δj​y\+V−1​\(1−at\)\.q\(x\_\{t\}=j\\mid x\_\{t\-1\}=y\)=a\_\{t\}\\delta\_\{jy\}\+V^\{\-1\}\(1\-a\_\{t\}\)\.Bayes’ rule gives

Qtuni​\(y∣j\)=\(at​δj​y\+1−atV\)​qt−1​\(y\)qt​\(j\)≥1−atV​qt​\(j\)​qt−1​\(y\)\.Q\_\{t\}^\{\\mathrm\{uni\}\}\(y\\mid j\)=\\frac\{\\left\(a\_\{t\}\\delta\_\{jy\}\+\\frac\{1\-a\_\{t\}\}\{V\}\\right\)q\_\{t\-1\}\(y\)\}\{q\_\{t\}\(j\)\}\\geq\\frac\{1\-a\_\{t\}\}\{Vq\_\{t\}\(j\)\}q\_\{t\-1\}\(y\)\.ThusQtuniQ\_\{t\}^\{\\mathrm\{uni\}\}has the shared componentqt−1q\_\{t\-1\}with mass

λtuni:=minj⁡1−atV​qt​\(j\),\\lambda\_\{t\}^\{\\mathrm\{uni\}\}:=\\min\_\{j\}\\frac\{1\-a\_\{t\}\}\{Vq\_\{t\}\(j\)\},assumingqt​\(j\)\>0q\_\{t\}\(j\)\>0\. By[Eq\.˜71](https://arxiv.org/html/2606.15327#A2.E71),

ηt,uni≤1−λtuni<1\.\\eta\_\{t,\\mathrm\{uni\}\}\\leq 1\-\\lambda\_\{t\}^\{\\mathrm\{uni\}\}<1\.\(72\)Uniform diffusion can therefore contract rollout mismatch, which is the sampling\-side repair advantage emphasized in the main text\.

Semantic diffusion and SemDLM\+\.For a semantic forward kernel

q​\(xt=j∣xt−1=y\)=at​δj​y\+\(1−at\)​stsem​\(j∣y\),q\(x\_\{t\}=j\\mid x\_\{t\-1\}=y\)=a\_\{t\}\\delta\_\{jy\}\+\(1\-a\_\{t\}\)s\_\{t\}^\{\\mathrm\{sem\}\}\(j\\mid y\),similarly, if the semantic diffusion is equiped with a global transitionνt\\nu\_\{t\}as a shared component:

stsem​\(j∣y\)≥αtsem​νt​\(j\),∀y,j\.s\_\{t\}^\{\\mathrm\{sem\}\}\(j\\mid y\)\\geq\\alpha\_\{t\}^\{\\mathrm\{sem\}\}\\nu\_\{t\}\(j\),\\qquad\\forall y,j\.Then

Qtsem​\(y∣j\)≥\(1−at\)​αtsem​νt​\(j\)qt​\(j\)​qt−1​\(y\),Q\_\{t\}^\{\\mathrm\{sem\}\}\(y\\mid j\)\\geq\\frac\{\(1\-a\_\{t\}\)\\alpha\_\{t\}^\{\\mathrm\{sem\}\}\\nu\_\{t\}\(j\)\}\{q\_\{t\}\(j\)\}q\_\{t\-1\}\(y\),so

ηt,sem≤1−λtsem<1,λtsem:=minj⁡\(1−at\)​αtsem​νt​\(j\)qt​\(j\)\.\\eta\_\{t,\\mathrm\{sem\}\}\\leq 1\-\\lambda\_\{t\}^\{\\mathrm\{sem\}\}<1,\\qquad\\lambda\_\{t\}^\{\\mathrm\{sem\}\}:=\\min\_\{j\}\\frac\{\(1\-a\_\{t\}\)\\alpha\_\{t\}^\{\\mathrm\{sem\}\}\\nu\_\{t\}\(j\)\}\{q\_\{t\}\(j\)\}\.\(73\)This serves another important factor that we need global transition in[Eq\.˜22](https://arxiv.org/html/2606.15327#S4.E22)\.

##### Cumulated Exposure Bias with Propagation\.

We then extend the one\-step analysis to the overall analysis of exposure bias\.

Assumeρt≤ρ\\rho\_\{t\}\\leq\\rhofor allttandηt≤η<1\\eta\_\{t\}\\leq\\eta<1\(this is the case for semantic and uniform diffusion\), then iteratingℬt−1≤η​ℬt\+ρ\{\\mathcal\{B\}\}\_\{t\-1\}\\leq\\eta\{\\mathcal\{B\}\}\_\{t\}\+\\rhofrom timet=\[0:T\]t=\[0:T\]gives

ℬt≤ηT−t​ℬT\+1−ηT−t1−η​ρ\.\{\\mathcal\{B\}\}\_\{t\}\\leq\\eta^\{T\-t\}\{\\mathcal\{B\}\}\_\{T\}\+\\frac\{1\-\\eta^\{T\-t\}\}\{1\-\\eta\}\\rho\.\(74\)Summing over all the time stepst=0,…,Tt=0,\\ldots,Tyields

∑t=0Tℬt≤ℬT1−η\+T​ρ1−η\.\\sum\_\{t=0\}^\{T\}\{\\mathcal\{B\}\}\_\{t\}\\leq\\frac\{\{\\mathcal\{B\}\}\_\{T\}\}\{1\-\\eta\}\+\\frac\{T\\rho\}\{1\-\\eta\}\.\(75\)IfqT=pTq\_\{T\}=p\_\{T\}, we getℬT=0\{\\mathcal\{B\}\}\_\{T\}=0, and therefore

∑t=0Tℬt=𝒪​\(T​ρ1−η\)\.\\sum\_\{t=0\}^\{T\}\{\\mathcal\{B\}\}\_\{t\}=\\mathcal\{O\}\\\!\\left\(\\frac\{T\\rho\}\{1\-\\eta\}\\right\)\.
For masking diffusion,ηt=1\\eta\_\{t\}=1\. The same recursion becomesℬt−1≤ℬt\+ρ\{\\mathcal\{B\}\}\_\{t\-1\}\\leq\{\\mathcal\{B\}\}\_\{t\}\+\\rho, so

ℬt≤ℬT\+\(T−t\)​ρ\{\\mathcal\{B\}\}\_\{t\}\\leq\{\\mathcal\{B\}\}\_\{T\}\+\(T\-t\)\\rhoand hence

∑t=0Tℬt≤\(T\+1\)​ℬT\+T​\(T\+1\)2​ρ\.\\sum\_\{t=0\}^\{T\}\{\\mathcal\{B\}\}\_\{t\}\\leq\(T\+1\)\{\\mathcal\{B\}\}\_\{T\}\+\\frac\{T\(T\+1\)\}\{2\}\\rho\.\(76\)WhenqT=pTq\_\{T\}=p\_\{T\}, this reduces to

∑t=0Tℬt=𝒪​\(T2​ρ\)\.\\sum\_\{t=0\}^\{T\}\{\\mathcal\{B\}\}\_\{t\}=\\mathcal\{O\}\(T^\{2\}\\rho\)\.This proves the comparison in Proposition[4](https://arxiv.org/html/2606.15327#Thmprop4): kernels with shared support, such as uniform diffusion and SemDLM\+ with global jumping, contract exposure bias across steps, while masking can accumulate it quadratically in the number of reverse steps\.

### B\.6Transition Kernel Dispersion and Variance

#### B\.6\.1Unifying Variance in sampling and approximation

We first unify the Training and Sampling Variance\.

𝒱​\(xt\)\\displaystyle\{\\mathcal\{V\}\}\(x\_\{t\}\):=𝔼q​\(x0∣xt\)​\[log⁡p¯​\(x0∣xt\)−𝔼S​log⁡p^S​\(x0∣xt\)\],\\displaystyle:=\\mathbb\{E\}\_\{q\(x\_\{0\}\\mid x\_\{t\}\)\}\\Big\[\\log\\bar\{p\}\(x\_\{0\}\\mid x\_\{t\}\)\-\\mathbb\{E\}\_\{S\}\\log\\hat\{p\}\_\{S\}\(x\_\{0\}\\mid x\_\{t\}\)\\Big\],\(77\)Per step sampling variance:=𝔼q​\(xt\)​\[𝒱​\(xt\)\]=𝔼q​\(xt,x0\)​\[log⁡p¯​\(x0∣xt\)−𝔼S​log⁡p^S​\(x0∣xt\)\],\\displaystyle:=\\mathbb\{E\}\_\{q\(x\_\{t\}\)\}\\big\[\{\\mathcal\{V\}\}\(x\_\{t\}\)\\big\]=\\mathbb\{E\}\_\{q\(x\_\{t\},x\_\{0\}\)\}\\Big\[\\log\\bar\{p\}\(x\_\{0\}\\mid x\_\{t\}\)\-\\mathbb\{E\}\_\{S\}\\log\\hat\{p\}\_\{S\}\(x\_\{0\}\\mid x\_\{t\}\)\\Big\],\(78\)Per\-step Training variance:=𝔼q​\(xt\)​\[log⁡p¯​\(xt\)−𝔼S​log⁡p^S​\(xt\)\]\.\\displaystyle:=\\mathbb\{E\}\_\{q\(x\_\{t\}\)\}\\Big\[\\log\\bar\{p\}\(x\_\{t\}\)\-\\mathbb\{E\}\_\{S\}\\log\\hat\{p\}\_\{S\}\(x\_\{t\}\)\\Big\]\.\(79\)
For any fixedxtx\_\{t\}\(and any fixedSS\), suppose

p¯​\(xt\)=∫p¯​\(x0∣xt\)​p¯​\(x0\)​𝑑x0,p^S​\(xt\)=∫p^S​\(x0∣xt\)​p^S​\(x0\)​𝑑x0\.\\bar\{p\}\(x\_\{t\}\)=\\int\\bar\{p\}\(x\_\{0\}\\mid x\_\{t\}\)\\bar\{p\}\(x\_\{0\}\)dx\_\{0\},\\hat\{p\}\_\{S\}\(x\_\{t\}\)=\\int\\hat\{p\}\_\{S\}\(x\_\{0\}\\mid x\_\{t\}\)\\hat\{p\}\_\{S\}\(x\_\{0\}\)dx\_\{0\}\.\(80\)Define the \(normalized\) reference measure

rp¯​\(x0∣xt\):=p¯​\(x0∣xt\)​p¯​\(x0\)p¯​\(xt\)\.r\_\{\\bar\{p\}\}\(x\_\{0\}\\mid x\_\{t\}\):=\\frac\{\\bar\{p\}\(x\_\{0\}\\mid x\_\{t\}\)\\bar\{p\}\(x\_\{0\}\)\}\{\\bar\{p\}\(x\_\{t\}\)\}\.\(81\)Then

p^S​\(xt\)p¯​\(xt\)=𝔼x0∼rp¯\(⋅∣xt\)​\[p^S​\(x0∣xt\)​p^S​\(x0\)p¯​\(x0∣xt\)​p¯​\(x0\)\]\.\\frac\{\\hat\{p\}\_\{S\}\(x\_\{t\}\)\}\{\\bar\{p\}\(x\_\{t\}\)\}=\\mathbb\{E\}\_\{x\_\{0\}\\sim r\_\{\\bar\{p\}\}\(\\cdot\\mid x\_\{t\}\)\}\\left\[\\frac\{\\hat\{p\}\_\{S\}\(x\_\{0\}\\mid x\_\{t\}\)\\hat\{p\}\_\{S\}\(x\_\{0\}\)\}\{\\bar\{p\}\(x\_\{0\}\\mid x\_\{t\}\)\\bar\{p\}\(x\_\{0\}\)\}\\right\]\.\(82\)By Jensen’s inequality \(concavity oflog\\log\),

log⁡p^S​\(xt\)p¯​\(xt\)=log⁡𝔼rp¯​\[a​\(x0\)\]≥𝔼rp¯​\[log⁡a​\(x0\)\],\\log\\frac\{\\hat\{p\}\_\{S\}\(x\_\{t\}\)\}\{\\bar\{p\}\(x\_\{t\}\)\}=\\log\\mathbb\{E\}\_\{r\_\{\\bar\{p\}\}\}\[a\(x\_\{0\}\)\]\\geq\\mathbb\{E\}\_\{r\_\{\\bar\{p\}\}\}\[\\log a\(x\_\{0\}\)\],\(83\)where

a​\(x0\):=p^S​\(x0∣xt\)​p^S​\(x0\)p¯​\(x0∣xt\)​p¯​\(x0\)\.a\(x\_\{0\}\):=\\frac\{\\hat\{p\}\_\{S\}\(x\_\{0\}\\mid x\_\{t\}\)\\hat\{p\}\_\{S\}\(x\_\{0\}\)\}\{\\bar\{p\}\(x\_\{0\}\\mid x\_\{t\}\)\\bar\{p\}\(x\_\{0\}\)\}\.\(84\)Equivalently, for eachxtx\_\{t\},

log⁡p¯​\(xt\)−log⁡p^S​\(xt\)\\displaystyle\\log\\bar\{p\}\(x\_\{t\}\)\-\\log\\hat\{p\}\_\{S\}\(x\_\{t\}\)≤𝔼x0∼rp¯\(⋅∣xt\)​\[log⁡p¯​\(x0∣xt\)−log⁡p^S​\(x0∣xt\)\+log⁡p¯​\(x0\)−log⁡p^S​\(x0\)\]\.\\displaystyle\\leq\\mathbb\{E\}\_\{x\_\{0\}\\sim r\_\{\\bar\{p\}\}\(\\cdot\\mid x\_\{t\}\)\}\\Big\[\\log\\bar\{p\}\(x\_\{0\}\\mid x\_\{t\}\)\-\\log\\hat\{p\}\_\{S\}\(x\_\{0\}\\mid x\_\{t\}\)\+\\log\\bar\{p\}\(x\_\{0\}\)\-\\log\\hat\{p\}\_\{S\}\(x\_\{0\}\)\\Big\]\.\(85\)Taking𝔼q​\(xt\)\\mathbb\{E\}\_\{q\(x\_\{t\}\)\}and then𝔼S\\mathbb\{E\}\_\{S\}yields the bound

B\\displaystyle B=𝔼q​\(xt\)​\[log⁡p¯​\(xt\)−𝔼S​log⁡p^S​\(xt\)\]\\displaystyle=\\mathbb\{E\}\_\{q\(x\_\{t\}\)\}\\Big\[\\log\\bar\{p\}\(x\_\{t\}\)\-\\mathbb\{E\}\_\{S\}\\log\\hat\{p\}\_\{S\}\(x\_\{t\}\)\\Big\]≤𝔼q​\(xt\)​𝔼S​𝔼x0∼rp¯\(⋅∣xt\)​\[log⁡p¯​\(x0∣xt\)−log⁡p^S​\(x0∣xt\)\+log⁡p¯​\(x0\)−log⁡p^S​\(x0\)\]\.\\displaystyle\\leq\\mathbb\{E\}\_\{q\(x\_\{t\}\)\}\\mathbb\{E\}\_\{S\}\\mathbb\{E\}\_\{x\_\{0\}\\sim r\_\{\\bar\{p\}\}\(\\cdot\\mid x\_\{t\}\)\}\\Big\[\\log\\bar\{p\}\(x\_\{0\}\\mid x\_\{t\}\)\-\\log\\hat\{p\}\_\{S\}\(x\_\{0\}\\mid x\_\{t\}\)\+\\log\\bar\{p\}\(x\_\{0\}\)\-\\log\\hat\{p\}\_\{S\}\(x\_\{0\}\)\\Big\]\.If in addition thex0x\_\{0\}\-marginals match, i\.e\.p^S​\(x0\)=p¯​\(x0\),∀S\\hat\{p\}\_\{S\}\(x\_\{0\}\)=\\bar\{p\}\(x\_\{0\}\),\\forall S, then[sec\.˜B\.6\.1](https://arxiv.org/html/2606.15327#A2.Ex42)reduces to

log⁡p¯​\(xt\)−log⁡p^S​\(xt\)≤𝔼x0∼rp¯\(⋅∣xt\)​\[log⁡p¯​\(x0∣xt\)−log⁡p^S​\(x0∣xt\)\],\\log\\bar\{p\}\(x\_\{t\}\)\-\\log\\hat\{p\}\_\{S\}\(x\_\{t\}\)\\leq\\mathbb\{E\}\_\{x\_\{0\}\\sim r\_\{\\bar\{p\}\}\(\\cdot\\mid x\_\{t\}\)\}\\Big\[\\log\\bar\{p\}\(x\_\{0\}\\mid x\_\{t\}\)\-\\log\\hat\{p\}\_\{S\}\(x\_\{0\}\\mid x\_\{t\}\)\\Big\],\(86\)and hence

B≤𝔼q​\(xt\)​𝔼S​𝔼x0∼rp¯\(⋅∣xt\)​\[log⁡p¯​\(x0∣xt\)−log⁡p^S​\(x0∣xt\)\]\.B\\leq\\mathbb\{E\}\_\{q\(x\_\{t\}\)\}\\mathbb\{E\}\_\{S\}\\mathbb\{E\}\_\{x\_\{0\}\\sim r\_\{\\bar\{p\}\}\(\\cdot\\mid x\_\{t\}\)\}\\Big\[\\log\\bar\{p\}\(x\_\{0\}\\mid x\_\{t\}\)\-\\log\\hat\{p\}\_\{S\}\(x\_\{0\}\\mid x\_\{t\}\)\\Big\]\.\(87\)
If moreoverq​\(x0∣xt\)=p¯​\(x0∣xt\)q\(x\_\{0\}\\mid x\_\{t\}\)=\\bar\{p\}\(x\_\{0\}\\mid x\_\{t\}\)\(so thatrp¯​\(x0∣xt\)=p¯​\(x0∣xt\)r\_\{\\bar\{p\}\}\(x\_\{0\}\\mid x\_\{t\}\)=\\bar\{p\}\(x\_\{0\}\\mid x\_\{t\}\)\), then

Per\-step Sampling Variance≤Per\-step Approximation Variance\.\\boxed\{\\text\{Per\-step Sampling Variance\}\\leq\\text\{Per\-step Approximation Variance\}\.\}\(88\)This concludes that, the per\-step sampling variance is strictly upper\-bounded by the per\-step training variance\. Thus, in what follows, we only analyze the training variance for simplicity\.

#### B\.6\.2The gradient guided variance

To connect the variance term to a more classical estimation\-variance picture, we rewrite the KL divergence as the combination of negative log\-likelihood and entropy\.

ℒ\\displaystyle\{\\mathcal\{L\}\}=−𝔼x∼pdata​\(x\)​\[log⁡pθ​\(x\)\]−Entropy\\displaystyle=\-\\mathbb\{E\}\_\{x\\sim p\_\{\\text\{data\}\}\(x\)\}\\left\[\\log p\_\{\\theta\}\(x\)\\right\]\-\\text\{Entropy\}\(89\)And define the per\-sample lossℓt​\(θ\):=−log⁡pθ​\(x0∣xt\)\\ell\_\{t\}\(\\theta\):=\-\\log p\_\{\\theta\}\(x\_\{0\}\\mid x\_\{t\}\), per\-step risksℒt​\(θ\):=𝔼xt​\[ℓt​\(θ\)\]\{\\mathcal\{L\}\}\_\{t\}\(\\theta\):=\\mathbb\{E\}\_\{x\_\{t\}\}\[\\ell\_\{t\}\(\\theta\)\]and use∇θ\\nabla\_\{\\theta\}as the gradient operator\. Letθ⋆\\theta^\{\\star\}minimize the population riskℒ​\(θ\)=𝔼t​\[ℒt​\(θ\)\]\{\\mathcal\{L\}\}\(\\theta\)=\\mathbb\{E\}\_\{t\}\[\\mathcal\{L\}\_\{t\}\(\\theta\)\], so that𝔼t​\[gt​\(θ⋆\)\]=0\\mathbb\{E\}\_\{t\}\[g\_\{t\}\(\\theta^\{\\star\}\)\]=0\. Atθ⋆\\theta^\{\\star\}, the law of total variance yields

Vart,xt​\(∇ℓt​\(θ∗\)\)\\displaystyle\\mathrm\{Var\}\_\{t,x\_\{t\}\}\\big\(\\nabla\\ell\_\{t\}\(\\theta^\{\*\}\)\\big\)\(90\)=\\displaystyle=𝔼t​\[Varxt​\(∇ℓt​\(θ∗\)\)\]\+Vart​\(𝔼xt​\[∇ℓt​\(θ∗\)\]\),\\displaystyle\\mathbb\{E\}\_\{t\}\\left\[\\mathrm\{Var\}\_\{x\_\{t\}\}\\big\(\\nabla\\ell\_\{t\}\(\\theta^\{\*\}\)\\big\)\\right\]\+\\mathrm\{Var\}\_\{t\}\\left\(\\mathbb\{E\}\_\{x\_\{t\}\}\\left\[\\nabla\\ell\_\{t\}\(\\theta^\{\*\}\)\\right\]\\right\),=\\displaystyle=𝔼t​\[Varxt​\(∇ℓt​\(θ∗\)\)\]⏟within\-step noise\+Vart​\(∇ℒt​\(θ⋆\)\)⏟between\-step heterogeneity\.\\displaystyle\\underbrace\{\\mathbb\{E\}\_\{t\}\\left\[\\mathrm\{Var\}\_\{x\_\{t\}\}\\big\(\\nabla\\ell\_\{t\}\(\\theta^\{\*\}\)\\big\)\\right\]\}\_\{\\text\{within\-step noise\}\}\+\\underbrace\{\\mathrm\{Var\}\_\{t\}\\left\(\\nabla\{\\mathcal\{L\}\}\_\{t\}\(\\theta^\{\\star\}\)\\right\)\}\_\{\\text\{between\-step heterogeneity\}\}\.The second termVart​\(∇ℒt​\(θ⋆\)\)\\mathrm\{Var\}\_\{t\}\\left\(\\nabla\{\\mathcal\{L\}\}\_\{t\}\(\\theta^\{\\star\}\)\\right\)is a*kernel\-induced heterogeneity*\. It quantifies how the gradient varies across different diffusion stepstt\.

Remark\.Crucially,∇ℒt​\(θ⋆\)\\nabla\{\\mathcal\{L\}\}\_\{t\}\(\\theta^\{\\star\}\)depends onQtQ\_\{t\}via samplingxtx\_\{t\}, hence a larger dispersion ofQtQ\_\{t\}will induce larger between\-t heterogeneity\. Other than the theoretical results,[sec\.˜5\.1](https://arxiv.org/html/2606.15327#S5.SS1.SSS0.Px1)empirically valid that the gradient variance is strongly correlated with the dispersion of the transition kernel across time t\.

Now we aim to rigorously derive that as DispersionQtQ\_\{t\}increases, Between\-ttHeterogeneity increases\.

We define the between\-ttheterogeneity as:

𝒱​\(Q\)≔Vart​\(∇ℒt​\(θ⋆;Q\)\)=𝔼t​\[‖∇ℒt​\(θ⋆;Q\)‖2\],\{\\mathcal\{V\}\}\(Q\)\\coloneqq\\mathrm\{Var\}\_\{t\}\\big\(\\nabla\{\\mathcal\{L\}\}\_\{t\}\(\\theta^\{\\star\};Q\)\\big\)=\\mathbb\{E\}\_\{t\}\\big\[\\\|\\nabla\{\\mathcal\{L\}\}\_\{t\}\(\\theta^\{\\star\};Q\)\\\|^\{2\}\\big\],\(91\)where the equality holds because𝔼t​\[∇ℒt​\(θ⋆;Q\)\]=𝟎\\mathbb\{E\}\_\{t\}\[\\nabla\{\\mathcal\{L\}\}\_\{t\}\(\\theta^\{\\star\};Q\)\]=\\mathbf\{0\}\.

We define the posterior entropy ash​\(t;Q\)≔𝔼xt∼qt​\(⋅;Qt\)​\[H​\(q​\(x0∣xt;Qt\)\)\]h\(t;Q\)\\coloneqq\\mathbb\{E\}\_\{x\_\{t\}\\sim q\_\{t\}\(\\cdot;Q\_\{t\}\)\}\\big\[H\\big\(q\(x\_\{0\}\\mid x\_\{t\};Q\_\{t\}\)\\big\)\\big\], whereH​\(⋅\)H\(\\cdot\)is Shannon entropy\. Then we define the kernel dispersion as𝒟​\(Q\)≔Vart​\(h​\(t;Q\)\)=𝔼t​\[h​\(t;Q\)2\]−\(𝔼t​\[h​\(t;Q\)\]\)2\.\\mathcal\{D\}\(Q\)\\coloneqq\\mathrm\{Var\}\_\{t\}\\big\(h\(t;Q\)\\big\)=\\mathbb\{E\}\_\{t\}\\big\[h\(t;Q\)^\{2\}\\big\]\-\\big\(\\mathbb\{E\}\_\{t\}\[h\(t;Q\)\]\\big\)^\{2\}\.This quantifies how posterior uncertainty fluctuates across diffusion steps\. Uniform diffusion yields large𝒟​\(Q\)\\mathcal\{D\}\(Q\)\(entropy spans\[0,log⁡\|𝒱\|\]\[0,\\log\|\{\\mathcal\{V\}\}\|\]\); semantic diffusion yields small𝒟​\(Q\)\\mathcal\{D\}\(Q\)\(entropy concentrated nearlog⁡k\\log k\), and Masking diffusion has the lowest dispersion\.

We make the following assumptions:

1\) the approximation error scales with posterior entropy, such that there existsc1\>0c\_\{1\}\>0such that for alltt,

𝔼xt\[∥pθ⋆\(⋅∣xt\)−q\(⋅∣xt;Qt\)∥1\]≥c1⋅h\(t;Q\)\.\\mathbb\{E\}\_\{x\_\{t\}\}\\big\[\\\|p\_\{\\theta^\{\\star\}\}\(\\cdot\\mid x\_\{t\}\)\-q\(\\cdot\\mid x\_\{t\};Q\_\{t\}\)\\\|\_\{1\}\\big\]\\geq c\_\{1\}\\cdot h\(t;Q\)\.\(92\)This is induced by the fact that high\-entropy posteriors \(induced by dispersedQtQ\_\{t\}\) are harder to approximate with finite\-capacity models

2\) The gradients are bounded and stable\.‖∇θfθ⋆​\(xt\)‖≤M\\\|\\nabla\_\{\\theta\}f\_\{\\theta^\{\\star\}\}\(x\_\{t\}\)\\\|\\leq Mfor allxtx\_\{t\}, and the direction of∇θfθ⋆​\(xt\)\\nabla\_\{\\theta\}f\_\{\\theta^\{\\star\}\}\(x\_\{t\}\)varies smoothly over semantically coherent neighborhoods\.

###### Proposition 8\. There exists a constantc=c1/M\>0c=c\_\{1\}/M\>0such that:𝒱​\(Q\)≥c2⋅𝒟​\(Q\)\.\{\\mathcal\{V\}\}\(Q\)\\geq c^\{2\}\\cdot\\mathcal\{D\}\(Q\)\.\(93\)Consequently, if two generators satisfy𝒟​\(QA\)\>𝒟​\(QB\)\\mathcal\{D\}\(Q^\{A\}\)\>\\mathcal\{D\}\(Q^\{B\}\), then𝒱​\(QA\)\>𝒱​\(QB\)\{\\mathcal\{V\}\}\(Q^\{A\}\)\>\{\\mathcal\{V\}\}\(Q^\{B\}\)\.

proof:We previously derived the gradient derivation for cross\-entropy loss, and we re\-formula it here\. \(in[Sec\.˜B\.4](https://arxiv.org/html/2606.15327#A2.SS4)\)

∇ℒt​\(θ⋆;Q\)=𝔼xt​\[𝔼x0∣xt​\[\(pθ⋆​\(x0∣xt\)−q​\(x0∣xt;Qt\)\)⊙∇θfθ⋆​\(xt\)\]\]=𝔼xt​\[𝜹t​\(xt;Q\)\],\\displaystyle\\begin\{aligned\} \\nabla\{\\mathcal\{L\}\}\_\{t\}\(\\theta^\{\\star\};Q\)&=\\mathbb\{E\}\_\{x\_\{t\}\}\\Big\[\\mathbb\{E\}\_\{x\_\{0\}\\mid x\_\{t\}\}\\big\[\(p\_\{\\theta^\{\\star\}\}\(x\_\{0\}\\mid x\_\{t\}\)\-q\(x\_\{0\}\\mid x\_\{t\};Q\_\{t\}\)\)\\odot\\nabla\_\{\\theta\}f\_\{\\theta^\{\\star\}\}\(x\_\{t\}\)\\big\]\\Big\]\\\\ &=\\mathbb\{E\}\_\{x\_\{t\}\}\\big\[\\bm\{\\delta\}\_\{t\}\(x\_\{t\};Q\)\\big\],\\end\{aligned\}\(94\)whereℓt​\(xt;Q\)\\ell\_\{t\}\(x\_\{t\};Q\)denotes the per\-sample gradient signal\. By Hölder’s inequality:

‖∇ℒt​\(θ⋆;Q\)‖≥\|𝔼xt​\[‖𝜹t​\(xt;Q\)‖\]\|≥c1⋅𝔼xt​\[H​\(q​\(x0∣xt;Qt\)\)\]⋅1M=c⋅h​\(t;Q\),c≔c1/M\>0\.\\displaystyle\\begin\{aligned\} \\\|\\nabla\{\\mathcal\{L\}\}\_\{t\}\(\\theta^\{\\star\};Q\)\\\|&\\geq\\big\|\\mathbb\{E\}\_\{x\_\{t\}\}\\big\[\\\|\\bm\{\\delta\}\_\{t\}\(x\_\{t\};Q\)\\\|\\big\]\\big\|\\\\ &\\geq c\_\{1\}\\cdot\\mathbb\{E\}\_\{x\_\{t\}\}\\big\[H\(q\(x\_\{0\}\\mid x\_\{t\};Q\_\{t\}\)\)\\big\]\\cdot\\frac\{1\}\{M\}\\\\ &=c\\cdot h\(t;Q\),\\quad c\\coloneqq c\_\{1\}/M\>0\.\\end\{aligned\}\(95\)Squaring both sides and taking expectation overtt:

𝒱​\(Q\)=𝔼t​\[‖∇ℒt​\(θ⋆;Q\)‖2\]≥c2⋅𝔼t​\[h​\(t;Q\)2\]=c2⋅\(Vart​\(h​\(t;Q\)\)\+\(𝔼t​\[h​\(t;Q\)\]\)2\)≥c2⋅Vart​\(h​\(t;Q\)\)=c2⋅𝒟​\(Q\)\.\\displaystyle\\begin\{aligned\} \{\\mathcal\{V\}\}\(Q\)&=\\mathbb\{E\}\_\{t\}\\big\[\\\|\\nabla\{\\mathcal\{L\}\}\_\{t\}\(\\theta^\{\\star\};Q\)\\\|^\{2\}\\big\]\\\\ &\\geq c^\{2\}\\cdot\\mathbb\{E\}\_\{t\}\\big\[h\(t;Q\)^\{2\}\\big\]\\\\ &=c^\{2\}\\cdot\\Big\(\\mathrm\{Var\}\_\{t\}\\big\(h\(t;Q\)\\big\)\+\\big\(\\mathbb\{E\}\_\{t\}\[h\(t;Q\)\]\\big\)^\{2\}\\Big\)\\\\ &\\geq c^\{2\}\\cdot\\mathrm\{Var\}\_\{t\}\\big\(h\(t;Q\)\\big\)\\\\ &=c^\{2\}\\cdot\\mathcal\{D\}\(Q\)\.\\end\{aligned\}\(96\)The directly completes the proof\.

Remark:𝒟​\(Q\)\\mathcal\{D\}\(Q\)explicitly captures howQtQ\_\{t\}’s structural dispersion, such as support size, entropy trajectory, propagates to gradient statistics\. This suggests that minimizing kernel dispersion𝒟​\(Q\)\\mathcal\{D\}\(Q\)during design can reduce between\-ttgradient heterogeneity\. This inspires us to design SemDLM\.

## Appendix CMore about Diffusion as CTMC

In[Sec\.˜2](https://arxiv.org/html/2606.15327#S2), we mentioned that the local transitionqt∣t−d​tq\_\{t\\mid t\-dt\}, the cumulative forwardqt∣0q\_\{t\\mid 0\}, and the generatorQtQ\_\{t\}are equivalent representations of the same forward process\. The equivalence is obtained via Bayesian rule:q​\(xt−d​t∣xt,x0\)∝q​\(xt∣xt−d​t\)​q​\(xt−d​t∣x0\)q\\left\(x\_\{t\-dt\}\\mid x\_\{t\},x\_\{0\}\\right\)\\propto q\\left\(x\_\{t\}\\mid x\_\{t\-dt\}\\right\)q\(x\_\{t\-dt\}\\mid x\_\{0\}\)\.

We thus present other details for the process\. We first detail the forward process and reverse process in CTMC\. The forward process is derived by theinfinitesimal generatorQtQ\_\{t\}, i\.e\.,d​qtd​t=Qt​qt,0≤t≤1\\frac\{dq\_\{t\}\}\{dt\}=Q\_\{t\}q\_\{t\},0\\leq t\\leq 1, where

Forward Process via Euler Sampling:​q​\(xt\+d​t=y∣xt=z\)=δz​y\+Qt​\(z,y\)​d​t\+O​\(d​t2\)\\text\{Forward Process via Euler Sampling: \}q\\left\(x\_\{t\+dt\}=y\\mid x\_\{t\}=z\\right\)=\\delta\_\{zy\}\+Q\_\{t\}\(z,y\)dt\+O\\left\(dt^\{2\}\\right\)\(97\)and the corresponding reverse process is

Reverse\-time Euler Step:​q​\(xt−d​t=z∣xt=y\)=δy​z\+Q¯t​\(y,z\)​d​t\+O​\(d​t2\),\\text\{Reverse\-time Euler Step: \}q\(x\_\{t\-dt\}=z\\mid x\_\{t\}=y\)=\\delta\_\{yz\}\+\\bar\{Q\}\_\{t\}\(y,z\)dt\+O\(dt^\{2\}\),With the reverse\-time infinitesimal generator

Q¯t​\(y,z\)=Qt​\(z,y\)​qt​\(z\)qt​\(y\),z≠y,Q¯t​\(y,y\)=−∑z≠yQ¯t​\(y,z\)\.\\bar\{Q\}\_\{t\}\(y,z\)=Q\_\{t\}\(z,y\)\\frac\{q\_\{t\}\(z\)\}\{q\_\{t\}\(y\)\},\\quad z\\neq y,\\qquad\\bar\{Q\}\_\{t\}\(y,y\)=\-\\sum\_\{z\\neq y\}\\bar\{Q\}\_\{t\}\(y,z\)\.where the density ratioqt​\(z\)/qt​\(y\)q\_\{t\}\(z\)/q\_\{t\}\(y\)is typically approximated by a learned score or posterior model in practice\.

Thus, it is clear that the local transitionqt∣t−d​tq\_\{t\\mid t\-dt\}, the cumulative forwardqt∣0q\_\{t\\mid 0\}, and the generatorQtQ\_\{t\}are equivalent representations of the same forward/reverse process\. For completeness, we list the other representation of masking, uniform and semantic diffusion\.

Absorbing \(Mask\) Transitiondefines an absorbing token\[𝙼𝙰𝚂𝙺\]\\mathtt\{\[MASK\]\}to makeQt​\(\[𝙼𝙰𝚂𝙺\],y\)=0Q\_\{t\}\(\\mathtt\{\[MASK\]\},y\)=0and

Qt​\(z,y\)=\{λ​\(t\),z≠\[𝙼𝙰𝚂𝙺\],y=\[𝙼𝙰𝚂𝙺\],−λ​\(t\),z≠\[𝙼𝙰𝚂𝙺\],y=z\.Q\_\{t\}\(z,y\)=\\begin\{cases\}\\lambda\(t\),&z\\neq\\mathtt\{\[MASK\]\},y=\\mathtt\{\[MASK\]\},\\\\ \-\\lambda\(t\),&z\\neq\\mathtt\{\[MASK\]\},y=z\.\\end\{cases\}\(98\)whereλ​\(t\)\\lambda\(t\)is the time schedule\. This induces a cumulated kernel directly fromx0x\_\{0\}toxtx\_\{t\}:

q​\(xt=j∣x0=i\)=αt​δi​j\+\(1−αt\)​δj,\[𝙼𝙰𝚂𝙺\],q\(x\_\{t\}=j\\mid x\_\{0\}=i\)=\\alpha\_\{t\}\\delta\_\{ij\}\+\(1\-\\alpha\_\{t\}\)\\delta\_\{j,\\mathtt\{\[MASK\]\}\},\(99\)whereαt=exp⁡\(−∫0tλ​\(s\)​𝑑s\)\\alpha\_\{t\}=\\exp\\\!\\left\(\-\\int\_\{0\}^\{t\}\\lambda\(s\)ds\\right\)andδi​j\\delta\_\{ij\}is the Kronecker delta\.

Uniform Transitionis another choice that has been shown to have favorable scaling behavior\(vonruette2025scalingbehaviordiscretediffusion\):

Qt​\(z,y\)=\{λ​\(t\)K,y≠z,−λ​\(t\)​\(1−1K\),y=z\.Q\_\{t\}\(z,y\)=\\begin\{cases\}\\frac\{\\lambda\(t\)\}\{K\},&y\\neq z,\\\\ \-\\lambda\(t\)\\big\(1\-\\frac\{1\}\{K\}\\big\),&y=z\.\\end\{cases\}\(100\)which has the following cumulated kernel:

q​\(xt=j∣x0=i\)=αt​δi​j\+1−αtK,∀i,j∈𝒱,q\(x\_\{t\}=j\\mid x\_\{0\}=i\)=\\alpha\_\{t\}\\delta\_\{ij\}\+\\frac\{1\-\\alpha\_\{t\}\}\{K\},\\quad\\forall i,j\\in\\mathcal\{V\},\(101\)with the same decay factorαt=exp⁡\(−∫0tλ​\(s\)​𝑑s\)\\alpha\_\{t\}=\\exp\\\!\\left\(\-\\int\_\{0\}^\{t\}\\lambda\(s\)ds\\right\)\.

Semantic TransitionLetKNN​\(z\)\\mathrm\{KNN\}\(z\)be thekknearest neighbors ofzzandw​\(z,y\)≥0w\(z,y\)\\geq 0be edge weights, the transition kernel is then,

Qt​\(z,y\)=\{λ​\(t\)​w​\(z,y\)∑y′∈KNN​\(z\)w​\(z,y′\),y∈KNN​\(z\),−λ​\(t\),y=z,0,otherwise\.Q\_\{t\}\(z,y\)=\\begin\{cases\}\\lambda\(t\)\\frac\{w\(z,y\)\}\{\\sum\_\{y^\{\\prime\}\\in\\mathrm\{KNN\}\(z\)\}w\(z,y^\{\\prime\}\)\},&y\\in\\mathrm\{KNN\}\(z\),\\\\ \-\\lambda\(t\),&y=z,\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}\(102\)This gives a forward kernel as,

q​\(xt=j∣x0=i\)=αt​δi​j\+1−αtkt​𝕀​\(j∈𝒩k​\(i\)\),q\(x\_\{t\}=j\\mid x\_\{0\}=i\)=\\alpha\_\{t\}\\delta\_\{ij\}\+\\frac\{1\-\\alpha\_\{t\}\}\{k\_\{t\}\}\\mathbb\{I\}\\bigl\(j\\in\\mathcal\{N\}\_\{k\}\(i\)\\bigr\),\(103\)where𝒩k​\(i\)\\mathcal\{N\}\_\{k\}\(i\)is the top\-kksemantic neighborhood of tokenii\.

## Appendix DAdditional Design for Semantic Diffusion

Algorithm 1Training of SemDLM1:Dataset

𝒟∼p0\\mathcal\{D\}\\sim p\_\{0\}
2:Trained model

fθ​\(xt,t\)f\_\{\\theta\}\(x\_\{t\},t\)
3:Initialize parameters

θ\\theta
4:whilenot convergeddo

5:Sample a minibatch

\{x0\(i\)\}i=1B∼𝒟\\\{x\_\{0\}^\{\(i\)\}\\\}\_\{i=1\}^\{B\}\\sim\\mathcal\{D\}
6:Sample

t∼𝒰​\(0,1\)t\\sim\\mathcal\{U\}\(0,1\)
7:Sample noisy states

xt\(i\)∼p​\(xt∣x0\(i\)\)x\_\{t\}^\{\(i\)\}\\sim p\(x\_\{t\}\\mid x\_\{0\}^\{\(i\)\}\)via Eq\. \([22](https://arxiv.org/html/2606.15327#S4.E22)\)

8:Compute

p0∣tθ\(⋅∣xt\(i\)\)=fθ\(xt\(i\),t\)p\_\{0\\mid t\}^\{\\theta\}\(\\cdot\\mid x\_\{t\}^\{\(i\)\}\)=f\_\{\\theta\}\(x\_\{t\}^\{\(i\)\},t\)
9:Update

θ\\thetaby minimizing Eq\. \([3](https://arxiv.org/html/2606.15327#S2.E3)\)

10:endwhile

Algorithm 2Sampling from SemDLM1:Reference distribution

p1p\_\{1\}, trained model

fθ​\(xt,t\)f\_\{\\theta\}\(x\_\{t\},t\), step size

Δ​t\\Delta t
2:Generated sample

x^0\\hat\{x\}\_\{0\}
3:Sample initial state

x^1∼p1\\hat\{x\}\_\{1\}\\sim p\_\{1\}
4:for

t=1,1−Δ​t,…,Δ​tt=1,1\-\\Delta t,\\dots,\\Delta tdo

5:Compute

p0∣tθ\(⋅∣x^t\)=fθ\(x^t,t\)p\_\{0\\mid t\}^\{\\theta\}\(\\cdot\\mid\\hat\{x\}\_\{t\}\)=f\_\{\\theta\}\(\\hat\{x\}\_\{t\},t\)
6:Sample

x~0∼p0∣tθ\(⋅∣x^t\)\\tilde\{x\}\_\{0\}\\sim p\_\{0\\mid t\}^\{\\theta\}\(\\cdot\\mid\\hat\{x\}\_\{t\}\)
7:Compute generator

Qθ​\(x^t∣x~0\)Q\_\{\\theta\}\(\\hat\{x\}\_\{t\}\\mid\\tilde\{x\}\_\{0\}\)
8:Sample

x^t−Δ​t∼x^t\+Qθ​\(x^t∣x~0\)​Δ​t\\hat\{x\}\_\{t\-\\Delta t\}\\sim\\hat\{x\}\_\{t\}\+Q\_\{\\theta\}\(\\hat\{x\}\_\{t\}\\mid\\tilde\{x\}\_\{0\}\)\\,\\Delta t
9:endfor

10:return

x^0\\hat\{x\}\_\{0\}

### D\.1Analysis of Semantic Basins

##### Posterior logit decomposition\.

Again, for a fixed positionii, let the current sampling state bex^t\\hat\{x\}\_\{t\}, and writex^ti=j\\hat\{x\}\_\{t\}^\{i\}=j\. The ideal denoising posterior satisfies

q​\(x0i=k∣x^t\)∝pdata​\(x0i=k∣x^t−i\)​qt​\(j∣x0i=k,x^t−i\)\.q\(x\_\{0\}^\{i\}=k\\mid\\hat\{x\}\_\{t\}\)\\propto p\_\{\\mathrm\{data\}\}\(x\_\{0\}^\{i\}=k\\mid\\hat\{x\}\_\{t\}^\{\-i\}\)q\_\{t\}\(j\\mid x\_\{0\}^\{i\}=k,\\hat\{x\}\_\{t\}^\{\-i\}\)\.\(104\)Thus, up to a normalization constant, the ideal logitsli⋆​\(k;x^t\)l\_\{i\}^\{\\star\}\(k;\\hat\{x\}\_\{t\}\)

li⋆​\(k;x^t\)=log⁡pdata​\(x0i=k∣x^t−i\)⏟contextual prior\+log⁡qt​\(x^ti∣x0i=k,x^t−i\)⏟local forward likelihood\+const\.l\_\{i\}^\{\\star\}\(k;\\hat\{x\}\_\{t\}\)=\\underbrace\{\\log p\_\{\\mathrm\{data\}\}\(x\_\{0\}^\{i\}=k\\mid\\hat\{x\}\_\{t\}^\{\-i\}\)\}\_\{\\text\{contextual prior\}\}\+\\underbrace\{\\log q\_\{t\}\(\\hat\{x\}\_\{t\}^\{i\}\\mid x\_\{0\}^\{i\}=k,\\hat\{x\}\_\{t\}^\{\-i\}\)\}\_\{\\text\{local forward likelihood\}\}\+\\mathrm\{const\}\.\(105\)The first term is the contextual prior, while the second term is the local forward likelihood\. During sampling, the contextx^t−i\\hat\{x\}\_\{t\}^\{\-i\}is generated by the model itself\. Hence the actual model logit can be written as

lθ,i​\(k;x^t\)=li⋆​\(k;x^t\)\+Δroll,i​\(k;x^t\)\+ϵi​\(k\),l\_\{\\theta,i\}\(k;\\hat\{x\}\_\{t\}\)=l\_\{i\}^\{\\star\}\(k;\\hat\{x\}\_\{t\}\)\+\\Delta\_\{\\mathrm\{roll\},i\}\(k;\\hat\{x\}\_\{t\}\)\+\\epsilon\_\{i\}\(k\),\(106\)whereΔroll,i\\Delta\_\{\\mathrm\{roll\},i\}denotes rollout\-induced logit bias\. The rollout bias can be locally approximated as

Δroll,i​\(k;x^t\)≈∑l∈VA​\(k,l\)​ni\(W\)​\(l\),with​A​\(k,l\)\>0\.\\Delta\_\{\\mathrm\{roll\},i\}\(k;\\hat\{x\}\_\{t\}\)\\approx\\sum\_\{l\\in V\}A\(k,l\)n\_\{i\}^\{\(W\)\}\(l\),\\text\{ with \}A\(k,l\)\>0\.

##### Why do semantically similar tokens induce positive contextual contribution?

A​\(k,l\)A\(k,l\)measures contextual association: whether the occurrence of tokenllin the recent context provides positive evidence for predicting tokenkkat positionii\.

At the data\-distribution level, one can define the contextual contribution as a log\-prior shift:

Ai​\(k,l\):=log⁡pdata\(x0i=k∣xtr=l,rest\)pdata\(x0i=k∣xtr=neutral,rest\)\.A\_\{i\}\(k,l\):=\\log\\frac\{p\_\{\\mathrm\{data\}\}\(x\_\{0\}^\{i\}=k\\mid x\_\{t\}^\{r\}=l,\\mathrm\{rest\}\)\}\{p\_\{\\mathrm\{data\}\}\(x\_\{0\}^\{i\}=k\\mid x\_\{t\}^\{r\}=\\mathrm\{neutral\},\\mathrm\{rest\}\)\}\.\(107\)Thus,Ai​\(k,l\)\>0A\_\{i\}\(k,l\)\>0means that observingllin the context increases the conditional prior probability ofkk\. More generally, tokens in the same semantic cluster tend to have positive pointwise contextual association under coherent natural language contexts\.

At the model level, the same effect can be understood through a local linearization of the Transformer logits\. Suppose the sampling logit is

lθ,i​\(k;x^t\)=uk⊤​hi​\(x^t\)\+bk,l\_\{\\theta,i\}\(k;\\hat\{x\}\_\{t\}\)=u\_\{k\}^\{\\top\}h\_\{i\}\(\\hat\{x\}\_\{t\}\)\+b\_\{k\},wherehi​\(x^t\)h\_\{i\}\(\\hat\{x\}\_\{t\}\)is the hidden state at positioniianduku\_\{k\}is the output vector for tokenkk\. Around a neutral baselinex¯t\\bar\{x\}\_\{t\}, the hidden state can be locally approximated as

hi​\(x^t\)≈hi​\(x¯t\)\+∑r∈WBi​r​ex^tr,h\_\{i\}\(\\hat\{x\}\_\{t\}\)\\approx h\_\{i\}\(\\bar\{x\}\_\{t\}\)\+\\sum\_\{r\\in W\}B\_\{ir\}e\_\{\\hat\{x\}\_\{t\}^\{r\}\},whereex^tre\_\{\\hat\{x\}\_\{t\}^\{r\}\}is the embedding of the token at positionrr, andBi​rB\_\{ir\}summarizes the local attention and feed\-forward influence from positionrrto positionii\. Substituting this into the logit gives

lθ,i​\(k;x^t\)−lθ,i​\(k;x¯t\)\\displaystyle l\_\{\\theta,i\}\(k;\\hat\{x\}\_\{t\}\)\-l\_\{\\theta,i\}\(k;\\bar\{x\}\_\{t\}\)≈∑r∈Wuk⊤​Bi​r​ex^tr\.\\displaystyle\\approx\\sum\_\{r\\in W\}u\_\{k\}^\{\\top\}B\_\{ir\}e\_\{\\hat\{x\}\_\{t\}^\{r\}\}\.\(108\)Hence, the contextual contribution of tokenllto the logit ofkkcan be approximated by

Aθ,i​\(k,l\)≈uk⊤​Bi​r​el\.A\_\{\\theta,i\}\(k,l\)\\approx u\_\{k\}^\{\\top\}B\_\{ir\}e\_\{l\}\.\(109\)Ifkkandllare semantically related, their embedding and output directions are often aligned in language models, makingAAcontribution more likely to be positive\.

##### Rollout\-induced positive feedback\.

The rollout\-induced logit bias can be locally approximated as

Δroll,i​\(k;x^t\)≈∑r∈WA​\(k,x^tr\)=∑l∈VA​\(k,l\)​ni\(W\)​\(l\)\.\\Delta\_\{\\mathrm\{roll\},i\}\(k;\\hat\{x\}\_\{t\}\)\\approx\\sum\_\{r\\in W\}A\(k,\\hat\{x\}\_\{t\}^\{r\}\)=\\sum\_\{l\\in V\}A\(k,l\)n\_\{i\}^\{\(W\)\}\(l\)\.\(110\)This approximation states that repeated contextual contributions accumulate in the logit\. IfA​\(k,l\)\>0A\(k,l\)\>0for semantically related tokensk,l∈Ck,l\\in C, then over\-production of clusterCCincreases the logits of tokens inCC\. Thus,

ni\(W\)\(C\)↑⇒Δroll,i\(k\)↑fork∈C⇒pθ\(x0i∈C∣x^t\)↑\.n\_\{i\}^\{\(W\)\}\(C\)\\uparrow\\Rightarrow\\Delta\_\{\\mathrm\{roll\},i\}\(k\)\\uparrow\\text\{ for \}k\\in C\\Rightarrow p\_\{\\theta\}\(x\_\{0\}^\{i\}\\in C\\mid\\hat\{x\}\_\{t\}\)\\uparrow\.This is the rollout\-induced positive feedback that drives semantic basin formation\.

##### Likelihood amplification\.

LetC⊆VC\\subseteq Vbe a semantic cluster and suppose the current tokenj=x^tij=\\hat\{x\}\_\{t\}^\{i\}belongs toCC\. For a semantic kernel, we haveqt​\(j∣k\)​large for​k∈Cq\_\{t\}\(j\\mid k\)\\ \\text\{large for \}k\\in Candqt​\(j∣k\)​small for​k∉Cq\_\{t\}\(j\\mid k\)\\ \\text\{small for \}k\\notin C\. Defineain:=infk∈Cqt​\(j∣k\)a\_\{\\mathrm\{in\}\}:=\\inf\_\{k\\in C\}q\_\{t\}\(j\\mid k\)andaout:=supk∉Cqt​\(j∣k\)\.a\_\{\\mathrm\{out\}\}:=\\sup\_\{k\\notin C\}q\_\{t\}\(j\\mid k\)\.Ifain\>aouta\_\{\\mathrm\{in\}\}\>a\_\{\\mathrm\{out\}\}, then the posterior odds of clusterCCare amplified relative to the contextual prior odds:

q​\(x0i∈C∣x^t\)q​\(x0i∉C∣x^t\)=∑k∈Cpdata​\(k∣x^t−i\)​qt​\(j∣k\)∑k∉Cpdata​\(k∣x^t−i\)​qt​\(j∣k\)≥ainaout⋅pdata​\(x0i∈C∣x^t−i\)pdata​\(x0i∉C∣x^t−i\)\.\\displaystyle\\frac\{q\(x\_\{0\}^\{i\}\\in C\\mid\\hat\{x\}\_\{t\}\)\}\{q\(x\_\{0\}^\{i\}\\notin C\\mid\\hat\{x\}\_\{t\}\)\}=\\frac\{\\sum\_\{k\\in C\}p\_\{\\mathrm\{data\}\}\(k\\mid\\hat\{x\}\_\{t\}^\{\-i\}\)q\_\{t\}\(j\\mid k\)\}\{\\sum\_\{k\\notin C\}p\_\{\\mathrm\{data\}\}\(k\\mid\\hat\{x\}\_\{t\}^\{\-i\}\)q\_\{t\}\(j\\mid k\)\}\\geq\\frac\{a\_\{\\mathrm\{in\}\}\}\{a\_\{\\mathrm\{out\}\}\}\\cdot\\frac\{p\_\{\\mathrm\{data\}\}\(x\_\{0\}^\{i\}\\in C\\mid\\hat\{x\}\_\{t\}^\{\-i\}\)\}\{p\_\{\\mathrm\{data\}\}\(x\_\{0\}^\{i\}\\notin C\\mid\\hat\{x\}\_\{t\}^\{\-i\}\)\}\.\(111\)Thus, the semantic likelihood itself increases the cluster\-level log\-odds\. For a highly local semantic kernel,aouta\_\{\\mathrm\{out\}\}can be very small, so the likelihood amplification can be large\.

Combining the semantic likelihood amplification in[Eq\.˜111](https://arxiv.org/html/2606.15327#A4.E111)and the rollout\-induced feedback in[Eq\.˜110](https://arxiv.org/html/2606.15327#A4.E110), semantic diffusion has two significant sources of cluster amplification, which makes semantic basins more severe in SemDLM\.

### D\.2Global Jumping as Basin Escape

We show why the global jumping kernel can mitigate semantic basin\. Consider a kernel,

qt​\(j∣k,c\)=αt​δk​j\+βt​νt​\(j\)\+\(1−αt−βt\)​stsem​\(j∣k,c\)\.q\_\{t\}\(j\\mid k,c\)=\\alpha\_\{t\}\\delta\_\{kj\}\+\\beta\_\{t\}\\nu\_\{t\}\(j\)\+\(1\-\\alpha\_\{t\}\-\\beta\_\{t\}\)s\_\{t\}^\{\\mathrm\{sem\}\}\(j\\mid k,c\)\.\(112\)The global component givesqt​\(j∣k,c\)≥βt​νt​\(j\)q\_\{t\}\(j\\mid k,c\)\\geq\\beta\_\{t\}\\nu\_\{t\}\(j\)As a result, even ifkkis outside the semantic neighborhood ofjj, it still receives non\-zero likelihood\.

Letj∈Cj\\in C,kin∈Ck\_\{\\mathrm\{in\}\}\\in Candkout∉Ck\_\{\\mathrm\{out\}\}\\notin C\. With global jumping,qt​\(j∣kout,c\)≥βt​νt​\(j\),q\_\{t\}\(j\\mid k\_\{\\mathrm\{out\}\},c\)\\geq\\beta\_\{t\}\\nu\_\{t\}\(j\),and thus the likelihood ratio is bounded by

qt​\(j∣kin,c\)qt​\(j∣kout,c\)≤αt​δkin​j\+βt​νt​\(j\)\+\(1−αt−βt\)​stsem​\(j∣kin,c\)βt​νt​\(j\)\.\\frac\{q\_\{t\}\(j\\mid k\_\{\\mathrm\{in\}\},c\)\}\{q\_\{t\}\(j\\mid k\_\{\\mathrm\{out\}\},c\)\}\\leq\\frac\{\\alpha\_\{t\}\\delta\_\{k\_\{\\mathrm\{in\}\}j\}\+\\beta\_\{t\}\\nu\_\{t\}\(j\)\+\(1\-\\alpha\_\{t\}\-\\beta\_\{t\}\)s\_\{t\}^\{\\mathrm\{sem\}\}\(j\\mid k\_\{\\mathrm\{in\}\},c\)\}\{\\beta\_\{t\}\\nu\_\{t\}\(j\)\}\.\(113\)Therefore, global jumping caps the log\-likelihood advantage of the local semantic cluster, which prevents the positive feedback introduced in[Sec\.˜4\.1](https://arxiv.org/html/2606.15327#S4.SS1)\.

### D\.3Semantic\-Frequency Penalty as Negative Feedback

We rewrite the semantic frequency penalty term:

mi\(W\)​\(k\)=∑l∈VS\+​\(k,l\)​ni\(W\)​\(l\),m\_\{i\}^\{\(W\)\}\(k\)=\\sum\_\{l\\in V\}S\_\{\+\}\(k,l\)n\_\{i\}^\{\(W\)\}\(l\),\(114\)whereS\+​\(k,l\)≥0S\_\{\+\}\(k,l\)\\geq 0measures positive semantic association\. with the semantic frequency penalty, the new logits are

l~θ,i​\(k\)=lθ,i​\(k\)−λfreq​ψ​\(ni\(W\)​\(k\)\)−λsem​ψ​\(mi\(W\)​\(k\)\),ψ​\(u\)=log⁡\(1\+u\)\.\\tilde\{l\}\_\{\\theta,i\}\(k\)=l\_\{\\theta,i\}\(k\)\-\\lambda\_\{\\mathrm\{freq\}\}\\psi\(n\_\{i\}^\{\(W\)\}\(k\)\)\-\\lambda\_\{\\mathrm\{sem\}\}\\psi\(m\_\{i\}^\{\(W\)\}\(k\)\),\\qquad\\psi\(u\)=\\log\(1\+u\)\.\(115\)For two tokenskkandll, the corrected log\-ratio satisfy

log⁡p~θ​\(k\)p~θ​\(l\)\\displaystyle\\log\\frac\{\\tilde\{p\}\_\{\\theta\}\(k\)\}\{\\tilde\{p\}\_\{\\theta\}\(l\)\}=log⁡pθ​\(k\)pθ​\(l\)−λfreq​\[ψ​\(ni\(W\)​\(k\)\)−ψ​\(ni\(W\)​\(l\)\)\]\\displaystyle=\\log\\frac\{p\_\{\\theta\}\(k\)\}\{p\_\{\\theta\}\(l\)\}\-\\lambda\_\{\\mathrm\{freq\}\}\\left\[\\psi\(n\_\{i\}^\{\(W\)\}\(k\)\)\-\\psi\(n\_\{i\}^\{\(W\)\}\(l\)\)\\right\]−λsem​\[ψ​\(mi\(W\)​\(k\)\)−ψ​\(mi\(W\)​\(l\)\)\]\.\\displaystyle\\quad\-\\lambda\_\{\\mathrm\{sem\}\}\\left\[\\psi\(m\_\{i\}^\{\(W\)\}\(k\)\)\-\\psi\(m\_\{i\}^\{\(W\)\}\(l\)\)\\right\]\.Therefore, a token receives lower relative odds if either the token itself has been over\-produced, or its semantic neighborhood has been over\-produced\. We can connect this penalty to the rollout\-induced positive feedback\. Suppose the semantic feedback satisfies

Δ¯roll​\(C\)−Δ¯roll​\(Cc\)≤ηsem​\[ψ​\(mi\(W\)​\(C\)\)−ψ​\(mi\(W\)​\(Cc\)\)\],\\bar\{\\Delta\}\_\{\\mathrm\{roll\}\}\(C\)\-\\bar\{\\Delta\}\_\{\\mathrm\{roll\}\}\(C^\{c\}\)\\leq\\eta\_\{\\mathrm\{sem\}\}\\left\[\\psi\(m\_\{i\}^\{\(W\)\}\(C\)\)\-\\psi\(m\_\{i\}^\{\(W\)\}\(C^\{c\}\)\)\\right\],\(116\)whereΔ¯roll​\(C\)\\bar\{\\Delta\}\_\{\\mathrm\{roll\}\}\(C\)denotes the average rollout\-induced logit shift for tokens inCC\. After applying the semantic penalty, the net cluster\-level semantic feedback is bounded by

\[Δ¯roll​\(C\)−Δ¯roll​\(Cc\)\]−λsem​\[ψ​\(mi\(W\)​\(C\)\)−ψ​\(mi\(W\)​\(Cc\)\)\]\\displaystyle\\left\[\\bar\{\\Delta\}\_\{\\mathrm\{roll\}\}\(C\)\-\\bar\{\\Delta\}\_\{\\mathrm\{roll\}\}\(C^\{c\}\)\\right\]\-\\lambda\_\{\\mathrm\{sem\}\}\\left\[\\psi\(m\_\{i\}^\{\(W\)\}\(C\)\)\-\\psi\(m\_\{i\}^\{\(W\)\}\(C^\{c\}\)\)\\right\]≤−\(λsem−ηsem\)​\[ψ​\(mi\(W\)​\(C\)\)−ψ​\(mi\(W\)​\(Cc\)\)\]\.\\displaystyle\\leq\-\(\\lambda\_\{\\mathrm\{sem\}\}\-\\eta\_\{\\mathrm\{sem\}\}\)\\left\[\\psi\(m\_\{i\}^\{\(W\)\}\(C\)\)\-\\psi\(m\_\{i\}^\{\(W\)\}\(C^\{c\}\)\)\\right\]\.Thus, whenλsem≥ηsem\\lambda\_\{\\mathrm\{sem\}\}\\geq\\eta\_\{\\mathrm\{sem\}\}, the semantic penalty cancels the positive semantic feedback\.

##### Pure frequency penalty as a special case\.

The exact frequency penalty is recovered by settingλsem=0\\lambda\_\{\\mathrm\{sem\}\}=0in[Eq\.˜115](https://arxiv.org/html/2606.15327#A4.E115), or equivalently by choosingS\+​\(k,l\)=𝕀​\{k=l\}S\_\{\+\}\(k,l\)=\\mathbb\{I\}\\\{k=l\\\}\. In this case,

mi\(W\)​\(k\)=ni\(W\)​\(k\),m\_\{i\}^\{\(W\)\}\(k\)=n\_\{i\}^\{\(W\)\}\(k\),and the semantic penalty reduces to an exact\-token repetition penalty\.

## Appendix EAdditional Experimental Details

### E\.1Practical Designs

##### Model setup\.

Followingarriola2025blockandsahoo2024simple, we use a Transformer backbone with rotary positional embeddings\. We adopt the small\-scale architecture ofarriola2025block, resulting in a 110M\-parameter model\.

##### Practical SemDLM\+ kernel\.

Since a full\-vocabulary semantic softmax in[Eq\.˜22](https://arxiv.org/html/2606.15327#S4.E22)is expensive for large vocabularies, we use a top\-kknearest neighborhoods as a practical approximation, i\.e\.

q​\(xt=j∣x0=i\)=αt​δi​j\+βt​νt​\(j\)\+1−αt−βtkt​𝕀​\(j∈𝒩kt​\(i\)\),q\(x\_\{t\}=j\\mid x\_\{0\}=i\)=\\alpha\_\{t\}\\delta\_\{ij\}\+\\beta\_\{t\}\\nu\_\{t\}\(j\)\+\\frac\{1\-\\alpha\_\{t\}\-\\beta\_\{t\}\}\{k\_\{t\}\}\\mathbb\{I\}\\bigl\(j\\in\\mathcal\{N\}\_\{k\_\{t\}\}\(i\)\\bigr\),\(117\)where𝒩kt​\(i\)\\mathcal\{N\}\_\{k\_\{t\}\}\(i\)is the top\-ktk\_\{t\}semantic neighborhood of tokenii\. We excludeiifrom𝒩kt​\(i\)\\mathcal\{N\}\_\{k\_\{t\}\}\(i\)so thatαt\\alpha\_\{t\}exclusively controls self\-retention\. A simple schedule is

αt=αmin\+\(1−αmin\)​\(1−t\)β,βt=βmax​tη,kt=1\+\(kmax−1\)​tγ,\\alpha\_\{t\}=\\alpha\_\{\\min\}\+\(1\-\\alpha\_\{\\min\}\)\(1\-t\)^\{\\beta\},\\qquad\\beta\_\{t\}=\\beta\_\{\\max\}t^\{\\eta\},\\qquad k\_\{t\}=1\+\(k\_\{\\max\}\-1\)t^\{\\gamma\},\(118\)withαmin∈\(0,1\)\\alpha\_\{\\min\}\\in\(0,1\)andβ,η,γ\>0\\beta,\\eta,\\gamma\>0\. Early diffusion \(t→1t\\to 1\) uses larger neighborhoods and a stronger shared transition term for exploration and overlap; late diffusion \(t→0t\\to 0\) increases self\-retention, shrinks the neighborhood, and can annealβt\\beta\_\{t\}downward for precise refinement\.

This kernel keeps the main properties of SemDLM\+: the semantic neighborhood keeps the posterior concentrated, while the global transition branch provides shared support and improves sampling\-side repair\. In implementation, we store only a finite top\-kmaxk\_\{\\max\}neighbor table and absorb the remaining probability mass into the global transition branch, which keeps memory usage manageable\.

##### Dataset\-specific global transition\.

We also test the unigram global transition For LM1B, a simple uniform transition is sufficient\. For OpenWebText, we use a unigram transition where the transition weights is determined by the word frequency in the dataset:

ντowt​\(j\)=𝕀​\{j∈𝒱validowt\}​ν^​\(j\)τ∑k∈𝒱validowtν^​\(k\)τ,0<τ≤1,\\nu\_\{\\tau\}^\{\\mathrm\{owt\}\}\(j\)=\\frac\{\\mathbb\{I\}\\\{j\\in\\mathcal\{V\}\_\{\\mathrm\{valid\}\}^\{\\mathrm\{owt\}\}\\\}\\,\\hat\{\\nu\}\(j\)^\{\\tau\}\}\{\\sum\_\{k\\in\\mathcal\{V\}\_\{\\mathrm\{valid\}\}^\{\\mathrm\{owt\}\}\}\\hat\{\\nu\}\(k\)^\{\\tau\}\},\\qquad 0<\\tau\\leq 1,\(119\)whereν^\\hat\{\\nu\}is the empirical unigram distribution and𝒱validowt\\mathcal\{V\}\_\{\\mathrm\{valid\}\}^\{\\mathrm\{owt\}\}removes special symbols, continuation wordpieces, punctuation\-only tokens, and extremely rare tokens\. This avoids the unnatural proposals produced by uniform transition on open\-domain text\.

##### Changed\-aware training objective\.

In practice, many positions remain unchanged under the forward kernel\. If they are weighted equally, the model can overuse a trivial copy shortcut\. We therefore upweight corrupted positions during training\. Letmi∈\{0,1\}m\_\{i\}\\in\\\{0,1\\\}indicate whether positioniiwas changed by the forward kernel\. We optimize

ℒtrain=∑iwi​ℓi∑iwi,wi=λchg​𝕀​\{mi=1\}\+λsame​𝕀​\{mi=0\},\\mathcal\{L\}\_\{\\mathrm\{train\}\}=\\frac\{\\sum\_\{i\}w\_\{i\}\\,\\ell\_\{i\}\}\{\\sum\_\{i\}w\_\{i\}\},\\qquad w\_\{i\}=\\lambda\_\{\\mathrm\{chg\}\}\\mathbb\{I\}\\\{m\_\{i\}=1\\\}\+\\lambda\_\{\\mathrm\{same\}\}\\mathbb\{I\}\\\{m\_\{i\}=0\\\},\(120\)withλchg≫λsame\\lambda\_\{\\mathrm\{chg\}\}\\gg\\lambda\_\{\\mathrm\{same\}\}\. This preserves the semantic denoising signal while keeping optimization stable\.

##### Blockwise predictor sampler\.

For generation, we use a semi\-autoregressive blockwise sampler\. At each step, the model predictspθ​\(x0∣xt\)p\_\{\\theta\}\(x\_\{0\}\\mid x\_\{t\}\), samples a truncated clean proposal, projects it to the next lower\-noise level using the same practical forward kernel, and updates only low\-confidence positions\. The replacement probability is approximated by

Pr⁡\(replace at position​i\)≈pt−pspt​\(1−pθ​\(xti∣xt\)\)ρ,\\Pr\(\\text\{replace at position \}i\)\\approx\\frac\{p\_\{t\}\-p\_\{s\}\}\{p\_\{t\}\}\\Big\(1\-p\_\{\\theta\}\(x\_\{t\}^\{i\}\\mid x\_\{t\}\)\\Big\)^\{\\rho\},\(121\)optionally with a freeze threshold for already confident tokens\.

### E\.2Addtional Figures

[Figure˜7](https://arxiv.org/html/2606.15327#A5.F7)draws the impact of sampling steps on generation quality\.

![Refer to caption](https://arxiv.org/html/2606.15327v1/x3.png)\(a\)LM1B\.
![Refer to caption](https://arxiv.org/html/2606.15327v1/x4.png)\(b\)OpenWebText\.

Figure 7:Generation PPL as the number of sampling steps increases\. SemDLM\+ shows stronger improvement with additional refinement steps, indicating better sampling\-side repair\.
### E\.3Computational Cost

The experiments were run on 8x NVIDIA A100\-SXM4\-80GB GPUs\. For LM1B experiments, the training time is typically 72 hours and for OWT the training time is 144 Hours\. The sampling takes approximately 5 mins in the same infrastructure\.