大型语言连续扩散模型
摘要
NVIDIA 推出 Sigma，这是首个大规模（3B/8B）连续扩散语言模型。它通过可调控的低维 ODE/SDE 轨迹对被高斯噪声破坏的词元嵌入进行去噪，在数学和编程基准测试上实现了与离散扩散模型和自回归模型相当的性能，同时支持在嵌入空间中进行引导以实现质量与多样性的权衡，并能实现高效的少步蒸馏。
arXiv:2610.02665v1 Announce Type: new
Abstract: Despite the success of discrete diffusion language models (dLMs) for fast parallel decoding, their non-smooth, high-dimensional space hinders trajectory steering for reasoning and inference acceleration. To overcome this, we present Sigma, the first large-scale (3B/8B) continuous dLM built on steerable, low-dimensional ODE/SDE latent trajectories. Trained blockwise via likelihood optimization, Sigma jointly denoises Gaussian-corrupted token embeddings while learning an optimal embedding geometry. To accelerate training, Sigma leverages pre-trained weights from autoregressive (AR) models for warm-starting. During inference, we identify classifier-free guidance and score temperature as essential for high-fidelity reasoning and coding. Across comprehensive math reasoning and coding evaluations against state-of-the-art discrete counterparts (masked dLMs and AR baselines), Sigma achieves competitive performance with discrete models on standard benchmarks (e.g., GSM8K, Minerva, HumanEval, MBPP) after pre-training and on challenging reasoning tasks (e.g., MATH-500, AIME) after supervised fine-tuning. Beyond performance parity, we uncover key structural properties unique to continuous dLMs: (i) embedding-space steering effectively governs the quality-diversity trade-off, yielding strong pass@k performance and (ii) continuous trajectories enable graceful degradation for low NFEs and efficient distillation. These establish continuous dLMs as a promising paradigm for efficient language generation.
查看缓存全文
缓存时间: 2026/10/05 10:01
# Large Language Continuous Diffusion Models Source: [https://arxiv.org/html/2610.02665](https://arxiv.org/html/2610.02665) Wei GuoAffiliation:NVIDIAAffiliation:Georgia Institute of Technology†Work done during internship at NVIDIA\.\*Core contributors, equal contribution\.⋆Equal supervision\.Correspondence to:[mmardani@nvidia\.com](mailto:[email protected])Jean\-Marie LemercierAffiliation:NVIDIASimon Welker Yonggan FuAffiliation:NVIDIAMohammad Mahdi KamaniAffiliation:NVIDIASajad NorouziAffiliation:NVIDIAJulius Berner Tomas GeffnerAffiliation:NVIDIAKarsten KreisAffiliation:NVIDIAYongxin ChenAffiliation:Georgia Institute of Technology†Work done during internship at NVIDIA\.\*Core contributors, equal contribution\.⋆Equal supervision\.Correspondence to:[mmardani@nvidia\.com](mailto:[email protected])Molei TaoAffiliation:Georgia Institute of Technology†Work done during internship at NVIDIA\.\*Core contributors, equal contribution\.⋆Equal supervision\.Correspondence to:[mmardani@nvidia\.com](mailto:[email protected])John Thickstun Pavlo MolchanovAffiliation:NVIDIAAffiliation:Cornell UniversityAnte JukićAffiliation:NVIDIAArash VahdatAffiliation:NVIDIAMorteza MardaniAffiliation:NVIDIA ###### Abstract Despite the success of discrete diffusion language models \(dLMs\) for fast parallel decoding, their non\-smooth, high\-dimensional space hinders trajectory steering for reasoning and inference acceleration\. To overcome this, we presentSigma, the first large\-scale \(3B/8B\) continuous dLM built on steerable, low\-dimensional ODE/SDE latent trajectories\. Trained blockwise via likelihood optimization,Sigmajointly denoises Gaussian\-corrupted token embeddings while learning an optimal embedding geometry\. To accelerate training,Sigmaleverages pre\-trained weights from autoregressive \(AR\) models for warm\-starting\. During inference, we identify classifier\-free guidance and score temperature as essential for high\-fidelity reasoning and coding\. Across comprehensive math reasoning and coding evaluations against state\-of\-the\-art discrete counterparts \(masked dLMs and AR baselines\),Sigmaachieves competitive performance with discrete models on standard benchmarks \(e\.g\., GSM8K, Minerva, HumanEval, MBPP\) after pre\-training and on challenging reasoning tasks \(e\.g\., MATH\-500, AIME\) after supervised fine\-tuning\. Beyond performance parity, we uncover key structural properties unique to continuous dLMs: \(i\) embedding\-space steering effectively governs the quality\-diversity trade\-off, yielding strong pass@kkperformance and \(ii\) continuous trajectories enable graceful degradation for low NFEs and efficient distillation\. These establish continuous dLMs as a promising paradigm for efficient language generation\. ## 1Introduction While diffusion language models \(dLMs\) enable fast parallel decoding compared to autoregressive \(AR\) baselines\([Inception Labs et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib3);[Song et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib2);[Fu et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib51)\), dominant discrete dLMs operate in high\-dimensional, categorical state spaces \(e\.g\.,V≥100KV\{\\geq\}100\\text\{K\}\) that suffer under limited step budgets and impede trajectory control for complex reasoning and coding tasks\. Continuous diffusion offers a principled way to overcome these structural limitations through smooth latent geometries that natively support structured editing\([Meng et al\., 2022](https://arxiv.org/html/2610.02665#bib.bib6)\), flexible ODE trajectory steering\([Lu et al\., 2022](https://arxiv.org/html/2610.02665#bib.bib4)\), and efficient few\-step distillation\([Salimans and Ho, 2022](https://arxiv.org/html/2610.02665#bib.bib9);[Song et al\., 2023](https://arxiv.org/html/2610.02665#bib.bib8)\)\. Recently, there has been growing interest in continuous diffusion and flow matching for text generation\([Gulrajani and Hashimoto, 2023](https://arxiv.org/html/2610.02665#bib.bib16);[Dieleman et al\., 2022](https://arxiv.org/html/2610.02665#bib.bib24);[Hu et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib47);[Lee et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib43);[Guo et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib56);[Liang et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib53);[Yang et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib60)\); see[App\.A](https://arxiv.org/html/2610.02665#A1)\. However, existing methods remain limited in scale \(<2B<2\\text\{B\}\) or focus onunconditionalgeneration, lacking evaluations on complex reasoning tasks\. Whether continuous dLMs scale to complex conditional reasoning remains open\. This is a non\-trivial hurdle because continuous models update block embeddings globally, in contrast to the factorized categorical denoising of discrete dLMs that leads to semi\-autoregressive token commitment per block\. To overcome these limitations, we presentSigma\(3B/8B\):scaling upGaussian diffusionmodels for language\. We propose a family of continuous dLMs operating over a 16\-dimensional embedding space trained blockwise via likelihood optimization\. Within each block, the model learns to jointly denoise Gaussian\-corrupted token embeddings and to structure an optimal embedding geometry\. To scale training, we warm\-start the model from existing pre\-trained AR weights\. For inference, we explore diffusion trajectory sampling, identifying embedding\-space Classifier\-Free Guidance \(CFG\) and Score Temperature \(ST\) as crucial ingredients for high\-fidelity reasoning and coding\. Figure 1:Left: Inference pipeline\.Sigmagenerates text blockwise via iterative diffusion denoising over 16D continuous latents \(𝐳t\\mathbf\{z\}\_\{t\}\) conditioned on a KV\-cached prefix \(𝐜\\mathbf\{c\}\), followed by categorical readout\. Predicted vocabulary probabilities𝐱θ\(𝐳t,t,𝐜\)\\mathbf\{x\}\_\{\\theta\}\(\\mathbf\{z\}\_\{t\},t;\\mathbf\{c\}\)map to mean predictions𝐞θ\(𝐳t,t,𝐜\)\\mathbf\{e\}\_\{\\theta\}\(\\mathbf\{z\}\_\{t\},t;\\mathbf\{c\}\)in 16D space via categorical reparameterization\. Clean embeddings \(dots\) lie on the 16D unit sphere, while𝐞θ\\mathbf\{e\}\_\{\\theta\}lies on or within it\.Middle: Smooth trajectory enables low\-NFE reasoning\.GSM8K accuracy vs\. NFEs using the optimal sampler per step budget \(PDD: Parallel Decoding Distillation\)\.Right: Scaling inference\-time compute\.Pass@kkvs\.kkon AIME 2025 \(solid curves: fixed config acrosskk; thin curves: alternative tested configs\)\.Through unified evaluations against state\-of\-the\-art discrete dLMs \(e\.g\., Nemotron\-Labs\-Diffusion\([Fu et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib51)\), LLaDA\([Nie et al\., 2025b](https://arxiv.org/html/2610.02665#bib.bib38)\), Dream\([Ye et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib12)\), SDAR\([Cheng et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib66)\)\), we show thatSigmaachieves competitive performance across standard math and coding benchmarks \(e\.g\., GSM8K, Minerva, HumanEval, MBPP\) and extends well to harder reasoning tasks after supervised fine\-tuning \(e\.g\., MATH\-500, AIME\)\. Beyond raw accuracy, we highlight two structural properties unique to continuous language diffusion: \(i\) flexible pass@kkexploration–exploitation control via CFG and ST hyperparameters; and \(ii\) graceful performance degradation under low NFE regimes \(16/32\) and efficient trajectory distillation for very low NFE regimes \(4/8\)\. Together, these properties position continuous diffusion as a promising foundation for fast inference in modern agentic AI\. Our key contributions are summarized as follows: - •First Large\-Scale Continuous dLM:We presentSigma\(3B/8B\), efficiently trained by warm\-starting from pre\-trained AR weights, and optimizing a regularized continuous diffusion ELBO\. - •Competitive Math & Coding:Across six base and ten instruct benchmarks,Sigmaachieves competitive performance with state\-of\-the\-art discrete dLMs/ARs on reasoning and coding\. - •Inference Steering \(CFG & ST\):By elucidating the inference design space, we show that embedding\-space CFG and ST effectively govern the quality\-diversity trade\-off with minimal latency overhead via single\-pass block\-causal attention packing\. - •Low\-NFE Robustness:With embedding regularization,Sigmasustains strong quality down to 16 NFEs per block, while trajectory distillation noticeably improves performance at 4–8 NFEs\. ## 2Background ##### Notation\. Let𝐱\{\\mathbf\{x\}\}be a sequence of lengthLLover a vocabulary𝒱\\mathcal\{V\}of sizeVV\. We identify𝐱\{\\mathbf\{x\}\}with a matrix in\{0,1\}L×V\\\{0,1\\\}^\{L\\times V\}whose rows𝐱1,…,𝐱L\{\\mathbf\{x\}\}^\{1\},\.\.\.,\{\\mathbf\{x\}\}^\{L\}are one\-hot vectors inℝV\\mathbb\{R\}^\{V\}\.ΔV\\Delta^\{V\}denotes the probability simplex over𝒱\\mathcal\{V\}\.⟨⋅,⋅⟩\\langle\\cdot,\\cdot\\rangledenotes the inner product\. NELBO stands for negative evidence lower bound\. ##### Plaid\. Continuous dLMs lift discrete tokens into a low\-dimensional Euclidean space and run Gaussian diffusion\. There have been multiple proposed design choices, see[App\.A](https://arxiv.org/html/2610.02665#A1)for an overview\. We adopt the embedding and diffusion process from Plaid\([Gulrajani and Hashimoto, 2023](https://arxiv.org/html/2610.02665#bib.bib16)\), a Variational Diffusion Model \(VDM,[Kingma et al\. \(2021\)](https://arxiv.org/html/2610.02665#bib.bib18)\) over token embeddings\. Let𝐄∈ℝV×de\{\\mathbf\{E\}\}\\in\\mathbb\{R\}^\{V\\times d\_\{e\}\}be a learnable embedding matrix whose rows lie on the unit sphere, withde≪Vd\_\{e\}\\ll V\. The embedding of𝐱\{\\mathbf\{x\}\}is given by𝐞:=𝐱𝐄∈ℝL×de\{\\mathbf\{e\}\}:=\{\\mathbf\{x\}\}\{\\mathbf\{E\}\}\\in\\mathbb\{R\}^\{L\\times d\_\{e\}\}\(i\.e\., standard embedding lookup\)\. The forward process applies variance\-preserving Gaussian noise on𝐞\{\\mathbf\{e\}\}: q\(𝐳t∣𝐱\)=𝒩\(αt𝐞,σt2𝐈\),αt,σt\>0,αt2\+σt2=1,t∈\[0,1\],\\displaystyle q\(\{\\mathbf\{z\}\}\_\{t\}\\mid\{\\mathbf\{x\}\}\)=\\mathcal\{N\}\(\\alpha\_\{t\}\{\\mathbf\{e\}\},\\ \\sigma\_\{t\}^\{2\}\{\\mathbf\{I\}\}\),\\qquad\\alpha\_\{t\},\\sigma\_\{t\}\>0,\\qquad\\alpha\_\{t\}^\{2\}\+\\sigma\_\{t\}^\{2\}=1,\\qquad t\\in\[0,1\],\(1\)withSNR\(t\)=αt2/σt2\\operatorname\{SNR\}\(t\)=\\alpha\_\{t\}^\{2\}/\\sigma\_\{t\}^\{2\}decreasing monotonically overt∈\[0,1\]t\\in\[0,1\]\. A denoiser𝐱θ\(𝐳t,t\)∈\(ΔV\)L⊂ℝL×V\{\\mathbf\{x\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\},t\)\\in\(\\Delta^\{V\}\)^\{L\}\\subset\\mathbb\{R\}^\{L\\times V\}with output prior returns a categorical distribution over the vocabulary at every position\. The reverse process is the exact Gaussian posteriorq\(𝐳s∣𝐳t,𝐱\)q\(\{\\mathbf\{z\}\}\_\{s\}\\mid\{\\mathbf\{z\}\}\_\{t\},\{\\mathbf\{x\}\}\)\([Kingma et al\., 2021](https://arxiv.org/html/2610.02665#bib.bib18), App\. A\.2\)with𝐱\{\\mathbf\{x\}\}replaced by𝐱θ\(𝐳t,t\)\{\\mathbf\{x\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\},t\)\. Its NELBO splits into 3 terms: prior KLℒprior\\mathcal\{L\}\_\{\\text\{prior\}\}, reconstruction lossℒrec\\mathcal\{L\}\_\{\\text\{rec\}\}, and diffusion lossℒdiff\\mathcal\{L\}\_\{\\text\{diff\}\}: −logpθ\(𝐱\)≤ℒNELBO\(𝐱\):=\\displaystyle\-\\log p\_\{\\theta\}\(\{\\mathbf\{x\}\}\)\\leq\\mathcal\{L\}\_\{\\text\{NELBO\}\}\(\{\\mathbf\{x\}\}\):=KL\(q\(𝐳1∣𝐱\)∥p\(𝐳1\)\)\+𝔼𝐳0∼q\(𝐳0∣𝐱\)\[∑l=1L−log⟨𝐱θl\(𝐳0,0\),𝐱l⟩\]\\displaystyle\\operatorname\{KL\}\(q\(\{\\mathbf\{z\}\}\_\{1\}\\mid\{\\mathbf\{x\}\}\)\\,\\\|\\,p\(\{\\mathbf\{z\}\}\_\{1\}\)\)\+\\E\_\{\{\\mathbf\{z\}\}\_\{0\}\\sim q\(\{\\mathbf\{z\}\}\_\{0\}\\mid\{\\mathbf\{x\}\}\)\}\\big\[\{\\textstyle\\sum\}\_\{l=1\}^\{L\}\-\\log\\langle\{\\mathbf\{x\}\}\_\{\\theta\}^\{l\}\(\{\\mathbf\{z\}\}\_\{0\},0\),\{\\mathbf\{x\}\}^\{l\}\\rangle\\big\]−12𝔼t∼Unif\[0,1\]𝔼𝐳t∼q\(𝐳t∣𝐱\)\[SNR′\(t\)‖𝐞θ\(𝐳t,t\)−𝐞‖2\],\\displaystyle\-\\tfrac\{1\}\{2\}\\ \\E\_\{t\\sim\\operatorname\{Unif\}\[0,1\]\}\\E\_\{\{\\mathbf\{z\}\}\_\{t\}\\sim q\(\{\\mathbf\{z\}\}\_\{t\}\\mid\{\\mathbf\{x\}\}\)\}\[\\operatorname\{SNR\}^\{\\prime\}\(t\)\\\|\{\\mathbf\{e\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\},t\)\-\{\\mathbf\{e\}\}\\\|^\{2\}\],\(2\)wherep\(𝐳1\)=𝒩\(𝟎,𝐈\)p\(\{\\mathbf\{z\}\}\_\{1\}\)=\\mathcal\{N\}\(\{\\mathbf\{0\}\},\{\\mathbf\{I\}\}\)and thecategorical reparameterization𝐞θ\(𝐳t,t\):=𝐱θ\(𝐳t,t\)𝐄∈ℝL×de\{\\mathbf\{e\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\},t\):=\{\\mathbf\{x\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\},t\)\{\\mathbf\{E\}\}\\in\\mathbb\{R\}^\{L\\times d\_\{e\}\}predicts the posterior mean embedding𝔼\[𝐞∣𝐳t,t\]\\E\[\{\\mathbf\{e\}\}\\mid\{\\mathbf\{z\}\}\_\{t\},t\]corresponding to𝐱θ\(𝐳t,t\)\{\\mathbf\{x\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\},t\)\. ##### Learned noise schedule\. The schedule admitsγ\(t\)=γ0\+\(γ1−γ0\)γ~\(t\)\\gamma\(t\)=\\gamma\_\{0\}\+\(\\gamma\_\{1\}\-\\gamma\_\{0\}\)\\tilde\{\\gamma\}\(t\)with learnable endpointsγ0<γ1\\gamma\_\{0\}<\\gamma\_\{1\}and a monotone interior shapeγ~∈\[0,1\]\\tilde\{\\gamma\}\\in\[0,1\], givingαt2=σ\(−γ\(t\)\)\\alpha\_\{t\}^\{2\}=\\sigma\(\-\\gamma\(t\)\),σt2=σ\(γ\(t\)\)\\sigma\_\{t\}^\{2\}=\\sigma\(\\gamma\(t\)\)andSNR\(t\)=e−γ\(t\)\\operatorname\{SNR\}\(t\)=\{\\mathrm\{e\}\}^\{\-\\gamma\(t\)\}\.111σ\(t\)=1/\(1\+e−t\)\\sigma\(t\)=1/\(1\+\{\\mathrm\{e\}\}^\{\-t\}\)is the sigmoid function\.The diffusion term depends on the endpoints alone and is invariant toγ~\(t\)\\tilde\{\\gamma\}\(t\), which instead controls its Monte\-Carlo \(MC\) variance\([Kingma et al\., 2021](https://arxiv.org/html/2610.02665#bib.bib18)\), so they are learned for different reasons\. Learning the interior\-shape distributes denoising difficulty overtt\([Yang et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib60)\)\. ##### Learned embeddings\. The NELBO[2](https://arxiv.org/html/2610.02665#S2.E2)depends on𝐄\{\\mathbf\{E\}\}via[1](https://arxiv.org/html/2610.02665#S2.E1), where embeddings become likelihood\-trained parameters, which separates Plaid from continuous dLMs built on fixed \(e\.g\., one\-hot\) encodings\. Not learning embeddings significantly worsens Plaid’s performance\([Yang et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib60)\)\. ##### Self\-conditioning \(SC\)\. Proposed in[Chen et al\. \(2023\)](https://arxiv.org/html/2610.02665#bib.bib21), SC lets the denoiser see its own previous estimate\. Let𝐱θ\(𝐳t,t,𝐞sc\)\{\\mathbf\{x\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\},t;\{\\mathbf\{e\}\}\_\{\\text\{sc\}\}\)denote a denoiser taking an additional predicted clean embedding𝐞sc∈ℝL×de\{\\mathbf\{e\}\}\_\{\\text\{sc\}\}\\in\\mathbb\{R\}^\{L\\times d\_\{e\}\}, and𝟎\{\\mathbf\{0\}\}for the empty condition \(omitted by default in our notation\)\. During training,25%25\\%of training samples are self\-conditioned: a first pass without gradient tracking produces𝐞sc:=𝐱θ\(𝐳t,t,𝟎\)𝐄\{\\mathbf\{e\}\}\_\{\\text\{sc\}\}:=\{\\mathbf\{x\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\},t;\{\\mathbf\{0\}\}\)\{\\mathbf\{E\}\}, and only the second pass,𝐱θ\(𝐳t,t,sg\(𝐞sc\)\)𝐄\{\\mathbf\{x\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\},t;\\operatorname\{sg\}\(\{\\mathbf\{e\}\}\_\{\\text\{sc\}\}\)\)\{\\mathbf\{E\}\}, receives gradients\. The remaining samples use𝟎\{\\mathbf\{0\}\}as SC input, so the denoiser learns both branches\. During validation, all samples enable SC\. Since𝐞sc\{\\mathbf\{e\}\}\_\{\\text\{sc\}\}is itself a function of\(𝐳t,t\)\(\{\\mathbf\{z\}\}\_\{t\},t\), the NELBO remains valid\. Notice that during training,25%25\\%SC increases the non\-embedding FLOPs by only about8%8\\%, which is minimal\. Furthermore, during inference, one directly feeds the previous\-step’s prediction as SC, which incurs no overhead\. ##### Scaling\. RePlaid\([Yang et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib60)\)realigns Plaid with the architectures of modern discrete dLMs/ARs and finds that, once aligned, continuous diffusion scales competitively with masked diffusion\. ## 3Large Language Continuous Diffusion Models \(a\)Training procedure ofSigma\. \(b\)Attention mask for training\. Figure 2:Training procedure ofSigma\(a\) and its transformer attention mask \(b\)for an example sequence𝐱\{\\mathbf\{x\}\}of lengthL=6L=6with block sizeL′=2L^\{\\prime\}=2\. During pre\-training, we feed the clean discrete tokens𝐱\\mathbf\{x\}to the AR stream \(a, left part\) and the noisy continuous embeddings𝐳\\mathbf\{z\}to the diffusion stream \(a, right part\)\. For the diffusion stream, each block is randomly chosen to either compute the reconstruction loss \(1st block,t=0t=0\) or the diffusion loss \(2nd & 3rd blocks,t∼Unif\[0,1\]t\\sim\\operatorname\{Unif\}\[0,1\]\); see[5](https://arxiv.org/html/2610.02665#S3.E5)\. Reconstruction loss is the cross entropy between the logits and the ground truth tokens, while for diffusion loss we use categorical reparameterization to obtain the mean embedding prediction and compute the mean squared error against the ground truth diffusion embeddings\.### 3\.1Blockwise Probabilistic Model Inspired by Nemotron\-Labs\-Diffusion \(NLD,[Fu et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib51)\), a state\-of\-the\-art open\-sourced masked dLM \(see[App\.C](https://arxiv.org/html/2610.02665#A3)for background\), we adopt a blockwise formulation for our continuous diffusion language model: the sequence𝐱\{\\mathbf\{x\}\}is partitioned intoBBcontiguous blocks\{𝐱\(b\)\}b=1B\\\{\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\\}\_\{\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}=1\}^\{B\}of sizeL′=L/BL^\{\\prime\}=L/B, where the model denoises one block at a time given its clean prefix\. As in[Arriola et al\. \(2025\)](https://arxiv.org/html/2610.02665#bib.bib36), we factorize the likelihood autoregressively*across*blocks and then bound each factor: −logpθ\(𝐱\)\\displaystyle\-\\log p\_\{\\theta\}\(\{\\mathbf\{x\}\}\)=∑b=1B−logpθ\(𝐱\(b\)∣𝐱\(<b\)\)≤∑b=1BℒNELBO\(𝐱\(b\);𝐱\(<b\)\)=:ℒBD\(𝐱\),\\displaystyle=\{\\textstyle\\sum\}\_\{b=1\}^\{B\}\-\\log p\_\{\\theta\}\(\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\mid\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)\\leq\{\\textstyle\\sum\}\_\{b=1\}^\{B\}\\mathcal\{L\}\_\{\\text\{NELBO\}\}\(\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\};\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)=:\\mathcal\{L\}\_\{\\text\{BD\}\}\(\{\\mathbf\{x\}\}\),\(3\)where we obtainℒNELBO\(𝐱\(b\),𝐱\(<b\)\)\\mathcal\{L\}\_\{\\text\{NELBO\}\}\(\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\};\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)by conditioning the denoiser on𝐱\(<b\)\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}as follows: ℒNELBO\(𝐱\(b\),𝐱\(<b\)\)\\displaystyle\\mathcal\{L\}\_\{\\text\{NELBO\}\}\(\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\};\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)=KL\(q\(𝐳1\(b\)∣𝐱\(b\)\)∥p\(𝐳1\(b\)\)\)\\displaystyle=\\operatorname\{KL\}\(q\(\{\\mathbf\{z\}\}\_\{1\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\mid\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\)\\,\\\|\\,p\(\{\\mathbf\{z\}\}\_\{1\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\)\)\+𝔼q\(𝐳0\(b\)∣𝐱\(b\)\)\[∑l=1L′−log⟨𝐱θl\(𝐳0\(b\),0;𝐱\(<b\)\),𝐱\(b,l\)⟩\]\\displaystyle\+\\E\_\{q\(\{\\mathbf\{z\}\}\_\{0\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\mid\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\)\}\\big\[\{\\textstyle\\sum\}\_\{l=1\}^\{L^\{\\prime\}\}\-\\log\\langle\{\\mathbf\{x\}\}\_\{\\theta\}^\{l\}\(\{\\mathbf\{z\}\}\_\{0\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},0;\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\),\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b,l\}\)\}\\rangle\\big\]−12𝔼t∼Unif\[0,1\]𝔼q\(𝐳t\(b\)∣𝐱\(b\)\)\[SNR′\(t\)‖𝐞θ\(𝐳t\(b\),t,𝐱\(<b\)\)−𝐞\(b\)‖2\]\.\\displaystyle\-\\tfrac\{1\}\{2\}\\E\_\{t\\sim\\operatorname\{Unif\}\[0,1\]\}\\E\_\{q\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\mid\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\)\}\[\\operatorname\{SNR\}^\{\\prime\}\(t\)\\\|\{\\mathbf\{e\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)\-\{\\mathbf\{e\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\\|^\{2\}\]\.\(4\) ##### Training\. The reconstruction loss att=0t=0and diffusion loss att∈\(0,1\)t\\in\(0,1\)do not share a single forward pass\. RePlaid evaluates one of them per𝐱\(b\)\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}via Horvitz–Thompson estimator\([Horvitz and Thompson, 1952](https://arxiv.org/html/2610.02665#bib.bib52)\)withR∼Bern\(prec\)R\\sim\\mathrm\{Bern\}\(p\_\{\\text\{rec\}\}\), i\.e\.,𝐱\(b\)\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}is assigned toℒrec\\mathcal\{L\}\_\{\\text\{rec\}\}w\.p\.prec∈\(0,1\)p\_\{\\text\{rec\}\}\\in\(0,1\): ℒNELBO\(𝐱\(b\),𝐱\(<b\)\)=𝔼R\[ℒprior\(𝐱\(b\)\)\+Rprecℒrec\(𝐱\(b\),𝐱\(<b\)\)\+1−R1−precℒdiff\(𝐱\(b\),𝐱\(<b\)\)\]\.\\displaystyle\\mathcal\{L\}\_\{\\text\{NELBO\}\}\(\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\};\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)=\\E\_\{R\}\\big\[\\mathcal\{L\}\_\{\\text\{prior\}\}\(\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\)\+\\tfrac\{R\}\{p\_\{\\text\{rec\}\}\}\\mathcal\{L\}\_\{\\text\{rec\}\}\(\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\};\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)\+\\tfrac\{1\-R\}\{1\-p\_\{\\text\{rec\}\}\}\\mathcal\{L\}\_\{\\text\{diff\}\}\(\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\};\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)\\big\]\.\(5\)Notice that this is an unbiased estimator ifℒdiff\\mathcal\{L\}\_\{\\text\{diff\}\}andℒrec\\mathcal\{L\}\_\{\\text\{rec\}\}are unbiased\. We setprec=0\.1p\_\{\\text\{rec\}\}=0\.1, empirically found to minimize the training objective variance of RePlaid\. To construct the unbiased estimator forℒBD\(𝐱\)\\mathcal\{L\}\_\{\\text\{BD\}\}\(\{\\mathbf\{x\}\}\)in[3](https://arxiv.org/html/2610.02665#S3.E3)for training, we sampleR\(b\)R^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}i\.i\.d\. for each block\. IfR\(b\)=0R^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}=0, we samplet\(b\)t^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}and𝐳t\(b\)\(b\)\{\\mathbf\{z\}\}\_\{t^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}; otherwise, we sett=0t=0and sample𝐳0\(b\)\{\\mathbf\{z\}\}\_\{0\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\. The noise schedule endpoints are learned to optimizeℒNELBO\(𝐱\(b\),𝐱\(<b\)\)\\mathcal\{L\}\_\{\\text\{NELBO\}\}\(\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\};\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)while the schedule interior shape is learned to optimize its MC variance, with both shared across blocks\. As mentioned before, we apply SC only to25%25\\%of samples \(full sequence with all blocks\)\. See[Alg\.1](https://arxiv.org/html/2610.02665#alg1)for the training algorithm\. ### 3\.2Pre\-Training NLD utilizes AR initialization and regularization to enhance downstream performance\([Fu et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib51)\)\. However, adapting these techniques to continuous dLMs likeSigmais non\-trivial, as the activation dynamics of continuous and discrete models differ fundamentally; it is thus unclear whether discrete AR initialization remains as effective for continuous representation space\. To establish a rigorous baseline, we first alignSigma’s data, architecture, and optimization with discrete dLMs for a controlled comparison at scale \([Sec\.3\.2\.1](https://arxiv.org/html/2610.02665#S3.SS2.SSS1)\)\. In[Sec\.3\.2\.2](https://arxiv.org/html/2610.02665#S3.SS2.SSS2), we evaluate the impact of AR initialization when pre\-trainingSigma\-3B with[3](https://arxiv.org/html/2610.02665#S3.E3), followed by an investigation of AR regularization for training our flagship 3B/8B models\. \(a\)From scratch \+ no AR reg\. Block size: 64\(b\)From AR init\. \+ no AR reg\. Block size: 64\(c\)From AR init\. \+ AR reg\. Block size: 32 Figure 3:Pre\-trainingSigma\-3Bfrom scratch \(a\), from AR initialization \(b\), and from AR initialization with AR regularization \(c\)\. PPL for AR and diffusion streams is computed viaℒar\(𝐱\)\\mathcal\{L\}\_\{\\text\{ar\}\}\(\{\\mathbf\{x\}\}\)[6](https://arxiv.org/html/2610.02665#S3.E6)andℒBD\(𝐱\)\\mathcal\{L\}\_\{\\text\{BD\}\}\(\{\\mathbf\{x\}\}\)[3](https://arxiv.org/html/2610.02665#S3.E3), respectively\. Early investigations \(a, b\) used block size 64; our flagship 3B model \(c\) used block size 32 to match the inference block size of NLD\. Proper AR initialization/regularization leads to the bestSigmamodel\.#### 3\.2\.1Unified Pre\-training of Continuous and Discrete dLMs ##### Data\. As with NLD, we adopt the open\-sourced pre\-training data from[Basant et al\. \(2025\)](https://arxiv.org/html/2610.02665#bib.bib50)\. We use the Mistral\-NeMo tokenizer \(V=131,072V=131\{,\}072\)\([Mistral AI, 2024](https://arxiv.org/html/2610.02665#bib.bib81)\), batch size512512, andL=4096L=4096\. ##### Architecture\. Sigmaadopts the transformer backbone of NLD and its AR baseline, ensuring the three families are parameter\- and FLOP\-matched\. As with NLD,Sigmauses a block causal attention mask \([Fig\.2\(b\)](https://arxiv.org/html/2610.02665#S3.F2.sf2)\)\.Sigmauniquely adds a unit\-norm 16\-dimensional diffusion embedding table with a linear up\-projection, a categorical reparameterization mappingVV\-dimensional probabilities to low\-dimensional soft embeddings, a self\-conditioning linear projection, Fourier time conditioning, and a learnable noise schedule \(see[Sec\.2](https://arxiv.org/html/2610.02665#S2)\)\. Together, these account for less than1%1\\%of the total parameters\. We provide thorough details in[App\.E](https://arxiv.org/html/2610.02665#A5)\. ##### Training\. We train our models for 150K steps, following the optimization procedure detailed in[App\.E](https://arxiv.org/html/2610.02665#A5)\. Given a batch size of 512, 150K steps correspond to over 300B tokens\. Training consumed ~9,600 and ~19,200 NVIDIA GB200 GPU hours for the 3B and 8B models, respectively\. #### 3\.2\.2AR Initialization and Regularization ##### AR Initialization\. To evaluate the impact of initialization, we compare pre\-trainingSigma\(i\) from scratch and \(ii\) from an AR baseline pretrained on 1T tokens\. Due to the dimension mismatch between high\-dimensional AR embeddings andSigma’s 16\-dimensional space, we randomly initialize the diffusion embedding layer\. As shown in[Figs\.3\(a\)](https://arxiv.org/html/2610.02665#S3.F3.sf1)and[3\(b\)](https://arxiv.org/html/2610.02665#S3.F3.sf2),Sigma\-3B \(L′=64L^\{\\prime\}\{=\}64\) with self\-conditioning achieves competitive perplexity relative to NLD\-3B \(L′=64L^\{\\prime\}\{=\}64\) under both regimes \(i\) and \(ii\)\. However, AR initialization significantly accelerates convergence and yields lower final training loss\. ##### AR Regularization\. Continuous diffusion training inherently suffers from high gradient variance and slow convergence\. To mitigate variance and accelerate optimization, we follow NLD by adding AR regularization, optimizing the joint objectiveℒar\+wdiffℒBD\\mathcal\{L\}\_\{\\text\{ar\}\}\+w\_\{\\text\{diff\}\}\\mathcal\{L\}\_\{\\text\{BD\}\}\([Fig\.3\(c\)](https://arxiv.org/html/2610.02665#S3.F3.sf3); see[App\.E](https://arxiv.org/html/2610.02665#A5)for thewdiffw\_\{\\text\{diff\}\}weighting schedule\), where ℒar=∑l=2L−log⟨𝐱θl\(𝐱<l\),𝐱l⟩\\mathcal\{L\}\_\{\\text\{ar\}\}=\{\\textstyle\\sum\}\_\{l=2\}^\{L\}\-\\log\\langle\{\\mathbf\{x\}\}\_\{\\theta\}^\{l\}\(\{\\mathbf\{x\}\}^\{<l\}\),\{\\mathbf\{x\}\}^\{l\}\\rangle\(6\)is computed over the AR stream using causal attention \([Fig\.2](https://arxiv.org/html/2610.02665#S3.F2)\)\. This architecture employs two distinct readout heads, with the AR regularization computed exclusively from the AR head\. While this dual\-head formulation improves auxiliary AR decoding capability, our primary objective remains maximizing diffusion decoding performance\. Finally, input sequences utilize AR embeddings for the AR stream and 16D diffusion embeddings for the diffusion stream \([Fig\.2](https://arxiv.org/html/2610.02665#S3.F2)\), facilitating AR initialization and efficient KV\-caching\. As shown in[Fig\.3\(c\)](https://arxiv.org/html/2610.02665#S3.F3.sf3), AR regularization significantly improves training stability; with a properly tunedwdiffw\_\{\\text\{diff\}\}, it even outperforms pure perplexity optimization for the diffusion stream\. This confirms that AR regularization stabilizes training while yielding a higher\-quality model\. Consequently, we adopt AR initialization, AR regularization, and a block size ofL′=32L^\{\\prime\}\{=\}32for all subsequentSigmaexperiments\. ## 4Language Generation via Continuous Diffusion Sampling Drawing on the success of continuous diffusion samplers using SDE/ODE solvers\([Song et al\., 2021b](https://arxiv.org/html/2610.02665#bib.bib26)\), we systematically explore decoding samplers for continuous dLMs, evaluating performance across standard mathematical reasoning and coding benchmarks\. ### 4\.1Block diffusion sampling To generate a response𝐫\{\\mathbf\{r\}\}given a prompt𝐩\{\\mathbf\{p\}\}, we denoise one block at a time\. Let𝐜:=\(𝐩,𝐫\(<b\)\)\{\\mathbf\{c\}\}:=\(\{\\mathbf\{p\}\},\{\\mathbf\{r\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)be the clean context available when generating blockbb, stored in an attention key\-value \(KV\) cache using the AR embeddings\. An offset block\-causal mask determines who can see whom \([Fig\.4\(a\)](https://arxiv.org/html/2610.02665#S4.F4.sf1)\)\. Positions in𝐜\{\\mathbf\{c\}\}use causal attention, so their KV can be cached and reused across diffusion steps, regardless of whether the prompt length is a multiple ofL′L^\{\\prime\}\. Positions in the noisy block𝐳t\(b\)\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}attend bidirectionally within the block and causally to the entire prefix𝐜\{\\mathbf\{c\}\}\. After a block is decoded, the AR embeddings of its committed tokens are appended to the causal KV cache for the next block during the next block’s first diffusion step, without incurring an extra NFE\. Sampling blockbbstarts from𝐳1\(b\)∼𝒩\(𝟎,𝐈\)\{\\mathbf\{z\}\}\_\{1\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\sim\\mathcal\{N\}\(\{\\mathbf\{0\}\},\{\\mathbf\{I\}\}\)and steps fromtttos<ts<talong a discretization of\[1,0\]\[1,0\]\. At each step the denoiser yields a predicted clean embedding𝐞^θ:=𝐞θ\(𝐳t\(b\),t,𝐜\)=𝐱θ\(𝐳t\(b\),t,𝐜\)𝐄\\hat\{\\mathbf\{e\}\}\_\{\\theta\}:=\{\\mathbf\{e\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\{\\mathbf\{c\}\}\)=\{\\mathbf\{x\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\{\\mathbf\{c\}\}\)\{\\mathbf\{E\}\}and the implied noise predictionϵ^θ:=1σt\(𝐳t\(b\)−αt𝐞^θ\)\\hat\{\\bm\{\\epsilon\}\}\_\{\\theta\}:=\\frac\{1\}\{\\sigma\_\{t\}\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\-\\alpha\_\{t\}\\hat\{\\mathbf\{e\}\}\_\{\\theta\}\)\. Below we introduce two canonical samplers for the reverse process\. For both samplers, after the final step, tokens are read out asargmax\\argmaxof the conditional logit output𝐱θ\(𝐳0\(b\),0,𝐜\)𝐄\{\\mathbf\{x\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{0\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},0;\{\\mathbf\{c\}\}\)\{\\mathbf\{E\}\}and appended to𝐜\{\\mathbf\{c\}\}\. See[Alg\.2](https://arxiv.org/html/2610.02665#alg2)for the complete sampling algorithm\. Note that, in subsequent analysis, we do not account for this extra decoding step as a part of the diffusion sampling budget\. ##### DDPM\. We use the ancestral kernel of VDM\([Kingma et al\., 2021](https://arxiv.org/html/2610.02665#bib.bib18), App\. A\.4\)\. It draws𝐳s\(b\)\{\\mathbf\{z\}\}\_\{s\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}from the closed\-form Gaussian posteriorq\(𝐳s\(b\)∣𝐳t\(b\),𝐱\(b\)\)q\(\{\\mathbf\{z\}\}\_\{s\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\mid\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\)with the clean embedding replaced by𝐞^θ\\hat\{\\mathbf\{e\}\}\_\{\\theta\}, 𝐳s\(b\)\\displaystyle\{\\mathbf\{z\}\}\_\{s\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}=\(1−c\)αsαt𝐳t\(b\)\+cαs𝐞^θ\+c\(1−αs2\)ϵ,ϵ∼𝒩\(𝟎,𝐈\),c=1−eγ\(s\)−γ\(t\)\.\\displaystyle=\(1\-c\)\\tfrac\{\\alpha\_\{s\}\}\{\\alpha\_\{t\}\}\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\+c\\,\\alpha\_\{s\}\\hat\{\\mathbf\{e\}\}\_\{\\theta\}\+\\sqrt\{c\(1\-\\alpha\_\{s\}^\{2\}\)\}\\,\{\\bm\{\\epsilon\}\},\\quad\{\\bm\{\\epsilon\}\}\\sim\\mathcal\{N\}\(\{\\mathbf\{0\}\},\{\\mathbf\{I\}\}\),\\penalty\\ c=1\-\{\\mathrm\{e\}\}^\{\\gamma\(s\)\-\\gamma\(t\)\}\.\(DDPM\) ##### DDIM\. One can also leverage the predictions𝐞^θ\\hat\{\\mathbf\{e\}\}\_\{\\theta\}andϵ^θ\\hat\{\\bm\{\\epsilon\}\}\_\{\\theta\}at𝐳t\(b\)\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}to deterministically update the state\([Song et al\., 2021a](https://arxiv.org/html/2610.02665#bib.bib31)\): 𝐳s\(b\)=αs𝐞^θ\+σsϵ^θ\.\\displaystyle\{\\mathbf\{z\}\}\_\{s\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}=\\alpha\_\{s\}\\hat\{\\mathbf\{e\}\}\_\{\\theta\}\+\\sigma\_\{s\}\\hat\{\\bm\{\\epsilon\}\}\_\{\\theta\}\.\(DDIM\) ### 4\.2Trajectory Steering To sharpen model predictions, we incorporate two reverse\-process steering methods to our blockwise continuous diffusion model with no impact to its training efficiency and minimal impact to its inference efficiency\. ##### Classifier\-free guidance \(CFG\)\. CFG\([Ho and Salimans, 2022](https://arxiv.org/html/2610.02665#bib.bib30)\)is the standard tool for class\-conditional or text\-to\-image diffusion models\. In CFG, one queries the model twice to obtain a conditional and an unconditional prediction and then linearly extrapolates them\. For our block diffusion model, the clean context𝐜\{\\mathbf\{c\}\}serves as the natural conditioning information, and CFG extrapolates away from the unconditional prediction for the current blockb\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}: 𝐞θw\(𝐳t\(b\),t,𝐜\):=𝐞θ\(𝐳t\(b\),t,∅\)\+w\[𝐞θ\(𝐳t\(b\),t,𝐜\)−𝐞θ\(𝐳t\(b\),t,∅\)\],\\displaystyle\{\\mathbf\{e\}\}^\{w\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\{\\mathbf\{c\}\}\):=\{\\mathbf\{e\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\\varnothing\)\+w\\,\\big\[\{\\mathbf\{e\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\{\\mathbf\{c\}\}\)\-\{\\mathbf\{e\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\\varnothing\)\\big\],\(CFG\)where∅\\varnothingdrops the entire clean context𝐜\{\\mathbf\{c\}\}\(so𝐳t\(b\)\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}attends only within its own block\) andw≥1w\\geq 1is the guidance strength \(w=1w=1falls back to the plain conditional prediction\)\. We bring two technical considerations\. First, at batch size11transformer inference is memory\-bound, so we pack conditional and unconditional predictions into a single forward pass at almost no wall\-clock cost\([Liu et al\., 2026b](https://arxiv.org/html/2610.02665#bib.bib39)\), using the packed mask of[Fig\.4\(a\)](https://arxiv.org/html/2610.02665#S4.F4.sf1)\. We measure this overhead to be less than5%5\\%per forward pass at block size3232and6464\([Sec\.G\.10](https://arxiv.org/html/2610.02665#A7.SS10)\)\. Second, unlike in image diffusion models and existing dLMs that require train\-time context dropout\([Potaptchik et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib44);[Hu et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib47)\),our CFG is a pure inference\-time trick: due to sequence packing in pre\-training, the first block typically starts in the middle of a document and naturally sees no context\. We discuss existing approaches in[App\.A](https://arxiv.org/html/2610.02665#A1)\. ##### Score temperature \(ST\)\. ST\([Dieleman et al\., 2022](https://arxiv.org/html/2610.02665#bib.bib24)\)is a heuristic that sharpens the conditional score itself, rescaling the score∇logpt\(𝐳t\(b\)\|𝐜\)\\nabla\\log p\_\{t\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\|\{\\mathbf\{c\}\}\)\(or equivalently, the noise prediction\) by1τ\\frac\{1\}\{\\tau\}forτ∈\(0,1\]\\tau\\in\(0,1\]\. One can show that it corresponds to the following operation on the mean prediction\. 𝐞θτ\(𝐳t\(b\),t,𝐜\):=𝐳t\(b\)αt\+1τ\[𝐞θ\(𝐳t\(b\),t,𝐜\)−𝐳t\(b\)αt\]\.\\displaystyle\{\\mathbf\{e\}\}^\{\\tau\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\{\\mathbf\{c\}\}\):=\\tfrac\{\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\}\{\\alpha\_\{t\}\}\+\\tfrac\{1\}\{\\tau\}\\big\[\{\\mathbf\{e\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\{\\mathbf\{c\}\}\)\-\\tfrac\{\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\}\{\\alpha\_\{t\}\}\\big\]\.\(ST\)Intuitively, since𝔼\[𝐳t\(b\)\|𝐱\(b\)\]=αt𝐞\(b\)\\E\[\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\|\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\]=\\alpha\_\{t\}\{\\mathbf\{e\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},1αt𝐳t\(b\)\\frac\{1\}\{\\alpha\_\{t\}\}\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}serves as the naive bad prediction to extrapolate from\. ST does not incur further model queries, andτ=1\\tau=1leaves the sampler unchanged\. We also recognize ST as a special case of Temporal Score Rescaling proposed in\([Xu et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib48)\)\. \(a\)Packed attention mask while generating blockbb\(L′=2L^\{\\prime\}=2\)\. Prefix𝐜\{\\mathbf\{c\}\}is causal \(KV\-cached\)\. Conditional𝐳t\(b\)\{\\mathbf\{z\}\}\_\{t\}^\{\(b\)\}attends to all of𝐜\{\\mathbf\{c\}\}and bidirectionally within the block; unconditional copy attends only to itself\.\(b\)One reverse stept→st\\to s\. CFG and ST mix the two denoisers into𝐞θw,τ\{\\mathbf\{e\}\}^\{w,\\tau\}\_\{\\theta\}; an update yields𝐳s\(b\)\{\\mathbf\{z\}\}\_\{s\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\. SC \(red\) feeds the previous conditional mean \(not sharpened by CFG or ST\) to the next conditional branch\. Figure 4:Inferencefor blockwise continuous diffusion: packed CFG attention \(a\) and one reverse step \(b\)\.Figure 5:CFG and ST tuningonSigma\-3B for DDPM sampling\. Combining both techniques yields larger gains over applying either in isolation, and markedly outperforms sampling without CFG or ST\. See[Fig\.10](https://arxiv.org/html/2610.02665#A7.F10)for the full tuning grid\.Finally, we remark that due to the commutativity of[CFG](https://arxiv.org/html/2610.02665#S4.Ex6)and[ST](https://arxiv.org/html/2610.02665#S4.Ex7), we can combine them into a single operation𝐞θw,τ\(𝐳t\(b\),t,𝐜\)\{\\mathbf\{e\}\}^\{w,\\tau\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\{\\mathbf\{c\}\}\), which is essentially using the score of the composed distribution\(pt\(⋅∣𝐜\)wpt\(⋅∣∅\)1−w\)1/τ\(p\_\{t\}\(\\cdot\\mid\{\\mathbf\{c\}\}\)^\{w\}p\_\{t\}\(\\cdot\\mid\\varnothing\)^\{1\-w\}\)^\{1/\\tau\}\([Prop\.1](https://arxiv.org/html/2610.02665#Thmproposition1)\)\. We observed that using either knob is weaker than using both of them, see[Fig\.5](https://arxiv.org/html/2610.02665#S4.F5)\. ##### Self\-conditioning \(SC\)\. At inference, SC is applied only to the conditional prediction: when querying the model at timess, the previous step’s conditional prediction𝐞θ\(𝐳t\(b\),t,𝐜\)\{\\mathbf\{e\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\{\\mathbf\{c\}\}\)\(before applying CFG and ST\) serves as the SC input for the conditional prediction, while the unconditional prediction receives𝟎\{\\mathbf\{0\}\}SC input \([Fig\.4\(b\)](https://arxiv.org/html/2610.02665#S4.F4.sf2)\)\. ##### Sharpening in logit space\. While both knobs above act on the mean embedding prediction obtained after the categorical reparameterization, we can also sharpen in logit space by applying the same two ideas before the softmax\. In[Sec\.G\.3](https://arxiv.org/html/2610.02665#A7.SS3), we show that the logit\-space CFG and temperature do provide sharpening, but the embedding\-space knobs provide stronger performance, hence we keep[CFG](https://arxiv.org/html/2610.02665#S4.Ex6)and[ST](https://arxiv.org/html/2610.02665#S4.Ex7)as the default option\. ### 4\.3Mathematical Reasoning and Coding Performance Table 1:Performance ofSigma\-3B \(base model; block size: 32\)on math reasoning and coding benchmarks\. Accuracy \(%\) is pass@1\. TheSigmadiffusion rows are reported as mean \(over 10 seeds\)±\\pmtwo standard errors\.Bold: the best value across all methods;underline: the best value within each mode\.MathCodingMethodGSM8KMinerva MathHumanEvalHumanEval\+MBPPMBPP\+Avg\.AR initialization83\.7066\.2851\.2248\.7860\.6077\.7864\.73*AR mode*Nemotron\-Labs\-Diffusion\-3B83\.7865\.0653\.0547\.5660\.8078\.8464\.85Sigma\-3B\(Ours\)83\.4065\.1057\.3251\.8359\.0076\.1965\.47*Diffusion mode*LLaDA\-8B70\.9627\.3032\.3227\.4440\.8051\.8541\.78Dream\-7B77\.1839\.6054\.8849\.3956\.8074\.6058\.74Nemotron\-Labs\-Diffusion\-3B83\.4063\.9656\.1053\.0557\.6074\.3464\.74Sigma\-3B\(Ours\)DDPM256256steps / block \(w=2,τ=0\.1w\{=\}2,\\tau\{=\}0\.1\)84\.50\(±\\pm0\.31\)64\.60\(±\\pm0\.28\)59\.45\(±\\pm1\.30\)53\.84\(±\\pm1\.29\)52\.32\(±\\pm0\.73\)71\.32\(±\\pm0\.91\)64\.34DDIM 32 steps / block \(w=3,τ=0\.3w\{=\}3,\\tau\{=\}0\.3\)81\.91\(±\\pm0\.32\)62\.76\(±\\pm0\.18\)56\.59\(±\\pm1\.31\)50\.67\(±\\pm1\.00\)47\.26\(±\\pm0\.46\)63\.49\(±\\pm0\.77\)60\.45DDIM 16 steps / block \(w=3,τ=0\.5w\{=\}3,\\tau\{=\}0\.5\)78\.08\(±\\pm0\.34\)60\.16\(±\\pm0\.21\)47\.50\(±\\pm1\.14\)43\.84\(±\\pm0\.94\)41\.84\(±\\pm1\.00\)59\.58\(±\\pm0\.98\)55\.17 Table 2:Performance ofSigma\-8B \(base model; block size: 32\)on math reasoning and coding benchmarks\. Accuracy \(%\) is pass@1\. TheSigmadiffusion rows are reported as mean \(over 10 seeds\)±\\pmtwo standard errors\.Bold: the best value across all methods;underline: the best value within each mode\.MathCodingMethodGSM8KMinerva MathHumanEvalHumanEval\+MBPPMBPP\+Avg\.AR initialization85\.9069\.1859\.1553\.0568\.2083\.3369\.80*AR mode*Llama\-3\.1\-8B54\.0618\.2235\.3728\.6648\.8061\.9041\.17Ministral3\-8B80\.2144\.5842\.6838\.4161\.6076\.9857\.41Qwen3\-8B86\.7352\.9464\.6356\.7169\.4083\.0768\.91Nemotron\-Labs\-Diffusion\-8B88\.2566\.0060\.3753\.0568\.2082\.5469\.73Sigma\-8B\(Ours\)86\.8866\.7251\.8348\.7864\.0080\.9566\.53*Diffusion mode*LLaDA\-8B70\.9627\.3032\.3227\.4440\.8051\.8541\.78Dream\-7B77\.1839\.6054\.8849\.3956\.8074\.6058\.74Nemotron\-Labs\-Diffusion\-8B87\.2665\.1662\.8057\.3267\.0081\.7570\.22Sigma\-8B\(Ours\)DDPM \(256 steps,w=2,τ=0\.1w=2,\\tau=0\.1\)89\.07\(±\\pm0\.28\)67\.47\(±\\pm0\.20\)61\.40\(±\\pm1\.73\)57\.44\(±\\pm1\.62\)64\.78\(±\\pm0\.54\)77\.83\(±\\pm0\.72\)69\.67DDIM \(32 steps,w=3,τ=0\.3w=3,\\tau=0\.3\)87\.78\(±\\pm0\.33\)67\.13\(±\\pm0\.15\)58\.54\(±\\pm1\.04\)53\.17\(±\\pm1\.27\)59\.54\(±\\pm0\.57\)74\.15\(±\\pm0\.43\)66\.72 We evaluate on two math and four coding benchmarks: GSM8K, Minerva Math, HumanEval, HumanEval\+, MBPP, and MBPP\+\. Results are reported in[Tab\.1](https://arxiv.org/html/2610.02665#S4.T1)\. In diffusion mode,Sigma\-3B uses DDPM sampling with 256 steps per block with EDMρ=7\\rho=7schedule,w=2,τ=0\.1w=2,\\tau=0\.1and SC\. See[Sec\.G\.1](https://arxiv.org/html/2610.02665#A7.SS1)for details on the harness, scoring pipeline, error bars, and the samplers for baselines\. ##### Results\. In AR mode,Sigma\-3B attains the best six\-task average across all methods; in diffusion mode, our DDPM\-256 run iscomparable to Nemotron\-Labs\-Diffusion\-3B on average and sets the best base\-model numbers on GSM8K, HumanEval, and HumanEval\+; it also outperforms LLaDA\-8B and Dream\-7B by22\.622\.6and5\.65\.6points on average respectively, despite being ~2×2\\timessmaller in parameters\. Empirically, we observelessleft\-to\-right bias within a block forSigmacompared with the confidence\-based masked dLM sampling \([Fig\.20](https://arxiv.org/html/2610.02665#A7.F20)\)\. To confirm the benefit of AR regularization \([Sec\.3\.2\.2](https://arxiv.org/html/2610.02665#S3.SS2.SSS2)\), we also trainedSigma\-3B with justℒBD\\mathcal\{L\}\_\{\\text\{BD\}\}[3](https://arxiv.org/html/2610.02665#S3.E3)withoutℒar\\mathcal\{L\}\_\{\\text\{ar\}\}[6](https://arxiv.org/html/2610.02665#S3.E6); results are uniformly worse across all tasks \([Fig\.11](https://arxiv.org/html/2610.02665#A7.F11)\)\. We refer readers to[Tab\.2](https://arxiv.org/html/2610.02665#S4.T2)for the same evaluation on 8B base models\. Figure 6:Pass@kkaccuracy on MBPP\.Askkgrows,Sigmaprefers higher ST and overtakes NLD afterk=32k=32\. ##### Pass@kkanalysis\. While pass@1 accuracy in[Tab\.1](https://arxiv.org/html/2610.02665#S4.T1)captures single\-draw quality, test\-time search and RL draw multiple rollouts for the same prompt, where diverse rollouts are needed for richer learning signals and increase the chance of discovering a correct solution\. Pass@kkdoes not directly measure diversity, but tests whether additional samples expand correct\-solution coverage\. We sweep CFG and ST forSigma\-3B and confidence threshold and sampling temperature for NLD, reporting each model’s best tested config at each budgetkk\. On MBPP,Sigmaovertakes NLD atk=32k=32and reaches91\.8%91\.8\\%versus87\.6%87\.6\\%atk=256k=256\([Fig\.6](https://arxiv.org/html/2610.02665#S4.F6)\)\. See[Sec\.G\.8](https://arxiv.org/html/2610.02665#A7.SS8)for the full study\. ## 5Few\-Step Generation Figure 7:Performance vs\. diffusion steps ofSigma\-3B andSigma\-SUSReg\-3B \(block size: 32\)\.All base models use DDIM sampling with uniform\-γ\\gammatime discretization, with best per\-NFE \(CFG, ST\) shared across the three tasks\. PDD models use their own DDIM\-based sampling and a common \(CFG, ST\) setting inherited at distillation, with self\-conditioning disabled in both the student and teacher\. Results on all six base tasks:[Tab\.8](https://arxiv.org/html/2610.02665#A7.T8)\.### 5\.1ODE Sampler and Time Discretization In previous results, we run 256 DDPM steps per block to establish the strongest performance\. In low\-NFE regime, it is generally known that deterministic samplers paired with well\-tuned time discretization lead to better performance\. To investigate the tradeoffs forSigma\-3B, we choose two representative tasks \(GSM8K and HumanEval\), and vary the sampler \(DDPM or DDIM\) and the time\-discretization across various diffusion sampling budgetsT∈\{4,8,16,32,64,128,256\}T\\in\\\{4,8,16,32,64,128,256\\\}\. For each sampler\-discretization\-budget combination, we sweep CFG and ST; we then report, for each combination, the CFG\-ST pair that achieves the best average accuracy over the two tasks\. ##### Results\. As shown in[Fig\.13](https://arxiv.org/html/2610.02665#A7.F13),DDIM substantially outperforms DDPM at low NFEswhile matching DDPM at high NFEs\. The uniform\-γ\\gammadiscretization \([Sec\.G\.6](https://arxiv.org/html/2610.02665#A7.SS6)\) performs the best for a wide range of budgets\. The best CFG\-ST pairs are shown in[Fig\.14](https://arxiv.org/html/2610.02665#A7.F14)\. Surprisingly, we found that higher\-order ODE solvers originally proposed for image diffusion models \(e\.g\.,[Zhang and Chen \(2023\)](https://arxiv.org/html/2610.02665#bib.bib5);[Lu et al\. \(2022\)](https://arxiv.org/html/2610.02665#bib.bib4)\) perform worse than DDIM, see[Sec\.G\.7](https://arxiv.org/html/2610.02665#A7.SS7)for the full study\. Using the same optimal hyperparameters from the two tasks, we evaluate the 16/32\-step performance across all six tasks in[Tabs\.1](https://arxiv.org/html/2610.02665#S4.T1)and[2](https://arxiv.org/html/2610.02665#S4.T2);[Tab\.8](https://arxiv.org/html/2610.02665#A7.T8)shows performance at other budgets\. ### 5\.2Embedding Regularization We observe that during training, embeddings collapse to ~4 effective dimensions in terms of principal components—a phenomenon not typically seen in standard autoregressive language model embeddings\. While the current model achieves strong high\-NFE performance, we hypothesize that regularizing the embeddings to be more uniformly distributed on the unit sphere will improve low\-NFE performance\. Our intuition is that representation collapse exacerbates token ambiguity, particularly when coarse numerical precision and low\-NFE sampling introduce prediction errors\. To mitigate this, we apply Sketched Uniform Spherical Regularization \(SUSReg,[Nicollier et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib80)\) to the embedding matrix𝐄\{\\mathbf\{E\}\}during pre\-training, penalizing the deviation between its empirical distribution and a uniform distribution on the unit sphere \(see[Sec\.E\.1](https://arxiv.org/html/2610.02665#A5.SS1)\)\. ##### Results\. As shown in[Figs\.7](https://arxiv.org/html/2610.02665#S5.F7)and[8](https://arxiv.org/html/2610.02665#A7.T8), the resultingSigma\-SUSReg\-3Bimproves few\-step and distillation resultscompared toSigma\-3B\. Interestingly, SUSReg brings nearly no penalty to the validation PPL, although the learned embedding geometry differs substantially from that learned without SUSReg \([Fig\.8](https://arxiv.org/html/2610.02665#A5.F8)\)\. ### 5\.3Parallel Decoding Distillation for Few\-step Generation Although DDIM has better performance than DDPM at reduced denoising steps, the pretrained model still loses quality at very low NFEs with a clear drop\-off below 16 NFEs \([Fig\.7](https://arxiv.org/html/2610.02665#S5.F7)\)\. We therefore distill the many\-step trajectory ofSigma\-3B into a few\-step student\. Our method adapts Parallel Decoding Distillation \(PDD,[Shaul et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib72)\) to blockwise continuous language diffusion, see[App\.I](https://arxiv.org/html/2610.02665#A9)for full details\. Due to the parallel nature of the student PDD heads, they cannot utilize recurrent computation, so we distill from the teacher without self\-conditioning \(\-SC\)\. During distillation, we amortize CFG into our student by training the single conditional student call to match the two\-call guided teacher directly, which reduces FLOPs during sampling\. ##### Results\. As shown in[Figs\.7](https://arxiv.org/html/2610.02665#S5.F7)and[8](https://arxiv.org/html/2610.02665#A7.T8),PDD is most helpful at 4 diffusion NFEsandpre\-training with SUSReg improves PDD performance\. ## 6Supervised Fine\-Tuning We apply supervised fine\-tuning \(SFT\) to the pre\-trained baseSigmawith the same training objective used during pre\-training, restricting to data on assistant responses\. The complete data processing, optimization recipe, and continuous\-diffusion\-specific controls are given in[App\.H](https://arxiv.org/html/2610.02665#A8)\. ##### Results\. As shown in[Tabs\.3](https://arxiv.org/html/2610.02665#S6.T3)and[4](https://arxiv.org/html/2610.02665#S6.T4), the average performance ofSigma\-Instruct ison par with NLD at both 3B and 8B scale, andsurpasses other open\-source masked diffusion models like Dream and LLaDA at the 8B scale\. For each sampler\-step combination, we re\-use the best CFG\-ST configuration found in[Secs\.4](https://arxiv.org/html/2610.02665#S4)and[5](https://arxiv.org/html/2610.02665#S5), indicating sampling hyperparameters are reasonably robust to SFT and scale\. Table 3:Performance ofSigma\-3B\-Instruct \(block size: 32\)\. Accuracy \(%\) is pass@1\. TheSigmadiffusion rows are reported as mean \(over 10 seeds\)±\\pmtwo SEs\. The four reference models at the top are AR models\. Baseline numbers are from[Fu et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib51)\. In*diffusion mode*,underlinemarks best value per task\.MathCodingScience & KnowledgeIFMethodGSM8KMath500AIME24AIME25HumanEvalMBPPLCB\-CPPMMLUGPQAIFEvalAvg\.Llama\-3\.2\-3B\-Instruct62\.0932\.600\.000\.0065\.8566\.939\.2562\.4527\.7869\.6939\.66Phi\-4\-mini\-Instruct76\.5770\.8010\.006\.6777\.4475\.139\.9153\.3740\.9171\.1649\.20Ministral3\-3B\-Instruct\-251287\.4171\.9012\.5012\.7164\.7968\.3912\.5063\.2128\.7957\.5547\.98Qwen3\-4B92\.1968\.4010\.6310\.6375\.9172\.4915\.5877\.0537\.1872\.2053\.23*AR mode*Nemotron\-Labs\-Diffusion\-3B\-Instruct87\.8777\.6023\.3316\.6776\.2271\.1621\.3771\.7039\.3969\.3955\.47Sigma\-3B\-Instruct \(Ours\)87\.1173\.0010\.006\.6772\.5672\.7513\.6671\.2438\.3866\.5151\.19*Diffusion mode*Nemotron\-Labs\-Diffusion\-3B\-Instruct88\.4074\.8016\.6710\.0074\.3973\.5416\.3072\.0633\.8468\.9352\.89Sigma\-3B\-Instruct \(Ours\)DDPM \(256 steps,w=2,τ=0\.1w\{=\}2,\\tau\{=\}0\.1\)52\.15DDIM \(32 steps,w=3,τ=0\.3w\{=\}3,\\tau\{=\}0\.3\)49\.86 Table 4:Performance ofSigma\-8B\-Instruct \(block size: 32\)\. Accuracy \(%\) is pass@1\. TheSigmadiffusion rows are reported as mean \(over 10 seeds\)±\\pmtwo SEs\. The three reference models at the top are AR models\. NLD\-8B\-Instruct’s diffusion mode results are reproduced under aligned protocol, while other baseline numbers are from[Fu et al\. \(2026, Tab\. 5\)](https://arxiv.org/html/2610.02665#bib.bib51)\. In*diffusion mode*,underlinemarks best value per task\.MathCodingScience & KnowledgeIFMethodGSM8KMath500AIME24AIME25HumanEvalMBPPLCB\-CPPMMLUGPQAIFEvalAvg\.Qwen2\.5\-7B\-Instruct91\.8975\.1013\.756\.8877\.4481\.5512\.3374\.8637\.1274\.5854\.55Qwen3\-8B92\.4284\.8030\.2122\.0881\.7181\.8821\.0976\.6649\.2487\.3862\.75Ministral3\-8B\-Instruct\-251292\.4283\.6027\.7124\.5871\.0478\.9720\.7673\.9042\.8764\.3158\.02*AR mode*Nemotron\-Labs\-Diffusion\-8B\-Instruct94\.0188\.0033\.3333\.3380\.4985\.1928\.8579\.8544\.4468\.6563\.61Sigma\-8B\-Instruct \(Ours\)91\.9678\.6016\.6713\.3376\.2283\.0721\.8177\.8943\.9480\.4658\.39*Diffusion mode*LLaDA\-8B Instruct79\.9139\.200\.000\.0049\.4041\.004\.1965\.5033\.3059\.9037\.24Dream\-7B Instruct81\.0043\.000\.003\.3355\.5058\.801\.2567\.0033\.0062\.5040\.54SDAR\-8B Chat91\.3078\.6016\.6710\.0078\.7072\.0013\.4478\.6040\.2061\.4054\.09Nemotron\-Labs\-Diffusion\-8B\-Instruct91\.5884\.6030\.0023\.3383\.5482\.0121\.1576\.6533\.3373\.0259\.92Sigma\-8B\-Instruct \(Ours\)DDPM \(256 steps,w=2,τ=0\.1w\{=\}2,\\tau\{=\}0\.1\)59\.27DDIM \(32 steps,w=3,τ=0\.3w\{=\}3,\\tau\{=\}0\.3\)57\.01 ## 7Conclusions We introducedSigma, a blockwise continuous dLM via efficient continual pre\-training from an AR checkpoint, together with a reverse\-diffusion sampling recipe that combines DDPM/DDIM samplers with CFG, ST, and SC\.Sigma\(3B/8B\), in diffusion mode, performs comparably with the same\-size masked dLM \(Nemotron\-Labs\-Diffusion\) trained under an aligned pipeline\. The same conclusions generally transfers to post\-SFT\. Moreover, we adapt a sphere\-normalized embedding regularization technique and few\-step distillation toSigma, which together enable non\-trivial 4/8\-step results\. We discuss limitations and future directions in[App\.B](https://arxiv.org/html/2610.02665#A2)\. ## References - Agarwalet al\.\(2025\)N\. Agarwal, A\. Ali, M\. Bala, Y\. Balaji, E\. Barker, T\. Cai, P\. Chattopadhyay, Y\. Chen, Y\. Cui, Y\. Ding,et al\.Cosmos world foundation model platform for physical AI\.arXiv preprint arXiv:2501\.03575\.Cited by:[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px4.p1.1)\. - Arriolaet al\.\(2025\)M\. Arriola, A\. Gokaslan, J\. T\. Chiu, Z\. Yang, Z\. Qi, J\. Han, S\. S\. Sahoo, and V\. KuleshovBlock diffusion: interpolating between autoregressive and diffusion language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=tyEyYT267x)Cited by:[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px7.p1.1),[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px9.p1.1),[§3\.1](https://arxiv.org/html/2610.02665#S3.SS1.p1.1)\. - Austinet al\.\(2021a\)J\. Austin, D\. D\. Johnson, J\. Ho, D\. Tarlow, and R\. van den BergStructured denoising diffusion models in discrete state\-spaces\.InAdvances in Neural Information Processing Systems,M\. Ranzato, A\. Beygelzimer, Y\. Dauphin, P\.S\. Liang, and J\. W\. Vaughan \(Eds\.\),Vol\.34,pp\. 17981–17993\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2021/file/958c530554f78bcd8e97125b70e6973d-Paper.pdf)Cited by:[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px1.p1.1)\. - Austinet al\.\(2021b\)J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le,et al\.Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[§G\.1](https://arxiv.org/html/2610.02665#A7.SS1.SSS0.Px1.p1.1)\. - Avdeyevet al\.\(2023\)P\. Avdeyev, C\. Shi, Y\. Tan, K\. Dudnyk, and J\. ZhouDirichlet diffusion score model for biological sequence generation\.InProceedings of the 40th International Conference on Machine Learning,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202,pp\. 1276–1301\.External Links:[Link](https://proceedings.mlr.press/v202/avdeyev23a.html)Cited by:[3rd item](https://arxiv.org/html/2610.02665#A1.I1.i3.p1.1)\. - Basantet al\.\(2025\)A\. Basant, A\. Khairnar, A\. Paithankar, A\. Khattar, A\. Renduchintala, A\. Malte, A\. Bercovich, A\. Hazare, A\. Rico, A\. Ficek,et al\.Nvidia nemotron nano 2: an accurate and efficient hybrid mamba\-transformer reasoning model\.arXiv preprint arXiv:2508\.14444\.Cited by:[§3\.2\.1](https://arxiv.org/html/2610.02665#S3.SS2.SSS1.Px1.p1.1)\. - Batzoliset al\.\(2026\)G\. Batzolis, M\. Girolami, and L\. AmbrogioniCoBit: language modeling with bitstream diffusion\.arXiv preprint arXiv:2605\.07013\.Cited by:[4th item](https://arxiv.org/html/2610.02665#A1.I1.i4.p1.1)\. - Bieet al\.\(2025\)T\. Bie, M\. Cao, K\. Chen, L\. Du, M\. Gong, Z\. Gong, Y\. Gu, J\. Hu, Z\. Huang, Z\. Lan,et al\.LLaDA2\.0: scaling up diffusion language models to 100B\.arXiv preprint arXiv:2512\.15745\.Cited by:[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px9.p1.1)\. - Bisket al\.\(2020\)Y\. Bisk, R\. Zellers, R\. Le Bras, J\. Gao, and Y\. ChoiPiqa: reasoning about physical commonsense in natural language\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[Appendix F](https://arxiv.org/html/2610.02665#A6.SS0.SSS0.Px1.p1.1)\. - Blakemanet al\.\(2025\)A\. Blakeman, A\. Grattafiori, A\. Basant, A\. Gupta, A\. Khattar, A\. Renduchintala, A\. Vavre, A\. Shukla, A\. Bercovich, A\. Ficek,et al\.Nemotron 3 Nano: open, efficient mixture\-of\-experts hybrid mamba\-transformer model for agentic reasoning\.arXiv preprint arXiv:2512\.20848\.Cited by:[§H\.1](https://arxiv.org/html/2610.02665#A8.SS1.SSS0.Px2.p1.1)\. - Boffiet al\.\(2025\)N\. M\. Boffi, M\. S\. Albergo, and E\. Vanden\-EijndenHow to build a consistency model: learning flow maps via self\-distillation\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=Di5apl8HSH)Cited by:[Appendix B](https://arxiv.org/html/2610.02665#A2.SS0.SSS0.Px5.p1.1)\. - Chandiramaniet al\.\(2026\)A\. Chandiramani, A\. Blakeman, A\. Olaoye, A\. Gupta, A\. Somasamudramath, A\. Khattar, A\. Adesoba, A\. Renduchintala, A\. Asif, A\. Agrawal,et al\.Nemotron 3 super: open, efficient mixture\-of\-experts hybrid mamba\-transformer model for agentic reasoning\.arXiv preprint arXiv:2604\.12374\.Cited by:[§H\.1](https://arxiv.org/html/2610.02665#A8.SS1.SSS0.Px2.p1.1)\. - Changet al\.\(2022\)H\. Chang, H\. Zhang, L\. Jiang, C\. Liu, and W\. T\. FreemanMaskGIT: masked generative image transformer\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,Cited by:[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px6.p1.1)\. - Chemseddineet al\.\(2026\)J\. Chemseddine, G\. Kornhardt, and G\. SteidlSpherical flows for sampling categorical data\.arXiv preprint arXiv:2605\.05629\.Cited by:[1st item](https://arxiv.org/html/2610.02665#A1.I1.i1.p1.1)\. - Chenet al\.\(2024\)B\. Chen, D\. M\. Monsó, Y\. Du, M\. Simchowitz, R\. Tedrake, and V\. SitzmannDiffusion forcing: next\-token prediction meets full\-sequence diffusion\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=yDo1ynArjj)Cited by:[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px4.p1.1)\. - Chenet al\.\(2026a\)H\. Chen, K\. Zhang, H\. Tan, L\. Guibas, G\. Wetzstein, and S\. Bipi\-Flow: policy\-based few\-step generation via imitation distillation\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=1vAte8dsap)Cited by:[Appendix B](https://arxiv.org/html/2610.02665#A2.SS0.SSS0.Px5.p1.1)\. - Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. D\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§G\.1](https://arxiv.org/html/2610.02665#A7.SS1.SSS0.Px1.p1.1)\. - Chenet al\.\(2023\)T\. Chen, R\. Zhang, and G\. HintonAnalog bits: generating discrete data using diffusion models with self\-conditioning\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=3itjR9QxFw)Cited by:[4th item](https://arxiv.org/html/2610.02665#A1.I1.i4.p1.1),[§2](https://arxiv.org/html/2610.02665#S2.SS0.SSS0.Px5.p1.1)\. - Chenet al\.\(2026b\)Y\. Chen, C\. Liang, H\. Sui, R\. Guo, C\. Cheng, J\. You, and G\. LiuLangFlow: continuous diffusion rivals discrete in language modeling\.arXiv preprint arXiv:2604\.11748\.Cited by:[1st item](https://arxiv.org/html/2610.02665#A1.I1.i1.p1.1)\. - Chenget al\.\(2024\)C\. Cheng, J\. Li, J\. Peng, and G\. LiuCategorical flow matching on statistical manifolds\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=5fybcQZ0g4)Cited by:[3rd item](https://arxiv.org/html/2610.02665#A1.I1.i3.p1.1)\. - Chenget al\.\(2026\)S\. Cheng, Y\. Bian, D\. Liu, Y\. Jiang, Y\. Liu, L\. Zhang, Q\. Yao, Z\. Tian, W\. Wang, Q\. Guo, K\. Chen, B\. Qi, and B\. ZhouSDAR: a synergistic diffusion\-AutoRegression paradigm for scalable sequence generation\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 22058–22075\.External Links:[Link](https://aclanthology.org/2026.findings-acl.1110/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.1110),ISBN 979\-8\-89176\-395\-1Cited by:[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px9.p1.1),[§1](https://arxiv.org/html/2610.02665#S1.p4.1)\. - Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have solved question answering? try ARC, the AI2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[Appendix F](https://arxiv.org/html/2610.02665#A6.SS0.SSS0.Px1.p1.1)\. - Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§G\.1](https://arxiv.org/html/2610.02665#A7.SS1.SSS0.Px1.p1.1)\. - Daviset al\.\(2026\)O\. Davis, A\. Filippova, P\. Ablin, V\. Turrisi, A\. Shidani, M\. Cuturi, and L\. BéthuneScaling categorical flow maps\.arXiv preprint arXiv:2605\.07820\.Cited by:[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px2.p1.1)\. - Daviset al\.\(2024\)O\. Davis, S\. Kessler, M\. Petrache, I\. I\. Ceylan, M\. M\. Bronstein, and J\. BoseFisher flow matching for generative modeling over discrete data\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=6jOScqwdHU)Cited by:[3rd item](https://arxiv.org/html/2610.02665#A1.I1.i3.p1.1)\. - Deschenaux and Gulcehre \(2026\)J\. Deschenaux and C\. GulcehreLanguage modeling with hyperspherical flows\.arXiv preprint arXiv:2605\.11125\.Cited by:[1st item](https://arxiv.org/html/2610.02665#A1.I1.i1.p1.1)\. - Dielemanet al\.\(2022\)S\. Dieleman, L\. Sartran, A\. Roshannai, N\. Savinov, Y\. Ganin, P\. H\. Richemond, A\. Doucet, R\. Strudel, C\. Dyer, C\. Durkan,et al\.Continuous diffusion for categorical data\.arXiv preprint arXiv:2211\.15089\.Cited by:[1st item](https://arxiv.org/html/2610.02665#A1.I1.i1.p1.1),[§1](https://arxiv.org/html/2610.02665#S1.p2.1),[§4\.2](https://arxiv.org/html/2610.02665#S4.SS2.SSS0.Px2.p1.1)\. - Fuet al\.\(2026\)Y\. Fu, L\. Whalen, A\. Garg, C\. Wu, M\. Khadkevich, N\. Oswald, E\. Xie, D\. Egert, S\. T\. Sreenivas, S\. Diao,et al\.Nemotron\-labs\-diffusion: a tri\-mode language model unifying autoregressive, diffusion, and self\-speculation decoding\.arXiv preprint arXiv:2607\.05722\.Cited by:[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px2.p1.1),[2nd item](https://arxiv.org/html/2610.02665#A3.I2.i2.p1.1),[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px6.p1.1),[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px8.p1.2),[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px9.p1.1),[Appendix E](https://arxiv.org/html/2610.02665#A5.SS0.SSS0.Px1.p1.1),[§E\.2](https://arxiv.org/html/2610.02665#A5.SS2.p2.1),[Table 5](https://arxiv.org/html/2610.02665#A5.T5),[Table 5](https://arxiv.org/html/2610.02665#A5.T5.8),[Appendix F](https://arxiv.org/html/2610.02665#A6.SS0.SSS0.Px3.p2.1),[§G\.1](https://arxiv.org/html/2610.02665#A7.SS1.SSS0.Px1.p1.1),[§G\.1](https://arxiv.org/html/2610.02665#A7.SS1.SSS0.Px5.p1.1),[§G\.8\.1](https://arxiv.org/html/2610.02665#A7.SS8.SSS1.Px1.p1.1),[§H\.1](https://arxiv.org/html/2610.02665#A8.SS1.p1.1),[§H\.2](https://arxiv.org/html/2610.02665#A8.SS2.SSS0.Px3.p1.1),[§H\.2](https://arxiv.org/html/2610.02665#A8.SS2.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2610.02665#S1.p1.1),[§1](https://arxiv.org/html/2610.02665#S1.p4.1),[§3\.1](https://arxiv.org/html/2610.02665#S3.SS1.p1.1),[§3\.2](https://arxiv.org/html/2610.02665#S3.SS2.p1.1),[Table 3](https://arxiv.org/html/2610.02665#S6.T3),[Table 3](https://arxiv.org/html/2610.02665#S6.T3.8),[Table 4](https://arxiv.org/html/2610.02665#S6.T4),[Table 4](https://arxiv.org/html/2610.02665#S6.T4.8)\. - Genget al\.\(2025\)Z\. Geng, M\. Deng, X\. Bai, J\. Z\. Kolter, and K\. HeMean flows for one\-step generative modeling\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=uWj4s7rMnR)Cited by:[Appendix B](https://arxiv.org/html/2610.02665#A2.SS0.SSS0.Px5.p1.1)\. - Gonget al\.\(2025\)S\. Gong, S\. Agarwal, Y\. Zhang, J\. Ye, L\. Zheng, M\. Li, C\. An, P\. Zhao, W\. Bi, J\. Han, H\. Peng, and L\. KongScaling diffusion language models via adaptation from autoregressive models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=j1tSLYKwg8)Cited by:[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px9.p1.1)\. - Gulrajani and Hashimoto \(2023\)I\. Gulrajani and T\. HashimotoLikelihood\-based diffusion language models\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=e2MCL6hObn)Cited by:[1st item](https://arxiv.org/html/2610.02665#A1.I1.i1.p1.1),[Appendix E](https://arxiv.org/html/2610.02665#A5.SS0.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2610.02665#S1.p2.1),[§2](https://arxiv.org/html/2610.02665#S2.SS0.SSS0.Px2.p1.1)\. - Guoet al\.\(2026\)H\. Guo, Q\. Zhao, Y\. Zhao, S\. Nie, R\. Zhu, Q\. Guo, F\. Wang, T\. Yang, H\. Zhao, G\. Wei,et al\.Continuous latent diffusion language model\.arXiv preprint arXiv:2605\.06548\.Cited by:[2nd item](https://arxiv.org/html/2610.02665#A1.I1.i2.p1.1),[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px2.p1.1),[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.02665#S1.p2.1)\. - Hanet al\.\(2023\)X\. Han, S\. Kumar, and Y\. TsvetkovSSD\-LM: semi\-autoregressive simplex\-based diffusion language model for text generation and modular control\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 11575–11596\.External Links:[Link](https://aclanthology.org/2023.acl-long.647/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.647)Cited by:[3rd item](https://arxiv.org/html/2610.02665#A1.I1.i3.p1.1)\. - Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[Appendix F](https://arxiv.org/html/2610.02665#A6.SS0.SSS0.Px1.p1.1)\. - Hoet al\.\(2022\)J\. Ho, T\. Salimans, A\. Gritsenko, W\. Chan, M\. Norouzi, and D\. FleetVideo diffusion models\.InAdvances in Neural Information Processing Systems,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),Vol\.35,pp\. 8633–8646\.External Links:[Document](https://dx.doi.org/10.52202/068431-0628),[Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/39235c56aef13fb05a6adc95eb9d8d66-Paper-Conference.pdf)Cited by:[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px4.p1.1)\. - Ho and Salimans \(2022\)J\. Ho and T\. SalimansClassifier\-free diffusion guidance\.External Links:2207\.12598,[Link](https://arxiv.org/abs/2207.12598)Cited by:[§G\.4](https://arxiv.org/html/2610.02665#A7.SS4.p2.1),[§4\.2](https://arxiv.org/html/2610.02665#S4.SS2.SSS0.Px1.p1.1)\. - Hoogeboomet al\.\(2021\)E\. Hoogeboom, D\. Nielsen, P\. Jaini, P\. Forré, and M\. WellingArgmax flows and multinomial diffusion: learning categorical distributions\.InAdvances in Neural Information Processing Systems,A\. Beygelzimer, Y\. Dauphin, P\. Liang, and J\. W\. Vaughan \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=6nbpPqUCIi7)Cited by:[4th item](https://arxiv.org/html/2610.02665#A1.I1.i4.p1.1)\. - Horvitz and Thompson \(1952\)D\. G\. Horvitz and D\. J\. ThompsonA generalization of sampling without replacement from a finite universe\.Journal of the American statistical Association47\(260\),pp\. 663–685\.Cited by:[§3\.1](https://arxiv.org/html/2610.02665#S3.SS1.SSS0.Px1.p1.1)\. - Huet al\.\(2026\)K\. Hu, L\. Qiu, Y\. Lu, H\. Zhao, T\. Li, Y\. Kim, J\. Andreas, and K\. HeELF: embedded language flows\.arXiv preprint arXiv:2605\.10938\.Cited by:[2nd item](https://arxiv.org/html/2610.02665#A1.I1.i2.p1.1),[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px3.p1.1),[§G\.4](https://arxiv.org/html/2610.02665#A7.SS4.p2.1),[§1](https://arxiv.org/html/2610.02665#S1.p2.1),[§4\.2](https://arxiv.org/html/2610.02665#S4.SS2.SSS0.Px1.p1.2)\. - Huanget al\.\(2025\)X\. Huang, Z\. Li, G\. He, M\. Zhou, and E\. ShechtmanSelf forcing: bridging the train\-test gap in autoregressive video diffusion\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=mSiN7i0BYH)Cited by:[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px4.p1.1)\. - Inception Labset al\.\(2025\)Inception Labs, S\. Khanna, S\. Kharbanda, S\. Li, H\. Varma, E\. Wang, S\. Birnbaum, Z\. Luo, Y\. Miraoui, A\. Palrecha,et al\.Mercury: ultra\-fast language models based on diffusion\.arXiv preprint arXiv:2506\.17298\.Cited by:[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px9.p1.1),[§1](https://arxiv.org/html/2610.02665#S1.p1.1)\. - Jo and Hwang \(2025\)J\. Jo and S\. J\. HwangContinuous diffusion model for language modeling\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=VGv5y60sXC)Cited by:[3rd item](https://arxiv.org/html/2610.02665#A1.I1.i3.p1.1)\. - Karraset al\.\(2022\)T\. Karras, M\. Aittala, T\. Aila, and S\. LaineElucidating the design space of diffusion\-based generative models\.InAdvances in Neural Information Processing Systems,A\. H\. Oh, A\. Agarwal, D\. Belgrave, and K\. Cho \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=k7FuTOWMOc7)Cited by:[3rd item](https://arxiv.org/html/2610.02665#A7.I1.i3.p1.1)\. - Karraset al\.\(2024\)T\. Karras, M\. Aittala, T\. Kynkäänniemi, J\. Lehtinen, T\. Aila, and S\. LaineGuiding a diffusion model with a bad version of itself\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=bg6fVPVs3s)Cited by:[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px3.p1.1),[§G\.4](https://arxiv.org/html/2610.02665#A7.SS4.p2.1)\. - Kingmaet al\.\(2021\)D\. Kingma, T\. Salimans, B\. Poole, and J\. HoVariational diffusion models\.Advances in neural information processing systems34,pp\. 21696–21707\.External Links:[Link](https://openreview.net/forum?id=2LdBqxc1Yv)Cited by:[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px3.p1.2),[Appendix D](https://arxiv.org/html/2610.02665#A4.p4.1),[§2](https://arxiv.org/html/2610.02665#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.02665#S2.SS0.SSS0.Px2.p1.2),[§2](https://arxiv.org/html/2610.02665#S2.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2610.02665#S4.SS1.SSS0.Px1.p1.1)\. - Leeet al\.\(2026\)C\. Lee, J\. Yoo, M\. Agarwal, S\. Shah, J\. Huang, A\. Raghunathan, S\. Hong, N\. M\. Boffi, and J\. KimFlow map language models: one\-step language modeling via continuous denoising\.arXiv preprint arXiv:2602\.16813\.Cited by:[1st item](https://arxiv.org/html/2610.02665#A1.I1.i1.p1.1),[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px3.p1.1),[§G\.4](https://arxiv.org/html/2610.02665#A7.SS4.p2.1),[§1](https://arxiv.org/html/2610.02665#S1.p2.1)\. - Lewkowyczet al\.\(2022\)A\. Lewkowycz, A\. J\. Andreassen, D\. Dohan, E\. Dyer, H\. Michalewski, V\. V\. Ramasesh, A\. Slone, C\. Anil, I\. Schlag, T\. Gutman\-Solo, Y\. Wu, B\. Neyshabur, G\. Gur\-Ari, and V\. MisraSolving quantitative reasoning problems with language models\.InAdvances in Neural Information Processing Systems,A\. H\. Oh, A\. Agarwal, D\. Belgrave, and K\. Cho \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=IFXTZERXdM7)Cited by:[§G\.1](https://arxiv.org/html/2610.02665#A7.SS1.SSS0.Px1.p1.1)\. - Liet al\.\(2025\)P\. Li, S\. Yan, J\. Cai, R\. Zhang, R\. An, Z\. Guo, and X\. GaoAdaptive classifier\-free guidance via dynamic low\-confidence masking\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=0EcomthIKh)Cited by:[§G\.3](https://arxiv.org/html/2610.02665#A7.SS3.p2.1)\. - Liet al\.\(2022\)X\. L\. Li, J\. Thickstun, I\. Gulrajani, P\. Liang, and T\. HashimotoDiffusion\-LM improves controllable text generation\.InAdvances in Neural Information Processing Systems,A\. H\. Oh, A\. Agarwal, D\. Belgrave, and K\. Cho \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=3s9IrEsjLyk)Cited by:[1st item](https://arxiv.org/html/2610.02665#A1.I1.i1.p1.1)\. - Lianget al\.\(2026\)J\. Liang, Y\. Liao, Y\. Cao, J\. Wei, K\. Li, W\. Tan, J\. Zhang, Z\. Cui, J\. Yang, L\. Guo,et al\.AURORA\-LM: autoencoding unified representation for continuous\-latent diffusion language modeling\.arXiv preprint arXiv:2608\.02602\.Cited by:[2nd item](https://arxiv.org/html/2610.02665#A1.I1.i2.p1.1),[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px2.p1.1),[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.02665#S1.p2.1)\. - Liuet al\.\(2026a\)A\. H\. Liu, K\. Khandelwal, S\. Subramanian, V\. Jouault, A\. Rastogi, A\. Sadé, A\. Jeffares, A\. Jiang, A\. Cahill, A\. Gavaudan,et al\.Ministral 3\.arXiv preprint arXiv:2601\.08584\.Cited by:[Appendix E](https://arxiv.org/html/2610.02665#A5.SS0.SSS0.Px1.p1.1)\. - Liuet al\.\(2026b\)J\. Liu, X\. Dong, Z\. Ye, R\. Mehta, Y\. Fu, V\. Singh, C\. Zhang, and P\. MolchanovTiDAR: think in diffusion, talk in autoregression\.InProceedings of Machine Learning and Systems,A\. Chowdhery and Z\. Jia \(Eds\.\),Vol\.8,pp\. 748–762\.External Links:[Link](https://proceedings.mlsys.org/paper_files/paper/2026/file/1367d856028f65a9555b0274db09e608-Paper-Conference.pdf)Cited by:[§4\.2](https://arxiv.org/html/2610.02665#S4.SS2.SSS0.Px1.p1.2)\. - Lovelaceet al\.\(2023\)J\. Lovelace, V\. Kishore, C\. Wan, E\. S\. Shekhtman, and K\. Q\. WeinbergerLatent diffusion for language generation\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=NKdtztladR)Cited by:[2nd item](https://arxiv.org/html/2610.02665#A1.I1.i2.p1.1)\. - Luet al\.\(2022\)C\. Lu, Y\. Zhou, F\. Bao, J\. Chen, C\. Li, and J\. ZhuDPM\-solver: a fast ODE solver for diffusion probabilistic model sampling in around 10 steps\.InAdvances in Neural Information Processing Systems,A\. H\. Oh, A\. Agarwal, D\. Belgrave, and K\. Cho \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=2uAaGwlP_V)Cited by:[§1](https://arxiv.org/html/2610.02665#S1.p1.1),[§5\.1](https://arxiv.org/html/2610.02665#S5.SS1.SSS0.Px1.p1.1)\. - Luet al\.\(2025\)C\. Lu, Y\. Zhou, F\. Bao, J\. Chen, C\. Li, and J\. ZhuDPM\-Solver\+\+: fast solver for guided sampling of diffusion probabilistic models\.Machine Intelligence Research22\(4\),pp\. 730–751\.Cited by:[§G\.7](https://arxiv.org/html/2610.02665#A7.SS7.SSS0.Px1.p3.1)\. - Menget al\.\(2022\)C\. Meng, Y\. He, Y\. Song, J\. Song, J\. Wu, J\. Zhu, and S\. ErmonSDEdit: guided image synthesis and editing with stochastic differential equations\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=aBsCjcPu_tE)Cited by:[§1](https://arxiv.org/html/2610.02665#S1.p1.1)\. - Meshchaninovet al\.\(2025\)V\. Meshchaninov, E\. Chimbulatov, A\. Shabalin, A\. Abramov, and D\. VetrovCosmos: compressed and smooth latent space for text diffusion modeling\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=Rv6Lz84FlZ)Cited by:[2nd item](https://arxiv.org/html/2610.02665#A1.I1.i2.p1.1)\. - Meshchaninovet al\.\(2026\)V\. Meshchaninov, A\. Shabalin, E\. Chimbulatov, N\. Gushchin, I\. Koziev, A\. Korotin, and D\. VetrovHow to train your latent diffusion language model jointly with the latent space\.arXiv preprint arXiv:2605\.07933\.Cited by:[2nd item](https://arxiv.org/html/2610.02665#A1.I1.i2.p1.1)\. - Mistral AI \(2024\)Mistral AIMistral NeMo\.Note:[https://mistral\.ai/news/mistral\-nemo/](https://mistral.ai/news/mistral-nemo/)Accessed: 2026\-09\-14Cited by:[§H\.1](https://arxiv.org/html/2610.02665#A8.SS1.SSS0.Px2.p1.1),[§3\.2\.1](https://arxiv.org/html/2610.02665#S3.SS2.SSS1.Px1.p1.1)\. - Nicollieret al\.\(2026\)L\. Nicollier, M\. Dunitz, M\. Pic, P\. Musé, E\. Meinhardt\-Llopis, and G\. FaccioloSPHERE\-JEPA: spherical prediction with homogeneous embeddings\.arXiv preprint arXiv:2605\.26900\.Cited by:[§E\.1](https://arxiv.org/html/2610.02665#A5.SS1.p2.1),[§5\.2](https://arxiv.org/html/2610.02665#S5.SS2.p1.1)\. - Nieet al\.\(2025a\)S\. Nie, F\. Zhu, C\. Du, T\. Pang, Q\. Liu, G\. Zeng, M\. Lin, and C\. LiScaling up masked diffusion models on text\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=WNvvwK0tut)Cited by:[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px9.p1.1)\. - Nieet al\.\(2025b\)S\. Nie, F\. Zhu, Z\. You, X\. Zhang, J\. Ou, J\. Hu, J\. Zhou, Y\. Lin, J\. Wen, and C\. LiLarge language diffusion models\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=KnqiC0znVF)Cited by:[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px2.p1.1),[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px3.p1.1),[1st item](https://arxiv.org/html/2610.02665#A3.I2.i1.p1.1),[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px6.p1.1),[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px9.p1.1),[Appendix F](https://arxiv.org/html/2610.02665#A6.SS0.SSS0.Px3.p1.1),[§G\.3](https://arxiv.org/html/2610.02665#A7.SS3.p2.1),[§1](https://arxiv.org/html/2610.02665#S1.p4.1)\. - Nisonoffet al\.\(2025\)H\. Nisonoff, J\. Xiong, S\. Allenspach, and J\. ListgartenUnlocking guidance for discrete state\-space diffusion and flow models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=XsgHl54yO7)Cited by:[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px3.p1.1)\. - Ouet al\.\(2025\)J\. Ou, S\. Nie, K\. Xue, F\. Zhu, J\. Sun, Z\. Li, and C\. LiYour absorbing discrete diffusion secretly models the conditional distributions of clean data\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=sMyXP8Tanm)Cited by:[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px1.p1.1),[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px3.p1.1),[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px4.p1.1),[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px4.p1.2),[Appendix F](https://arxiv.org/html/2610.02665#A6.SS0.SSS0.Px3.p1.1)\. - Potaptchiket al\.\(2026\)P\. Potaptchik, J\. Yim, A\. Saravanan, P\. Holderrieth, E\. Vanden\-Eijnden, and M\. S\. AlbergoDiscrete flow maps\.arXiv preprint arXiv:2604\.09784\.Cited by:[1st item](https://arxiv.org/html/2610.02665#A1.I1.i1.p1.1),[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px3.p1.1),[§G\.4](https://arxiv.org/html/2610.02665#A7.SS4.p2.1),[§4\.2](https://arxiv.org/html/2610.02665#S4.SS2.SSS0.Px1.p1.2)\. - Robbins \(1992\)H\. E\. RobbinsAn empirical Bayes approach to statistics\.InBreakthroughs in Statistics: Foundations and Basic Theory,S\. Kotz and N\. L\. Johnson \(Eds\.\),pp\. 388–394\.External Links:ISBN 978\-1\-4612\-0919\-5,[Document](https://dx.doi.org/10.1007/978-1-4612-0919-5%5F26),[Link](https://doi.org/10.1007/978-1-4612-0919-5_26)Cited by:[§G\.11](https://arxiv.org/html/2610.02665#A7.SS11.p4.1.1)\. - Sabouret al\.\(2025\)A\. Sabour, S\. Fidler, and K\. KreisAlign your flow: scaling continuous\-time flow map distillation\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=pzHuesCvcO)Cited by:[Appendix B](https://arxiv.org/html/2610.02665#A2.SS0.SSS0.Px5.p1.1)\. - Sahooet al\.\(2024\)S\. S\. Sahoo, M\. Arriola, Y\. Schiff, A\. Gokaslan, E\. M\. Marroquin, J\. T\. Chiu, A\. M\. Rush, and V\. KuleshovSimple and effective masked diffusion language models\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=L4uaAR4ArM)Cited by:[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px1.p1.1),[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px3.p1.1),[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px3.p1.2),[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px4.p1.1)\. - Sahooet al\.\(2025\)S\. S\. Sahoo, J\. Deschenaux, A\. Gokaslan, G\. Wang, J\. T\. Chiu, and V\. KuleshovThe diffusion duality\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=9P9Y8FOSOk)Cited by:[4th item](https://arxiv.org/html/2610.02665#A1.I1.i4.p1.1)\. - Sakaguchiet al\.\(2020\)K\. Sakaguchi, R\. Le Bras, C\. Bhagavatula, and Y\. ChoiWinoGrande: an adversarial winograd schema challenge at scale\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 8732–8740\.Cited by:[Appendix F](https://arxiv.org/html/2610.02665#A6.SS0.SSS0.Px1.p1.1)\. - Salimans and Ho \(2022\)T\. Salimans and J\. HoProgressive distillation for fast sampling of diffusion models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=TIdIXIpzhoI)Cited by:[§1](https://arxiv.org/html/2610.02665#S1.p1.1)\. - Schiffet al\.\(2025\)Y\. Schiff, S\. S\. Sahoo, H\. Phung, G\. Wang, S\. Boshar, H\. Dalla\-torre, B\. P\. de Almeida, A\. M\. Rush, T\. Pierrot, and V\. KuleshovSimple guidance mechanisms for discrete diffusion models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=i5MrJ6g5G1)Cited by:[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px3.p1.1),[§G\.3](https://arxiv.org/html/2610.02665#A7.SS3.p2.1)\. - Shaulet al\.\(2026\)N\. Shaul, C\. Liu, A\. Vahdat, and J\. BernerParallel decoding distillation for fast image and video generation\.arXiv preprint arXiv:2607\.26004\.Cited by:[Appendix B](https://arxiv.org/html/2610.02665#A2.SS0.SSS0.Px5.p1.1),[§I\.1](https://arxiv.org/html/2610.02665#A9.SS1.p1.1),[§I\.2](https://arxiv.org/html/2610.02665#A9.SS2.p1.1),[§I\.5](https://arxiv.org/html/2610.02665#A9.SS5.SSS0.Px2.p1.1),[§5\.3](https://arxiv.org/html/2610.02665#S5.SS3.p1.1),[footnote 4](https://arxiv.org/html/2610.02665#footnote4)\. - Shiet al\.\(2024\)J\. Shi, K\. Han, Z\. Wang, A\. Doucet, and M\. TitsiasSimplified and generalized masked diffusion for discrete data\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=xcqSOfHt4g)Cited by:[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px1.p1.1),[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px3.p1.1),[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px4.p1.1)\. - Songet al\.\(2021a\)J\. Song, C\. Meng, and S\. ErmonDenoising diffusion implicit models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=St1giarCHLP)Cited by:[§4\.1](https://arxiv.org/html/2610.02665#S4.SS1.SSS0.Px2.p1.1)\. - Songet al\.\(2023\)Y\. Song, P\. Dhariwal, M\. Chen, and I\. SutskeverConsistency models\.InProceedings of the 40th International Conference on Machine Learning,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202,pp\. 32211–32252\.External Links:[Link](https://proceedings.mlr.press/v202/song23a.html)Cited by:[Appendix B](https://arxiv.org/html/2610.02665#A2.SS0.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2610.02665#S1.p1.1)\. - Songet al\.\(2021b\)Y\. Song, J\. Sohl\-Dickstein, D\. P\. Kingma, A\. Kumar, S\. Ermon, and B\. PooleScore\-based generative modeling through stochastic differential equations\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=PxTIG12RRHS)Cited by:[§4](https://arxiv.org/html/2610.02665#S4.p1.1)\. - Songet al\.\(2025\)Y\. Song, Z\. Zhang, C\. Luo, P\. Gao, F\. Xia, H\. Luo, Z\. Li, Y\. Yang, H\. Yu, X\. Qu,et al\.Seed diffusion: a large\-scale diffusion language model with high\-speed inference\.arXiv preprint arXiv:2508\.02193\.Cited by:[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px9.p1.1),[§1](https://arxiv.org/html/2610.02665#S1.p1.1)\. - Strudelet al\.\(2022\)R\. Strudel, C\. Tallec, F\. Altché, Y\. Du, Y\. Ganin, A\. Mensch, W\. Grathwohl, N\. Savinov, S\. Dieleman, L\. Sifre,et al\.Self\-conditioned embedding diffusion for text generation\.arXiv preprint arXiv:2211\.04236\.Cited by:[1st item](https://arxiv.org/html/2610.02665#A1.I1.i1.p1.1)\. - Taeet al\.\(2025\)J\. Tae, H\. Ivison, S\. Kumar, and A\. CohanTESS 2: a large\-scale generalist diffusion language model\.pp\. 21171–21188\.External Links:[Link](https://aclanthology.org/2025.acl-long.1029/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1029),ISBN 979\-8\-89176\-251\-0Cited by:[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px2.p1.1)\. - Wuet al\.\(2026\)Y\. Wu, J\. Cheng, L\. Yu, P\. Thakuria, R\. Brekelmans, E\. E\. Papalexakis, and G\. V\. SteegDiscrete stochastic localization for non\-autoregressive generation\.arXiv preprint arXiv:2602\.16169\.Cited by:[1st item](https://arxiv.org/html/2610.02665#A1.I1.i1.p1.1),[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px2.p1.1)\. - Xieet al\.\(2025\)J\. Xie, W\. Mao, Z\. Bai, D\. J\. Zhang, W\. Wang, K\. Q\. Lin, Y\. Gu, Z\. Chen, Z\. Yang, and M\. Z\. ShouShow\-o: one single transformer to unify multimodal understanding and generation\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=o6Ynz6OIQ6)Cited by:[Appendix B](https://arxiv.org/html/2610.02665#A2.SS0.SSS0.Px5.p1.1)\. - Xuet al\.\(2026\)Y\. Xu, Y\. Wu, S\. Park, Z\. Zhou, and S\. TulsianiTemporal score rescaling for temperature sampling in diffusion and flow models\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=WbfRe0cheB)Cited by:[§4\.2](https://arxiv.org/html/2610.02665#S4.SS2.SSS0.Px2.p1.2)\. - Yanget al\.\(2025\)L\. Yang, Y\. Tian, B\. Li, X\. Zhang, K\. Shen, Y\. Tong, and M\. WangMMaDA: multimodal large diffusion language models\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=wczmXLuLGd)Cited by:[Appendix B](https://arxiv.org/html/2610.02665#A2.SS0.SSS0.Px5.p1.1)\. - Yanget al\.\(2026\)Z\. Yang, W\. Guo, S\. Zhang, S\. S\. Sahoo, Y\. Chen, A\. Vahdat, M\. Mardani, and J\. ThickstunContinuous diffusion scales competitively with discrete diffusion for language\.arXiv preprint arXiv:2605\.18530\.Cited by:[1st item](https://arxiv.org/html/2610.02665#A1.I1.i1.p1.1),[Appendix E](https://arxiv.org/html/2610.02665#A5.SS0.SSS0.Px2.p1.1),[Appendix E](https://arxiv.org/html/2610.02665#A5.SS0.SSS0.Px5.p1.1),[§E\.1](https://arxiv.org/html/2610.02665#A5.SS1.p2.1),[§1](https://arxiv.org/html/2610.02665#S1.p2.1),[§2](https://arxiv.org/html/2610.02665#S2.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2610.02665#S2.SS0.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2610.02665#S2.SS0.SSS0.Px6.p1.1)\. - Yeet al\.\(2025\)J\. Ye, Z\. Xie, L\. Zheng, J\. Gao, Z\. Wu, X\. Jiang, Z\. Li, and L\. KongDream 7B: diffusion large language models\.arXiv preprint arXiv:2508\.15487\.Cited by:[1st item](https://arxiv.org/html/2610.02665#A3.I2.i1.p1.1),[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px6.p1.1),[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px9.p1.1),[§1](https://arxiv.org/html/2610.02665#S1.p4.1)\. - Yinet al\.\(2024\)T\. Yin, M\. Gharbi, T\. Park, R\. Zhang, E\. Shechtman, F\. Durand, and W\. T\. FreemanImproved distribution matching distillation for fast image synthesis\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=tQukGCDaNT)Cited by:[Appendix B](https://arxiv.org/html/2610.02665#A2.SS0.SSS0.Px5.p1.1)\. - Yinet al\.\(2025\)T\. Yin, Q\. Zhang, R\. Zhang, W\. T\. Freeman, F\. Durand, E\. Shechtman, and X\. HuangFrom Slow Bidirectional to Fast Autoregressive Video Diffusion Models\.In2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.,Los Alamitos, CA, USA,pp\. 22963–22974\.External Links:ISSN,[Document](https://dx.doi.org/10.1109/CVPR52734.2025.02138),[Link](https://doi.ieeecomputersociety.org/10.1109/CVPR52734.2025.02138)Cited by:[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px4.p1.1)\. - Youet al\.\(2026\)Z\. You, S\. Nie, X\. Zhang, J\. ZHOU, Z\. Lu, J\. Wen, and C\. LiLLaDA\-V: large language diffusion models with visual instruction tuning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 10093–10105\.Cited by:[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px9.p1.1)\. - Yuet al\.\(2026\)L\. Yu, Y\. Wu, Y\. Fu, S\. Xiong, R\. Brekelmans, H\. Liu, Y\. Dong, and G\. V\. SteegDSL\-LLaDA: scaling continuous denoising to 8B masked diffusion LMs\.arXiv preprint arXiv:2606\.01024\.Cited by:[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px2.p1.1)\. - Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,Florence, Italy,pp\. 4791–4800\.External Links:[Link](https://aclanthology.org/P19-1472/)Cited by:[Appendix F](https://arxiv.org/html/2610.02665#A6.SS0.SSS0.Px1.p1.1)\. - Zhang and Chen \(2023\)Q\. Zhang and Y\. ChenFast sampling of diffusion models with exponential integrator\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Loek7hfb46P)Cited by:[§G\.7](https://arxiv.org/html/2610.02665#A7.SS7.SSS0.Px1.p4.1),[§G\.7](https://arxiv.org/html/2610.02665#A7.SS7.SSS0.Px1.p5.1),[§G\.7](https://arxiv.org/html/2610.02665#A7.SS7.SSS0.Px1.p6.1),[§5\.1](https://arxiv.org/html/2610.02665#S5.SS1.SSS0.Px1.p1.1)\. - Zhouet al\.\(2025\)C\. Zhou, L\. YU, A\. Babu, K\. Tirumala, M\. Yasunaga, L\. Shamis, J\. Kahn, X\. Ma, L\. Zettlemoyer, and O\. LevyTransfusion: predict the next token and diffuse images with one multi\-modal model\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SI2hI0frk6)Cited by:[Appendix B](https://arxiv.org/html/2610.02665#A2.SS0.SSS0.Px5.p1.1)\. - Zhuet al\.\(2026a\)F\. Zhu, R\. Wang, S\. Nie, X\. Zhang, C\. Wu, J\. Zhou, Y\. Lin, J\. Wen, and C\. LiLLaDA 1\.5: variance\-reduced preference optimization for large language diffusion models\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 11425–11460\.External Links:[Link](https://aclanthology.org/2026.acl-long.524/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.524),ISBN 979\-8\-89176\-390\-6Cited by:[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px9.p1.1)\. - Zhuet al\.\(2025\)F\. Zhu, Z\. You, Y\. Xing, Z\. Huang, L\. Liu, Y\. Zhuang, G\. Lu, K\. Wang, X\. Wang, L\. Wei,et al\.LLaDA\-MoE: a sparse MoE diffusion language model\.arXiv preprint arXiv:2509\.24389\.Cited by:[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px9.p1.1)\. - Zhuet al\.\(2026b\)Y\. Zhu, J\. Shi, C\. Ge, H\. Tan, Y\. Xu, W\. Zhu, J\. Kuen, K\. Goswami, R\. Jain, Y\. Chen,et al\.FLARE: diffusion for hybrid language model\.arXiv preprint arXiv:2606\.01774\.Cited by:[Appendix C](https://arxiv.org/html/2610.02665#A3.SS0.SSS0.Px9.p1.1)\. - Zhuanget al\.\(2026\)S\. Zhuang, Y\. Ai, J\. Han, X\. Li, H\. Huang, X\. Yue, X\. Hu, K\. Xu, Y\. Wang, and H\. ChenBitLM: unlocking multi\-token language generation with bitwise continuous diffusion\.arXiv preprint arXiv:2605\.11577\.Cited by:[4th item](https://arxiv.org/html/2610.02665#A1.I1.i4.p1.1),[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px2.p1.1),[Appendix A](https://arxiv.org/html/2610.02665#A1.SS0.SSS0.Px3.p1.1)\. ## Contents ## Appendix ARelated Work \[Back to[Sec\.1](https://arxiv.org/html/2610.02665#S1)or[Sec\.2](https://arxiv.org/html/2610.02665#S2)in the main paper\.\] This appendix provides a literature review on continuous diffusion language models, the family to whichSigmabelongs\. ##### Continuous diffusion language models\. Continuous dLMs lift discrete tokens into a Euclidean space and run a Gaussian \(or Riemannian\) diffusion there\. A central attraction for doing so is that the machinery developed for image diffusion, such as efficient ODE solvers, guidance, and distillation techniques, transfers with little modification\. Existing methods differ mainly in*where*the diffusion process lives: - •Diffusing*token embeddings*\. Diffusion\-LM\([Li et al\., 2022](https://arxiv.org/html/2610.02665#bib.bib19)\)pioneered this approach with Gaussian diffusion over learned word embeddings and a rounding step back to tokens; CDCD\([Dieleman et al\., 2022](https://arxiv.org/html/2610.02665#bib.bib24)\)refined the objective with score interpolation and a categorical cross\-entropy loss over softmax logits, and self\-conditioned embedding diffusion\([Strudel et al\., 2022](https://arxiv.org/html/2610.02665#bib.bib23)\)added self\-conditioning on the previous denoising estimate\. Plaid\([Gulrajani and Hashimoto, 2023](https://arxiv.org/html/2610.02665#bib.bib16)\)tied these ideas together around a likelihood target by pairing low\-dimensional sphere\-normalized embeddings with a learned noise schedule and categorical reparameterization; RePlaid\([Yang et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib60)\)aligns this recipe with modern DiT architectures and shows it scales competitively with discrete dLMs\. Meanwhile, LangFlow\([Chen et al\., 2026b](https://arxiv.org/html/2610.02665#bib.bib42)\)follows the same learnable\-embedding but uses cross\-entropy training loss, while[Wu et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib96)adopts a stochastic\-localization formulation\.𝕊\\mathbb\{S\}\-FLM\([Deschenaux and Gulcehre, 2026](https://arxiv.org/html/2610.02665#bib.bib46)\)and Spherical Flows\([Chemseddine et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib95)\)likewise use learnable token embeddings but further constrain the diffusion trajectory itself to the embedding hypersphere\. A parallel strand fixes the embedding to be*one\-hot*rather than learned: FLM\([Lee et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib43)\)and DFM\([Potaptchik et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib44)\)run flow matching from a Gaussian source to one\-hot vocabulary targets inℝV\\mathbb\{R\}^\{V\}, and further distill the flow into a few\-step flow map for fast generation\. - •Diffusing an*autoencoded latent*\. Here, a text autoencoder produces a compact continuous latent variable and diffusion is run in that latent space\. LD4LG\([Lovelace et al\., 2023](https://arxiv.org/html/2610.02665#bib.bib25)\)runs latent diffusion on a frozen encoder–decoder \(BART and T5\); COSMOS\([Meshchaninov et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib11)\), ELF\([Hu et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib47)\)Cola\-DLM\([Guo et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib56)\), and AURORA\-LM\([Liang et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib53)\)scale this axis with different latent designs and training objectives; LDLM\([Meshchaninov et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib49)\)learns the encoder and the diffusion prior jointly rather than sequentially\. - •Diffusing on the*simplex or other statistical manifolds*\. These methods parameterize a categorical distribution over the vocabulary and diffuse its parameters directly, keeping the noise process on\-manifold and avoiding a rounding step\. SSD\-LM\([Han et al\., 2023](https://arxiv.org/html/2610.02665#bib.bib22)\)runs simplex diffusion in a semi\-autoregressive block structure, Dirichlet diffusion\([Avdeyev et al\., 2023](https://arxiv.org/html/2610.02665#bib.bib28)\)uses a Jacobi diffusion process on the probability simplex, and later variants\([Cheng et al\., 2024](https://arxiv.org/html/2610.02665#bib.bib13);[Davis et al\., 2024](https://arxiv.org/html/2610.02665#bib.bib14);[Jo and Hwang, 2025](https://arxiv.org/html/2610.02665#bib.bib15)\)extend the geometry to Fisher–Rao and Riemannian flow matching\. - •Reaching discrete data through*dequantization or a Gaussian–discrete duality*\. Argmax flows\([Hoogeboom et al\., 2021](https://arxiv.org/html/2610.02665#bib.bib17)\)lift categorical data into a continuous space via argmax\-based variational dequantization, so continuous flows can be trained under a discrete likelihood; analog bits\([Chen et al\., 2023](https://arxiv.org/html/2610.02665#bib.bib21)\), BitLM\([Zhuang et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib54)\), and CoBit\([Batzolis et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib1)\)instead map each token to a bitstring and diffuse from there; Duo\([Sahoo et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib37)\)establishes a duality between continuous Gaussian and uniform\-state discrete diffusion, letting the two families share training and sampling machinery\. Sigmasits in the token\-embedding class, inheriting the learnable embedding and variational diffusion process of Plaid and RePlaid\. ##### Scaling continuous dLMs to billions of parameters\. Recent work pushes continuous dLMs into the billion\-parameter regime, but along axes different from ours: for example, Cola\-DLM\([Guo et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib56)\)\(∼\\sim2B, hierarchical latent\-prior modeling\), AURORA\-LM\([Liang et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib53)\)\(1B, flow matching over an autoencoded latent\), and Categorical Flow Maps\([Davis et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib90)\)\(1\.7B, self\-distilled for few\-step generation\)\. A separate line changes the base representation and scales through an existing LM backbone: BitLM\([Zhuang et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib54)\)places a bitwise diffusion head atop a causal LM up to 8B, TESS 2\([Tae et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib55)\)adapts an autoregressive Mistral\-7B into a generalist diffusion model with a simplex cross\-entropy loss, and DSL\-LLaDA\([Yu et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib97)\)scales continuous denoising via discrete stochastic localization\([Wu et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib96)\)on top of the LLaDA\-8B masked\-diffusion backbone\([Nie et al\., 2025b](https://arxiv.org/html/2610.02665#bib.bib38)\)\. In contrast,Sigmais, to our knowledge, the first likelihood\-trained continuous dLM at 3B/8B, and the first evaluated head\-to\-head with a matched masked dLM\([Fu et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib51)\)on conditional reasoning and code\. ##### CFG in dLMs\. Continuous dLMs realize the guidance condition in two ways\. When there is no prompt \(e\.g\., unconditional generation\), several works repurpose the self\-conditioning signal as the condition: ELF\([Hu et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib47)\)and AURORA\-LM\([Liang et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib53)\)extrapolate between predictions made with and without the previous\-step estimate\. For conditional generation, DFM\([Potaptchik et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib44)\), AURORA\-LM\([Liang et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib53)\), Cola\-DLM\([Guo et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib56)\), and BitLM\([Zhuang et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib54)\)extrapolate between a prompt\-conditioned and an unconditional prediction, as in image diffusion models\. A cost for this is that enabling it requires dropping the prompt context during training\. FLM\([Lee et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib43)\)also leverages autoguidance\([Karras et al\., 2024](https://arxiv.org/html/2610.02665#bib.bib64)\)for prediction sharpening\. For discrete dLMs, LLaDA\([Nie et al\., 2025b](https://arxiv.org/html/2610.02665#bib.bib38)\)popularized an “unsupervised” CFG that runs one forward with full prompt and one with the prompt masked, then linearly extrapolates the two logit vectors before renormalizing; this has since been refined by more general discrete guidance mechanisms\([Schiff et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib29);[Nisonoff et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib32)\)\. Our steering \([Sec\.4\.2](https://arxiv.org/html/2610.02665#S4.SS2)\) differs on three points: \(i\) the guidance condition is the clean prefix itself, and the unconditional branch simply drops that prefix so the noisy block attends only within itself; \(ii\) the extrapolation acts on the predicted clean embedding \(equivalently the score\), not on logits; \(iii\) it is a pure inference\-time operation that requires no train\-time dropout because sequence packing leaves the first block without context\. ##### Autoregressive and block video diffusion\. Our block\-causal factorization \(i\.e\., autoregressive across blocks, bidirectional Gaussian denoising within a block over a KV\-cacheable clean prefix\) closely mirrors recent video generation models that couple causal structure across frames with denoising within them\([Chen et al\., 2024](https://arxiv.org/html/2610.02665#bib.bib61);[Yin et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib62);[Huang et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib63)\)\. The bidirectional video\-diffusion lineage\([Ho et al\., 2022](https://arxiv.org/html/2610.02665#bib.bib34)\)was the original target of this conversion, and the autoregressive arm of Cosmos\([Agarwal et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib35)\)likewise structures video generation causally across frames\. Diffusion Forcing\([Chen et al\., 2024](https://arxiv.org/html/2610.02665#bib.bib61)\)trains a causal model with independent per\-frame noise levels, unifying next\-token prediction with full\-sequence diffusion and enabling stable rollouts past the training horizon\. CausVid\([Yin et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib62)\)and Self Forcing\([Huang et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib63)\)convert bidirectional video diffusion into causal, KV\-cached autoregressive generators for fast streaming synthesis while curbing error accumulation along the rollout\. These works share our central design tension \(i\.e\., coupling causal, autoregressive structure with within\-block diffusion\) and several of their techniques \(per\-block noise schedules, rolling KV caches, distribution\-matching distillation\) transfer naturally to the language setting\. ## Appendix BLimitations and Future Work \[Back to[Sec\.7](https://arxiv.org/html/2610.02665#S7)in the main paper\.\] Despite the promising performance of continuous dLMs for language reasoning, several limitations remain\. In this section, we outline these key challenges alongside promising directions for future research\. ##### Inference Speed and Few\-Step Trajectory Distillation\. Continuous dLM decoding remains slower per generated token at high step counts compared to fully optimized autoregressive baselines\. Under low step budgets, DDIM\-4/8 exhibits a major performance drop relative to DDPM\-256 \([Tab\.1](https://arxiv.org/html/2610.02665#S4.T1)\)\. We observe that neither higher\-order ODE solvers \([Sec\.G\.7](https://arxiv.org/html/2610.02665#A7.SS7)\) nor commit\-stride sampling \([Sec\.G\.5](https://arxiv.org/html/2610.02665#A7.SS5)\) fully bridge this gap\. To address this, we explore preliminary on\-policy and off\-policy Parallel Decoding Distillation \(PDD\), observing encouraging performance gains\. Significant headroom remains in optimizing PDD head parameterizations, training objectives, and policy learning strategies\. Given that standard DDIM sampling with 16 NFEs maintains decent performance, effective distillation holds substantial potential for4×4\\times–8×8\\timesacceleration, leveraging insights from continuous trajectory distillation in visual domains\. Beyond PDD, extending trajectory distillation to consistency and flow\-matching frameworks provides a promising path toward real\-time continuous dLM generation\. Finally, since our model involves self\-conditioning, how to distill it into the few\-step generative model remains an open question\. ##### Encoder–Decoder Robustness\. We observe that diffusion\-based ELBO training tends to collapse the continuous embedding geometry because the readout decoder \(the linear logit projection\) is trained predominantly to reconstruct clean, final\-step token embeddings\. Consequently, the decoder becomes hypersensitive to subtle continuous perturbations\. This sensitivity becomes severe when latent representations fall out\-of\-distribution \(OOD\), which occurs frequently during early denoising stages, low\-NFE sampling, trajectory distillation, or aggressive score steering via CFG and ST \([Fig\.15](https://arxiv.org/html/2610.02665#A7.F15)\)\. This latent\-to\-discrete mismatch might induce decoding errors that bottleneck reasoning performance, especially in code generation where a single syntax error yields complete benchmark failure\. To mitigate embedding collapse and strengthen decoder robustness, future research could explore auxiliary cross\-entropy losses evaluated across all diffusion timesteps or robustify the readout decoder via VAE\-style pre\-training prior to diffusion alignment\. ##### Memory Overhead and Blockwise KV\-Caching Dynamics\. Operating blockwise continuous diffusion requires evaluating the transformer denoiser acrossNNreverse steps per continuous block\. Although blockwise generation offers parallel decoding potential, repeatedly updating and querying key\-value \(KV\) states across intermediate noise timesteps increases peak memory bandwidth demand compared to standard single\-pass autoregressive KV\-caching\. Optimizing persistent KV\-cache management, state\-reuse across intermediate diffusion steps, and extending blockwise continuous diffusion to ultra\-long context windows \(\>32k\>32\\text\{k\}tokens\) remain important systems challenges for future hardware deployments\. ##### Reward\-Driven Inference\-Time Scaling and Post\-Training\. Both inference\-time reward steering and reward\-driven post\-training remain largely unexplored for continuous dLMs\. At inference, our CFG \+ ST framework extends naturally to reward\-model guidance, sequential Monte Carlo over the denoising SDE, and tilted sampling toward verifier\-preferred regions\. For post\-training, continuous dLMs can in principle inherit the RL toolbox developed for image and video diffusion; whether it transfers from perceptual rewards to language verifiers \(math checkers, unit tests, preference models\) is an open direction\. ##### Multimodal Reasoning and Unified Latents\. Because the probability\-flow ODE is natively well\-defined in our continuous formulation, standard continuous image\-diffusion techniques such as trajectory matching\([Song et al\., 2023](https://arxiv.org/html/2610.02665#bib.bib8);[Geng et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib71);[Sabour et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib73);[Boffi et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib75);[Chen et al\., 2026a](https://arxiv.org/html/2610.02665#bib.bib74);[Shaul et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib72)\)and distribution matching\([Yin et al\., 2024](https://arxiv.org/html/2610.02665#bib.bib79)\)can be directly imported into language modeling\. Continuous diffusion is a natural fit for unified multimodal generation architectures\([Zhou et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib77);[Xie et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib78);[Yang et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib76)\)that bridge text with images, video, or audio\. ExtendingSigmato joint multimodal continuous trajectories presents a compelling path forward\. ## Appendix CAdditional Background: Masked Diffusion Language Models \[Back to[Sec\.3\.1](https://arxiv.org/html/2610.02665#S3.SS1)in the main paper\.\] ##### Notation\. We restate masked diffusion models \(MDMs,[Austin et al\. \(2021a\)](https://arxiv.org/html/2610.02665#bib.bib10);[Sahoo et al\. \(2024\)](https://arxiv.org/html/2610.02665#bib.bib20);[Shi et al\. \(2024\)](https://arxiv.org/html/2610.02665#bib.bib7);[Ou et al\. \(2025\)](https://arxiv.org/html/2610.02665#bib.bib27)\) in the notation of[Secs\.2](https://arxiv.org/html/2610.02665#S2)and[3\.1](https://arxiv.org/html/2610.02665#S3.SS1)\. Many notations are shared and carry their usual meaning: the clean sequence𝐱\{\\mathbf\{x\}\}, identified with its one\-hot matrix in\{0,1\}L×V\\\{0,1\\\}^\{L\\times V\}with rows𝐱1,…,𝐱L\{\\mathbf\{x\}\}^\{1\},\\ldots,\{\\mathbf\{x\}\}^\{L\}; the vocabulary𝒱\\mathcal\{V\}of sizeVV\(now including mask token𝐦\{\\mathbf\{m\}\}\); the simplexΔV\\Delta^\{V\}; and the denoiser𝐱θ\{\\mathbf\{x\}\}\_\{\\theta\}, which in either paradigm maps a noisy state to a per\-position categorical distribution in\(ΔV\)L⊂ℝL×V\(\\Delta^\{V\}\)^\{L\}\\subset\\mathbb\{R\}^\{L\\times V\}\. However, there are a few key differences: - •The latent𝐳t\{\\mathbf\{z\}\}\_\{t\}\. Continuous: a Gaussian\-noised embedding𝐳t∈ℝL×de\{\\mathbf\{z\}\}\_\{t\}\\in\\mathbb\{R\}^\{L\\times d\_\{e\}\}[1](https://arxiv.org/html/2610.02665#S2.E1)\. Masked: a partially masked token sequence𝐳t∈𝒱L\{\\mathbf\{z\}\}\_\{t\}\\in\\mathcal\{V\}^\{L\}[7](https://arxiv.org/html/2610.02665#A3.E7), identified with its one\-hot matrix like𝐱\{\\mathbf\{x\}\}\. The denoiser𝐱θ\(𝐳t,t\)\{\\mathbf\{x\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\},t\)and reverse chainpθp\_\{\\theta\}consume whichever of the two is in force\. - •The noise scheduleαt\\alpha\_\{t\}\. Continuous: the variance\-preserving signal scale,αt2\+σt2=1\\alpha\_\{t\}^\{2\}\+\\sigma\_\{t\}^\{2\}=1, learned throughγ\(t\)\\gamma\(t\)\(see[1](https://arxiv.org/html/2610.02665#S2.E1)\)\. Masked: the probability that a position is still kept unmasked \(i\.e\., clean\) at timett, decreasing fromα0=1\\alpha\_\{0\}=1toα1=0\\alpha\_\{1\}=0and fixed to be the linear scheduleαt=1−t\\alpha\_\{t\}=1\-tby default\. - •The endpointst=0t=0andt=1t=1\. Masked: exact\.α0=1\\alpha\_\{0\}=1gives the clean sequence𝐱\{\\mathbf\{x\}\}andα1=0\\alpha\_\{1\}=0the fully masked one, so the bound needs neither a prior nor a reconstruction term\. Continuous: approximate\. The learned endpointsγ0<γ1\\gamma\_\{0\}<\\gamma\_\{1\}leave a finiteSNR\\operatorname\{SNR\}at both ends, so𝐳0\{\\mathbf\{z\}\}\_\{0\}is a minimally noisy latent that still has to be decoded and𝐳1\{\\mathbf\{z\}\}\_\{1\}is only approximately𝒩\(𝟎,𝐈\)\\mathcal\{N\}\(\{\\mathbf\{0\}\},\{\\mathbf\{I\}\}\)\. The embedding objects𝐞\{\\mathbf\{e\}\},𝐄\{\\mathbf\{E\}\},ded\_\{e\}and the signal\-to\-noise scheduleSNR\(t\)\\operatorname\{SNR\}\(t\)have no masked counterpart and do not appear below\. ##### Forward process\. MDMs construct the noising process by random masking and generate samples by iteratively unmasking from a fully masked sequence\. The marginalq\(𝐳t∣𝐱\)q\(\{\\mathbf\{z\}\}\_\{t\}\\mid\{\\mathbf\{x\}\}\)factorizes across positions and independently keeps each token clean with probabilityαt\\alpha\_\{t\}or replaces it with the mask token𝐦\{\\mathbf\{m\}\}: q\(𝐳tl∣𝐱l\)=Cat\(𝐳tl,αt𝐱l\+\(1−αt\)𝐦\)\.\\displaystyle q\(\{\\mathbf\{z\}\}\_\{t\}^\{l\}\\mid\{\\mathbf\{x\}\}^\{l\}\)=\\operatorname\{Cat\}\(\{\\mathbf\{z\}\}\_\{t\}^\{l\};\\alpha\_\{t\}\\,\{\\mathbf\{x\}\}^\{l\}\+\(1\-\\alpha\_\{t\}\)\\,\{\\mathbf\{m\}\}\)\.\(7\)The process is*absorbing*: once a position is masked, it remains masked for the rest of the process\. Combined with[7](https://arxiv.org/html/2610.02665#A3.E7), this pins down the two\-time conditional for any0≤s<t≤10\\leq s<t\\leq 1: q\(𝐳sl∣𝐳tl,𝐱l\)=\{Cat\(𝐳sl;𝐳tl\),𝐳tl≠𝐦,Cat\(𝐳sl,αs−αt1−αt𝐱l\+1−αs1−αt𝐦\),𝐳tl=𝐦,\\displaystyle q\(\{\\mathbf\{z\}\}\_\{s\}^\{l\}\\mid\{\\mathbf\{z\}\}\_\{t\}^\{l\},\{\\mathbf\{x\}\}^\{l\}\)=\\begin\{cases\}\\operatorname\{Cat\}\\,\(\{\\mathbf\{z\}\}\_\{s\}^\{l\};\{\\mathbf\{z\}\}\_\{t\}^\{l\}\),&\{\\mathbf\{z\}\}\_\{t\}^\{l\}\\neq\{\\mathbf\{m\}\},\\\\\[2\.0pt\] \\operatorname\{Cat\}\\big\(\{\\mathbf\{z\}\}\_\{s\}^\{l\};\\frac\{\\alpha\_\{s\}\-\\alpha\_\{t\}\}\{1\-\\alpha\_\{t\}\}\\,\{\\mathbf\{x\}\}^\{l\}\+\\frac\{1\-\\alpha\_\{s\}\}\{1\-\\alpha\_\{t\}\}\\,\{\\mathbf\{m\}\}\\big\),&\{\\mathbf\{z\}\}\_\{t\}^\{l\}=\{\\mathbf\{m\}\},\\end\{cases\}\(8\)where the first case copies an already\-unmasked position unchanged\. ##### Denoiser and sequence\-level NELBO\. As in[Sec\.2](https://arxiv.org/html/2610.02665#S2), a denoiser𝐱θ\(𝐳t,t\)∈\(ΔV\)L⊂ℝL×V\{\\mathbf\{x\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\},t\)\\in\(\\Delta^\{V\}\)^\{L\}\\subset\\mathbb\{R\}^\{L\\times V\}returns a categorical distribution over the vocabulary at each masked position, the only difference being that𝐳t\{\\mathbf\{z\}\}\_\{t\}is now the masked state of[7](https://arxiv.org/html/2610.02665#A3.E7)rather than a noisy embedding\. Since it predicts*clean*tokens, it places no mass on the mask token,⟨𝐱θl\(𝐳t,t\),𝐦⟩=0\\langle\{\\mathbf\{x\}\}\_\{\\theta\}^\{l\}\(\{\\mathbf\{z\}\}\_\{t\},t\),\{\\mathbf\{m\}\}\\rangle=0, so it can be substituted for𝐱l\{\\mathbf\{x\}\}^\{l\}wherever the latter appears\. Substituting𝐱θ\{\\mathbf\{x\}\}\_\{\\theta\}for the one\-hot𝐱\{\\mathbf\{x\}\}in[8](https://arxiv.org/html/2610.02665#A3.E8)to define the reverse chainpθp\_\{\\theta\}and, by taking the continuous\-time limit, one obtains the NELBO as a weighted cross\-entropy on the masked positions\([Sahoo et al\., 2024](https://arxiv.org/html/2610.02665#bib.bib20);[Shi et al\., 2024](https://arxiv.org/html/2610.02665#bib.bib7);[Ou et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib27)\): −logpθ\(𝐱\)≤ℒMDLM\(𝐱\):=𝔼t∼Unif\[0,1\]𝔼q\(𝐳t∣𝐱\)\[αt′1−αt∑l=1L𝟏𝐳tl=𝐦log⟨𝐱θl\(𝐳t,t\),𝐱l⟩\]\.\\displaystyle\-\\log p\_\{\\theta\}\(\{\\mathbf\{x\}\}\)\\leq\\mathcal\{L\}\_\{\\text\{MDLM\}\}\(\{\\mathbf\{x\}\}\):=\\E\_\{t\\sim\\operatorname\{Unif\}\[0,1\]\}\\E\_\{q\(\{\\mathbf\{z\}\}\_\{t\}\\mid\{\\mathbf\{x\}\}\)\}\\Big\[\\frac\{\\alpha\_\{t\}^\{\\prime\}\}\{1\-\\alpha\_\{t\}\}\\sum\_\{l=1\}^\{L\}\\mathbf\{1\}\_\{\{\\mathbf\{z\}\}\_\{t\}^\{l\}=\{\\mathbf\{m\}\}\}\\log\\langle\{\\mathbf\{x\}\}\_\{\\theta\}^\{l\}\(\{\\mathbf\{z\}\}\_\{t\},t\),\{\\mathbf\{x\}\}^\{l\}\\rangle\\Big\]\.\(9\)Givenαt=1−t\\alpha\_\{t\}=1\-t, the weight becomes−1t\-\\frac\{1\}\{t\}\. The loss is invariant to the shape ofαt\\alpha\_\{t\}and only depends on the endpointsα0=1\\alpha\_\{0\}=1andα1=0\\alpha\_\{1\}=0\([Kingma et al\., 2021](https://arxiv.org/html/2610.02665#bib.bib18);[Sahoo et al\., 2024](https://arxiv.org/html/2610.02665#bib.bib20)\)\. Interestingly, among several choices, the linear schedule minimizes the variance of the loss\([Sahoo et al\., 2024](https://arxiv.org/html/2610.02665#bib.bib20), E\.1\.1\)\. ##### Optimal denoiser\. Letθ⋆\\theta^\{\\star\}be the minimizer of[9](https://arxiv.org/html/2610.02665#A3.E9)\. The bound is a cross entropy, so𝐱θ⋆\{\\mathbf\{x\}\}\_\{\\theta^\{\\star\}\}is the posterior over the clean token given the partially masked state\([Sahoo et al\., 2024](https://arxiv.org/html/2610.02665#bib.bib20);[Shi et al\., 2024](https://arxiv.org/html/2610.02665#bib.bib7);[Ou et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib27)\); in the one\-hot notation this posterior*is*a conditional mean, exactly as𝐞θ\{\\mathbf\{e\}\}\_\{\\theta\}predicts𝔼\[𝐞∣𝐳t,t\]\\E\[\{\\mathbf\{e\}\}\\mid\{\\mathbf\{z\}\}\_\{t\},t\]in[Sec\.2](https://arxiv.org/html/2610.02665#S2): 𝐱θ⋆l\(𝐳t,t\)=𝔼\[𝐱l∣𝐳t\]∈ΔV\.\\displaystyle\{\\mathbf\{x\}\}\_\{\\theta^\{\\star\}\}^\{l\}\(\{\\mathbf\{z\}\}\_\{t\},t\)=\\E\\big\[\{\\mathbf\{x\}\}^\{l\}\\mid\{\\mathbf\{z\}\}\_\{t\}\\big\]\\in\\Delta^\{V\}\.\(10\)The NELBO scores masked positions only, so it constrains𝐱θ⋆l\{\\mathbf\{x\}\}\_\{\\theta^\{\\star\}\}^\{l\}only where𝐳tl=𝐦\{\\mathbf\{z\}\}\_\{t\}^\{l\}=\{\\mathbf\{m\}\}\. At the remaining positions𝐳tl\{\\mathbf\{z\}\}\_\{t\}^\{l\}already reveals𝐱l\{\\mathbf\{x\}\}^\{l\}, so the same expression returns𝔼\[𝐱l∣𝐳t\]=𝐳tl\\E\[\{\\mathbf\{x\}\}^\{l\}\\mid\{\\mathbf\{z\}\}\_\{t\}\]=\{\\mathbf\{z\}\}\_\{t\}^\{l\}: the denoiser copies its input, which is what the absorbing property of[8](https://arxiv.org/html/2610.02665#A3.E8)assumes\. The right\-hand side of optimal denoiser[10](https://arxiv.org/html/2610.02665#A3.E10)does not explicitly depend onttand involves it only through𝐳t\{\\mathbf\{z\}\}\_\{t\}, i\.e\., conditioned on the masked state, the time carries no further information about𝐱\{\\mathbf\{x\}\}\([Ou et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib27)\)\. Models in this family accordingly take the state alone as input, and we write𝐱θ\(𝐳t\)\{\\mathbf\{x\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}\), dropping the time argument, from here on\. ##### Ancestral reverse sampling\. Sampling starts from the fully masked sequence𝐳1=\(𝐦,…,𝐦\)\{\\mathbf\{z\}\}\_\{1\}=\(\{\\mathbf\{m\}\},\\ldots,\{\\mathbf\{m\}\}\)and steps along a discretization1=t0\>t1\>⋯\>tN=01=t\_\{0\}\>t\_\{1\}\>\\cdots\>t\_\{N\}=0by iterating[8](https://arxiv.org/html/2610.02665#A3.E8)with𝐱θ\{\\mathbf\{x\}\}\_\{\\theta\}in place of𝐱\{\\mathbf\{x\}\}: pθ\(𝐳sl∣𝐳t\)=\{Cat\(𝐳sl;𝐳tl\),𝐳tl≠𝐦,Cat\(𝐳sl,αs−αt1−αt𝐱θl\(𝐳t\)\+1−αs1−αt𝐦\),𝐳tl=𝐦\.\\displaystyle p\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{s\}^\{l\}\\mid\{\\mathbf\{z\}\}\_\{t\}\)=\\begin\{cases\}\\operatorname\{Cat\}\(\{\\mathbf\{z\}\}\_\{s\}^\{l\};\{\\mathbf\{z\}\}\_\{t\}^\{l\}\),&\{\\mathbf\{z\}\}\_\{t\}^\{l\}\\neq\{\\mathbf\{m\}\},\\\\\[2\.0pt\] \\operatorname\{Cat\}\\big\(\{\\mathbf\{z\}\}\_\{s\}^\{l\};\\frac\{\\alpha\_\{s\}\-\\alpha\_\{t\}\}\{1\-\\alpha\_\{t\}\}\\,\{\\mathbf\{x\}\}\_\{\\theta\}^\{l\}\(\{\\mathbf\{z\}\}\_\{t\}\)\+\\frac\{1\-\\alpha\_\{s\}\}\{1\-\\alpha\_\{t\}\}\\,\{\\mathbf\{m\}\}\\big\),&\{\\mathbf\{z\}\}\_\{t\}^\{l\}=\{\\mathbf\{m\}\}\.\\end\{cases\}\(11\)In other words, at each step and each masked positionll, independently, one decides whether to unmask this position with probabilityαs−αt1−αt\\frac\{\\alpha\_\{s\}\-\\alpha\_\{t\}\}\{1\-\\alpha\_\{t\}\}\. If decided to unmask, the token is sampled from the distribution𝐱θl\(𝐳t\)∈ΔV\{\\mathbf\{x\}\}\_\{\\theta\}^\{l\}\(\{\\mathbf\{z\}\}\_\{t\}\)\\in\\Delta^\{V\}\. Once unmasked, a token is frozen for the rest of the trajectory\. ##### Confidence\-based \(greedy\) sampling\. Ancestral sampling with a large vocabulary and few steps often commits low\-probability tokens early and cannot revisit them\.*Confidence\-based*samplers\([Chang et al\., 2022](https://arxiv.org/html/2610.02665#bib.bib58);[Nie et al\., 2025b](https://arxiv.org/html/2610.02665#bib.bib38);[Ye et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib12);[Fu et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib51)\)choose which positions to unmask by predicted confidence, and a temperatureT∈\[0,1\]T\\in\[0,1\]\(typically zero, i\.e\., deterministic argmax decoding\) when sampling from the categorical distribution\. Two variants are commonly used: - •*Top\-kkper step\.*Fix a budgetkik\_\{i\}at stepiiso that their sum is the sequence length\. At stepii, commit thekik\_\{i\}masked positions with the highest confidencecl:=‖𝐱θl\(𝐳ti\)‖∞c^\{l\}:=\\\|\{\\mathbf\{x\}\}\_\{\\theta\}^\{l\}\(\{\\mathbf\{z\}\}\_\{t\_\{i\}\}\)\\\|\_\{\\infty\}, and sample these positions from the probability distribution𝐱θl\(𝐳ti\)\{\\mathbf\{x\}\}\_\{\\theta\}^\{l\}\(\{\\mathbf\{z\}\}\_\{t\_\{i\}\}\)with temperatureTT\. WhenT=0T=0, this is the default sampler of LLaDA\([Nie et al\., 2025b](https://arxiv.org/html/2610.02665#bib.bib38)\)and Dream\([Ye et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib12)\)\. The LLaDA and Dream baselines reported in[Tabs\.2](https://arxiv.org/html/2610.02665#S4.T2)and[4](https://arxiv.org/html/2610.02665#S6.T4)are generated withk1=…=kL=1k\_\{1\}=\.\.\.=k\_\{L\}=1\(i\.e\., unmasking one position at a time with the highest confidence\)\. - •*Threshold per step\.*Fix a confidence thresholdϑ∈\(0,1\)\\vartheta\\in\(0,1\)\. At stepii, commit every masked position withcl≥ϑc^\{l\}\\geq\\vartheta, and sample these positions from the probability distribution𝐱θl\(𝐳ti\)\{\\mathbf\{x\}\}\_\{\\theta\}^\{l\}\(\{\\mathbf\{z\}\}\_\{t\_\{i\}\}\)with temperatureTT\. WhenT=0T=0, this is the greedy sampler of Nemotron\-Labs\-Diffusion\([Fu et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib51)\)and defines our masked baseline in[Tabs\.1](https://arxiv.org/html/2610.02665#S4.T1),[3](https://arxiv.org/html/2610.02665#S6.T3),[2](https://arxiv.org/html/2610.02665#S4.T2)and[4](https://arxiv.org/html/2610.02665#S6.T4)usingϑ∈\{0\.8,0\.9\}\\vartheta\\in\\\{0\.8,0\.9\\\}as the threshold\. Like ancestral sampling, these samplers preserve the absorbing property that an unmasked token stays fixed for the rest of the trajectory\. ##### Block diffusion\. Similar to[Sec\.3\.1](https://arxiv.org/html/2610.02665#S3.SS1), the sequence𝐱\{\\mathbf\{x\}\}is partitioned intoBBcontiguous blocks\{𝐱\(b\)\}b=1B\\\{\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\\}\_\{\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}=1\}^\{B\}of lengthL′=L/BL^\{\\prime\}=L/B, and the model denoises one block at a time given its clean prefix\. Exactly as in[3](https://arxiv.org/html/2610.02665#S3.E3),[Arriola et al\. \(2025\)](https://arxiv.org/html/2610.02665#bib.bib36)factorizes the likelihood autoregressively*across*blocks and then bounds each factor, now with the masked bound in place of the continuous one: −logpθ\(𝐱\)\\displaystyle\-\\log p\_\{\\theta\}\(\{\\mathbf\{x\}\}\)=∑b=1B−logpθ\(𝐱\(b\)∣𝐱\(<b\)\)≤∑b=1BℒMDLM\(𝐱\(b\);𝐱\(<b\)\)=:ℒBD\-MDLM\(𝐱\),\\displaystyle=\\sum\_\{b=1\}^\{B\}\-\\log p\_\{\\theta\}\(\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\mid\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)\\leq\\sum\_\{b=1\}^\{B\}\\mathcal\{L\}\_\{\\text\{MDLM\}\}\(\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\};\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)=:\\mathcal\{L\}\_\{\\text\{BD\-MDLM\}\}\(\{\\mathbf\{x\}\}\),where we obtainℒMDLM\(𝐱\(b\),𝐱\(<b\)\)\\mathcal\{L\}\_\{\\text\{MDLM\}\}\(\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\};\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)by noising blockb\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}alone and conditioning the denoiser in[9](https://arxiv.org/html/2610.02665#A3.E9)on the clean prefix𝐱\(<b\)\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}: ℒMDLM\(𝐱\(b\),𝐱\(<b\)\)\\displaystyle\\mathcal\{L\}\_\{\\text\{MDLM\}\}\(\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\};\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)=\\displaystyle=𝔼t∼Unif\[0,1\]𝔼q\(𝐳t\(b\)∣𝐱\(b\)\)\[αt′1−αt∑l=1L′𝟏𝐳t\(b,l\)=𝐦log⟨𝐱θl\(𝐳t\(b\),𝐱\(<b\)\),𝐱\(b,l\)⟩\]\.\\displaystyle\\E\_\{t\\sim\\operatorname\{Unif\}\[0,1\]\}\\E\_\{q\(\{\\mathbf\{z\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\_\{t\}\\mid\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\)\}\\Big\[\\frac\{\\alpha\_\{t\}^\{\\prime\}\}\{1\-\\alpha\_\{t\}\}\\sum\_\{l=1\}^\{L^\{\\prime\}\}\\mathbf\{1\}\_\{\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\},l\)\}=\{\\mathbf\{m\}\}\}\\log\\langle\{\\mathbf\{x\}\}\_\{\\theta\}^\{l\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\};\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\),\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\},l\)\}\\rangle\\Big\]\.\(12\) ##### Training objective of NLD\. A single Monte\-Carlo sample of[12](https://arxiv.org/html/2610.02665#A3.E12)per block \(one timet\(b\)∼Unif\[0,1\]t^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\sim\\operatorname\{Unif\}\[0,1\]and a correspondingly sampled mask assignment, abbreviated as𝐳\(b\):=𝐳t\(b\)\(b\)\{\\mathbf\{z\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}:=\{\\mathbf\{z\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\_\{t^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\}\) gives the NELBO estimator ℒ^BD\-MDLM\(𝐱\)=∑b=1Bαt\(b\)′1−αt\(b\)∑l=1L′𝟏𝐳\(b,l\)=𝐦log⟨𝐱θl\(𝐳\(b\),𝐱\(<b\)\),𝐱\(b,l\)⟩,\\displaystyle\\widehat\{\\mathcal\{L\}\}\_\{\\text\{BD\-MDLM\}\}\(\{\\mathbf\{x\}\}\)\\;=\\;\\sum\_\{b=1\}^\{B\}\\frac\{\\alpha\_\{t^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\}^\{\\prime\}\}\{1\-\\alpha\_\{t^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\}\}\\sum\_\{l=1\}^\{L^\{\\prime\}\}\\mathbf\{1\}\_\{\{\\mathbf\{z\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\},l\)\}=\{\\mathbf\{m\}\}\}\\log\\langle\{\\mathbf\{x\}\}\_\{\\theta\}^\{l\}\(\{\\mathbf\{z\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\};\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\),\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\},l\)\}\\rangle,\(13\)whose weight is−1t\(b\)\-\\frac\{1\}\{t^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\}, so each masked token contributes its cross entropy divided by its masking probability\. NLD\([Fu et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib51)\)instead draws onettper𝐱\{\\mathbf\{x\}\}and shares it across theBBblocks rather than drawingt\(b\)t^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}independently\. For validation we want the NELBO per token, whose exponential is the reported perplexity; over a batch𝐱1,…,𝐱N\{\\mathbf\{x\}\}\_\{1\},\\ldots,\{\\mathbf\{x\}\}\_\{N\}this normalizes by the total number of tokens: ℒBD\-VAL=1NL∑n=1Nℒ^BD\-MDLM\(𝐱n\)\.\\displaystyle\\mathcal\{L\}\_\{\\text\{BD\-VAL\}\}\\;=\\;\\frac\{1\}\{NL\}\\sum\_\{n=1\}^\{N\}\\widehat\{\\mathcal\{L\}\}\_\{\\text\{BD\-MDLM\}\}\(\{\\mathbf\{x\}\}\_\{n\}\)\.\(14\)NLD, however, trains with a different objective: the*global loss average*normalizes by the number of masked tokens actually drawn \(Nmask\(𝐱\)=∑b=1B∑l=1L′𝟏𝐳\(b,l\)=𝐦N\_\{\\text\{mask\}\}\(\{\\mathbf\{x\}\}\)\\;=\\;\\sum\_\{b=1\}^\{B\}\\sum\_\{l=1\}^\{L^\{\\prime\}\}\\mathbf\{1\}\_\{\{\\mathbf\{z\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\},l\)\}=\{\\mathbf\{m\}\}\}\), giving the NELBO per*masked*token, which is not a bound on the per\-token likelihood: ℒBD\-GLA=∑n=1Nℒ^BD\-MDLM\(𝐱n\)∑n=1NNmask\(𝐱n\)\.\\displaystyle\\mathcal\{L\}\_\{\\text\{BD\-GLA\}\}\\;=\\;\\frac\{\\sum\_\{n=1\}^\{N\}\\widehat\{\\mathcal\{L\}\}\_\{\\text\{BD\-MDLM\}\}\(\{\\mathbf\{x\}\}\_\{n\}\)\}\{\\sum\_\{n=1\}^\{N\}N\_\{\\text\{mask\}\}\(\{\\mathbf\{x\}\}\_\{n\}\)\}\.\(15\)The two share a numerator and differ only by the batch’s realized masking rate, ℒBD\-VAL=ℒBD\-GLA⋅∑n=1NNmask\(𝐱n\)NL,𝔼\[Nmask\(𝐱\)L\]=𝔼t∼Unif\[0,1\]\(1−αt\)=12,\\displaystyle\\mathcal\{L\}\_\{\\text\{BD\-VAL\}\}\\;=\\;\\mathcal\{L\}\_\{\\text\{BD\-GLA\}\}\\cdot\\frac\{\\sum\_\{n=1\}^\{N\}N\_\{\\text\{mask\}\}\(\{\\mathbf\{x\}\}\_\{n\}\)\}\{NL\},\\qquad\\E\\\!\\left\[\\frac\{N\_\{\\text\{mask\}\}\(\{\\mathbf\{x\}\}\)\}\{L\}\\right\]=\\E\_\{t\\sim\\operatorname\{Unif\}\[0,1\]\}\(1\-\\alpha\_\{t\}\)=\\frac\{1\}\{2\},the last equality under the linear schedule\. SoℒBD\-GLA\\mathcal\{L\}\_\{\\text\{BD\-GLA\}\}overstates the per\-token NELBO by roughly a factor of two\. Both denominators are constant with respect toθ\\theta, so the two objectives differ only in scale: that scale is the fixedNLNLforℒBD\-VAL\\mathcal\{L\}\_\{\\text\{BD\-VAL\}\}and the random∑nNmask\(𝐱n\)\\sum\_\{n\}N\_\{\\text\{mask\}\}\(\{\\mathbf\{x\}\}\_\{n\}\)forℒBD\-GLA\\mathcal\{L\}\_\{\\text\{BD\-GLA\}\}\. ##### Scaling\. Following the scaling study of masked dLM on text by[Nie et al\. \(2025a\)](https://arxiv.org/html/2610.02665#bib.bib40), the field has scaled along two lines\. The*from\-scratch*line trains a masked dLM directly on raw data: LLaDA\-8B\([Nie et al\., 2025b](https://arxiv.org/html/2610.02665#bib.bib38)\)rivals AR models of the same size, and LLaDA\-MoE\([Zhu et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib68)\)pushes the objective into a sparse mixture\-of\-experts backbone at 1\.4B active parameters\. The LLaDA\-8B backbone has been further extended along orthogonal axes without additional pre\-training scaling: LLaDA 1\.5\([Zhu et al\., 2026a](https://arxiv.org/html/2610.02665#bib.bib69)\)adds variance\-reduced preference alignment and LLaDA\-V\([You et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib70)\)adds visual instruction tuning\. The*AR\-adaptation*line instead starts from a pretrained AR checkpoint and performances continual pre\-training to turn it into a masked dLM\. Dream\-7B\([Ye et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib12)\)converts Qwen2\.5\-7B into a masked dLM via the DiffuLLaMA shift\-operation trick\([Gong et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib94)\)\. The remainder of this line instead converts the AR checkpoint into a blockwise masked dLM\([Arriola et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib36)\)that preserves KV caching across blocks, a conversion far cheaper than pre\-training from scratch: Nemotron\-Labs\-Diffusion\([Fu et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib51)\), FLARE\([Zhu et al\., 2026b](https://arxiv.org/html/2610.02665#bib.bib59)\), SDAR\([Cheng et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib66)\)at 1\.7∼\\sim30B \(including a 30B MoE\), and most recently LLaDA 2\.0\-mini/flash\([Bie et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib67)\)at 16B and 100B \(MoE\), the largest open masked dLM to date\. Closed\-source frontier deployments such as Mercury\([Inception Labs et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib3)\)and Seed Diffusion\([Song et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib2)\)also fall into this block masked dLM paradigm\. ## Appendix DTraining and Inference Algorithms \[Back to[Sec\.3\.2](https://arxiv.org/html/2610.02665#S3.SS2)or[Sec\.4](https://arxiv.org/html/2610.02665#S4)in the main paper\.\] We collect the two procedures described in the main paper: - •[Alg\.1](https://arxiv.org/html/2610.02665#alg1)is one optimization step of the blockwise NELBO[3](https://arxiv.org/html/2610.02665#S3.E3)and[4](https://arxiv.org/html/2610.02665#S3.Ex2)under the Horvitz–Thompson estimator[5](https://arxiv.org/html/2610.02665#S3.E5)\([Sec\.3\.1](https://arxiv.org/html/2610.02665#S3.SS1)\); - •[Alg\.2](https://arxiv.org/html/2610.02665#alg2)is blockwise reverse sampling \([Sec\.4\.1](https://arxiv.org/html/2610.02665#S4.SS1)\) with CFG, ST, SC, and commit stride \([Sec\.4\.2](https://arxiv.org/html/2610.02665#S4.SS2)\)\. Algorithm 1Sigmatraining step for one sequence1:sequence 𝐱∈\{0,1\}L×V\{\\mathbf\{x\}\}\\in\\\{0,1\\\}^\{L\\times V\}; block size L′L^\{\\prime\}, number of blocks B=L/L′B=L/L^\{\\prime\}; backbone parameters θ\\theta; embedding matrix 𝐄∈ℝV×de\{\\mathbf\{E\}\}\\in\\mathbb\{R\}^\{V\\times d\_\{e\}\}; noise schedule endpoints γ0<γ1\\gamma\_\{0\}<\\gamma\_\{1\}and monotone interior shape γ~ψ:\[0,1\]→\[0,1\]\\tilde\{\\gamma\}\_\{\\psi\}:\[0,1\]\\to\[0,1\], giving γ\(t\)=γ0\+\(γ1−γ0\)γ~ψ\(t\)\\gamma\(t\)=\\gamma\_\{0\}\+\(\\gamma\_\{1\}\-\\gamma\_\{0\}\)\\tilde\{\\gamma\}\_\{\\psi\}\(t\); reconstruction probability prec∈\(0,1\)p\_\{\\text\{rec\}\}\\in\(0,1\)\(default 0\.10\.1\); self\-conditioning probability psc∈\(0,1\)p\_\{\\text\{sc\}\}\\in\(0,1\)\(default 0\.250\.25\); diffusion loss weight wdiff\>0w\_\{\\text\{diff\}\}\>0and optional AR\-loss weight war≥0w\_\{\\text\{ar\}\}\\geq 0; optional SUSReg weight λsusreg≥0\\lambda\_\{\\text\{susreg\}\}\\geq 0with MMslices and PPEpps–Pulley quadrature points on \[−tmax,tmax\]\[\-t\_\{\\max\},t\_\{\\max\}\]\(defaults in[Sec\.E\.1](https://arxiv.org/html/2610.02665#A5.SS1)\) 2: 𝐞←𝐱𝐄∈ℝL×de\{\\mathbf\{e\}\}\\leftarrow\{\\mathbf\{x\}\}\{\\mathbf\{E\}\}\\in\\mathbb\{R\}^\{L\\times d\_\{e\}\}⊳\\trianglerightEmbedding representation, each row has unit\-norm 3:for b=1,…,Bb=1,\\dots,Bdo⊳\\trianglerightindependent noise level and estimator branch per block 4: R\(b\)∼Bern\(prec\)R^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\sim\\mathrm\{Bern\}\(p\_\{\\text\{rec\}\}\) 5: t\(b\)←0t^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\leftarrow 0if R\(b\)=1R^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}=1else t\(b\)∼Unif\[0,1\]t^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\sim\\operatorname\{Unif\}\[0,1\] 6: ϵ\(b\)∼𝒩\(𝟎,𝐈\)\{\\bm\{\\epsilon\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\sim\\mathcal\{N\}\(\{\\mathbf\{0\}\},\{\\mathbf\{I\}\}\), 𝐳\(b\)←αt\(b\)𝐞\(b\)\+σt\(b\)ϵ\(b\)\{\\mathbf\{z\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\leftarrow\\alpha\_\{t^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\}\{\\mathbf\{e\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\+\\sigma\_\{t^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\}\{\\bm\{\\epsilon\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}⊳\\trianglerightGaussian noising[1](https://arxiv.org/html/2610.02665#S2.E1) 7:endfor 8: 𝐞sc←𝟎\{\\mathbf\{e\}\}\_\{\\text\{sc\}\}\\leftarrow\{\\mathbf\{0\}\} 9:if u∼Unif\[0,1\]<pscu\\sim\\operatorname\{Unif\}\[0,1\]<p\_\{\\text\{sc\}\}then⊳\\trianglerightSC pass carries no gradient 10: 𝐞sc←sg\(𝐱θ\(\{𝐳\(b\)\},\{t\(b\)\},𝐱,𝟎\)𝐄\)\{\\mathbf\{e\}\}\_\{\\text\{sc\}\}\\leftarrow\\operatorname\{sg\}\\big\(\{\\mathbf\{x\}\}\_\{\\theta\}\(\\\{\{\\mathbf\{z\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\\},\\\{t^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\\};\{\\mathbf\{x\}\},\{\\mathbf\{0\}\}\)\\,\{\\mathbf\{E\}\}\\big\) 11:endif 12: 𝐱θ←𝐱θ\(\{𝐳\(b\)\},\{t\(b\)\},𝐱,𝐞sc\)\{\\mathbf\{x\}\}\_\{\\theta\}\\leftarrow\{\\mathbf\{x\}\}\_\{\\theta\}\(\\\{\{\\mathbf\{z\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\\},\\\{t^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\\};\{\\mathbf\{x\}\},\{\\mathbf\{e\}\}\_\{\\text\{sc\}\}\), 𝐞θ←𝐱θ𝐄\{\\mathbf\{e\}\}\_\{\\theta\}\\leftarrow\{\\mathbf\{x\}\}\_\{\\theta\}\{\\mathbf\{E\}\}⊳\\trianglerightone forward for all blocks; block\-causal mask 13:for b=1,…,Bb=1,\\dots,Bdo 14: ℒrec\(b\)←−∑l=1L′log⟨𝐱θ\(b,l\),𝐱\(b,l\)⟩\\mathcal\{L\}\_\{\\text\{rec\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\leftarrow\-\\sum\_\{l=1\}^\{L^\{\\prime\}\}\\log\\langle\{\\mathbf\{x\}\}\_\{\\theta\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\},l\)\},\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\},l\)\}\\rangle⊳\\trianglerightused only whenR\(b\)=1R^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}=1, i\.e\.,t\(b\)=0t^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}=0 15: ℒdiff\(b\)←−12SNR′\(t\(b\)\)‖𝐞θ\(b\)−𝐞\(b\)‖2\\mathcal\{L\}\_\{\\text\{diff\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\leftarrow\-\\frac\{1\}\{2\}\\,\\operatorname\{SNR\}^\{\\prime\}\(t^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\)\\,\\\|\{\\mathbf\{e\}\}\_\{\\theta\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\-\{\\mathbf\{e\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\\|^\{2\}⊳\\trianglerightused only whenR\(b\)=0R^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}=0 16: ℒprior\(b\)←L′2\[de\(σ12−1−logσ12\)\+α12\]\\mathcal\{L\}\_\{\\text\{prior\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\leftarrow\\tfrac\{L^\{\\prime\}\}\{2\}\\big\[d\_\{e\}\(\\sigma\_\{1\}^\{2\}\-1\-\\log\\sigma\_\{1\}^\{2\}\)\+\\alpha\_\{1\}^\{2\}\\big\]⊳\\trianglerightclosed form ofKL\(q\(𝐳1\(b\)∣𝐱\(b\)\)∥p\(𝐳1\(b\)\)\)\\operatorname\{KL\}\\big\(q\(\{\\mathbf\{z\}\}\_\{1\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\mid\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\)\\,\\\|\\,p\(\{\\mathbf\{z\}\}\_\{1\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\)\\big\) 17: ℒ\(b\)←1−R\(b\)1−precℒdiff\(b\)\+R\(b\)precℒrec\(b\)\+ℒprior\(b\)\\mathcal\{L\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\leftarrow\\frac\{1\-R^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\}\{1\-p\_\{\\text\{rec\}\}\}\\mathcal\{L\}\_\{\\text\{diff\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\+\\frac\{R^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\}\{p\_\{\\text\{rec\}\}\}\\mathcal\{L\}\_\{\\text\{rec\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\+\\mathcal\{L\}\_\{\\text\{prior\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}⊳\\trianglerightHorvitz–Thompson estimator[5](https://arxiv.org/html/2610.02665#S3.E5) 18:endfor 19: ℒBD←∑b=1Bℒ\(b\)\\mathcal\{L\}\_\{\\text\{BD\}\}\\leftarrow\\sum\_\{b=1\}^\{B\}\\mathcal\{L\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}⊳\\trianglerightblock\-diffusion NELBO estimator[3](https://arxiv.org/html/2610.02665#S3.E3) 20:if λsusreg\>0\\lambda\_\{\\text\{susreg\}\}\>0then 21: ℒBD←ℒBD\+λsusregLs\(𝐄\)\\mathcal\{L\}\_\{\\text\{BD\}\}\\leftarrow\\mathcal\{L\}\_\{\\text\{BD\}\}\+\\lambda\_\{\\text\{susreg\}\}\\,L\\,s\(\{\\mathbf\{E\}\}\)⊳\\trianglerightSUSReg loss[16](https://arxiv.org/html/2610.02665#A5.E16) 22:endif 23: ℒ←wdiffℒBD\+warℒar\(𝐱\)\\mathcal\{L\}\\leftarrow w\_\{\\text\{diff\}\}\\,\\mathcal\{L\}\_\{\\text\{BD\}\}\+w\_\{\\text\{ar\}\}\\,\\mathcal\{L\}\_\{\\text\{ar\}\}\(\{\\mathbf\{x\}\}\)⊳\\trianglerightjoint objective \([Sec\.3\.2\.2](https://arxiv.org/html/2610.02665#S3.SS2.SSS2)\) 24: ℒvar←∑b:R\(b\)=0\(ℒdiff\(b\)\)2\\mathcal\{L\}\_\{\\text\{var\}\}\\leftarrow\\sum\_\{b\\,:\\,R^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}=0\}\(\\mathcal\{L\}\_\{\\text\{diff\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\)^\{2\}⊳\\trianglerightNoise schedule interior shape learned to minimize MC variance 25:update θ\\thetawith ∇θℒ\\nabla\_\{\\theta\}\\mathcal\{L\}⊳\\trianglerightbackbone group, peak lr10−510^\{\-5\} 26:update \(𝐄,γ0,γ1\)\(\{\\mathbf\{E\}\},\\gamma\_\{0\},\\gamma\_\{1\}\)with ∇\(𝐄,γ0,γ1\)ℒ\\nabla\_\{\(\{\\mathbf\{E\}\},\\gamma\_\{0\},\\gamma\_\{1\}\)\}\\mathcal\{L\}⊳\\trianglerightdiffusion group, peak lr10−210^\{\-2\}, no weight decay 27:update ψ\\psiwith ∇ψℒvar\\nabla\_\{\\psi\}\\mathcal\{L\}\_\{\\text\{var\}\}⊳\\trianglerightsame as diffusion group At training, allBBblocks are denoised in a*single*forward pass: the offset block\-causal mask of[Fig\.2\(b\)](https://arxiv.org/html/2610.02665#S3.F2.sf2)makes blockbb’s noisy latent attend to the clean prefix𝐱\(<b\)\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}and bidirectionally within itself, so a per\-block timet\(b\)t^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}and a per\-block Horvitz–Thompson coinR\(b\)R^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}cost no extra compute\. At inference, the conditional and unconditional branches are likewise packed into one forward as in[Fig\.4\(a\)](https://arxiv.org/html/2610.02665#S4.F4.sf1), so the guidance in[28](https://arxiv.org/html/2610.02665#A7.E28)costs slightly more wall\-clock time per step \([Tab\.20](https://arxiv.org/html/2610.02665#A7.T20)\) at batch size one\. We further clarify on the learning of the noise schedule, since its two parts are optimized for two different objectives\. The endpointsγ0,γ1\\gamma\_\{0\},\\gamma\_\{1\}enterℒdiff\\mathcal\{L\}\_\{\\text\{diff\}\}throughSNR′\(t\)\\operatorname\{SNR\}^\{\\prime\}\(t\)and\(αt,σt\)\(\\alpha\_\{t\},\\sigma\_\{t\}\), so they receive an ordinary NELBO gradient and are trained to tighten the bound while the interior shapeγ~ψ\\tilde\{\\gamma\}\_\{\\psi\}does not\. As noted in[Sec\.2](https://arxiv.org/html/2610.02665#S2), the diffusion term is invariant to it in expectation, so∇ψℒ\\nabla\_\{\\psi\}\\mathcal\{L\}is zero on average and carries only noise\. Following[Kingma et al\. \(2021\)](https://arxiv.org/html/2610.02665#bib.bib18),ψ\\psiis instead trained to minimize the Monte\-Carlo variance of the diffusion estimator, which for a mean\-invariant estimator reduces to minimizing its second momentℒvar\\mathcal\{L\}\_\{\\text\{var\}\}in[Alg\.1](https://arxiv.org/html/2610.02665#alg1)\. Both parts sit in the same separate optimizer group as the diffusion embedding, at peak learning rate10−210^\{\-2\}without weight decay \([Sec\.3\.2\.2](https://arxiv.org/html/2610.02665#S3.SS2.SSS2)\)\. Algorithm 2Blockwise reverse sampling ofSigmawith CFG, ST, SC, and commit stride1:prompt 𝐩\{\\mathbf\{p\}\}; block size L′L^\{\\prime\}; commit stride S∈\{1,…,L′\}S\\in\\\{1,\\dots,L^\{\\prime\}\\\}\( S=L′S=L^\{\\prime\}recovers plain block decoding,[Sec\.G\.5](https://arxiv.org/html/2610.02665#A7.SS5)\); total generation length TgenT\_\{\\text\{gen\}\}; steps NNwith discretization 1=t0\>t1\>⋯\>tN=01=t\_\{0\}\>t\_\{1\}\>\\cdots\>t\_\{N\}=0\([Sec\.G\.6](https://arxiv.org/html/2610.02665#A7.SS6)\); guidance weight w≥1w\\geq 1; score temperature τ∈\(0,1\]\\tau\\in\(0,1\]; sampler ∈\{DDPM,DDIM\}\\in\\\{\\text\{DDPM\},\\text\{DDIM\}\\\} 2: 𝐜←𝐩\{\\mathbf\{c\}\}\\leftarrow\{\\mathbf\{p\}\}; populate the causal KV cache from 𝐩\{\\mathbf\{p\}\}; b←0b\\leftarrow 0 3:while \|𝐫\|<Tgen\|\{\\mathbf\{r\}\}\|<T\_\{\\text\{gen\}\}do⊳\\triangleright\|𝐫\|\|\{\\mathbf\{r\}\}\|: number of tokens committed so far 4: b←b\+1b\\leftarrow b\+1 5: 𝐳t0\(b\)∼𝒩\(𝟎,𝐈\)∈ℝL′×de\{\\mathbf\{z\}\}\_\{t\_\{0\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\sim\\mathcal\{N\}\(\{\\mathbf\{0\}\},\{\\mathbf\{I\}\}\)\\in\\mathbb\{R\}^\{L^\{\\prime\}\\times d\_\{e\}\}, 𝐞sc←𝟎\{\\mathbf\{e\}\}\_\{\\text\{sc\}\}\\leftarrow\{\\mathbf\{0\}\} 6:for i=1,…,Ni=1,\\dots,Ndo 7: t←ti−1t\\leftarrow t\_\{i\-1\}, s←tis\\leftarrow t\_\{i\} 8: \(𝐞c,𝐞∅\)←\(𝐞θ\(𝐳t\(b\),t,𝐜,𝐞sc\),𝐞θ\(𝐳t\(b\),t,∅,𝟎\)\)\(\{\\mathbf\{e\}\}\_\{\\mathrm\{c\}\},\{\\mathbf\{e\}\}\_\{\\varnothing\}\)\\leftarrow\(\{\\mathbf\{e\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\{\\mathbf\{c\}\},\{\\mathbf\{e\}\}\_\{\\text\{sc\}\}\),\\;\{\\mathbf\{e\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\\varnothing,\{\\mathbf\{0\}\}\)\)⊳\\trianglerightone packed forward,[Fig\.4\(a\)](https://arxiv.org/html/2610.02665#S4.F4.sf1) 9: 𝐞sc←𝐞c\{\\mathbf\{e\}\}\_\{\\text\{sc\}\}\\leftarrow\{\\mathbf\{e\}\}\_\{\\mathrm\{c\}\}⊳\\trianglerightSC reuses the*unsteered*conditional mean,[Fig\.4\(b\)](https://arxiv.org/html/2610.02665#S4.F4.sf2) 10: 𝐞^θ←wτ𝐞c\+1−wτ𝐞∅\+\(1−1τ\)𝐳t\(b\)αt\\hat\{\\mathbf\{e\}\}\_\{\\theta\}\\leftarrow\\frac\{w\}\{\\tau\}\{\\mathbf\{e\}\}\_\{\\mathrm\{c\}\}\+\\frac\{1\-w\}\{\\tau\}\{\\mathbf\{e\}\}\_\{\\varnothing\}\+\\big\(1\-\\frac\{1\}\{\\tau\}\\big\)\\frac\{\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\}\{\\alpha\_\{t\}\}⊳\\trianglerightEmbedding mean prediction after CFG & ST,[28](https://arxiv.org/html/2610.02665#A7.E28) 11:ifsampler is DDIMthen 12: ϵ^θ←1σt\(𝐳t\(b\)−αt𝐞^θ\)\\hat\{\\bm\{\\epsilon\}\}\_\{\\theta\}\\leftarrow\\frac\{1\}\{\\sigma\_\{t\}\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\-\\alpha\_\{t\}\\hat\{\\mathbf\{e\}\}\_\{\\theta\}\), 𝐳s\(b\)←αs𝐞^θ\+σsϵ^θ\{\\mathbf\{z\}\}\_\{s\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\leftarrow\\alpha\_\{s\}\\hat\{\\mathbf\{e\}\}\_\{\\theta\}\+\\sigma\_\{s\}\\hat\{\\bm\{\\epsilon\}\}\_\{\\theta\}⊳\\triangleright[DDIM](https://arxiv.org/html/2610.02665#S4.Ex5) 13:else 14: κ←1−eγ\(s\)−γ\(t\)\\kappa\\leftarrow 1\-\{\\mathrm\{e\}\}^\{\\gamma\(s\)\-\\gamma\(t\)\}, ϵ∼𝒩\(𝟎,𝐈\)\{\\bm\{\\epsilon\}\}\\sim\\mathcal\{N\}\(\{\\mathbf\{0\}\},\{\\mathbf\{I\}\}\) 15: 𝐳s\(b\)←\(1−κ\)αsαt𝐳t\(b\)\+καs𝐞^θ\+κ\(1−αs2\)ϵ\{\\mathbf\{z\}\}\_\{s\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\leftarrow\(1\-\\kappa\)\\frac\{\\alpha\_\{s\}\}\{\\alpha\_\{t\}\}\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\+\\kappa\\,\\alpha\_\{s\}\\hat\{\\mathbf\{e\}\}\_\{\\theta\}\+\\sqrt\{\\kappa\(1\-\\alpha\_\{s\}^\{2\}\)\}\\,\{\\bm\{\\epsilon\}\}⊳\\triangleright[DDPM](https://arxiv.org/html/2610.02665#S4.Ex4) 16:endif 17:endfor 18: kb←min\(S,Tgen−\|𝐫\|\)k\_\{b\}\\leftarrow\\min\(S,\\;T\_\{\\text\{gen\}\}\-\|\{\\mathbf\{r\}\}\|\)⊳\\trianglerighttokens to commit: the last window may commit fewer thanSS 19: 𝐫\(b,l\)←argmaxv𝐱θl,v\(𝐳tN\(b\),tN,𝐜,𝐞sc\)\{\\mathbf\{r\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\},l\)\}\\leftarrow\\argmax\_\{v\}\{\\mathbf\{x\}\}\_\{\\theta\}^\{l,v\}\(\{\\mathbf\{z\}\}\_\{t\_\{N\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t\_\{N\};\{\\mathbf\{c\}\},\{\\mathbf\{e\}\}\_\{\\text\{sc\}\}\), l∈\[kb\]l\\in\[k\_\{b\}\]⊳\\trianglerightargmax decoding 20: 𝐜←\(𝐜,𝐫\(b\)\)\{\\mathbf\{c\}\}\\leftarrow\(\{\\mathbf\{c\}\},\{\\mathbf\{r\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\); append the KV of 𝐫\(b\)\{\\mathbf\{r\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}to the causal cache; discard the tail⊳\\trianglerighttheL′−kbL^\{\\prime\}\-k\_\{b\}tail positions are re\-drawn as fresh noise in the next window 21:endwhile 22:return 𝐫=\(𝐫\(1\),…,𝐫\(b\)\)\{\\mathbf\{r\}\}=\(\{\\mathbf\{r\}\}^\{\(1\)\},\\dots,\{\\mathbf\{r\}\}^\{\(b\)\}\) The outer loop of[Alg\.2](https://arxiv.org/html/2610.02665#alg2)runsB:=⌈Tgen/S⌉B:=\\lceil T\_\{\\text\{gen\}\}/S\\rceiltimes, and each window costsN\+1N\+1denoiser forward passes \(NNdiffusion steps plus one decoding readout\), for a total ofB\(N\+1\)B\(N\+1\)forward passes\. The effective NFE per committed token is thus≈\(N\+1\)/S\\approx\(N\+1\)/S, see[Sec\.G\.5](https://arxiv.org/html/2610.02665#A7.SS5)for a full compute\-vs\-quality sweep\. Settingw=1w=1disables CFG,τ=1\\tau=1disables ST, and never updating𝐞sc\{\\mathbf\{e\}\}\_\{\\text\{sc\}\}\(i\.e\., leaving it as𝟎\{\\mathbf\{0\}\}\) disables SC; each of these recovers a strictly weaker sampler, as shown in[Fig\.10](https://arxiv.org/html/2610.02665#A7.F10)\. ## Appendix EImplementation Details and Additional Results for Pre\-training \[Back to[Sec\.3\.2](https://arxiv.org/html/2610.02665#S3.SS2)in the main paper\.\] ##### Architecture\. Following NLD\([Fu et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib51)\), we adopt the transformer architecture of Ministral\([Liu et al\., 2026a](https://arxiv.org/html/2610.02665#bib.bib57)\): - •3B: 26 layers, width 3072, FFN 9216, 32 query and 8 key/value heads \(head dimension 128\), SwiGLU, RMSNorm, YaRN\-scaled rotary embeddings \(θ=106\\theta=10^\{6\}\), and untied embeddings\. This adds up to 3\.0B non\-embedding and 3\.8B total parameters\. - •8B: 34 layers, width 4096, FFN 14336, same 32 query and 8 key/value heads \(head dimension 128\) and same SwiGLU, RMSNorm, YaRN rotary embeddings and untied embeddings\. This adds up to 7\.4B non\-embedding and 8\.5B total parameters\. ##### Optimization\. Forγ~\\tilde\{\\gamma\}and𝐄\{\\mathbf\{E\}\}, we use AdamW withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95, no weight decay \(rows of𝐄\{\\mathbf\{E\}\}unit\-norm\), and peak learning rateηpeak=1×10−2\\eta\_\{\\text\{peak\}\}=1\\times 10^\{\-2\};γ0,γ1\\gamma\_\{0\},\\gamma\_\{1\}use the same setting but with weight decay0\.10\.1\([Yang et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib60)\)\. For other parameters, we follow NLD and use AdamW withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95, weight decay0\.10\.1, andηpeak=1×10−5\\eta\_\{\\text\{peak\}\}=1\\times 10^\{\-5\}\. For all parameters, we use gradient clipping value1\.01\.0\. The learning rate schedule is Warmup\-Stable\-Decay over 150K steps:1%1\\%linear warmup toηpeak\\eta\_\{\\text\{peak\}\}, constant through step 120K, then exponential decay to10%10\\%ofηpeak\\eta\_\{\\text\{peak\}\}\. ##### Self\-conditioning \(SC\)\. SC injects the model’s own detached clean\-embedding prediction into the projected noised input stream with a bias\-free, zero\-initialized linear layer mappingℝde\\mathbb\{R\}^\{d\_\{e\}\}to the hidden widthℝd\\mathbb\{R\}^\{d\}: a first pass with𝐞sc=𝟎\{\\mathbf\{e\}\}\_\{\\text\{sc\}\}=\{\\mathbf\{0\}\}, then a second pass withdesg\(𝐞sc\)\\sqrt\{d\_\{e\}\}\\,\\operatorname\{sg\}\(\{\\mathbf\{e\}\}\_\{\\text\{sc\}\}\)through that linear layer\. ##### Time conditioning\. For diffusion stream, a singleγ\\gammais sampled per block and shared across its tokens\. We encodeγ\\gammausing fixed sinusoidal features with frequenciesexp\(linspace\(−5,5,32\)\)\\exp\(\\mathrm\{linspace\}\(\-5,5,32\)\), concatenating\[sin,cos\]\[\\sin,\\cos\]into 64 dimensions\. A bias\-free, zero\-initialized linear layer projects this encoding to the hidden width, where it is added to the projected noised input\. ##### Numerical precision\. All parameters are stored inBF16, and the transformer also computes inBF16through the forward and backward pass, its linear logit heads included; gradient accumulation and the data\-parallel all\-reduce run inFP32\. The small modules outside the transformer upcast their weights at use\. Among these, most computes inFP32: the diffusion embeddings and their Gaussian corruption and up\-projection, the sinusoidalγ\\gammafeatures and their up\-projection, and the SC up\-projection\. The noise schedule computes inFP64, covering the endpointsγ0,γ1\\gamma\_\{0\},\\gamma\_\{1\}, the minus log\-SNRγ\\gammaand its derivative, and theαt,σt\\alpha\_\{t\},\\sigma\_\{t\}derived from them\. These dropFP64at three points:γ\\gammabefore the sinusoidal features,αt,σt\\alpha\_\{t\},\\sigma\_\{t\}at the corruption, and the variance rescaling before the up\-projection for𝐳t\{\\mathbf\{z\}\}\_\{t\}\. These modules incur negligible FLOPs, so the extra precision is a cheap precaution followed from[Gulrajani and Hashimoto \(2023\)](https://arxiv.org/html/2610.02665#bib.bib16);[Yang et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib60)\. The optimizer keepsFP32master weights and Adam moments, sharded across data\-parallel ranks \(distributed optimizer\)\. ##### AR regularization\. We searched overwdiff∈\{0\.3,1\.0\}w\_\{\\text\{diff\}\}\\in\\\{0\.3,1\.0\\\}\. Interestingly, whenwdiff=1\.0w\_\{\\text\{diff\}\}=1\.0, adding the AR regularization even improved diffusion PPL \([Fig\.3\(c\)](https://arxiv.org/html/2610.02665#S3.F3.sf3)\)\. However, we found that switching towdiff=0\.3w\_\{\\text\{diff\}\}=0\.3at 200B tokens offers the best convergence for both losses \([Fig\.3\(c\)](https://arxiv.org/html/2610.02665#S3.F3.sf3)\); this is our flagship 3BSigmamodel which we evaluate in subsequent sections\. We also trained our 8BSigmamodel using this configuration\. ### E\.1Sketched Uniform Spherical Regularization \(SUSReg\) \[Back to[Sec\.5\.2](https://arxiv.org/html/2610.02665#S5.SS2)in the main paper\.\] To avoid theV=131072V=131072embedding vectors inde=16d\_\{e\}=16dimensional unit sphere being too tightly clustered for accurate decoding \(as observed in their low\-rank structure in[Yang et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib60)\), we propose to add an optional regularization term based on SUSReg\([Nicollier et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib80)\)to the embeddings, which regularizes their distributionp𝐄:=1V∑v=1Vδ𝐞vp\_\{\\mathbf\{E\}\}:=\\frac\{1\}\{V\}\\sum\_\{v=1\}^\{V\}\\delta\_\{\{\\mathbf\{e\}\}\_\{v\}\}towardUnif\(𝕊de−1\)\\operatorname\{Unif\}\(\\mathbb\{S\}^\{d\_\{e\}\-1\}\), the uniform distribution on the unit sphere inℝde\\mathbb\{R\}^\{d\_\{e\}\}\. ForMMdirections𝐚m∼Unif\(𝕊de−1\)\\mathbf\{a\}\_\{m\}\\sim\\operatorname\{Unif\}\(\\mathbb\{S\}^\{d\_\{e\}\-1\}\), we compare the empirical characteristic function of each one\-dimensional projection, ϕ𝐚m\(t\)=1V∑veit⟨𝐚m,𝐞v⟩,\\displaystyle\\phi\_\{\\mathbf\{a\}\_\{m\}\}\(t\)=\\frac\{1\}\{V\}\\sum\_\{v\}\{\\mathrm\{e\}\}^\{\\mathrm\{i\}t\\langle\\mathbf\{a\}\_\{m\},\{\\mathbf\{e\}\}\_\{v\}\\rangle\},to the closed\-form target characteristic function of one\-dimensional projection ofUnif\(𝕊de−1\)\\operatorname\{Unif\}\(\\mathbb\{S\}^\{d\_\{e\}\-1\}\), ϕde\(t\)=∫−11cos\(tx\)ρde\(x\)𝑑x,ρde\(x\)=Γ\(de2\)πΓ\(de−12\)\(1−x2\)de−32𝟏\|x\|≤1,\\displaystyle\\phi\_\{d\_\{e\}\}\(t\)=\\int\_\{\-1\}^\{1\}\\cos\(tx\)\\rho\_\{d\_\{e\}\}\(x\)\\,\{\\mathrm\{d\}\}x,\\quad\\rho\_\{d\_\{e\}\}\(x\)=\\frac\{\\Gamma\(\\frac\{d\_\{e\}\}\{2\}\)\}\{\\sqrt\{\\pi\}\\Gamma\(\\frac\{d\_\{e\}\-1\}\{2\}\)\}\(1\-x^\{2\}\)^\{\\frac\{d\_\{e\}\-3\}\{2\}\}\{\\mathbf\{1\}\}\_\{\|x\|\\leq 1\},under an Epps–Pulley Gaussian\-weightedL2L^\{2\}metric: s\(𝐄\):=VM∑m=1M∫−tmaxtmax\|ϕ𝐚m\(t\)−ϕde\(t\)\|2e−t2/2dt\.\\displaystyle s\(\{\\mathbf\{E\}\}\):=\\frac\{V\}\{M\}\\sum\_\{m=1\}^\{M\}\\int\_\{\-t\_\{\\max\}\}^\{t\_\{\\max\}\}\\bigl\|\\phi\_\{\\mathbf\{a\}\_\{m\}\}\(t\)\-\\phi\_\{d\_\{e\}\}\(t\)\\bigr\|^\{2\}\{\\mathrm\{e\}\}^\{\-t^\{2\}/2\}\\,\{\\mathrm\{d\}\}t\.\(16\)The multiplierVVensures that𝔼𝐚1:M\[s\(𝐄\)\]\\E\_\{\\mathbf\{a\}\_\{1:M\}\}\[s\(\{\\mathbf\{E\}\}\)\], asV→∞V\\to\\infty, converges to a limit that is independent ofVVand only depends onded\_\{e\}\. We addλsusregs\(𝐄\)\\lambda\_\{\\text\{susreg\}\}s\(\{\\mathbf\{E\}\}\)per tokento the block NELBOℒBD\(𝐱\)\\mathcal\{L\}\_\{\\text\{BD\}\}\(\{\\mathbf\{x\}\}\)[3](https://arxiv.org/html/2610.02665#S3.E3)on the diffusion arm before combining with the AR auxiliary, so the SUSReg\-to\-ELBO gradient ratio on𝐄\{\\mathbf\{E\}\}is exactlyλsusreg\\lambda\_\{\\text\{susreg\}\}regardless of the joint weightswdiff,warw\_\{\\text\{diff\}\},w\_\{\\text\{ar\}\}\([Alg\.1](https://arxiv.org/html/2610.02665#alg1)\)\. At the values used in our 3B pre\-training run,λsusreg=0\.005\\lambda\_\{\\text\{susreg\}\}=0\.005,M=512M=512directions, and thett\-integral discretized onP=17P=17trapezoidal points over\[−5,5\]\[\-5,5\]; the resulting SUSReg contribution to the reported loss iswdiffλsusregs\(𝐄\)w\_\{\\text\{diff\}\}\\lambda\_\{\\text\{susreg\}\}s\(\{\\mathbf\{E\}\}\)\. We remark that the cost for computings\(𝐄\)s\(\{\\mathbf\{E\}\}\),O\(VM\(de\+P\)\)O\(VM\(d\_\{e\}\+P\)\), is negligible compared with the cost of transformer forward pass\. Figure 8:Density plot of a random𝕊15→𝕊2\\mathbb\{S\}^\{15\}\\to\\mathbb\{S\}^\{2\}Mollweide projection of all learned embeddings living on𝕊15\\mathbb\{S\}^\{15\}\. ### E\.2Ablation Study: MDLM Initialization for Pre\-training \[Back to[Sec\.3\.2](https://arxiv.org/html/2610.02665#S3.SS2)in the main paper\.\] Sigma\-3B featured in our main results is warm\-started from an AR checkpoint \(“AR initialization” in[Tab\.1](https://arxiv.org/html/2610.02665#S4.T1)\) and then trained for300300B tokens under the joint diffusion \+ AR objective \([Sec\.3\.2\.2](https://arxiv.org/html/2610.02665#S3.SS2.SSS2)\)\. We ablate that choice against a masked diffusion language model \(MDLM\) warm start: identical300300B\-token recipe, run instead from the released Nemotron\-Labs\-Diffusion\-3B checkpoint of[Fu et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib51), which is our main baseline in[Tab\.1](https://arxiv.org/html/2610.02665#S4.T1)\. All other knobs are held fixed\. \(a\)Validation Loss \(AR\)\(b\)Validation Loss \(Diffusion\) Figure 9:Pre\-trainingSigma\-3Bwith block size 32 and AR regularization from AR initialization \(AR init\) and from MDLM initialization \(MDLM init\)\. PPL for AR stream \(a\) and diffusion stream \(b\) are computed usingℒar\(𝐱\)\\mathcal\{L\}\_\{\\text\{ar\}\}\(\{\\mathbf\{x\}\}\)[6](https://arxiv.org/html/2610.02665#S3.E6)andℒBD\(𝐱\)\\mathcal\{L\}\_\{\\text\{BD\}\}\(\{\\mathbf\{x\}\}\)[3](https://arxiv.org/html/2610.02665#S3.E3), respectively\. MDLM initialization has an initial advantage that dies off during learning rate decay\.##### Evaluation setup\. In[Tab\.5](https://arxiv.org/html/2610.02665#A5.T5), we evaluate the performance of the two trainedSigma\-3B models at the two CFG\-ST configurations of[Tab\.1](https://arxiv.org/html/2610.02665#S4.T1), both with self\-conditioning enabled: DDPM \(256256steps, EDMρ=7\\rho\{=\}7schedule,w=2w=2,τ=0\.1\\tau=0\.1\) and DDIM \(3232steps, uniform\-γ\\gammaschedule,w=3w=3,τ=0\.3\\tau=0\.3\)\. Numbers are mean±\\pmtwo standard errors over1010sampling lanes, forming a95%95\\%confidence interval of the mean\. Table 5:Diffusion initialization ablation forSigma\-3B pre\-training\.Same six generative reasoning tasks as[Tab\.1](https://arxiv.org/html/2610.02665#S4.T1)\. The*AR init*rows are copied verbatim from[Tab\.1](https://arxiv.org/html/2610.02665#S4.T1)\. The*MDLM init*rows use the same300300B\-token training recipe but warm\-started from the Nemotron\-Labs\-Diffusion\-3B checkpoint of[Fu et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib51)\.Δ=\\Delta=MDLM−\-AR\. All metrics are mean±\\pmtwo standard errors over1010sampling lanes\.Avg\.is the unweighted mean over the six tasks\.SamplerInitGSM8KMinerva MathHumanEvalHumanEval\+MBPPMBPP\+Avg\.AR64\.34DDPM \(256256steps,w=2,τ=0\.1w\{=\}2,\\tau\{=\}0\.1\)MDLM64\.67Δ\\Delta\-0\.05\+0\.77\-2\.32\-0\.97\+3\.24\+1\.30\+0\.33AR60\.45DDIM \(3232steps,w=3,τ=0\.3w\{=\}3,\\tau\{=\}0\.3\)MDLM61\.64Δ\\Delta\+1\.80\+1\.47\-3\.60\-0\.49\+3\.98\+3\.97\+1\.19 ##### Results\. MDLM initialization has a small but consistent win on average at both configurations, but the sign of the change is task\-dependent rather than uniform\. MBPP and MBPP\+ strongly favor MDLM initialization, while HumanEval and HumanEval\+ show opposite trends\. ## Appendix FImplementation Details and Additional Results for Likelihood\-based Benchmarks \[Back to[Sec\.3\.2](https://arxiv.org/html/2610.02665#S3.SS2)in the main paper\.\] Table 6:Likelihood\-based multiple\-choice benchmarks \(3B Base\)\.Accuracy \(%\) per task\.*AR mode*ranks candidates with the AR likelihood from the AR stream;*diffusion mode*ranks them by the variational bound of the diffusion stream:[17](https://arxiv.org/html/2610.02665#A6.Ex13)forSigma, the count\-based masked diffusion NELBO[19](https://arxiv.org/html/2610.02665#A6.E19)for Nemotron\-Labs\-Diffusion\.Underline: the best within each mode\.Bold: the best value across all methods\.MethodARC\-eARC\-cHellaSwagPIQAWinoGrandeMMLUAvg\.Chance25\.0025\.0025\.0050\.0050\.0025\.0033\.33AR initialization80\.3957\.4271\.7478\.1870\.6470\.7471\.52*AR mode*Nemotron\-Labs\-Diffusion\-3B78\.7055\.4670\.3978\.2470\.3269\.6170\.46Sigma\-3B\(Ours\)79\.2955\.2969\.9877\.6469\.4668\.5070\.03*Diffusion mode*Nemotron\-Labs\-Diffusion\-3B78\.0752\.9964\.9073\.7868\.7569\.7068\.03Sigma\-3B\(Ours\)78\.9654\.6165\.6473\.6169\.9367\.6668\.40 Table 7:Likelihood\-based multiple\-choice benchmarks \(8B Base\)\.Accuracy \(%\) per task\.*AR mode*ranks candidates with the AR likelihood from the AR stream;*diffusion mode*ranks them by the variational bound of the diffusion stream:[17](https://arxiv.org/html/2610.02665#A6.Ex13)forSigma, the count\-based masked diffusion NELBO[19](https://arxiv.org/html/2610.02665#A6.E19)for Nemotron\-Labs\-Diffusion\.Underline: the best within each mode\.Bold: the best value across all methods\.MethodARC\-eARC\-cHellaSwagPIQAWinoGrandeMMLUAvg\.Chance25\.0025\.0025\.0050\.0050\.0025\.0033\.33AR initialization83\.8061\.5277\.0580\.0376\.4876\.8475\.95*AR mode*Nemotron\-Labs\-Diffusion\-8B82\.3259\.7376\.0979\.7675\.9374\.8874\.78Sigma\-8B\(Ours\)82\.2860\.6775\.7080\.4176\.4874\.5875\.02*Diffusion mode*Nemotron\-Labs\-Diffusion\-8B80\.8158\.6271\.4777\.3173\.8876\.0173\.02Sigma\-8B\(Ours\)80\.8558\.1969\.0376\.4475\.2273\.9672\.28 ##### Benchmarks\. We evaluate on ARC\-e/c\([Clark et al\., 2018](https://arxiv.org/html/2610.02665#bib.bib82)\), HellaSwag\([Zellers et al\., 2019](https://arxiv.org/html/2610.02665#bib.bib83)\), PIQA\([Bisk et al\., 2020](https://arxiv.org/html/2610.02665#bib.bib84)\), WinoGrande\([Sakaguchi et al\., 2020](https://arxiv.org/html/2610.02665#bib.bib85)\), and MMLU\([Hendrycks et al\., 2021](https://arxiv.org/html/2610.02665#bib.bib86)\)\. Per Open LLM Leaderboard convention we reportacc\_norm\(normalized by character length\) on the four length\-varying 0\-shot tasks and plainaccon WinoGrande and MMLU\. These benchmarks score a candidate response𝐫\{\\mathbf\{r\}\}against a prompt𝐩\{\\mathbf\{p\}\}by comparinglogpθ\(𝐫∣𝐩\)\\log p\_\{\\theta\}\(\{\\mathbf\{r\}\}\\mid\{\\mathbf\{p\}\}\)across candidates\. ##### Scoring our continuous method\. Sigmadoes not give likelihood in closed form, but a surrogate based on the NELBO: taking the prompt as the clean prefix and the candidate as the block to be denoised, and then applying[4](https://arxiv.org/html/2610.02665#S3.Ex2)verbatim with𝐩\{\\mathbf\{p\}\}as𝐱\(<b\)\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}and𝐫\{\\mathbf\{r\}\}as𝐱\(b\)\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}: −logpθ\(𝐫∣𝐩\)≤ℒNELBO\(𝐫,𝐩\)\\displaystyle\-\\log p\_\{\\theta\}\(\{\\mathbf\{r\}\}\\mid\{\\mathbf\{p\}\}\)\\leq\\mathcal\{L\}\_\{\\text\{NELBO\}\}\(\{\\mathbf\{r\}\};\{\\mathbf\{p\}\}\)=KL\(q\(𝐳1∣𝐫\)∥p\(𝐳1\)\)\+𝔼q\(𝐳0∣𝐫\)\[∑l=1\|𝐫\|−log⟨𝐱θl\(𝐳0,0;𝐩\),𝐫l⟩\]\\displaystyle=\\operatorname\{KL\}\(q\(\{\\mathbf\{z\}\}\_\{1\}\\mid\{\\mathbf\{r\}\}\)\\,\\\|\\,p\(\{\\mathbf\{z\}\}\_\{1\}\)\)\+\\E\_\{q\(\{\\mathbf\{z\}\}\_\{0\}\\mid\{\\mathbf\{r\}\}\)\}\\big\[\{\\textstyle\\sum\}\_\{l=1\}^\{\|\{\\mathbf\{r\}\}\|\}\-\\log\\langle\{\\mathbf\{x\}\}\_\{\\theta\}^\{l\}\(\{\\mathbf\{z\}\}\_\{0\},0;\{\\mathbf\{p\}\}\),\{\\mathbf\{r\}\}^\{l\}\\rangle\\big\]−12𝔼t∼Unif\[0,1\]𝔼q\(𝐳t∣𝐫\)\[SNR′\(t\)‖𝐞θ\(𝐳t,t,𝐩\)−𝐞‖2\]\.\\displaystyle\-\\tfrac\{1\}\{2\}\\ \\E\_\{t\\sim\\operatorname\{Unif\}\[0,1\]\}\\E\_\{q\(\{\\mathbf\{z\}\}\_\{t\}\\mid\{\\mathbf\{r\}\}\)\}\[\\operatorname\{SNR\}^\{\\prime\}\(t\)\\\|\{\\mathbf\{e\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\},t;\{\\mathbf\{p\}\}\)\-\{\\mathbf\{e\}\}\\\|^\{2\}\]\.\(17\) In practice, we evaluate[17](https://arxiv.org/html/2610.02665#A6.Ex13)with128128stratified samplestk=k\+u128t\_\{k\}=\\frac\{k\+u\}\{128\},k=0,…,127k=0,\\dots,127,u∼Unif\[0,1\]u\\sim\\operatorname\{Unif\}\[0,1\]and88reconstruction samples\. ##### Scoring the masked baseline via count\-based NELBO\. Here we use the notation for masked diffusion defined in[App\.C](https://arxiv.org/html/2610.02665#A3)\. Let𝐩\{\\mathbf\{p\}\}be the prompt and𝐫\{\\mathbf\{r\}\}be a candidate response\. We generate the masked states by the*mask set*, instead of randomly masking each position with probabilitytt: forM⊆\{1,…,L𝐫\}M\\subseteq\\\{1,\\dots,L\_\{\\mathbf\{r\}\}\\\}, whereL𝐫:=\|𝐫\|L\_\{\\mathbf\{r\}\}:=\|\{\\mathbf\{r\}\}\|, we obtain𝐳M\{\\mathbf\{z\}\}\_\{M\}from masking indices inMMof𝐫\{\\mathbf\{r\}\}, and denoteC\(M\):=∑l∈M−log⟨𝐱θl\(𝐳M;𝐩\),𝐫l⟩C\(M\):=\\sum\_\{l\\in M\}\-\\log\\langle\{\\mathbf\{x\}\}\_\{\\theta\}^\{l\}\(\{\\mathbf\{z\}\}\_\{M\};\{\\mathbf\{p\}\}\),\\,\{\\mathbf\{r\}\}^\{l\}\\rangletheir cross entropy\. Underαt=1−t\\alpha\_\{t\}=1\-t,[9](https://arxiv.org/html/2610.02665#A3.E9)gives the continuous\-time bound; integratingttout yields an average over the*number*of masked tokens\([Ou et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib27);[Nie et al\., 2025b](https://arxiv.org/html/2610.02665#bib.bib38)\): −logpθ\(𝐫∣𝐩\)≤ℒMDLM\(𝐫,𝐩\)\\displaystyle\-\\log p\_\{\\theta\}\(\{\\mathbf\{r\}\}\\mid\{\\mathbf\{p\}\}\)\\;\\leq\\;\\mathcal\{L\}\_\{\\text\{MDLM\}\}\(\{\\mathbf\{r\}\};\{\\mathbf\{p\}\}\)=𝔼t∼Unif\[0,1\]\[1t𝔼M∼Bern\(t\)⊗L𝐫C\(M\)\]\\displaystyle\\;=\\;\\E\_\{t\\sim\\operatorname\{Unif\}\[0,1\]\}\\Big\[\\frac\{1\}\{t\}\\,\\E\_\{M\\sim\\mathrm\{Bern\}\(t\)^\{\\otimes L\_\{\\mathbf\{r\}\}\}\}C\(M\)\\Big\]\(18\)=𝔼n∼Unif\{1,…,L𝐫\}\[L𝐫n𝔼M∼Unif\(\[L𝐫\]n\)C\(M\)\],\\displaystyle\\;=\\;\\E\_\{n\\sim\\operatorname\{Unif\}\\\{1,\\ldots,L\_\{\\mathbf\{r\}\}\\\}\}\\Big\[\\frac\{L\_\{\\mathbf\{r\}\}\}\{n\}\\,\\E\_\{M\\sim\\operatorname\{Unif\}\\binom\{\[L\_\{\\mathbf\{r\}\}\]\}\{n\}\}C\(M\)\\Big\],\(19\)where\(\[L𝐫\]n\)\\binom\{\[L\_\{\\mathbf\{r\}\}\]\}\{n\}means the set of allnn\-element subsets of\[L𝐫\]:=\{1,2,…,L𝐫\}\[L\_\{\\mathbf\{r\}\}\]:=\\\{1,2,\.\.\.,L\_\{\\mathbf\{r\}\}\\\}\.[19](https://arxiv.org/html/2610.02665#A6.E19)is a Rao–Blackwellized low\-variance estimator of[18](https://arxiv.org/html/2610.02665#A6.E18), manifested in the following three major differences:*\(i\)*The1t\\frac\{1\}\{t\}weight becomesL𝐫n≤L𝐫\\frac\{L\_\{\\mathbf\{r\}\}\}\{n\}\\leq L\_\{\\mathbf\{r\}\}, so rare small\-ttdraws that leave one or two masked tokens would not cause the variance of the estimator to blow up\.*\(ii\)*Bernoulli masking leaves𝐫\{\\mathbf\{r\}\}wholly unmasked with probability\(1−t\)L𝐫\(1\-t\)^\{L\_\{\\mathbf\{r\}\}\}, which may be common for short responses and wastes a forward pass, while then≥1n\\geq 1constraint avoids this\.*\(iii\)*Asnnis finite, we can similarly use stratified sampling as thett\-grid above, spreading counts evenly over\{1,…,L𝐫\}\\\{1,\\ldots,L\_\{\\mathbf\{r\}\}\\\}from one uniform offset, so every batch covers all masking rates\. In practice, we use[19](https://arxiv.org/html/2610.02665#A6.E19)with128128low\-discrepancy drawsnkn\_\{k\}and standard random sampling forMkM\_\{k\}for Nemotron\-Labs\-Diffusion\([Fu et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib51)\)in[Tabs\.6](https://arxiv.org/html/2610.02665#A6.T6)and[7](https://arxiv.org/html/2610.02665#A6.T7)\. On MMLU candidates are single tokens, soL𝐫=n=1L\_\{\\mathbf\{r\}\}=n=1and[19](https://arxiv.org/html/2610.02665#A6.E19)is exact after one forward pass, while[18](https://arxiv.org/html/2610.02665#A6.E18)would still be averaging1t\\frac\{1\}\{t\}\. We found[19](https://arxiv.org/html/2610.02665#A6.E19)to work better than[18](https://arxiv.org/html/2610.02665#A6.E18)in early experiments\. ##### Length generalization\. Diffusion training denoises blocks of exactlyL′L^\{\\prime\}tokens from packed sequences of lengthLL, whereas candidates are typically short and occasionally longer thanL′L^\{\\prime\}\. For each\(𝐩,𝐫\)\(\{\\mathbf\{p\}\},\{\\mathbf\{r\}\}\)pair, we apply causal attention over𝐩\{\\mathbf\{p\}\}and bidirectional attention over𝐫\{\\mathbf\{r\}\}, assuming that the bidirectional attention can generalize to response length other thanL′L^\{\\prime\}\. The attention map follows the conditional branch in[Fig\.4\(a\)](https://arxiv.org/html/2610.02665#S4.F4.sf1)\. ## Appendix GImplementation Details and Additional Results for Inference \[Back to[Secs\.4](https://arxiv.org/html/2610.02665#S4)and[5](https://arxiv.org/html/2610.02665#S5)in the main paper\.\] Table 8:Low\-NFE performance ofSigma\-3B \(base model; block size 32\) with DDIM and SUSReg embedding regularizationon math reasoning and coding benchmarks\. Accuracy \(%\) is Pass@1\. TheSigmarows are reported as mean \(over 10 seeds\)±\\pmtwo standard errors\. All results use the uniform\-γ\\gammaschedule\.Boldmarks the best value per task;underlinemarks the better ofSigma\-3Band\+\+SUSReg within each step budget\. For DDIM 256 steps, guidance settings were tuned according to the best 6\-task average accuracy\. For other runs \(except PDD\), settings were selected by tuning on 3\-task average using GSM8K, HumanEval and MBPP\. Within this table, all DDIM runs use SC; PDD does not use SC due to limitations discussed in[Sec\.5\.3](https://arxiv.org/html/2610.02665#S5.SS3)\.MathCodingMethod \(Diffusion Mode\)GSM8KMinerva MathHumanEvalHumanEval\+MBPPMBPP\+Avg\.Δ\\DeltaAvg\.Nemotron\-Labs\-Diffusion\-3B83\.4063\.9656\.1053\.0557\.6074\.3464\.74DDIM 256 steps / blockSigma\-3B\(w=3,τ=0\.2w\{=\}3,\\tau\{=\}0\.2\)63\.88\+ SUSReg \(w=3,τ=0\.2w\{=\}3,\\tau\{=\}0\.2\)64\.34\+0\.46DDIM 32 steps / blockSigma\-3B\(w=3,τ=0\.3w\{=\}3,\\tau\{=\}0\.3\)60\.41\+ SUSReg \(w=4,τ=0\.4w\{=\}4,\\tau\{=\}0\.4\)61\.85\+1\.44DDIM 16 steps / blockSigma\-3B\(w=3,τ=0\.5w\{=\}3,\\tau\{=\}0\.5\)55\.17\+ SUSReg \(w=3,τ=0\.4w\{=\}3,\\tau\{=\}0\.4\)58\.50\+3\.33DDIM 8 steps / blockSigma\-3B\(w=3,τ=0\.8w\{=\}3,\\tau\{=\}0\.8\)37\.06\+ SUSReg \(w=3,τ=0\.7w\{=\}3,\\tau\{=\}0\.7\)42\.76\+5\.70DDIM 4 steps / blockSigma\-3B\(w=2,τ=0\.9w\{=\}2,\\tau\{=\}0\.9\)7\.74\+ SUSReg \(w=2,τ=0\.9w\{=\}2,\\tau\{=\}0\.9\)11\.80\+4\.06PDD 8 steps / blockSigma\-3B\(w=3,τ=0\.3w\{=\}3,\\tau\{=\}0\.3\)42\.12\+ SUSReg \(w=3,τ=0\.3w\{=\}3,\\tau\{=\}0\.3\)46\.79\+4\.67PDD 4 steps / blockSigma\-3B\(w=4,τ=0\.4w\{=\}4,\\tau\{=\}0\.4\)31\.74\+ SUSReg \(w=4,τ=0\.4w\{=\}4,\\tau\{=\}0\.4\)35\.45\+3\.71 ### G\.1Reasoning and Coding Benchmark Setup \[Back to[Secs\.4](https://arxiv.org/html/2610.02665#S4)and[5](https://arxiv.org/html/2610.02665#S5)in the main paper\.\] We describe the harness, scoring pipeline, and sampler that produce[Tabs\.1](https://arxiv.org/html/2610.02665#S4.T1),[2](https://arxiv.org/html/2610.02665#S4.T2)and[8](https://arxiv.org/html/2610.02665#A7.T8)\. The scoring pipeline is consistent for all base models and generation modes\. ##### Benchmarks and few\-shot budgets\. We use six tasks fromlm\-evaluation\-harnessfollowing[Fu et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib51): GSM8K \(8\-shot, 1319 questions\)\([Cobbe et al\., 2021](https://arxiv.org/html/2610.02665#bib.bib45)\), Minerva Math \(4\-shot, 5000 questions222Number of questions in each sub\-domain: algebra 1187, counting/prob 474, geometry 479, intermediate algebra 903, number theory 540, prealgebra 871, precalc 546\.\)\([Lewkowycz et al\., 2022](https://arxiv.org/html/2610.02665#bib.bib87)\), HumanEval and HumanEval\+ \(0\-shot, 164 questions\)\([Chen et al\., 2021](https://arxiv.org/html/2610.02665#bib.bib88)\), and MBPP and MBPP\+ \(3\-shot, 500 and 378 questions, respectively\)\([Austin et al\., 2021b](https://arxiv.org/html/2610.02665#bib.bib89)\)\. The per\-task generation budget is 256 tokens for GSM8K and 512 tokens for the other five tasks\. ##### Scoring\. GSM8K is scored by exact\-match on the numerical answer, extracted from the model’s response by a permissive regex that keeps the last number appearing in the completion \(stripping separators and trailing punctuation\)\. Minerva Math is scored by symbolic equivalence after canonicalizing boxed and inline expressions\. HumanEval and HumanEval\+ report post\-processed pass@1: the completion is stripped out of the Markdown code fence, an AST\-based longest\-valid\-code extraction is applied, and the resulting program is executed against the task test cases in the container\. MBPP and MBPP\+ report the harness pass@1 directly without this post\-processing step, matching the convention of prior work\. ##### Decoding configuration\. Unless overridden in the corresponding row of[Tab\.1](https://arxiv.org/html/2610.02665#S4.T1), allSigmadiffusion\-mode runs use block sizeL′=32L^\{\\prime\}=32,BF16model weights at inference time, argmax readout at the final diffusion step, and operation of SC follows[Fig\.4\(b\)](https://arxiv.org/html/2610.02665#S4.F4.sf2)\. The DDPM\-256 row uses the ancestral sampler of[DDPM](https://arxiv.org/html/2610.02665#S4.Ex4)on the EDMρ=7\\rho=7schedule \([Sec\.G\.6](https://arxiv.org/html/2610.02665#A7.SS6)\); the DDIM\-32 row uses the deterministic sampler of[DDIM](https://arxiv.org/html/2610.02665#S4.Ex5)on the uniform\-γ\\gammaschedule; the row\-specific guidance strengthwwand score temperatureτ\\tauappear in the table caption\. Commit stride is the block size unless otherwise specified\. ##### Seed error bars\. EachSigmadiffusion row aggregates1010independent sampling lanes as follows\. For questionii, we independently draw1010responses under the same decoder with1010different seeds\. Letsi,js\_\{i,j\}be the score of thejj\-th response to questionii\. We first average over questions to obtain1010per\-lane scoressj=meani\(si,j\)s\_\{j\}=\\operatorname\{mean\}\_\{i\}\(s\_\{i,j\}\)forj=1,…,10j=1,\\dots,10, and then obtains¯=meanj\(sj\)\\bar\{s\}=\\operatorname\{mean\}\_\{j\}\(s\_\{j\}\)together with the across\-lane standard deviationσ^=stdj\(sj\)\\hat\{\\sigma\}=\\operatorname\{std\}\_\{j\}\(s\_\{j\}\)\. The half\-widths in[Tab\.1](https://arxiv.org/html/2610.02665#S4.T1)are210σ^\\frac\{2\}\{\\sqrt\{10\}\}\\hat\{\\sigma\}, an approximate95%95\\%confidence interval for the mean\. ##### Baseline decoders\. The AR\-mode for NLD andSigmarow uses greedy argmax at each step; NLD’s diffusion\-mode row uses the confidence\-threshold greedy sampler of[Fu et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib51)\(block sizeL′=32L^\{\\prime\}=32;argmax\\argmaxreadout at every position whose top\-1 probability exceeds a threshold; we report the better of thresholds0\.80\.8and0\.90\.9per task\)\. For LLaDA\-8B and Dream\-7B we quote each model’s best diffusion\-mode number without the confidence\-threshold rule: at each denoising step the single position with the highest top\-1 confidence is unmasked\. We refer readers to[App\.C](https://arxiv.org/html/2610.02665#A3)for details of inference algorithms for masked diffusion models\. ### G\.2Sweeping Classifier\-Free Guidance and Score Temperature \[Back to[Sec\.4\.2](https://arxiv.org/html/2610.02665#S4.SS2)or[Sec\.4\.3](https://arxiv.org/html/2610.02665#S4.SS3)in the main paper\.\] In[Figs\.10](https://arxiv.org/html/2610.02665#A7.F10)and[11](https://arxiv.org/html/2610.02665#A7.F11), we present the full tables for CFG and ST sweep for ourSigma\-3B models trained with and without AR regularization, respectively\. Figure 10:Sweeping CFG and ST forSigma\-3B trained with AR regularization\. Accuracies are averaged over 10 random seeds\.Red: per\-task best;orange: within 1 point of per\-task best;green: best average across 6 tasks\.For this model, per\-task best and best average align on 5 out of 6 tasks\.In the exception of Minerva Math \(subset; random10%10\\%from each subtask\), per\-task best is still within 1 point from best average\. We report best average \(CFG=2\\text\{CFG\}=2,ST=0\.1\\text\{ST\}=0\.1\) in[Fig\.5](https://arxiv.org/html/2610.02665#S4.F5)and[Tab\.1](https://arxiv.org/html/2610.02665#S4.T1)\.Figure 11:Sweeping CFG and ST forSigma\-3B trainedwithoutAR regularization\. Accuracies are averaged over 10 random seeds\.Red: per\-task best;orange: within 1 point of per\-task best;green: best average across 6 tasks\. Both per\-task best and best average are worse thanSigma\-3B trained AR regularization \([Fig\.10](https://arxiv.org/html/2610.02665#A7.F10)\)\. ### G\.3Ablation Study: Steering in Logit Space Instead of Embedding Space \[Back to[Sec\.4\.2](https://arxiv.org/html/2610.02665#S4.SS2)in the main paper\.\] Both steering knobs introduced in[Sec\.4\.2](https://arxiv.org/html/2610.02665#S4.SS2)act on the predicted clean embedding𝐞θ∈ℝL′×de\{\\mathbf\{e\}\}\_\{\\theta\}\\in\\mathbb\{R\}^\{L^\{\\prime\}\\times d\_\{e\}\}, equivalently on the score or velocity\. Recall that our denoiser leveragescategorical reparameterization, i\.e\.,𝐞θ=softmax\(ℓθ\)𝐄=𝐱θ𝐄\{\\mathbf\{e\}\}\_\{\\theta\}=\\operatorname\{softmax\}\(\{\\bm\{\\ell\}\}\_\{\\theta\}\)\{\\mathbf\{E\}\}=\{\\mathbf\{x\}\}\_\{\\theta\}\{\\mathbf\{E\}\}, whereℓθ∈ℝL′×V\{\\bm\{\\ell\}\}\_\{\\theta\}\\in\\mathbb\{R\}^\{L^\{\\prime\}\\times V\}are the predicted logits, andsoftmax\\operatorname\{softmax\}operates in the vocabulary dimension\. Thus, a natural alternative is to apply the same operations toℓθ\{\\bm\{\\ell\}\}\_\{\\theta\}before thesoftmax\\operatorname\{softmax\}\. This is the form of guidance used by discrete dLMs, which does not have embedding\-space prediction to steer\([Nie et al\., 2025b](https://arxiv.org/html/2610.02665#bib.bib38);[Li et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib65);[Schiff et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib29)\)\([App\.A](https://arxiv.org/html/2610.02665#A1)\)\. ##### Logit\-space CFG \(LCFG\)\. Reusing the notations in[Sec\.G\.4](https://arxiv.org/html/2610.02665#A7.SS4), with the same fully unconditional bad prediction, we do softmax on the extrapolated logits: 𝐞θw,logit:=softmax\(ℓθ∅,𝟎\+w\[ℓθ\+−ℓθ∅,𝟎\]\)𝐄,w≥1,\\displaystyle\{\\mathbf\{e\}\}^\{w,\\text\{logit\}\}\_\{\\theta\}:=\\operatorname\{softmax\}\(\{\\bm\{\\ell\}\}^\{\\varnothing,\{\\mathbf\{0\}\}\}\_\{\\theta\}\+w\\,\[\{\\bm\{\\ell\}\}^\{\+\}\_\{\\theta\}\-\{\\bm\{\\ell\}\}^\{\\varnothing,\{\\mathbf\{0\}\}\}\_\{\\theta\}\]\)\{\\mathbf\{E\}\},\\qquad w\\geq 1,against our default𝐞θw=𝐞θ∅,𝟎\+w\[𝐞θ\+−𝐞θ∅,𝟎\]\{\\mathbf\{e\}\}^\{w\}\_\{\\theta\}=\{\\mathbf\{e\}\}^\{\\varnothing,\{\\mathbf\{0\}\}\}\_\{\\theta\}\+w\[\{\\mathbf\{e\}\}^\{\+\}\_\{\\theta\}\-\{\\mathbf\{e\}\}^\{\\varnothing,\{\\mathbf\{0\}\}\}\_\{\\theta\}\]of[CFG](https://arxiv.org/html/2610.02665#S4.Ex6), which extrapolates after the softmax\. The two coincide atw=1w=1\(i\.e\., no guidance\)\. ##### Logit temperature \(LT\)\. This is the analogue of ST in logit space, which rescales the conditional logits themselves: 𝐞θT:=softmax\(ℓθ\+T\)𝐄,T∈\[0,1\],\\displaystyle\{\\mathbf\{e\}\}^\{T\}\_\{\\theta\}:=\\operatorname\{softmax\}\\Big\(\\frac\{\{\\bm\{\\ell\}\}^\{\+\}\_\{\\theta\}\}\{T\}\\Big\)\{\\mathbf\{E\}\},\\qquad T\\in\[0,1\],whereT=1T=1recovers the conditional mean embedding prediction \(i\.e\., no guidance\), and smallerTTsharpens\. TheT→0T\\to 0limit is the maximum a posteriori \(MAP\) denoiser𝐞θ0=𝐄\[argmaxℓθ\+\]\{\\mathbf\{e\}\}^\{0\}\_\{\\theta\}=\{\\mathbf\{E\}\}\[\\argmax\{\\bm\{\\ell\}\}^\{\+\}\_\{\\theta\}\], which snaps the prediction onto a single vocabulary embedding\. We remark that both operations change the trajectory only; tokens are still read out as theargmax\\argmaxof the conditional logits at the final stept=0t=0, exactly as in[Secs\.4\.1](https://arxiv.org/html/2610.02665#S4.SS1)and[2](https://arxiv.org/html/2610.02665#alg2)\. Also, LCFG does not cost extra compute compared with the embedding\-space counterpart, since we need to get both conditional and unconditional logit predictions\. ##### Experimental setup\. We use the setting identical to the default in[Sec\.G\.4](https://arxiv.org/html/2610.02665#A7.SS4)\. We use the sameSigma\-3B checkpoint with block length3232, the full GSM8K and HumanEval evaluation sets, and generate 10 responses per problem\. DDIM uses6464steps/block on a uniform\-γ\\gammaschedule withτ=0\.3\\tau=0\.3, and DDPM uses256256steps/block on the EDMρ=7\\rho=7schedule andτ=1\\tau=1\. Table 9:Logit\-space versus embedding\-space CFG\. GSM8K is accuracy \(%\) and HumanEval is execution\-based pass@1 \(%\)\. Entries are mean±\\pmtwo standard errors over the 10 generations per prompt, an approximate95%95\\%confidence interval for the mean, andboldmarks the best entry in each row\. The last column is the best value over the*full*sweep of[Tabs\.12](https://arxiv.org/html/2610.02665#A7.T12)and[13](https://arxiv.org/html/2610.02665#A7.T13), which continues pastw=5w=5for the embedding rows\. The logit rows were swept only tow=5w=5as they already peak atw=2w=2and fall monotonically after that\.SettingGuidance spacew=1w=1w=2w=2w=3w=3w=4w=4w=5w=5Tuned bestGSM8K, DDIM6464steps,τ=0\.3\\tau=0\.3Logit\(w=2w=2\)Embedding \(ours\)\(w=5w=5\)GSM8K, DDPM256256steps,τ=1\\tau=1Logit\(w=2w=2\)Embedding \(ours\)\(w=10w=10\)HumanEval, DDIM6464steps,τ=0\.3\\tau=0\.3Logit\(w=2w=2\)Embedding \(ours\)\(w=4w=4\)HumanEval, DDPM256256steps,τ=1\\tau=1Logit\(w=2w=2\)Embedding \(ours\)\(w=9w=9\) Table 10:Logit temperature, GSM8K accuracy \(%\) with DDPM,256256steps/block, EDMρ=7\\rho=7schedule, no CFG\. Score entries are mean±\\pmtwo standard errors over the 10 generations per prompt, an approximate95%95\\%confidence interval for the mean\.T=1T=1is the unsharpened MMSE denoiser and is the same configuration as thew=1w=1logit\-space entry of[Tab\.9](https://arxiv.org/html/2610.02665#A7.T9)\. The bottom row is LCFG at the matched weightw=1Tw=\\frac\{1\}\{T\}, repeated from[Tab\.9](https://arxiv.org/html/2610.02665#A7.T9); the two are the same operation if the unconditional logits are uniform over the vocabulary, so the gap isolates what logit\-space guidance adds over plain sharpening\.TT1112\\frac\{1\}\{2\}13\\frac\{1\}\{3\}14\\frac\{1\}\{4\}15\\frac\{1\}\{5\}110\\frac\{1\}\{10\}00\(MAP\)LT atTTLCFG atw=1Tw=\\frac\{1\}\{T\}——Δ\\Delta\(LT−\-LCFG\)0\.00\+0\.12\+3\.91\+9\.49\+18\.34——Table 11:Composing logit temperature with the other two knobs: GSM8K accuracy \(%\) with DDIM,6464steps/block, uniform\-γ\\gammaschedule, STτ=0\.3\\tau=0\.3\. Entries are mean±\\pmtwo standard errors over the 10 generations per prompt, an approximate95%95\\%confidence interval for the mean\.Boldmarks the best entry in each row\.TT1112\\frac\{1\}\{2\}13\\frac\{1\}\{3\}15\\frac\{1\}\{5\}00\(MAP\)No guidance \(w=1w=1\)Embedding CFG atw=5w=5 ##### LCFG peaks early and tops out lower\. [Tab\.9](https://arxiv.org/html/2610.02665#A7.T9)shows that LCFG peaks atw=2w=2in all four settings and then declines monotonically, while embedding\-space CFG keeps improving out tow=4∼10w=4\\sim 10\. Comparing each at its own optimum, logit space gives a marginal gain of\+0\.30\+0\.30on GSM8K/DDIM but loses by5\.685\.68,2\.442\.44, and17\.2617\.26points in the other three settings\. ##### Logit temperature sharpens effectively, and the MAP denoiser does not collapse\. [Tab\.10](https://arxiv.org/html/2610.02665#A7.T10)shows sharpening alone is worth a great deal on DDPM, lifting GSM8K from52\.4452\.44atT=1T=1to79\.1179\.11atT=15T=\\frac\{1\}\{5\}\. The curve then stays flat all the way to the hard endpoint: the entire range fromT=13T=\\frac\{1\}\{3\}toT=0T=0spans only0\.50\.5points, andT=0T=0itself scores78\.6178\.61\. Committing byargmax\\argmaxat high noise therefore costs nothing measurable here, and the on\-manifold denoiser is a viable operating point rather than a degenerate one\. ##### What logit\-space guidance adds over sharpening is neutral at best\. Note that if the unconditional branch predicted zero \(i\.e\., a uniform distribution over the vocabulary\), logit CFG at weightwwwill be equivalent to logit temperature atT=1wT=\\frac\{1\}\{w\}\. The gap between the two therefore measures the contribution of the structured part of the unconditional logits, which is the entire content of logit\-space guidance\. The bottom rows of[Tab\.10](https://arxiv.org/html/2610.02665#A7.T10)show that contribution is negligible atw=2w=2and then strictly harmful at largerww\. So logit\-space CFG is best understood as a more expensive but still poorly behaved sharpener\. ##### Logit temperature composes with ST, but adds nothing on top of CFG\. LT and ST sharpen different objects: LT tempers the categorical posterior and then takes its mean, so its prediction stays inside the convex hull of𝐄\{\\mathbf\{E\}\}and saturates at the MAP vertex, whereas ST extrapolates the mean prediction away from𝐳tαt\\frac\{\{\\mathbf\{z\}\}\_\{t\}\}\{\\alpha\_\{t\}\}and can go outside the unit ball\.[Tab\.11](https://arxiv.org/html/2610.02665#A7.T11)confirms they do compose: on DDIM, where ST=0\.3=0\.3is already active, adding LT atT=12T=\\frac\{1\}\{2\}lifts GSM8K for9\.429\.42points that score temperature alone does not reach\. However, when CFG and ST are already on, adding a third sharpener LT harms accuracy\. ##### Conclusion\. Logit\-space steering does sharpen, and logit temperature in particular is a simple knob requiring no second branch as in[Fig\.4\(a\)](https://arxiv.org/html/2610.02665#S4.F4.sf1)\. However, it does not outperform the default embedding\-space CFG in general\. We therefore keep the embedding\-space formulation of[CFG](https://arxiv.org/html/2610.02665#S4.Ex6)and[ST](https://arxiv.org/html/2610.02665#S4.Ex7)throughout\. ### G\.4Ablation Study: Choosing the Bad Prediction for CFG with Self\-conditioning \[Back to[Sec\.4\.2](https://arxiv.org/html/2610.02665#S4.SS2)in the main paper\.\] Standard CFG\([Ho and Salimans, 2022](https://arxiv.org/html/2610.02665#bib.bib30)\), also following the idea of auto\-guidance\([Karras et al\., 2024](https://arxiv.org/html/2610.02665#bib.bib64)\), extrapolates a conditional prediction away from a “bad” prediction to sharpen the model’s prediction\. Self\-conditioning introduces a second source of conditioning, as is used in existing continuous dLMs \(e\.g\.,[Lee et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib43);[Potaptchik et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib44);[Hu et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib47); see[App\.A](https://arxiv.org/html/2610.02665#A1)for detailed discussion\)\. We define the following notations in this subsection: 𝐞θ\+\\displaystyle\{\\mathbf\{e\}\}^\{\+\}\_\{\\theta\}:=𝐞θ\(𝐳t\(b\),t,𝐜,𝐞sc\),\\displaystyle:=\{\\mathbf\{e\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\{\\mathbf\{c\}\},\{\\mathbf\{e\}\}\_\{\\text\{sc\}\}\),𝐞θ𝐜,𝟎\\displaystyle\{\\mathbf\{e\}\}^\{\{\\mathbf\{c\}\},\{\\mathbf\{0\}\}\}\_\{\\theta\}:=𝐞θ\(𝐳t\(b\),t,𝐜,𝟎\),\\displaystyle:=\{\\mathbf\{e\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\{\\mathbf\{c\}\},\{\\mathbf\{0\}\}\),𝐞θ∅,𝟎\\displaystyle\{\\mathbf\{e\}\}^\{\\varnothing,\{\\mathbf\{0\}\}\}\_\{\\theta\}:=𝐞θ\(𝐳t\(b\),t,∅,𝟎\),\\displaystyle:=\{\\mathbf\{e\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\\varnothing,\{\\mathbf\{0\}\}\),where the good prediction𝐞θ\+\{\\mathbf\{e\}\}^\{\+\}\_\{\\theta\}is the conditional prediction that uses both the clean context and SC; for either bad prediction𝐞θ−∈\{𝐞θ𝐜,𝟎,𝐞θ∅,𝟎\}\{\\mathbf\{e\}\}^\{\-\}\_\{\\theta\}\\in\\\{\{\\mathbf\{e\}\}^\{\{\\mathbf\{c\}\},\{\\mathbf\{0\}\}\}\_\{\\theta\},\{\\mathbf\{e\}\}^\{\\varnothing,\{\\mathbf\{0\}\}\}\_\{\\theta\}\\\}, the guided mean embedding prediction is 𝐞θw=𝐞θ\+\+\(w−1\)\(𝐞θ\+−𝐞θ−\)\.\\displaystyle\{\\mathbf\{e\}\}^\{w\}\_\{\\theta\}=\{\\mathbf\{e\}\}^\{\+\}\_\{\\theta\}\+\(w\-1\)\(\{\\mathbf\{e\}\}^\{\+\}\_\{\\theta\}\-\{\\mathbf\{e\}\}^\{\-\}\_\{\\theta\}\)\.Thecontext\-preservingbaseline keeps the prompt and previously generated blocks in both branches and sets𝐞θ−←𝐞θ𝐜,𝟎\{\\mathbf\{e\}\}^\{\-\}\_\{\\theta\}\\leftarrow\{\\mathbf\{e\}\}^\{\{\\mathbf\{c\}\},\{\\mathbf\{0\}\}\}\_\{\\theta\}; it therefore extrapolates only away from the prediction made without SC\. Our proposed method in[Sec\.4\.2](https://arxiv.org/html/2610.02665#S4.SS2)instead uses the fully unconditional branch𝐞θ−←𝐞θ∅,𝟎\{\\mathbf\{e\}\}^\{\-\}\_\{\\theta\}\\leftarrow\{\\mathbf\{e\}\}^\{\\varnothing,\{\\mathbf\{0\}\}\}\_\{\\theta\}, removing both clean context and SC\. ##### Experimental setup\. We compare the two choices using the sameSigma\-3B checkpoint with block length3232, full GSM8K and HumanEval evaluation sets, and 10 samples per problem\. The DDIM sweep uses6464steps per block, uniform\-γ\\gammaschedule, and STτ=0\.3\\tau=0\.3; the DDPM sweep uses256256steps per block, EDMρ=7\\rho=7schedule, and STτ=1\\tau=1\. We remark that the guidance is inactive atw=1w=1and the two choices coincide\. Table 12:GSM8K accuracy \(%\) when choosing the bad prediction for CFG with SC\. “Context/no\-SC” retains clean context but removes SC \(i\.e\.,𝐞θ−←𝐞θ𝐜,𝟎\{\\mathbf\{e\}\}^\{\-\}\_\{\\theta\}\\leftarrow\{\\mathbf\{e\}\}^\{\{\\mathbf\{c\}\},\{\\mathbf\{0\}\}\}\_\{\\theta\}\); ours removes both \(i\.e\.,𝐞θ−←𝐞θ∅,𝟎\{\\mathbf\{e\}\}^\{\-\}\_\{\\theta\}\\leftarrow\{\\mathbf\{e\}\}^\{\\varnothing,\{\\mathbf\{0\}\}\}\_\{\\theta\}\)\. Score entries are mean±\\pmtwo standard errors over the 10 sampling lanes, an approximate95%95\\%confidence interval for the mean;Δ\\Deltais ours−\-context/no\-SC\.DDIM, 64 steps/block, uniform\-γ\\gammaschedule,τ=0\.3\\tau=0\.3 Guidanceww12345681216Context/no\-SC68\.13±\\pm0\.3872\.48±\\pm0\.2773\.99±\\pm0\.4774\.37±\\pm0\.3575\.12±\\pm0\.5176\.13±\\pm0\.4176\.19±\\pm0\.4475\.60±\\pm0\.5674\.09±\\pm0\.52Ours68\.13±\\pm0\.3882\.52±\\pm0\.2182\.84±\\pm0\.2883\.04±\\pm0\.2083\.19±\\pm0\.1883\.01±\\pm0\.3082\.78±\\pm0\.4081\.26±\\pm0\.4078\.00±\\pm0\.32Δ\\Delta0\.00\+10\.05\+8\.85\+8\.67\+8\.07\+6\.88\+6\.59\+5\.66\+3\.91 DDPM, 256 steps/block, EDMρ=7\\rho=7schedule,τ=1\\tau=1 Guidanceww12345678910Context/no\-SC53\.09±\\pm0\.8562\.43±\\pm0\.9765\.44±\\pm0\.7767\.53±\\pm0\.5868\.45±\\pm0\.7768\.45±\\pm0\.5667\.66±\\pm0\.6167\.82±\\pm1\.1767\.09±\\pm0\.4866\.76±\\pm0\.56Ours53\.09±\\pm0\.8577\.12±\\pm0\.6680\.04±\\pm0\.3980\.99±\\pm0\.4881\.30±\\pm0\.4182\.00±\\pm0\.4781\.54±\\pm0\.6482\.05±\\pm0\.2982\.41±\\pm0\.5182\.43±\\pm0\.50Δ\\Delta0\.00\+14\.69\+14\.60\+13\.46\+12\.85\+13\.55\+13\.88\+14\.23\+15\.32\+15\.67 Table 13:HumanEval execution\-based pass@1 \(%\) when choosing the bad prediction for CFG with SC\. “Context/no\-SC” retains clean context but removes SC \(i\.e\.,𝐞θ−←𝐞θ𝐜,𝟎\{\\mathbf\{e\}\}^\{\-\}\_\{\\theta\}\\leftarrow\{\\mathbf\{e\}\}^\{\{\\mathbf\{c\}\},\{\\mathbf\{0\}\}\}\_\{\\theta\}\); ours removes both \(i\.e\.,𝐞θ−←𝐞θ∅,𝟎\{\\mathbf\{e\}\}^\{\-\}\_\{\\theta\}\\leftarrow\{\\mathbf\{e\}\}^\{\\varnothing,\{\\mathbf\{0\}\}\}\_\{\\theta\}\)\. Score entries are mean±\\pmtwo standard errors over the 10 sampling lanes, an approximate95%95\\%confidence interval for the mean;Δ\\Deltais ours−\-context/no\-SC\.DDIM, 64 steps/block, uniform\-γ\\gammaschedule,τ=0\.3\\tau=0\.3 Guidanceww12345681216Context/no\-SC42\.93±\\pm0\.6646\.65±\\pm1\.0850\.00±\\pm0\.8150\.85±\\pm0\.6150\.91±\\pm0\.8650\.43±\\pm0\.6649\.39±\\pm1\.3043\.23±\\pm1\.2040\.85±\\pm2\.07Ours42\.93±\\pm0\.6653\.96±\\pm0\.9556\.04±\\pm1\.1656\.22±\\pm0\.8354\.09±\\pm1\.2254\.51±\\pm0\.8954\.02±\\pm0\.7843\.66±\\pm1\.5231\.22±\\pm1\.62Δ\\Delta0\.00\+7\.31\+6\.04\+5\.37\+3\.18\+4\.08\+4\.63\+0\.43\-9\.63 DDPM, 256 steps/block, EDMρ=7\\rho=7schedule,τ=1\\tau=1 Guidanceww12345678910Context/no\-SC13\.23±\\pm1\.5020\.85±\\pm2\.2925\.73±\\pm1\.5127\.93±\\pm1\.6629\.57±\\pm1\.0329\.76±\\pm1\.7531\.40±\\pm2\.0732\.38±\\pm1\.8832\.50±\\pm2\.2630\.00±\\pm1\.35Ours13\.23±\\pm1\.5032\.62±\\pm1\.9840\.55±\\pm1\.7944\.63±\\pm1\.0746\.71±\\pm1\.8246\.22±\\pm2\.1245\.24±\\pm1\.4945\.61±\\pm1\.0346\.95±\\pm1\.7345\.37±\\pm1\.63Δ\\Delta0\.00\+11\.77\+14\.82\+16\.70\+17\.14\+16\.46\+13\.84\+13\.23\+14\.45\+15\.37 ##### Results\. As shown in[Tabs\.12](https://arxiv.org/html/2610.02665#A7.T12)and[13](https://arxiv.org/html/2610.02665#A7.T13), the fully unconditional bad prediction \(ours\) is consistently stronger in the useful guidance range: it almost wins every matched CFG weight in sweep on both tasks and both samplers, and, tuned over the sweep, reaches a higher ceiling in each task\. Thus, we conclude that the advantage shown here is not an artifact of using different guidance search ranges\. ### G\.5Ablation Study: Commit\-Stride Sampling \[Back to[Sec\.4](https://arxiv.org/html/2610.02665#S4)in the main paper\.\] Beyond blockwise decoding ofL′L^\{\\prime\}tokens per denoising window, we further introduce a*commit stride*S∈\{1,…,L′\}S\\in\\\{1,\\dots,L^\{\\prime\}\\\}\. Each denoising window still runs the same blockwise reverse process over a block of fresh Gaussian noise with lengthL′=32L^\{\\prime\}=32, but only the leadingS∈\{8,16,32\}S\\in\\\{8,16,32\\\}tokens are committed back into the causal KV cache; the remainingL′−SL^\{\\prime\}\-Stail positions are discarded and re\-drawn as fresh noise in the next window \(which is shifted forward bySSabsolute positions\)\. SettingS←L′S\\leftarrow L^\{\\prime\}recovers the default block\-decoding method\. The pseudocode for commit stride can be found in[Alg\.2](https://arxiv.org/html/2610.02665#alg2)\. This is a pure inference\-time technique that further improves the output quality even under matched compute budgets, as we will see below\. ##### Setup\. We evaluate on the base models of[Tabs\.1](https://arxiv.org/html/2610.02665#S4.T1)and[2](https://arxiv.org/html/2610.02665#S4.T2):Sigma\-3B andSigma\-8B, both with block sizeL′=32L^\{\\prime\}=32\. Decoding invariants match the diffusion\-mode rows of[Tab\.1](https://arxiv.org/html/2610.02665#S4.T1)\. For each stride, CFG weightwwand STτ\\tauare chosen based on the number of DDIM steps per window:\(w,τ\)=\(3,0\.2\)\(w,\\tau\)=\(3,0\.2\)at≥64\\geq 64steps,\(3,0\.3\)\(3,0\.3\)at3232steps,\(3,0\.5\)\(3,0\.5\)at1616steps, and\(3,0\.8\)\(3,0\.8\)at88steps, following the recommendation in[Fig\.14](https://arxiv.org/html/2610.02665#A7.F14)\. All cells generate1010samples for each prompt and report mean±\\pmtwo standard errors \([Sec\.G\.1](https://arxiv.org/html/2610.02665#A7.SS1)\)\. ##### Nominal vs\. true NFE\. For simplicity, we match compute cost by*nominal*NFE, defined as the number of denoiser forward passes required to commitL′=32L^\{\\prime\}=32tokens under the given \(steps, stride\) pair: Nominal NFE:=NL′S,\\text\{Nominal\\penalty\\ NFE\}\\;:=\\;\\frac\{NL^\{\\prime\}\}\{S\},whereNNis the number of diffusion steps per denoising window and the\+1\+1endpointargmax\\argmaxreadout is dropped so that the count is exactly proportional to the diffusion\-step budget\. Since every denoising window operates on the*same*L′L^\{\\prime\}\-wide noisy block appended to the same\-length clean prefix, one forward pass costs the same FLOPs regardless ofSS, and two rows sharing a nominal\-NFE bucket in[Tabs\.14](https://arxiv.org/html/2610.02665#A7.T14)and[15](https://arxiv.org/html/2610.02665#A7.T15)are FLOP\-matched \(up to a1/N1/Ncorrection from the shared readout\)\. Table 14:Commit\-stride ablation onSigma\-3B\.Same six generative reasoning tasks as[Tab\.1](https://arxiv.org/html/2610.02665#S4.T1)\.*Nom\. NFE*is the nominal number of forward passes per3232committed tokens \(i\.e\.,steps/stride×32\\text\{steps\}/\\text\{stride\}\\times 32\)\. All DDIM rows use the uniform\-γ\\gammaschedule; the DDPM row uses the EDMρ=7\\rho=7schedule\. Rows within each nominal\-NFE bucket are directly compute\-comparable\. Mean±\\pmtwo standard errors across1010sampling lanes\.Boldmarks the best value per column across the stride sweep\.SamplerNom\. NFEStepsStrideGSM8KMinerva MathHumanEvalHumanEval\+MBPPMBPP\+Avg\.DDPM \(w=2,τ=0\.1w\{=\}2,\\tau\{=\}0\.1\)2562563264\.34DDIM \(w=3,τ=0\.2w\{=\}3,\\tau\{=\}0\.2\)2561281665\.60DDIM \(w=3,τ=0\.2w\{=\}3,\\tau\{=\}0\.2\)25664867\.00DDIM \(w=3,τ=0\.2w\{=\}3,\\tau\{=\}0\.2\)1281283263\.10DDIM \(w=3,τ=0\.2w\{=\}3,\\tau\{=\}0\.2\)128641665\.17DDIM \(w=3,τ=0\.3w\{=\}3,\\tau\{=\}0\.3\)12832866\.55DDIM \(w=3,τ=0\.2w\{=\}3,\\tau\{=\}0\.2\)64643261\.71DDIM \(w=3,τ=0\.3w\{=\}3,\\tau\{=\}0\.3\)64321664\.08DDIM \(w=3,τ=0\.5w\{=\}3,\\tau\{=\}0\.5\)6416864\.70DDIM \(w=3,τ=0\.3w\{=\}3,\\tau\{=\}0\.3\)32323260\.45DDIM \(w=3,τ=0\.5w\{=\}3,\\tau\{=\}0\.5\)32161661\.82DDIM \(w=3,τ=0\.8w\{=\}3,\\tau\{=\}0\.8\)328857\.76 Table 15:Commit\-stride ablation onSigma\-8B\.Same six tasks as[Tab\.2](https://arxiv.org/html/2610.02665#S4.T2); conventions match[Tab\.14](https://arxiv.org/html/2610.02665#A7.T14)\.Boldmarks the best value per column across the stride sweep\.SamplerNom\. NFEStepsStrideGSM8KMinerva MathHumanEvalHumanEval\+MBPPMBPP\+Avg\.DDPM \(w=2,τ=0\.1w\{=\}2,\\tau\{=\}0\.1\)2562563269\.66DDIM \(w=3,τ=0\.2w\{=\}3,\\tau\{=\}0\.2\)2561281670\.33DDIM \(w=3,τ=0\.2w\{=\}3,\\tau\{=\}0\.2\)25664870\.42DDIM \(w=3,τ=0\.2w\{=\}3,\\tau\{=\}0\.2\)1281283267\.87DDIM \(w=3,τ=0\.2w\{=\}3,\\tau\{=\}0\.2\)128641669\.46DDIM \(w=3,τ=0\.3w\{=\}3,\\tau\{=\}0\.3\)12832870\.09DDIM \(w=3,τ=0\.2w\{=\}3,\\tau\{=\}0\.2\)64643267\.62DDIM \(w=3,τ=0\.3w\{=\}3,\\tau\{=\}0\.3\)64321668\.33DDIM \(w=3,τ=0\.5w\{=\}3,\\tau\{=\}0\.5\)6416868\.31DDIM \(w=3,τ=0\.3w\{=\}3,\\tau\{=\}0\.3\)32323266\.72DDIM \(w=3,τ=0\.5w\{=\}3,\\tau\{=\}0\.5\)32161666\.63DDIM \(w=3,τ=0\.8w\{=\}3,\\tau\{=\}0\.8\)328862\.38 We present the results for commit\-stride sampling forSigma\-3B andSigma\-8B in[Tabs\.14](https://arxiv.org/html/2610.02665#A7.T14)and[15](https://arxiv.org/html/2610.02665#A7.T15), respectively\. The tables sweep all three stridesS∈\{8,16,32\}S\\in\\\{8,16,32\\\}within each nominal\-NFE bucket\{256,128,64,32\}\\\{256,128,64,32\\\}\. At the top NFE \(nominal256256\), the DDIM6464steps / stride88row is the best on both models: it beats the tuned DDPM\-256 baseline of[Tabs\.1](https://arxiv.org/html/2610.02665#S4.T1)and[2](https://arxiv.org/html/2610.02665#S4.T2)by\+2\.66\+2\.66average points on 3B and\+0\.76\+0\.76on 8B, with the largest per\-task gains on the coding benchmarks such as MBPP\. The same ordering holds down to around nominal6464: committing fewer tokens per window \(i\.e\., smaller strideSS\) improves the average monotonically\. At nominal128128NFE, DDIM3232steps / stride88reaches within a point of the top budget on both sizes, and still above the full\-stride DDPM\-256 baselines\. The benefit of a smaller stride disappears once it forces the per\-window step count below roughly1616: at nominal3232, stride88leaves only88steps per window, and its coarse time grid collapses quality due to larger discretization error, falling below the stride\-1616and stride\-3232rows at the same budget\. ### G\.6Ablation Study: Time Discretization \[Back to[Sec\.5\.1](https://arxiv.org/html/2610.02665#S5.SS1)in the main paper\.\] Figure 12:Left: thelearned noise scheduleγ\(t\)\\gamma\(t\)forSigma\-3B\. Right: Differenttime discretization schemesfor 16\-step generation\. Note that an extra decoding step is needed att=0t=0\.Figure 13:DDIM substantially outperforms DDPM at low NFEs,while matching DDPM at high NFEs\.Figure 14:Best CFG\-ST configuration vs\. diffusion sampling stepsfor the best average score of GSM8K & HumanEval under uniform\-γ\\gammadiscretization\.We mainly consider the following discretization schemes\. LetNNbe the total number of steps and assume theii\-th step \(i=1,…,Ni=1,\.\.\.,N\) goes fromti−1t\_\{i\-1\}toti<ti−1t\_\{i\}<t\_\{i\-1\}\. - •Learned: arithmetic progression in time:ti=1−iNt\_\{i\}=1\-\\frac\{i\}\{N\}\. - •Uniform\-γ\\gamma: arithmetic progression inγ\\gamma, the minus log\-SNR:γ\(ti\)=γmax\+iN\(γmin−γmax\)\\gamma\(t\_\{i\}\)=\\gamma\_\{\\max\}\+\\frac\{i\}\{N\}\(\\gamma\_\{\\min\}\-\\gamma\_\{\\max\}\)\. - •EDMρ\\rho\([Karras et al\., 2022](https://arxiv.org/html/2610.02665#bib.bib33)\): first convert the VP schedule to VE by definingσ~t:=σtαt\\tilde\{\\sigma\}\_\{t\}:=\\frac\{\\sigma\_\{t\}\}\{\\alpha\_\{t\}\}as the essential standard deviation at timett\. The grid points are chosen so thatσ~ti1/ρ\\tilde\{\\sigma\}\_\{t\_\{i\}\}^\{1/\\rho\}is an arithmetic progression, i\.e\.,σ~ti=\(σ~max1/ρ\+iN\(σ~min1/ρ−σ~max1/ρ\)\)ρ\\tilde\{\\sigma\}\_\{t\_\{i\}\}=\\left\(\\tilde\{\\sigma\}\_\{\\max\}^\{1/\\rho\}\+\\frac\{i\}\{N\}\(\\tilde\{\\sigma\}\_\{\\min\}^\{1/\\rho\}\-\\tilde\{\\sigma\}\_\{\\max\}^\{1/\\rho\}\)\\right\)^\{\\rho\}\. Note that asρ→∞\\rho\\to\\infty, this converges to the uniform\-γ\\gammascheme \([Prop\.2](https://arxiv.org/html/2610.02665#Thmproposition2)\)\. We visualize the three schemes in[Fig\.12](https://arxiv.org/html/2610.02665#A7.F12)\. Empirically, we find that for high NFE sampling, both the uniform\-γ\\gammaand EDMρ=7\\rho=7schemes perform similarly, while uniform\-γ\\gammagenerally performs the best for low NFE sampling\. See[Fig\.13](https://arxiv.org/html/2610.02665#A7.F13)for the few\-step reasoning accuracy of DDPM and DDIM over GSM8K and HumanEval\. ### G\.7Higher\-order ODE Solvers \[Back to[Sec\.5\.1](https://arxiv.org/html/2610.02665#S5.SS1)in the main paper\.\] In this subsection, we test if existing higher\-order probability\-flow ODE \(PFODE\) solvers improve low\-NFE sampling\. All methods use the sameSigma\-3B checkpoint, block length3232, uniform\-γ\\gammatime discretization \([Sec\.G\.6](https://arxiv.org/html/2610.02665#A7.SS6)\), self\-conditioning on, and DDIM\-selected CFG and ST configurations as in[Fig\.14](https://arxiv.org/html/2610.02665#A7.F14): at88steps, we usew=3,τ=0\.8w=3,\\tau=0\.8; at1616steps, we usew=3,τ=0\.5w=3,\\tau=0\.5\. As described in[Sec\.G\.1](https://arxiv.org/html/2610.02665#A7.SS1), we generate1010responses per problem and report the mean±\\pmtwo standard errors\. We match compute across methods at88or1616sampling\-loop denoiser calls \(the “Steps” column of[Tab\.16](https://arxiv.org/html/2610.02665#A7.T16)\); adding the finalargmax\\argmaxdecoding attN=0t\_\{N\}=0\([Alg\.2](https://arxiv.org/html/2610.02665#alg2)\) gives a total NFE of99or1717\. Midpoint spends two evaluations per outer transition so performsS/2S/2transitions atSSsteps, while the other four methods performSStransitions\. Table 16:Higher\-order solver comparison onSigma\-3B\. “Steps” is the number of denoiser calls in the sampling loop, and NFE==Steps\+1\+1is the total denoiser count including the finalargmax\\argmaxdecoding; both are matched across methods within each row\. Each entry is mean±\\pmtwo standard errors over 10 responses per question, forming an approximate95%95\\%confidence interval\.Boldmarks the best method in each row\.TaskStepsNFEDDIMDPM\-Solver\+\+\(2M\)ρ\\rhoAB1\-DEISρ\\rhoAB2\-DEISρ\\rhoRK\-DEISGSM8K89GSM8K1617HumanEval89HumanEval1617MBPP89MBPP1617 ##### Update forms\. We write the discretization time points as1=t0\>t1\>⋯\>tN=01=t\_\{0\}\>t\_\{1\}\>\\cdots\>t\_\{N\}=0and describe transitioni→i\+1i\\to i\+1\. For conciseness, letαi=αti\\alpha\_\{i\}=\\alpha\_\{t\_\{i\}\},σi=σti\\sigma\_\{i\}=\\sigma\_\{t\_\{i\}\},λi=logαiσi=−γ\(ti\)2\\lambda\_\{i\}=\\log\\frac\{\\alpha\_\{i\}\}\{\\sigma\_\{i\}\}=\-\\frac\{\\gamma\(t\_\{i\}\)\}\{2\}, and defineρi=σiαi\\rho\_\{i\}=\\frac\{\\sigma\_\{i\}\}\{\\alpha\_\{i\}\}\. The model prediction at the current point is𝐞^i=𝐞θ\(𝐳i,ti,𝐜\)\\hat\{\\mathbf\{e\}\}\_\{i\}=\{\\mathbf\{e\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{i\},t\_\{i\};\{\\mathbf\{c\}\}\)with implied noise predictionϵ^i=1σi\(𝐳i−αi𝐞^i\)\\hat\{\\bm\{\\epsilon\}\}\_\{i\}=\\frac\{1\}\{\\sigma\_\{i\}\}\(\{\\mathbf\{z\}\}\_\{i\}\-\\alpha\_\{i\}\\hat\{\\mathbf\{e\}\}\_\{i\}\)\. Unless stated otherwise, we assume both quantities already carry the CFG and ST transformations of[CFG](https://arxiv.org/html/2610.02665#S4.Ex6)and[ST](https://arxiv.org/html/2610.02665#S4.Ex7)\. The DDIM update in[DDIM](https://arxiv.org/html/2610.02665#S4.Ex5)is 𝐳i\+1DDIM=αi\+1𝐞^i\+σi\+1ϵ^i\.\\displaystyle\{\\mathbf\{z\}\}\_\{i\+1\}^\{\\text\{DDIM\}\}=\\alpha\_\{i\+1\}\\hat\{\\mathbf\{e\}\}\_\{i\}\+\\sigma\_\{i\+1\}\\hat\{\\bm\{\\epsilon\}\}\_\{i\}\.\(20\)This is the zeroth\-order exponential integral discretization of the PFODE, in either prediction form as described below: - •In theϵ\{\\bm\{\\epsilon\}\}\-prediction form, setting𝐲:=𝐳α\{\\mathbf\{y\}\}:=\\frac\{\{\\mathbf\{z\}\}\}\{\\alpha\}givesd𝐲dρ=ϵ^θ\\frac\{\{\\mathrm\{d\}\}\{\\mathbf\{y\}\}\}\{\{\\mathrm\{d\}\}\\rho\}=\\hat\{\\bm\{\\epsilon\}\}\_\{\\theta\}, hence 𝐳i\+1=αi\+1αi𝐳i\+αi\+1∫ρiρi\+1ϵ^θ\(ρ\)𝑑ρ\.\\displaystyle\{\\mathbf\{z\}\}\_\{i\+1\}=\\frac\{\\alpha\_\{i\+1\}\}\{\\alpha\_\{i\}\}\{\\mathbf\{z\}\}\_\{i\}\+\\alpha\_\{i\+1\}\\int\_\{\\rho\_\{i\}\}^\{\\rho\_\{i\+1\}\}\\hat\{\\bm\{\\epsilon\}\}\_\{\\theta\}\(\\rho\)\\,\{\\mathrm\{d\}\}\\rho\.\(21\) - •In the𝐞\{\\mathbf\{e\}\}\-prediction form, setting𝐯:=𝐳σ\{\\mathbf\{v\}\}:=\\frac\{\{\\mathbf\{z\}\}\}\{\\sigma\}andu=ασu=\\frac\{\\alpha\}\{\\sigma\}givesd𝐯du=𝐞^θ\\frac\{\{\\mathrm\{d\}\}\{\\mathbf\{v\}\}\}\{\{\\mathrm\{d\}\}u\}=\\hat\{\\mathbf\{e\}\}\_\{\\theta\}, hence 𝐳i\+1=σi\+1σi𝐳i\+σi\+1∫uiui\+1𝐞^θ\(u\)𝑑u\.\\displaystyle\{\\mathbf\{z\}\}\_\{i\+1\}=\\frac\{\\sigma\_\{i\+1\}\}\{\\sigma\_\{i\}\}\{\\mathbf\{z\}\}\_\{i\}\+\\sigma\_\{i\+1\}\\int\_\{u\_\{i\}\}^\{u\_\{i\+1\}\}\\hat\{\\mathbf\{e\}\}\_\{\\theta\}\(u\)\\,\{\\mathrm\{d\}\}u\.\(22\) Freezing either learned field on the interval recovers[20](https://arxiv.org/html/2610.02665#A7.E20)\. DPM\-Solver\+\+\(2M\)\([Lu et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib41)\)uses the𝐞\{\\mathbf\{e\}\}\-prediction form and linearly extrapolates𝐞^θ\\hat\{\\mathbf\{e\}\}\_\{\\theta\}from the two most recent grid points inλ\\lambda\. Withhi=λi\+1−λi\>0h\_\{i\}=\\lambda\_\{i\+1\}\-\\lambda\_\{i\}\>0andri=hi−1hir\_\{i\}=\\frac\{h\_\{i\-1\}\}\{h\_\{i\}\}, 𝐝i\\displaystyle\\mathbf\{d\}\_\{i\}=\{𝐞^i,i=0,\(1\+12ri\)𝐞^i−12ri𝐞^i−1,i\>0,\\displaystyle=\\begin\{cases\}\\hat\{\\mathbf\{e\}\}\_\{i\},&i=0,\\\\ \\Big\(1\+\\frac\{1\}\{2r\_\{i\}\}\\Big\)\\hat\{\\mathbf\{e\}\}\_\{i\}\-\\frac\{1\}\{2r\_\{i\}\}\\hat\{\\mathbf\{e\}\}\_\{i\-1\},&i\>0,\\end\{cases\}\(23\)𝐳i\+1DPM\\displaystyle\{\\mathbf\{z\}\}\_\{i\+1\}^\{\\text\{DPM\}\}=σi\+1σi𝐳i\+αi\+1\(1−e−hi\)𝐝i\.\\displaystyle=\\frac\{\\sigma\_\{i\+1\}\}\{\\sigma\_\{i\}\}\{\\mathbf\{z\}\}\_\{i\}\+\\alpha\_\{i\+1\}\(1\-\{\\mathrm\{e\}\}^\{\-h\_\{i\}\}\)\\mathbf\{d\}\_\{i\}\.\(24\)Equivalently, this is𝐳i\+1DDIM\+αi\+1\(1−e−hi\)12ri\(𝐞^i−𝐞^i−1\)\{\\mathbf\{z\}\}\_\{i\+1\}^\{\\text\{DDIM\}\}\+\\alpha\_\{i\+1\}\(1\-\{\\mathrm\{e\}\}^\{\-h\_\{i\}\}\)\\frac\{1\}\{2r\_\{i\}\}\(\\hat\{\\mathbf\{e\}\}\_\{i\}\-\\hat\{\\mathbf\{e\}\}\_\{i\-1\}\)fori\>0i\>0\. The second\-orderρ\\rhoAB1\-DEIS solver\([Zhang and Chen, 2023](https://arxiv.org/html/2610.02665#bib.bib5)\)333DEIS indexes its Adams–Bashforth \(AB\) methods by the degree of the extrapolating polynomial, thus therr\-th order polynomial provides convergence orderr\+1r\+1\.instead uses the natural quadrature coordinate in[21](https://arxiv.org/html/2610.02665#A7.E21)\. Letki=ρi\+1−ρik\_\{i\}=\\rho\_\{i\+1\}\-\\rho\_\{i\}andpi=ρi−ρi−1p\_\{i\}=\\rho\_\{i\}\-\\rho\_\{i\-1\}\. The first transition uses DDIM\. After one history value is available, 𝐳i\+1AB2=𝐳i\+1DDIM\+αi\+1ki22pi\(ϵ^i−ϵ^i−1\)\.\\displaystyle\{\\mathbf\{z\}\}\_\{i\+1\}^\{\\text\{AB2\}\}=\{\\mathbf\{z\}\}\_\{i\+1\}^\{\\text\{DDIM\}\}\+\\frac\{\\alpha\_\{i\+1\}k\_\{i\}^\{2\}\}\{2p\_\{i\}\}\\left\(\\hat\{\\bm\{\\epsilon\}\}\_\{i\}\-\\hat\{\\bm\{\\epsilon\}\}\_\{i\-1\}\\right\)\.\(25\) The third\-orderρ\\rhoAB2\-DEIS solver\([Zhang and Chen, 2023](https://arxiv.org/html/2610.02665#bib.bib5)\)is the corresponding three\-point method ford𝐲dρ=ϵ^θ\\frac\{\{\\mathrm\{d\}\}\{\\mathbf\{y\}\}\}\{\{\\mathrm\{d\}\}\\rho\}=\\hat\{\\bm\{\\epsilon\}\}\_\{\\theta\}\. It uses DDIM on the first transition, theρ\\rhoAB1\-DEIS rule in[25](https://arxiv.org/html/2610.02665#A7.E25)on the second transition, and then integrates the quadratic Lagrange interpolant through the newest three noise predictions\. Withqi=ρi−1−ρi−2q\_\{i\}=\\rho\_\{i\-1\}\-\\rho\_\{i\-2\}, b0,i\\displaystyle b\_\{0,i\}=1\+\(2pi\+qi\)ki2pi\(pi\+qi\)\+ki23pi\(pi\+qi\),\\displaystyle=1\+\\frac\{\(2p\_\{i\}\+q\_\{i\}\)k\_\{i\}\}\{2p\_\{i\}\(p\_\{i\}\+q\_\{i\}\)\}\+\\frac\{k\_\{i\}^\{2\}\}\{3p\_\{i\}\(p\_\{i\}\+q\_\{i\}\)\},b1,i\\displaystyle b\_\{1,i\}=−\(pi\+qi\)ki2piqi−ki23piqi,\\displaystyle=\-\\frac\{\(p\_\{i\}\+q\_\{i\}\)k\_\{i\}\}\{2p\_\{i\}q\_\{i\}\}\-\\frac\{k\_\{i\}^\{2\}\}\{3p\_\{i\}q\_\{i\}\},b2,i\\displaystyle b\_\{2,i\}=piki2qi\(pi\+qi\)\+ki23qi\(pi\+qi\),\\displaystyle=\\frac\{p\_\{i\}k\_\{i\}\}\{2q\_\{i\}\(p\_\{i\}\+q\_\{i\}\)\}\+\\frac\{k\_\{i\}^\{2\}\}\{3q\_\{i\}\(p\_\{i\}\+q\_\{i\}\)\},and 𝐳i\+1AB3=𝐳i\+1DDIM\+αi\+1ki\[\(b0,i−1\)ϵ^i\+b1,iϵ^i−1\+b2,iϵ^i−2\]\.\\displaystyle\{\\mathbf\{z\}\}\_\{i\+1\}^\{\\text\{AB3\}\}=\{\\mathbf\{z\}\}\_\{i\+1\}^\{\\text\{DDIM\}\}\+\\alpha\_\{i\+1\}k\_\{i\}\\left\[\(b\_\{0,i\}\-1\)\\hat\{\\bm\{\\epsilon\}\}\_\{i\}\+b\_\{1,i\}\\hat\{\\bm\{\\epsilon\}\}\_\{i\-1\}\+b\_\{2,i\}\\hat\{\\bm\{\\epsilon\}\}\_\{i\-2\}\\right\]\.By construction,b0,i\+b1,i\+b2,i=1b\_\{0,i\}\+b\_\{1,i\}\+b\_\{2,i\}=1, so the update reduces to DDIM whenϵ^i−2=ϵ^i−1=ϵ^i\\hat\{\\bm\{\\epsilon\}\}\_\{i\-2\}=\\hat\{\\bm\{\\epsilon\}\}\_\{i\-1\}=\\hat\{\\bm\{\\epsilon\}\}\_\{i\}\. On aρ\\rho\-uniform grid, the weights collapse to the classical\(2312,−1612,512\)\\left\(\\frac\{23\}\{12\},\-\\frac\{16\}\{12\},\\frac\{5\}\{12\}\\right\)\. Finally,ρ\\rhoRK\-DEIS\([Zhang and Chen, 2023](https://arxiv.org/html/2610.02665#bib.bib5)\)is explicit midpoint Runge\-Kutta method in the same integrating\-factor state𝐲=𝐳α\{\\mathbf\{y\}\}=\\frac\{\{\\mathbf\{z\}\}\}\{\\alpha\}\. It uses the arithmetic midpointρm,i=12\(ρi\+ρi\+1\)\\rho\_\{m,i\}=\\frac\{1\}\{2\}\(\\rho\_\{i\}\+\\rho\_\{i\+1\}\), converts it back toγm,i=2logρm,i\\gamma\_\{m,i\}=2\\log\\rho\_\{m,i\}and therefore to\(αm,i,σm,i\)\(\\alpha\_\{m,i\},\\sigma\_\{m,i\}\), predicts an intermediate state, 𝐳m,i=αm,i\(𝐳iαi\+\(ρm,i−ρi\)ϵ^i\),\\displaystyle\{\\mathbf\{z\}\}\_\{m,i\}=\\alpha\_\{m,i\}\\Big\(\\frac\{\{\\mathbf\{z\}\}\_\{i\}\}\{\\alpha\_\{i\}\}\+\(\\rho\_\{m,i\}\-\\rho\_\{i\}\)\\hat\{\\bm\{\\epsilon\}\}\_\{i\}\\Big\),\(26\)queries the denoiser at\(𝐳m,i,tm,i\)\(\{\\mathbf\{z\}\}\_\{m,i\},t\_\{m,i\}\)to obtainϵ^m,i\\hat\{\\bm\{\\epsilon\}\}\_\{m,i\}, and completes the full interval as 𝐳i\+1midpoint=αi\+1\(𝐳iαi\+\(ρi\+1−ρi\)ϵ^m,i\)=𝐳i\+1DDIM\+αi\+1\(ρi\+1−ρi\)\(ϵ^m,i−ϵ^i\)\.\\displaystyle\{\\mathbf\{z\}\}\_\{i\+1\}^\{\\text\{midpoint\}\}=\\alpha\_\{i\+1\}\\Big\(\\frac\{\{\\mathbf\{z\}\}\_\{i\}\}\{\\alpha\_\{i\}\}\+\(\\rho\_\{i\+1\}\-\\rho\_\{i\}\)\\hat\{\\bm\{\\epsilon\}\}\_\{m,i\}\\Big\)=\{\\mathbf\{z\}\}\_\{i\+1\}^\{\\text\{DDIM\}\}\+\\alpha\_\{i\+1\}\(\\rho\_\{i\+1\}\-\\rho\_\{i\}\)\(\\hat\{\\bm\{\\epsilon\}\}\_\{m,i\}\-\\hat\{\\bm\{\\epsilon\}\}\_\{i\}\)\.\(27\) ##### Results\. [Tab\.16](https://arxiv.org/html/2610.02665#A7.T16)shows that none of these higher\-order updates improves accuracy over DDIM at the tested low\-NFE sampling settings\. The closest alternative isρ\\rhoAB1\-DEIS at88steps, but it still trails DDIM on GSM8K and HumanEval and is much worse on MBPP\. Increasing the budget from88to1616steps improves DDIM on every task, while the higher\-order solvers do not improve reliably\.ρ\\rhoAB2\-DEIS andρ\\rhoRK\-DEIS become especially unstable at1616steps\. ##### Trajectory diagnosis\. We perform a trajectory diagnosis by running different solvers from the same initial Gaussian noise𝐳1\{\\mathbf\{z\}\}\_\{1\}given the same prompt and few\-shot examples, saving the complete block\-00trajectory\. We use the first3232questions of either GSM8K or HumanEval dataset, draw eight generations per question, and use three CFG–ST configurations\(w,τ\)∈\{\(3,0\.4\),\(3,0\.6\),\(2,0\.7\)\}\(w,\\tau\)\\in\\\{\(3,0\.4\),\(3,0\.6\),\(2,0\.7\)\\\}, each with self\-conditioning on and off\. This gives22tasks×\\times33CFG–ST pairs×\\times22self\-conditioning configurations=12=12settings\. For each setting, the initial eight\-lane noise tensor is identical across solvers and step budgets\. We evaluate runs with8,16,32,64,128,8,16,32,64,128,and256256sampling steps together with a paired DDIM\-512512high\-resolution reference as ground truth\. Recall that all embedding vectors in𝐄\{\\mathbf\{E\}\}are unit\-norm, and thus ideally our terminal state𝐳0\{\\mathbf\{z\}\}\_\{0\}should be close to the unit sphere\. Panels \(a\) and \(b\) of[Fig\.15](https://arxiv.org/html/2610.02665#A7.F15)measure the geometry of the terminal latent states\. Panel \(a\) averages\|‖𝐳0‖−1\|\|\\\|\{\\mathbf\{z\}\}\_\{0\}\\\|\-1\|within each setting and then equally across the1212settings\. For panel \(b\), we compute the nearest\-embedding cosine similaritymaxkcos\(𝐳0,𝐄k\)\\max\_\{k\}\\cos\(\{\\mathbf\{z\}\}\_\{0\},\{\\mathbf\{E\}\}\_\{k\}\), take its fifth percentile within each setting, and then average the1212setting\-level percentiles\. This tail statistic reveals endpoints stranded between vocabulary vertices\. Error bars are standard errors across settings\. At88steps, the mean off\-sphere distance is0\.0140\.014for DDIM, but1\.291\.29forρ\\rhoAB1\-DEIS and26\.226\.2for DPM\-Solver\+\+\(2M\); their fifth\-percentile nearest\-embedding cosine is0\.9940\.994,0\.8510\.851, and0\.8450\.845, respectively\. Thus, coarse higher\-order extrapolation can move𝐳0\{\\mathbf\{z\}\}\_\{0\}far outside the clean embedding manifold\. Because the reconstruction term is applied only att=0t=0[4](https://arxiv.org/html/2610.02665#S3.Ex2), the model is not trained to decode such off\-manifold terminal states, this causes a large decoding error\. By3232steps, all three methods return close to the sphere and to a vocabulary vertex\. Figure 15:Endpoint geometry under low\-step ODE solvers\.\(a\) Terminal distance from the unit\-sphere vocabulary manifold\. \(b\) Fifth\-percentile cosine similarity to the nearest vocabulary embedding\. At88steps, higher\-order extrapolation leaves endpoints far off\-manifold and between vocabulary vertices, whereas these differences largely disappear by3232steps\. ##### Implications and future work\. This negative result suggests that low\-NFE inference forSigmais not limited only by the formal order of the ODE integrator\. Under strong CFG, ST, and SC, the effective denoising field can be less smooth than the trajectories for which standard image\-diffusion solvers were designed\. In this regime, formal integration order alone is not a reliable predictor of downstream quality, and DDIM remains the most robust choice among the solvers tested\. We leave exploration of more stable and efficient higher\-order ODE solvers for future work\. ### G\.8Pass@kkEvaluation \[Back to[Tab\.2](https://arxiv.org/html/2610.02665#S4.T2)in the main paper\.\] We study inference\-time scaling in two distinct evaluation regimes\. First, we compare the pretrained 3B models on MBPP, Minerva Geometry, and GSM8K under few\-shot prompting\. Second, we compare the instruct 3B models on AIME 2025 under the non\-thinking protocol of[Tab\.3](https://arxiv.org/html/2610.02665#S6.T3)\. ##### Metric\. For a problem withcccorrect answers amongnngenerations, we use the standard unbiased estimator pass@k=1−\(n−ck\)\(nk\)\\text\{pass@\}k=1\-\\frac\{\\binom\{n\-c\}\{k\}\}\{\\binom\{n\}\{k\}\}and average it over problems to obtain the pass@kkaccuracy on a evaluation dataset\. Cross\-family uncertainty is computed from paired per\-problem differences and reported as a95%95\\%confidence interval\. #### G\.8\.1Base Model: MBPP Figure 16:Pass@kkon MBPP\.*Left:*at eachkk, each family uses the best configuration in its tested sampler grid;Sigmaovertakes NLD atk=32k=32and reaches0\.9180\.918versus0\.8760\.876atk=256k=256\.*Right:*one fixed high\-coverage configuration per family, selected by the one reaching the highest pass@256 accuracy:Sigma\(w=2,τ=1\.0\)\(w=2,\\tau=1\.0\)and NLD\(ϑ=0\.9,T=1\.0\)\(\\vartheta=0\.9,T=1\.0\)\. The\+4\.2\+4\.2point improvement atk=256k=256has a paired95%95\\%CI of\[\+1\.7,\+6\.7\]\[\+1\.7,\+6\.7\], with the interval excluding zero fromk=64k=64onward\.##### Setup\. We compare the pretrained 3BSigmaand NLD\([Fu et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib51)\)models on all 500 MBPP problems, drawing 256 generations per problem for every configuration\. Both models use block size 32\.Sigmauses DDPM with 256 steps and the EDMρ=7\\rho=7schedule, sweeping CFGw∈\{1,2,3\}w\\in\\\{1,2,3\\\}and STτ∈\{0\.1,0\.4,0\.7,1\.0\}\\tau\\in\\\{0\.1,0\.4,0\.7,1\.0\\\}; NLD uses 32 transfer steps per block, sweeping confidence thresholdϑ∈\{0\.8,0\.9\}\\vartheta\\in\\\{0\.8,0\.9\\\}and temperatureT∈\{0\.3,0\.6,1\.0\}T\\in\\\{0\.3,0\.6,1\.0\\\}\. Candidate draws use different random seeds while preserving identical prompts and few\-shot examples across configurations, as in[Sec\.G\.1](https://arxiv.org/html/2610.02665#A7.SS1)\. ##### Results\. The left panel of[Fig\.16](https://arxiv.org/html/2610.02665#A7.F16)shows the best performance available in each tested sampler grid at every generation budget\. Askkgrows, theSigmasetting carrying this curve moves fromτ=0\.1\\tau=0\.1to0\.40\.4,0\.70\.7, and finally1\.01\.0, shifting from single\-sample quality toward broader coverage\. Because the tested grids differ in size, this is a best\-of\-grid comparison; the right panel provides the complementary fixed\-configuration result\. One fixedSigmaconfiguration reaches the same pass@256 value of0\.9180\.918, compared with0\.8760\.876for NLD\. ##### String\-diversity diagnostic\. We further investigate exact\-string repetition as a partial proxy for output diversity, after stripping leading and trailing whitespaces\. At the selected fixed MBPP configurations,Sigmarepeats 10\.4% of its 256 output strings, compared with 53\.1% for NLD\. Although NLD produces more correct samples per problem on average \(139\.5 versus 91\.5\),Sigmaproduces more distinct correct strings \(71\.4 versus 46\.9\), consistent with its later saturation and higher problem coverage\. #### G\.8\.2Base Model: Minerva Geometry Figure 17:Pass@kkon Minerva Geometry\.*Left:*the best tested configuration in each family at everykk;Sigmaleads fromk=8k=8onward and reaches0\.9540\.954versus0\.9420\.942atk=512k=512\.*Right:*fixed high\-coverage configurationsSigma\(w=2,τ=1\.0\)\(w=2,\\tau=1\.0\)and NLD\(ϑ=0\.9,T=1\.0\)\(\\vartheta=0\.9,T=1\.0\)\. Their\+1\.3\+1\.3point difference atk=512k=512is directional but unresolved, with a paired 95% CI of\[−0\.3,\+2\.8\]\[\-0\.3,\+2\.8\]\.##### Setup\. We next compare the same 3B base models on the 479 Minerva Geometry problems\. Each tested configuration has 256 generations per problem, and the highlighted configurations are extended to 512\.Sigmanow uses DDIM with 32 steps and the uniform\-γ\\gammaschedule, sweepingw∈\{1,2,3\}w\\in\\\{1,2,3\\\}andτ∈\{0\.4,0\.7,1\.0\}\\tau\\in\\\{0\.4,0\.7,1\.0\\\}plus\(w,τ\)∈\{\(4,0\.7\),\(5,0\.7\)\}\(w,\\tau\)\\in\\\{\(4,0\.7\),\(5,0\.7\)\\\}; NLD uses 32 transfer steps withϑ=0\.9\\vartheta=0\.9andT∈\{0\.3,0\.6,1\.0\}T\\in\\\{0\.3,0\.6,1\.0\\\}, as we have observed in the previous experiment on MBPP thatϑ=0\.9\\vartheta=0\.9almost always outperformsϑ=0\.8\\vartheta=0\.8under the same temperature\. ##### Results\. In the left panel of[Fig\.17](https://arxiv.org/html/2610.02665#A7.F17),Sigmaleads by0\.10\.1point atk=8k=8, and the gap grows monotonically to1\.31\.3points atk=512k=512\. The setting carrying its best\-tested curve moves fromτ=0\.4\\tau=0\.4to0\.70\.7and then1\.01\.0\. The fixed comparison in the right panel has the same direction at highkk, but unlike MBPP, its confidence interval still spans zero\. ##### Matched\-pass@1 controls on Minerva Geometry\. We next hold single\-sample quality fixed on Geometry\. Increasing CFG atτ=0\.7\\tau=0\.7movesSigmainto the pass@1 band of NLD’s best sampler\(ϑ=0\.9,T=1\)\(\\vartheta=0\.9,T=1\), producing three matched operating points, including one whose pass@1 differs by only0\.00050\.0005\. As shown in[Tab\.17](https://arxiv.org/html/2610.02665#A7.T17), all threeSigmasettings trace essentially the same pass@kkcurve as best\-tuned NLD: beyond pass@1, every difference is within0\.0070\.007\. Table 17:Sigmaties the best\-tuned NLD sampler\(ϑ=0\.9,T=1\)\(\\vartheta=0\.9,T=1\)at matched pass@1 on Minerva Geometry\.Entries are paired differences in pass@kk\(Sigma−\-NLD\) with 512 independent generations per problem for every setting\.Sigmasampling configurationk=1k=1k=8k=8k=32k=32k=64k=64k=128k=128k=256k=256k=512k=512w=3,τ=0\.7w=3,\\tau=0\.7−0\.0080\-0\.0080−0\.002\-0\.002\+0\.001\+0\.0010\.0000\.000−0\.003\-0\.003−0\.002\-0\.002\+0\.006\+0\.006w=4,τ=0\.7w=4,\\tau=0\.7−0\.0025\-0\.0025−0\.001\-0\.001\+0\.002\+0\.002\+0\.001\+0\.001−0\.001\-0\.001−0\.004\-0\.004−0\.004\-0\.004w=5,τ=0\.7w=5,\\tau=0\.7\+0\.0005\+0\.0005\+0\.001\+0\.001\+0\.005\+0\.005\+0\.004\+0\.004−0\.001\-0\.001−0\.005\-0\.005−0\.004\-0\.004 ##### Comparison with lower\-temperature NLD samplers\. At the same pass@1, the outcome is stronger against NLD configurations with lower sampling temperature\. As[Tab\.18](https://arxiv.org/html/2610.02665#A7.T18)shows,Sigmaimproves pass@256 by1\.91\.9to6\.66\.6points across all five matched comparisons, with every paired interval excluding zero\. In short,Sigmadominates NLD when the latter is poorly tuned for coverage and ties its best\-tuned high\-pass@kksampler\. Table 18:Sigmaoutperforms lower\-temperature NLD samplers at matched pass@1 on Minerva Geometry\.The final columns report the paired pass@256 difference \(Sigma−\-NLD\) and its 95% confidence interval\.Sigmavs\. NLDΔ\\Deltapass@1Δ\\Deltapass@25695% CIw=4,τ=0\.7w=4,\\tau=0\.7vs\.ϑ=0\.9,T=0\.3\\vartheta=0\.9,T=0\.3−0\.0103\-0\.0103\+0\.066\\mathbf\{\+0\.066\}\[\+0\.044,\+0\.088\]\[\+0\.044,\+0\.088\]w=5,τ=0\.7w=5,\\tau=0\.7vs\.ϑ=0\.9,T=0\.3\\vartheta=0\.9,T=0\.3−0\.0073\-0\.0073\+0\.065\\mathbf\{\+0\.065\}\[\+0\.042,\+0\.088\]\[\+0\.042,\+0\.088\]w=3,τ=0\.7w=3,\\tau=0\.7vs\.ϑ=0\.9,T=0\.6\\vartheta=0\.9,T=0\.6−0\.0115\-0\.0115\+0\.024\\mathbf\{\+0\.024\}\[\+0\.008,\+0\.041\]\[\+0\.008,\+0\.041\]w=4,τ=0\.7w=4,\\tau=0\.7vs\.ϑ=0\.9,T=0\.6\\vartheta=0\.9,T=0\.6−0\.0064\-0\.0064\+0\.020\\mathbf\{\+0\.020\}\[\+0\.003,\+0\.037\]\[\+0\.003,\+0\.037\]w=5,τ=0\.7w=5,\\tau=0\.7vs\.ϑ=0\.9,T=0\.6\\vartheta=0\.9,T=0\.6−0\.0034\-0\.0034\+0\.019\\mathbf\{\+0\.019\}\[\+0\.002,\+0\.036\]\[\+0\.002,\+0\.036\] #### G\.8\.3Base Model: GSM8K ##### Setup and results\. We repeat the comparison on GSM8K \(1319 problems, 8\-shot\), drawing 256 generations per problem for every configuration;Sigmauses DDIM with3232steps and the uniform\-γ\\gammaschedule, and NLD uses3232transfer steps per block\. We observe that GSM8K is nearly saturated for both families, so the family separation lives entirely in the hard tail\. We report the full per\-configuration results in[Tab\.19](https://arxiv.org/html/2610.02665#A7.T19)and the best\-of\-family comparison in[Fig\.18](https://arxiv.org/html/2610.02665#A7.F18)\. Fixing one high\-coverage configuration per family,Sigma\(w=2,τ=1\.0\)\(w=2,\\tau=1\.0\)against NLD’s best sampler\(ϑ=0\.9,T=1\.0\)\(\\vartheta=0\.9,T=1\.0\), reproduces the same pattern as MBPP and Geometry: NLD leads at smallkk, the curves cross neark=16k=16, andSigmais ahead fromk=32k=32onward\. The paired per\-problem advantage is\+0\.7\+0\.7points and its95%95\\%confidence interval excludes zero at*every*k≥32k\\geq 32throughk=256k=256\(\+0\.0072\[\+0\.003,\+0\.012\]\+0\.0072\\,\[\+0\.003,\+0\.012\]atk=32k=32;\+0\.0068\[\+0\.002,\+0\.012\]\+0\.0068\\,\[\+0\.002,\+0\.012\]atk=256k=256\)\.Sigma’s selected configuration also solves13141314of13191319problems at least once versus13051305for NLD\. Table 19:Full pass@kkresults on GSM8K, sorted by pass@256\. “Solved” is the number of problems answered correctly at least once in 256 generations\.ModelConfigurationPass@1Pass@8Pass@32Pass@64Pass@256SolvedSigmaw=2,τ=1\.0w\{=\}2,\\tau\{=\}1\.00\.6510\.9420\.9800\.9880\.9961314NLDϑ=0\.9,T=1\.0\\vartheta\{=\}0\.9,T\{=\}1\.00\.8230\.9500\.9730\.9810\.9891305Sigmaw=3,τ=0\.7w\{=\}3,\\tau\{=\}0\.70\.8060\.9380\.9650\.9740\.9871302NLDϑ=0\.9,T=0\.3\\vartheta\{=\}0\.9,T\{=\}0\.30\.8360\.9130\.9370\.9460\.9641271Sigmaw=2,τ=0\.4w\{=\}2,\\tau\{=\}0\.40\.7990\.8780\.9060\.9160\.9311228Figure 18:Sigmaovertakes NLD in the GSM8K hard tail\.Best\-of\-family fixed configuration:Sigma\(w=2,τ=1\.0\)\(w\{=\}2,\\tau\{=\}1\.0\)versus NLD’s best sampler\(ϑ=0\.9,T=1\.0\)\(\\vartheta\{=\}0\.9,T\{=\}1\.0\)\.*Left:*the pass@kkcurves; GSM8K saturates high, so the curves cross neark=16k=16and the gap is small\.*Right:*the paired per\-problem difference \(Sigma−\-NLD\) with its95%95\\%CI, where the effect is legible: negative atk≤8k\\leq 8, then positive with the interval excluding zero at everyk≥32k\\geq 32\. ##### Base\-model summary\. Across MBPP, Minerva Geometry, and GSM8K, CFG and ST expose a broad exploration–exploitation frontier\. The high\-kkadvantage survives with one fixed configuration on MBPP and GSM8K, while the matched\-pass@1 Geometry controls show parity with NLD’s best\-tuned sampler and gains only against its lower\-temperature settings\. #### G\.8\.4Instruct Models: AIME 2025 Figure 19:Sigmaleads NLD at everykkon AIME 2025, with a margin that grows withkk\.*Left:*all eight tested configurations atn=256n=256; the bold curve of each family is its best setting by pass@256\. FourSigmasettings exceed every NLD setting atk=256k=256, and the fifth ties NLD’s best \(ϑ=0\.9,T=0\.2\\vartheta=0\.9,T=0\.2\)\.*Right:*paired per\-problem differences against each NLD temperature, with the95%95\\%interval of the highlighted pair shaded\. The lead is consistent in direction at everykkagainst all three NLD samplers\.The study above uses the pretrained checkpoints under few\-shot prompting\. Because pass@kkis often read as a proxy for the headroom available to reinforcement learning, we repeat the comparison on the*instruct*models, under the same non\-thinking protocol as the benchmark table in[Tab\.3](https://arxiv.org/html/2610.02665#S6.T3)\(harness described in[Sec\.H\.2](https://arxiv.org/html/2610.02665#A8.SS2)\), on AIME 2025\. ##### Protocol and sampling settings\. We use the same unbiased estimator as above, averaged over the 30 AIME 2025 problems, and draw 256 generations per problem for every configuration of both families\. All cross\-family comparisons are paired per problem and reported with a95%95\\%confidence interval\. Both models use block size 32\.Sigmauses DDIM with 32 steps and the uniform\-γ\\gammaschedule, sweeping the five settings\(w,τ\)∈\{\(3,0\.3\),\(4,0\.3\),\(4,0\.7\),\(5,0\.7\),\(3,1\.0\)\}\(w,\\tau\)\\in\\\{\(3,0\.3\),\\,\(4,0\.3\),\\,\(4,0\.7\),\\,\(5,0\.7\),\\,\(3,1\.0\)\\\}; NLD uses confidence\-thresholded masked decoding withϑ=0\.9\\vartheta=0\.9andT∈\{0\.2,0\.6,1\.0\}T\\in\\\{0\.2,0\.6,1\.0\\\}\. The response budgets are 4096 tokens forSigmaand 16384 for NLD, but the shorterSigmacap is not binding since the reasonable generations \(i\.e\., the samples that are not degenerate, such as repeating the same pattern without emitting EOS token\) are all within this length range for both models\. ##### A consistent lead that widens with the budget\. The left panel of[Fig\.19](https://arxiv.org/html/2610.02665#A7.F19)shows that the bestSigmasetting\(w=3,τ=0\.3\)\(w=3,\\tau=0\.3\)leads NLD’s best sampler\(ϑ=0\.9,T=0\.2\)\(\\vartheta=0\.9,T=0\.2\)ateverykk, from\+1\.7\+1\.7points atk=1k=1to\+10\.0\+10\.0points atk=256k=256\. It is also robust to the choice ofSigmasetting: atk=256k=256, four of the five sweep points, namely\(w,τ\)=\(3,0\.3\)\(w,\\tau\)=\(3,0\.3\)and\(4,0\.7\)\(4,0\.7\)at0\.6330\.633,\(4,0\.3\)\(4,0\.3\)and\(3,1\.0\)\(3,1\.0\)at0\.5670\.567, exceed every NLD point, and the fifth,\(5,0\.7\)\(5,0\.7\)at0\.5330\.533, ties NLD’s best setting\(ϑ=0\.9,T=0\.2\)\(\\vartheta=0\.9,T=0\.2\)at0\.5330\.533\. Against NLD’s other two temperatures the paired gap atk=256k=256is\+13\.3\+13\.3points in both cases\. We remark that, unlike the base\-model result, the gain here does not come at the cost of single\-sample quality:Sigmaleads on pass@1 as well\. ##### Statistical resolution\. The right panel of[Fig\.19](https://arxiv.org/html/2610.02665#A7.F19)shows the limit of this evidence\. Of the 27 paired comparisons \(nine values ofkkagainst three NLD samplers\), only\(w=3,τ=0\.3\)\(w=3,\\tau=0\.3\)against\(ϑ=0\.9,T=0\.6\)\(\\vartheta=0\.9,T=0\.6\)atk=256k=256excludes zero Since AIME 2025 has only 30 problems, the width of these intervals is set by the number of problems rather than the number of generations, and drawing more samples per problem cannot narrow them\. ### G\.9Reasoning Trajectory Analysis \[Back to[Sec\.4\.3](https://arxiv.org/html/2610.02665#S4.SS3)in the main paper\.\] See[Fig\.20](https://arxiv.org/html/2610.02665#A7.F20)for the commit order analysis between continuous and discrete dLMs\. Figure 20:Sigmatend to commit early positions of a block later than MDLM and the final positions earlier\.The sampler used is 256\-step DDPM \(w=2,τ=0\.1w=2,\\tau=0\.1\) as in[Tab\.1](https://arxiv.org/html/2610.02665#S4.T1)\. Generation order averaged across multiple documents and blocks\. ForSigma, the commitment steptit\_\{i\}of positioniiis defined as the smallestttsuch that for every stept′≥tt^\{\\prime\}\\geq tup to the final readout, \(i\) the argmax of𝐱θ\\mathbf\{x\}\_\{\\theta\}equals the finally decoded token, and \(ii\) that argmax is assigned probability at least0\.90\.9\. The commitment order of positioniiis the rank oftit\_\{i\}amongt1,…,t32\{t\_\{1\},\\dots,t\_\{32\}\}within its block, in ascending order\. Positions sharing a commitment step share the lowest rank of the tie\. ### G\.10Wall\-clock Cost of the Packed CFG Forward \[Back to[Sec\.4\.2](https://arxiv.org/html/2610.02665#S4.SS2)in the main paper\.\] Table 20:Wall\-clock cost per denoiser evaluation \(NFE\) of packing the conditional and unconditional CFG branches into one forward pass, at batch size11\. Times are in milliseconds on H100\.Δ\\Deltais the time cost of packed conditional and unconditional predictions \(packed CFG\)−\-only conditional prediction \(no CFG\), reported as mean±\\pmtwo standard errors over the1616questions, an approximate95%95\\%confidence interval for the mean\. The*transformer forward*row shows that doubling the query length fromL′L^\{\\prime\}to2L′2L^\{\\prime\}leaves both the KV cache and the weight traffic unchanged, so the extra work hides entirely under the memory\-bandwidth floor\. The residual cost is when passing through the logit head and computing the categorical reparameterization𝐞θ=softmax\(ℓθ\)𝐄\{\\mathbf\{e\}\}\_\{\\theta\}=\\operatorname\{softmax\}\(\{\\bm\{\\ell\}\}\_\{\\theta\}\)\{\\mathbf\{E\}\}, which is independent of how the two branches are batched\.Block SizeStageNo CFG \(ms\)Packed CFG \(ms\)Δ\\DeltaΔ\(%\)\\Delta\\ \(\\%\)L′=32L^\{\\prime\}=32Transformer forward18\.0618\.0618\.3818\.38\+0\.322±0\.181\+0\.322\\pm 0\.181\+1\.8\+1\.8Logit head \+ cat\. reparam\.1\.021\.021\.501\.50\+0\.481±0\.008\+0\.481\\pm 0\.008\+47\.2\+47\.2Total per NFE19\.0819\.0819\.8919\.89\+0\.804±0\.186\+0\.804\\pm 0\.186\+4\.2\+4\.2L′=64L^\{\\prime\}=64Transformer forward18\.2318\.2318\.2118\.21−0\.021±0\.159\-0\.021\\pm 0\.159−0\.1\-0\.1Logit head \+ cat\. reparam\.1\.081\.081\.601\.60\+0\.519±0\.006\+0\.519\\pm 0\.006\+48\.3\+48\.3Total per NFE19\.3119\.3119\.8019\.80\+0\.498±0\.162\+0\.498\\pm 0\.162\+2\.6\+2\.6 We measure the wall\-clock cost of the packed CFG forward with attention map \([Fig\.4\(a\)](https://arxiv.org/html/2610.02665#S4.F4.sf1)\) at batch size11compared with the unguided forward\. We decode the same set of1616GSM8K questions \(88\-shot, prompt length729∼814729\\sim 814\) with the DDIM sampler at3232steps per block\. To keep the comparison exact, each run generates only the*first*block, so that every timed forward sees a KV cache which is precisely the prompt\. Each question therefore requires32\+132\+1forward passes \([Alg\.2](https://arxiv.org/html/2610.02665#alg2)\)\. Our implementation evaluates the mean embedding prediction𝐞θ=𝐱θ𝐄\{\\mathbf\{e\}\}\_\{\\theta\}=\{\\mathbf\{x\}\}\_\{\\theta\}\{\\mathbf\{E\}\}on every step \(including the final one att=0t=0which is the argmax readout of the logits that does not require mean embedding prediction\), so wall\-clock cost is not diluted by that step\. The very first denoiser query of each process involves a one\-time CUDA initialization which takes a longer time, so is excluded to avoid biasing the measurement\. All runs are performed on one H100, usingSigma\-3B with precisionBF16\. [Tab\.20](https://arxiv.org/html/2610.02665#A7.T20)confirms our claim\. AtL′=32L^\{\\prime\}=32the packed transformer forward costs\+1\.8%\+1\.8\\%over the unguided one, and atL′=64L^\{\\prime\}=64it is free to within measurement noise\. The per\-NFE overhead is dominated by computing the second set of logits, doing softmax, and multiplying by the embedding matrix rather than by attention\. ### G\.11Proofs \[Back to[Sec\.4\.2](https://arxiv.org/html/2610.02665#S4.SS2)in the main paper\.\] ###### Proposition 1\(Guidance and temperature commute\)\. Applying[ST](https://arxiv.org/html/2610.02665#S4.Ex7)to each branch and then[CFG](https://arxiv.org/html/2610.02665#S4.Ex6)yields the same mean prediction as applying[CFG](https://arxiv.org/html/2610.02665#S4.Ex6)before[ST](https://arxiv.org/html/2610.02665#S4.Ex7), 𝐞θw,τ\(𝐳t\(b\),t,𝐜\):=wτ𝐞θ\(𝐳t\(b\),t,𝐜\)\+1−wτ𝐞θ\(𝐳t\(b\),t,∅\)\+\(1−1τ\)𝐳t\(b\)αt\.\\displaystyle\{\\mathbf\{e\}\}^\{w,\\tau\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\{\\mathbf\{c\}\}\):=\\frac\{w\}\{\\tau\}\{\\mathbf\{e\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\{\\mathbf\{c\}\}\)\+\\frac\{1\-w\}\{\\tau\}\{\\mathbf\{e\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\\varnothing\)\+\\Big\(1\-\\frac\{1\}\{\\tau\}\\Big\)\\frac\{\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\}\{\\alpha\_\{t\}\}\.\(28\)The associated score is that ofpt\(𝐳t\(b\)∣𝐜\)wτpt\(𝐳t\(b\)∣∅\)1−wτp\_\{t\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\mid\{\\mathbf\{c\}\}\)^\{\\frac\{w\}\{\\tau\}\}\\,p\_\{t\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\mid\\varnothing\)^\{\\frac\{1\-w\}\{\\tau\}\}\. ###### Proof\. Write𝐞c:=𝐞θ\(𝐳t\(b\),t,𝐜\)\{\\mathbf\{e\}\}\_\{\\mathrm\{c\}\}:=\{\\mathbf\{e\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\{\\mathbf\{c\}\}\),𝐞∅:=𝐞θ\(𝐳t\(b\),t,∅\)\{\\mathbf\{e\}\}\_\{\\varnothing\}:=\{\\mathbf\{e\}\}\_\{\\theta\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},t;\\varnothing\), andμt:=𝐳t\(b\)αt\\mu\_\{t\}:=\\frac\{\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\}\{\\alpha\_\{t\}\}\. Rearranging[CFG](https://arxiv.org/html/2610.02665#S4.Ex6)and[ST](https://arxiv.org/html/2610.02665#S4.Ex7), we have the following affine maps for CFG and ST, respectively: G\(𝐞c,𝐞∅\):=μt\+\[w\(𝐞c−μt\)\+\(1−w\)\(𝐞∅−μt\)\],T\(𝐞\):=μt\+1τ\(𝐞−μt\)\.G\(\{\\mathbf\{e\}\}\_\{\\mathrm\{c\}\},\{\\mathbf\{e\}\}\_\{\\varnothing\}\):=\\mu\_\{t\}\+\\big\[w\(\{\\mathbf\{e\}\}\_\{\\mathrm\{c\}\}\-\\mu\_\{t\}\)\+\(1\-w\)\(\{\\mathbf\{e\}\}\_\{\\varnothing\}\-\\mu\_\{t\}\)\\big\],\\qquad T\(\{\\mathbf\{e\}\}\):=\\mu\_\{t\}\+\\frac\{1\}\{\\tau\}\(\{\\mathbf\{e\}\}\-\\mu\_\{t\}\)\.\(29\)Note thatG\(μt,μt\)=μtG\(\\mu\_\{t\},\\mu\_\{t\}\)=\\mu\_\{t\}andT\(μt\)=μtT\(\\mu\_\{t\}\)=\\mu\_\{t\}\. Therefore, T\(G\(𝐞c,𝐞∅\)\)\\displaystyle T\(G\(\{\\mathbf\{e\}\}\_\{\\mathrm\{c\}\},\{\\mathbf\{e\}\}\_\{\\varnothing\}\)\)=μt\+1τ\[w\(𝐞c−μt\)\+\(1−w\)\(𝐞∅−μt\)\]\\displaystyle=\\mu\_\{t\}\+\\frac\{1\}\{\\tau\}\\big\[w\(\{\\mathbf\{e\}\}\_\{\\mathrm\{c\}\}\-\\mu\_\{t\}\)\+\(1\-w\)\(\{\\mathbf\{e\}\}\_\{\\varnothing\}\-\\mu\_\{t\}\)\\big\]=w\[μt\+1τ\(𝐞c−μt\)\]\+\(1−w\)\[μt\+1τ\(𝐞∅−μt\)\]\\displaystyle=w\\big\[\\mu\_\{t\}\+\\frac\{1\}\{\\tau\}\(\{\\mathbf\{e\}\}\_\{\\mathrm\{c\}\}\-\\mu\_\{t\}\)\\big\]\+\(1\-w\)\\big\[\\mu\_\{t\}\+\\frac\{1\}\{\\tau\}\(\{\\mathbf\{e\}\}\_\{\\varnothing\}\-\\mu\_\{t\}\)\\big\]=wT\(𝐞c\)\+\(1−w\)T\(𝐞∅\)=G\(T\(𝐞c\),T\(𝐞∅\)\),\\displaystyle=w\\,T\(\{\\mathbf\{e\}\}\_\{\\mathrm\{c\}\}\)\+\(1\-w\)\\,T\(\{\\mathbf\{e\}\}\_\{\\varnothing\}\)\\;=\\;G\(T\(\{\\mathbf\{e\}\}\_\{\\mathrm\{c\}\}\),T\(\{\\mathbf\{e\}\}\_\{\\varnothing\}\)\),so CFG and ST commute\. Expanding the middle line recovers[28](https://arxiv.org/html/2610.02665#A7.E28)\. Recall from[1](https://arxiv.org/html/2610.02665#S2.E1)that𝐳t\(b\)\|𝐞\(b\)∼𝒩\(αt𝐞\(b\),σt2𝐈\)\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\mid\{\\mathbf\{e\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\sim\\mathcal\{N\}\(\\alpha\_\{t\}\{\\mathbf\{e\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},\\sigma\_\{t\}^\{2\}\{\\mathbf\{I\}\}\)is a Gaussian transition kernel\. Tweedie’s formula\([Robbins, 1992](https://arxiv.org/html/2610.02665#bib.bib93)\)gives an affine correspondence between any mean prediction𝐞\{\\mathbf\{e\}\}and the marginal score𝐬:=∇𝐳t\(b\)logpt\(𝐳t\(b\)∣⋅\)\{\\mathbf\{s\}\}:=\\nabla\_\{\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\}\\log p\_\{t\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\mid\\cdot\): 𝐬=αtσt2\(𝐞−μt\),𝐞=μt\+σt2αt𝐬\.\{\\mathbf\{s\}\}=\\frac\{\\alpha\_\{t\}\}\{\\sigma\_\{t\}^\{2\}\}\\big\(\{\\mathbf\{e\}\}\-\\mu\_\{t\}\\big\),\\qquad\{\\mathbf\{e\}\}=\\mu\_\{t\}\+\\frac\{\\sigma\_\{t\}^\{2\}\}\{\\alpha\_\{t\}\}\\,\{\\mathbf\{s\}\}\. Therefore, taking an affine combination of mean predictions in[29](https://arxiv.org/html/2610.02665#A7.E29)is the same as taking the same affine combination of the corresponding scores, and scaling the offset by1τ\\frac\{1\}\{\\tau\}is the same as scaling the score by1τ\\frac\{1\}\{\\tau\}\. Write𝐬c:=∇𝐳t\(b\)logpt\(𝐳t\(b\)∣𝐜\)\{\\mathbf\{s\}\}\_\{\\mathrm\{c\}\}:=\\nabla\_\{\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\}\\log p\_\{t\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\mid\{\\mathbf\{c\}\}\)and𝐬∅\{\\mathbf\{s\}\}\_\{\\varnothing\}analogously; then the score of[28](https://arxiv.org/html/2610.02665#A7.E28)is 1τ\(w𝐬c\+\(1−w\)𝐬∅\)\\displaystyle\\frac\{1\}\{\\tau\}\\big\(w\\,\{\\mathbf\{s\}\}\_\{\\mathrm\{c\}\}\+\(1\-w\)\\,\{\\mathbf\{s\}\}\_\{\\varnothing\}\\big\)=wτ∇𝐳t\(b\)logpt\(𝐳t\(b\)∣𝐜\)\+1−wτ∇𝐳t\(b\)logpt\(𝐳t\(b\)∣∅\)\\displaystyle=\\frac\{w\}\{\\tau\}\\,\\nabla\_\{\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\}\\log p\_\{t\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\mid\{\\mathbf\{c\}\}\)\+\\frac\{1\-w\}\{\\tau\}\\,\\nabla\_\{\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\}\\log p\_\{t\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\mid\\varnothing\)=∇𝐳t\(b\)log\(pt\(𝐳t\(b\)∣𝐜\)wτpt\(𝐳t\(b\)∣∅\)1−wτ\),\\displaystyle=\\nabla\_\{\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\}\\log\\big\(p\_\{t\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\mid\{\\mathbf\{c\}\}\)^\{\\frac\{w\}\{\\tau\}\}\\,p\_\{t\}\(\{\\mathbf\{z\}\}\_\{t\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\mid\\varnothing\)^\{\\frac\{1\-w\}\{\\tau\}\}\\big\),and hence the claim on the associated score is proved\. ∎ ###### Proposition 2\(EDM\-ρ\\rhoschedule converges to uniform\-γ\\gammaschedule whenρ→∞\\rho\\to\\infty\)\. FixN≥1N\\geq 1and endpoints0<σ~min<σ~max0<\\tilde\{\\sigma\}\_\{\\min\}<\\tilde\{\\sigma\}\_\{\\max\}, whereσ~t=σtαt\\tilde\{\\sigma\}\_\{t\}=\\frac\{\\sigma\_\{t\}\}\{\\alpha\_\{t\}\}and the VP noise schedule follows[Sec\.2](https://arxiv.org/html/2610.02665#S2)\. For every0≤i≤N0\\leq i\\leq N, the EDM\-ρ\\rhoand uniform\-γ\\gammagrid points of[Sec\.G\.6](https://arxiv.org/html/2610.02665#A7.SS6)satisfy σ~tiEDM\(ρ\):=\(σ~max1/ρ\+iN\(σ~min1/ρ−σ~max1/ρ\)\)ρ→ρ→∞σ~max1−i/Nσ~mini/N=:σ~tiuniform\-γ\.\\displaystyle\\tilde\{\\sigma\}\_\{t\_\{i\}\}^\{\\text\{EDM\($\\rho$\)\}\}:=\(\\tilde\{\\sigma\}\_\{\\max\}^\{1/\\rho\}\+\\tfrac\{i\}\{N\}\(\\tilde\{\\sigma\}\_\{\\min\}^\{1/\\rho\}\-\\tilde\{\\sigma\}\_\{\\max\}^\{1/\\rho\}\)\)^\{\\rho\}\\;\\xrightarrow\[\\rho\\to\\infty\]\{\}\\;\\tilde\{\\sigma\}\_\{\\max\}^\{1\-i/N\}\\tilde\{\\sigma\}\_\{\\min\}^\{i/N\}=:\\tilde\{\\sigma\}\_\{t\_\{i\}\}^\{\\text\{uniform\-$\\gamma$\}\}\.\(30\)In other words, the EDM\-ρ\\rhodiscretization converges pointwise to the uniform\-γ\\gammadiscretization whenρ→∞\\rho\\to\\infty\. ###### Proof\. The right\-hand side of[30](https://arxiv.org/html/2610.02665#A7.E30)is exactly the uniform\-γ\\gammagrid on\[σ~min,σ~max\]\[\\tilde\{\\sigma\}\_\{\\min\},\\tilde\{\\sigma\}\_\{\\max\}\]: becauseσ~t2=σt2αt2=1SNR\(t\)=eγ\(t\)\\tilde\{\\sigma\}\_\{t\}^\{2\}=\\frac\{\\sigma\_\{t\}^\{2\}\}\{\\alpha\_\{t\}^\{2\}\}=\\frac\{1\}\{\\operatorname\{SNR\}\(t\)\}=\{\\mathrm\{e\}\}^\{\\gamma\(t\)\}, the definitionγ\(ti\)=γmax\+iN\(γmin−γmax\)\\gamma\(t\_\{i\}\)=\\gamma\_\{\\max\}\+\\tfrac\{i\}\{N\}\(\\gamma\_\{\\min\}\-\\gamma\_\{\\max\}\)is equivalent tologσ~ti\\log\\tilde\{\\sigma\}\_\{t\_\{i\}\}being the arithmetic progression\(1−iN\)logσ~max\+iNlogσ~min\(1\-\\tfrac\{i\}\{N\}\)\\log\\tilde\{\\sigma\}\_\{\\max\}\+\\tfrac\{i\}\{N\}\\log\\tilde\{\\sigma\}\_\{\\min\}, which exponentiates toσ~max1−i/Nσ~mini/N\\tilde\{\\sigma\}\_\{\\max\}^\{1\-i/N\}\\,\\tilde\{\\sigma\}\_\{\\min\}^\{i/N\}\. To prove the limit, letu:=iN∈\[0,1\]u:=\\frac\{i\}\{N\}\\in\[0,1\],p:=1ρ\>0p:=\\frac\{1\}\{\\rho\}\>0,a:=σ~maxa:=\\tilde\{\\sigma\}\_\{\\max\},b:=σ~minb:=\\tilde\{\\sigma\}\_\{\\min\}\. The EDM\-ρ\\rhogrid point is the weighted power mean σ~tiEDM\(ρ\)=\(\(1−u\)ap\+ubp\)1/p,\\displaystyle\\tilde\{\\sigma\}\_\{t\_\{i\}\}^\{\\text\{EDM\($\\rho$\)\}\}=\(\(1\-u\)a^\{p\}\+ub^\{p\}\)^\{1/p\},so it suffices to show that this converges to the weighted geometric meana1−ubua^\{1\-u\}b^\{u\}asp↘0p\\searrow 0\. Takinglog\\logand performing Taylor expansionap=1\+ploga\+O\(p2\)a^\{p\}=1\+p\\log a\+O\(p^\{2\}\)aroundp=0p=0, we have \(1−u\)ap\+ubp\\displaystyle\(1\-u\)a^\{p\}\+ub^\{p\}=1\+p\(\(1−u\)loga\+ulogb\)\+O\(p2\),\\displaystyle=1\+p\(\(1\-u\)\\log a\+u\\log b\)\+O\(p^\{2\}\),⟹log\(\(1−u\)ap\+ubp\)\\displaystyle\\implies\\log\(\(1\-u\)a^\{p\}\+ub^\{p\}\)=p\(\(1−u\)loga\+ulogb\)\+O\(p2\),\\displaystyle=p\(\(1\-u\)\\log a\+u\\log b\)\+O\(p^\{2\}\),Dividing bypp, logσ~tiEDM\(ρ\)=\(1−u\)loga\+ulogb\+O\(p\)→log\(a1−ubu\),\\displaystyle\\log\\tilde\{\\sigma\}\_\{t\_\{i\}\}^\{\\text\{EDM\($\\rho$\)\}\}=\(1\-u\)\\log a\+u\\log b\+O\(p\)\\to\\log\(a^\{1\-u\}b^\{u\}\),which is exactly thelog\\logof the right\-hand side of[30](https://arxiv.org/html/2610.02665#A7.E30)\. ∎ ## Appendix HImplementation Details and Additional Results for Supervised Fine\-Tuning \[Back to[Sec\.6](https://arxiv.org/html/2610.02665#S6)in the main paper\.\] ### H\.1Supervised Fine\-Tuning Setup Our SFT pipeline follows the joint diffusion–AR design of Nemotron\-Labs\-Diffusion \(NLD\)\([Fu et al\., 2026](https://arxiv.org/html/2610.02665#bib.bib51)\): the prompt remains clean, supervision is restricted to the response, and the AR and diffusion objectives retain coefficients1\.01\.0and0\.30\.3, respectively, following the training objective in[Sec\.3\.2](https://arxiv.org/html/2610.02665#S3.SS2)\. ##### Initialization and objective\. We initialize from the 150K\-step pretrainedSigma\-\{3B, 8B\} checkpoints, which have block lengthL′=32L^\{\\prime\}=32, separate diffusion and AR readout heads, and self\-conditioning\. We fine\-tune the full transformer and both heads\. Let𝒜\\mathcal\{A\}denote the supervised answer positions, including the terminal EOS padding described below\. The SFT objective is ℒsft=0\.3∑i∈𝒜ℒBD,i\|𝒜\|\+∑i∈𝒜ℒar,i\|𝒜\|,\\displaystyle\\mathcal\{L\}\_\{\\text\{sft\}\}=0\.3\\,\\frac\{\\sum\_\{i\\in\\mathcal\{A\}\}\\mathcal\{L\}\_\{\\text\{BD\},i\}\}\{\|\\mathcal\{A\}\|\}\+\\frac\{\\sum\_\{i\\in\\mathcal\{A\}\}\\mathcal\{L\}\_\{\\text\{ar\},i\}\}\{\|\\mathcal\{A\}\|\},\(31\)whereℒBD\\mathcal\{L\}\_\{\\text\{BD\}\}is the continuous block NELBO of[3](https://arxiv.org/html/2610.02665#S3.E3)and[4](https://arxiv.org/html/2610.02665#S3.Ex2), including its diffusion, reconstruction, and prior terms, andℒar\\mathcal\{L\}\_\{\\text\{ar\}\}is next\-token cross\-entropy from the clean stream[6](https://arxiv.org/html/2610.02665#S3.E6)\. Each numerator and denominator is summed globally over data\-parallel ranks before the two terms are combined\. The implementation maintains separate denominators for the two objectives; they are equal in this setup because position zero is an unsupervised system token and both objectives otherwise use the same answer mask\. As in pre\-training \([Sec\.3\.2](https://arxiv.org/html/2610.02665#S3.SS2)\), we estimate the reconstruction and diffusion terms with the Horvitz–Thompson estimator in[5](https://arxiv.org/html/2610.02665#S3.E5): independently for each block, we select reconstruction att=0t=0with probabilityprec=0\.1p\_\{\\mathrm\{rec\}\}=0\.1and otherwise drawt∼Unif\[0,1\]t\\sim\\operatorname\{Unif\}\[0,1\]\. Gaussian noise is applied only in embedding space; the token IDs always remain clean\. Prompt and padding positions have zero loss and are pinned to their clean embeddings; answer blocks see the clean context𝐜=\(𝐩,𝐫\(<b\)\)\{\\mathbf\{c\}\}=\(\{\\mathbf\{p\}\},\{\\mathbf\{r\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)of[Sec\.4\.1](https://arxiv.org/html/2610.02665#S4.SS1)\. ##### Data and preprocessing\. Following NLD, our controlled experiment uses the same single\-turn SFT dataset of[Chandiramani et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib91)\. Every data sample has system, user, and assistant turns; the system message is empty\. The corpus is the non\-thinking variant: responses either begin with an empty<think\></think\>field \(92\.1%92\.1\\%of the samples\) or does not have think tags at all\. We use the same Mistral\-NeMo\([Mistral AI, 2024](https://arxiv.org/html/2610.02665#bib.bib81)\)tokenizer \(V=131,072V=131\{,\}072\) and the Nemotron Nano chat template\([Blakeman et al\., 2025](https://arxiv.org/html/2610.02665#bib.bib92)\)\. One key difference betweenSigma\(continuous diffusion\) and NLD \(masked diffusion\) is thatSigmarequires one well\-defined diffusion time per block, and thus cannot place prompt and answer tokens in the same noisy block\. Therefore, during training, weleft\-padeach prompt by at most 31 unsupervised tokens so that the answer begins on the absolute 32\-token block grid\. We do not insert padding between prompt and answer, and inference does not pad the prompt either due to fully causal attention for the prompt\. After the answer, supervised EOS tokens complete its final 32\-token block; the batch is then right\-padded with unsupervised PAD tokens to a multiple of 1024\. This coarser batch alignment limits the number of compiled FlexAttention sequence shapes without changing the block factorization\. Main SFT optimization configurations are listed in[Tab\.21](https://arxiv.org/html/2610.02665#A8.T21), across all model sizes\. Table 21:SFT optimization settings forSigma\.SettingValueSequence length / block length40964096/3232Global / micro batch size256256/11sequenceTraining duration80008000steps \(≈2\.4\\approx 2\.4epochs\)OptimizerAdamW,\(β1,β2\)=\(0\.9,0\.98\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.9,0\.98\),ϵ=10−5\\epsilon=10^\{\-5\}Learning rate / weight decay2\.5×10−62\.5\\times 10^\{\-6\}/0\.10\.1Schedule5050\-step linear warmup, then constantGradient clipping / precision1\.01\.0/BF16Self\-conditioning25%25\\%of microbatches each step following pre\-trainingParallelismFSDP2; TP=\{\}=\{\}CP=\{\}=\{\}1; no sequence packing ##### Frozen\-special\-parameter default and ablation\. In the main SFT run, we freeze the learned noise\-schedule networkγ~ψ\(t\)\\tilde\{\\gamma\}\_\{\\psi\}\(t\)and its endpointsγ0,γ1\\gamma\_\{0\},\\gamma\_\{1\}, as well as the diffusion embedding matrix𝐄\{\\mathbf\{E\}\}\. This freezes only the continuous\-diffusion geometry; the transformer, input and conditioning projections, self\-conditioning projection, and both output heads remain trainable at the base learning rate\. Our ablation unfreezes the schedule, endpoints, and diffusion embedding matrix while holding all other settings fixed\. Following pre\-training \([App\.E](https://arxiv.org/html/2610.02665#A5)\), these parameters use a1000×1000\\timeslearning\-rate multiplier, giving a peak rate of2\.5×10−32\.5\\times 10^\{\-3\}under the same scheduler; weight decay is zero for the schedule network and diffusion embedding matrix, while the two scalar endpoints retain the base weight decay of0\.10\.1\. ### H\.2Evaluation Setup and Frozen\-Parameter Ablation The instruct evaluations in[Tabs\.3](https://arxiv.org/html/2610.02665#S6.T3)and[4](https://arxiv.org/html/2610.02665#S6.T4)share the diffusion decoder, answer extraction, and seed\-error\-bar procedure of the base\-model harness \([Sec\.G\.1](https://arxiv.org/html/2610.02665#A7.SS1)\); we describe here only what differs at the instruct operating point\. ##### Benchmarks\. We evaluate on ten tasks \(numbers in brackets represent the number of problems\): GSM8K \(1319\), MATH\-500 \(500\), AIME 2024 and AIME 2025 \(30 each\), HumanEval \(164\), MBPP \(378\), LiveCodeBench\-CPP \(454\), MMLU \(14042\), GPQA \(198\), and IFEval \(541\)\. Prompts follow the Nemotron Nano chat template used in SFT training, and the assistant turn is generated in diffusion mode with block lengthL′=32L^\{\\prime\}=32\. ##### Scoring\. Math tasks \(GSM8K, MATH\-500, AIME\) are scored by symbolic equivalence on the final\\boxed\{\}answer; GSM8K additionally accepts the last number in the completion\. HumanEval and MBPP report executed pass@1@1on the base test suites \(not the\+\+variants\)\. LiveCodeBench\-CPP compiles and runs each extracted C\+\+ program against its tests\. GPQA and MMLU are scored by the extracted multiple\-choice letter; both are prompted for free\-form reasoning ending inAnswer: \\boxed\{A/B/C/D\}, and the grader reads the*last*\\boxed\{\}in the completion, falling back to a regular expression when the model emits none\. IFEval reports the instruction\-level loose\-following rate from theinstruction\_following\_evalcheckers, which is the metric the reference models in[Tabs\.3](https://arxiv.org/html/2610.02665#S6.T3)and[4](https://arxiv.org/html/2610.02665#S6.T4)report\. ##### Baselines\. All baseline numbers in[Tabs\.3](https://arxiv.org/html/2610.02665#S6.T3)and[4](https://arxiv.org/html/2610.02665#S6.T4)are copied from[Fu et al\. \(2026, Tabs\. 5, 7\)](https://arxiv.org/html/2610.02665#bib.bib51), except that NLD\-8B’s diffusion mode results are our reproduction under aligned protocol \(without format\-injection tricks\)\.[Fu et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib51)report SDAR\-8B Chat at two confidence\-threshold operating points \(TPF=1\.00=1\.00and TPF=1\.75=1\.75\); we use the former as it is stronger on 8 of the 10 tasks \(the two exceptions, MMLU and HumanEval, differ by≤0\.6\\leq 0\.6points\)\. ##### Decoding algorithms\. We report two operating points, both selected on the base model \([Tab\.1](https://arxiv.org/html/2610.02665#S4.T1)\) and both with self\-conditioning enabled\. The DDPM\-256 rows use the ancestral sampler on the EDMρ=7\\rho=7schedule with guidancew=2w=2and score temperatureτ=0\.1\\tau=0\.1; the DDIM\-32 rows use the deterministic sampler on the uniform\-γ\\gammaschedule withw=3w=3andτ=0\.3\\tau=0\.3\. ##### Generation budget\. Unlike in[Fu et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib51)who used the same1683416834budget for all tasks, the maximum number of generated tokens forSigmais set per benchmark rather than uniformly:15361536for GSM8K,25602560for HumanEval, MBPP, LiveCodeBench\-CPP, and IFEval,25602560for MMLU, and40964096for AIME 2024, AIME 2025, MATH\-500, and GPQA\. These budgets are sized to the per\-lane graded\-answer\-length distribution: on our small\-scale smoke runs, no correct answer used a budget beyond40964096, and the long tail of length is dominated by degenerate repetition that a larger budget does not convert into correct answers\. ##### Seed error bars\. As in the base\-model harness, each row draws1010independent sampling lanes, averages over questions within each lane to obtain1010per\-lane scores, and reports the mean with half\-width210σ^\\frac\{2\}\{\\sqrt\{10\}\}\\hat\{\\sigma\}, whereσ^\\hat\{\\sigma\}is the across\-lane standard deviation\. ##### Frozen\-versus\-unfrozen ablation\. [Tab\.22](https://arxiv.org/html/2610.02665#A8.T22)compares the frozen default of[Tab\.3](https://arxiv.org/html/2610.02665#S6.T3)against the unfrozen variant of the previous paragraph, which trains the noise schedule, itsγ0,γ1\\gamma\_\{0\},\\gamma\_\{1\}endpoints, and the diffusion embedding matrix at a1000×1000\\timeslearning\-rate multiplier as in pre\-training \([App\.E](https://arxiv.org/html/2610.02665#A5)\)\. We run the comparison at both sampler operating points of[Tab\.3](https://arxiv.org/html/2610.02665#S6.T3)\. Under DDPM\-256, the two configurations are within the seed interval on seven of ten tasks; under DDIM\-32, the ordering of the average reverses, with the unfrozen model ahead\. As the two configurations are close in performance in general, we keep the frozen configuration as the simpler default for 8B\. Table 22:Freezing the continuous\-diffusion geometry during SFT\.3B instruct performance, mean±\\pmtwo standard errors over1010sampling lanes, under both sampler operating points of[Tab\.3](https://arxiv.org/html/2610.02665#S6.T3)\.*Frozen*is the default of[Tab\.3](https://arxiv.org/html/2610.02665#S6.T3): the noise schedule, its endpoints, and the diffusion embedding matrix are held fixed\.*Unfrozen*trains all three at a1000×1000\\timeslearning\-rate multiplier\.Avg\.is the unweighted mean over the ten tasks\. The frozen/unfrozen ordering of the average is not stable across the two samplers, and in general the gap is within the seed interval\.MathCodingMultiple ChoiceIFConfigurationGSM8KMath500AIME24AIME25HumanEvalMBPPLCB\-CPPMMLUGPQAIFEvalAvg\.*DDPM \(256256steps,w=2,τ=0\.1w\{=\}2,\\tau\{=\}0\.1\)*Frozen \(default\)52\.15Unfrozen \(1000×1000\\timesLR\)51\.65*DDIM \(3232steps,w=3,τ=0\.3w\{=\}3,\\tau\{=\}0\.3\)*Frozen \(default\)49\.86Unfrozen \(1000×1000\\timesLR\)50\.84 ## Appendix IImplementation Details and Additional Results for Parallel Decoding Distillation \[Back to[Sec\.5\.3](https://arxiv.org/html/2610.02665#S5.SS3)in the main paper\.\] ### I\.1Parallel Predictions and Intra\-Segment Rollout PDD reduces the number of expensive denoiser evaluations without discarding the teacher’s fine integration grid\. Following[Shaul et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib72), we let1=t0\>t1\>⋯\>tG=01=t\_\{0\}\>t\_\{1\}\>\\cdots\>t\_\{G\}=0be a decreasing time grid\. A conventional DDIM sampler queries the denoiser once per interval and therefore requiresGGNFEs\. For a target budgetM<GM<G, PDD instead partitions the grid intoMM*segments*,444We use*segment*rather than the term*block*from[Shaul et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib72)to avoid confusion with the language\-model diffusion blocks\.each spanningΔ=G/M\\Delta=G/Mfine intervals\. At the start of segmentkk, whereik=kΔi\_\{k\}=k\\Delta, the student evaluates its transformer backbone once and emitsΔ\\Deltaraw clean\-embedding predictions required within that segment: 𝐞ϕ\(𝐳ik,tik,𝐱\(<b\)\)=\{𝐞^ik,jS,raw\}j=0Δ−1\.\\displaystyle\{\\mathbf\{e\}\}\_\{\\phi\}\(\{\\mathbf\{z\}\}\_\{i\_\{k\}\},t\_\{i\_\{k\}\};\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)=\\\{\\hat\{\\mathbf\{e\}\}^\{S,\\text\{raw\}\}\_\{i\_\{k\},j\}\\\}\_\{j=0\}^\{\\Delta\-1\}\.\(32\)Herejjis the local offset within the segment, while the corresponding prediction head is indexed by the absolute fine\-grid positionik\+ji\_\{k\}\+j; we describe this parameterization in[Sec\.I\.2](https://arxiv.org/html/2610.02665#A9.SS2)\. AllΔ\\Deltaoutputs share the same backbone representation and differ only through lightweight output heads\. Because these heads are evaluated in parallel, they cannot consume recurrent self\-conditioning from earlier local nodes; we thereforedisable self\-conditioningfor both teacher and student during distillation\. Starting from the segment input, these predictions are then composed through the ordinary DDIM recursion[DDIM](https://arxiv.org/html/2610.02665#S4.Ex5): 𝐳~ik,0\\displaystyle\\tilde\{\\mathbf\{z\}\}\_\{\{i\_\{k\}\},0\}=𝐳ik,\\displaystyle=\{\\mathbf\{z\}\}\_\{i\_\{k\}\},𝐳~ik,j\+1\\displaystyle\\tilde\{\\mathbf\{z\}\}\_\{\{i\_\{k\}\},j\+1\}=αik,j\+1𝐞^Sik,j\+σik,j\+1⋅𝐳~ik,j−αik,j𝐞^ik,jSσik,j,j=0,1,…,Δ−1,\\displaystyle=\\alpha\_\{\{i\_\{k\}\},j\+1\}\\hat\{\\mathbf\{e\}\}^\{S\}\_\{i\_\{k\},j\}\+\\sigma\_\{\{i\_\{k\}\},j\+1\}\\cdot\\frac\{\\tilde\{\\mathbf\{z\}\}\_\{\{i\_\{k\}\},j\}\-\\alpha\_\{\{i\_\{k\}\},j\}\\hat\{\\mathbf\{e\}\}^\{S\}\_\{i\_\{k\},j\}\}\{\\sigma\_\{\{i\_\{k\}\},j\}\},\\penalty\\ j=0,1,\.\.\.,\\Delta\-1,\(33\)whereαik,j=αtik\+j\\alpha\_\{\{i\_\{k\}\},j\}=\\alpha\_\{t\_\{i\_\{k\}\+j\}\},σik,j=σtik\+j\\sigma\_\{\{i\_\{k\}\},j\}=\\sigma\_\{t\_\{i\_\{k\}\+j\}\}, and𝐞^ik,jS\\hat\{\\mathbf\{e\}\}^\{S\}\_\{i\_\{k\},j\}denotes the guided student prediction, constructed as follows\. The raw prediction in[32](https://arxiv.org/html/2610.02665#A9.E32)includes the CFG behavior distilled from the teacher but precedes score temperature \(ST\)\. Before each solver update, we apply ST analytically using the state𝐳~ik,j\\tilde\{\\mathbf\{z\}\}\_\{\{i\_\{k\}\},j\}reached at that local node according to the local DDIM rollout[33](https://arxiv.org/html/2610.02665#A9.E33), 𝐞^ik,jS\\displaystyle\\hat\{\\mathbf\{e\}\}^\{S\}\_\{i\_\{k\},j\}=1τ𝐞^ik,jS,raw\+\(1−1τ\)𝐳~ik,jαik,j\.\\displaystyle=\\frac\{1\}\{\\tau\}\\hat\{\\mathbf\{e\}\}^\{S,\\text\{raw\}\}\_\{i\_\{k\},j\}\+\\big\(1\-\\frac\{1\}\{\\tau\}\\big\)\\frac\{\\tilde\{\\mathbf\{z\}\}\_\{\{i\_\{k\}\},j\}\}\{\\alpha\_\{\{i\_\{k\}\},j\}\}\.\(34\) Thus, PDD parallelizes the expensive denoiser predictions, not the inexpensive state updates within a segment\. Only the DDIM endpoint𝐳~ik,Δ\\tilde\{\\mathbf\{z\}\}\_\{\{i\_\{k\}\},\\Delta\}is passed to the next transformer evaluation as𝐳ik\+1\{\\mathbf\{z\}\}\_\{i\_\{k\+1\}\}\. ### I\.2Residual Parallel Heads Learning to predict an independent clean embedding at every fine\-grid node would discard the strong prediction already produced by the pretrained denoiser at each segment start\. We instead retain the original diffusion readout as an anchor and parameterize every parallel prediction as a residual transformation of it\. Following[Shaul et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib72), each residual head is associated with an absolute fine\-grid index\. Segmentkktherefore selects Rik,Rik\+1,…,Rik\+Δ−1\.\\displaystyle R\_\{i\_\{k\}\},R\_\{i\_\{k\}\+1\},\\ldots,R\_\{i\_\{k\}\+\\Delta\-1\}\.\(35\)LetHϕ\(𝐳ik,tik,𝐱\(<b\)\)H\_\{\\phi\}\(\{\\mathbf\{z\}\}\_\{i\_\{k\}\},t\_\{i\_\{k\}\};\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)be the shared transformer representation from[32](https://arxiv.org/html/2610.02665#A9.E32)\. Its original diffusion readout, which serves as the anchor, is 𝐞^ikS,anchor=softmax\(WoutHϕ\(𝐳ik,tik,𝐱\(<b\)\)\+ℓprior\(𝐳ik,tik\)\)𝐄,\\displaystyle\\hat\{\\mathbf\{e\}\}\_\{i\_\{k\}\}^\{S,\\text\{anchor\}\}=\\operatorname\{softmax\}\(W\_\{\\text\{out\}\}H\_\{\\phi\}\(\{\\mathbf\{z\}\}\_\{i\_\{k\}\},t\_\{i\_\{k\}\};\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)\+\{\\bm\{\\ell\}\}^\{\\text\{prior\}\}\(\{\\mathbf\{z\}\}\_\{i\_\{k\}\},t\_\{i\_\{k\}\}\)\)\{\\mathbf\{E\}\},\(36\)whereWoutW\_\{\\text\{out\}\}is the vocabulary projection andℓprior\(𝐳ik,tik\)\{\\bm\{\\ell\}\}^\{\\text\{prior\}\}\(\{\\mathbf\{z\}\}\_\{i\_\{k\}\},t\_\{i\_\{k\}\}\)denotes the output\-prior logits\. For local offsetjj, the corresponding absolute head produces \(𝐬ik,j,𝐫ik,j\)\\displaystyle\(\\mathbf\{s\}\_\{i\_\{k\},j\},\\mathbf\{r\}\_\{i\_\{k\},j\}\)=Rik\+j\(Hϕ\(𝐳ik,tik,𝐱\(<b\)\)\),\\displaystyle=R\_\{i\_\{k\}\+j\}\(H\_\{\\phi\}\(\{\\mathbf\{z\}\}\_\{i\_\{k\}\},t\_\{i\_\{k\}\};\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)\),𝐪ik,j\\displaystyle\\mathbf\{q\}\_\{i\_\{k\},j\}=exp\(2tanh\(𝐬ik,j\)\),\\displaystyle=\\exp\(2\\tanh\(\\mathbf\{s\}\_\{i\_\{k\},j\}\)\),𝐞^ik,jS,raw\\displaystyle\\hat\{\\mathbf\{e\}\}\_\{i\_\{k\},j\}^\{S,\\text\{raw\}\}=𝐪ik,j⊙𝐞^ikS,anchor\+𝐫ik,j\.\\displaystyle=\\mathbf\{q\}\_\{i\_\{k\},j\}\\odot\\hat\{\\mathbf\{e\}\}\_\{i\_\{k\}\}^\{S,\\text\{anchor\}\}\+\\mathbf\{r\}\_\{i\_\{k\},j\}\.\(37\)EachRiR\_\{i\}is an embedding\-space MLP whose final layer is initialized at zero, so every parallel head initially reproduces the pretrained anchor prediction\. The student therefore begins from the pretrained denoiser rather than from randomly initialized parallel predictions, and each residual head only needs to model how the clean prediction evolves along the local trajectory\. The additive residual and scaling also allow predictions to leave the convex hull of all token embeddings, which is necessary for matching extrapolated CFG\-guided teacher predictions\. ### I\.3Distillation Objective and Optimization Throughout this subsection, every teacher evaluation is treated as a fixed target: the teacher parameters are frozen, and any student\-generated input state is detached at the teacher\-query boundary\. We omit stop\-gradient operators from the notation for simplicity\. In contrast, the student evaluation and its intra\-segment rollout are constructed with gradients enabled\. We distill CFG into the conditional student, while leaving ST as the analytic transformation in[34](https://arxiv.org/html/2610.02665#A9.E34)\. At a student\-visited state𝐳~ik,j\\tilde\{\\mathbf\{z\}\}\_\{\{i\_\{k\}\},j\}, the frozen teacher’s CFG\-guided prediction before ST is 𝐞^ik,jT,raw\\displaystyle\\hat\{\\mathbf\{e\}\}^\{T,\\text\{raw\}\}\_\{i\_\{k\},j\}=𝐞T\(𝐳~ik,j,tik,j,∅\)\+w\[𝐞T\(𝐳~ik,j,tik,j,𝐱\(<b\)\)−𝐞T\(𝐳~ik,j,tik,j,∅\)\]\.\\displaystyle=\{\\mathbf\{e\}\}\_\{T\}\(\\tilde\{\\mathbf\{z\}\}\_\{\{i\_\{k\}\},j\},t\_\{\{i\_\{k\}\},j\};\\varnothing\)\+w\[\{\\mathbf\{e\}\}\_\{T\}\(\\tilde\{\\mathbf\{z\}\}\_\{\{i\_\{k\}\},j\},t\_\{\{i\_\{k\}\},j\};\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)\-\{\\mathbf\{e\}\}\_\{T\}\(\\tilde\{\\mathbf\{z\}\}\_\{\{i\_\{k\}\},j\},t\_\{\{i\_\{k\}\},j\};\\varnothing\)\]\.\(38\)The student receives only the conditional prefix𝐱\(<b\)\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}, so matching this target amortizes the teacher’s conditional and unconditional evaluations into a single student call\. Crucially, both𝐞^ik,jS,raw\\hat\{\\mathbf\{e\}\}^\{S,\\text\{raw\}\}\_\{i\_\{k\},j\}and𝐞^ik,jT,raw\\hat\{\\mathbf\{e\}\}^\{T,\\text\{raw\}\}\_\{i\_\{k\},j\}denote predictions*before*ST; ST affects the states visited by the local rollout but is not part of the regression target\. For each language\-model diffusion blockb\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}, we sample a segment starttikt\_\{i\_\{k\}\}uniformly from the candidate starts and draw𝐳ik\{\\mathbf\{z\}\}\_\{i\_\{k\}\}from the VP forward marginal conditioned on the clean data embedding\. One student evaluation produces the parallel predictions in[32](https://arxiv.org/html/2610.02665#A9.E32), from which we construct the differentiable local rollout in[33](https://arxiv.org/html/2610.02665#A9.E33)\. We always supervise the first local node,j1=0j\_\{1\}=0, to anchor the segment\-start prediction, and sample the remainingP−1P\-1indices\{j2,…,jP\}\\\{j\_\{2\},\\ldots,j\_\{P\}\\\}uniformly without replacement from\{1,…,Δ−1\}\\\{1,\\ldots,\\Delta\-1\\\}\. The PDD objective compares the raw student and CFG\-guided teacher predictions at these student\-visited states: ℒPDD\\displaystyle\\mathcal\{L\}\_\{\\text\{PDD\}\}=1P∑p=1P𝔼\[‖𝐞^ik,jpS,raw−𝐞^ik,jpT,raw‖2\]\.\\displaystyle=\\frac\{1\}\{P\}\\sum\_\{p=1\}^\{P\}\\E\[\\\|\\hat\{\\mathbf\{e\}\}^\{S,\\text\{raw\}\}\_\{i\_\{k\},j\_\{p\}\}\-\\hat\{\\mathbf\{e\}\}^\{T,\\text\{raw\}\}\_\{i\_\{k\},j\_\{p\}\}\\\|^\{2\}\]\.\(39\)This construction is intra\-segment on\-policy: although the segment start is sampled from the forward marginal, all later supervised states within the segment are generated by the current student\. The teacher therefore corrects the student where it locally visits, rather than only on teacher or forward\-process states\. Because trajectory distillation supervises only the trajectory and only on a sparse set of guided states, we additionally match the teacher’s categorical reconstruction and autoregressive distributions to preserve the pretrained model’s behavior\. We use forward KL divergences on the corresponding logits: ℒteacher\-rec=𝔼𝐳0\(b\)∼q\(𝐳0\(b\)∣𝐱\(b\)\)∑l=1L′KL\(𝐱Tl\(𝐳0\(b\),0;𝐱\(<b\)\)∥𝐱ϕl\(𝐳0\(b\),0;𝐱\(<b\)\)\)\.\\mathcal\{L\}\_\{\\text\{teacher\-rec\}\}=\\E\_\{\{\\mathbf\{z\}\}\_\{0\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\sim q\(\{\\mathbf\{z\}\}\_\{0\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\\mid\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\}\)\}\\sum\_\{l=1\}^\{L^\{\\prime\}\}\\operatorname\{KL\}\(\{\\mathbf\{x\}\}\_\{T\}^\{l\}\(\{\\mathbf\{z\}\}\_\{0\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},0;\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)\\,\\\|\\,\{\\mathbf\{x\}\}\_\{\\phi\}^\{l\}\(\{\\mathbf\{z\}\}\_\{0\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\}\)\},0;\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)\)\.\(40\)ℒteacher\-ar=∑l=1L′KL\(𝐱TARTl\(𝐱\(b,<l\);𝐱\(<b\)\)∥𝐱ϕARl\(𝐱\(b,<l\);𝐱\(<b\)\)\)\.\\mathcal\{L\}\_\{\\text\{teacher\-ar\}\}=\\sum\_\{l=1\}^\{L^\{\\prime\}\}\\operatorname\{KL\}\(\{\\mathbf\{x\}\}\_\{T\_\{\\text\{AR\}\}^\{T\}\}^\{l\}\(\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\},<l\)\};\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)\\,\\\|\\,\{\\mathbf\{x\}\}\_\{\\phi\_\{\\text\{AR\}\}\}^\{l\}\(\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}b\},<l\)\};\{\\mathbf\{x\}\}^\{\(\{\\color\[rgb\]\{0,0\.4297,0\.2148\}<b\}\)\}\)\)\.\(41\) By construction, these losses are zero as distillation begins\. Together, this yields the combined objective: ℒdistill=λPDDℒPDD\+λteacher\-recℒteacher\-rec\+λteacher\-arℒteacher\-ar,\\displaystyle\\mathcal\{L\}\_\{\\text\{distill\}\}=\\lambda\_\{\\text\{PDD\}\}\\,\\mathcal\{L\}\_\{\\text\{PDD\}\}\+\\lambda\_\{\\text\{teacher\-rec\}\}\\,\\mathcal\{L\}\_\{\\text\{teacher\-rec\}\}\+\\lambda\_\{\\text\{teacher\-ar\}\}\\,\\mathcal\{L\}\_\{\\text\{teacher\-ar\}\},\(42\)Teacher and student are initialized from the same checkpoint, after which the teacher remains frozen\. We also freeze the embedding matrix and noise schedule so that the teacher trajectory retains a fixed meaning throughout distillation\. The student transformer is adapted through LoRA, while the residual heads are optimized directly with a separate learning rate\. This separation allows the new heads to learn local corrections without destabilizing the pretrained backbone or diffusion calibration\. Full training details and hyperparameters are given in[Sec\.I\.5](https://arxiv.org/html/2610.02665#A9.SS5)\. ### I\.4Few\-Step Inference For a requested NFE budgetMM, we evaluate the student atMMsegment starts in sequence\. At segmentkk, the model uses the absolute headsRik,…,Rik\+ΔM−1R\_\{i\_\{k\}\},\\ldots,R\_\{i\_\{k\}\+\\Delta\_\{M\}\-1\}, whereΔM=G/M\\Delta\_\{M\}=G/M, and advances the state through[34](https://arxiv.org/html/2610.02665#A9.E34)and[33](https://arxiv.org/html/2610.02665#A9.E33)\. Only the segment endpoint is passed to the next transformer evaluation\. The student therefore reduces the sequential neural\-network depth fromGGevaluations toMM, while retaining fine\-grid DDIM integration between evaluations\. ### I\.5Training and Parameterization Details ##### Training details\. We initialize the student from baseSigma\-3B \(with or without SUSReg regularization, ablation results are shown in[Fig\.7](https://arxiv.org/html/2610.02665#S5.F7)\), which also serves as the frozen distillation teacher\. The embedding lookup, vocabulary projection and noise schedule \(including terminal points\) are frozen\. The transformer backbone is adapted with LoRA of rank128128and scaling factor256256using learning rate10−610^\{\-6\}\. We use a global bank of3232parallel residual heads parameterized by three\-layer MLPs of width384384with standard LayerNorm, which use a separate parameter group with learning rate2\.5×10−62\.5\\times 10^\{\-6\}\. We use AdamW with weight decay 1\.0,β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999and a batch size of256256\. For all parameters, we use gradient clipping value 1\.0\. The learning rate schedule is Warmup\-Stable\-Decay over 10K steps: 5% linear warmup to the nominal learning rate value, constant through step 8000, then exponential decay to 10% of nominal learning rate value\. We use a fixed clean context of512512tokens, and train on a signal block of3232tokens\. We noticed that this hardly impacts distillation results compared to training on all possible context lengths\{0,32,…,4064\}\\\{0,32,\\dots,4064\\\}used for pre\-training, while simplifying and significantly accelerating distillation\. Our student is trained flexibly atMmin=4M\_\{\\mathrm\{min\}\}=4andMmax=8M\_\{\\mathrm\{max\}\}=8NFE\. For NFE\-4 results in[Fig\.7](https://arxiv.org/html/2610.02665#S5.F7)in the main paper, the teacher trajectory uses classifier\-free guidancew=4w=4and score temperatureτ=0\.4\\tau=0\.4swhile for NFE\-8 results, the best guidance setting withw=3w=3andτ=0\.3\\tau=0\.3\. These were the only guidance settings investigated and were informed by the self\-conditioning\-disabled teacher DDIM performance at 32 NFEs\. We superviseP=8P=8sampled downstream nodes per segment\. Inℒdistill\\mathcal\{L\}\_\{\\text\{distill\}\}[42](https://arxiv.org/html/2610.02665#A9.E42), the PDD objective component has weightλPDD=5\.0\\lambda\_\{\\text\{PDD\}\}=5\.0, the teacher\-KL regularization on diffusion stream has weightλteacher\-rec=0\.1\\lambda\_\{\\text\{teacher\-rec\}\}=0\.1and teacher\-KL regularization on the clean AR stream has weightλteacher\-ar=0\.3\\lambda\_\{\\text\{teacher\-ar\}\}=0\.3\. These weights were chosen to balance the respective gradient norms\. ##### Flexible\-NFE head sharing\. Following[Shaul et al\. \(2026\)](https://arxiv.org/html/2610.02665#bib.bib72), we train one student for a set of inference budgets ℳ=\{Mmin,2Mmin,…,Mmax\},Mmax=2aMmin\.\\displaystyle\\mathcal\{M\}=\\\{M\_\{\\min\},2M\_\{\\min\},\\ldots,M\_\{\\max\}\\\},\\qquad M\_\{\\max\}=2^\{a\}M\_\{\\min\}\.\(43\)For budgetM∈ℳM\\in\\mathcal\{M\}, one student evaluation advancesΔM=G/M\\Delta\_\{M\}=G/Mfine\-grid intervals before the transformer is evaluated again\. Because the residual heads are indexed by absolute fine\-grid position, segmentkkselects the contiguous sliceRik,…,Rik\+ΔM−1R\_\{i\_\{k\}\},\\ldots,R\_\{i\_\{k\}\+\\Delta\_\{M\}\-1\}\. ChangingMMchanges the segment boundaries and slice lengths while reusing the same backbone and bank of absolute heads\. A single student can therefore be evaluated at several NFE budgets without an additional NFE embedding or a separately distilled model for each budget\. During training, NFE budgets are stratified across global microbatches so that all values inℳ\\mathcal\{M\}receive comparable supervision\. This exposes the model both to longer, more difficult intra\-segment rollouts at low NFE and to shorter rollouts at higher NFE\. ##### Additional results We synthesize the results in[Fig\.7](https://arxiv.org/html/2610.02665#S5.F7)in the main paper for GSM8K, HumanEval and MBPP\. All\-task results are displayed in[Tab\.8](https://arxiv.org/html/2610.02665#A7.T8)\. PDD achieves a clear 2x acceleration from NFE\-16 to NFE\-8 relative to its self\-conditioning\-disabled teacher\. Compared to the self\-conditioning\-enabledSigma\-3B\-SUSReg model, which obtains the best results for NFE\>\>16, PDD achieves a 2×\\timesacceleration from NFE\-8 to NFE\-4, where it boasts a 68% GSM8K flexible\-extract accuracy at NFE=4 while the non\-distilled models achieve at most 32%\. Coding is empirically more NFE\-sensitive: PDD allows a more graceful decay as NFE decreases, but NFE\-4 pass@1 still falls short of the performance at NFE\>\>16\. Importantly, initializing the student and frozen teacher from a checkpoint regularized with SUSReg yields much better distillation performance: \+4\.67pp at NFE\-8 and \+3\.71pp at NFE\-4\. This confirms the importance of regularizing the embedding geometry in such a few\-NFE setup\. ## Appendix JExample Outputs Below, we present example outputs from the pretrained and instruct 3B models to showcase the reasoning capabilities of the models across multiple tasks\. We only make minimal modifications to the original outputs \(e\.g\., correcting the unpairedLaTeXbrackets that otherwise break the compilation, and changing markdown items toLaTeXitemize environment for better visual appearance\)\. ### J\.1Sigma\-Base\-3B GSM8K \(8 shot\): DDIM,88steps/block, CFG22, ST0\.70\.7Q:There are 66 fish in the fish tank\. One\-third of the fish have red stripes, and 5/11 of the remaining fish have blue stripes\. Altogether, how many fish have red stripes and blue stripes?A:There are 66 fish originally\. One\-third of them have red stripes, which is 66 \* \(1/3\) = 22 fish\. That leaves 66 \- 22 = 44 fish remaining\. 5/11 of the remaining fish have blue stripes, which is 44 \* \(5/11\) = 20 fish\. So there are 22 \+ 20 = 42 fish with red or blue stripes\. The answer is 42\. MBPP \(3 shot\): DDIM,88steps/block, CFG22, ST0\.70\.7Q:Write a function to count all the distinct pairs having a difference of k in any array\.A:[⬇](data:text/plain;base64,ZGVmIGNvdW50X3BhaXJzKG51bXMsIG4sIGspOgogICAgY291bnQgPSAwCiAgICBwYWlycyA9IHNldCgpCiAgICBmb3IgbnVtIGluIG51bXM6CiAgICAgIGlmIG51bSArIGsgaW4gcGFpcnM6CiAgICAgICAgY291bnQgKz0gMQogICAgICBpZiBudW0gLSBrIGluIHBhaXJzOgogICAgICAgIGNvdW50ICs9IDEKICAgICAgcGFpcnMuYWRkKG51bSkKICAgIHJldHVybiBjb3VudA==)defcount\_pairs\(nums,n,k\):count=0pairs=set\(\)fornuminnums:ifnum\+kinpairs:count\+=1ifnum\-kinpairs:count\+=1pairs\.add\(num\)returncount HumanEval \(0 shot\): DDIM,3232steps/block, CFG22, ST0\.70\.7Q:[⬇](data:text/plain;base64,ZnJvbSB0eXBpbmcgaW1wb3J0IExpc3QKCgpkZWYgaGFzX2Nsb3NlX2VsZW1lbnRzKG51bWJlcnM6IExpc3RbZmxvYXRdLCB0aHJlc2hvbGQ6IGZsb2F0KSAtPiBib29sOgogICAgIiIiIENoZWNrIGlmIGluIGdpdmVuIGxpc3Qgb2YgbnVtYmVycywgYXJlIGFueSB0d28gbnVtYmVycyBjbG9zZXIgdG8gZWFjaCBvdGhlciB0aGFuCiAgICBnaXZlbiB0aHJlc2hvbGQuCiAgICA+Pj4gaGFzX2Nsb3NlX2VsZW1lbnRzKFsxLjAsIDIuMCwgMy4wXSwgMC41KQogICAgRmFsc2UKICAgID4+PiBoYXNfY2xvc2VfZWxlbWVudHMoWzEuMCwgMi44LCAzLjAsIDQuMCwgNS4wLCAyLjBdLCAwLjMpCiAgICBUcnVlCiAgICAiIiI=)fromtypingimportListdefhas\_close\_elements\(numbers:List\[float\],threshold:float\)\-\>bool:"""Checkifingivenlistofnumbers,areanytwonumbersclosertoeachotherthangiventhreshold\.\>\>\>has\_close\_elements\(\[1\.0,2\.0,3\.0\],0\.5\)False\>\>\>has\_close\_elements\(\[1\.0,2\.8,3\.0,4\.0,5\.0,2\.0\],0\.3\)True"""A:[⬇](data:text/plain;base64,ICAgIGZvciBpIGluIHJhbmdlKGxlbihudW1iZXJzKSk6CiAgICAgICAgZm9yIGogaW4gcmFuZ2UoaSArIDEsIGxlbihudW1iZXJzKSk6CiAgICAgICAgICAgIGlmIGFicyhudW1iZXJzW2ldIC0gbnVtYmVyc1tqXSkgPCB0aHJlc2hvbGQ6CiAgICAgICAgICAgICAgICByZXR1cm4gVHJ1ZQogICAgcmV0dXJuIEZhbHNl)foriinrange\(len\(numbers\)\):forjinrange\(i\+1,len\(numbers\)\):ifabs\(numbers\[i\]\-numbers\[j\]\)<threshold:returnTruereturnFalse Minerva math \(4 shot\): DDIM,3232steps/block, CFG22, ST0\.70\.7Q:In right triangleABCABC, shown below,cosB=610\\cos\{B\}=\\frac\{6\}\{10\}\. What istanC\\tan\{C\}?[⬇](data:text/plain;base64,W2FzeV0KZHJhdygoMCwwKS0tKDEwLDApLS0oMy42LDQuOCktLWN5Y2xlLGJsYWNrK2xpbmV3aWR0aCgxKSk7CmRyYXcocmlnaHRhbmdsZW1hcmsoKDAsMCksKDMuNiw0LjgpLCgxMCwwKSwyMCksYmxhY2srbGluZXdpZHRoKDEpKTsKbGFiZWwoIiRDJCIsKDEwLDApLEUpOwpsYWJlbCgiJEEkIiwoMy42LDQuOCksTik7CmxhYmVsKCIkQiQiLCgwLDApLFcpOwpsYWJlbCgiMTAiLCgwLDApLS0oMTAsMCksUyk7ClsvYXN5XQ==)\[asy\]draw\(\(0,0\)\-\-\(10,0\)\-\-\(3\.6,4\.8\)\-\-cycle,black\+linewidth\(1\)\);draw\(rightanglemark\(\(0,0\),\(3\.6,4\.8\),\(10,0\),20\),black\+linewidth\(1\)\);label\("$C$",\(10,0\),E\);label\("$A$",\(3\.6,4\.8\),N\);label\("$B$",\(0,0\),W\);label\("10",\(0,0\)\-\-\(10,0\),S\);\[/asy\]A:In right triangleABCABCwithcosB=610\\cos\{B\}=\\frac\{6\}\{10\}, we first determinesinB\\sin\{B\}using the Pythagorean identity:sin2B=1−cos2B=1−\(610\)2=1−36100=64100\.\\sin^\{2\}\{B\}=1\-\\cos^\{2\}\{B\}=1\-\\left\(\\frac\{6\}\{10\}\\right\)^\{2\}=1\-\\frac\{36\}\{100\}=\\frac\{64\}\{100\}\.Taking the positive square root \(sinceBBis in the first quadrant\), we get:sinB=810=45\.\\sin\{B\}=\\frac\{8\}\{10\}=\\frac\{4\}\{5\}\.Next,tanB=sinBcosB=45610=45×106=4030=43\\tan\{B\}=\\frac\{\\sin\{B\}\}\{\\cos\{B\}\}=\\frac\{\\frac\{4\}\{5\}\}\{\\frac\{6\}\{10\}\}=\\frac\{4\}\{5\}\\times\\frac\{10\}\{6\}=\\frac\{40\}\{30\}=\\frac\{4\}\{3\}\.In a right triangle,tanC=cotB=1tanB=143=34\\tan\{C\}=\\cot\{B\}=\\frac\{1\}\{\\tan\{B\}\}=\\frac\{1\}\{\\frac\{4\}\{3\}\}=\\frac\{3\}\{4\}\.Thus, the value oftanC\\tan\{C\}is34\\boxed\{\\frac\{3\}\{4\}\}\.Final Answer: The final answer is34\\frac\{3\}\{4\}\. I hope it is correct\. ### J\.2Sigma\-3B\-Instruct AIME 24: DDIM,3232steps/block, CFG33, ST0\.30\.3Q:Jen enters a lottery by picking44distinct numbers fromS=\{1,2,3,⋯,9,10\}\.S=\\\{1,2,3,\\cdots,9,10\\\}\.44numbers are randomly chosen fromS\.S\.She wins a prize if at least two of her numbers were22of the randomly chosen numbers, and wins the grand prize if all four of her numbers were the randomly chosen numbers\. The probability of her winning the grand prize given that she won a prize ismn\\tfrac\{m\}\{n\}wheremmandnnare relatively prime positive integers\. Findm\+nm\+n\.A:The probability of winning the grand prize given that Jen won a prize is calculated as follows\.Jen wins a prize if at least two of her chosen numbers are among the four randomly selected numbers\. The grand prize is won if all four of her chosen numbers are among the four chosen numbers\.The total number of ways to choose four numbers from the setS=\{1,2,3,…,10\}S=\\\{1,2,3,\\dots,10\\\}is\(104\)=210\\binom\{10\}\{4\}=210\.The event that Jen wins a prize occurs when at least two of her four chosen numbers are in the four chosen numbers\. The number of favorable outcomes for winning a prize is found by considering the cases where at least two of her four chosen numbers are included in the four chosen numbers\. This is equivalent to the number of ways to choose the four chosen numbers such that at least two of Jen’s four chosen numbers are in the four chosen numbers\.The number of ways to choose the four chosen numbers with at least two of Jen’s four chosen numbers is calculated by summing the cases for exactly a numbers of Jen’s numbers being included, wherea=2,3,4,4a=2,3,4,4:•Fora=2a=2:\(42\)×\(62\)=6×15=90\\binom\{4\}\{2\}\\times\\binom\{6\}\{2\}=6\\times 15=90•Fora=3a=3:\(43\)×\(61\)=4×6=24\\binom\{4\}\{3\}\\times\\binom\{6\}\{1\}=4\\times 6=24•Fora=4a=4:\(44\)×\(60\)=1×1=1\\binom\{4\}\{4\}\\times\\binom\{6\}\{0\}=1\\times 1=1Summing these gives the total number of outcomes where Jen wins a prize:90\+24\+1=11590\+24\+1=115\.The event that Jen wins the grand prize occurs when all four of her chosen numbers are in the four chosen numbers, which happens only whena=4a=4\. The number of such outcomes is 1\.The conditional probability of winning the grand prize given that she won a prize is the ratio of the number of outcomes for winning the grand prize to the number of outcomes for winning a prize, which is1115\\frac\{1\}\{115\}\.The fraction1115\\frac\{1\}\{115\}is in simplest terms, as 1 and 115 are relatively prime\. Thus,m=1m=1andn=115n=115, andm\+n=116m\+n=116\.116 AIME 25: DDIM,3232steps/block, CFG33, ST0\.30\.3Q:Six pointsA,B,C,D,E,A,B,C,D,E,andFFlie in a straight line in that order\. Suppose thatGGis a point not on the line and thatAC=26,BD=22,CE=31,DF=33,AF=73,CG=40,AC=26,BD=22,CE=31,DF=33,AF=73,CG=40,andDG=30\.DG=30\.Find the area of△BGE\.\\triangle BGE\.A:The pointsAA,BB,CC,DD,EE, andFFlie on a straight line in that order\. Assign coordinates to the points such thatAAisa=0a=0,BBbb,CCcc,DDatdd,EEatee, andFFatff, with0<b<c<d<e<f0<b<c<d<e<f\.Given:•AC=26AC=26, soc=26c=26\.•AF=73AF=73, sof=73f=73\.•BD=22BD=22, sod−b=22d\-b=22\.•CE=31CE=31, soe−c=31e\-c=31, givinge=26\+31=57e=26\+31=57\.•DF=33DF=33, sof−d=33f\-d=33, givingd=73−33=40d=73\-33=40\.•CG=40CG=40andDG=30DG=30, whereGGis not on the line\.Fromd=40d=40andd−b=22d\-b=22, solve forb=40−22=18b=40\-22=18\. Frome=57e=57andf=73f=73, solve forf=73−57=16f=73\-57=16\. Thus, the coordinates areA=0A=0,B=18B=18,C=26C=26,D=40D=40,E=57E=57,F=73F=73\.LetGGbe at\(x,y\)\(x,y\)\. The distances give:•CG=40CG=40:\(x−26\)2\+y2=1600\(x\-26\)^\{2\}\+y^\{2\}=1600\.•DG=30DG=30:\(x−40\)2\+y2=900\(x\-40\)^\{2\}\+y^\{2\}=900\.Subtract the equations:\(x−26\)2\+y2−\[\(x−40\)2\+y2\]=1600−900\(x\-26\)^\{2\}\+y^\{2\}\-\[\(x\-40\)^\{2\}\+y^\{2\}\]=1600\-900\(x2−52x\+676\)−\(x2−80x\+1600\)=700\(x^\{2\}\-52x\+676\)\-\(x^\{2\}\-80x\+1600\)=70028x−924=70028x\-924=70028x=162428x=1624x=162428=4067\.x=\\frac\{1624\}\{28\}=\\frac\{406\}\{7\}\.Substitutex=4067x=\\frac\{406\}\{7\}into\(x−40\)2\+y2=900\(x\-40\)^\{2\}\+y^\{2\}=900:\(4067−40\)2\+y2=900\\left\(\\frac\{406\}\{7\}\-40\\right\)^\{2\}\+y^\{2\}=900\(4067−2807\)2\+y2=900\\left\(\\frac\{406\}\{7\}\-\\frac\{280\}\{7\}\\right\)^\{2\}\+y^\{2\}=900\(1267\)2\+y2=900\\left\(\\frac\{126\}\{7\}\\right\)^\{2\}\+y^\{2\}=900\(1267\)2=182=324\\left\(\\frac\{126\}\{7\}\\right\)^\{2\}=18^\{2\}=324324\+y2=900324\+y^\{2\}=900y2=576y^\{2\}=576y=±24\(since area depends on\|y\|\)\.y=\\pm 24\\hskip 8\.50012pt\(\\text\{since area depends on \}\|y\|\)\.The area of△BGE\\triangle BGEwith verticesB\(18,0\)B\(18,0\),G\(4067,y\)G\\left\(\\frac\{406\}\{7\},y\\right\), andE\(57,0\)E\(57,0\)is found using the formula for the area of a triangle with baseBEBEand height\|y\|\|y\|:•BaseBE=\|57−18=39BE=\|57\-18=39\.•Height is\|y\|=24\|y\|=24\.•Area =12×39×24=12×936=468\\frac\{1\}\{2\}\\times 39\\times 24=\\frac\{1\}\{2\}\\times 936=468\.Alternatively, using the shoelace formula:Area=12\|18\(y−0\)\+4067\(0−0\)\+57\(0−y\)\|=12\|18y−57y\|=12\|−39y\|=12×39\|y\|=392×24=468,\\text\{Area\}=\\frac\{1\}\{2\}\\left\|18\\left\(y\-0\\right\)\+\\frac\{406\}\{7\}\(0\-0\)\+57\(0\-y\)\\right\|=\\frac\{1\}\{2\}\\left\|18y\-57y\\right\|=\\frac\{1\}\{2\}\|\-39y\|=\\frac\{1\}\{2\}\\times 39\|y\|=\\frac\{39\}\{2\}\\times 24=468,since\|y\|=24\|y\|=24\.The base of△BGE\\triangle BGEis\|57−18\|=39\|57\-18\|=39, and the height is\|y\|=24\|y\|=24, so the area is12×39×24=468\\frac\{1\}\{2\}\\times 39\\times 24=468\.The area is consistent regardless of the sign ofyy, and all conditions are satisfied withy=24y=24ory=−24y=\-24\.Thus, the area of△BGE\\triangle BGEis 468\.468 GPQA \(Quantum Mechanics, Hard graduate level\): DDIM,3232steps/block, CFG33, ST0\.30\.3Q:A common approximation made in many\-body nuclear calculations of some interaction \(for example an electroweak interaction\) is to suppose that nucleons in a nucleus behave in the same way as free nucleons\. Which of the following four assumptions jointly imply this “impulse” approximation?1\.The interaction current only interacts with individual nucleons\.2\.The nucleus is transparent apart from the selected nucleon\.3\.The quarks internal to the selected nucleon are non\-relativistic\.4\.The interaction proceeds as if the selected nucleon experiences no binding forces\.A\)1, 2, 3B\)1, 3, 4C\)2, 3, 4D\)1, 2, 4A:The “impulse” approximation in many\-body nuclear calculations assumes that the behavior of a selected nucleon in a nucleus is similar to that of a free nucleon\. This approximation is based on the idea that the interaction on the selected nucleon is not significantly affected by the binding forces from the other particles in the nucleus\.To justify this approximation, the following assumptions must ensure that the interaction on the selected nucleon is not influenced by the binding forces from the other nucleons:•Assumption 1: The interaction current only interacts with individual nucleons\. This ensures that the interaction does not depend on the collective state of the nucleus but rather only on the individual nucleons\.•Assumption 2: The nucleus is transparent apart from the selected nucleon\. This means that the interaction on the selected nucleon is not altered by the presence of the other nucleons, allowing it to be treated as a free nucleon\.•Assumption 4: The interaction proceeds as if the selected nucleon experiences no binding forces\. This directly states that the binding forces from other nucleons are ignored, allowing the selected nucleon to behave like a free nucleon\.•Assumption 3\(“quarks internal to the selected nucleon are non\-relativistic”\) is not relevant to this approximation, as it pertains to the internal structure of the nucleon, not the external environment in the nucleus\.Thus, the assumptions that jointly imply the impulse approximation are1, 2, and 4\.Answer: D
相似文章
@NVIDIAAI: 大多数语言模型一次只生成一个token。我们刚刚发布了Nemotron-Labs-Diffusion，一个扩散语言模型系列…
NVIDIA发布了Nemotron-Labs-Diffusion，这是一个扩散语言模型系列，可以并行生成多个token，从而实现更快的推理和更好的GPU利用率，模型规模从3B到14B，包括视觉语言变体。
LangFlow：连续扩散在语言建模中可与离散扩散相媲美
LangFlow提出了首个可与离散扩散方法相媲美的连续扩散语言模型，挑战了长期以来认为连续扩散在语言建模中劣于离散扩散的观点。该工作引入了基于最优Gumbel噪声调度等关键要素，并展示了与离散扩散基线相比具有竞争力的困惑度和迁移学习性能。
Hierarchical Continuous Diffusion Language Models
论文提出了 Hierarchical Continuous Diffusion Language Models（HC-DLM），将离散 token 生成与连续隐变量轨迹耦合到单个去噪过程中，该过程的训练目标源自 token 似然的变分下界。在 Sudoku、Countdown 和 LM1B 任务上，HC-DLM 优于离散与连续扩散基线模型。
令牌时间连续扩散用于语言建模
本文介绍了令牌时间连续扩散（Token Time Continuous Diffusion，简称TTCD），这是一种新型扩散语言模型，它在连续空间中运行，采用每个令牌的时间步，在条件生成和数独求解方面以高加速比优于离散模型。
用于优化离散扩散语言模型的漂移目标
本文提出TokenDrift，一种漂移目标方法，通过将分类预测提升至连续语义空间进行反对称漂移，从而优化离散扩散语言模型。在固定去噪步数下，该方法显著提升了生成质量。