CaRE Compute-aware Remasking Evaluation Protocol for Masked Diffusion Language Models

# Compute-aware Remasking Evaluation Protocol for Masked Diffusion Language Models
Source: [https://arxiv.org/html/2607.24763](https://arxiv.org/html/2607.24763)
Yash Shah1 Arizona State University yshah124@asu\.edu&Abhijit Chakraborty211footnotemark:1 MongoDB abhijit\.chakraborty@mongodb\.com&Vivek Gupta1 Arizona State University vgupt140@asu\.edu

###### Abstract

Masked diffusion language models are outpacing the evaluation standards needed to reliably interpret their progress\. MDLMs are now competitive with autoregressive LMs, yet seven recent remasking papers evaluate under incompatible settings, rendering their strategy rankings largely incomparable\. Existing studies vary nominal step counts, metrics, and sampling temperatures, but no work jointly controls these factors, leaving open whether reported gains reflect algorithms or evaluation artifacts\. We present Care111https://anonymous\.4open\.science/w/CaRE\-846C/, We release the protocol, implementation, and leaderboard to ensure future remasking claims are reproducible and robust\., a compute\-aware evaluation framework for auditing masked\-discrete MDLM remasking by standardizing actual NFE, multi\-metric reporting, and stochasticity\. We release the protocol, implementation, and leaderboard to ensure future remasking claims are reproducible and robust\. Applied to 7 remasking strategies across LLaDA\-8B\-Base and Dream\-7B\-Base, 4 stochasticity levels, and 3 step budgets on OpenWebText and LM1B, Careshows that \(i\) temperature explains the majority of MAUVE variance \(η2=0\.91\\eta^\{2\}\{=\}0\.91\), \(ii\) compute\-matched comparisons reverse several published strategy rankings, and \(iii\) informed remasking and stochastic unmasking are in tension, with high\-entropy remasking reducing MAUVE by0\.2960\.296at 256 steps andunmask\_temp=0\.25\{=\}0\.25\(p=0\.020p\{=\}0\.020\)\. A Careleaderboard covers 12 open\-weight MDLMs \(150M–8B parameters\), where the interaction direction holds across architectures and scales\. These results reveal that current MDLM evaluations can systematically conflate algorithmic improvements with hidden choices of compute and stochasticity, and Carereleases a seven\-point protocol and implementation to make future remasking claims reproducible\.

## 1Introduction

For over a decade, autoregressive language models have treated left\-to\-right generation less as a design choice than as a default\. Masked diffusion language models \(MDLMs\) challenge this assumption by reformulating generation as iterative parallel denoising over the full sequence\. The emergence of large\-scale MDLMs such as LLaDA\-8B\[[18](https://arxiv.org/html/2607.24763#bib.bib2)\]and Dream\-7B\[[30](https://arxiv.org/html/2607.24763#bib.bib3)\]shows that this paradigm has moved from theoretical curiosity to a competitive alternative to autoregressive LMs\.Throughout this paper, “remasking” refers to mask re\-introduction during denoising in*masked discrete\-token*diffusion language models; continuous\-state and flow\-based DLMs use a different inference paradigm and are out of scope \(§[A](https://arxiv.org/html/2607.24763#A1)\)\.

![Refer to caption](https://arxiv.org/html/2607.24763v1/x1.png)Figure 1:The evaluation gap: seven remasking papers evaluate under different conditions, producing contradictory rankings\. Three confounds \(compute, metric, stochasticity\) each independently reverse conclusions\. Controlling all three jointly reveals their interaction \(p=0\.020p\{=\}0\.020;nonevs\.high\_entropy, LLaDA\-8B\-Base, OWT, 256 steps,t=0\.25t\{=\}0\.25, 3 seeds, pairedtt\-test\)\.#### The evaluation crisis\.

In the last six months, seven remasking strategies\[[28](https://arxiv.org/html/2607.24763#bib.bib4),[32](https://arxiv.org/html/2607.24763#bib.bib5),[13](https://arxiv.org/html/2607.24763#bib.bib6),[9](https://arxiv.org/html/2607.24763#bib.bib7),[11](https://arxiv.org/html/2607.24763#bib.bib8),[1](https://arxiv.org/html/2607.24763#bib.bib9),[12](https://arxiv.org/html/2607.24763#bib.bib10)\]have been proposed to improve MDLM inference by selectively re\-introducing masks during denoising\. Each is evaluated under different step budgets, metrics, and sampling temperatures, producing contradictory rankings where the same strategy is reported as best in one paper and worst in another \(Figure[1](https://arxiv.org/html/2607.24763#S1.F1)\)\. The image generation community faced a similar situation before Clean\-FID\[[20](https://arxiv.org/html/2607.24763#bib.bib45)\]standardized evaluation, and machine translation before SacreBLEU\[[23](https://arxiv.org/html/2607.24763#bib.bib25)\]enforced tokenizer consistency\. In both cases, part of the apparent progress was later attributed to evaluation inconsistency rather than algorithmic improvement\. MDLMs are now at a similar inflection point\. The hidden interaction\.Individual confounds are documented in prior work: PPL–MAUVE disagreement\[[22](https://arxiv.org/html/2607.24763#bib.bib24)\], temperature sensitivity\[[3](https://arxiv.org/html/2607.24763#bib.bib22),[8](https://arxiv.org/html/2607.24763#bib.bib23)\], NFE formalization\[[19](https://arxiv.org/html/2607.24763#bib.bib20)\], and decoding\-hyperparameter dominance\[[10](https://arxiv.org/html/2607.24763#bib.bib29)\]\. The community has flagged the problem\[[21](https://arxiv.org/html/2607.24763#bib.bib15),[34](https://arxiv.org/html/2607.24763#bib.bib12),[29](https://arxiv.org/html/2607.24763#bib.bib13),[24](https://arxiv.org/html/2607.24763#bib.bib11)\]but without joint control\. Carecombines these strands and reveals that*informed remasking and stochastic unmasking are in consistent tension*\. Stochastic unmasking is the dominant diversity lever, improving MAUVE from0\.660\.66to0\.980\.98with a single temperature change—a gain of0\.320\.32that exceeds the maximum gap between any two strategies at the same temperature\. Yet at higher compute budgets, high\-entropy remasking collapses this benefit by0\.2960\.296MAUVE points \(p=0\.020p\{=\}0\.020\) through persistent token churn that narrows distributional coverage while preserving surface\-level diversity\. To our knowledge, no concurrent MDLM evaluation surfaces this interaction, because detecting it requires controlling all three confounds simultaneously\. Contributions\.\(1\) Interaction finding: informed remasking and stochastic unmasking are in consistent tension \(p=0\.020p\{=\}0\.020\); a three\-way ANOVA shows temperature dominates \(η2=0\.91\\eta^\{2\}\{=\}0\.91\) with significant strategy×\\timestemperature interaction \(η2=0\.47\\eta^\{2\}\{=\}0\.47\)\. Unlike prior frameworks that control at most two confounds \(Table[2](https://arxiv.org/html/2607.24763#S3.T2)\), detecting this interaction requires compute, metric, and stochasticity to be controlled*jointly*, which is why it has not appeared in earlier MDLM evaluation work\.\(2\) Compute\-matched reversal: strategy rankings flip when actual NFE is equated, on both unconditional generation and HumanEval\.\(3\) Seven\-point protocol and leaderboard: a reproducible standard validated on OpenWebText and LM1B, with a public leaderboard over twelve open\-weight MDLMs\. A frontier overlay \(§[4\.5](https://arxiv.org/html/2607.24763#S4.SS5)\) shows the interaction is invisible in PPL\-entropy space, positioning Careas complementary to Generative Frontiers\[[24](https://arxiv.org/html/2607.24763#bib.bib11)\]\.

## 2The CareFramework and Protocol

Careprovides a unifiedRemaskingSamplerwith four design principles: \(1\) automatic NFE tracking per generation; \(2\) four default metrics \(PPL via Llama\-3\-8B, MAUVE, Self\-BLEU, Distinct\-3\); \(3\)unmask\_tempas a required parameter with sweep support; \(4\) character\-level English filter \(ASCII≥\\geq0\.85, CJK≤\\leq0\.02\)\. Adding a model takes∼\\sim50 LOC; adding a strategy takes∼\\sim20 LOC\. Appendix[C\.2](https://arxiv.org/html/2607.24763#A3.SS2)gives the completeHighEntropyimplementation as shipped in the released codebase, showing that a full remasking strategy fits in roughly 20 lines\. Prompt formats for unconditional generation, HumanEval, and GSM8K follow standardlm\-evalconfigurations; representative templates appear in Appendix[C\.1](https://arxiv.org/html/2607.24763#A3.SS1)\. The full experimental scope spans 12 open\-weight MDLMs \(150M–8B parameters\), 5 evaluation settings, and 7 remasking strategies; details appear in Section[4](https://arxiv.org/html/2607.24763#S4)\.

Table 1:Seven\-point Careprotocol\. Each omission independently reverses published conclusions\.
## 3Related Work

#### Remasking strategies\.

ReMDM\[[28](https://arxiv.org/html/2607.24763#bib.bib4)\]identifies PPL hacking and advocates MAUVE, but applies nucleus sampling as an on/off toggle without isolating stochasticity\. CoRe\[[32](https://arxiv.org/html/2607.24763#bib.bib5)\]uses compute\-matched baselines for a single strategy; Prism\[[2](https://arxiv.org/html/2607.24763#bib.bib19)\]scales test\-time compute via hierarchical NFE; UnMaskFork\[[17](https://arxiv.org/html/2607.24763#bib.bib46)\]finds temperature\-based scaling degrades MDLMs\. Carecodifies these ad\-hoc practices and reveals the interaction they cannot detect\. What Careadds over existing frameworks\.Existing frameworks each control at most two of the three confounds Careidentifies\.HELM\[[15](https://arxiv.org/html/2607.24763#bib.bib26)\]is multi\-metric but targets autoregressive LLMs with no notion of NFE or remasking\.Generative Frontiers\[[24](https://arxiv.org/html/2607.24763#bib.bib11)\]treats temperature as a curve parameter and decomposes generation into PPL–entropy space, but does not compare remasking strategies and leaves NFE implicit \(Section[4\.5](https://arxiv.org/html/2607.24763#S4.SS5)\)\.dLLM\[[34](https://arxiv.org/html/2607.24763#bib.bib12)\]flags hyperparameter sensitivity but provides no remasking strategy\-comparison protocol\.ReMDM\[[28](https://arxiv.org/html/2607.24763#bib.bib4)\]controls NFE and advocates MAUVE but conflates stochasticity with nucleus sampling on/off, leaving rankings entangled with the variable Careidentifies as dominant \(η2=0\.91\\eta^\{2\}\{=\}0\.91\)\. Careis, to our knowledge, the first framework to jointly control compute, metric, and stochasticity in a single runnable protocol \(Table[2](https://arxiv.org/html/2607.24763#S3.T2)\) and to expose their interaction as a measurable object; DARE\[[29](https://arxiv.org/html/2607.24763#bib.bib13)\]and d3LLM\[[25](https://arxiv.org/html/2607.24763#bib.bib14)\]are complementary post\-training and parallelism frameworks\.

Table 2:Framework comparison\. Careis the only one to jointly control all three confounds and identify their interaction\.Evaluation methodology\.Clean\-FID\[[20](https://arxiv.org/html/2607.24763#bib.bib45)\]and SacreBLEU\[[23](https://arxiv.org/html/2607.24763#bib.bib25)\]are Care’s direct analogs\. LM Eval Harness\[[6](https://arxiv.org/html/2607.24763#bib.bib27)\]standardizes broader LLM evaluation; Zheng et al\.\[[33](https://arxiv.org/html/2607.24763#bib.bib31)\]identify fp32 truncation as implicit temperature in MDMs; Holtzman et al\.\[[10](https://arxiv.org/html/2607.24763#bib.bib29)\]show decoding hyperparameters dominate AR generation\.

## 4Validation Experiments

#### Evaluation scope\.

Experiments span12 open\-weight MDLMs\(150M–8B parameters; dense, MoE, and Soft\-Masked variants\) and5 evaluation settings: \(1\) unconditional generation on OpenWebText\[[7](https://arxiv.org/html/2607.24763#bib.bib32)\]and LM1B\[[4](https://arxiv.org/html/2607.24763#bib.bib33)\]\(MAUVE, PPL, Self\-BLEU, Distinct\-3\); \(2\) code generation on HumanEval \(pass@1\); \(3–5\) arithmetic, commonsense, and multi\-step reasoning on GSM8K\[[5](https://arxiv.org/html/2607.24763#bib.bib44)\], HellaSwag\[[31](https://arxiv.org/html/2607.24763#bib.bib34)\], and BBH\[[27](https://arxiv.org/html/2607.24763#bib.bib49)\]—evaluated via log\-likelihood scoring as sampling\-invariant sanity checks\. Additional experiments and setup has been dicussed in Appendix[C](https://arxiv.org/html/2607.24763#A3)\. Strategies cover all remasking methods supported by LLaDA and Dream \(Appendix[B](https://arxiv.org/html/2607.24763#A2)\)\.

### 4\.1Confound 1: Nominal Steps≠\\neqActual Compute

If strategies are compared at the same nominal step count without controlling actual NFE, any quality difference could reflect a compute advantage rather than an algorithmic one\.

At 256 nominal steps, actual NFE ranges from 128 \(none\) to 513 \(running\_conf\.\)—a4×4\\timesdifference \(Table[3](https://arxiv.org/html/2607.24763#S4.T3)\)\. Fornone, which never reintroduces masks, each of theL=128L\{=\}128output positions is unmasked exactly once; the model therefore requires exactly 128 forward passes regardless of any nominal step count above 128, explaining the factor\-of\-two between the nominal budget \(256\) and actual NFE \(128\)\. Strategies that remask already\-decided tokens incur additional forward passes proportional to their remasking rate\.

Table 3:Actual NFE at 256 nominal steps, LLaDA\-8B\-Base\.Compute\-matched quality\.Fixing actual NFE at\{128,256,512\}\\\{128,256,512\\\}shows thatnonedominates MAUVE at every budget, and the gap widens with more compute \(Table[4](https://arxiv.org/html/2607.24763#S4.T4)\)\.

Table 4:Compute\-matched quality \(LLaDA\-8B\-Base, OWT, prefix=64, gen=128,t=0\.0t\{=\}0\.0, English\-filtered; PPL via Llama\-3\-8B\)\.nonerows use 3\-seed mean \(seeds 1–3\);randomandhigh\_entropyrows are single\-seed \(seed 1\)\. These are separate experimental runs from Table[5](https://arxiv.org/html/2607.24763#S4.T5): strategies are run at different nominal step counts to equate actual NFE across configurations\.
### 4\.2Confound 2: PPL and MAUVE Prefer Different Strategies

Even with compute matched, a single metric can award different winners—confirming that metric choice is an independent confound, not a proxy for the same underlying quality\.

At 128 deterministic steps,high\_entropyachieves the best MAUVE \(0\.612\) but not the best PPL;noneachieves the best PPL but not the best MAUVE \(Table[5](https://arxiv.org/html/2607.24763#S4.T5)\)\. Rankings flip across step budgets\. The GPT\-2\-XL AR reference achieves MAUVE=0\.742\{=\}0\.742on OWT, calibrating absolute scale\. Figure[2](https://arxiv.org/html/2607.24763#S4.F2)shows that stochastic configurations dominate the Pareto frontier\.

Table 5:LLaDA\-8B\-Base, OWT, prefix=64, gen=128, single seed, deterministic \(t=0\.0t\{=\}0\.0\), English\-filtered\. PPL via Llama\-3\-8B\.Blue=best MAUVE,green=best PPL\. GPT\-2\-XL AR: MAUVE=0\.742\{=\}0\.742\.![Refer to caption](https://arxiv.org/html/2607.24763v1/x2.png)Figure 2:LLaDA\-8B\-Base, OWT\. MAUVE vs PPL across step budgets \(64/128/256\) and temperatures\. Open markers = stochastic \(t\>0t\>0\); filled = deterministic \(t=0\.0t\{=\}0\.0\)\. Stochastic configurations dominate the Pareto frontier at every PPL level\.
### 4\.3Confound 3: Stochastic Unmasking Dominates Strategy Choice

Stochasticity is the most overlooked confound: a single temperature parameter change produces quality shifts that exceed the entire range of strategy differences, rendering any strategy comparison at uncontrolled temperature uninformative\.

Varyingunmask\_tempfrom 0\.0 to 0\.1 on LLaDA improves MAUVE from 0\.66 to 0\.98 while PPL degrades from 9\.6 to 17\.6 \(Table[6](https://arxiv.org/html/2607.24763#S4.T6)\)\. At 128 and 256 steps, this0\.320\.32gain exceeds the maximum gap between any two strategies at the same temperature \(Table[5](https://arxiv.org/html/2607.24763#S4.T5)\); at 64 steps the strategy gap reaches 0\.347 \(nonevs\.high\_entropy\), but the temperature effect remains the dominant variance source in either case \(η2=0\.91\\eta^\{2\}\{=\}0\.91, Table[8](https://arxiv.org/html/2607.24763#S4.T8)\)\. Confirmed over 3 seeds att=0\.25t\{=\}0\.25:0\.948±0\.0210\.948\{\\pm\}0\.021vs\.0\.652±0\.0250\.652\{\\pm\}0\.025\(p=0\.020p\{=\}0\.020\)\.

Table 6:MAUVE across temperatures \(LLaDA\-8B\-Base, OWT, prefix=64, gen=128, English\-filtered; MAUVE via GPT\-2 XL backbone,K=500K\{=\}500\)\.±\\pm= 3\-seed mean/SE \(seeds 1–3, pairedtt\-test\); cells without±\\pmare single\-seed \(seed 1\)\. The bolded cell \(none@256,t=0\.25t\{=\}0\.25\) marks the headline interaction test\.Red=collapse\.MAUVE saturation considerations\.The MAUVE values reached at moderate stochasticity \(0\.948 fornone@256 att=0\.25t\{=\}0\.25\) exceed our GPT\-2\-XL AR reference \(0\.742\)\. Two interpretations are consistent: \(a\) stochastic MDLM sampling produces a genuinely better distributional match than AR nucleus sampling, or \(b\) MAUVE partially saturates at the high end\. The interaction finding compares MDLM configurations against each other rather than an absolute scale, so it is robust to either interpretation: even under partial saturation, the0\.2960\.296gap reflects a real difference in distributional behavior under controlled conditions\.

![Refer to caption](https://arxiv.org/html/2607.24763v1/x3.png)Figure 3:MAUVE vs actual NFE across stochasticity levels \(LLaDA\-8B\-Base, OWT, English\-filtered\)\. Att=0\.0t\{=\}0\.0all strategies cluster low \(<<0\.7\)\. Att≥0\.1t\\geq 0\.1,nonerises to0\.9\+0\.9\{\+\};randomshows moderate improvement \(0\.8–0\.98 depending on temperature\);high\_entropycollapses at high NFE despite moderate gains at lower compute\.
### 4\.4The Interaction That Only Joint Control Reveals

Whilenonemaintains MAUVE\>\>0\.92 at all temperatures,high\_entropycollapses to 0\.56–0\.65 at 256 steps witht≥0\.25t\\geq 0\.25\. At 256 steps,t=0\.25t\{=\}0\.25: gap=0\.296±0\.042\{=\}0\.296\\pm 0\.042\(paired SE\),t​\(2\)=7\.06t\(2\)\{=\}7\.06,p=0\.020p\{=\}0\.020\(Table[7](https://arxiv.org/html/2607.24763#S4.T7)\)\.

In the main\-text experiments we therefore focus on two remasking configurations: a no\-remasking baseline \(none\) and an aggressive informed strategy \(high\_entropy\)\. These span the extremes of the compute–churn trade\-off under the Careprotocol:noneuses the least NFE and leaves tokens fixed once denoised, whilehigh\_entropyconsumes the most NFE and remasks the highest\-entropy positions at every step\. Intermediate strategies fall between these endpoints and appear in the full seven\-strategy comparison \(Appendix[C\.10](https://arxiv.org/html/2607.24763#A3.SS10)\); we omit them from most main\-text plots for clarity and to highlight the trade\-off between the two practically relevant extremes\.

Table 7:MAUVE gap \(none−\-high\_entropy\), LLaDA\-8B\-Base, OWT\. Cells with±\\pmuse paired SE on 3\-seed differences \(seeds 1–3\); single\-seed cells use seed 1\. Significant at 256 steps,t=0\.25t\{=\}0\.25\(p=0\.020p\{=\}0\.020, pairedtt\-test,d​f=2df\{=\}2\)\.Formal decomposition\.A three\-way ANOVA confirms the finding \(Table[8](https://arxiv.org/html/2607.24763#S4.T8)\): temperature \(η2=0\.91\\eta^\{2\}\{=\}0\.91\) dominates strategy \(η2=0\.80\\eta^\{2\}\{=\}0\.80\), and strategy×\\timestemperature is significant \(p=0\.002p\{=\}0\.002,η2=0\.47\\eta^\{2\}\{=\}0\.47\)\. A 95% percentile bootstrap CI for thet=0\.25t\{=\}0\.25, 256\-step gap \(10k resamples over 3 paired seeds\) is\[0\.243,0\.356\]\[0\.243,0\.356\], excluding zero\.

Table 8:Three\-way ANOVA on MAUVE \(LLaDA\-8B\-Base, seeds 1–3;nonevs\.high\_entropy, 4 temperatures×\\times3 step budgets;n=72n\{=\}72seed–configuration cells, Residuald​f=48df\{=\}48\)\.This interaction is the paper’s central finding: it is structurally undetectable unless all three confounds are controlled simultaneously, which is why it has not appeared in prior MDLM evaluation work\.

Mechanism\.high\_entropyreplaces 4\.3% of tokens per step \(vs\. 0% fornone\), achieving only 2,522 stable token\-step cells vs\. 8,129—3\.2×3\.2\\timesless stability\. Persistent churn pushes samples toward the model’s mode, collapsing distributional coverage while preserving surface diversity \(Fig\.[4](https://arxiv.org/html/2607.24763#S4.F4)\)\.

![Refer to caption](https://arxiv.org/html/2607.24763v1/x4.png)Figure 4:Token stability heatmap \(LLaDA\-8B\-Base, 128 steps,t=0\.25t\{=\}0\.25\)\. Blue=stable, white=masked, orange=newly unmasked, red=remasked\.none\(left\) settles tokens early;high\_entropy\(right\) exhibits persistent late\-stage churn\.Dose\-response ablation\.Sweeping the remask fraction at fixed strategy and temperature \(Table[9](https://arxiv.org/html/2607.24763#S4.T9)\) shows MAUVE degrades monotonically with churn volume; the dose\-response is consistent with churn driving the collapse, though the ablation is single\-seed and a matched\-volume comparison ofrandomvs\.high\_entropyis left for future work\.

Table 9:Dose\-response ablation \(LLaDA\-8B\-Base, OWT, single seed\): increasing the remask fraction degrades MAUVE while PPL and Distinct\-3 stay in narrow ranges \(high\_entropy@256,t=0\.25t\{=\}0\.25\)\.
### 4\.5Generative Frontier Overlay: What Frontier Analysis Misses

A practitioner consulting only frontier plots at this operating point would concludehigh\_entropyandnoneare interchangeable—and would deploy the strategy that costs3\.4×3\.4\\timesmore compute to produce worse distributional coverage\. Generative Frontiers\[[24](https://arxiv.org/html/2607.24763#bib.bib11)\]treats temperature as a curve parameter\. Plotting Care’s data in this format \(Fig\.[5](https://arxiv.org/html/2607.24763#S4.F5), left\) reveals a blind spot: att=0\.25t\{=\}0\.25, 256 steps,noneandhigh\_entropyoccupy overlapping regions of the PPL–MAUVE plane\. The interaction surfaces only when MAUVE is decomposed by strategy×\\timestemperature \(Fig\.[5](https://arxiv.org/html/2607.24763#S4.F5), right\), where the0\.2960\.296gap is visually unambiguous\.

![Refer to caption](https://arxiv.org/html/2607.24763v1/x5.png)Figure 5:LLaDA\-8B\-Base, OWT\. Generative frontier overlay\.*Left*: PPL–MAUVE frontier curves\. Att=0\.25t\{=\}0\.25, 256 steps,noneandhigh\_entropyoverlap\.*Right*: MAUVE by strategy shows the gap clearly \(p=0\.020p\{=\}0\.020\)\.
### 4\.6Generalization and Downstream Evaluation

Cross\-architecture \(LLaDA\-MoE\)\.Both findings replicate \(Table[10](https://arxiv.org/html/2607.24763#S4.T10)\):none@256jumps from0\.4470\.447\(t=0\.0t\{=\}0\.0\) to0\.9540\.954\(t=0\.1t\{=\}0\.1\),\+0\.507\+0\.507; interaction gap=0\.161\{=\}0\.161att=0\.1t\{=\}0\.1, 256 steps\.

Table 10:LLaDA\-MoE\-7B\-A1B \(∼\{\\sim\}1B active params\), OWT, prefix=64, gen=128, single seed, English\-filtered\. Rows show 128\- and 256\-step configurations att=0\.0t\{=\}0\.0andt=0\.1t\{=\}0\.1\.Cross\-family \(Dream\)\.Dream\-7B\-Base confirms metric and budget dependence \(Appendix Table[15](https://arxiv.org/html/2607.24763#A3.T15)\)\. Figure[6](https://arxiv.org/html/2607.24763#S4.F6)visualizes the full interaction: stochastic unmasking liftsnonebut collapseshigh\_entropyin MAUVE, while PPL shows the reverse\.

![Refer to caption](https://arxiv.org/html/2607.24763v1/x6.png)Figure 6:Stochasticity interaction, LLaDA\-8B\-Base, OWT, 256 steps\. \(a\) MAUVE: stochastic unmasking \(dashed\) liftsnonebut collapseshigh\_entropy\. \(b\) PPL: stochastic unmasking hurts fluency; informed remasking compensates\.Language filter and LM1B\.high\_entropyatt=0\.5t\{=\}0\.5, 256 steps rejects 10% of OWT samples vs\. 2\.4% fornone; LM1B achieves98−99%98\{\-\}99\\%acceptance throughout, validating the filter detects real multilingual leakage rather than noise \(Appendix Tables[19](https://arxiv.org/html/2607.24763#A3.T19),[20](https://arxiv.org/html/2607.24763#A3.T20)\)\.

Downstream—HumanEval\.The NFE confound extends to code generation \(Table[11](https://arxiv.org/html/2607.24763#S4.T11)\)\. At nominal steps,high\_entropyappears to outperformnone\(31\.7 vs\. 28\.3 pass@1\), but uses3\.4×3\.4\\timesmore compute\. At matched NFE=437\{=\}437,nonematches or exceeds all strategies\. The reported±\\pm0\.3 SEs reflect seed variance only; benchmark\-level binomial uncertainty atn=100n\{=\}100is≈±4\.7\{\\approx\}\\pm 4\.7pp, so the32\.032\.0vs\.31\.731\.7within\-block difference is within sampling noise\. The substantive finding is the*nominal\-vs\-compute\-matched reversal*\(block\-to\-block contrast\)\.

Table 11:HumanEval pass@1 \(%\), LLaDA\-8B\-Base, via lm\-eval\[[6](https://arxiv.org/html/2607.24763#bib.bib27)\],n=100n\{=\}100, 3\-seed mean±\\pmSE \(seed variance only; benchmark\-level binomial SE≈4\.7\{\\approx\}4\.7pp\)\.Other benchmarks\.HellaSwag\[[31](https://arxiv.org/html/2607.24763#bib.bib34)\]\(70\.7%70\.7\\%; reported:70\.5%70\.5\\%\) and BBH\[[27](https://arxiv.org/html/2607.24763#bib.bib49)\]\(45\.5%45\.5\\%; reported:45\.045\.0–49\.7%49\.7\\%\) are evaluated via log\-likelihood scoring rather than sampling, so they are independent ofunmask\_temp, step budget, and remasking strategy by construction\. Their inclusion serves as a sanity check: stable scores across all configurations confirm that Care’s manipulations affect only sampling behavior, not the underlying model’s parametric knowledge\. GSM8K\[[5](https://arxiv.org/html/2607.24763#bib.bib44)\]\(n=200n\{=\}200\) yields5555–64%64\\%accuracy across configurations, with no significant differences\.

### 4\.7The CareLeaderboard

Table[12](https://arxiv.org/html/2607.24763#S4.T12)reports the leaderboard for six primary open\-weight MDLMs at 256 nominal steps under the seven\-point protocol\. The interaction gap quantifies how severely each model is affected by the remasking×\\timesstochasticity confound\.

Table 12:CareLeaderboard, Tier 1 \(256 nominal steps, English\-filtered, OWT prefix=64, gen=128; PPL via Llama\-3\-8B; MAUVE via GPT\-2 XL backbone,K=500K\{=\}500; 3\-seed mean except where noted; HE = HumanEval pass@1 \(%\)\)\. Tier 2 entries appear in Appendix Table[21](https://arxiv.org/html/2607.24763#A3.T21)\.†MAUVE att=0\.25t\{=\}0\.25except where noted\.∗MoE att=0\.1t\{=\}0\.1\(highest\-stochasticity setting tested for this architecture\)\.‡Dream usesentropystrategy in place ofhigh\_entropy\.

Three observations: \(i\) the gap is positive for*every*Tier 1 model, meaningnonebeatshigh\_entropyunder matched stochastic unmasking across architectures, scales, and post\-training regimes; \(ii\) the gap’s*magnitude*spans nearly an order of magnitude \(∼\\sim9×\\times, from 0\.034 for Dream\-7B\-Base to 0\.296 for LLaDA\-8B\-Base\), so the confound’s severity is model\-dependent; \(iii\) Appendix Table[21](https://arxiv.org/html/2607.24763#A3.T21)extends to six Tier 2 entries spanning 0\.5B–8B parameters and training variants, where the interaction direction holds across all twelve\.

## 5Conclusion

Masked diffusion language models advance faster than evaluation practices\. We introduced Care, a compute\-aware framework with NFE, multi\-metric reporting, and stochasticity as controlled variables\. Applied to LLaDA and Dream on OpenWebText and LM1B, Careshows stochasticity dominates strategy effects \(η2=0\.91\\eta^\{2\}\{=\}0\.91\), rankings reverse under compute\-matched comparisons, and remasking interacts with stochastic unmasking, hiding or inverting gains \(p=0\.020p\{=\}0\.020\)\. A leaderboard of twelve MDLMs confirms stable interaction across architectures\.

## References

- \[1\]H\. Asano, T\. Kozuno, K\. Saito, and Y\. Baba\(2026\)Where\-to\-unmask: ground\-truth\-guided unmasking order learning for masked diffusion language models\.arXiv preprint arXiv:2602\.09501\.Cited by:[Appendix A](https://arxiv.org/html/2607.24763#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.24763#S1.SS0.SSS0.Px1.p1.9)\.
- \[2\]J\. Bai, Y\. Li, Y\. Zhu, Y\. Xin, Q\. Shi, A\. Feng, X\. Liu, M\. Tao, J\. Xue, X\. Li,et al\.\(2026\)Prism: efficient test\-time scaling via hierarchical search and self\-verification for discrete diffusion language models\.arXiv preprint arXiv:2602\.01842\.Cited by:[Appendix A](https://arxiv.org/html/2607.24763#A1.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2607.24763#S3.SS0.SSS0.Px1.p1.1)\.
- \[3\]H\. Chang, H\. Zhang, L\. Jiang, C\. Liu, and W\. T\. Freeman\(2022\)Maskgit: masked generative image transformer\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 11315–11325\.Cited by:[§1](https://arxiv.org/html/2607.24763#S1.SS0.SSS0.Px1.p1.9)\.
- \[4\]C\. Chelba, T\. Mikolov, M\. Schuster, Q\. Ge, T\. Brants, P\. T\. Koehn, and T\. Robinson\(2013\)One billion word benchmark for measuring progress in statistical language modeling\.InInterspeech,External Links:[Link](https://api.semanticscholar.org/CorpusID:14136307)Cited by:[§4](https://arxiv.org/html/2607.24763#S4.SS0.SSS0.Px1.p1.1)\.
- \[5\]K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4](https://arxiv.org/html/2607.24763#S4.SS0.SSS0.Px1.p1.1),[§4\.6](https://arxiv.org/html/2607.24763#S4.SS6.p5.8)\.
- \[6\]L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. L\. Noac’h,et al\.\(2023\)A framework for few\-shot language model evaluation\.Zenodo10256836\.External Links:[Link](https://zenodo.org/records/10256836)Cited by:[§C\.1](https://arxiv.org/html/2607.24763#A3.SS1.p3.1),[§3](https://arxiv.org/html/2607.24763#S3.SS0.SSS0.Px1.p2.1),[Table 11](https://arxiv.org/html/2607.24763#S4.T11)\.
- \[7\]A\. Gokaslan and V\. Cohen\(2019\)OpenWebText corpus\.Note:[http://Skylion007\.github\.io/OpenWebTextCorpus](http://skylion007.github.io/OpenWebTextCorpus)Cited by:[§4](https://arxiv.org/html/2607.24763#S4.SS0.SSS0.Px1.p1.1)\.
- \[8\]S\. Hayakawa, Y\. Takida, M\. Imaizumi, H\. Wakaki, and Y\. Mitsufuji\(2025\)Demystifying maskgit sampler and beyond: adaptive order selection in masked diffusion\.arXiv preprint arXiv:2510\.04525\.Cited by:[§1](https://arxiv.org/html/2607.24763#S1.SS0.SSS0.Px1.p1.9)\.
- \[9\]H\. He, K\. Renz, Y\. Cao, and A\. Geiger\(2025\)Mdpo: overcoming the training\-inference divide of masked diffusion language models\.arXiv preprint arXiv:2508\.13148\.Cited by:[Appendix A](https://arxiv.org/html/2607.24763#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.24763#S1.SS0.SSS0.Px1.p1.9)\.
- \[10\]A\. Holtzman, J\. Buys, L\. Du, M\. Forbes, and Y\. Choi\(2019\)The curious case of neural text degeneration\.ArXivabs/1904\.09751\.External Links:[Link](https://api.semanticscholar.org/CorpusID:127986954)Cited by:[§1](https://arxiv.org/html/2607.24763#S1.SS0.SSS0.Px1.p1.9),[§3](https://arxiv.org/html/2607.24763#S3.SS0.SSS0.Px1.p2.1)\.
- \[11\]Z\. Huang, Y\. Wang, Z\. Chen, and G\. Qi\(2025\)Don’t settle too early: self\-reflective remasking for diffusion language models\.arXiv preprint arXiv:2509\.23653\.Cited by:[Appendix A](https://arxiv.org/html/2607.24763#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.24763#S1.SS0.SSS0.Px1.p1.9)\.
- \[12\]J\. Kim, J\. Geuter, D\. Alvarez\-Melis, S\. Kakade, and S\. Chen\(2026\)Stop training for the worst: progressive unmasking accelerates masked diffusion training\.arXiv preprint arXiv:2602\.10314\.Cited by:[Appendix A](https://arxiv.org/html/2607.24763#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.24763#S1.SS0.SSS0.Px1.p1.9)\.
- \[13\]J\. Kim, S\. Kim, T\. Lee, D\. Z\. Pan, H\. Kim, S\. Kakade, and S\. Chen\(2025\)Fine\-tuning masked diffusion for provable self\-correction\.arXiv preprint arXiv:2510\.01384\.Cited by:[§1](https://arxiv.org/html/2607.24763#S1.SS0.SSS0.Px1.p1.9)\.
- \[14\]J\. Li, M\. Galley, C\. Brockett, J\. Gao, and W\. B\. Dolan\(2016\)A diversity\-promoting objective function for neural conversation models\.InProceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies,pp\. 110–119\.Cited by:[Appendix B](https://arxiv.org/html/2607.24763#A2.SS0.SSS0.Px3.p1.1)\.
- \[15\]P\. Liang, R\. Bommasani, T\. Lee, D\. Tsipras, D\. Soylu, M\. Yasunaga, Y\. Zhang, D\. Narayanan, Y\. Wu, A\. Kumar,et al\.\(2022\)Holistic evaluation of language models\.arXiv preprint arXiv:2211\.09110\.Cited by:[§3](https://arxiv.org/html/2607.24763#S3.SS0.SSS0.Px1.p1.1),[Table 2](https://arxiv.org/html/2607.24763#S3.T2.6.2.1.1)\.
- \[16\]C\. Lu, Y\. Zhou, F\. Bao, J\. Chen, C\. Li, and J\. Zhu\(2022\)Dpm\-solver: a fast ode solver for diffusion probabilistic model sampling in around 10 steps\.Advances in neural information processing systems35,pp\. 5775–5787\.Cited by:[Appendix C](https://arxiv.org/html/2607.24763#A3.SS0.SSS0.Px2.p1.1)\.
- \[17\]K\. Misaki and T\. Akiba\(2026\)UnMaskFork: test\-time scaling for masked diffusion via deterministic action branching\.arXiv preprint arXiv:2602\.04344\.Cited by:[§3](https://arxiv.org/html/2607.24763#S3.SS0.SSS0.Px1.p1.1)\.
- \[18\]S\. Nie, F\. Zhu, Z\. You, X\. Zhang, J\. Ou, J\. Hu, J\. Zhou, Y\. Lin, J\. Wen, and C\. Li\(2025\)Large language diffusion models\.arXiv preprint arXiv:2502\.09992\.Cited by:[§1](https://arxiv.org/html/2607.24763#S1.p1.1)\.
- \[19\]J\. Ou, S\. Nie, K\. Xue, F\. Zhu, J\. Sun, Z\. Li, and C\. Li\(2024\)Your absorbing discrete diffusion secretly models the conditional distributions of clean data\.arXiv preprint arXiv:2406\.03736\.Cited by:[Appendix B](https://arxiv.org/html/2607.24763#A2.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2607.24763#S1.SS0.SSS0.Px1.p1.9)\.
- \[20\]G\. Parmar, R\. Zhang, and J\. Zhu\(2022\)On aliased resizing and surprising subtleties in gan evaluation\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 11410–11420\.Cited by:[§1](https://arxiv.org/html/2607.24763#S1.SS0.SSS0.Px1.p1.9),[§3](https://arxiv.org/html/2607.24763#S3.SS0.SSS0.Px1.p2.1)\.
- \[21\]H\. Peng, P\. Liu, Z\. Dong, D\. Cheng, J\. Li, Y\. Tang, S\. Wang, and W\. X\. Zhao\(2025\)How efficient are diffusion language models? a critical examination of efficiency evaluation practices\.arXiv preprint arXiv:2510\.18480\.Cited by:[§1](https://arxiv.org/html/2607.24763#S1.SS0.SSS0.Px1.p1.9)\.
- \[22\]K\. Pillutla, S\. Swayamdipta, R\. Zellers, J\. Thickstun, S\. Welleck, Y\. Choi, and Z\. Harchaoui\(2021\)Mauve: measuring the gap between neural text and human text using divergence frontiers\.Advances in Neural Information Processing Systems34,pp\. 4816–4828\.Cited by:[Appendix B](https://arxiv.org/html/2607.24763#A2.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2607.24763#S1.SS0.SSS0.Px1.p1.9)\.
- \[23\]M\. Post\(2018\)A call for clarity in reporting bleu scores\.InProceedings of the third conference on machine translation: Research papers,pp\. 186–191\.Cited by:[§1](https://arxiv.org/html/2607.24763#S1.SS0.SSS0.Px1.p1.9),[§3](https://arxiv.org/html/2607.24763#S3.SS0.SSS0.Px1.p2.1)\.
- \[24\]P\. Pynadath, J\. Shi, and R\. Zhang\(2026\)Generative frontiers: why evaluation matters for diffusion language models\.arXiv preprint arXiv:2604\.02718\.Cited by:[Appendix B](https://arxiv.org/html/2607.24763#A2.SS0.SSS0.Px3.p1.1),[Appendix C](https://arxiv.org/html/2607.24763#A3.SS0.SSS0.Px1.p1.10),[§1](https://arxiv.org/html/2607.24763#S1.SS0.SSS0.Px1.p1.9),[§3](https://arxiv.org/html/2607.24763#S3.SS0.SSS0.Px1.p1.1),[Table 2](https://arxiv.org/html/2607.24763#S3.T2.6.4.3.1),[§4\.5](https://arxiv.org/html/2607.24763#S4.SS5.p1.4)\.
- \[25\]Y\. Qian, J\. Su, L\. Hu, P\. Zhang, Z\. Deng, P\. Zhao, and H\. Zhang\(2026\)D3LLM: ultra\-fast diffusion llm using pseudo\-trajectory distillation\.arXiv preprint arXiv:2601\.07568\.Cited by:[§3](https://arxiv.org/html/2607.24763#S3.SS0.SSS0.Px1.p1.1)\.
- \[26\]J\. Song, C\. Meng, and S\. Ermon\(2020\)Denoising diffusion implicit models\.arXiv preprint arXiv:2010\.02502\.Cited by:[Appendix C](https://arxiv.org/html/2607.24763#A3.SS0.SSS0.Px2.p1.1)\.
- \[27\]M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. Le, E\. Chi, D\. Zhou,et al\.\(2023\)Challenging big\-bench tasks and whether chain\-of\-thought can solve them\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 13003–13051\.Cited by:[§4](https://arxiv.org/html/2607.24763#S4.SS0.SSS0.Px1.p1.1),[§4\.6](https://arxiv.org/html/2607.24763#S4.SS6.p5.8)\.
- \[28\]G\. Wang, Y\. Schiff, S\. S\. Sahoo, and V\. Kuleshov\(2025\)Remasking discrete diffusion models with inference\-time scaling\.arXiv preprint arXiv:2503\.00307\.Cited by:[Appendix A](https://arxiv.org/html/2607.24763#A1.SS0.SSS0.Px1.p1.1),[Appendix B](https://arxiv.org/html/2607.24763#A2.SS0.SSS0.Px3.p1.1),[Appendix C](https://arxiv.org/html/2607.24763#A3.SS0.SSS0.Px1.p1.10),[§1](https://arxiv.org/html/2607.24763#S1.SS0.SSS0.Px1.p1.9),[§3](https://arxiv.org/html/2607.24763#S3.SS0.SSS0.Px1.p1.1),[Table 2](https://arxiv.org/html/2607.24763#S3.T2.6.5.4.1)\.
- \[29\]J\. Yang, Y\. Jiang, X\. Hu, S\. Cheng, B\. Qi, and J\. Shao\(2026\)DARE: diffusion large language models alignment and reinforcement executor\.arXiv preprint arXiv:2604\.04215\.Cited by:[§1](https://arxiv.org/html/2607.24763#S1.SS0.SSS0.Px1.p1.9),[§3](https://arxiv.org/html/2607.24763#S3.SS0.SSS0.Px1.p1.1)\.
- \[30\]J\. Ye, Z\. Xie, L\. Zheng, J\. Gao, Z\. Wu, X\. Jiang, Z\. Li, and L\. Kong\(2025\)Dream 7b: diffusion large language models\.arXiv preprint arXiv:2508\.15487\.Cited by:[§1](https://arxiv.org/html/2607.24763#S1.p1.1)\.
- \[31\]R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi\(2019\)HellaSwag: can a machine really finish your sentence?\.InAnnual Meeting of the Association for Computational Linguistics,External Links:[Link](https://api.semanticscholar.org/CorpusID:159041722)Cited by:[§4](https://arxiv.org/html/2607.24763#S4.SS0.SSS0.Px1.p1.1),[§4\.6](https://arxiv.org/html/2607.24763#S4.SS6.p5.8)\.
- \[32\]K\. Zhai, S\. Mollah, Z\. Wang, and M\. Shah\(2026\)CoRe: context\-robust remasking for diffusion language models\.arXiv preprint arXiv:2602\.04096\.Cited by:[Appendix A](https://arxiv.org/html/2607.24763#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.24763#S1.SS0.SSS0.Px1.p1.9),[§3](https://arxiv.org/html/2607.24763#S3.SS0.SSS0.Px1.p1.1)\.
- \[33\]K\. Zheng, Y\. Chen, H\. Mao, M\. Liu, J\. Zhu, and Q\. Zhang\(2024\)Masked diffusion models are secretly time\-agnostic masked models and exploit inaccurate categorical sampling\.ArXivabs/2409\.02908\.External Links:[Link](https://api.semanticscholar.org/CorpusID:272397565)Cited by:[Appendix A](https://arxiv.org/html/2607.24763#A1.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2607.24763#S3.SS0.SSS0.Px1.p2.1)\.
- \[34\]Z\. Zhou, L\. Chen, H\. Tong, and D\. Song\(2026\)Dllm: simple diffusion language modeling\.arXiv preprint arXiv:2602\.22661\.Cited by:[§1](https://arxiv.org/html/2607.24763#S1.SS0.SSS0.Px1.p1.9),[§3](https://arxiv.org/html/2607.24763#S3.SS0.SSS0.Px1.p1.1),[Table 2](https://arxiv.org/html/2607.24763#S3.T2.6.3.2.1)\.
- \[35\]Y\. Zhu, S\. Lu, L\. Zheng, J\. Guo, W\. Zhang, J\. Wang, and Y\. Yu\(2018\)Texygen: a benchmarking platform for text generation models\.InThe 41st international ACM SIGIR conference on research & development in information retrieval,pp\. 1097–1100\.Cited by:[Appendix B](https://arxiv.org/html/2607.24763#A2.SS0.SSS0.Px3.p1.1)\.

## Broader Impact

Careis an evaluation framework rather than a new model or capability, so its societal effects are mediated by how the research community uses it\.

#### Positive impact\.

Standardized, compute\-aware evaluation reduces the rate at which apparent algorithmic progress turns out to reflect evaluation artifacts\. The Clean\-FID and SacreBLEU precedents suggest that domain\-specific evaluation standards can recover years of overstated gains and redirect community effort toward genuine improvements\. For MDLMs specifically, our compute\-matched results indicate that several recent reported gains may shrink or reverse under fair comparison; making this visible early reduces wasted compute and research effort\. The leaderboard infrastructure lowers the barrier for new MDLM proposals to be compared honestly against prior work\.

#### Negative impacts and mitigations\.

*Leaderboard gaming*: public leaderboards can incentivize hyperparameter overfitting on reported metrics\. We mitigate this with multi\-metric reporting, stochasticity sweeps that surface sensitivity, and verified\-vs\-pending status flags\.*Gatekeeping*: standardized protocols can become barriers to entry\. We have kept Careaccessible \(∼\\sim50 LOC per added model,∼\\sim20 per added strategy; reproducible on a single 8\-GPU node\) and treat each of the seven practices as independently reportable rather than all\-or\-nothing\.*Scope*: Careaddresses inference\-time evaluation and does not speak to training\-data, content\-moderation, or deployment\-safety questions; the MDLMs we evaluate inherit the safety, fairness, and bias properties of their training corpora, which our findings neither improve nor worsen\.

#### Dual\-use\.

The framework does not increase MDLM capabilities, scale, or speed; it only changes how generations are measured\. Honest evaluation can in principle accelerate development of more capable MDLMs by clarifying which research directions work, but the alternative— continued evaluation incoherence—carries its own costs\. We view the dual\-use risk as low and substantially outweighed by the scientific\-integrity benefits\.

## Appendix AEthical consideration and Future Work

#### Scope: discrete masked diffusion only\.

Careis scoped tomasked discrete\-token diffusion language models—the only family where remasking, NFE accounting, and distributional metrics over a finite vocabulary are jointly well\-defined\. This is a design choice, not a gap: the seven remasking papers motivating Careare all masked discrete\-token models\[[28](https://arxiv.org/html/2607.24763#bib.bib4),[32](https://arxiv.org/html/2607.24763#bib.bib5),[2](https://arxiv.org/html/2607.24763#bib.bib19),[9](https://arxiv.org/html/2607.24763#bib.bib7),[11](https://arxiv.org/html/2607.24763#bib.bib8),[1](https://arxiv.org/html/2607.24763#bib.bib9),[12](https://arxiv.org/html/2607.24763#bib.bib10)\]\. The protocol does not apply to continuous\-state diffusion LMs \(e\.g\., Diffusion\-LM, PLAID\) or flow\-based LMs, which require different compute accounting and metrics\.

#### Statistical caveats\.

The three confounds we study are not exhaustive—numerical precision\[[33](https://arxiv.org/html/2607.24763#bib.bib31)\], prompt format, and sequence length also matter but produce smaller measured effects\. Headline tests use 3 seeds, corroborated by a bootstrap CI \(Section[4\.4](https://arxiv.org/html/2607.24763#S4.SS4)\); larger seed counts would strengthen marginal claims\. The interaction analysis focuses onnonevs\.high\_entropy; the remaining five strategies appear in the deterministic seven\-strategy comparison \(Appendix[C\.10](https://arxiv.org/html/2607.24763#A3.SS10)\) but are not re\-evaluated at the stochastic headline configuration\.

#### Future work\.

Two natural extensions are beyond our current scope\. For continuous diffusion LMs, an analog of Carewould replace remasking with embedding\-space perturbations and use tokenization\-free distributional metrics\. For flow\-based LMs, compute accounting requires integrating over flow time rather than counting forward passes\. A unified compute\-aware evaluation framework spanning masked\-discrete, continuous, and flow\-mapping LMs is a natural next step\.

## Appendix BTaxonomy

#### Strategy taxonomy\.

The seven strategies supported by LLaDA are categorized based on their selection criteria:no\-op\(none, serving as the baseline\);stochastic\(random, independent of state\);confidence\-based\(low\_confidence,running\_confidence\);entropy\-based\(high\_entropy,conf\_entropy\); andagreement\-based\(agreement\)\. Dream introducesorigin/entropy/maskgit, which are similar but not identical \(§[4](https://arxiv.org/html/2607.24763#S4)\)\.

#### Confound taxonomy\.

Caremanages three aspects simultaneously:compute\(nominal steps versus actual NFE\),metric\(PPL/MAUVE/diversity\), andstochasticity\(unmask\_tempvalues of 0\.0, 0\.1, 0\.25, 0\.5\)\. Each of these can independently alter rankings; the interaction discussed in §[4\.4](https://arxiv.org/html/2607.24763#S4.SS4)is positioned on the strategy×\\timesstochasticity axis\.

#### Metric taxonomy and choice rationale\.

Careemploys four metrics, each identifying a unique failure mode that others might overlook:PPL\(evaluated by Llama\-3\-8B\) for fluency, though it is susceptible to PPL manipulation\[[28](https://arxiv.org/html/2607.24763#bib.bib4)\];MAUVE\[[22](https://arxiv.org/html/2607.24763#bib.bib24)\]for distributional alignment, which is the current benchmark in MDLM evaluation\[[19](https://arxiv.org/html/2607.24763#bib.bib20),[24](https://arxiv.org/html/2607.24763#bib.bib11)\];Self\-BLEU\[[35](https://arxiv.org/html/2607.24763#bib.bib47)\]for assessing pairwise diversity; andDistinct\-3\[[14](https://arxiv.org/html/2607.24763#bib.bib48)\]for evaluating vocabulary richness\. Empirical evidence shows that PPL and MAUVE rankings can reverse \(Section[4\.2](https://arxiv.org/html/2607.24763#S4.SS2)\); Self\-BLEU and MAUVE show divergence \(Section[4\.4](https://arxiv.org/html/2607.24763#S4.SS4)\), indicating that no single metric is sufficient\. BERTScore, ROUGE\-nn, and evaluations based solely on perplexity were excluded because they fail to capture the distributional failure modes that are central to this study\. The protocol mandates reporting all four metrics together\.

## Appendix CExtended Experiments and Set Up

#### Setup\.

We use a 64\-token prefix from OpenWebText or LM1B and generate 128 tokens; PPL is computed via Llama\-3\-8B\. All generations are filtered by a character\-level English filter \(ASCII≥\\geq0\.85\)\. Key results use 3 seeds with pairedtt\-tests; seed variance reflects sampling stochasticity only \(fixed pretrained checkpoints\)\. Per\-configuration standard errors use sample SD /3\\sqrt\{3\}\(𝚍𝚍𝚘𝚏=1\\mathtt\{ddof\}\{=\}1\); gap SEs are computed on paired seed\-wise differences\. MAUVE uses the standard implementation withK=500K\{=\}500and a GPT\-2 XL backbone\. On LLaDA,unmask\_tempscales Gumbel noise on unmasking logits \(t=0t\{=\}0selects argmax\); Dream uses an analogousalg\_temp\. Prompt examples appear in Appendix[C\.1](https://arxiv.org/html/2607.24763#A3.SS1)\. For inference\-only evaluation, three seeds provide sufficient resolution to detect the effect sizes we report: the key interaction gap \(0\.2960\.296MAUVE, paired SE=0\.042=0\.042\) yieldst​\(2\)=7\.06t\(2\)\{=\}7\.06, with a bootstrapped 95% CI of\[0\.243,0\.356\]\[0\.243,0\.356\]excluding zero \(§[4\.4](https://arxiv.org/html/2607.24763#S4.SS4)\)\. This practice is consistent with inference\-time evaluation in recent diffusion LM work\[[28](https://arxiv.org/html/2607.24763#bib.bib4),[24](https://arxiv.org/html/2607.24763#bib.bib11)\], where three seeds are standard given the high per\-run compute cost\. While three seeds is sufficient to detect the large effect sizes in our headline results, future work evaluating marginal strategies with smaller expected gaps should scale to five or more seeds\. Cross\-family comparisons inherit the small implementation differences between LLaDA and Dream\.

#### Why we use NFE and not wall\-clock time\.

We measure compute via actual NFE rather than wall\-clock time\. NFE counts the forward passes a strategy consumes per generation—the quantity directly controlled by the sampler—and is hardware invariant\. NFE\-based comparison is standard in diffusion sampler evaluation\[[26](https://arxiv.org/html/2607.24763#bib.bib50),[16](https://arxiv.org/html/2607.24763#bib.bib51)\], where quality\-versus\-NFE curves are the canonical efficiency plot, and has been adopted in recent MDLM work\. Wall\-clock time conflates algorithmic cost with implementation details \(batching, kernels, hardware generation\) that vary across labs\. Because per\-step forward passes dominate MDLM runtime, NFE differences translate to proportional latency differences on matched hardware; Section[A](https://arxiv.org/html/2607.24763#A1)discusses limits of this approximation\.

### C\.1Prompt Templates

All unconditional generation experiments use a 64\-token prefix sampled from the held\-out OpenWebText test split \(or LM1B test split for the cross\- dataset check\), followed by 128 tokens of generation\.

> Prefix \(64 tokens, OWT example\): ‘‘The development of large\-scale language models has been one of the most significant advances in artificial intelligence over the past decade\. Researchers have explored several architectures, training paradigms, and inference techniques in pursuit of’’ Generation budget:128 tokens\. Stop conditions:EOS token, or 128\-token cap reached\.

For HumanEval, prompts follow the standardlm\-eval\[[6](https://arxiv.org/html/2607.24763#bib.bib27)\]format: function signature plus docstring, with the model expected to complete the function body\. We use the unmodified release version of the benchmark\. For GSM8K, prompts use the 4\-shot in\-context format from the originallm\-evalconfiguration; we do not modify the in\-context exemplars\.

### C\.2Customizing Carefor a New Strategy

Adding a new remasking strategy requires implementing one method on theRemaskingSamplerinterface\. The base class handles NFE accounting, language filtering, multi\-metric evaluation, and seed control\. The example below is the implementation ofHighEntropyas it ships in the released framework:

classHighEntropy\(RemaskingStrategy\):

"""Re\-maskthetop\-khighest\-entropypositionsperstep\."""

def\_\_init\_\_\(self,fraction=0\.15\):

self\.fraction=fraction

defselect\(self,logits,history,step\):

probs=logits\.softmax\(dim=\-1\)

entropy=\-\(probs\*probs\.log\(\)\)\.sum\(dim=\-1\)

k=int\(self\.fraction\*logits\.shape\[1\]\)

returntopk\_mask\(entropy,k=k\)

A complete strategy is approximately 20 lines of code\. Adding a new model backbone takes approximately 50 lines—a thin wrapper around the model’s native sampling API that routes per\-step logits and history into theRemaskingSampler\. See the released codebase for additional worked examples of all seven strategies and the three Dream variants\.

### C\.3Token Replacement Rate

![Refer to caption](https://arxiv.org/html/2607.24763v1/x7.png)Figure 7:Token replacement rate per denoising step \(LLaDA\-8B\-Base,t=0\.25t\{=\}0\.25, 128 steps, 10 samples\)\.
### C\.4Diversity Metrics

![Refer to caption](https://arxiv.org/html/2607.24763v1/x8.png)Figure 8:Diversity at 256 steps \(LLaDA\-8B\-Base, OWT\)\. \(a\) Self\-BLEU:high\_entropyhas lowest \(most pairwise\-diverse\) yet worst MAUVE\. \(b\) Distinct\-3 vocabulary collapses att≥0\.25t\\geq 0\.25, exposing distributional collapse despite surface diversity\.
### C\.5Polar Area Charts

![Refer to caption](https://arxiv.org/html/2607.24763v1/x9.png)Figure 9:Polar area charts: deterministic vs stochastic \(LLaDA\-8B\-Base, OWT, 128 steps, 3\-seed mean\)\. Each axis shows one of the four core metrics normalized to \[0,1\]\.
### C\.6Dream Pareto Frontier

![Refer to caption](https://arxiv.org/html/2607.24763v1/x10.png)Figure 10:Dream\-7B MAUVE vs PPL Pareto\.
### C\.7Three\-Seed Analysis

Table 13:Three\-seed MAUVE on English\-filtered results \(LLaDA\-8B\-Base, OWT, prefix=64, gen=128\)\.
### C\.8LLaDA\-MoE Cross\-Architecture

Table 14:LLaDA\-MoE\-7B\-A1B \(∼\{\\sim\}1B active params\), OWT, prefix=64, gen=128, single seed, English\-filtered\. Rows show 128\- and 256\-step configurations att=0\.0t\{=\}0\.0andt=0\.1t\{=\}0\.1\.
### C\.9Dream PPL–MAUVE Results

Table 15:Dream\-7B\-Base results, native API \(OWT, prefix=64, gen=128, English\-filtered, single seed\)\.origin= Dream’s no\-remask baseline;ent= entropy\-based remasking;mg= MaskGIT\-style\.Green=best PPL,blue=best MAUVE\.
### C\.10Full 7\-Strategy Comparison

We report all seven LLaDA\-supported remasking strategies here for completeness; the main text usesnoneandhigh\_entropyas a running pair because they define the extremes of compute and churn under the Careprotocol\.

Table 16:All 7 strategies at 128 steps \(LLaDA\-8B\-Base, OWT, deterministic, prefix=32, seed 1\)\. Prefix=32 was used for this exploratory comparison; all main\-body experiments use prefix=64\. Absolute MAUVE values differ; strategy rank order is consistent\.
### C\.11PPL Across Temperatures

Table 17:PPL across temperatures \(LLaDA\-8B\-Base, Llama\-3\-8B evaluator, English\-filtered\)\.
### C\.12Downstream Results \(GSM8K\)

Table 18:GSM8K accuracy \(%\), LLaDA\-Instruct, 4\-shot,n=200n\{=\}200, SE≈3\.5\{\\approx\}3\.5%\.
### C\.13English Filter Acceptance Rates

Table 19:OWT acceptance rates \(%\), LLaDA\-8B\-Base\.Table 20:OWT vs\. LM1B acceptance rates \(%\), LLaDA\-8B\-Base\.
### C\.14Full CareLeaderboard \(Tier 1 \+ Tier 2\)

Table[21](https://arxiv.org/html/2607.24763#A3.T21)reports the full leaderboard\. Tier 1 rows reproduce Table[12](https://arxiv.org/html/2607.24763#S4.T12); Tier 2 extends across architecture and scale\. The interaction direction holds across all twelve models\.

Table 21:Full Careleaderboard \(256 nominal steps, English\-filtered, OWT prefix=64, gen=128; PPL via Llama\-3\-8B; MAUVE via GPT\-2 XL backbone,K=500K\{=\}500; 3\-seed mean except where noted\)\. Interaction gap = MAUVE\(none\)−\-MAUVE\(high\_ent\.\) att=0\.25t\{=\}0\.25\.∗MoE measured att=0\.1t\{=\}0\.1\.‡Dreamentropystrategy\. SM = Soft\-Masked\. “—” PPL for Dream\-7B\-Base reflects Dream’s native API not exposing intermediate logits compatible with the Llama\-3\-8B external PPL evaluator; MAUVE is unaffected\.