Speculative Refinement: A Hybrid Autoregressive Diffusion Decoding Strategy and Its Behavior Across Benchmarks

# Speculative Refinement: A Hybrid Autoregressive Diffusion Decoding Strategy and Its Behavior Across Benchmarks
Source: [https://arxiv.org/html/2606.27474](https://arxiv.org/html/2606.27474)
Aditi Gupta IIIT Hyderabad, India aditi\.gu@research\.iiit\.ac\.in&Neel Mishra Microsoft mishraneel99@gmail\.comKushagra Trivedi IIIT Hyderabad, India kushagra\.trivedi@students\.iiit\.ac\.in&Pawan Kumar IIIT Hyderabad, India pawan\.kumar@iiit\.ac\.in

###### Abstract

How should we evaluate generation systems that combine autoregressive \(AR\) and diffusion decoding? We study this question through*Speculative Refinement*\(SpecRef\), a training\-free hybrid method that warm\-starts a masked diffusion language model from an AR draft using entropy\-guided selective masking\. Evaluating SpecRef across six benchmarks \(HumanEval, MBPP, GSM8K, BBH, ARC\-Challenge, HellaSwag\) with three distinct evaluation protocols \(execution\-based pass@1, exact\-match, log\-likelihood scoring\), we surface several findings relevant beyond our specific system: \(1\) code benchmarks conflate*structural discovery*with*logical correctness*: providing a syntactic scaffold lifts accuracy from near zero to over 20% without changing the model, indicating that much of the baseline failure is structural; \(2\) a*refinement tension*phenomenon where multi\-stage correction degrades already\-correct tokens, exposing benchmark saturation ceilings invisible to single\-model evaluation; \(3\) log\-likelihood and generative evaluation produce different model rankings for the same model pair, suggesting they measure different capabilities; \(4\) standard Python post\-processing silently breaks code evaluation for non\-AR generators\. These observations apply to any multi\-stage or non\-autoregressive generation pipeline and point toward more diagnostic evaluation practices\. Codes are[here](https://github.com/misterpawan/specref-eval.git)\.

Speculative Refinement: A Hybrid Autoregressive Diffusion Decoding Strategy and Its Behavior Across Benchmarks

Aditi GuptaIIIT Hyderabad, Indiaaditi\.gu@research\.iiit\.ac\.inNeel MishraMicrosoftmishraneel99@gmail\.com

Kushagra TrivediIIIT Hyderabad, Indiakushagra\.trivedi@students\.iiit\.ac\.inPawan KumarIIIT Hyderabad, Indiapawan\.kumar@iiit\.ac\.in

## 1Introduction

An 8\-billion\-parameter diffusion language model scores 0% on HumanEval when given 16 denoising steps\. Hand it the same problem with a syntactic scaffold from a 2\.7B autoregressive drafter, and it scores over 20% without any retraining\. The model did not suddenly learn to code; the benchmark was testing whether it could discover Python’s indentation rules from scratch, and that turned out to be the hard part\. This is one of several findings that emerged when we tried to evaluate a hybrid generation system using standard benchmarks and discovered that the benchmarks themselves were not designed for this\.

Diffusion language models \(dLMs\) have moved rapidly from research curiosity to a competitive generation paradigm\. LLaDA\(Nieet al\.,[2025](https://arxiv.org/html/2606.27474#bib.bib9)\)demonstrated that masked diffusion scales to 8B parameters and matches LLaMA\-3 on in\-context learning\. MDLM\(Sahooet al\.,[2024](https://arxiv.org/html/2606.27474#bib.bib22)\)showed that simple masked diffusion with modern training recipes closes much of the perplexity gap with AR models\. Dream 7B\(Yeet al\.,[2025](https://arxiv.org/html/2606.27474#bib.bib25)\)further narrowed this gap while retaining diffusion\-native capabilities like parallel generation\. On the commercial side, Inception Labs’ Mercury\(Labset al\.,[2025](https://arxiv.org/html/2606.27474#bib.bib26)\)achieved over 1,000 tokens per second on NVIDIA H100 GPUs while matching the quality of speed\-optimized AR models, demonstrating that dLMs are viable at production scale\. These models offer structural advantages that AR models lack: bidirectional context at every step, arbitrary position infilling, and the ability to correct multiple tokens in parallel\. The question is no longer whether dLMs can work, but how to combine their strengths with the sequential planning ability of AR models\.

Several recent works have begun exploring this direction\.Christopheret al\.\([2025](https://arxiv.org/html/2606.27474#bib.bib14)\)used diffusion as a drafter for speculative decoding\.Horvitzet al\.\([2024](https://arxiv.org/html/2606.27474#bib.bib15)\)conditioned diffusion paraphrasers on AR output\. Warm\-starting diffusion from corrupted inputs is well\-studied in images\(Menget al\.,[2022](https://arxiv.org/html/2606.27474#bib.bib10)\)and was recently formalized for language\(Scholz and Turner,[2025](https://arxiv.org/html/2606.27474#bib.bib13)\)\. These efforts suggest that hybrid AR\-diffusion pipelines are a promising research direction, but they also raise a question that has received less attention:*how should we evaluate them?*

The NLP community has built mature evaluation infrastructure\(Gehrmannet al\.,[2021](https://arxiv.org/html/2606.27474#bib.bib16); Lianget al\.,[2023](https://arxiv.org/html/2606.27474#bib.bib24)\): execution\-based code benchmarks\(Chenet al\.,[2021](https://arxiv.org/html/2606.27474#bib.bib17)\), math reasoning suites\(Cobbeet al\.,[2021](https://arxiv.org/html/2606.27474#bib.bib19)\), logical deduction tasks\(Suzgunet al\.,[2023](https://arxiv.org/html/2606.27474#bib.bib23)\), and LLM\-as\-judge protocols\(Zhenget al\.,[2023](https://arxiv.org/html/2606.27474#bib.bib20)\)\. But this infrastructure was built for left\-to\-right autoregressive \(AR\) decoding\(Vaswaniet al\.,[2017](https://arxiv.org/html/2606.27474#bib.bib1)\)\. Post\-processing scripts assume tokens arrive in order\. Execution sandboxes assume well\-formed syntax\. Log\-likelihood scoring assumes a single forward pass\. These assumptions break quietly when applied to dLMs\(Nieet al\.,[2025](https://arxiv.org/html/2606.27474#bib.bib9); Austinet al\.,[2021a](https://arxiv.org/html/2606.27474#bib.bib6); Louet al\.,[2024](https://arxiv.org/html/2606.27474#bib.bib8)\), speculative decoding pipelines\(Leviathanet al\.,[2023](https://arxiv.org/html/2606.27474#bib.bib2); Chenet al\.,[2023](https://arxiv.org/html/2606.27474#bib.bib3)\), and hybrid systems that combine both\. Benchmark contamination\(Xuet al\.,[2024](https://arxiv.org/html/2606.27474#bib.bib21)\)is already a known concern; we show that*protocol mismatch*is equally serious and largely unexamined\.

We expose these issues through*Speculative Refinement*\(SpecRef\), a training\-free hybrid method that uses a small AR model \(Phi\-2, 2\.7B\) to draft an initial answer, then selectively masks the least\-confident tokens based on their Shannon entropy and lets a large diffusion model \(LLaDA\-8B\(Nieet al\.,[2025](https://arxiv.org/html/2606.27474#bib.bib9)\)\) refine them \(Figure[1](https://arxiv.org/html/2606.27474#S1.F1)\)\. The method works with any off\-the\-shelf AR drafter and requires no training\. Evaluating it across six benchmarks with three evaluation protocols surfaced findings that go well beyond our specific system\.

#### Contributions\.

To the best of our knowledge, this is the first systematic study that pairs an AR language model as a drafter with a masked diffusion model \(LLaDA\) as a refiner, and evaluates the resulting hybrid across multiple benchmarks and evaluation protocols\. Our main contributions are evaluation insights rather than the method itself:

- •Structural discovery vs\. logical correctness\.Code benchmarks \(HumanEval\(Chenet al\.,[2021](https://arxiv.org/html/2606.27474#bib.bib17)\), MBPP\(Austinet al\.,[2021b](https://arxiv.org/html/2606.27474#bib.bib18)\)\) conflate two capabilities: discovering syntactic structure \(indentation, brackets, function signatures\) and writing correct logic\. Providing an AR scaffold separates these: accuracy jumps from near 0% to 20%\+ at the same diffusion step budget \(Table[1](https://arxiv.org/html/2606.27474#S4.T1)\), indicating that a substantial portion of the standalone diffusion baseline’s failure on code is structural rather than logical\.
- •Refinement tension\.On benchmarks where the AR drafter is already accurate \(BBH\(Suzgunet al\.,[2023](https://arxiv.org/html/2606.27474#bib.bib23)\): 82\.8%\), multi\-stage refinement can*decrease*accuracy \(to 80\.4%\) because the refiner overwrites correct tokens\. This ceiling effect is invisible when evaluating either model alone\.
- •PPL vs\. generative evaluation gap\.On multiple\-choice tasks \(ARC\-Challenge, HellaSwag\), log\-likelihood scoring produces rankings that differ from those observed on generative benchmarks for the same model pair \(Table[2](https://arxiv.org/html/2606.27474#S4.T2)\), suggesting that PPL\-mode and generative evaluation measure different capabilities in diffusion models\.
- •Post\-processing pitfalls\.Standard code post\-processing \(\.strip\(\),textwrap\.dedent\(\)\) silently destroys Python indentation for non\-AR generators, lowering pass@1 scores\. We document this and other practical pitfalls encountered when adapting AR\-era evaluation to hybrid systems \(Section[3\.3](https://arxiv.org/html/2606.27474#S3.SS3)\)\.

We also describe SpecRef itself \(Section[2](https://arxiv.org/html/2606.27474#S2)\) since it is needed to interpret the evaluation findings, and report that it improves absolute accuracy by up to 25\.8% over standalone diffusion on code generation while cutting wall\-clock latency by up to2\.43×2\.43\\times\.

AR DrafterPhi\-2, 2\.7Bccdeff\(x\):retx\+1x~0\\tilde\{x\}\_\{0\}EntropyMaskingdeff\(MM:MMxτx\_\{\\tau\}RefinerLLaDA\-8Bdeff\(n\):retn\*2x0x\_\{0\}SSstepskeptmaskedrefined

Figure 1:SpecRef pipeline\. The AR drafter generates a candidate; per\-token entropy identifies uncertain positions \(yellow/red\), which are replaced with\[MASK\]\. The diffusion refiner denoises from this warm\-started state inSSsteps\.

## 2Background: Hybrid AR\-Diffusion Generation

We describe the components of our hybrid system, which serves as the evaluation subject throughout the paper\.

### 2\.1AR drafter and uncertainty estimation

Letccdenote a prompt andx0=\(x0,1,…,x0,L\)x\_\{0\}=\(x\_\{0,1\},\\dots,x\_\{0,L\}\)be a sequence ofLLtokens from vocabularyVV\. An AR drafter with parametersϕ\\phidefines a left\-to\-right factored distribution:

qϕ​\(x0∣c\)=∏i=1Lqϕ​\(x0,i∣c,x0,<i\)\.q\_\{\\phi\}\(x\_\{0\}\\mid c\)=\\prod\_\{i=1\}^\{L\}q\_\{\\phi\}\(x\_\{0,i\}\\mid c,x\_\{0,<i\}\)\.\(1\)At each positionii, we record the Shannon entropy as a measure of uncertainty:

ui\\displaystyle u\_\{i\}≜H\(qϕ\(⋅∣c,x0,<i\)\)\\displaystyle\\triangleq H\(q\_\{\\phi\}\(\\cdot\\mid c,x\_\{0,<i\}\)\)=−∑vqϕ​\(v∣⋅\)​log⁡qϕ​\(v∣⋅\)\.\\displaystyle=\-\\\!\\sum\_\{v\}q\_\{\\phi\}\(v\\mid\\cdot\)\\log q\_\{\\phi\}\(v\\mid\\cdot\)\.\(2\)A high\-entropy position means the AR model is unsure about that token, making it a natural candidate for diffusion refinement\. In our experiments we usemicrosoft/phi\-2\(2\.7B parameters\), but the method works with any AR model that outputs probability logits\.

### 2\.2Masked diffusion language modeling

A masked diffusion LM such as LLaDA\(Nieet al\.,[2025](https://arxiv.org/html/2606.27474#bib.bib9)\)defines a forward process that progressively masks a clean token sequence toward a fully corrupted state using a token\-level masking kernel:

qt​\(xt,i∣x0,i\)=\{x0,iwith prob\.​\(1−αt\)\[MASK\]with prob\.​αt,q\_\{t\}\(x\_\{t,i\}\\mid x\_\{0,i\}\)=\\begin\{cases\}x\_\{0,i\}&\\text\{with prob\.\}\\ \(1\-\\alpha\_\{t\}\)\\\\ \\texttt\{\[MASK\]\}&\\text\{with prob\.\}\\ \\alpha\_\{t\}\\end\{cases\},\(3\)whereαt∈\[0,1\]\\alpha\_\{t\}\\in\[0,1\]is a monotone noise schedule withα0=0\\alpha\_\{0\}=0\(clean\) andαT=1\\alpha\_\{T\}=1\(fully masked\)\. The learned reverse processpθ​\(xt−1∣xt,c\)p\_\{\\theta\}\(x\_\{t\-1\}\\mid x\_\{t\},c\)is a bidirectional BERT\-style Transformer \(LLaDA\-8B in our case\) that attends to all positions at once and iteratively denoises fromt=Tt=Ttot=0t=0\. Unlike AR models, it can condition on “future” tokens at positionj\>ij\>i, and it can infill arbitrary subsets of tokens\.

### 2\.3SpecRef: warm\-start with selective masking

The main bottleneck when sampling from a fully masked prior is*structure discovery*: att=Tt=T, every token is\[MASK\], so the model must recover code skeletons, reasoning chains, and answer formats from scratch\. With a small step budgetSS, it often fails to produce syntactically valid output\.

SpecRef warm\-starts the diffusion model from an AR draft instead\. It samples a draftx~0∼qϕ\(⋅∣c\)\\tilde\{x\}\_\{0\}\\sim q\_\{\\phi\}\(\\cdot\\mid c\), then applies the forward corruption kernel at timeτ\\tau:

xτ∼qτ\(⋅∣x~0\),x\_\{\\tau\}\\sim q\_\{\\tau\}\(\\cdot\\mid\\tilde\{x\}\_\{0\}\),\(4\)and runs the learned reverse process fromτ\\tauto0:

xt−1∼pθ\(⋅∣xt,c\)fort=τ,τ−1,…,1\.x\_\{t\-1\}\\sim p\_\{\\theta\}\(\\cdot\\mid x\_\{t\},c\)\\quad\\text\{for\}\\ t=\\tau,\\tau\{\-\}1,\\dots,1\.\(5\)In practice, we use a subset of time indices \(e\.g\., DDIM\-style\(Songet al\.,[2022](https://arxiv.org/html/2606.27474#bib.bib11)\)\) to reduce the number of forward passes toS≪τS\\ll\\tau\.

Rather than masking globally, SpecRef masks*selectively*: only the topk%k\\%highest\-entropy positions are replaced with\[MASK\]\. Two heuristics extend the mask setℐ\\mathcal\{I\}:

1. 1\.Math\-aware block expansion\(e=2e=2\): positions aligned to numeric or math\-operator characters are identified via regex\. All tokens within±2\\pm 2positions are added:ℐ←ℐ∪\{i∣\|i−j\|≤2,j∈ℳ\}\\mathcal\{I\}\\leftarrow\\mathcal\{I\}\\cup\\\{i\\mid\|i\-j\|\\leq 2,\\;j\\in\\mathcal\{M\}\\\}, whereℳ\\mathcal\{M\}indexes math\-character positions\. This ensures arithmetic context is regenerated as a unit\.
2. 2\.Tail truncation\(M=150M=150\): all tokens past index 150 are unconditionally masked to force the diffusion model to regenerate conclusions, where AR drift is worst\.

Token scores for the top\-kkselection are:

mi=clip​\(uilog⁡\|V\|,0,1\),m\_\{i\}=\\mathrm\{clip\}\\\!\\left\(\\frac\{u\_\{i\}\}\{\\log\|V\|\},\\,0,\\,1\\right\),\(6\)whereuiu\_\{i\}is mapped to refiner\-token positions via character\-level offset alignment \(needed because the drafter and refiner use different tokenizers\)\. We usek=60%k=60\\%throughout\. Algorithm[1](https://arxiv.org/html/2606.27474#alg1)gives the full procedure\.

Algorithm 1Statistical SpecRef \(Top\-kkEntropy Masking\)0:Prompt

cc; AR drafter

qϕq\_\{\\phi\}; refiner

pθp\_\{\\theta\}; mask percentile

kk; math\-expansion window

ee; tail truncation

MM\.

0:Refined sample:

x0x\_\{0\}
1:Sample draft:

x~0∼qϕ\(⋅∣c\)\\tilde\{x\}\_\{0\}\\sim q\_\{\\phi\}\(\\cdot\\mid c\)
2:Extract logit entropies:

ui=H\(qϕ\(⋅∣c,x~0,<i\)\)u\_\{i\}=H\(q\_\{\\phi\}\(\\cdot\\mid c,\\tilde\{x\}\_\{0,<i\}\)\)
3:Compute the set of top

k%k\\%tokens by entropy:

ℐtop\\mathcal\{I\}\_\{\\text\{top\}\}
4:Math Block Expansion: Let

ℳ\\mathcal\{M\}be indices of tokens aligned to numeric/math characters\. Set

ℐ=ℐtop∪\{i∣\|i−j\|≤e,j∈ℳ\}\\mathcal\{I\}=\\mathcal\{I\}\_\{\\text\{top\}\}\\cup\\\{i\\mid\|i\-j\|\\leq e,\\;j\\in\\mathcal\{M\}\\\}
5:Tail Truncation: Set

ℐ=ℐ∪\{i∣i\>M\}\\mathcal\{I\}=\\mathcal\{I\}\\cup\\\{i\\mid i\>M\\\}
6:Form intermediate mask state

xτx\_\{\\tau\}:

xτ,i=\{\[MASK\]if​i∈ℐx~0,iotherwisex\_\{\\tau,i\}=\\begin\{cases\}\\texttt\{\[MASK\]\}&\\text\{if \}i\\in\\mathcal\{I\}\\\\ \\tilde\{x\}\_\{0,i\}&\\text\{otherwise\}\\end\{cases\}
7:forrefinement steps

SSdo

8:Sample proposal:

xt−1′∼pθ\(⋅∣xt,c\)x^\{\\prime\}\_\{t\-1\}\\sim p\_\{\\theta\}\(\\cdot\\mid x\_\{t\},c\)
9:endfor

10:return

x0x\_\{0\}

### 2\.4Compute cost and the handoff point

The total cost isL​CAR\+S​CDL\\,C\_\{\\text\{AR\}\}\+S\\,C\_\{\\text\{D\}\}, whereCARC\_\{\\text\{AR\}\}is the amortized cost of one AR token \(negligible under vLLM serving\) andCDC\_\{\\text\{D\}\}is the cost of one diffusion forward pass over allLLpositions\. A standalone diffusion sampler withNNsteps costsN​CDN\\,C\_\{\\text\{D\}\}; SpecRef targetsS≪NS\\ll N\.

The mask ratiok%k\\%\(equivalently the warm\-start timeτ\\tau\) controls a tradeoff:k→0k\\to 0preserves the entire draft;k→100%k\\to 100\\%reduces to standard diffusion from scratch\. We usek=60%k=60\\%: enough tokens survive to carry global structure, while enough are masked for correction\. On GSM8K \(RTX 4090\), this yields 6\.79s per query vs\. 16\.49s for standalone LLaDA atS=64S=64, a2\.43×2\.43\\timesspeedup\.

## 3Evaluation Setup

### 3\.1Models and infrastructure

All experiments ran on the 8×\\timesNVIDIA V100, 16GB each\. The refiner \(LLaDA\-8B\-Instruct\) is split across 2 GPUs via pipeline parallelism\. The drafter \(Phi\-2, 2\.7B\) runs under vLLM with 4\-bit quantization; its latency is under 1% of total generation time\.

### 3\.2Evaluation protocols

We deliberately chose benchmarks that require three different evaluation protocols, since the protocol itself turned out to be a significant variable\.

#### Protocol A: Execution\-based \(pass@1\)\.

HumanEval\(Chenet al\.,[2021](https://arxiv.org/html/2606.27474#bib.bib17)\)and MBPP\(Austinet al\.,[2021b](https://arxiv.org/html/2606.27474#bib.bib18)\)\. Generated Python code is executed against ground\-truth unit tests in a sandbox with a 10\-secondSIGALRMtimeout \(infinite loops are common in early diffusion steps\)\. We report pass@1\.

#### Protocol B: Exact match with parsing\.

GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2606.27474#bib.bib19)\)and MATH \(supplementary, evaluated on a 5,000\-sample subset\)\. Models are prompted to produce answers in\\boxed\{\.\.\.\}format\. We extract the final answer with regex and compute exact\-match accuracy\.

#### Protocol C: Log\-likelihood scoring \(PPL\-mode\)\.

ARC\-Challenge and HellaSwag\. Each answer choice is formatted ascontext \+ choice\_text\. The conditional log\-likelihood is estimated via 8\-round Monte Carlo masking \(randomly masking a subset of continuation tokens per round, scoring under the model, and averaging\), following the LLaDA evaluation protocol\(Nieet al\.,[2025](https://arxiv.org/html/2606.27474#bib.bib9)\)\. The highest\-scoring choice is selected\. This protocol is diffusion\-step\-independent: no iterative denoising is performed, giving a4×4\\timesspeedup over generative evaluation\.

### 3\.3Evaluation pitfalls encountered

Adapting standard AR\-era evaluation to a hybrid system revealed several pitfalls that would affect any non\-AR or multi\-stage generator:

1. 1\.Post\-processing breaks indentation\.Standard code post\-processing \(\.strip\(\),textwrap\.dedent\(\)\) destroys Python’s semantic indentation\. For diffusion outputs, which may contain irregular whitespace patterns unlike AR outputs, this lowered pass@1 scores significantly\. We replaced these with stop\-sequence truncation \(\[\\ndef, \\nclass, \\nif \_\_name\_\_\]\)\.
2. 2\.Infinite loops require sandboxing\.At low step counts \(S≤16S\\leq 16\), diffusion models frequently produce syntactically valid but non\-terminating code\. Without theSIGALRMtimeout, evaluation hangs indefinitely, a problem AR models rarely cause\.
3. 3\.Proxy metrics mislead for PPL\-mode benchmarks\.During development, we tried string\-matching the first 40 characters of the target as a proxy for PPL\-mode benchmarks\. The proxy and true metric disagreed substantially, producing misleading intermediate results\.
4. 4\.Tokenizer mismatch\.The drafter and refiner use different tokenizers\. Entropy scores must be mapped between tokenizations via character\-level offset alignment; naive token\-index transfer produces incorrect masks and unreliable evaluation\.

## 4Results

We organize results around evaluation findings rather than per\-dataset performance\. All numerical results are in Tables[1](https://arxiv.org/html/2606.27474#S4.T1)–[3](https://arxiv.org/html/2606.27474#S4.T3); we do not repeat numbers already in the tables except when interpreting them\.

### 4\.1What code benchmarks actually test: structural discovery

Table 1:Step\-wise scaling on generative benchmarks\. SpecRef sets a quality floor at low step counts \(S=8,16S=8,16\) where standalone diffusion fails\. All results usemicrosoft/phi\-2as drafter andGSAI\-ML/LLaDA\-8B\-Instructas refiner\.\* Note: The MATH evaluation was performed on a 5,000\-sample representative subset of the final test suite\.

The most striking pattern in Table[1](https://arxiv.org/html/2606.27474#S4.T1)is on code benchmarks\. Standalone LLaDA produces 0% valid code on HumanEval atS≤16S\\leq 16and only 8\.70% atS=64S=64\. But when warm\-started from an AR draft, SpecRef reaches 20\.12% at just 8 steps, without changing the diffusion model at all\.

This gap tells us something about what HumanEval and MBPP actually measure\. This suggests the diffusion model at 8B parameters is not entirely incapable of code logic; rather, it cannot discover the right syntactic structure \(indentation, brackets, function signatures\) from a fully masked starting point in a few steps\. Once that structure is provided, accuracy jumps immediately\. These benchmarks appear to conflate two distinct capabilities:*structural discovery*and*logical correctness*\. For diffusion models at low step counts, the structural component accounts for a large share of the failure\.

A near\-zero score on HumanEval does not mean a diffusion model “cannot code\.” It may mean the model cannot discover Python’s formatting conventions quickly enough\. Separating structural and logical evaluation \(e\.g\., providing scaffolds and measuring only logic fill\-in\) would give more diagnostic information\.

On MBPP the pattern is even clearer: SpecRef’s absolute gain expands to 25\.80% atS=64S=64\(33\.00% vs\. 7\.20%\)\. On GSM8K, SpecRef atS=64S=64reaches 54\.28%, outperforming both the standalone drafter \(47\.38%\) and refiner \(44\.81%\), suggesting complementary strengths on math reasoning\.

### 4\.2Refinement tension: when correction hurts

A second finding concerns what happens when the AR drafter is already accurate\. On BBH Boolean Expressions\(Suzgunet al\.,[2023](https://arxiv.org/html/2606.27474#bib.bib23)\), the drafter scores 82\.80%\. SpecRef improves over the standalone diffusion baseline at every step count but plateaus at 80\.40% forS≥32S\\geq 32, below the drafter itself\. The same pattern appears on MATH, where the baseline edges ahead of SpecRef atS=64S=64\(22\.32% vs\. 21\.38%\)\.

We call this*refinement tension*: the entropy\-based mask sometimes selects tokens that the drafter already had correct, and the refiner replaces them with worse alternatives\. The drafter’s entropy is a noisy signal of actual error; on saturated tasks, high entropy does not always mean the token is wrong\.

This phenomenon is invisible when evaluating either model in isolation\. It only appears in the pipeline, and it reveals a benchmark saturation ceiling: once a component model is near\-ceiling, aggregate pipeline scores can*decrease*\. For evaluation of any multi\-stage system \(not just SpecRef\), this argues for reporting component\-level scores alongside pipeline scores\. It also suggests that adaptive, per\-instance mask rate selection \(loweringkkwhen the drafter is already confident\) could mitigate the effect, though we leave this to future work\.

### 4\.3The log\-likelihood vs\. generative evaluation disconnect

Table 2:Likelihood \(PPL\) performance\. SpecRef in PPL mode matches the baseline because both use log\-likelihood scoring on the full sequence\.Table[2](https://arxiv.org/html/2606.27474#S4.T2)shows PPL\-mode evaluation on ARC\-Challenge and HellaSwag\. Two observations are worth noting\.

First, PPL\-mode is diffusion\-step\-independent\. The same score emerges regardless of step count, because log\-likelihood is computed in a single masked forward pass, not through iterative denoising\. This makes PPL\-mode evaluation4×4\\timescheaper on our cluster, but it also means PPL\-mode and generative evaluation measure fundamentally different things\. Whether a model can*score*a correct answer highly is not the same as whether it can*generate*that answer\.

Second, the AR drafter outperforms LLaDA on HellaSwag in PPL mode \(54\.94% vs\. 45\.62%\), yet on generative benchmarks \(Table[1](https://arxiv.org/html/2606.27474#S4.T1)\) the relationship between the two models is more complex and step\-dependent\. While these are different tasks, the pattern suggests that PPL\-mode rankings do not straightforwardly transfer to generative settings\. For diffusion models specifically, PPL\-mode may overstate or understate generation ability depending on the task\.

### 4\.4Latency\-quality tradeoffs

Table 3:Latency \(seconds per query\) on GSM8K, NVIDIA RTX 4090\. SpecRef times include both the AR drafting pass \(vLLM\) and the diffusion steps\. Because fewer diffusion steps are needed to match baseline quality, total wall\-clock time drops\.Table[3](https://arxiv.org/html/2606.27474#S4.T3)shows that SpecRefS=64S=64is both more accurate and2\.43×2\.43\\timesfaster than standalone LLaDA at the same step count\. AtS=16S=16SpecRef, it is slower \(the drafting pass dominates when diffusion steps are few\) but far more accurate\.

For evaluation methodology, the lesson is thatreporting only accuracy is insufficientfor comparing generation paradigms with different cost structures\. Step count, wall\-clock time, and FLOPs all affect the comparison\. Table[1](https://arxiv.org/html/2606.27474#S4.T1)shows accuracy swings of over 30 percentage points across step counts for the same model; a single\-step\-count comparison could be arbitrarily misleading\.

## 5Discussion: Lessons for Evaluating Hybrid Generation

#### 1\. Report evaluation protocol details exhaustively\.

Post\-processing choices alone can change code evaluation scores by large margins \(Section[3\.3](https://arxiv.org/html/2606.27474#S3.SS3)\)\. Sandbox timeouts, stop\-sequence selection, and tokenizer alignment details are not minor implementation notes; they are evaluation\-critical parameters\. We recommend treating them with the same care as hyperparameters\.

#### 2\. Benchmark scores conflate multiple capabilities\.

Code benchmarks test structural discovery and logical correctness simultaneously\. For AR models, this distinction rarely matters \(structure comes for free from left\-to\-right generation\), but for diffusion and hybrid models, it is the dominant factor\. Providing syntactic scaffolds as a controlled experiment could separate these components and give more diagnostic evaluations\.

#### 3\. Multi\-stage systems need component\-level reporting\.

Aggregate pipeline scores can mask component\-level regression \(the refinement tension effect\)\. We recommend reporting drafter\-only, refiner\-only, and pipeline scores together, along with an analysis of where the pipeline helps and where it hurts\.

#### 4\. PPL\-mode and generative evaluation are not interchangeable\.

They measure different aspects of model capability and can rank the same models differently\. Papers that use only one mode should be explicit about this limitation\.

#### 5\. Step count and compute budget must be reported\.

Diffusion model accuracy varies by 30\+ pp across step counts \(Table[1](https://arxiv.org/html/2606.27474#S4.T1)\)\. A score without a step count is incomplete; accuracy\-vs\-compute curves should replace single numbers\.

## 6Related Work

GEM\(Gehrmannet al\.,[2021](https://arxiv.org/html/2606.27474#bib.bib16)\)and HELM\(Lianget al\.,[2023](https://arxiv.org/html/2606.27474#bib.bib24)\)established shared evaluation infrastructure; LLM\-as\-judge methods\(Zhenget al\.,[2023](https://arxiv.org/html/2606.27474#bib.bib20)\)added scalable evaluation beyond reference matching; and benchmark contamination\(Xuet al\.,[2024](https://arxiv.org/html/2606.27474#bib.bib21)\)has motivated living benchmarks\.Deschenaux and Gulcehre \([2024](https://arxiv.org/html/2606.27474#bib.bib12)\)noted evaluation challenges for discrete diffusion but did not systematically document them, our work fills this gap\. Continuous\-space diffusion for text was introduced byLiet al\.\([2022](https://arxiv.org/html/2606.27474#bib.bib7)\); discrete extensions\(Austinet al\.,[2021a](https://arxiv.org/html/2606.27474#bib.bib6); Hoet al\.,[2020](https://arxiv.org/html/2606.27474#bib.bib4); Songet al\.,[2021](https://arxiv.org/html/2606.27474#bib.bib5)\)led to SEDD\(Louet al\.,[2024](https://arxiv.org/html/2606.27474#bib.bib8)\)and MDLM\(Sahooet al\.,[2024](https://arxiv.org/html/2606.27474#bib.bib22)\)\. Speculative decoding\(Leviathanet al\.,[2023](https://arxiv.org/html/2606.27474#bib.bib2); Chenet al\.,[2023](https://arxiv.org/html/2606.27474#bib.bib3)\)accelerates AR generation via draft\-then\-verify; SpecRef inverts this pattern, adapting warm\-start diffusion\(Menget al\.,[2022](https://arxiv.org/html/2606.27474#bib.bib10); Scholz and Turner,[2025](https://arxiv.org/html/2606.27474#bib.bib13)\)to discrete masked diffusion with a confidence\-aware corruption policy\.

## 7Conclusion

We used SpecRef as a lens to examine how standard benchmarks behave when generation is no longer purely autoregressive, surfacing four findings: code benchmarks conflate structural discovery with logical correctness; multi\-stage pipelines can degrade accuracy through refinement tension; log\-likelihood and generative evaluation are not interchangeable; and standard post\-processing silently breaks evaluation for non\-AR generators\. We hope these observations contribute to more diagnostic evaluation practices as generation architectures continue to diversify\.

## Limitations

Our evaluation uses a single drafter\-refiner pair \(Phi\-2 2\.7B \+ LLaDA 8B\); quantitative magnitudes are specific to this combination, though the qualitative findings likely generalize\. Phi\-2’s 2,048\-token context window prevented evaluation on long\-prompt benchmarks \(e\.g\., GPQA\)\. The fixed mask ratek=60%k=60\\%was not tuned per task; adaptive masking may reduce refinement tension\.

## Acknowledgement

This work was carried out at IIIT, Hyderabad\. We thank the insitute for logistics and compute resources\. For GPUs rentals, we used vast\.ai, we acknowledge the seamless access\. This work was funded by Qualcomm faculty award at IIIT, Hyderabad for funding the first author\. We thank Qualcomm for such unrestricted grant to support research\.

## Ethics Statement

SpecRef is a decoding strategy; it does not change training data or model capabilities\. It does, however, reduce the cost of generating text, which can amplify both beneficial and harmful uses\. Risks include:

- •Scaling misuse:faster generation can facilitate spam, disinformation, or automated harassment\.
- •Bias propagation:refinement can preserve biases from either model, and selective masking may ‘ "lock in" problematic phrasing from the draft\.
- •Over\-trust in refinement:users may assume “refined" outputs are more factual than they are\.

Mitigations include maintaining standard safety filters, auditing outputs for bias/toxicity, and using evaluation prompts that stress factuality and sensitive domains\.

## References

- Structured denoising diffusion models in discrete state\-spaces\.InProceedings of the 35th International Conference on Neural Information Processing Systems,NIPS ’21,Red Hook, NY, USA\.External Links:ISBN 9781713845393Cited by:[§1](https://arxiv.org/html/2606.27474#S1.p4.1),[§6](https://arxiv.org/html/2606.27474#S6.p1.1)\.
- J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. Sutton \(2021b\)Program synthesis with large language models\.External Links:2108\.07732,[Link](https://arxiv.org/abs/2108.07732)Cited by:[1st item](https://arxiv.org/html/2606.27474#S1.I1.i1.p1.1),[§3\.2](https://arxiv.org/html/2606.27474#S3.SS2.SSS0.Px1.p1.1)\.
- C\. Chen, S\. Borgeaud, G\. Irving, J\. Lespiau, L\. Sifre, and J\. Jumper \(2023\)Accelerating large language model decoding with speculative sampling\.Note:2302\.01318External Links:[Link](https://arxiv.org/abs/2302.01318)Cited by:[§1](https://arxiv.org/html/2606.27474#S1.p4.1),[§6](https://arxiv.org/html/2606.27474#S6.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. Zaremba \(2021\)Evaluating large language models trained on code\.External Links:2107\.03374,[Link](https://arxiv.org/abs/2107.03374)Cited by:[1st item](https://arxiv.org/html/2606.27474#S1.I1.i1.p1.1),[§1](https://arxiv.org/html/2606.27474#S1.p4.1),[§3\.2](https://arxiv.org/html/2606.27474#S3.SS2.SSS0.Px1.p1.1)\.
- J\. K\. Christopher, B\. R\. Bartoldson, T\. Ben\-Nun, M\. Cardei, B\. Kailkhura, and F\. Fioretto \(2025\)Speculative diffusion decoding: accelerating language generation through diffusion\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 12042–12059\.External Links:[Link](https://aclanthology.org/2025.naacl-long.601/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.601),ISBN 979\-8\-89176\-189\-6Cited by:[§1](https://arxiv.org/html/2606.27474#S1.p3.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.Note:2110\.14168External Links:[Link](https://arxiv.org/abs/2110.14168)Cited by:[§1](https://arxiv.org/html/2606.27474#S1.p4.1),[§3\.2](https://arxiv.org/html/2606.27474#S3.SS2.SSS0.Px2.p1.1)\.
- J\. Deschenaux and C\. Gulcehre \(2024\)Promises, outlooks and challenges of diffusion language modeling\.Note:arXiv: 2406\.11473External Links:[Link](https://arxiv.org/abs/2406.11473)Cited by:[§6](https://arxiv.org/html/2606.27474#S6.p1.1)\.
- S\. Gehrmann, T\. Adewumi, K\. Aggarwal, P\. S\. Ammanamanchi, A\. Aremu, A\. Bosselut, K\. R\. Chandu, M\. Clinciu, D\. Das, K\. Dhole, W\. Du, E\. Durmus, O\. Dušek, C\. C\. Emezue, V\. Gangal, C\. Garbacea, T\. Hashimoto, Y\. Hou, Y\. Jernite, H\. Jhamtani, Y\. Ji, S\. Jolly, M\. Kale, D\. Kumar, F\. Ladhak, A\. Madaan, M\. Maddela, K\. Mahajan, S\. Mahamood, B\. P\. Majumder, P\. H\. Martins, A\. McMillan\-Major, S\. Mille, E\. van Miltenburg, M\. Nadeem, S\. Narayan, V\. Nikolaev, A\. Niyongabo Rubungo, S\. Osei, A\. Parikh, L\. Perez\-Beltrachini, N\. R\. Rao, V\. Raunak, J\. D\. Rodriguez, S\. Santhanam, J\. Sedoc, T\. Sellam, S\. Shaikh, A\. Shimorina, M\. A\. Sobrevilla Cabezudo, H\. Strobelt, N\. Subramani, W\. Xu, D\. Yang, A\. Yerukola, and J\. Zhou \(2021\)The GEM benchmark: natural language generation, its evaluation and metrics\.InProceedings of the First Workshop on Natural Language Generation, Evaluation, and Metrics \(GEM\),A\. Bosselut, E\. Durmus, V\. P\. Gangal, S\. Gehrmann, Y\. Jernite, L\. Perez\-Beltrachini, S\. Shaikh, and W\. Xu \(Eds\.\),Online,pp\. 96–120\.External Links:[Link](https://aclanthology.org/2021.gem-1.10/),[Document](https://dx.doi.org/10.18653/v1/2021.gem-1.10)Cited by:[§1](https://arxiv.org/html/2606.27474#S1.p4.1),[§6](https://arxiv.org/html/2606.27474#S6.p1.1)\.
- J\. Ho, A\. Jain, and P\. Abbeel \(2020\)Denoising diffusion probabilistic models\.InProceedings of the 34th International Conference on Neural Information Processing Systems,NIPS ’20,Red Hook, NY, USA\.External Links:ISBN 9781713829546Cited by:[§6](https://arxiv.org/html/2606.27474#S6.p1.1)\.
- Z\. Horvitz, A\. Patel, C\. Callison\-Burch, Z\. Yu, and K\. McKeown \(2024\)ParaGuide: guided diffusion paraphrasers for plug\-and\-play textual style transfer\.InProceedings of the Thirty\-Eighth AAAI Conference on Artificial Intelligence and Thirty\-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence,AAAI’24/IAAI’24/EAAI’24\.External Links:ISBN 978\-1\-57735\-887\-9,[Link](https://doi.org/10.1609/aaai.v38i16.29780),[Document](https://dx.doi.org/10.1609/aaai.v38i16.29780)Cited by:[§1](https://arxiv.org/html/2606.27474#S1.p3.1)\.
- I\. Labs, S\. Khanna, S\. Kharbanda, S\. Li, H\. Varma, E\. Wang, S\. Birnbaum, Z\. Luo, Y\. Miraoui, A\. Palrecha, S\. Ermon, A\. Grover, and V\. Kuleshov \(2025\)Mercury: ultra\-fast language models based on diffusion\.Note:arXiv: 2506\.17298External Links:[Link](https://arxiv.org/abs/2506.17298)Cited by:[§1](https://arxiv.org/html/2606.27474#S1.p2.1)\.
- Y\. Leviathan, M\. Kalman, and Y\. Matias \(2023\)Fast inference from transformers via speculative decoding\.InProceedings of the 40th International Conference on Machine Learning,ICML’23\.Cited by:[§1](https://arxiv.org/html/2606.27474#S1.p4.1),[§6](https://arxiv.org/html/2606.27474#S6.p1.1)\.
- X\. L\. Li, J\. Thickstun, I\. Gulrajani, P\. Liang, and T\. B\. Hashimoto \(2022\)Diffusion\-lm improves controllable text generation\.InProceedings of the 36th International Conference on Neural Information Processing Systems,NIPS ’22,Red Hook, NY, USA\.External Links:ISBN 9781713871088Cited by:[§6](https://arxiv.org/html/2606.27474#S6.p1.1)\.
- P\. Liang, R\. Bommasani, T\. Lee, D\. Tsipras, D\. Soylu, M\. Yasunaga, Y\. Zhang, D\. Narayanan, Y\. Wu, A\. Kumar, B\. Newman, B\. Yuan, B\. Yan, C\. Zhang, C\. Cosgrove, C\. D\. Manning, C\. Ré, D\. Acosta\-Navas, D\. A\. Hudson, E\. Zelikman, E\. Durmus, F\. Ladhak, F\. Rong, H\. Ren, H\. Yao, J\. Wang, K\. Santhanam, L\. Orr, L\. Zheng, M\. Yuksekgonul, M\. Suzgun, N\. Kim, N\. Guha, N\. Chatterji, O\. Khattab, P\. Henderson, Q\. Huang, R\. Chi, S\. M\. Xie, S\. Santurkar, S\. Ganguli, T\. Hashimoto, T\. Icard, T\. Zhang, V\. Chaudhary, W\. Wang, X\. Li, Y\. Mai, Y\. Zhang, and Y\. Koreeda \(2023\)Holistic evaluation of language models\.Note:2211\.09110External Links:[Link](https://arxiv.org/abs/2211.09110)Cited by:[§1](https://arxiv.org/html/2606.27474#S1.p4.1),[§6](https://arxiv.org/html/2606.27474#S6.p1.1)\.
- A\. Lou, C\. Meng, and S\. Ermon \(2024\)Discrete diffusion modeling by estimating the ratios of the data distribution\.Note:arXiv: 2310\.16834External Links:[Link](https://arxiv.org/abs/2310.16834)Cited by:[§1](https://arxiv.org/html/2606.27474#S1.p4.1),[§6](https://arxiv.org/html/2606.27474#S6.p1.1)\.
- C\. Meng, Y\. He, Y\. Song, J\. Song, J\. Wu, J\. Zhu, and S\. Ermon \(2022\)SDEdit: guided image synthesis and editing with stochastic differential equations\.Note:arXiv: 2108\.01073External Links:[Link](https://arxiv.org/abs/2108.01073)Cited by:[§1](https://arxiv.org/html/2606.27474#S1.p3.1),[§6](https://arxiv.org/html/2606.27474#S6.p1.1)\.
- S\. Nie, F\. Zhu, Z\. You, X\. Zhang, J\. Ou, J\. Hu, J\. Zhou, Y\. Lin, J\. Wen, and C\. Li \(2025\)Large language diffusion models\.External Links:2502\.09992,[Link](https://arxiv.org/abs/2502.09992)Cited by:[§1](https://arxiv.org/html/2606.27474#S1.p2.1),[§1](https://arxiv.org/html/2606.27474#S1.p4.1),[§1](https://arxiv.org/html/2606.27474#S1.p5.1),[§2\.2](https://arxiv.org/html/2606.27474#S2.SS2.p1.8),[§3\.2](https://arxiv.org/html/2606.27474#S3.SS2.SSS0.Px3.p1.1)\.
- S\. S\. Sahoo, M\. Arriola, Y\. Schiff, A\. Gokaslan, E\. Marroquin, J\. T\. Chiu, A\. Rush, and V\. Kuleshov \(2024\)Simple and effective masked diffusion language models\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[§1](https://arxiv.org/html/2606.27474#S1.p2.1),[§6](https://arxiv.org/html/2606.27474#S6.p1.1)\.
- J\. Scholz and R\. E\. Turner \(2025\)Warm starts accelerate conditional diffusion\.Note:arXiv: 2507\.09212External Links:[Link](https://arxiv.org/abs/2507.09212)Cited by:[§1](https://arxiv.org/html/2606.27474#S1.p3.1),[§6](https://arxiv.org/html/2606.27474#S6.p1.1)\.
- J\. Song, C\. Meng, and S\. Ermon \(2022\)Denoising diffusion implicit models\.Note:arXiv: 2010\.02502External Links:[Link](https://arxiv.org/abs/2010.02502)Cited by:[§2\.3](https://arxiv.org/html/2606.27474#S2.SS3.p2.5)\.
- Y\. Song, J\. Sohl\-Dickstein, D\. P\. Kingma, A\. Kumar, S\. Ermon, and B\. Poole \(2021\)Score\-based generative modeling through stochastic differential equations\.Note:arXiv: 2011\.13456External Links:[Link](https://arxiv.org/abs/2011.13456)Cited by:[§6](https://arxiv.org/html/2606.27474#S6.p1.1)\.
- M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. Le, E\. Chi, D\. Zhou, and J\. Wei \(2023\)Challenging BIG\-bench tasks and whether chain\-of\-thought can solve them\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 13003–13051\.External Links:[Link](https://aclanthology.org/2023.findings-acl.824/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.824)Cited by:[2nd item](https://arxiv.org/html/2606.27474#S1.I1.i2.p1.1),[§1](https://arxiv.org/html/2606.27474#S1.p4.1),[§4\.2](https://arxiv.org/html/2606.27474#S4.SS2.p1.2)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.InProceedings of the 31st International Conference on Neural Information Processing Systems,NIPS’17,Red Hook, NY, USA,pp\. 6000–6010\.External Links:ISBN 9781510860964Cited by:[§1](https://arxiv.org/html/2606.27474#S1.p4.1)\.
- R\. Xu, Z\. Wang, R\. Fan, and P\. Liu \(2024\)Benchmarking benchmark leakage in large language models\.Note:arXiv: 2404\.18824External Links:[Link](https://arxiv.org/abs/2404.18824)Cited by:[§1](https://arxiv.org/html/2606.27474#S1.p4.1),[§6](https://arxiv.org/html/2606.27474#S6.p1.1)\.
- J\. Ye, Z\. Xie, L\. Zheng, J\. Gao, Z\. Wu, X\. Jiang, Z\. Li, and L\. Kong \(2025\)Dream 7b: diffusion large language models\.Note:arXiv: 2508\.15487External Links:[Link](https://arxiv.org/abs/2508.15487)Cited by:[§1](https://arxiv.org/html/2606.27474#S1.p2.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.InProceedings of the 37th International Conference on Neural Information Processing Systems,NIPS ’23,Red Hook, NY, USA\.Cited by:[§1](https://arxiv.org/html/2606.27474#S1.p4.1),[§6](https://arxiv.org/html/2606.27474#S6.p1.1)\.