Roll Out and Roll Back: Diffusion LLMs are Their Own Efficiency Teachers

# Roll Out and Roll Back: Diffusion LLMs are Their Own Efficiency Teachers
Source: [https://arxiv.org/html/2605.16941](https://arxiv.org/html/2605.16941)
Fanqin Zeng∗, Feng Hong∗, Geng Yu, Huangjie Zheng, Xiaofeng Cao, Ya Zhang, Bo Han, Yanfeng Wang, Jiangchao Yao†Fanqin Zeng and Feng Hong contributed equally to this work \(marked by∗\*\)\. The corresponding author is Jiangchao Yao \(marked by†\\dagger\), and the correspondence email is Sunarker@sjtu\.edu\.cn\.Fanqin Zeng, Feng Hong, Geng Yu, Ya Zhang, Yanfeng Wang, Jiangchao Yao are with Shanghai Jiao Tong University, Shanghai, China\. Huangjie Zheng is with Apple MLR, Cupertino, CA, USA\. Xiaofeng Cao is with Tongji University, Shanghai, China\. Bo Han is with Hong Kong Baptist University, Hong Kong, China, and RIKEN, Japan\.

###### Abstract

Diffusion Large Language Models \(DLLMs\) promise fast parallel generation, yet open\-source DLLMs still face a severe quality–speed trade\-off: accelerating decoding by revealing multiple tokens often causes substantial quality degradation\. We attribute this dilemma to a train–inference mismatch amplified by irreversible decoding\. While training reconstructs tokens from randomly corrupted states, efficient inference requires an adaptive denoising order, where easier tokens are revealed earlier and context\-dependent ones are deferred\. This view motivates two complementary methods: an inference\-time method that makes parallel decoding revokable, and a training\-time extension that distills the reliable order exposed by this revokable process\. Accordingly, we first propose Wide\-In, Narrow\-Out \(WINO\), a training\-free decoding algorithm that enables revokable parallel generation\. WINO aggressively drafts multiple tokens, verifies generated tokens with enriched global context, and re\-masks unreliable ones for later refinement\. Building on this discovered order, we further introduce WINO\+, which injects the verified denoising trajectories produced by WINO into model parameters, aligning training with efficient inference\. Experiments on LLaDA and MMaDA show that WINO improves both quality and efficiency, while WINO\+ further strengthens this progression\. On GSM8K, WINO improves accuracy from 73\.24% to 75\.82% with a 6\.10×\\timesstep reduction, and WINO\+ further achieves 76\.58% with a 6\.83×\\timesreduction\. On Flickr30K, WINO\+ reaches a 16\.22×\\timesstep reduction with improved CIDEr\. These results demonstrate that DLLMs can serve as their own efficiency teachers by first discovering reliable denoising orders through revokable decoding and then learning to follow them for faster generation\. Code is available at[https://github\.com/Feng\-Hong/WINO\-DLLM/tree/WINO\-plus](https://github.com/Feng-Hong/WINO-DLLM/tree/WINO-plus)\.

## IIntroduction

Autoregressive \(AR\) large language models\[radford2018improving,radford2019language\], such as the GPT series\[openai2022chatgpt\], have shown impressive performance in a wide range of language tasks\. However, their foundational token\-by\-token generation mechanism introduces inherent limitations, including severe inference latency, susceptibility to error propagation\[DBLP:journals/corr/abs\-2310\-12397,DBLP:journals/corr/abs\-2310\-08118\], and challenges in maintaining global coherence\[mei2025surveycontextengineeringlarge\]\. In response, Diffusion Large Language Models \(DLLMs\) have emerged as a compelling non\-autoregressive alternative, architected to overcome these bottlenecks\. By generating tokens simultaneously\[DBLP:conf/nips/LiTGLH22\], DLLMs theoretically enable massive inference acceleration, while their native bidirectional attention offers improved consistency\. The immense potential of DLLMs has also been showcased by proprietary, closed\-source systems \(*e\.g\.*, Mercury Coder\[inceptionlabs2025mercury\]and Gemini Diffusion\[deepmind2025geminidiffusion\]\), which have demonstrated astonishing speeds exceeding 1,000 tokens per second, serving as a powerful proof\-of\-concept\.

Despite this promise, the performance of open\-source DLLMs has been still disappointing\. One critical bottleneck is that they are caught in a severe quality\-speed trade\-off dilemma\. Specifically, to achieve high\-quality output, these models are often forced to decode slowly, generating just one token at a time, which negates their primary architectural advantage\. As shown in Fig\.[1](https://arxiv.org/html/2605.16941#S1.F1), attempting to accelerate inference by generating multiple tokens in parallel invariably leads to a significant degradation in output quality\[DBLP:journals/corr/abs\-2502\-09992,DBLP:journals/corr/abs\-2506\-00413\]\. This stark trade\-off has largely prevented the open\-source DLLMs from becoming a viable, high\-performance alternative to their AR counterparts\.

We attribute this trade\-off to a train–inference mismatch in DLLMs, which is further amplified by the irreversibility of standard decoding\[DBLP:conf/nips/SahooASGMCRK24,DBLP:conf/iclr/OuNXZSLL25\]\. During training, masked diffusion models reconstruct tokens from randomly corrupted states, where the recovery order is implicitly randomized\. During inference, however, generation follows a progressive denoising trajectory, in which different tokens have different context requirements and thus a natural order preference\. Easier tokens should be revealed earlier, while more context\-dependent tokens should be delayed until sufficient context is available\. Standard decoding cannot correct violations of this order preference\. Once a token is decoded, it is fixed and cannot be revised, even when richer context becomes available later\. As a result, tokens revealed too early under insufficient context may introduce errors that are preserved and propagated throughout the remaining trajectory\. Thus, the quality\-speed trade\-off stems from revealing tokens in a suboptimal order under an irreversible decoding process\.

To address this problem from the inference side, we first propose Wide\-In, Narrow\-Out \(WINO\), a novel decoding algorithm that enables revokable decoding for DLLMs\. WINO employs a draft\-and\-verify procedure that operates in parallel\. At each step, a draft module aggressively proposes multiple new tokens based on a lenient threshold \(the “Wide\-In”\)\. Concurrently, a verify module leverages the newly enriched global context to re\-evaluate all previously generated tokens\. Any token that fails a stricter verification check is re\-masked for refinement in a future step \(the “Narrow\-Out”\)\. This mechanism brings two merits: 1\) it breaks the irreversibility of conventional decoding, allowing tokens revealed too early to be revised when richer context becomes available; 2\) it permits more aggressive token generation in each diffusion step, yielding faster inference without sacrificing quality\. Importantly, WINO requires no additional training and can be directly applied to existing DLLMs\.

![Refer to caption](https://arxiv.org/html/2605.16941v1/x1.png)Figure 1:Demonstration of speedup and performance improvement of WINO over standard decoding and naive parallel sampling evaluated on GSM8K with LLaDA and Flickr30K with MMaDA\. The standard decoding unmasks 1 token per decoding step, while the naive parallel sampling unmasksM\(\>1\)M\(\>1\)tokens per decoding step\. We setM=4M=4for GSM8K andM=8M=8for Flickr30K\.While WINO mitigates order\-induced errors during inference, its draft–verify–fallback process also exposes a trajectory\-level signal that can reduce the training\-side mismatch\. Each fallback indicates that a token has likely been revealed too early, while the finalization step of a token reflects a more suitable stage for revealing it\. Based on this signal, we propose WINO\+, a trajectory\-injection framework that transfers the WINO\-derived order into model parameters\. WINO\+ runs WINO offline, extracts token\-level finalization steps, and constructs trajectory\-guided training samples\. Instead of reconstructing randomly selected masked tokens, WINO\+ trains the model to reproduce the verified order: earlier\-finalized tokens are supervised earlier, while later\-finalized tokens remain masked until their corresponding trajectory step\. This replaces random reconstruction with trajectory\-ordered denoising, making training more aligned with efficient inference\.

We conduct extensive experiments on both language and vision\-language benchmarks using representative open\-source DLLMs, including LLaDA\[DBLP:journals/corr/abs\-2502\-09992\]and MMaDA\[DBLP:journals/corr/abs\-2505\-15809\]\. The results show a clear progression from WINO to WINO\+\. WINO accelerates decoding at inference time by making parallel generation revokable\. WINO\+ further improves efficiency from the training side by teaching the model to follow the verified generation order discovered by WINO, reducing the dependence on online fallback during inference\. For example, on GSM8K, WINO improves accuracy from 73\.24% to 75\.82% with a 6\.10×\\timesreduction in decoding steps, while WINO\+ further improves accuracy to 76\.58% with a 6\.83×\\timesstep reduction\. On Flickr30K, WINO achieves a 10\.05×\\timesstep reduction, and WINO\+ further improves CIDEr to 63\.38 with a 16\.22×\\timesstep reduction\. These results show that DLLMs can serve as their own efficiency teachers: their revokable decoding process first improves inference directly, and then produces verified trajectories from which the model can learn a better denoising order\. Our contributions are summarized as follows:

- •We identify a train–inference mismatch behind the quality\-speed trade\-off of DLLMs: standard training relies on randomly corrupted states, whereas efficient inference requires an adaptive denoising order\. Under irreversible decoding, violations of this order preference lead to accumulated errors\.
- •We propose WINO, a training\-free and plug\-and\-play decoding algorithm\. Through a parallel draft–verify–fallback mechanism, WINO enables aggressive token generation while revising prematurely revealed tokens, thereby improving both decoding speed and generation quality\.
- •We further propose WINO\+, a trajectory\-injection framework that transfers the verified generation order discovered by WINO into model parameters\. WINO\+ extracts token\-level finalization steps from WINO trajectories and trains the model with trajectory\-ordered denoising instead of random reconstruction\.
- •Extensive experiments on language and vision\-language benchmarks show that WINO consistently accelerates inference while preserving or improving quality, and WINO\+ further enhances both efficiency and performance by learning from WINO\-derived trajectories\.

## IIRelated Work

### II\-ADiffusion\-based Language Models

Diffusion models\[DBLP:journals/corr/Sohl\-DicksteinW15,DBLP:conf/nips/HoJA20,DBLP:conf/iclr/0011SKKEP21\], originally popularized in image generation\[DBLP:conf/cvpr/RombachBLEO22,DBLP:conf/icml/NicholDRSMMSC22,DBLP:conf/nips/SahariaCSLWDGLA22\], have recently gained attention as an alternative to autoregressive language models \(ARLMs\) for text generation\. Early diffusion work\[DBLP:journals/corr/Sohl\-DicksteinW15\]first studied this expansion from continuous domain to discrete domain\. Subsequently, D3PM\[DBLP:conf/nips/AustinJHTB21\]provides a general framework which models the diffusion forward process as a discrete state Markov chain defined by the multiplication of specific transition matrices over discrete time steps\. A subsequent CTMC\-based approach\[DBLP:conf/nips/CampbellBBRDD22\]later expands D3PM to a continuous time setting, utilizing the theory of continuous time Markov chain\(CTMC\)\. More recently, research on masked diffusion models\(MDMs\)\[DBLP:conf/nips/ShiHWDT24\]derived from the absorbing state diffusion in D3PM has shown promising results both in small\-scale models \(*e\.g\.*, MDLM\[DBLP:conf/nips/SahooASGMCRK24\]and RADD\[DBLP:conf/iclr/OuNXZSLL25\]\) and large\-scale implementations \(*e\.g\.*, LLaDA\[DBLP:journals/corr/abs\-2502\-09992\]and Dream\[dream2025\]\)\. Extending this line of work, MMaDA\[DBLP:journals/corr/abs\-2505\-15809\]introduces a novel class of multimodal large diffusion models featuring a shared probabilistic formulation and a modality\-agnostic architecture\.

### II\-BDLLM Acceleration Techniques

The existing acceleration study for DLLMs falls into two directions: KV cache and sampling compression\. The former targets to build the KV cache for DLLMs due to its bidirectional full attention mechanism, unlike the causal attention of ARLMs\. Typical works like Block Diffusion\[DBLP:conf/iclr/ArriolaGCYQHSK25\], Fast\-dLLM\-cache\[DBLP:journals/corr/abs\-2505\-22618\]and dLLM\-cache\[DBLP:journals/corr/abs\-2506\-06295\]respectively explore different caching mechanisms, which shows promising performance for speedup\. Note that this direction is out of the scope of our work here\. The latter direction focuses on optimizing the sampling process itself\. For the classic low\-confidence remasking strategy, several works have introduced novel sampling strategies to dynamically adjust the number of tokens predicted in parallel, thereby improving inference efficiency\. Fast\-dLLM\-parallel\[DBLP:journals/corr/abs\-2505\-22618\]adopts a straightforward approach by selecting tokens with confidence scores exceeding a predefined threshold\. Meanwhile, the entropy\-bounded \(EB\) sampler\[DBLP:journals/corr/abs\-2505\-24857\], as a drop\-in replacement for conventional samplers, leverages an entropy\-based unmasking procedure to dynamically decode multiple tokens per step while maintaining a predefined error tolerance\. Although our WINO brings the acceleration promise due to sampling compression, different from these works, we explore to address the inherent limitation of standard decoding in DLLMs\.

## IIIPreliminary: Decoding Process for DLLMs

Given a promptXX, a DLLM generates a responseY=\[y1,y2,…,yL\]Y=\[y\_\{1\},y\_\{2\},\\ldots,y\_\{L\}\]with a pre\-defined response lengthLL\. The response sequence is initialized as all special mask tokens,Y\(0\)=\[\[MASK\],\[MASK\],…,\[MASK\]\]Y^\{\(0\)\}=\[\\mathchoice\{\\hbox to40\.36pt\{\\vbox to14\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 20\.17781pt\\lower\-4\.9pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-19\.77782pt\}\{\-4\.5pt\}\{39\.55563pt\}\{14\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-17\.77782pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\displaystyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to40\.36pt\{\\vbox to14\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 20\.17781pt\\lower\-4\.9pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-19\.77782pt\}\{\-4\.5pt\}\{39\.55563pt\}\{14\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-17\.77782pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\textstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to32\.96pt\{\\vbox to11\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-4\.15pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-16\.07994pt\}\{\-3\.75pt\}\{32\.15988pt\}\{11\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-14\.07994pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to28\.5pt\{\\vbox to9\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-3\.65pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-13\.85088pt\}\{\-3\.25pt\}\{27\.70175pt\}\{9\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-11\.85088pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptscriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\},\\mathchoice\{\\hbox to40\.36pt\{\\vbox to14\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 20\.17781pt\\lower\-4\.9pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-19\.77782pt\}\{\-4\.5pt\}\{39\.55563pt\}\{14\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-17\.77782pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\displaystyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to40\.36pt\{\\vbox to14\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 20\.17781pt\\lower\-4\.9pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-19\.77782pt\}\{\-4\.5pt\}\{39\.55563pt\}\{14\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-17\.77782pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\textstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to32\.96pt\{\\vbox to11\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-4\.15pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-16\.07994pt\}\{\-3\.75pt\}\{32\.15988pt\}\{11\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-14\.07994pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to28\.5pt\{\\vbox to9\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-3\.65pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-13\.85088pt\}\{\-3\.25pt\}\{27\.70175pt\}\{9\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-11\.85088pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptscriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\},\\ldots,\\mathchoice\{\\hbox to40\.36pt\{\\vbox to14\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 20\.17781pt\\lower\-4\.9pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-19\.77782pt\}\{\-4\.5pt\}\{39\.55563pt\}\{14\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-17\.77782pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\displaystyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to40\.36pt\{\\vbox to14\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 20\.17781pt\\lower\-4\.9pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-19\.77782pt\}\{\-4\.5pt\}\{39\.55563pt\}\{14\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-17\.77782pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\textstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to32\.96pt\{\\vbox to11\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-4\.15pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-16\.07994pt\}\{\-3\.75pt\}\{32\.15988pt\}\{11\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-14\.07994pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to28\.5pt\{\\vbox to9\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-3\.65pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-13\.85088pt\}\{\-3\.25pt\}\{27\.70175pt\}\{9\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-11\.85088pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptscriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\]\. The decoding process iteratively refines the response sequenceY\(k\)Y^\{\(k\)\}over a total ofKKdenoising steps\. In the following, we detail the case ofK=LK=L\(*i\.e\.*, decoding one token per step\), as existing models typically achieve optimal performance under this setting\[DBLP:journals/corr/abs\-2502\-09992\]\.

At stepkk, the goal of decoding is to refine the sequenceY\(k−1\)Y^\{\(k\-1\)\}intoY\(k\)Y^\{\(k\)\}\. Given the token vocabularyVVand the model parameterized withθ\\theta, the model estimates the probability distribution over the response sequence aspθ​\(Y^\|X,Y\(k−1\)\)p\_\{\\theta\}\(\\hat\{Y\}\|X,Y^\{\(k\-1\)\}\)\. As a common example, in high\-confidence greedy decoding,Y\(k\)Y^\{\(k\)\}is obtained by unmasking the most confident\[MASK\]token based onY\(k−1\)Y^\{\(k\-1\)\},*i\.e\.*,

l\(k\)=arg⁡maxl∈\{l\|yl\(k−1\)=\[MASK\]\}\(maxv∈V⁡pθ​\(y^l=v\|X,Y\(k−1\)\)\),yl\(k\)=\{arg​maxv∈V⁡pθ​\(y^l=v\|X,Y\(k−1\)\),if​l=l\(k\),yl\(k−1\),otherwise,∀l∈\{1,2,…,L\}\.\\begin\{array\}\[\]\{@\{\}l@\{\\;\}c@\{\\;\}l@\{\}\}l^\{\(k\)\}&=&\\displaystyle\\mathop\{\\arg\\max\}\_\{l\\in\\\{l\|y^\{\(k\-1\)\}\_\{l\}=\\mathchoice\{\\hbox to32\.96pt\{\\vbox to11\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-4\.15pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-16\.07994pt\}\{\-3\.75pt\}\{32\.15988pt\}\{11\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-14\.07994pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\displaystyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to32\.96pt\{\\vbox to11\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-4\.15pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-16\.07994pt\}\{\-3\.75pt\}\{32\.15988pt\}\{11\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-14\.07994pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\textstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to22\.22pt\{\\vbox to9\.7pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-3\.62497pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-10\.71112pt\}\{\-3\.22498pt\}\{21\.42224pt\}\{8\.89996pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-8\.71112pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to17\.24pt\{\\vbox to8\.3pt\{\\pgfpicture\\makeatletter\\hbox\{\\quad\\lower\-3\.275pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-8\.22221pt\}\{\-2\.875pt\}\{16\.44443pt\}\{7\.5pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-6\.22221pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptscriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\\\}\}\\left\(\\max\_\{v\\in V\}p\_\{\\theta\}\(\\hat\{y\}\_\{l\}=v\|X,Y^\{\(k\-1\)\}\)\\right\),\\\\\[3\.00003pt\] y^\{\(k\)\}\_\{l\}&=&\\begin\{cases\}\\operatorname\*\{arg\\,max\}\\limits\_\{v\\in V\}p\_\{\\theta\}\(\\hat\{y\}\_\{l\}=v\|X,Y^\{\(k\-1\)\}\),&\\text\{if \}l=l^\{\(k\)\},\\\\ y^\{\(k\-1\)\}\_\{l\},&\\text\{otherwise\},\\end\{cases\}\\\\\[\-1\.00006pt\] &&\\hskip 2\.5pt\\forall l\\in\\\{1,2,\\dots,L\\\}\.\\end\{array\}\(1\)
After completing allKKdecoding steps, the final generated response isY=Y\(K\)Y=Y^\{\(K\)\}\. Existing DLLMs, such as LLaDA\[DBLP:journals/corr/abs\-2502\-09992\]and MMaDA\[DBLP:journals/corr/abs\-2505\-15809\], can also accelerate the decoding process via naive parallel sampling by generating multiple tokens \(*e\.g\.*, 2 or 4\) per step\. However, empirical results reveal that such strategies often result in substantial performance degradation, limiting their practical effectiveness despite the computational speedup from using fewer decoding steps\[DBLP:journals/corr/abs\-2502\-09992\]\.

Semi\-Autoregressive Diffusion Decoding\.This strategy is widely adopted by DLLMs like LLaDA\[DBLP:journals/corr/abs\-2502\-09992\]and MMaDA\[DBLP:journals/corr/abs\-2505\-15809\], which involves splitting the response sequence into multiple blocks and decoding them sequentially from left to right\. Within each block, the typical diffusion decoding strategy described above is applied\.

We also provide the objective function for standard DLLM training in Appendix A as background reference\.

## IVRoll Out: WINO Algorithm

### IV\-AKey Limitation of Decoding Process for DLLMs

While architecturally suited for parallelism, DLLMs still suffer from a severe quality\-speed trade\-off in effective multi\-token decoding\. As discussed in the Introduction, this trade\-off stems from a train–inference mismatch: training reconstructs tokens from randomly corrupted states, whereas efficient inference requires an adaptive denoising order\. During progressive denoising, different tokens have different context requirements\. Aggressive parallel decoding may therefore reveal context\-dependent tokens too early, before sufficient context is available to support reliable prediction\.

This problem is further amplified by the irreversibility of standard decoding\. Once a token is decoded, it is fixed and cannot be revised, even when richer context becomes available later\. As a result, prematurely revealed tokens may be preserved and propagated through the remaining trajectory, degrading quality as decoding parallelism increases\.

To address this problem from the inference side, we propose to break the irreversibility of standard decoding and introduce*revokable decoding*\. The key idea is to allow DLLMs to generate multiple tokens aggressively while retaining the ability to re\-evaluate previously generated tokens under newly enriched global context\. As more context emerges during generation, the model can correct its preliminary predictions\. Such a mechanism effectively addresses the core conflict by marrying the efficiency of parallel generation with the accuracy of context\-driven corrections\.

![Refer to caption](https://arxiv.org/html/2605.16941v1/x2.png)Figure 2:An overview of WINO and illustration of our designed attention mask\. The green squares denote 1, the grey squares denote 0, and “Pos ID” is short for position ID\. Verified tokens refer to tokens in the promptXXor previously decoded blocks\. Draft tokens denote tokens in the current block that are unmasked up to the current decoding step\.\[MASK\]\(shadow draft\) refer to tokens in the shadow block whose position IDs correspond to the draft tokens while\[MASK\]\(shadow mask\) refer to the remaining tokens in the shadow block\.
### IV\-BIterative Refinement via Parallel Draft\-and\-Verify

Motivated by the above analysis and the design intuition, we propose a parallel Draft\-and\-Verify framework to enable revokable decoding for more efficient and higher\-quality generation in DLLMs\.

Specifically, our framework performs two modules in parallel at each decoding step: 1\) Draft: aggressively unmasks multiple\[MASK\]tokens into candidate meaningful tokens; 2\) Verify: evaluates all currently unmasked tokens and re\-masks those deemed low\-quality for further refinement\. We adopt the most common and general semi\-autoregressive decoding paradigm to present our method\. When the block length equals the generation length, this setting becomes equivalent to full diffusion decoding over the entire sequence\.

#### IV\-B1Drafting

We denote the entire sequence asY=\[Yleft,Ycur,Yright\]Y=\[Y\_\{\\text\{left\}\},\{Y\}\_\{\\text\{cur\}\},Y\_\{\\text\{right\}\}\], whereYleftY\_\{\\text\{left\}\}contains the promptXXand the previously decoded blocks,Ycur=\[ycur,1,…,ycur,Lb\]\{Y\}\_\{\\text\{cur\}\}=\[\{y\}\_\{\\text\{cur\},1\},\\ldots,\{y\}\_\{\\text\{cur\},L\_\{b\}\}\]represents the current block being decoded, andYrightY\_\{\\text\{right\}\}denotes the remaining blocks to be decoded\. Here,LbL\_\{b\}is the block length\. At thekk\-th decoding step, instead of decoding a fixed number of tokens, we perform aggressive multi\-token parallel decoding based on a confidence thresholdτ1\\tau\_\{1\}:

ycur,l\(k\)=arg​maxv∈V⁡pθ​\(y^cur,l=v∣Y\),if​maxv∈V⁡pθ​\(y^cur,l=v∣Y\)\>τ1and​ycur,l\(k−1\)=\[MASK\]\.\{y\}\_\{\\text\{cur\},l\}^\{\(k\)\}=\\operatorname\*\{arg\\,max\}\\limits\_\{v\\in V\}p\_\{\\theta\}\(\\hat\{\{y\}\}\_\{\\text\{cur\},l\}=v\\mid Y\),\\quad\\text\{if \}\\begin\{subarray\}\{c\}\\max\\limits\_\{v\\in V\}p\_\{\\theta\}\(\\hat\{\{y\}\}\_\{\\text\{cur\},l\}=v\\mid Y\)\>\\tau\_\{1\}\\\\ \\text\{and \}\{y\}^\{\(k\-1\)\}\_\{\\text\{cur\},l\}=\\mathchoice\{\\hbox to32\.96pt\{\\vbox to11\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 16\.47993pt\\lower\-4\.15pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-16\.07994pt\}\{\-3\.75pt\}\{32\.15988pt\}\{11\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-14\.07994pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\displaystyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to32\.96pt\{\\vbox to11\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 16\.47993pt\\lower\-4\.15pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-16\.07994pt\}\{\-3\.75pt\}\{32\.15988pt\}\{11\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-14\.07994pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\textstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to22\.22pt\{\\vbox to9\.7pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-3\.62497pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-10\.71112pt\}\{\-3\.22498pt\}\{21\.42224pt\}\{8\.89996pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-8\.71112pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to17\.24pt\{\\vbox to8\.3pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-3\.275pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-8\.22221pt\}\{\-2\.875pt\}\{16\.44443pt\}\{7\.5pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-6\.22221pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptscriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\\end\{subarray\}\.\(2\)
Here, a relatively low confidence thresholdτ1\\tau\_\{1\}is adopted to allow more possible tokens to be decoded at each step, which will achieve the acceleration if only a few tokens among them are revoked during the verification module detailed in the next section\. This will be demonstrated in the experiments\.

#### IV\-B2Verification

The design principle of the verification module is to utilize the increasingly enriched semantic context at each decoding step—relative to earlier steps, to evaluate the quality of previously unmasked tokens\. By re\-masking low\-quality tokens, the decoding process becomes revokable and amenable for the proper early error correction\.

To realize effective quality verification about the decoded tokens, we design an auxiliary shadow block consisting entirely of\[MASK\],Yshad=\[\[MASK\]\]×LbY\_\{\\text\{shad\}\}=\[\\ \\mathchoice\{\\hbox to40\.36pt\{\\vbox to14\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 20\.17781pt\\lower\-4\.9pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-19\.77782pt\}\{\-4\.5pt\}\{39\.55563pt\}\{14\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-17\.77782pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\displaystyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to40\.36pt\{\\vbox to14\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 20\.17781pt\\lower\-4\.9pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-19\.77782pt\}\{\-4\.5pt\}\{39\.55563pt\}\{14\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-17\.77782pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\textstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to32\.96pt\{\\vbox to11\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-4\.15pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-16\.07994pt\}\{\-3\.75pt\}\{32\.15988pt\}\{11\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-14\.07994pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to28\.5pt\{\\vbox to9\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-3\.65pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-13\.85088pt\}\{\-3\.25pt\}\{27\.70175pt\}\{9\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-11\.85088pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptscriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\\ \]\\times L\_\{b\}\. This block is appended to the sequenceYY, resulting in an extended sequenceY~=\[Yleft,Ycur,Yright,Yshad\]\\tilde\{Y\}=\[Y\_\{\\text\{left\}\},\{Y\}\_\{\\text\{cur\}\},Y\_\{\\text\{right\}\},Y\_\{\\text\{shad\}\}\]\. We carefully design the position IDs and attention mask associated withYshadY\_\{\\text\{shad\}\}to ensure that its output can effectively verify the quality of the tokens decoded at the corresponding positions inYcurY\_\{\\text\{cur\}\}\.

Position IDs\.AlthoughYshadY\_\{\\text\{shad\}\}is appended to the right end of the sequence, we assign it the same position IDs asYcurY\_\{\\text\{cur\}\}\. Thus, the output ofYshadY\_\{\\text\{shad\}\}corresponds to the same positions asYcurY\_\{\\text\{cur\}\}, enabling position\-wise verification\.

Attention Mask\.As illustrated in Fig\.[2](https://arxiv.org/html/2605.16941#S4.F2), we carefully design the attention mask after incorporatingYshadY\_\{\\text\{shad\}\}into the sequenceY~\\tilde\{Y\}\. Specifically, tokens inYleftY\_\{\\text\{left\}\},YcurY\_\{\\text\{cur\}\}, andYrightY\_\{\\text\{right\}\}can freely attend to each other, but they are not allowed to attend toYshadY\_\{\\text\{shad\}\}\. In contrast, each token inYshadY\_\{\\text\{shad\}\}is allowed to attend to all tokens except its corresponding position inYcurY\_\{\\text\{cur\}\}\.

With the above design of position IDs and attention masks, we achieve the following properties:

- •For any token in the current blockYcurY\_\{\\text\{cur\}\}, appendingYshadY\_\{\\text\{shad\}\}does not affect the model’s output\. Formally, pθ​\(y^cur,l\|Y\)=pθ​\(y^cur,l\|Y~\)\.p\_\{\\theta\}\(\\hat\{\{y\}\}\_\{\\text\{cur\},l\}\|Y\)=p\_\{\\theta\}\(\\hat\{\{y\}\}\_\{\\text\{cur\},l\}\|\\tilde\{Y\}\)\.
- •For any token inYshadY\_\{\\text\{shad\}\}, the following properties hold\. For example, consider the tokenyshad,3y\_\{\\text\{shad\},3\}in Fig\.[2](https://arxiv.org/html/2605.16941#S4.F2), which is assigned position ID 6\. - –It shares the same position ID asycur,3y\_\{\\text\{cur\},3\}, and is allowed to attend toYleftY\_\{\\text\{left\}\}andYrightY\_\{\\text\{right\}\}; - –It is explicitly prevented from attending toycur,3y\_\{\\text\{cur\},3\}, effectively avoiding information leakage during verification; - –For all other positions inYcurY\_\{\\text\{cur\}\}, each position is attended by exactly one decoded token \(fromYcurY\_\{\\text\{cur\}\}\) and one\[MASK\]inYshadY\_\{\\text\{shad\}\}\. The former provides progressively richer contextual semantics during decoding, while the latter serves to regularize the confidence of decoded tokens inYcurY\_\{\\text\{cur\}\}, reflecting the uncertainty and the need for potential refinement\.

With the specially designed position IDs and the attention mask described above, the verification module can be formally expressed as:

ycur,l\(k\)=\[MASK\],if​pθ​\(y^shad,l=ycur,l\(k−1\)∣Y~\)<τ2and​ycur,l\(k−1\)≠\[MASK\]\.\{y\}\_\{\\text\{cur\},l\}^\{\(k\)\}=\\mathchoice\{\\hbox to40\.36pt\{\\vbox to14\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 20\.17781pt\\lower\-4\.9pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-19\.77782pt\}\{\-4\.5pt\}\{39\.55563pt\}\{14\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-17\.77782pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\displaystyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to40\.36pt\{\\vbox to14\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 20\.17781pt\\lower\-4\.9pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-19\.77782pt\}\{\-4\.5pt\}\{39\.55563pt\}\{14\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-17\.77782pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\textstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to32\.96pt\{\\vbox to11\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-4\.15pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-16\.07994pt\}\{\-3\.75pt\}\{32\.15988pt\}\{11\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-14\.07994pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to28\.5pt\{\\vbox to9\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-3\.65pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-13\.85088pt\}\{\-3\.25pt\}\{27\.70175pt\}\{9\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-11\.85088pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptscriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\},\\quad\\text\{if \}\\begin\{subarray\}\{c\}p\_\{\\theta\}\(\\hat\{\{y\}\}\_\{\\text\{shad\},l\}=\{y\}\_\{\\text\{cur\},l\}^\{\(k\-1\)\}\\mid\\tilde\{Y\}\)<\\tau\_\{2\}\\\\ \\text\{and \}\{y\}\_\{\\text\{cur\},l\}^\{\(k\-1\)\}\\neq\\mathchoice\{\\hbox to32\.96pt\{\\vbox to11\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 16\.47993pt\\lower\-4\.15pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-16\.07994pt\}\{\-3\.75pt\}\{32\.15988pt\}\{11\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-14\.07994pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\displaystyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to32\.96pt\{\\vbox to11\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 16\.47993pt\\lower\-4\.15pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-16\.07994pt\}\{\-3\.75pt\}\{32\.15988pt\}\{11\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-14\.07994pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\textstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to22\.22pt\{\\vbox to9\.7pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-3\.62497pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-10\.71112pt\}\{\-3\.22498pt\}\{21\.42224pt\}\{8\.89996pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-8\.71112pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to17\.24pt\{\\vbox to8\.3pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-3\.275pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-8\.22221pt\}\{\-2\.875pt\}\{16\.44443pt\}\{7\.5pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-6\.22221pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptscriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\\end\{subarray\}\.\(3\)whereτ2\\tau\_\{2\}is the confidence threshold for verification\.

#### IV\-B3Overall Procedure

In summary, at decoding stepkk, our framework enables both the drafting and verification processes to be completed in a single forward pass:

ycur,l\(k\)=\{arg​maxv∈V⁡pθ​\(y^cur,l=v∣Y~\),if​maxv∈V⁡pθ​\(y^cur,l=v∣Y~\)\>τ1and​ycur,l\(k−1\)=\[MASK\],\[MASK\],if​pθ​\(y^shad,l=ycur,l\(k−1\)∣Y~\)<τ2and​ycur,l\(k−1\)≠\[MASK\],ycur,l\(k−1\),otherwise\.\\small\{y\}\_\{\\text\{cur\},l\}^\{\(k\)\}=\\begin\{cases\}\\operatorname\*\{arg\\,max\}\\limits\_\{v\\in V\}p\_\{\\theta\}\(\\hat\{\{y\}\}\_\{\\text\{cur\},l\}=v\\mid\\tilde\{Y\}\),&\\text\{if \}\\begin\{subarray\}\{c\}\\max\\limits\_\{v\\in V\}p\_\{\\theta\}\(\\hat\{\{y\}\}\_\{\\text\{cur\},l\}=v\\mid\\tilde\{Y\}\)\>\\tau\_\{1\}\\\\ \\text\{and \}\{y\}^\{\(k\-1\)\}\_\{\\text\{cur\},l\}=\\mathchoice\{\\hbox to32\.96pt\{\\vbox to11\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 16\.47993pt\\lower\-4\.15pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-16\.07994pt\}\{\-3\.75pt\}\{32\.15988pt\}\{11\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-14\.07994pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\displaystyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to32\.96pt\{\\vbox to11\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 16\.47993pt\\lower\-4\.15pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-16\.07994pt\}\{\-3\.75pt\}\{32\.15988pt\}\{11\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-14\.07994pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\textstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to22\.22pt\{\\vbox to9\.7pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-3\.62497pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-10\.71112pt\}\{\-3\.22498pt\}\{21\.42224pt\}\{8\.89996pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-8\.71112pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to17\.24pt\{\\vbox to8\.3pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-3\.275pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-8\.22221pt\}\{\-2\.875pt\}\{16\.44443pt\}\{7\.5pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-6\.22221pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptscriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\\end\{subarray\},\\\\ \\mathchoice\{\\hbox to37\.68pt\{\\vbox to13\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 18\.84222pt\\lower\-4\.65pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-18\.44223pt\}\{\-4\.25pt\}\{36\.88446pt\}\{13\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-16\.44223pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\displaystyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to37\.68pt\{\\vbox to13\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 18\.84222pt\\lower\-4\.65pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-18\.44223pt\}\{\-4\.25pt\}\{36\.88446pt\}\{13\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-16\.44223pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\textstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to31\.86pt\{\\vbox to11\.1pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-3\.97499pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-15\.53023pt\}\{\-3\.575pt\}\{31\.06046pt\}\{10\.29999pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-13\.53023pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to20\.8pt\{\\vbox to9\.3pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-3\.525pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-10\.0pt\}\{\-3\.125pt\}\{20\.0pt\}\{8\.5pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-8\.0pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptscriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\},&\\text\{if \}\\begin\{subarray\}\{c\}p\_\{\\theta\}\(\\hat\{\{y\}\}\_\{\\text\{shad\},l\}=\{y\}\_\{\\text\{cur\},l\}^\{\(k\-1\)\}\\mid\\tilde\{Y\}\)<\\tau\_\{2\}\\\\ \\text\{and \}\{y\}\_\{\\text\{cur\},l\}^\{\(k\-1\)\}\\neq\\mathchoice\{\\hbox to32\.96pt\{\\vbox to11\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 16\.47993pt\\lower\-4\.15pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-16\.07994pt\}\{\-3\.75pt\}\{32\.15988pt\}\{11\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-14\.07994pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\displaystyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to32\.96pt\{\\vbox to11\.8pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 16\.47993pt\\lower\-4\.15pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-16\.07994pt\}\{\-3\.75pt\}\{32\.15988pt\}\{11\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-14\.07994pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\textstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to22\.22pt\{\\vbox to9\.7pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-3\.62497pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-10\.71112pt\}\{\-3\.22498pt\}\{21\.42224pt\}\{8\.89996pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-8\.71112pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\{\\hbox to17\.24pt\{\\vbox to8\.3pt\{\\pgfpicture\\makeatletter\\hbox\{\\qquad\\lower\-3\.275pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\}\{ \{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\{\}\{\}\{\{ \{\}\{\}\}\}\{ \{\}\{\}\} \{\{\}\{\{\}\}\}\{\{\}\{\}\}\{\}\{\{\}\{\}\} \{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\{\}\\pgfsys@rect\{\-8\.22221pt\}\{\-2\.875pt\}\{16\.44443pt\}\{7\.5pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-6\.22221pt\}\{0\.0pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{$\\scriptscriptstyle\\text\{\{\\color\[rgb\]\{\.5,\.5,\.5\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{\.5,\.5,\.5\}\\pgfsys@color@gray@stroke\{\.5\}\\pgfsys@color@gray@fill\{\.5\}\[MASK\]\}\}$\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\\hbox to0\.0pt\{\}\{\{ \{\}\{\}\{\}\{\}\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\}\\end\{subarray\},\\\\ \{y\}\_\{\\text\{cur\},l\}^\{\(k\-1\)\},&\\text\{otherwise\}\.\\end\{cases\}\(4\)
We iteratively refine the entireYcurY\_\{\\text\{cur\}\}using the procedure in Eq\. \([4](https://arxiv.org/html/2605.16941#S4.E4)\), until all tokens inYcurY\_\{\\text\{cur\}\}are no longer\[MASK\]\. We set the drafting thresholdτ1\\tau\_\{1\}and the verification thresholdτ2\\tau\_\{2\}such thatτ1<τ2\\tau\_\{1\}<\\tau\_\{2\}\. A lowerτ1\\tau\_\{1\}accelerates the decoding process by allowing more tokens to be generated in parallel, while a higherτ2\\tau\_\{2\}ensures the quality of the final output by enforcing stricter acceptance criteria\. We refer to this design philosophy as*"Wide\-In, Narrow\-Out"*, abbreviated as WINO in short\.

Beyond online decoding, WINO also provides a reliable signal for training\-time alignment\. During draft and verification, a token may be revealed, revoked, and later revealed again under richer context\.

This behavior indicates that the first high\-confidence prediction is not always the proper time to expose a token\. Instead, the step from which a token stays stable reflects a more reliable generation order\. In Section[V](https://arxiv.org/html/2605.16941#S5), we use this verified order to construct trajectory\-guided training samples, so that the model can learn the denoising order discovered by WINO from its own\.

## VRoll Back: Learning from Verified WINO Trajectories

WINO improves DLLM inference by making parallel decoding revokable\. However, its benefit is obtained online: the model still needs draft and verification to identify which tokens are safe to reveal\. This suggests a natural training objective\. If the model can learn the verified generation order discovered by WINO, it can reveal reliable tokens earlier and defer uncertain positions without relying heavily on online rollback\.

We therefore propose WINO\+, a trajectory guided training method that internalizes the verified denoising order produced by WINO\. For each training instance, we first run WINO offline and record when each generated token becomes stable\. We then construct training states that follow this stable order\. The model is trained to predict the tokens scheduled for the current step, while keeping later tokens uncertain when they are not yet reliable\. In this way, WINO\+ aligns model training with the efficient inference process required by DLLMs\. The overall pipeline is illustrated in Fig\.[3](https://arxiv.org/html/2605.16941#S5.F3)\.

### V\-AVerified Trajectories from WINO

Given an input promptXXand its ground truthgg, WINO produces a denoising trajectory𝒯W=\(Y\(0\),Y\(1\),…,Y\(K\)\)\\mathcal\{T\}^\{W\}=\(Y^\{\(0\)\},Y^\{\(1\)\},\\ldots,Y^\{\(K\)\}\), whereY\(0\)Y^\{\(0\)\}is fully masked on the response side andY\(K\)Y^\{\(K\)\}is the final decoded response\. We retain a trajectory only when its final response is judged correct by the task evaluator with respect togg\. For example, in reasoning tasks, the extracted answer fromY\(K\)Y^\{\(K\)\}must match the ground truth answer; in code generation, the code extracted must pass the corresponding tests\. For each retained trajectory, we use its final response as the training target and denote it byY†=\[y1†,…,yL†\]Y^\{\\dagger\}=\[y^\{\\dagger\}\_\{1\},\\ldots,y^\{\\dagger\}\_\{L\}\], namelyY†=Y\(K\)Y^\{\\dagger\}=Y^\{\(K\)\}\.

For each response positionll, we define its WINO finalization step astl=min⁡\{t∣∀j≥t,yl\(j\)=yl†\}t\_\{l\}=\\min\\left\\\{t\\mid\\forall j\\geq t,\\;y\_\{l\}^\{\(j\)\}=y\_\{l\}^\{\\dagger\}\\right\\\}\. For prompt positions, we settl=0t\_\{l\}=0\. Thus,tlt\_\{l\}records the earliest step after which positionllremains equal to its final generated token\. A smallertlt\_\{l\}means that the token can be reliably revealed earlier, while a largertlt\_\{l\}means that the token requires more context before it becomes stable\. The offline trajectory\-collection stage in[Fig\.˜3](https://arxiv.org/html/2605.16941#S5.F3)illustrates how WINO rollouts are filtered and converted into token\-level finalization steps\.

The set of finalization steps gives a token wise generation order verified by WINO\. Unlike random masking used in standard DLLM training, this order reflects the actual decoding process under aggressive parallel generation and verification\. It therefore provides direct supervision for when each token should be revealed\.

### V\-BTrajectory Guided Training States

From each retained trajectory, we construct training states according to the extracted finalization steps\. Let𝒮=\{tl∣tl\>0\}\\mathcal\{S\}=\\\{t\_\{l\}\\mid t\_\{l\}\>0\\\}be the set of response side finalization steps\. For eacht∈𝒮t\\in\\mathcal\{S\}we construct a partially revealed sequenceY~\(t−1\)\\widetilde\{Y\}^\{\(t\-1\)\}by revealing only the tokens finalized before steptt:

y~l\(t−1\)=\{yl†,tl<t,\[MASK\],tl≥t\.\\widetilde\{y\}^\{\(t\-1\)\}\_\{l\}=\\begin\{cases\}y^\{\\dagger\}\_\{l\},&t\_\{l\}<t,\\\\ \\texttt\{\[MASK\]\},&t\_\{l\}\\geq t\.\\end\{cases\}\(5\)This state represents the context available immediately before the tokens finalized at stepttshould be revealed\.

We defineAt=\{l∣tl=t\}A\_\{t\}=\\\{l\\mid t\_\{l\}=t\\\}as the positions to be revealed at the current step, andBt=\{l∣tl\>t\}B\_\{t\}=\\\{l\\mid t\_\{l\}\>t\\\}as the positions that should remain masked for later steps\. Each tuple\(X,Y~\(t−1\),At,Bt,Y†\)\(X,\\widetilde\{Y\}^\{\(t\-1\)\},A\_\{t\},B\_\{t\},Y^\{\\dagger\}\)is added to the trajectory guided dataset𝒟traj\\mathcal\{D\}\_\{\\mathrm\{traj\}\}\. This construction converts a WINO rollout into supervised training states that preserve the verified denoising order\. The order\-aware dataset\-construction stage in[Fig\.˜3](https://arxiv.org/html/2605.16941#S5.F3)provides an example of how finalization steps are transformed into partially revealed training states\.

![Refer to caption](https://arxiv.org/html/2605.16941v1/x3.png)Figure 3:Overview of WINO\+\. Stage 1 uses WINO as an offline teacher to collect verification\-guided trajectories and derive token finalization steps\. Stage 2 converts the final response and its finalization\-step vector into order\-aware training states, where only tokens finalized before the current step are revealed\. Stage 3 post\-trains the base DLLM with trajectory\-consistency losses:ℒtok\\mathcal\{L\}\_\{\\mathrm\{tok\}\}supervises current\-step targets,ℒdefer\\mathcal\{L\}\_\{\\mathrm\{defer\}\}suppresses premature high\-confidence errors on deferred positions, andℒsharp\\mathcal\{L\}\_\{\\mathrm\{sharp\}\}sharpens correct but low\-confidence predictions\.
### V\-CTrajectory Consistency Objective

We train the base DLLM on𝒟traj\\mathcal\{D\}\_\{\\mathrm\{traj\}\}\. For a training tuple at steptt, letpθ,l\(t\)\(⋅\)=pθ\(⋅∣X,Y~\(t−1\)\)p\_\{\\theta,l\}^\{\(t\)\}\(\\cdot\)=p\_\{\\theta\}\(\\cdot\\mid X,\\widetilde\{Y\}^\{\(t\-1\)\}\)denote the prediction at positionll\. We usey^l\(t\)\\hat\{y\}^\{\(t\)\}\_\{l\}andcl\(t\)c\_\{l\}^\{\(t\)\}to denote its greedy prediction and confidence, respectively\.

The primary goal is to make the model follow the verified order extracted from WINO\. This gives a basic trajectory alignment objective with two complementary parts\. First, positions inAtA\_\{t\}should be revealed at the current step, so we supervise them with cross entropy\. Second, positions inBtB\_\{t\}should still be deferred\. If the model produces a high confidence incorrect prediction on these deferred positions, the prediction may cause premature revealing during parallel decoding\. We therefore suppress such predictions by maximizing their entropy\. Concretely, we defineRt=\{l∈Bt∣y^l\(t\)≠yl†,cl\(t\)≥τ1\}R\_\{t\}=\\\{l\\in B\_\{t\}\\mid\\hat\{y\}^\{\(t\)\}\_\{l\}\\neq y\_\{l\}^\{\\dagger\},\\;c\_\{l\}^\{\(t\)\}\\geq\\tau\_\{1\}\\\}\. The basic trajectory alignment loss is:

ℒbase\(t\)=−1\|At\|​∑l∈Atlog⁡pθ,l\(t\)​\(yl†\)⏟ℒtok\(t\)−1max⁡\(1,\|Rt\|\)​∑l∈RtH​\(pθ,l\(t\)\)⏟ℒdefer\(t\)\.\\small\\mathcal\{L\}\_\{\\mathrm\{base\}\}^\{\(t\)\}=\\underbrace\{\-\\frac\{1\}\{\|A\_\{t\}\|\}\\sum\_\{l\\in A\_\{t\}\}\\log p\_\{\\theta,l\}^\{\(t\)\}\(y\_\{l\}^\{\\dagger\}\)\}\_\{\\mathcal\{L\}\_\{\\mathrm\{tok\}\}^\{\(t\)\}\}\-\\underbrace\{\\frac\{1\}\{\\max\(1,\|R\_\{t\}\|\)\}\\sum\_\{l\\in R\_\{t\}\}H\\\!\\left\(p\_\{\\theta,l\}^\{\(t\)\}\\right\)\}\_\{\\mathcal\{L\}\_\{\\mathrm\{defer\}\}^\{\(t\)\}\}\.\(6\)Here,H​\(p\)=−∑v∈𝒱p​\(v\)​log⁡p​\(v\)H\(p\)=\-\\sum\_\{v\\in\\mathcal\{V\}\}p\(v\)\\log p\(v\)denotes the Shannon entropy of the predictive distribution over the vocabulary𝒱\\mathcal\{V\}\.ℒtok\(t\)\\mathcal\{L\}\_\{\\mathrm\{tok\}\}^\{\(t\)\}teaches what tokens should be revealed at the current verified step, whileℒdefer\(t\)\\mathcal\{L\}\_\{\\mathrm\{defer\}\}^\{\(t\)\}discourages high confidence errors on tokens that should remain masked\.

Although this basic objective already transfers the WINO derived order, it does not explicitly improve the confidence of correct predictions that are close to being revealable\. To further speed up inference, we add a confidence refinement term\. LetCt=\{l∈At∣y^l\(t\)=yl†,cl\(t\)<τ2\}C\_\{t\}=\\\{l\\in A\_\{t\}\\mid\\hat\{y\}^\{\(t\)\}\_\{l\}=y\_\{l\}^\{\\dagger\},\\;c\_\{l\}^\{\(t\)\}<\\tau\_\{2\}\\\}denote the positions whose predictions are correct but still below the verification threshold\. We sharpen these predictions by minimizing their entropy, denoted as:

ℒsharp\(t\)=1max⁡\(1,\|Ct\|\)​∑l∈CtH​\(pθ,l\(t\)\)\.\\mathcal\{L\}\_\{\\mathrm\{sharp\}\}^\{\(t\)\}=\\frac\{1\}\{\\max\(1,\|C\_\{t\}\|\)\}\\sum\_\{l\\in C\_\{t\}\}H\(p\_\{\\theta,l\}^\{\(t\)\}\)\.\(7\)
The final WINO\+ objective is:

ℒ\(t\)=ℒbase\(t\)\+λ​ℒsharp\(t\),\\mathcal\{L\}^\{\(t\)\}=\\mathcal\{L\}\_\{\\mathrm\{base\}\}^\{\(t\)\}\+\\lambda\\mathcal\{L\}\_\{\\mathrm\{sharp\}\}^\{\(t\)\},\(8\)whereλ\\lambdais the sharp loss weight hyperparameter\. The overall training objective is the expectation ofℒ\(t\)\\mathcal\{L\}^\{\(t\)\}over all tuples in𝒟traj\\mathcal\{D\}\_\{\\mathrm\{traj\}\}\.

As shown in WINO\+ post training of Fig\.[3](https://arxiv.org/html/2605.16941#S5.F3), this formulation separates the roles of the three losses\. The token loss and the defer loss form the basic trajectory alignment objective with fixed unit weights\. The sharp loss is used only as a confidence refinement term, controlled byλ\\lambda\. Thus, WINO\+ first learns the verified denoising order discovered by WINO, and then further sharpens correct current step predictions to enable faster token revealing during inference\.

At inference time, WINO\+ uses the same semi\-autoregressive decoding interface as the base DLLM\. The auxiliary shadow block and online verification are not required\. Because the model has been trained to follow the WINO derived order, more tokens can be safely revealed in each step, leading to faster decoding while preserving generation quality\.

TABLE I:Performance and inference speedup comparison on diverse language benchmarks\.BenchmarkMethodAccuracyStepsStepTPSTPSReductionSpeedupGSM8KMath ReasoningLLaDA73\.242561\.00×\\times17\.761\.00×\\timesWINO75\.82 \(\+2\.58\)41\.93 \(\-214\.07\)6\.10×\\times100\.53 \(\+82\.77\)5\.66×\\timesWINO\+76\.58 \(\+3\.34\)37\.47 \(\-218\.53\)6\.83×\\times121\.86 \(\+104\.1\)6\.86×\\timesMATH\-500Math ReasoningLLaDA32\.002561\.00×\\times17\.621\.00×\\timesWINO34\.20 \(\+2\.20\)74\.44 \(\-181\.56\)3\.44×\\times55\.86 \(\+38\.24\)3\.17×\\timesWINO\+34\.40 \(\+2\.40\)65\.62 \(\-190\.38\)3\.90×\\times64\.78 \(\+47\.16\)3\.68×\\timesHumanEvalCode GenerationLLaDA37\.802561\.00×\\times14\.521\.00×\\timesWINO42\.07 \(\+4\.27\)93\.32 \(\-162\.68\)2\.74×\\times37\.19 \(\+22\.67\)2\.56×\\timesWINO\+42\.68 \(\+4\.88\)82\.71 \(\-173\.29\)3\.10×\\times37\.80 \(\+23\.28\)2\.60×\\timesMBPPCode GenerationLLaDA36\.402561\.00×\\times18\.521\.00×\\timesWINO36\.40 \(\+0\.00\)96\.57 \(\-159\.43\)2\.65×\\times45\.39 \(\+26\.87\)2\.45×\\timesWINO\+37\.20 \(\+0\.80\)67\.53 \(\-188\.47\)3\.79×\\times64\.75 \(\+46\.23\)3\.50×\\timesCountdownLogical ReasoningLLaDA24\.212561\.00×\\times17\.221\.00×\\timesWINO33\.20 \(\+8\.99\)105\.88 \(\-150\.12\)2\.41×\\times38\.97 \(\+21\.75\)2\.26×\\timesWINO\+48\.05 \(\+23\.84\)63\.47 \(\-192\.53\)4\.03×\\times71\.48 \(\+54\.26\)4\.15×\\timesSudokuLogical ReasoningLLaDA14\.232561\.00×\\times11\.611\.00×\\timesWINO15\.20 \(\+0\.97\)131\.96 \(\-124\.04\)1\.94×\\times21\.11 \(\+9\.50\)1\.82×\\timesWINO\+18\.37 \(\+4\.14\)64\.77 \(\-191\.23\)3\.95×\\times48\.85 \(\+37\.24\)4\.21×\\timesARC\-ECommonsense ReasoningLLaDA59\.132561\.00×\\times17\.261\.00×\\timesWINO81\.19 \(\+22\.06\)40\.19 \(\-215\.81\)6\.37×\\times101\.61 \(\+84\.35\)5\.89×\\timesWINO\+84\.97 \(\+25\.84\)24\.86 \(\-231\.14\)10\.30×\\times178\.75 \(\+161\.49\)10\.36×\\timesARC\-CCommonsense ReasoningLLaDA51\.872561\.00×\\times17\.101\.00×\\timesWINO73\.89 \(\+22\.02\)47\.41 \(\-208\.59\)5\.40×\\times85\.42 \(\+68\.32\)5\.00×\\timesWINO\+80\.60 \(\+28\.73\)28\.78 \(\-227\.22\)8\.90×\\times150\.61 \(\+133\.51\)8\.81×\\times

## VIExperiments

### VI\-AExperiment Setup

#### VI\-A1Datasets and Baselines

We conduct experiments to evaluate WINO and WINO\+ across different types of tasks and domains\. Specifically, for the language domain, we compare WINO and WINO\+ with the standard decoding of LLaDA on eight tasks: GSM8K\[DBLP:journals/corr/abs\-2110\-14168\], MATH\-500\[DBLP:conf/nips/HendrycksBKABTS21\], HumanEval\[DBLP:journals/corr/abs\-2107\-03374\], MBPP\[DBLP:journals/corr/abs\-2108\-07732\], Countdown\[DBLP:journals/corr/abs\-2504\-12216\], Sudoku\[DBLP:journals/corr/abs\-2504\-12216\], ARC\-E\[DBLP:journals/corr/abs\-1803\-05457\], and ARC\-C\[DBLP:journals/corr/abs\-1803\-05457\], covering four categories of textual generation tasks, including math reasoning, code generation, logical reasoning, and commonsense reasoning\. For the vision\-language domain, we evaluate WINO and WINO\+ against the standard decoding of MMaDA\[DBLP:journals/corr/abs\-2505\-15809\]on six multimodal understanding tasks: Flickr30k\[DBLP:journals/tacl/YoungLHH14\], AI2D\[DBLP:journals/corr/KembhaviSKSHF16\], MATH\-Vision\[DBLP:conf/nips/WangPSLRZZL24\], MathVista\[DBLP:conf/iclr/LuBX0LH0CG024\], MMMU\[DBLP:conf/cvpr/YueNZ0LZSJRSWYY24\], and ScienceQA\[DBLP:conf/nips/LuMX0CZTCK22\], spanning four types of multimodal tasks—captioning, chart understanding, math reasoning, and multi\-discipline reasoning\. For clarity, we test on the validation set of MMMU and the official testmini subset of MathVista\.

#### VI\-A2Evaluation Details

All benchmarks are evaluated in a zero\-shot manner, except Sudoku, which is evaluated in a 4\-shot setting\. We use CIDEr\[DBLP:conf/cvpr/VedantamZP15\]for Flickr30k and accuracy for all remaining benchmarks\. To assess inference efficiency, we report the required decoding steps and Tokens Per Second \(TPS\) for the standard baselines, WINO, and WINO\+, averaged over all samples in each benchmark\. We adopt the open\-sourced[LLaDA\-8B\-Instruct](https://huggingface.co/GSAI-ML/LLaDA-8B-Instruct)for language benchmarks and[MMaDA\-8B\-MixCoT](https://huggingface.co/Gen-Verse/MMaDA-8B-MixCoT)for vision\-language tasks\. For all evaluated decoding variants, including the standard baselines, WINO, and WINO\+, we employ the semi\-autoregressive sampling strategy introduced in LLaDA\[DBLP:journals/corr/abs\-2502\-09992\], where the output sequence is partitioned into multiple blocks and generated from left to right\. Unless specified otherwise, we set the generation length to 256 and the block length to 128 for all methods\. For WINO, we set the verification thresholdτ2\\tau\_\{2\}to 0\.9 and tune the drafting thresholdτ1\\tau\_\{1\}from\{0\.5,0\.6,0\.7\}\\\{0\.5,0\.6,0\.7\\\}\. For WINO\+, we use the same generation and block\-wise decoding configuration, but replace the draft\-and\-verify rule with a single\-confidence\-threshold parallel decoding strategy, where the confidence threshold is tuned from\[0\.5,0\.9\]\[0\.5,0\.9\]\.

#### VI\-A3WINO\+ Training

WINO\+ is obtained by LoRA\-based post\-training\[DBLP:conf/iclr/HuSWALWWC22\]on WINO\-guided trajectories\. For LLaDA\-8B\-Instruct, we construct trajectories from GSM8K\[DBLP:journals/corr/abs\-2110\-14168\]and Countdown\-Tasks\-3to4\[DBLP:journals/corr/abs\-2504\-12216\]; for MMaDA\-8B\-MixCoT, we use IconQA\[DBLP:conf/nips/LuQCXZZYLZ21\]\. LoRA adapters are inserted into the attention and MLP projection layers, with rank and scaling factor set to 128\. Training is conducted with bf16 mixed precision and AdamW optimization\[loshchilov2019decoupled\]\. Full details on data construction, trajectory filtering, loss weights, and model\-specific optimization settings are provided in Appendix B\.

### VI\-BMain Results

TABLE II:Performance and inference speedup comparison across diverse multi\-modal understanding and reasoning benchmarks\. We use CIDEr for Flickr30k and accuracy for other benchmarks\.BenchmarkMethodPerformanceStepsStepTPSTPSReductionSpeedupFlickr30kCaptioningMMaDA53\.672561\.00×\\times6\.411\.00×\\timesWINO53\.83 \(\+0\.16\)25\.47 \(\-230\.53\)10\.05×\\times55\.11 \(\+48\.70\)8\.60×\\timesWINO\+63\.38 \(\+9\.71\)15\.78 \(\-240\.22\)16\.22×\\times106\.07 \(\+99\.66\)16\.55×\\timesAI2DChart UnderstandingMMaDA54\.862561\.00×\\times6\.311\.00×\\timesWINO57\.19 \(\+2\.33\)30\.90 \(\-225\.10\)8\.30×\\times46\.04 \(\+39\.73\)7\.30×\\timesWINO\+66\.61 \(\+11\.75\)26\.15 \(\-229\.85\)9\.79×\\times62\.00 \(\+55\.69\)9\.83×\\timesMATH\-VisionMath ReasoningMMaDA8\.552561\.00×\\times6\.221\.00×\\timesWINO9\.57 \(\+1\.02\)44\.69 \(\-211\.31\)5\.73×\\times31\.17 \(\+24\.95\)5\.01×\\timesWINO\+13\.42 \(\+4\.87\)41\.06 \(\-214\.94\)6\.23×\\times37\.46 \(\+31\.24\)6\.02×\\timesMathVista\-miniMath ReasoningMMaDA31\.102561\.00×\\times6\.211\.00×\\timesWINO31\.40 \(\+0\.30\)33\.45 \(\-222\.55\)7\.65×\\times41\.96 \(\+35\.75\)6\.76×\\timesWINO\+31\.40 \(\+0\.30\)19\.41 \(\-236\.59\)13\.19×\\times82\.16 \(\+75\.95\)13\.23×\\timesMMMU\-valMulti\-discipline ReasoningMMaDA18\.562561\.00×\\times6\.021\.00×\\timesWINO24\.00 \(\+5\.44\)38\.47 \(\-217\.53\)6\.65×\\times36\.13 \(\+30\.11\)6\.00×\\timesWINO\+28\.11 \(\+9\.55\)26\.25 \(\-229\.75\)9\.75×\\times54\.18 \(\+48\.16\)9\.00×\\timesScienceQAMulti\-discipline ReasoningMMaDA30\.892561\.00×\\times6\.071\.00×\\timesWINO42\.24 \(\+11\.35\)28\.12 \(\-227\.88\)9\.10×\\times49\.45 \(\+43\.38\)8\.15×\\timesWINO\+53\.84 \(\+22\.95\)23\.26 \(\-232\.74\)11\.01×\\times66\.89 \(\+60\.82\)11\.02×\\times

#### VI\-B1Performance and speedup on text generation

We report the performance, decoding steps and TPS of LLaDA, WINO, and WINO\+ on language benchmarks in Table[I](https://arxiv.org/html/2605.16941#S5.T1)\. Compared with standard LLaDA decoding, WINO achieves better accuracy with far fewer decoding steps on most benchmarks, except for MBPP where it matches the baseline performance\. For instance, WINO improves GSM8K accuracy by 2\.58%, while reducing the average decoding steps from 256 to 41\.93, corresponding to a 6\.10×\\timesstep reduction and a 5\.66×\\timesTPS speedup\. Similar trends can also be observed on MATH\-500, HumanEval, Countdown, Sudoku, ARC\-E, and ARC\-C, showing that the draft\-and\-verify mechanism can improve generation quality while preserving the parallel decoding advantage of DLLMs\. WINO\+ further improves the trade\-off by injecting WINO\-guided trajectories into the model\. It achieves higher accuracy than both LLaDA and WINO on all eight textual tasks, while requiring fewer decoding steps than WINO on every benchmark\. Notably, WINO\+ improves Countdown from 24\.21% to 48\.05%, ARC\-E from 59\.13% to 84\.97%, and ARC\-C from 51\.87% to 80\.60%, with 4\.15×\\times, 10\.36×\\times, and 8\.81×\\timesTPS speedups, respectively\. These results demonstrate that the draft\-and\-verify mechanism improves generation quality and inference efficiency, and that its induced trajectories can be further internalized by post\-training\.

#### VI\-B2Performance and speedup on multimodal understanding and reasoning

We further assess the performance and efficiency gains of WINO and WINO\+ on multimodal benchmarks in Table[II](https://arxiv.org/html/2605.16941#S6.T2)\. Compared with vanilla MMaDA, WINO achieves consistent and substantial speedup across all benchmarks\. For example, it obtains a 10\.05×\\timesstep reduction and an 8\.60×\\timesTPS speedup on Flickr30k while maintaining comparable captioning quality\. On AI2D, MMMU\-val, and ScienceQA, WINO improves performance by 2\.33, 5\.44, and 11\.35 points, respectively, with large efficiency gains\. WINO\+ further strengthens this trend\. It improves Flickr30k CIDEr from 53\.67 to 63\.38 with a 16\.55×\\timesTPS speedup, and improves ScienceQA accuracy by 22\.95 points with an 11\.02×\\timesTPS speedup\. On MMMU\-val, it improves accuracy from 18\.56% to 28\.11% while reducing the average decoding steps to 26\.25\. These results show that revocable parallel decoding also benefits multimodal DLLMs, where visual and textual contexts jointly affect token reliability\. The further gains of WINO\+ indicate that the verified trajectories discovered by WINO can serve as effective supervision, helping the model internalize token\-level generation order for a better quality–efficiency trade\-off\.

TABLE III:Experiment results on different generation lengths and full diffusion setting, respectively\.BenchmarkGenerationBlockMethodAccuracyStepsStepTPSTPSLengthLengthReductionSpeedup*Different Generation Lengths*GSM8K256128LLaDA73\.242561\.00×\\times17\.761\.00×\\timesWINO75\.82 \(\+2\.58\)41\.936\.10×\\times100\.535\.66×\\times512128LLaDA74\.605121\.00×\\times11\.841\.00×\\timesWINO79\.91 \(\+5\.31\)68\.537\.47×\\times82\.646\.98×\\timesMMMU\-val256128MMaDA18\.562561\.00×\\times6\.021\.00×\\timesWINO24\.00 \(\+5\.44\)38\.476\.65×\\times36\.136\.00×\\times512128MMaDA18\.445121\.00×\\times5\.011\.00×\\timesWINO23\.44 \(\+5\.00\)64\.827\.90×\\times35\.016\.99×\\times*Full Diffusion*GSM8K256256LLaDA34\.342561\.00×\\times17\.731\.00×\\timesWINO58\.22 \(\+23\.88\)38\.776\.60×\\times93\.615\.28×\\times128128LLaDA58\.601281\.00×\\times23\.231\.00×\\timesWINO62\.32 \(\+3\.72\)23\.955\.34×\\times114\.294\.92×\\timesMMMU\-val256256MMaDA17\.222561\.00×\\times6\.111\.00×\\timesWINO22\.44 \(\+5\.22\)24\.9410\.26×\\times50\.038\.19×\\times128128MMaDA15\.331281\.00×\\times6\.701\.00×\\timesWINO23\.11 \(\+7\.78\)19\.146\.69×\\times39\.945\.96×\\times

TABLE IV:Experiment results on the variant of WINO without the verification module\.BenchmarkMethodAccuracyStepsStepTPSTPSReductionSpeedupGSM8KLLaDA73\.242561\.00×\\times17\.761\.00×\\timesOnly Draft \(τ1=0\.6\\tau\_\{1\}=0\.6\)70\.2834\.797\.36×\\times130\.897\.37×\\timesOnly Draft \(τ1=0\.9\\tau\_\{1\}=0\.9\)72\.3381\.393\.15×\\times56\.123\.16×\\timesWINO75\.8241\.936\.10×\\times100\.535\.66×\\timesMMMU\-valMMaDA18\.562561\.00×\\times6\.021\.00×\\timesOnly Draft \(τ1=0\.6\\tau\_\{1\}=0\.6\)19\.8935\.637\.18×\\times43\.227\.18×\\timesOnly Draft \(τ1=0\.9\\tau\_\{1\}=0\.9\)18\.5679\.743\.21×\\times19\.383\.22×\\timesWINO24\.0038\.476\.65×\\times36\.136\.00×\\times

### VI\-CAnalysis and Ablation Studies of WINO

#### VI\-C1On different generation length

In Table[III](https://arxiv.org/html/2605.16941#S6.T3), we evaluate the performance of WINO with experiments on different generation lengths, where the block lengthLbL\_\{b\}is fixed to 128 and the baselines unmask 1 token every decoding step \(to achieve their best generation performance\)\. When the generation length is set to 512, WINO still achieves comparable or better task performance with significantly fewer decoding steps, demonstrating the effectiveness of WINO across different generation lengths\.

#### VI\-C2On full diffusion decoding \(instead of semi\-autoregressive decoding\)

In Table[III](https://arxiv.org/html/2605.16941#S6.T3), we compare the baselines and WINO applying full diffusion decoding, which means the block lengthLbL\_\{b\}is set equal to the generation length\. Compared to results on the semi\-autoregressive decoding in Table[I](https://arxiv.org/html/2605.16941#S5.T1)and Table[II](https://arxiv.org/html/2605.16941#S6.T2), WINO demonstrates substantially strong accuracy gains under the full diffusion setting\. Notably, while LLaDA suffers a substantial accuracy drop on GSM8K with full diffusion decoding, WINO maintains reasonable performance with far fewer decoding steps\. These results indicate that WINO unlocks significantly greater potential for boosting model performance and computational efficiency when applied in full diffusion decoding scenarios\.

#### VI\-C3Comparison with naive parallel sampling

The decoding process of existing DLLMs can be sped up by generating multiple tokens per step,*i\.e\.*, naive parallel sampling\. However, directly increasing the fixed number of generated tokens per step for DLLMs leads to significant performance degradation\. For instance, on GSM8K, accuracy drops from 73\.24% with 256 steps \(1 token/step\) to 71\.11% with 128 steps \(2 tokens/step\), and further down to 64\.67% with 64 steps \(4 tokens/step\)\. In contrast, the draft\-and\-verify procedure of WINO enables flexible decoding during the generation process, achieving 75\.82% accuracy with only 41\.93 steps on average, corresponding to a 6\.10×\\timesspeedup, thereby substantially improving task performance while accelerating inference\.

#### VI\-C4Ablation on verification module

We conduct an ablation study on a variant of WINO that excludes the verification module, implemented by setting the verification thresholdτ2\\tau\_\{2\}to zero\. As presented in Table[IV](https://arxiv.org/html/2605.16941#S6.T4), this variant exhibits significant performance degradation across both benchmarks compared to WINO\. Specifically, when the drafting thresholdτ1\\tau\_\{1\}is small \(corresponding to 0\.6 in the table\), more candidate tokens are unmasked per decoding step, which naturally introduces a higher proportion of unreliable tokens and ultimately compromises output quality\. Conversely, whenτ1\\tau\_\{1\}is large \(corresponding to 0\.9 in the table\), fewer candidate tokens are unmasked per decoding step, which can mitigate error propagation but at the expense of computational efficiency\. Crucially, without the verification module, the generation process lacks a mechanism to correct erroneous predictions\. As a result, even with a largeτ1\\tau\_\{1\}, the model may fail to achieve comparable performance, underscoring the necessity of the verification module in maintaining generation quality\.

![Refer to caption](https://arxiv.org/html/2605.16941v1/x4.png)Figure 4:Ablation of the drafting thresholdτ1\\tau\_\{1\}and verification thresholdτ2\\tau\_\{2\}in WINO\. Results are shown for LLaDA\-based WINO on GSM8K and MMaDA\-based WINO on MMMU\-val; each plot varies one threshold while fixing the other, and reports accuracy together with step\-reduction ratio\.
#### VI\-C5Effect of threshold tuning

In Fig\.[4](https://arxiv.org/html/2605.16941#S6.F4), we present the evaluation results of WINO with varying drafting thresholdτ1\\tau\_\{1\}and verification thresholdτ2\\tau\_\{2\}\. Our experiments suggest that WINO consistently outperforms baselines across different benchmarks and theτ1\\tau\_\{1\}andτ2\\tau\_\{2\}values in terms of both task performance and inference efficiency\. As theτ1\\tau\_\{1\}value decreases, more candidate tokens are unmasked at each decoding step, thereby accelerating inference by reducing the required decoding steps\. However, this comes at the cost of introducing more unreliable predictions, which may place a greater burden on the verification module to correct errors\. Empirically, we find that setting the value of drafting thresholdτ1\\tau\_\{1\}within the range of 0\.5 to 0\.7 achieves an optimal balance, maintaining competitive task performance while preserving efficient generation\. The verification thresholdτ2\\tau\_\{2\}controls the strictness of the verification process and thus influences decoding speed\. Since the performance is relatively robust toτ2\\tau\_\{2\}, we fixτ2=0\.9\\tau\_\{2\}=0\.9in all experiments, while leaving open the possibility of further tuning this parameter for even better performance and speedup\.

#### VI\-C6Relation between speedup and task complexity

As shown in Table[I](https://arxiv.org/html/2605.16941#S5.T1)and Table[II](https://arxiv.org/html/2605.16941#S6.T2), we observe a consistent positive correlation between the degree of speedup and task performance across all benchmarks\. For instance, WINO achieves a 10\.05×\\timesstep reduction on the relatively simple captioning task Flickr30k, compared to only 5\.73×\\timesstep reduction on the more challenging math reasoning benchmark MATH\-Vision\. This is because models can, in principle, solve tasks they are more proficient at with lower computational cost, leaving greater room for acceleration under our decoding method\. And since models are typically more confident when handling easier tasks, each decoding step in WINO tends to yield a larger number of effective tokens\. To further investigate this, we evaluate the decoding steps of WINO across subsets of the MATH\-500 benchmark categorized by difficulty levels\. As shown in Fig\.[5](https://arxiv.org/html/2605.16941#S6.F5), WINO achieves progressively greater acceleration as the difficulty decreases, highlighting its capability to adaptively optimize inference speed based on task complexity\.

![Refer to caption](https://arxiv.org/html/2605.16941v1/x5.png)Figure 5:Decoding steps of WINO on subsets of the MATH benchmark with varied difficulty levels\.

### VI\-DAnalysis and Ablation Studies of WINO\+

#### VI\-D1WINO\+ improves base models under the same decoding budget

To isolate the effect of trajectory injection on the model itself, we further evaluate the trained WINO\+ model under the standard decoding strategy, using the same 256 decoding steps as the base model\. As shown in Table[V](https://arxiv.org/html/2605.16941#S6.T5), WINO\+ consistently outperforms the corresponding base model under identical decoding budgets\. On GSM8K, WINO\+ improves the accuracy from 73\.24% to 79\.23%, and on MMMU\-val, it improves the accuracy from 18\.56% to 28\.44%\. Since both models use the same standard decoding procedure and the same number of decoding steps, these gains cannot be attributed to a different inference algorithm or a larger decoding budget\. Instead, the results indicate that WINO\-guided trajectory injection improves the denoising behavior of the model itself, leading to better generation quality\.

#### VI\-D2On injected trajectories

We compare different trajectory choices for trajectory injection in WINO\+ and summarize the results in Table[VI](https://arxiv.org/html/2605.16941#S6.T6)\. The random\-trajectory variant constructs intermediate supervision from randomly selected token\-revealing orders, while the WINO\-trajectory variant uses the verification\-guided revealing steps produced by WINO\. As shown in the table, the quality of the injected trajectory plays a critical role\. On GSM8K, replacing random trajectories with WINO trajectories improves the accuracy from 72\.63% to 76\.58%, while reducing the average decoding steps from 46\.69 to 37\.47\. A similar trend is observed on MMMU\-val, where the accuracy increases from 26\.67% to 28\.11% and the average decoding steps decrease from 45\.90 to 26\.25\. These results suggest that the improvement of WINO\+ is not simply due to adding intermediate partially masked states during training\. Instead, the trajectory produced by WINO carries useful order information from its draft–verify–fallback process\. By using such trajectories for training, WINO\+ provides a more suitable denoising order than random reconstruction, making the training process better aligned with efficient inference\.

TABLE V:Accuracy comparison between the base model and the trained WINO\+ model under standard decoding\.BenchmarkModelAccuracyStepsGSM8KLLaDA73\.24256WINO\+79\.23256MMMU\-valMMaDA18\.56256WINO\+28\.44256

TABLE VI:Ablation study of trajectories\.BenchmarkTrajectoryAccuracyStepsStepTPSTPSReductionSpeedupGSM8KRandom trajectory72\.6346\.695\.48×\\times96\.625\.44×\\timesWINO trajectory76\.5837\.476\.83×\\times121\.866\.86×\\timesMMMU\-valRandom trajectory26\.6745\.905\.58×\\times31\.275\.19×\\timesWINO trajectory28\.1126\.259\.75×\\times54\.189\.00×\\times

TABLE VII:Training objective ablation of WINO\+ with different loss components\.MethodBenchmarkTraining LossAccuracyStepsStepTPSTPSℒtok\\mathcal\{L\}\_\{\\mathrm\{tok\}\}ℒdefer\\mathcal\{L\}\_\{\\mathrm\{defer\}\}ℒsharp\\mathcal\{L\}\_\{\\mathrm\{sharp\}\}ReductionSpeedupLLaDAGSM8K✓\\checkmark73\.1642\.286\.05×\\times109\.056\.14×\\times✓\\checkmark✓\\checkmark75\.5939\.606\.46×\\times115\.626\.51×\\times✓\\checkmark✓\\checkmark72\.7140\.146\.38×\\times114\.916\.47×\\times✓\\checkmark✓\\checkmark✓\\checkmark76\.5837\.476\.83×\\times121\.866\.86×\\timesMMaDAMMMU\-val✓\\checkmark25\.5644\.985\.69×\\times31\.995\.31×\\times✓\\checkmark✓\\checkmark26\.0033\.707\.60×\\times42\.417\.06×\\times✓\\checkmark✓\\checkmark25\.2235\.637\.18×\\times41\.296\.86×\\times✓\\checkmark✓\\checkmark✓\\checkmark28\.1126\.259\.75×\\times54\.189\.00×\\times

#### VI\-D3On WINO\+ training objectives

We study the contribution of each training objective in WINO\+ and report the results in Table[VII](https://arxiv.org/html/2605.16941#S6.T7)\. Using onlyℒtok\\mathcal\{L\}\_\{\\mathrm\{tok\}\}already brings clear acceleration, showing that the WINO\-derived revealing order provides useful supervision\. However, it only specifies which tokens to decode at each step, without constraining positions that should remain masked\. Addingℒdefer\\mathcal\{L\}\_\{\\mathrm\{defer\}\}suppresses high\-confidence wrong predictions on deferred positions, improving the accuracy from 73\.16% to 75\.59% on GSM8K and from 25\.56% to 26\.00% on MMMU\-val\. This indicates that learning which tokens should not be revealed is also important\. Finally, when combined withℒtok\\mathcal\{L\}\_\{\\mathrm\{tok\}\}andℒdefer\\mathcal\{L\}\_\{\\mathrm\{defer\}\},ℒsharp\\mathcal\{L\}\_\{\\mathrm\{sharp\}\}further refines the confidence of reliable predictions\. The full objective achieves the best trade\-off, reaching 76\.58% with 37\.47 steps on GSM8K and 28\.11% with 26\.25 steps on MMMU\-val\.

![Refer to caption](https://arxiv.org/html/2605.16941v1/x6.png)Figure 6:Ablation of the loss\-balancing coefficientλ\\lambdain WINO\+\. The plots report average accuracy and decoding\-step reduction across language and multimodal benchmarks under differentλ\\lambdavalues\.
#### VI\-D4On loss weight

We study the effect of the sharpening loss by varyingλ\\lambdaover\{0\.02,0\.1,0\.5,1\.0\}\\\{0\.02,0\.1,0\.5,1\.0\\\}\. Fig\.[6](https://arxiv.org/html/2605.16941#S6.F6)reports the averaged accuracy and decoding\-step reduction over language benchmarks and multimodal benchmarks\. As shown in the figure,λ=0\.1\\lambda=0\.1achieves the best accuracy on both language and multimodal benchmarks, suggesting that a moderate sharpening signal helps the model confidently reveal stable tokens early\. Increasing the weight further improves step reduction, but hurts accuracy by encouraging overly aggressive parallel revelation\. Thus, we useλ=0\.1\\lambda=0\.1as the default setting, which offers the best trade\-off between quality and efficiency\.

### VI\-EEfficiency and Case Study

#### VI\-E1GPU memory usage

We compare the peak GPU memory usage of the standard baselines, WINO, and WINO\+ in Fig\.[7](https://arxiv.org/html/2605.16941#S6.F7)\. To facilitate efficient verification of unmasked tokens, WINO introduces an auxiliary shadow block whose size equals the specified block lengthLbL\_\{b\}of the semi\-autoregressive decoding process\. Therefore, WINO induces additional GPU memory cost due to the longer effective sequence length\. Nevertheless, this overhead remains marginal compared to the baselines across the benchmarks\. For instance, on GSM8K, WINO increases GPU memory usage by only 2\.4% compared to standard LLaDA decoding, from 16\.18 GiB to 16\.57 GiB\. In contrast, WINO\+ removes the shadow block at inference time by injecting the stabilized\-prediction behavior into the model parameters, reducing the memory usage to 15\.42 GiB on GSM8K, lower than both WINO and LLaDA\. Across both language and multimodal benchmarks, WINO\+ consistently achieves the lowest memory footprint\. This indicates that WINO\+ converts the online verification behavior of WINO into a more memory\-efficient inference procedure, rather than trading additional memory for speed\.

![Refer to caption](https://arxiv.org/html/2605.16941v1/x7.png)Figure 7:Peak GPU memory usage of the baseline DLLMs, WINO, and WINO\+ across representative language and multimodal benchmarks\.![Refer to caption](https://arxiv.org/html/2605.16941v1/x8.png)Figure 8:Decoding trace on a GSM8K example for standard decoding, WINO, and WINO\+\. The figure shows selected intermediate and final outputs at different decoding steps\. Red text indicates incorrect intermediate or final generated results, green text indicates the correct final result, and \[MASK\] denotes positions that remain undecoded\.
#### VI\-E2Case Study

To conduct a fine\-grained examination of decoding dynamics, we present a GSM8K example in Fig\.[8](https://arxiv.org/html/2605.16941#S6.F8)\. The standard decoding baseline may produce erroneous tokens at early decoding stages\. Since these generated tokens remain unchanged in subsequent decoding steps, the false contextual information can propagate throughout the generation process and eventually lead to low\-quality outputs\. In contrast, WINO enables dynamic refinement through its iterative draft\-and\-verify mechanism: unstable predictions can be identified, remasked, and regenerated when richer context becomes available, thereby mitigating error accumulation\.

The behavior of WINO\+ further shows that such dynamic refinement can be transferred into the model through trajectory injection\. Rather than using shadow block for online verification, WINO\+ learns from the trajectories produced by WINO and forms a more suitable denoising order during training\. As shown in the example, WINO\+ keeps uncertain reasoning\-critical positions masked until sufficient context is available, while generating stable tokens earlier\. As a result, it reaches the correct answer with substantially fewer decoding steps than both the standard baseline and WINO\. This case illustrates how WINO\-derived trajectories help WINO\+ preserve the refinement benefit of WINO while enabling a simpler and faster inference process\.

## VIIConclusion

In this paper, we studied the quality and efficiency trade\-off in open\-source Diffusion Large Language Models and attributed it to a train\-inference mismatch amplified by irreversible decoding\. While standard training reconstructs tokens from randomly corrupted states, efficient inference requires an adaptive denoising order that commits reliable tokens early and defers context\-dependent ones\. To address this issue, we proposed WINO, a training\-free revokable decoding algorithm with a Wide\-In, Narrow\-Out draft, verify, and fallback mechanism\. WINO enables aggressive parallel drafting while re\-evaluating generated tokens under enriched bidirectional context and re\-masking unreliable ones, thereby improving both generation quality and decoding efficiency\. We further introduced WINO\+, a trajectory\-injection framework that distills the verified finalization order discovered by WINO into model parameters\. By replacing random reconstruction with trajectory\-ordered denoising, WINO\+ teaches the model when each token should be revealed or deferred and reduces the need for online rollback\. Experiments on LLaDA and MMaDA across language and vision\-language benchmarks show that WINO consistently improves parallel decoding, while WINO\+ further strengthens the quality\-efficiency trade\-off by internalizing verified denoising trajectories\. These results suggest that DLLMs can serve as their own efficiency teachers, using revokable decoding to discover reliable generation orders and trajectory injection to learn them for faster, higher\-quality generation\.

## References

## Appendix AStandard DLLM Training Objective

Given a promptXXand a reference responseY⋆=\[y1⋆,…,yL⋆\]Y^\{\\star\}=\[y\_\{1\}^\{\\star\},\\ldots,y\_\{L\}^\{\\star\}\], standard DLLM training samples a masking ratioρ∼𝒰​\(0,1\)\\rho\\sim\\mathcal\{U\}\(0,1\)and independently masks each response token with probabilityρ\\rho, while keeping the prompt unchanged\. LetY~\(ρ\)\\widetilde\{Y\}^\{\(\\rho\)\}denote the corrupted response, and letMρ=\{l∈\[L\]∣y~l\(ρ\)=\[MASK\]\}M\_\{\\rho\}=\\\{l\\in\[L\]\\mid\\widetilde\{y\}^\{\(\\rho\)\}\_\{l\}=\\texttt\{\[MASK\]\}\\\}be the set of masked response positions, where\[L\]=\{1,…,L\}\[L\]=\\\{1,\\ldots,L\\\}\. Following the standard masked diffusion formulation used in LLaDA and MMaDA\[DBLP:conf/nips/ShiHWDT24,DBLP:conf/nips/SahooASGMCRK24,DBLP:journals/corr/abs\-2502\-09992,DBLP:journals/corr/abs\-2505\-15809\], the objective is

ℒstd​\(θ\)=−𝔼​\[1ρ​∑l∈Mρlog⁡pθ​\(yl⋆∣X,Y~\(ρ\)\)\],\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{std\}\}\(\\theta\)=\-\\mathbb\{E\}\\left\[\\frac\{1\}\{\\rho\}\\sum\_\{l\\in M\_\{\\rho\}\}\\log p\_\{\\theta\}\\left\(y\_\{l\}^\{\\star\}\\mid X,\\widetilde\{Y\}^\{\(\\rho\)\}\\right\)\\right\],\(9\)where the expectation is taken over\(X,Y⋆\)∼𝒟\(X,Y^\{\\star\}\)\\sim\\mathcal\{D\},ρ∼𝒰​\(0,1\)\\rho\\sim\\mathcal\{U\}\(0,1\), andY~\(ρ\)\\widetilde\{Y\}^\{\(\\rho\)\}sampled from the corresponding masking process\. Here,Y⋆Y^\{\\star\}denotes the reference response from the training data\. This loss supervises what token should fill each masked position, but not when the token should be revealed\.

## Appendix BMore Training Details for WINO\+

We provide additional training details for WINO\+\. All WINO\+ variants are trained with parameter\-efficient LoRA fine\-tuning\[DBLP:conf/iclr/HuSWALWWC22\]on WINO\-guided trajectories using bf16 mixed precision\. For LLaDA\-8B\-Instruct, we collect WINO trajectories from GSM8K\[DBLP:journals/corr/abs\-2110\-14168\]and Countdown\-Tasks\-3to4\[DBLP:journals/corr/abs\-2504\-12216\], retaining only samples with correct final answers\. This yields about 5\.6K GSM8K pairs and 0\.4K Countdown pairs before step\-wise expansion\. We train the model sequentially, first on the easier GSM8K trajectories and then on the more challenging Countdown trajectories\. For MMaDA\-8B\-MixCoT, we construct trajectories from a subset of the IconQA\[DBLP:conf/nips/LuQCXZZYLZ21\]training set and obtain about 0\.7K question\-answer pairs before expansion\. All trajectory data are constructed from training\-set samples only\.

For each retained sample, the final WINO response is used as the target, and all WINO steps are expanded as ordered training instances\. Offline trajectories use block size 128,τ1=0\.6\\tau\_\{1\}=0\.6, andτ2=0\.9\\tau\_\{2\}=0\.9\. WINO\+ does not use the shadow\-block verifier at inference\.

TABLE VIII:Optimization configuration for WINO\+ training\.ConfigurationLLaDA\-8B\-InstructMMaDA\-8B\-MixCoTPrecisionbf16bf16OptimizerAdamWAdamWLearning rate2×10−52\\times 10^\{\-5\}1×10−41\\times 10^\{\-4\}Weight decay0\.010\.05LR schedulerconstantcosineTraining length6 epochs1500 stepsBatch size11Gradient accumulation816Max grad norm1\.01\.0Warmup steps–500Minimum LR scale–0\.1LoRA adapters are inserted intoq\_proj,k\_proj,v\_proj,o\_proj,gate\_proj,up\_proj, anddown\_proj\. The LoRA rank and scaling factor are both set to 128, and the bias term is not adapted\. We use a LoRA dropout of 0\.0 for LLaDA and 0\.05 for MMaDA\. The balance weightλ\\lambdain the WINO\+ training objective is set to 0\.1 for all experiments\. The optimization configurations are summarized in Table[VIII](https://arxiv.org/html/2605.16941#A2.T8)\.

All WINO\+ training experiments are conducted on 8 NVIDIA RTX 3090 GPUs with DeepSpeed\-based distributed optimization\[rajbhandari2020zero\]\. We use CPU offloading and communication\-overlap optimization to reduce memory overhead during LoRA fine\-tuning\.