Zarya: A Hybrid Autoregressive--Masked Diffusion Language Model with Flexible Training and Dual-Mode Inference

# Zarya: A Hybrid Autoregressive–Masked Diffusion Language Model with Flexible Training and Dual-Mode Inference
Source: [https://arxiv.org/html/2609.19868](https://arxiv.org/html/2609.19868)
###### Abstract

Autoregressive language models \(ARMs\) are constrained by sequential, left\-to\-right generation, while masked diffusion models \(MDMs\) enable parallel decoding but suffer from high computational overhead due to the inability to reuse Key\-Value \(KV\) cache and from incoherent generation arising from learning dependencies over an intractable space of token combinations\. We introduceZarya, a family of hybrid language models that jointly optimizes an autoregressive \(AR\) objective and a masked\-diffusion objective within a single architecture\.Zaryastructures training data into variable\-size*slots*and employs a curriculum that gradually increases slot granularity, enabling a smooth transition from fine\-grained AR learning to coarse\-grained diffusion learning\. At inference,Zaryaprovides two distinct decoding paradigms through a unified interface: \(i\) MDM sampling with first\-hitting denoising, and \(ii\)*slotted speculative decoding*that interleaves inter\-slot diffusion\-based selection with intra\-slot autoregressive infilling, achieving full KV cache reuse\. The training and inference regimes are fully decoupled, allowing a model trained with any configuration to be deployed in either mode\. Extensive configurability — including grouped noise patterns \(Prefix Completion, Fill\-In\-the\-Prefix, Fill\-In\-the\-Middle\), ordered sampling schedules, and noise\-level permutation strategies — enables flexible research exploration\. We release Zarya models publicly in sizes 0\.6B, 1\.7B, and 4B, demonstrating performance on standard benchmarks while offering a principled integration of autoregressive and diffusion paradigms\.

## 1Introduction

Autoregressive models \(ARMs\) have achieved remarkable success in a wide range of natural language tasks[Brown et al\., 2020](https://arxiv.org/html/2609.19868#bib.bib2);[OpenAI, 2023](https://arxiv.org/html/2609.19868#bib.bib7);[Touvron et al\., 2023](https://arxiv.org/html/2609.19868#bib.bib9)\. However, their sequential, left\-to\-right decoding fundamentally limits inference throughput, preventing parallelization[Chen et al\., 2023](https://arxiv.org/html/2609.19868#bib.bib4);[Cai et al\., 2024](https://arxiv.org/html/2609.19868#bib.bib3)\. Masked diffusion models \(MDMs\) offer a compelling alternative by enabling parallel generation through an iterative denoising process without a fixed generation order[Li et al\., 2026b](https://arxiv.org/html/2609.19868#bib.bib6)\. However, most MDMs suffer from two critical drawbacks: \(i\) they exclude Key\-Value \(KV\) caching, incurring high computational overhead during inference; and \(ii\) they learn dependencies over an intractable space of token combinations, leading to incoherent generation[Li et al\., 2026a](https://arxiv.org/html/2609.19868#bib.bib5)\.

Recent hybrid approaches have sought to bridge these paradigms\. Block Diffusion \(BD3\-LMs\)[Arriola et al\., 2025](https://arxiv.org/html/2609.19868#bib.bib1)interpolates between AR and MDM by grouping tokens into fixed\-size blocks and unmasking them from left to right, but the block size is fixed and lacks flexibility\. Esoteric Language Models \(Eso\-LMs\)[Sahoo et al\., 2026](https://arxiv.org/html/2609.19868#bib.bib8)fuse AR and MDM paradigms at the loss level, enabling KV caching for MDMs and achieving faster inference than contemporary MDMs\.ReFusion[Li et al\., 2026a](https://arxiv.org/html/2609.19868#bib.bib5)elevates parallel decoding from the token level to a higher*slot level*, interleaving inter\-slot diffusion\-based selection with intra\-slot autoregressive infilling, outperforming Qwen3\-8B[Yang et al\., 2025](https://arxiv.org/html/2609.19868#bib.bib10)on GSM8K[Cobbe et al\., 2021](https://arxiv.org/html/2609.19868#bib.bib11)and MBPP[Austin et al\., 2021](https://arxiv.org/html/2609.19868#bib.bib13)while being 2\.33× faster on average\.

We introduceZarya, a novel hybrid architecture with the following key contributions:

1. 1\.Training with gradually increasing slot length:We partition the predicted sequence into fixed\-length, consecutive sub\-sequences, referred to as slots\. The size of the slots gradually increased during training\. For each sequence, we randomly mask several slots, reorder the input so that clean slots precede masked ones, and also permute the original order of both masked and clean slots\. The model simultaneously learns AR next\-token prediction on visible slots for sequential generation, and a denoising loss on the masked slots for context\-aware parallel reconstruction\.
2. 2\.Training–inference decoupling:Unlike prior work where the training configuration dictates the inference mode,Zaryaallows any trained model to be deployed in either MDM sampling or slotted speculative decoding mode via a single inference flag, offering higher flexibility\. Both modes are fully using KV cache with causal attention masks

## 2Related Work

##### Block Diffusion \(BD3\-LMs\)\.

BD3\-LMsinterpolate between AR and MDM by grouping tokens into blocks and unmasking them left\-to\-right[Arriola et al\., 2025](https://arxiv.org/html/2609.19868#bib.bib1)\. However, the block size is fixed, limiting flexibility\.

##### Esoteric Language Models \(Eso\-LMs\)\.

Eso\-LMsfuse AR and MDM paradigms using causal attention, enabling exact likelihood computation and KV caching for MDMs[Sahoo et al\., 2026](https://arxiv.org/html/2609.19868#bib.bib8)\. They achieve low perplexities among diffusion models on One Billion WordsLM1B;[Chelba et al\., 2014](https://arxiv.org/html/2609.19868#bib.bib14)and OpenWebTextOWT;[Gokaslan et al\., 2019](https://arxiv.org/html/2609.19868#bib.bib15)datasets\.

##### ReFusion\.

ReFusionintroduces slot\-level parallel decoding, elevating generation from tokens to fixed\-length slots[Li et al\., 2026a](https://arxiv.org/html/2609.19868#bib.bib5)\. It interleaves inter\-slot diffusion\-based selection with intra\-slot autoregressive infilling, reordering newly generated slots ahead of remaining masks after each iteration\. This design unlocks full KV cache reuse and reduces learning complexity from an intractable token combination space to a manageable slot\-level permutation space\. However,ReFusioninference is tied only for prefix completion tasks\.

## 3Zarya Architecture

### 3\.1Model Backbone and Configuration

Zaryawraps a Qwen3 backbone[Yang et al\., 2025](https://arxiv.org/html/2609.19868#bib.bib10)with a customZaryaclass registered with Hugging Face’sAutoModelandAutoConfigsystems\. We initializeZaryafrom the Qwen3\-0\.6B, Qwen3\-1\.7B, and Qwen3\-4B checkpoints, respectively, and fine\-tune it for 1 epoch on a diverse 32M\-sample instructional SFT dataset \(approximately 37\.7B tokens\) covering mathematics, coding, and general instruction\-following tasks\. We releaseZaryain three sizes \(see[Table 1](https://arxiv.org/html/2609.19868#S3.T1)\)\.

Because the Qwen3 backbone retains causal attention, masked positions cannot attend to future masked positions\. Thus, the diffusion objective used byZaryais a causal masked\-reconstruction objective rather than the fully bidirectional masked\-token objective commonly used in masked diffusion language models\. The reordering of visible and masked positions ensures that all masked positions can attend to the visible prefix while preserving the causal attention pattern and KV\-cache compatibility\.

Table 1:Model architecture of Zarya modelsThe architecture is parameterized with diffusion\-specific hyperparameters, which extends common AR model parameters:

- •α0\\alpha\_\{0\}\(default 0\.25\) andϵ\\epsilon\(default 0\.001\): parameters of the linear noise scheduleαt=α0​\(1−t\)\\alpha\_\{t\}=\\alpha\_\{0\}\(1\-t\);
- •λ=diffusion\_loss\_proportion\\lambda=\\texttt\{diffusion\\\_loss\\\_proportion\}\(default 0\.5\): weighting between MDM and AR losses;
- •sequential\_shuffle/diffusion\_shuffle: control slot/token shuffling for each phase;
- •ordered\_sampling: monotonically increasespmaskp\_\{\\text\{mask\}\}left\-to\-right;
- •grouped\_noiseandmax\_span\_length: enable Prefix Completion, Fill\-in\-the\-Prefix, Fill\-in\-the\-Middle, and random\-span masking patterns;
- •noise\_sorting: reorders tokens by mask/unmask state before the forward pass\.

## 4Training

### 4\.1Slotted Training

Whenslotted\_training=True, theforward\_process\(\)transforms each batch as follows \(this regime is heavily inspired by[Li et al\. \(2026a\)](https://arxiv.org/html/2609.19868#bib.bib5)\):

1. 1\.Slot partitioning:Each answer is split into slots of sizeslot\_size\(fromslot\_size\_set, e\.g\., for released model checkpoints\[2,4,8,16,32,64\]\[2,4,8,16,32,64\]\)\.
2. 2\.Mask sampling:For each example in batch, sample mask probability \(pmaskp\_\{\\text\{mask\}\}\) uniformly\. This determines what fraction of slots will be masked \(treated as the diffusion task\)\.
3. 3\.Slot assignment: - •*AR slots*: tokens remain visible; the model predicts the next token within each slot, yieldingℒseq\\mathcal\{L\}\_\{\\text\{seq\}\}\. - •*MDM slots*: all tokens are replaced with the special⟨mdm\_mask⟩\\langle\\texttt\{mdm\\\_mask\}\\rangletoken; the model reconstructs the original tokens, yieldingℒdif\\mathcal\{L\}\_\{\\text\{dif\}\}\. Each token inside slot assignedpmaskp\_\{\\text\{mask\}\}value to be used in per\-token normalization as1pmask\\frac\{1\}\{p\_\{\\text\{mask\}\}\}weighting\.

The final loss is a linear combination:

ℒ=λ⋅ℒdif\+\(1−λ\)⋅ℒseq,\\mathcal\{L\}=\\lambda\\cdot\\mathcal\{L\}\_\{\\text\{dif\}\}\+\(1\-\\lambda\)\\cdot\\mathcal\{L\}\_\{\\text\{seq\}\},\(1\)whereλ\\lambdais diffusion loss proportion\.

This design enables the model to simultaneously learn next\-token prediction \(AR\) and masked\-token reconstruction \(MDM\) on the*same*input, with explicit slot boundaries providing a structured inductive bias\.

Optional flags include:

- •ordered\_sampling: inside each slot modify per\-tokenpmaskp\_\{\\text\{mask\}\}so that it increases left\-to\-right, asℒdif\\mathcal\{L\}\_\{\\text\{dif\}\}is scaled with the value of1pmask\\frac\{1\}\{p\_\{\\text\{mask\}\}\}, this makes tne model learn that correct prediction of the slot beginning tokens is more important;

#### 4\.1\.1Slot\-Size Curriculum

Slot sizes evolve during training viaslot\_step\_borders\(epoch or step thresholds\)\. For example, with

slot\_size\_set=\[2,4,8,16,32,64\],\\displaystyle=\[2,4,8,16,32,64\],slot\_step\_borders=\[0\.06,0\.2,0\.4,0\.6,0\.8,1\.0\],\\displaystyle=\[0\.06,0\.2,0\.4,0\.6,0\.8,1\.0\],the slot size gradually increases from 2 to 64 over the course of training\. This curriculum eases the model from fine\-grained AR learning \(small slots, many predictions\) to coarse\-grained diffusion learning \(large slots, holistic reconstruction\), providing a smooth transition between paradigms\. Worth noting that increase in slot size also increases calculated loss \(see[Figure 1](https://arxiv.org/html/2609.19868#S4.F1)\)\.

![Refer to caption](https://arxiv.org/html/2609.19868v1/images/Zarya-4B-train-loss.png)Figure 1:Loss curve during training of Zarya\-4B model\. Step\-by\-step increasing of slot size causes increase of loss too

### 4\.2Non\-Slotted Training

Whenslotted\_training=False, the model runs objective over raw sequences without slot partitioning \(this regime is heavily inspired by[Sahoo et al\. \(2026\)](https://arxiv.org/html/2609.19868#bib.bib8)\):

1. 1\.For each example in batch,sample noise level\(tt\) uniformly\.
2. 2\.Construct noisy input\. Noise schedule isαt=α0​\(1−t\)\\alpha\_\{t\}=\\alpha\_\{0\}\(1\-t\)and the per\-token mask probability ispmask=1−αtp\_\{\\text\{mask\}\}=1\-\\alpha\_\{t\}\.
3. 3\.Runtwo separate forward passes: - •*sequential phase*: clean sequencex0x\_\{0\}, predict only tokens at masked positions \(others set toignore\_index\)\. - •*diffusion phase*: noisy sequencextx\_\{t\}, reconstruct masked tokens\.
4. 4\.Final loss is linear combination of losses at sequential and diffusion phases, calculated the same way as in[Section 4\.1](https://arxiv.org/html/2609.19868#S4.SS1)\.

Optional flags include:

- •noise\_sorting: reorders tokens by mask/unmask state before both forward passes, with logits permuted back afterward;
- •grouped\_noise: masks contiguous spans \(Prefix Completion, Fill\-in\-the\-Prefix, Fill\-in\-the\-Middle, random spans\) up tomax\_span\_length;
- •ordered\_sampling: offsetsttper token position so thatpmaskp\_\{\\text\{mask\}\}increases left\-to\-right\.

## 5Inference: Dual\-Mode Unified Interface

Zaryaprovides two distinct decoding paradigms through a singlemodel\.generate\(\)call, routed viageneration\_config\.slotted\_generation\. Critically, the training and inference regimes are fully decoupled: a model trained withslotted\_training=Falsecan still be deployed withslotted\_generation=True, and vice versa\.

### 5\.1Mode A: MDM Sampling

Whenslotted\_generation=False, inference code executes the first\-hitting denoising process[Zheng et al\., 2025](https://arxiv.org/html/2609.19868#bib.bib16);[Sahoo et al\., 2026](https://arxiv.org/html/2609.19868#bib.bib8)\. Starting from the prompt padded with⟨mdm\_mask⟩\\langle\\texttt\{mdm\\\_mask\}\\rangletokens up tomax\_length, the model iteratively reveals tokens:

1. 1\.Mask budget planning:\_tokens\_unmasked\_per\_step\(\)determines how many masks to reveal per step\.α0\\alpha\_\{0\}is the expected fraction of masked tokens generated with diffusion process\. - •If number of discretization steps is set toT\>0T\>0: exactlyTTdiffusion steps are used\. Binomial distribution is used to calculate number of masked tokens to denoise through diffusion process and tokens thats left after that to denoise sequentially \(this mode is heavily inspired by[Sahoo et al\. \(2026, Appendix B\.5\)](https://arxiv.org/html/2609.19868#bib.bib8)\)\. - •If number of discretization steps is set toT=0T=0\(ignoring noise calculations\):TTsteps are auto\-calculated as14\\frac\{1\}\{4\}of masked tokens\.
2. 2\.Reordering:Input sequence is reordered so that masked tokens are always after unmasked\.
3. 3\.Per\-step sampling from categorical distribution:At each step, the model receives the progressively filled sequence \(with KV cache reuse\) and yields logits for masked positions\. Gumbel noise\([Gumbel, 1935](https://arxiv.org/html/2609.19868#bib.bib17);[Zheng et al\., 2025](https://arxiv.org/html/2609.19868#bib.bib16), Appendix F\)is added for categorical sampling, and standard sampling parameters \(temperature,top\_p,repetition\_penalty\) are honored\.
4. 4\.Restoration:After all steps, the sequence is restored to the original token order\.

The KV cache can be reused because the sequence is reordered so that tokens whose values are fixed at a given denoising step precede the remaining masked positions\. Under causal attention, the cached prefix states therefore remain unchanged when masked positions are progressively filled\. This property would not hold for a bidirectional masked\-diffusion attention pattern, where changing any previously masked token could affect the representations of other masked positions\.

### 5\.2Mode B: Slotted Speculative Decoding

Whenslotted\_generation=True, inference code achieves parallelization by elevating decoding units from tokens to slots, fully reusing KV cache to avoid recomputation\. This mode follows theReFusionparadigm[Li et al\., 2026a](https://arxiv.org/html/2609.19868#bib.bib5):

1. 1\.Reordering:Before the first forward pass, input sequence is reordered so that masked tokens are always after unmasked\.
2. 2\.Block construction:max\_new\_tokenscount of masked tokens is divided intoserial\_num\_blocksblocks of lengthblock\_size=⌊max\_new\_tokens/serial\_num\_blocks⌋\\texttt\{block\\\_size\}=\\lfloor\\texttt\{max\\\_new\\\_tokens\}/\\texttt\{serial\\\_num\\\_blocks\}\\rfloor\. Within each block, tokens are grouped into slots of sizeslot\_size\. Ifmax\_new\_tokensis small,serial\_num\_blocksis forced to 1 to prevent zero\-length blocks\.
3. 3\.Draft phase:An MDM forward pass drafts tokens for all slots in the current block in parallel\. The confidence of each slot is estimated as the probability of its first token\.
4. 4\.Sampling from categorical distribution:Iftemperatureis positive, apply Gumbel noise to logits for sampling from categorical distributions\.
5. 5\.Slot selection:Slots with confidence exceedingslot\_thresholdare accepted immediately\. If no slots are confident enough, select the most confident one, so we always have at least one slot to process further\.
6. 6\.Verification phase:Selected slots undergo an AR verification forward pass\. Tokens with probability exceedingtoken\_thresholdare accepted; those below are iteratively refined in a speculative loop\.
7. 7\.KV cache update:The cache is updated incrementally with accepted tokens, avoiding recomputation for verified prefixes\.
8. 8\.Restoration:After all steps, the sequence is restored to the original token order\.

### 5\.3Parameter Decoupling Summary

[Table 2](https://arxiv.org/html/2609.19868#S5.T2)summarizes the scope of key configuration parameters\.

Table 2:Scope of configuration parameters across training and inference modes\.ParameterTrainingInferenceslotted\_training✓—slot\_size\_set/slot\_step\_borders✓—diffusion\_loss\_proportion\(λ\\lambda\)✓—noise\_sorting✓—ordered\_sampling✓—grouped\_noise/max\_span\_length✓—add\_loss\_path/scale\_by\_batch✓—slotted\_generation—✓\(routes Mode A/B\)TT\(diffusion steps\)—✓\(Mode A\)sequential\_shuffle/diffusion\_shuffle✓✓\(Mode A only\)slot\_size/serial\_num\_blocks—✓\(Mode B\)slot\_threshold/token\_threshold—✓\(Mode B\)temperature/top\_p/repetition\_penalty—✓\(both modes\)

## 6Evaluation

We evaluate on standard benchmarks including GSM8K \(mathematical reasoning\)[Cobbe et al\., 2021](https://arxiv.org/html/2609.19868#bib.bib11), HellaSwag \(commonsense reasoning\)[Zellers et al\., 2019](https://arxiv.org/html/2609.19868#bib.bib12), IFEval \(instruction following\)[Zhou et al\., 2023](https://arxiv.org/html/2609.19868#bib.bib18), and MBPP \(code generation\)[Austin et al\., 2021](https://arxiv.org/html/2609.19868#bib.bib13)\. Preliminary results for Zarya\-0\.6B are shown in[Table 3](https://arxiv.org/html/2609.19868#S6.T3)\. Hardware info: GPU A100; GPU driver CUDA version 13\.2; GPU driver version 595\.71\.05; Docker info: Torch: 2\.9\.0\+cu128; Transformers: 5\.12\.1; CUDNN in torch: 91002; lm\-eval 0\.4\.12; Inference info: BF16, apply chat template,`slotted\_generation=true`,`slot\_size=16`,`serial\_num\_blocks=4`,`slot\_threshold=0\.9`,`token\_threshold=0\.4`

Table 3:Preliminary evaluation results for Zarya\-0\.6B
## 7Conclusion

We introducedZarya, a hybrid AR–MDM language model that combines slotted training, dual\-mode inference, and extensive configurability\. By structuring data into slots and gradually increasing slot size during training,Zaryaachieves a principled integration of autoregressive and diffusion paradigms\. The unified inference interface supports both MDM sampling and slotted speculative decoding mode with full KV cache reuse\.Zaryarepresents a step toward flexible, efficient language models that leverage the strengths of both generation paradigms\. We release models at three sizes and provide a comprehensive training framework for further research\.

## Limitations

WhileZaryademonstrates strong potential, several limitations remain:

1. 1\.Threshold sensitivity:The performance of slotted speculative decoding depends critically on slot selection threshold and token selection threshold\. Poorly tuned thresholds can lead to low acceptance rates or quality degradation\. An adaptive thresholding mechanism is needed\.
2. 2\.Mode\-specific ignorance:Researchers must remember thatordered\_samplingandnoise\_sortingconfiguration options do*nothing*during inference\. If these are tuned heavily during training, users may see no effect at inference time\.

## Acknowledgments

We thank the open\-source community for providing the foundation upon which this work is built\.

## References

- Arriolaet al\.\(2025\)M\. Arriola, S\. S\. Sahoo, A\. Gokaslan, Z\. Yang, Z\. Qi, J\. Han, J\. T\. Chiu, and V\. KuleshovBlock diffusion: interpolating between autoregressive and diffusion language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=tyEyYT267x)Cited by:[§1](https://arxiv.org/html/2609.19868#S1.p2.1),[§2](https://arxiv.org/html/2609.19868#S2.SS0.SSS0.Px1.p1.1)\.
- Austinet al\.\(2021\)J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. SuttonProgram synthesis with large language models\.External Links:2108\.07732,[Link](https://arxiv.org/abs/2108.07732)Cited by:[§1](https://arxiv.org/html/2609.19868#S1.p2.1),[§6](https://arxiv.org/html/2609.19868#S6.p1.1)\.
- Brownet al\.\(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. AmodeiLanguage models are few\-shot learners\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\. F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 1877–1901\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2609.19868#S1.p1.1)\.
- Caiet al\.\(2024\)T\. Cai, Y\. Li, Z\. Geng, H\. Peng, J\. D\. Lee, D\. Chen, and T\. DaoMedusa: simple LLM inference acceleration framework with multiple decoding heads\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,External Links:[Link](https://openreview.net/forum?id=PEpbUobfJv)Cited by:[§1](https://arxiv.org/html/2609.19868#S1.p1.1)\.
- Chelbaet al\.\(2014\)C\. Chelba, T\. Mikolov, M\. Schuster, Q\. Ge, T\. Brants, P\. Koehn, and T\. RobinsonOne billion word benchmark for measuring progress in statistical language modeling\.External Links:1312\.3005,[Link](https://arxiv.org/abs/1312.3005)Cited by:[§2](https://arxiv.org/html/2609.19868#S2.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2023\)C\. Chen, S\. Borgeaud, G\. Irving, J\. Lespiau, L\. Sifre, and J\. JumperAccelerating large language model decoding with speculative sampling\.External Links:2302\.01318,[Link](https://arxiv.org/abs/2302.01318)Cited by:[§1](https://arxiv.org/html/2609.19868#S1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.External Links:2110\.14168,[Link](https://arxiv.org/abs/2110.14168)Cited by:[§1](https://arxiv.org/html/2609.19868#S1.p2.1),[§6](https://arxiv.org/html/2609.19868#S6.p1.1)\.
- Gokaslanet al\.\(2019\)A\. Gokaslan, V\. Cohen, E\. Pavlick, and S\. TellexOpenWebText corpus\.External Links:[Link](http://skylion007.github.io/OpenWebTextCorpus)Cited by:[§2](https://arxiv.org/html/2609.19868#S2.SS0.SSS0.Px2.p1.1)\.
- Gumbel \(1935\)E\. J\. GumbelLes valeurs extrêmes des distributions statistiques\.5\(2\),pp\. 115–158\.External Links:[Link](https://www.numdam.org/item/AIHP_1935__5_2_115_0/)Cited by:[item 3](https://arxiv.org/html/2609.19868#S5.I1.i3.p1.1)\.
- Liet al\.\(2026a\)J\. Li, J\. Guan, W\. Wu, and C\. LiReFusion: a diffusion large language model with parallel autoregressive decoding\.InInternational Conference on Learning Representations,C\. Vondrick, B\. Hariharan, C\. Raffel, L\. Pinto, D\. Yang, and A\. Faust \(Eds\.\),Vol\.2026,pp\. 53846–53869\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/file/585979c057a1b30796cf317063559638-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2609.19868#S1.p1.1),[§1](https://arxiv.org/html/2609.19868#S1.p2.1),[§2](https://arxiv.org/html/2609.19868#S2.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2609.19868#S4.SS1.p1.1),[§5\.2](https://arxiv.org/html/2609.19868#S5.SS2.p1.1)\.
- Liet al\.\(2026b\)T\. Li, M\. Chen, B\. Guo, and Z\. ShenA survey on diffusion language models\.External Links:2508\.10875,[Link](https://arxiv.org/abs/2508.10875)Cited by:[§1](https://arxiv.org/html/2609.19868#S1.p1.1)\.
- OpenAI \(2023\)OpenAIGPT\-4 technical report\.External Links:2303\.08774,[Link](https://arxiv.org/abs/2303.08774)Cited by:[§1](https://arxiv.org/html/2609.19868#S1.p1.1)\.
- Sahooet al\.\(2026\)S\. S\. Sahoo, Z\. Yang, Y\. Akhauri, J\. Liu, D\. Singh, Z\. Cheng, Z\. Liu, E\. P\. Xing, J\. Thickstun, and A\. VahdatEsoteric language models: bridging autoregressive and masked diffusion llms\.InICLR 2026 Workshop on Multimodal Intelligence,External Links:[Link](https://openreview.net/forum?id=CKrPJveQIr)Cited by:[§1](https://arxiv.org/html/2609.19868#S1.p2.1),[§2](https://arxiv.org/html/2609.19868#S2.SS0.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2609.19868#S4.SS2.p1.1),[1st item](https://arxiv.org/html/2609.19868#S5.I1.i1.I1.i1.p1.1),[§5\.1](https://arxiv.org/html/2609.19868#S5.SS1.p1.1)\.
- Touvronet al\.\(2023\)H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale, D\. Bikel, L\. Blecher, C\. C\. Ferrer, M\. Chen, G\. Cucurull, D\. Esiobu, J\. Fernandes, J\. Fu, W\. Fu, B\. Fuller, C\. Gao, V\. Goswami, N\. Goyal, A\. Hartshorn, S\. Hosseini, R\. Hou, H\. Inan, M\. Kardas, V\. Kerkez, M\. Khabsa, I\. Kloumann, A\. Korenev, P\. S\. Koura, M\. Lachaux, T\. Lavril, J\. Lee, D\. Liskovich, Y\. Lu, Y\. Mao, X\. Martinet, T\. Mihaylov, P\. Mishra, I\. Molybog, Y\. Nie, A\. Poulton, J\. Reizenstein, R\. Rungta, K\. Saladi, A\. Schelten, R\. Silva, E\. M\. Smith, R\. Subramanian, X\. E\. Tan, B\. Tang, R\. Taylor, A\. Williams, J\. X\. Kuan, P\. Xu, Z\. Yan, I\. Zarov, Y\. Zhang, A\. Fan, M\. Kambadur, S\. Narang, A\. Rodriguez, R\. Stojnic, S\. Edunov, and T\. ScialomLlama 2: open foundation and fine\-tuned chat models\.External Links:2307\.09288,[Link](https://arxiv.org/abs/2307.09288)Cited by:[§1](https://arxiv.org/html/2609.19868#S1.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§1](https://arxiv.org/html/2609.19868#S1.p2.1),[§3\.1](https://arxiv.org/html/2609.19868#S3.SS1.p1.1)\.
- Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 4791–4800\.External Links:[Document](https://dx.doi.org/10.18653/v1/P19-1472),[Link](https://aclanthology.org/P19-1472/)Cited by:[§6](https://arxiv.org/html/2609.19868#S6.p1.1)\.
- Zhenget al\.\(2025\)K\. Zheng, Y\. Chen, H\. Mao, M\. Liu, J\. Zhu, and Q\. ZhangMasked diffusion models are secretly time\-agnostic masked models and exploit inaccurate categorical sampling\.InInternational Conference on Learning Representations,Y\. Yue, A\. Garg, N\. Peng, F\. Sha, and R\. Yu \(Eds\.\),Vol\.2025,pp\. 63186–63227\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/9e3b203e72c4e058de26d02a92a81844-Paper-Conference.pdf)Cited by:[item 3](https://arxiv.org/html/2609.19868#S5.I1.i3.p1.1),[§5\.1](https://arxiv.org/html/2609.19868#S5.SS1.p1.1)\.
- Zhouet al\.\(2023\)J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. HouInstruction\-following evaluation for large language models\.External Links:2311\.07911,[Link](https://arxiv.org/abs/2311.07911)Cited by:[§6](https://arxiv.org/html/2609.19868#S6.p1.1)\.