Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding
Summary
Set Diffusion introduces a new class of language models that interpolates between autoregressive and diffusion models by factorizing token generation over flexible-position, flexible-length token sets. This enables faster decoding and flexible token ordering, achieving better speed-quality tradeoffs on reasoning, summarization, and unconditional generation tasks.
View Cached Full Text
Cached at: 07/03/26, 05:43 AM
# Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding Source: [https://arxiv.org/abs/2607.01775](https://arxiv.org/abs/2607.01775) [View PDF](https://arxiv.org/pdf/2607.01775) > Abstract:Discrete diffusion models have steadily improved in quality relative to autoregressive \(AR\) models\. However, these models are normally constrained to fixed\-length generation and do not support key\-value \(KV\) caching\. Block diffusion partially bridges diffusion and AR by generating token blocks left\-to\-right, but its fixed\-size sequential blocks limit decoding flexibility and parallelism\. Here, we present a new class of language models, set diffusion, comprised of \(i\) a likelihood parameterization that factorizes over flexible\-position, flexible\-length token sets and \(ii\) a set\-causal diffusion architecture that supports KV cache updates after every inference step\. By factorizing over token sets instead of fixed\-size blocks, tokens can be decoded in arbitrarily\-ordered sets, including sliding\-window sets, enabling faster inference and support for any\-order decoding\. Set diffusion achieves better speed\-quality tradeoffs on mathematical reasoning, summarization, and unconditional generation compared to prior diffusion language models while offering stronger infilling performance than block diffusion\. We provide the code, along with the model weights and blog post on the project page:[this https URL](https://m-arriola.com/setdlms/) ## Submission history From: Marianne Arriola \[[view email](https://arxiv.org/show-email/ed38dc4f/2607.01775)\] **\[v1\]**Thu, 2 Jul 2026 06:45:43 UTC \(552 KB\)
Similar Articles
DiffRetriever: Parallel Representative Tokens for Retrieval with Diffusion Language Models
This paper introduces DiffRetriever, a method that uses diffusion language models to generate multiple representative tokens in parallel for efficient information retrieval, outperforming autoregressive baselines in speed and accuracy.
Supportive Token Revealing for Fast Diffusion Language Model Decoding
This paper proposes AXON, a training-free module that improves the quality-latency trade-off of discrete diffusion language model decoding by intelligently selecting 'anchor' tokens to reveal first, using attention, uncertainty, and confidence signals to support subsequent denoising steps. Experiments on reasoning and code-generation benchmarks show AXON reduces function evaluations while maintaining or improving accuracy.
Drifting Objectives for Refining Discrete Diffusion Language Models
This paper introduces TokenDrift, a drifting objective that refines discrete diffusion language models by lifting categorical predictions to a continuous semantic space for anti-symmetric drifting, significantly improving generation quality under a fixed number of denoising steps.
Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis
This paper proposes Nemotron-Labs-Diffusion-Image, a masked discrete diffusion model for high-resolution text-to-image synthesis, introducing a token-editing mechanism and grouped cross-entropy objective to improve token refinement and training efficiency.
Speculative Refinement: A Hybrid Autoregressive Diffusion Decoding Strategy and Its Behavior Across Benchmarks
Introduces Speculative Refinement (SpecRef), a training-free hybrid decoding strategy that warm-starts a masked diffusion language model from an autoregressive draft using entropy-guided selective masking. Evaluated across six benchmarks, it reveals that code benchmarks conflate structural discovery with logical correctness, identifies a refinement tension phenomenon, and shows that evaluation protocols can produce different model rankings.