Reinforcing the Generation Order of Multimodal Masked Diffusion Models

# Reinforcing the Generation Order of Multimodal Masked Diffusion Models
Source: [https://arxiv.org/html/2607.08056](https://arxiv.org/html/2607.08056)
Yidong Ouyang1, Zhe Wang2, Sourav Bhabesh2, Dmitriy Bespalov2 1University of California, Los Angeles2AGI Foundations for AWS

###### Abstract

Diffusion Language Models \(DLMs\) have recently achieved substantial progress in natural language generation tasks\. Recent research demonstrates that adaptive token generation ordering can significantly improve performance in mathematical reasoning and code synthesis applications\. In this work, we investigate the optimization of generation order for both text\-to\-image synthesis and multimodal understanding\. We first establish that, unlike structured problems in language generation such as Sudoku puzzles, model logits alone are insufficient for determining optimal generation sequences in text\-to\-image generation and multimodal understanding\. To address this challenge, we introduce a learnable control module trained via Group Relative Policy Optimization \(GRPO\) to determine the generation order\. Our results demonstrate that learning this control block substantially improves both text\-to\-image alignment and multimodal understanding in DLMs\. In particular, it enhances the model’s ability to capture fine\-grained spatial relationships in generated images while also strengthening performance on multimodal reasoning and comprehension tasks\. We evaluate our framework on GenEval, an object\-focused benchmark for text\-to\-image alignment, where it achieves 4\.08% relative improvements\. In addition, experiments on VLMEvalKit confirm 4\.85% relative improvements in multimodal understanding, highlighting the broad effectiveness of our approach\.

## 1Introduction

Masked diffusion models \(MDMs\)\(Austinet al\.,[2021](https://arxiv.org/html/2607.08056#bib.bib2); Hoogeboomet al\.,[2021](https://arxiv.org/html/2607.08056#bib.bib3); Sunet al\.,[2022](https://arxiv.org/html/2607.08056#bib.bib5); Louet al\.,[2023](https://arxiv.org/html/2607.08056#bib.bib7); Arriolaet al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib6); Ouet al\.,[2024](https://arxiv.org/html/2607.08056#bib.bib10); von Rutteet al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib11); Shiet al\.,[2024](https://arxiv.org/html/2607.08056#bib.bib12); Nieet al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib13); Gatet al\.,[2024](https://arxiv.org/html/2607.08056#bib.bib14); Campbellet al\.,[2024](https://arxiv.org/html/2607.08056#bib.bib15)\)have recently emerged as a powerful class of generative models for discrete data, achieving promising success in language generation tasks\. Unlike traditional autoregressive models, which follow a fixed left\-to\-right decoding scheme, MDMs are trained to solve any\-order infilling problems and can generate tokens in essentially arbitrary orders during inference\. This flexibility allows MDMs to better handle partially observed inputs and to exploit non\-sequential dependencies among tokens\.

As MDMs gained popularity, the community has begun to recognize that the generation order—the sequence in which unobserved tokens are produced—plays a crucial role in determining the final output quality\(Kimet al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib20); Huanget al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib19)\)\. Early approaches typically selected generation positions at random, but recent work has shown that order selection can be made more informed by leveraging intermediate model predictions\. For example, Zheng et al\.\(Zhenget al\.,[2023](https://arxiv.org/html/2607.08056#bib.bib21)\)introduced the Top\-KKstrategy, which prioritizes positions with the highest predicted token probabilities\. Similarly,Kimet al\.\([2025](https://arxiv.org/html/2607.08056#bib.bib20)\)proposed the Top\-KKmargin strategy, ranking positions by the absolute difference between the top two predicted probabilities\. These strategies, by exploiting the model’s confidence estimates, significantly improve performance in structured reasoning tasks such as Sudoku solving\.

However, existing studies on generation order have been confined to language generation tasks\. With the recent development of multimodal masked diffusion models\(Yanget al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib17)\), which extend MDMs to text\-to\-image generation and multimodal understanding, a natural question arises: Does the generation order matter in multimodal settings, and if so, how should it be determined? Our preliminary experiments reveal that strategies such as the Top\-KKmargin, while highly effective for symbolic reasoning, fail to yield improvements in image generation quality and multimodal understanding\. This discrepancy highlights an important gap: multimodal tasks require order control mechanisms that go beyond simple confidence\-based heuristics\.

To address this, we propose a control block for dynamically determining the generation order in multimodal masked diffusion models\. Unlike heuristic approaches that rely solely on logits, the control block is learnable and can be trained to optimize the generation process directly via Group Relative Policy Optimization \(GRPO\)\. This enables the model to adapt its generation order policy to the specific requirements of image synthesis, capturing complex interdependencies between spatially distributed visual tokens\.

We evaluate our approach on the GenEval benchmark, a challenging suite designed for rigorous assessment of text\-to\-image generation, and VLMEvalKit benchmark, a comprehensive evaluation framework for multimodal understanding covering visual question answering, reasoning, and perception\. Experimental results demonstrate that our method outperforms existing order selection strategies in this setting, confirming the importance of learning\-based control for multimodal generation\.

Our contributions are as follows:

- •We observe existing confidence\-based strategies do not yield improvement in image generation quality and multimodal reasoning\.
- •We propose a control block to determine the generation order and optimize it via Group Relative Policy Optimization \(GRPO\)\.
- •We provide extensive empirical evaluation on the GenEval benchmark and VLMEvalKit benchmark with 4\.08% and 4\.85% relative improvement respectively, demonstrating the effectiveness and generality of our approach\.

## 2Preliminary

In this section, we first introduce the background of masked diffusion models and then summarize a widely used Group Relative Policy Optimization\.

### 2\.1Masked Diffusion Model

Masked diffusion models \(MDMs\)\(Austinet al\.,[2021](https://arxiv.org/html/2607.08056#bib.bib2); Hoogeboomet al\.,[2021](https://arxiv.org/html/2607.08056#bib.bib3); Sunet al\.,[2022](https://arxiv.org/html/2607.08056#bib.bib5); Louet al\.,[2023](https://arxiv.org/html/2607.08056#bib.bib7); Arriolaet al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib6); Ouet al\.,[2024](https://arxiv.org/html/2607.08056#bib.bib10); von Rutteet al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib11); Shiet al\.,[2024](https://arxiv.org/html/2607.08056#bib.bib12); Nieet al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib13)\)are defined by a forward \(noising\) process and a reverse \(denoising\) process\. In the forward process, tokens in a sequence are progressively replaced with a special mask token until the sequence is fully masked\. The reverse process learns to reconstruct the original sequence starting from the fully masked state\.

#### Forward process

Given a clean sequencexT∈𝒮ℒx\_\{T\}\\in\\mathcal\{S\}^\{\\mathcal\{L\}\}sampled from the data distributionpdatap\_\{\\mathrm\{data\}\}111We usexTx\_\{T\}to denote the clean data sample instead of the more common convention wherex0x\_\{0\}denotes the clean data sample\. This notation is adopted to align with the GRPO\-based optimization\., where𝒮\\mathcal\{S\}denotes the finite token space, and a noise levelt∈\{0,1,⋯,T\}t\\in\\\{0,1,\\cdots,T\\\}, the forward process is defined asxt∼qt\|T\(⋅\|xT\)x\_\{t\}\\sim q\_\{t\|T\}\(\\cdot\|x\_\{T\}\)\. The corruption process independently masks each coordinate:

qt\|T​\(xt\|xT\)=∏i=1ℒqt\|T​\(xti\|xTi\),q\_\{t\|T\}\(x\_\{t\}\|x\_\{T\}\)=\\prod\_\{i=1\}^\{\\mathcal\{L\}\}q\_\{t\|T\}\(x\_\{t\}^\{i\}\|x\_\{T\}^\{i\}\),where

qt\|T​\(xti\|xTi\)=Cat⁡\(αt​exTi\+\(1−αt\)​em\)\.q\_\{t\|T\}\(x\_\{t\}^\{i\}\|x\_\{T\}^\{i\}\)=\\operatorname\{Cat\}\\left\(\\alpha\_\{t\}e\_\{x\_\{T\}^\{i\}\}\+\(1\-\\alpha\_\{t\}\)e\_\{m\}\\right\)\.Here,αt\\alpha\_\{t\}is a predefined noise schedule satisfyingα0=0\\alpha\_\{0\}=0andαT=1\\alpha\_\{T\}=1\. The vectorexTie\_\{x\_\{T\}^\{i\}\}denotes the one\-hot representation of tokenxTix\_\{T\}^\{i\}, andeme\_\{m\}denotes the one\-hot representation of the mask tokenmm\. Equivalently, each token remains unchanged with probabilityαt\\alpha\_\{t\}and is replaced by the mask token with probability1−αt1\-\\alpha\_\{t\}\.

#### Reverse process

The reverse transition conditioned on the original sequence is defined asqt\|t−1​\(xt\|xt−1,xT\)q\_\{t\|t\-1\}\(x\_\{t\}\|x\_\{t\-1\},x\_\{T\}\)\. Due to the independence across coordinates, it factorizes as

qt\|t−1​\(xt\|xt−1,xT\)=∏i=1ℒqt\|t−1​\(xti\|xt−1i,xTi\),q\_\{t\|t\-1\}\(x\_\{t\}\|x\_\{t\-1\},x\_\{T\}\)=\\prod\_\{i=1\}^\{\\mathcal\{L\}\}q\_\{t\|t\-1\}\(x\_\{t\}^\{i\}\|x\_\{t\-1\}^\{i\},x\_\{T\}^\{i\}\),where

qt\|t−1​\(xti\|xt−1i,xTi\)=\{Cat⁡\(ext−1i\),xt−1i≠m,Cat⁡\(1−αt1−αt−1​em\+αt−αt−11−αt−1​exTi\),xt−1i=m\.q\_\{t\|t\-1\}\(x\_\{t\}^\{i\}\|x\_\{t\-1\}^\{i\},x\_\{T\}^\{i\}\)=\\begin\{cases\}\\operatorname\{Cat\}\(e\_\{x\_\{t\-1\}^\{i\}\}\),&x\_\{t\-1\}^\{i\}\\neq m,\\\\\[6\.0pt\] \\operatorname\{Cat\}\\left\(\\frac\{1\-\\alpha\_\{t\}\}\{1\-\\alpha\_\{t\-1\}\}e\_\{m\}\+\\frac\{\\alpha\_\{t\}\-\\alpha\_\{t\-1\}\}\{1\-\\alpha\_\{t\-1\}\}e\_\{x\_\{T\}^\{i\}\}\\right\),&x\_\{t\-1\}^\{i\}=m\.\\end\{cases\}

#### Training objective

The model is trained by minimizing the weighted cross\-entropy objective

ℒθ=−∑t=1Tαt−αt−11−αt​𝔼xT∼pdataxt∼qt\|T\(⋅\|xT\)​\[∑i=1ℒδm​\(xti\)​exTi⊤​log⁡pθ​\(xTi\|xt,t\)\],\\mathcal\{L\}\_\{\\theta\}=\-\\sum\_\{t=1\}^\{T\}\\frac\{\\alpha\_\{t\}\-\\alpha\_\{t\-1\}\}\{1\-\\alpha\_\{t\}\}\\,\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}x\_\{T\}\\sim p\_\{\\mathrm\{data\}\}\\\\ x\_\{t\}\\sim q\_\{t\|T\}\(\\cdot\|x\_\{T\}\)\\end\{subarray\}\}\\left\[\\sum\_\{i=1\}^\{\\mathcal\{L\}\}\\delta\_\{m\}\(x\_\{t\}^\{i\}\)e\_\{x\_\{T\}^\{i\}\}^\{\\top\}\\log p\_\{\\theta\}\(x\_\{T\}^\{i\}\|x\_\{t\},t\)\\right\],whereδx​\(z\)\\delta\_\{x\}\(z\)denotes Kronecker delta satisfyingδx​\(z\)=1\\delta\_\{x\}\(z\)=1ifx=zx=z\. In practice, the denoising model is often implemented without explicit time conditioning, i\.e\.,pθ​\(xT\|xt,t\)=pθ​\(xT\|xt\)p\_\{\\theta\}\(x\_\{T\}\|x\_\{t\},t\)=p\_\{\\theta\}\(x\_\{T\}\|x\_\{t\}\), since the masked inputxtx\_\{t\}implicitly encodes the noise levelttthrough the proportion of masked tokens\.

#### Sampling process

Sampling proceeds by simulating the reverse process fromt=0t=0tot=Tt=T\. Starting from a fully masked sequencex0x\_\{0\}, we iteratively generatextkx\_\{t\_\{k\}\}fromxtk−1x\_\{t\_\{k\-1\}\}\. At each step, the model first predicts the clean sequence

x^T∼pθ\(⋅∣xtk−1\),\\hat\{x\}\_\{T\}\\sim p\_\{\\theta\}\(\\cdot\\mid x\_\{t\_\{k\-1\}\}\),and then samples

xtk∼qtk∣tk−1\(⋅∣xtk−1,x^T\)\.x\_\{t\_\{k\}\}\\sim q\_\{t\_\{k\}\\mid t\_\{k\-1\}\}\\big\(\\cdot\\mid x\_\{t\_\{k\-1\}\},\\hat\{x\}\_\{T\}\\big\)\.
The transition is applied independently across coordinates, where only masked positions are updated\. Repeating this procedure progressively removes masks and yields samplexTx\_\{T\}\.

### 2\.2GRPO

Policy gradient methods have been widely adopted in the post\-training stage to enhance the performance of large models\(Ouyanget al\.,[2022](https://arxiv.org/html/2607.08056#bib.bib22)\)\. While Proximal Policy Optimization \(PPO\)\(Schulmanet al\.,[2017](https://arxiv.org/html/2607.08056#bib.bib23)\)remains a dominant approach in online RL, it requires learning an additional value functionVV, leading to increased computational overhead\. Group Relative Policy Optimization \(GRPO\)\(Shaoet al\.,[2024](https://arxiv.org/html/2607.08056#bib.bib24)\)provides a more efficient alternative by estimating advantages via group statistics\.

In the multimodal setting, we consider a conditioning inputc=\(xtext,ximg\)c=\(x^\{\\text\{text\}\},x^\{\\text\{img\}\}\), wherextextx^\{\\text\{text\}\}denotes the textual prompt andximgx^\{\\text\{img\}\}denotes the visual input \(which may be empty for text\-to\-image generation\)\. For each conditioning inputcc, GRPO samples a group ofGGoutputs\{o1,…,oG\}\\\{o\_\{1\},\\ldots,o\_\{G\}\\\}from the old policyπθold\(⋅∣c\)\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\cdot\\mid c\)\. Each outputoio\_\{i\}receives a scalar rewardrir\_\{i\}\. The advantage is defined as

Ai=ri−mean⁡\(\{rj\}j=1G\)std⁡\(\{rj\}j=1G\)\.A\_\{i\}=\\frac\{r\_\{i\}\-\\operatorname\{mean\}\\left\(\\\{r\_\{j\}\\\}\_\{j=1\}^\{G\}\\right\)\}\{\\operatorname\{std\}\\left\(\\\{r\_\{j\}\\\}\_\{j=1\}^\{G\}\\right\)\}\.The GRPO objective is given by

ℒGRPO​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{GRPO\}\}\(\\theta\)=𝔼c∼𝒟o1,…,oG∼πθold\(⋅∣c\)\[1G∑i=1G1\|oi\|∑k=1\|oi\|min\(ρikAi,clip\(ρik,1−ε,1\+ε\)Ai\)\\displaystyle=\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}c\\sim\\mathcal\{D\}\\\\ o\_\{1\},\\ldots,o\_\{G\}\\sim\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\cdot\\mid c\)\\end\{subarray\}\}\\Bigg\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{\|o\_\{i\}\|\}\\sum\_\{k=1\}^\{\|o\_\{i\}\|\}\\min\\Big\(\\rho\_\{i\}^\{k\}A\_\{i\},\\,\\operatorname\{clip\}\(\\rho\_\{i\}^\{k\},1\-\\varepsilon,1\+\\varepsilon\)A\_\{i\}\\Big\)−βDKL\(πθ\(⋅∣c,oi<k\)∥πref\(⋅∣c,oi<k\)\)\],\\displaystyle\\hskip 100\.0pt\-\\beta\\,D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\_\{\\theta\}\(\\cdot\\mid c,o\_\{i\}^\{<k\}\)\\,\\big\\\|\\,\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid c,o\_\{i\}^\{<k\}\)\\right\)\\Bigg\],whereρik=πθ​\(oik∣c,oi<k\)πθold​\(oik∣c,oi<k\)\\rho\_\{i\}^\{k\}=\\frac\{\\pi\_\{\\theta\}\(o\_\{i\}^\{k\}\\mid c,o\_\{i\}^\{<k\}\)\}\{\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(o\_\{i\}^\{k\}\\mid c,o\_\{i\}^\{<k\}\)\}\. The clipping parameterε\\varepsiloncontrols the update magnitude, whileβ\\betaweights the KL regularization toward the reference policyπref\\pi\_\{\\text\{ref\}\}\.

#### Diffusion policy optimization

A key challenge is estimating likelihoods under masked diffusion policies\. InZhaoet al\.\([2025](https://arxiv.org/html/2607.08056#bib.bib16)\), the author adopt a mean\-field approximation of sequence likelihood\. Letϕπθ​\(ok∣c′\)\\phi^\{\\pi\_\{\\theta\}\}\(o^\{k\}\\mid c^\{\\prime\}\)andϕπθ​\(o∣c′\)\\phi^\{\\pi\_\{\\theta\}\}\(o\\mid c^\{\\prime\}\)denote the estimated per\-token and sequence probabilities, respectively, wherec′c^\{\\prime\}is a noised version of the multimodal condition obtained via a masking operator:c′∼masking⁡\(c\)\.c^\{\\prime\}\\sim\\operatorname\{masking\}\(c\)\.The resulting diffu\-GRPO objective is

ℒdiffu\-GRPO​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\text\{diffu\-GRPO\}\}\(\\theta\)=𝔼c∼𝒟,c′∼masking⁡\(c\)o1,…,oG∼πθold\(⋅∣c\)\[1G∑i=1G1\|oi\|∑k=1\|oi\|min\(ρ~ikAi,clip\(ρ~ik,1−ε,1\+ε\)Ai\)\\displaystyle=\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}c\\sim\\mathcal\{D\},\\,c^\{\\prime\}\\sim\\operatorname\{masking\}\(c\)\\\\ o\_\{1\},\\ldots,o\_\{G\}\\sim\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\cdot\\mid c\)\\end\{subarray\}\}\\Bigg\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{\|o\_\{i\}\|\}\\sum\_\{k=1\}^\{\|o\_\{i\}\|\}\\min\\Big\(\\tilde\{\\rho\}\_\{i\}^\{k\}A\_\{i\},\\,\\operatorname\{clip\}\(\\tilde\{\\rho\}\_\{i\}^\{k\},1\-\\varepsilon,1\+\\varepsilon\)A\_\{i\}\\Big\)−βDKL\(ϕπθ\(⋅∣c′\)∥ϕπref\(⋅∣c′\)\)\],\\displaystyle\\hskip 100\.0pt\-\\beta\\,D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\phi^\{\\pi\_\{\\theta\}\}\(\\cdot\\mid c^\{\\prime\}\)\\,\\\|\\,\\phi^\{\\pi\_\{\\text\{ref\}\}\}\(\\cdot\\mid c^\{\\prime\}\)\\right\)\\Bigg\],whereρ~ik=ϕπθ​\(oik∣c′\)ϕπθold​\(oik∣c′\)\.\\tilde\{\\rho\}\_\{i\}^\{k\}=\\frac\{\\phi^\{\\pi\_\{\\theta\}\}\(o\_\{i\}^\{k\}\\mid c^\{\\prime\}\)\}\{\\phi^\{\\pi\_\{\\theta\_\{\\text\{old\}\}\}\}\(o\_\{i\}^\{k\}\\mid c^\{\\prime\}\)\}\.Recent work such as\(Yanget al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib17)\)improves this framework by introducing structured noising strategies that selectively mask generated tokens while preserving the multimodal condition\. InHuanget al\.\([2025](https://arxiv.org/html/2607.08056#bib.bib19)\); Wanget al\.\([2025b](https://arxiv.org/html/2607.08056#bib.bib59)\), trajectory information is further leveraged for post\-training of diffusion language models\.

## 3Method

Existing order selection strategies, such as Top\-KKand Top\-KKmargin\(Zhenget al\.,[2023](https://arxiv.org/html/2607.08056#bib.bib21); Kimet al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib20)\), rely solely on the model’s output logits to rank token positions\. While effective for structured language tasks such as Sudoku solving, we empirically find that these heuristics do not transfer to multimodal settings: logit confidence at a given position does not reliably reflect its importance to the overall image layout or visual coherence\. This motivates a*learned*control mechanism that can capture the complex spatial dependencies inherent in visual token generation\.

In this section, we introduce a*control block*that adaptively determines the generation order at each denoising step\. The control block employs the Unmask Policy Module \(UPM\)\(Huanget al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib19)\)to assign a ranking scorehθ,tih\_\{\\theta,t\}^\{i\}to each tokeniiat steptt\. Based on these scores, a Plackett\-Luce policy\(Ragain and Ugander,[2018](https://arxiv.org/html/2607.08056#bib.bib47); Niuet al\.,[2012](https://arxiv.org/html/2607.08056#bib.bib48)\)sequentially samples a top\-KKunmask set𝒰t\\mathcal\{U\}\_\{t\}without replacement, such that higher\-scoring tokens are preferentially unmasked first\.

### 3\.1Control Block for Generation Order

Letℳt\\mathcal\{M\}\_\{t\}denote the set of masked tokens remaining after steptt, i\.e\.,ℳt=ℳt−1∖𝒰n\\mathcal\{M\}\_\{t\}=\\mathcal\{M\}\_\{t\-1\}\\setminus\\mathcal\{U\}\_\{n\}\. The probability of selecting a specific unmask set𝒰t\\mathcal\{U\}\_\{t\}is

πθ,tunmask​\(𝒰t∣xt−1\)=∏k=1Kexp⁡\(hθ,tut​\(k\)\)∑j=kKexp⁡\(hθ,tut​\(j\)\)\+∑j∈ℳtexp⁡\(hθ,tut​\(j\)\)\.\\pi\_\{\\theta,t\}^\{\\text\{unmask\}\}\(\\mathcal\{U\}\_\{t\}\\mid x\_\{t\-1\}\)=\\prod\_\{k=1\}^\{K\}\\frac\{\\exp\(h\_\{\\theta,t\}^\{u\_\{t\}\(k\)\}\)\}\{\\sum\_\{j=k\}^\{K\}\\exp\(h\_\{\\theta,t\}^\{u\_\{t\}\(j\)\}\)\+\\sum\_\{j\\in\\mathcal\{M\}\_\{t\}\}\\exp\(h\_\{\\theta,t\}^\{u\_\{t\}\(j\)\}\)\}\.The UPM takes as input the hidden states of the final transformer layer at steptt, together with the step indexttand a per\-token mask indicator, and outputs ranking scoreshθ,tih\_\{\\theta,t\}^\{i\}\. TheKKtokens with the highest scores are selected to form the unmask set𝒰t\\mathcal\{U\}\_\{t\}\. Token prediction is then performed by the main multimodal diffusion model \(MMaDA blocks\), yielding

πθ,ttoken​\(xt∣xt−1,𝒰t\)=∏i∈𝒰tpθ,t​\(xti∣xt−1\),\\pi\_\{\\theta,t\}^\{\\text\{token\}\}\(x\_\{t\}\\mid x\_\{t\-1\},\\mathcal\{U\}\_\{t\}\)=\\prod\_\{i\\in\\mathcal\{U\}\_\{t\}\}p\_\{\\theta,t\}\(x\_\{t\}^\{i\}\\mid x\_\{t\-1\}\),wherepθ,t​\(xti∣xt−1\)p\_\{\\theta,t\}\(x\_\{t\}^\{i\}\\mid x\_\{t\-1\}\)is a categorical distribution over the vocabulary or image patch values\. The complete step policy factorizes as the product of these two components:

πθ,t​\(xt∣xt−1\)=πθ,tunmask​\(𝒰t∣xt−1\)⋅πθ,ttoken​\(xt∣xt−1,𝒰t\)\.\\pi\_\{\\theta,t\}\(x\_\{t\}\\mid x\_\{t\-1\}\)=\\pi\_\{\\theta,t\}^\{\\text\{unmask\}\}\(\\mathcal\{U\}\_\{t\}\\mid x\_\{t\-1\}\)\\cdot\\pi\_\{\\theta,t\}^\{\\text\{token\}\}\(x\_\{t\}\\mid x\_\{t\-1\},\\mathcal\{U\}\_\{t\}\)\.Figure[1](https://arxiv.org/html/2607.08056#S3.F1)illustrates the overall generation pipeline, in which the control block governs the generation order at each denoising step\.

### 3\.2Training the Control Block via GRPO

We formulate the control block as a policy over the reverse diffusion process and optimize it via Group Relative Policy Optimization \(GRPO\)\.

#### GRPO Objective\.

Given a multimodal inputccandGGsampled trajectories\{xtg\}t=0T\\\{x\_\{t\}^\{g\}\\\}\_\{t=0\}^\{T\}, each trajectory receives a scalar rewardrg=reward\_func​\(c,xTg\)r^\{g\}=\\texttt\{reward\\\_func\}\(c,x\_\{T\}^\{g\}\)\. Advantages are estimated from group statistics:

Ag=rg−mean​\(r1:G\)std​\(r1:G\)\.A^\{g\}=\\frac\{r^\{g\}\-\\mathrm\{mean\}\(r^\{1:G\}\)\}\{\\mathrm\{std\}\(r^\{1:G\}\)\}\.The control policy is optimized by minimizing the advantage\-weighted likelihood ratio:

ℒθ,t=−1G​∑g=1Gmin⁡\(ρθ,tg​Ag,clip⁡\(ρθ,tg,1−ε,1\+ε\)​Ag\),ρθ,tg=πθ,t​\(xtg∣xt−1g\)πold,t​\(xtg∣xt−1g\),\\mathcal\{L\}\_\{\\theta,t\}=\-\\frac\{1\}\{G\}\\sum\_\{g=1\}^\{G\}\\min\\\!\\Big\(\\rho\_\{\\theta,t\}^\{g\}\\,A^\{g\},\\;\\operatorname\{clip\}\\\!\\big\(\\rho\_\{\\theta,t\}^\{g\},\\,1\-\\varepsilon,\\,1\+\\varepsilon\\big\)\\,A^\{g\}\\Big\),\\qquad\\rho\_\{\\theta,t\}^\{g\}=\\frac\{\\pi\_\{\\theta,t\}\(x\_\{t\}^\{g\}\\mid x\_\{t\-1\}^\{g\}\)\}\{\\pi\_\{\\text\{old\},t\}\(x\_\{t\}^\{g\}\\mid x\_\{t\-1\}^\{g\}\)\},\(1\)with gradients accumulated across all diffusion steps:

∇θℒθ=∑t=1T∇θℒθ,t\.\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\theta\}=\\sum\_\{t=1\}^\{T\}\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\theta,t\}\.Unlike the single\-step density ratio estimation adopted inZhaoet al\.\([2025](https://arxiv.org/html/2607.08056#bib.bib16)\); Yanget al\.\([2025](https://arxiv.org/html/2607.08056#bib.bib17)\), we leverage full trajectory information by accumulating policy gradients over allNNdenoising steps\. This allows the control block to receive training signal from the complete generation trajectory rather than a single transition, enabling more effective optimization of the generation order\.

![Refer to caption](https://arxiv.org/html/2607.08056v1/figures/control_block.png)Figure 1:Generation pipeline of the diffusion language model with a control block for determining the generation order\.During inference, the control block and the diffusion model operate jointly at each denoising step\. At steptt, the control block first computes ranking scoreshθ,tih\_\{\\theta,t\}^\{i\}for all masked tokens inℳt−1\\mathcal\{M\}\_\{t\-1\}, and samples a subset𝒰t\\mathcal\{U\}\_\{t\}ofKKtokens according toπθ,tunmask\\pi\_\{\\theta,t\}^\{\\text\{unmask\}\}\. The diffusion model then predicts the value of each selected token viapθ,t\(⋅∣xt−1\)p\_\{\\theta,t\}\(\\cdot\\mid x\_\{t\-1\}\), yieldingxtix\_\{t\}^\{i\}for eachi∈𝒰ti\\in\\mathcal\{U\}\_\{t\}\. The masked set is updated asℳt=ℳt−1∖𝒰t\\mathcal\{M\}\_\{t\}=\\mathcal\{M\}\_\{t\-1\}\\setminus\\mathcal\{U\}\_\{t\}, and the process repeats for allTTsteps to produce the final samplexTx\_\{T\}\. The complete procedure is summarized in Algorithm[1](https://arxiv.org/html/2607.08056#alg1)\.

Algorithm 1Training Control Block for Multimodal Diffusion Language Model1:Model

θ\\theta, dataset

𝒟\\mathcal\{D\}, reward functionreward\_func

2:while

θ\\thetanot convergedandmax epochs not reacheddo

3:Sample batch of inputs

c∼𝒟c\\sim\\mathcal\{D\}
4:for

g=1g=1to

GGdo

5:Initialize

x0gx\_\{0\}^\{g\}with

ccand mask tokens

6:for

t=1t=1to

TTdo

7:Compute ranking scores

hθ,th\_\{\\theta,t\}for each token

8:Sample unmask set

𝒰t∼Plackett\-Luce​\(hθ,t,K\)\\mathcal\{U\}\_\{t\}\\sim\\text\{Plackett\-Luce\}\(h\_\{\\theta,t\},K\)
9:Sample tokens

xtg,i∼pθ,ti\(⋅∣xt−1g\)x\_\{t\}^\{g,i\}\\sim p\_\{\\theta,t\}^\{i\}\(\\cdot\\mid x\_\{t\-1\}^\{g\}\)for

i∈𝒰ti\\in\\mathcal\{U\}\_\{t\}
10:endfor

11:Compute reward

rg←reward\_func​\(c,xTg\)r^\{g\}\\leftarrow\\texttt\{reward\\\_func\}\(c,x\_\{T\}^\{g\}\)
12:endfor

13:Compute advantages

AgA^\{g\}using group statistics

14:for

n=1n=1to

TTdo

15:Compute

πθ,t​\(xtg∣xt−1g\)\\pi\_\{\\theta,t\}\(x\_\{t\}^\{g\}\\mid x\_\{t\-1\}^\{g\}\)
16:Compute per\-step loss

ℒθ,t\\mathcal\{L\}\_\{\\theta,t\}and gradient

17:endfor

18:Update

θ\\thetausing accumulated gradients

∑t=1T∇θℒθ,t\\sum\_\{t=1\}^\{T\}\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\theta,t\}
19:endwhile

## 4Experimental Results

In this section, we evaluate our proposed method on compositional image generation and human preference alignment tasks\. We first describe the experimental setup, followed by comprehensive benchmark results and ablation studies across both text\-to\-image generation in Section[4\.1](https://arxiv.org/html/2607.08056#S4.SS1)and multimodal understanding in Section[4\.2](https://arxiv.org/html/2607.08056#S4.SS2)\. Implementation details including optimizer settings, hyperparameters, and hardware configuration are provided in Appendix[A](https://arxiv.org/html/2607.08056#A1)\.

### 4\.1Text\-to\-Image Generation

#### Compositional Image Generation

We evaluate our method on the GenEval benchmark\(Ghoshet al\.,[2023](https://arxiv.org/html/2607.08056#bib.bib25)\), a comprehensive suite designed to assess text\-to\-image \(T2I\) models under complex compositional prompts involving object counting, spatial relations, and attribute binding\. The benchmark comprises six challenging sub\-tasks: Single Object, Two Objects, Counting, Colors, Position, and Color Attributes\.

FollowingLiuet al\.\([2025](https://arxiv.org/html/2607.08056#bib.bib26)\), we generate training prompts using the official GenEval scripts, which apply templates and random combinations to construct a diverse prompt dataset\. Evaluation employs the official pipeline for object detection, color identification, and spatial relation inference\.

To prevent data leakage, we strictly deduplicate the test set by treating prompts that differ only in object order \(e\.g\., “a photo ofAAandBB” vs\. “a photo ofBBandAA”\) as identical variants and removing them from training\. Based on the base model’s initial task accuracy, we set the prompt sampling ratio as Position : Counting : Attribute Binding : Colors : Two Objects : Single Object =7:5:3:1:1:07:5:3:1:1:0\.

We employ rule\-based rewards for different tasks:

- •Counting:r=1−\|Ngen−Nref\|Nrefr=1\-\\frac\{\|N\_\{\\text\{gen\}\}\-N\_\{\\text\{ref\}\}\|\}\{N\_\{\\text\{ref\}\}\}, providing partial reward when object count is correct;
- •Position/Color:Full reward when both count and position/color specifications match exactly\.

Following the same training pipeline as\(Liuet al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib26); Liet al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib28); Xueet al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib49); Heet al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib50)\), we utilize the mixture reward to learn the control block\. We adopt PickScore\(Kirstainet al\.,[2023](https://arxiv.org/html/2607.08056#bib.bib27)\)as our reward model to align model outputs with human judgments\. PickScore is trained on large\-scale human\-annotated pairwise comparisons and scores each image\-prompt pair based on multiple criteria, including prompt alignment and visual quality\.

#### Results on GenEval

Table[1](https://arxiv.org/html/2607.08056#S4.T1)presents our results on the GenEval benchmark\. Although confidence\-based methods such as Top\-K margin perform well on Sudoku puzzles, we find that such strategies do not yield improvement in image generation quality\. Even when training with PickScore reward and evaluating on GenEval metrics, our method still achieves superior performance, which demonstrates the generalization ability of the control block\. Our approach achieves the best overall performance when trained with in\-distribution rewards, representing significant improvements in patial positioning and multi\-object composition compared to baseline methods\. Notably, we observe substantial gains in the Two Objects \(\+0\.08\) and Position \(\+0\.07\) tasks, which we attribute to improved generation order control that better handles spatial relationships and multi\-object compositions\.

Table 1:Evaluation results on GenEval benchmark\. All metrics represent accuracy scores, with higher values indicating better performance\.MethodGenEval BenchmarkSingle Obj\.Two Obj\.CountingColorsPositionColor Attr\.OverallMMaDA\-COT Top\-K0\.930\.470\.310\.810\.160\.260\.49MMaDA\-COT Top\-K Margin0\.880\.490\.290\.790\.150\.280\.48MMaDA\-Ours \(PickScore\)0\.950\.450\.340\.790\.170\.300\.50MMaDA\-Ours \(GenEval\)0\.950\.550\.250\.830\.230\.260\.51

### 4\.2Multimodal Understanding

#### Benchmark Setup

We train the control block on the VLAA\-thinker dataset\(Chenet al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib51)\), focusing on thedigitandmultiple\-choice questionsubsets\. The training objective adopts a weighted reward function defined as

ℛ=2×ℛcorrectness\+0\.4×ℛformat,\\mathcal\{R\}=2\\times\\mathcal\{R\}\_\{\\text\{correctness\}\}\+0\.4\\times\\mathcal\{R\}\_\{\\text\{format\}\},whereℛcorrectness\\mathcal\{R\}\_\{\\text\{correctness\}\}evaluates whether the final answer is correct, andℛformat\\mathcal\{R\}\_\{\\text\{format\}\}evaluates whether the response contains a Chain\-of\-Thought enclosed within the<think\>tags\. We evaluate our method on a comprehensive suite of multimodal benchmarks using theVLMEvalKitframework\. This evaluation covers diverse aspects of multimodal reasoning, perception, and language understanding, ensuring robust comparison with existing approaches\.

#### Evaluation Datasets

We conduct experiments on six widely\-adopted benchmarks, each targeting different multimodal capabilities:

- •GQA\(Hudson and Manning,[2019](https://arxiv.org/html/2607.08056#bib.bib42)\): Large\-scale visual question answering focusing on compositional reasoning over real\-world images\.
- •MMMU\(Yueet al\.,[2024](https://arxiv.org/html/2607.08056#bib.bib43)\): Multimodal massive multitask benchmark spanning science, mathematics, and humanities domains for knowledge\-intensive reasoning evaluation\.
- •MMB\(Liuet al\.,[2024](https://arxiv.org/html/2607.08056#bib.bib44)\): Comprehensive multimodal benchmark for language\-vision models with diverse visual understanding tasks\.
- •SEED\(Geet al\.,[2024](https://arxiv.org/html/2607.08056#bib.bib34)\): Multi\-granularity comprehension benchmark targeting fine\-grained visual\-textual alignment\.
- •MathVista\_MINI\(Luet al\.,[2024](https://arxiv.org/html/2607.08056#bib.bib45)\): Challenging multimodal mathematical reasoning dataset requiring symbolic and visual integration\.
- •COCO\_VAL\(Linet al\.,[2014](https://arxiv.org/html/2607.08056#bib.bib46)\): MS\-COCO validation split for evaluating image\-text understanding and generation consistency\.

#### Multimodal Understanding Results

Table[2](https://arxiv.org/html/2607.08056#S4.T2)summarizes quantitative results across all evaluated datasets\. Similarly, confidence\-based methods such as Top\-K margin do not achieve consistent improvement over the baseline method\. Our approach achieves the best results on four of six benchmarks\. Consistent improvements are observed on MathVista\_MINI \(\+1\.3\) and COCO\_VAL \(\+0\.7\), both of which involve open\-ended questions and complex reasoning processes compared to multiple\-choice benchmarks\. These results validate that optimizing generation order effectively enhances multimodal understanding and reasoning capabilities\.

Table 2:Evaluation results on multimodal understanding benchmarks\.MethodGQAMMMUMMBSEEDMathVista\_MINICOCO\_VALMMaDA\-COT48\.580\.270\.240\.5227\.727\.6MMaDA\-COT Top\-K Margin47\.550\.240\.370\.5428\.726\.3MMaDA\-Ours47\.610\.310\.240\.5729\.028\.3

### 4\.3Ablation Studies and Analysis

#### Generation Order Optimization Analysis

To better understand the effect of our method on the generation process, we visualize the generation trajectories for both text\-to\-image generation and multimodal understanding tasks in Figure[2](https://arxiv.org/html/2607.08056#S4.F2)and Figure[3](https://arxiv.org/html/2607.08056#S4.F3)\. For text\-to\-image generation, our approach improves intermediate results across the trajectory, leading to higher\-quality final outputs\. For multimodal understanding, while the baseline model—despite being allowed up to 256 output tokens—often degenerates to a single\-token response \(e\.g\., “5” illustrated in Figure[3](https://arxiv.org/html/2607.08056#S4.F3)\), our control block enables more detailed reasoning\. The model is able to correctly solve counting problems and explicitly name objects to support its reasoning\. These results demonstrate that optimizing the generation order not only enhances the model’s ability to capture fine\-grained spatial relationships in images but also strengthens its performance on multimodal reasoning and comprehension tasks\.

![Refer to caption](https://arxiv.org/html/2607.08056v1/figures/trajectory_t2i.png)Figure 2:Trajectory visualization on text\-to\-image generation\.![Refer to caption](https://arxiv.org/html/2607.08056v1/figures/trajectory_mmu.png)Figure 3:Trajectory visualization on multimodal understanding\.
#### Qualitative Comparison

Figure[4](https://arxiv.org/html/2607.08056#S4.F4)presents qualitative comparisons of generated samples for both text\-to\-image generation and multimodal understanding tasks\. Our method produces more coherent and contextually appropriate outputs, particularly in scenarios requiring complex compositional reasoning\.

![Refer to caption](https://arxiv.org/html/2607.08056v1/figures/synthetic_t2i.png)\(a\)Text\-to\-Image Generation Results
![Refer to caption](https://arxiv.org/html/2607.08056v1/figures/synthetic_mmu.png)\(b\)Multimodal Understanding Results

Figure 4:Qualitative comparison of generated samples\. Our method demonstrates superior understanding and generation quality across both text\-to\-image synthesis and multimodal reasoning tasks\.

## 5Related work

#### Masked diffusion models

Masked diffusion models \(discrete diffusion model with absorbing state\) were first proposed by extending the transition probability from a Gaussian distribution used in the continuous diffusion model to a multinormal distribution\(Sohl\-Dicksteinet al\.,[2015](https://arxiv.org/html/2607.08056#bib.bib1); Austinet al\.,[2021](https://arxiv.org/html/2607.08056#bib.bib2); Hoogeboomet al\.,[2021](https://arxiv.org/html/2607.08056#bib.bib3)\)\. Then, continuous Time framework was established byCampbellet al\.\([2022](https://arxiv.org/html/2607.08056#bib.bib4)\); Sunet al\.\([2022](https://arxiv.org/html/2607.08056#bib.bib5)\)\. A line of works\(Louet al\.,[2023](https://arxiv.org/html/2607.08056#bib.bib7); Arriolaet al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib6); Shiet al\.,[2024](https://arxiv.org/html/2607.08056#bib.bib12); Ouet al\.,[2024](https://arxiv.org/html/2607.08056#bib.bib10); Zhenget al\.,[2024](https://arxiv.org/html/2607.08056#bib.bib9); Sahooet al\.,[2024](https://arxiv.org/html/2607.08056#bib.bib8); von Rutteet al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib11)\)proposed simplified objective to train masked diffusion model\. Discrete flow matching was proposed inGatet al\.\([2024](https://arxiv.org/html/2607.08056#bib.bib14)\); Campbellet al\.\([2024](https://arxiv.org/html/2607.08056#bib.bib15)\)\. InNieet al\.\([2025](https://arxiv.org/html/2607.08056#bib.bib13)\), the authors scale up the masked diffusion model to 8B, and Yang et al\.\(Yanget al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib17); Xinet al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib52); Wanget al\.,[2025a](https://arxiv.org/html/2607.08056#bib.bib18)\)generalize the masked diffusion model and discrete flow matching on multimodal tasks\.

#### Control over generation order

Recent work has increasingly recognized the importance of studying the generation order for masked diffusion model\(Kimet al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib20); Huanget al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib19)\)\. Instead of randomly selecting a set of positions to generate, Zheng et al\.\(Zhenget al\.,[2023](https://arxiv.org/html/2607.08056#bib.bib21)\)propose Top K strategy, i\.e\., the generation order is selected by the maximum probability of predicted token in each position\. InKimet al\.\([2025](https://arxiv.org/html/2607.08056#bib.bib20)\), the authors propose top K margin strategy, i\.e\., the absolute difference between the two most probable values at each position\. They show that by leveraging the information for the logits, they can dramatically improve the performance of MDM on math puzzles like Sudoku\.

## 6Conclusion

We have investigated the problem of generation order in multimodal masked diffusion models for text\-to\-image generation\. While prior logits\-based strategies such as Top\-KKmargin are effective in language reasoning tasks, we showed that they fail to improve image generation quality\. To overcome this, we proposed a learnable control block optimized with Group Relative Policy Optimization \(GRPO\), enabling adaptive and task\-aware order selection\. Experiments on the GenEval benchmark and VLMEvalKit benchmarks demonstrate improvements over existing baselines, confirming the importance of learning\-based order policies in multimodal diffusion models\.

## References

- M\. Arriola, A\. Gokaslan, J\. T\. Chiu, Z\. Yang, Z\. Qi, J\. Han, S\. S\. Sahoo, and V\. Kuleshov \(2025\)Block diffusion: interpolating between autoregressive and diffusion language models\.ArXivabs/2503\.09573\.External Links:[Link](https://api.semanticscholar.org/CorpusID:276938178)Cited by:[§1](https://arxiv.org/html/2607.08056#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.08056#S2.SS1.p1.1),[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px1.p1.1)\.
- J\. Austin, D\. D\. Johnson, J\. Ho, D\. Tarlow, and R\. van den Berg \(2021\)Structured denoising diffusion models in discrete state\-spaces\.ArXivabs/2107\.03006\.External Links:[Link](https://api.semanticscholar.org/CorpusID:235755106)Cited by:[§1](https://arxiv.org/html/2607.08056#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.08056#S2.SS1.p1.1),[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px1.p1.1)\.
- A\. Campbell, J\. Benton, V\. D\. Bortoli, T\. Rainforth, G\. Deligiannidis, and A\. Doucet \(2022\)A continuous time framework for discrete denoising models\.ArXivabs/2205\.14987\.External Links:[Link](https://api.semanticscholar.org/CorpusID:249192370)Cited by:[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px1.p1.1)\.
- A\. Campbell, J\. Yim, R\. Barzilay, T\. Rainforth, and T\. Jaakkola \(2024\)Generative flows on discrete state\-spaces: enabling multimodal flows with applications to protein co\-design\.ArXiv\.External Links:[Link](https://api.semanticscholar.org/CorpusID:267523194)Cited by:[§1](https://arxiv.org/html/2607.08056#S1.p1.1),[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px1.p1.1)\.
- H\. Chen, H\. Tu, F\. Wang, H\. Liu, X\. Tang, X\. Du, Y\. Zhou, and C\. Xie \(2025\)SFT or rl? an early investigation into training r1\-like reasoning large vision\-language models\.ArXivabs/2504\.11468\.External Links:[Link](https://api.semanticscholar.org/CorpusID:277824294)Cited by:[§4\.2](https://arxiv.org/html/2607.08056#S4.SS2.SSS0.Px1.p1.3)\.
- I\. Gat, T\. Remez, N\. Shaul, F\. Kreuk, R\. T\. Q\. Chen, G\. Synnaeve, Y\. Adi, and Y\. Lipman \(2024\)Discrete flow matching\.ArXivabs/2407\.15595\.External Links:[Link](https://api.semanticscholar.org/CorpusID:271328803)Cited by:[§1](https://arxiv.org/html/2607.08056#S1.p1.1),[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px1.p1.1)\.
- Y\. Ge, S\. Zhao, J\. Zhu, Y\. Ge, K\. Yi, L\. Song, C\. Li, X\. Ding, and Y\. Shan \(2024\)SEED\-x: multimodal models with unified multi\-granularity comprehension and generation\.ArXivabs/2404\.14396\.External Links:[Link](https://api.semanticscholar.org/CorpusID:269293538)Cited by:[4th item](https://arxiv.org/html/2607.08056#S4.I2.i4.p1.1)\.
- D\. Ghosh, H\. Hajishirzi, and L\. Schmidt \(2023\)GenEval: an object\-focused framework for evaluating text\-to\-image alignment\.ArXivabs/2310\.11513\.External Links:[Link](https://api.semanticscholar.org/CorpusID:264288728)Cited by:[§4\.1](https://arxiv.org/html/2607.08056#S4.SS1.SSS0.Px1.p1.1)\.
- X\. He, S\. Fu, Y\. Zhao, W\. Li, J\. Yang, D\. Yin, F\. Rao, and B\. Zhang \(2025\)TempFlow\-grpo: when timing matters for grpo in flow models\.ArXivabs/2508\.04324\.Cited by:[§4\.1](https://arxiv.org/html/2607.08056#S4.SS1.SSS0.Px1.p5.1)\.
- E\. Hoogeboom, D\. Nielsen, P\. Jaini, P\. Forr’e, and M\. Welling \(2021\)Argmax flows and multinomial diffusion: learning categorical distributions\.InNeural Information Processing Systems,External Links:[Link](https://api.semanticscholar.org/CorpusID:235262511)Cited by:[§1](https://arxiv.org/html/2607.08056#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.08056#S2.SS1.p1.1),[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px1.p1.1)\.
- Z\. Huang, Z\. Chen, Z\. Wang, T\. Li, and G\. Qi \(2025\)Reinforcing the diffusion chain of lateral thought with diffusion language models\.ArXivabs/2505\.10446\.External Links:[Link](https://api.semanticscholar.org/CorpusID:278635983)Cited by:[§1](https://arxiv.org/html/2607.08056#S1.p2.2),[§2\.2](https://arxiv.org/html/2607.08056#S2.SS2.SSS0.Px1.p1.5),[§3](https://arxiv.org/html/2607.08056#S3.p2.5),[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px2.p1.1)\.
- D\. A\. Hudson and C\. D\. Manning \(2019\)GQA: a new dataset for real\-world visual reasoning and compositional question answering\.2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 6693–6702\.External Links:[Link](https://api.semanticscholar.org/CorpusID:152282269)Cited by:[1st item](https://arxiv.org/html/2607.08056#S4.I2.i1.p1.1)\.
- J\. Kim, K\. Shah, V\. Kontonis, S\. M\. Kakade, and S\. Chen \(2025\)Train for the worst, plan for the best: understanding token ordering in masked diffusions\.ArXivabs/2502\.06768\.External Links:[Link](https://api.semanticscholar.org/CorpusID:276250574)Cited by:[§1](https://arxiv.org/html/2607.08056#S1.p2.2),[§3](https://arxiv.org/html/2607.08056#S3.p1.2),[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px2.p1.1)\.
- Y\. Kirstain, A\. Polyak, U\. Singer, S\. Matiana, J\. Penna, and O\. Levy \(2023\)Pick\-a\-pic: an open dataset of user preferences for text\-to\-image generation\.ArXivabs/2305\.01569\.External Links:[Link](https://api.semanticscholar.org/CorpusID:258437096)Cited by:[§4\.1](https://arxiv.org/html/2607.08056#S4.SS1.SSS0.Px1.p5.1)\.
- J\. Li, Y\. Cui, T\. Huang, Y\. Ma, C\. Fan, M\. Yang, and Z\. Zhong \(2025\)MixGRPO: unlocking flow\-based grpo efficiency with mixed ode\-sde\.ArXivabs/2507\.21802\.Cited by:[§4\.1](https://arxiv.org/html/2607.08056#S4.SS1.SSS0.Px1.p5.1)\.
- T\. Lin, M\. Maire, S\. J\. Belongie, J\. Hays, P\. Perona, D\. Ramanan, P\. Dollár, and C\. L\. Zitnick \(2014\)Microsoft coco: common objects in context\.InEuropean Conference on Computer Vision,External Links:[Link](https://api.semanticscholar.org/CorpusID:14113767)Cited by:[6th item](https://arxiv.org/html/2607.08056#S4.I2.i6.p1.1)\.
- J\. Liu, G\. Liu, J\. Liang, Y\. Li, J\. Liu, X\. Wang, P\. Wan, D\. Zhang, and W\. Ouyang \(2025\)Flow\-grpo: training flow matching models via online rl\.ArXivabs/2505\.05470\.External Links:[Link](https://api.semanticscholar.org/CorpusID:278394068)Cited by:[§4\.1](https://arxiv.org/html/2607.08056#S4.SS1.SSS0.Px1.p2.1),[§4\.1](https://arxiv.org/html/2607.08056#S4.SS1.SSS0.Px1.p5.1)\.
- Y\. Liu, H\. Duan, Y\. Zhang, B\. Li, S\. Zhang, W\. Zhao, Y\. Yuan, J\. Wang, C\. He, Z\. Liu,et al\.\(2024\)Mmbench: is your multi\-modal model an all\-around player?\.InEuropean conference on computer vision,pp\. 216–233\.Cited by:[3rd item](https://arxiv.org/html/2607.08056#S4.I2.i3.p1.1)\.
- A\. Lou, C\. Meng, and S\. Ermon \(2023\)Discrete diffusion modeling by estimating the ratios of the data distribution\.InInternational Conference on Machine Learning,External Links:[Link](https://api.semanticscholar.org/CorpusID:264451832)Cited by:[§1](https://arxiv.org/html/2607.08056#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.08056#S2.SS1.p1.1),[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px1.p1.1)\.
- P\. Lu, H\. Bansal, T\. Xia, J\. Liu, C\. Li, H\. Hajishirzi, H\. Cheng, K\. Chang, M\. Galley, and J\. Gao \(2024\)MathVista: evaluating mathematical reasoning of foundation models in visual contexts\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[5th item](https://arxiv.org/html/2607.08056#S4.I2.i5.p1.1)\.
- S\. Nie, F\. Zhu, Z\. You, X\. Zhang, J\. Ou, J\. Hu, J\. Zhou, Y\. Lin, J\. Wen, and C\. Li \(2025\)Large language diffusion models\.ArXivabs/2502\.09992\.External Links:[Link](https://api.semanticscholar.org/CorpusID:276395038)Cited by:[§1](https://arxiv.org/html/2607.08056#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.08056#S2.SS1.p1.1),[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px1.p1.1)\.
- S\. Niu, Y\. Lan, J\. Guo, and X\. Cheng \(2012\)A new probabilistic model for top\-k ranking problem\.Proceedings of the 21st ACM international conference on Information and knowledge management\.External Links:[Link](https://api.semanticscholar.org/CorpusID:13969766)Cited by:[§3](https://arxiv.org/html/2607.08056#S3.p2.5)\.
- J\. Ou, S\. Nie, K\. Xue, F\. Zhu, J\. Sun, Z\. Li, and C\. Li \(2024\)Your absorbing discrete diffusion secretly models the conditional distributions of clean data\.ArXivabs/2406\.03736\.External Links:[Link](https://api.semanticscholar.org/CorpusID:270285911)Cited by:[§1](https://arxiv.org/html/2607.08056#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.08056#S2.SS1.p1.1),[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px1.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. E\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. F\. Christiano, J\. Leike, and R\. J\. Lowe \(2022\)Training language models to follow instructions with human feedback\.ArXivabs/2203\.02155\.External Links:[Link](https://api.semanticscholar.org/CorpusID:246426909)Cited by:[§2\.2](https://arxiv.org/html/2607.08056#S2.SS2.p1.1)\.
- S\. Ragain and J\. Ugander \(2018\)Choosing to rank\.ArXivabs/1809\.05139\.External Links:[Link](https://api.semanticscholar.org/CorpusID:52280181)Cited by:[§3](https://arxiv.org/html/2607.08056#S3.p2.5)\.
- S\. S\. Sahoo, M\. Arriola, Y\. Schiff, A\. Gokaslan, E\. Marroquin, J\. T\. Chiu, A\. Rush, and V\. Kuleshov \(2024\)Simple and effective masked diffusion language models\.ArXivabs/2406\.07524\.External Links:[Link](https://api.semanticscholar.org/CorpusID:270380319)Cited by:[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px1.p1.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.ArXivabs/1707\.06347\.External Links:[Link](https://api.semanticscholar.org/CorpusID:28695052)Cited by:[§2\.2](https://arxiv.org/html/2607.08056#S2.SS2.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.ArXivabs/2402\.03300\.External Links:[Link](https://api.semanticscholar.org/CorpusID:267412607)Cited by:[§2\.2](https://arxiv.org/html/2607.08056#S2.SS2.p1.1)\.
- J\. Shi, K\. Han, Z\. Wang, A\. Doucet, and M\. K\. Titsias \(2024\)Simplified and generalized masked diffusion for discrete data\.ArXivabs/2406\.04329\.External Links:[Link](https://api.semanticscholar.org/CorpusID:270286107)Cited by:[§1](https://arxiv.org/html/2607.08056#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.08056#S2.SS1.p1.1),[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px1.p1.1)\.
- J\. N\. Sohl\-Dickstein, E\. A\. Weiss, N\. Maheswaranathan, and S\. Ganguli \(2015\)Deep unsupervised learning using nonequilibrium thermodynamics\.ArXivabs/1503\.03585\.External Links:[Link](https://api.semanticscholar.org/CorpusID:14888175)Cited by:[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px1.p1.1)\.
- H\. Sun, L\. Yu, B\. Dai, D\. Schuurmans, and H\. Dai \(2022\)Score\-based continuous\-time discrete diffusion models\.ArXivabs/2211\.16750\.External Links:[Link](https://api.semanticscholar.org/CorpusID:254096040)Cited by:[§1](https://arxiv.org/html/2607.08056#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.08056#S2.SS1.p1.1),[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px1.p1.1)\.
- D\. von Rutte, J\. Fluri, Y\. Ding, A\. Orvieto, B\. Scholkopf, and T\. Hofmann \(2025\)Generalized interpolating discrete diffusion\.ArXivabs/2503\.04482\.External Links:[Link](https://api.semanticscholar.org/CorpusID:276813388)Cited by:[§1](https://arxiv.org/html/2607.08056#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.08056#S2.SS1.p1.1),[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px1.p1.1)\.
- J\. Wang, Y\. Lai, A\. Li, S\. Zhang, J\. Sun, N\. Kang, C\. Wu, Z\. Li, and P\. Luo \(2025a\)FUDOKI: discrete flow\-based unified understanding and generation via kinetic\-optimal velocities\.ArXivabs/2505\.20147\.External Links:[Link](https://api.semanticscholar.org/CorpusID:278911369)Cited by:[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px1.p1.1)\.
- Y\. Wang, L\. Yang, B\. Li, Y\. Tian, K\. Shen, and M\. Wang \(2025b\)Revolutionizing reinforcement learning framework for diffusion large language models\.ArXivabs/2509\.06949\.External Links:[Link](https://api.semanticscholar.org/CorpusID:281203470)Cited by:[§2\.2](https://arxiv.org/html/2607.08056#S2.SS2.SSS0.Px1.p1.5)\.
- Y\. Xin, Q\. Qin, S\. Luo, K\. Zhu, J\. Yan, Y\. Tai, J\. Lei, Y\. Cao, K\. Wang, Y\. Wang, J\. Bai, Q\. Yu, D\. Jiang, Y\. Pu, H\. Chen, L\. Zhuo, J\. He, G\. Luo, T\. Li, M\. Hu, J\. Ye, S\. Ye, B\. Zhang, C\. Xu, W\. Wang, H\. Li, G\. Zhai, T\. Xue, B\. Fu, X\. Liu, Y\. Qiao, and Y\. Liu \(2025\)Lumina\-dimoo: an omni diffusion large language model for multi\-modal generation and understanding\.ArXivabs/2510\.06308\.External Links:[Link](https://api.semanticscholar.org/CorpusID:281891687)Cited by:[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px1.p1.1)\.
- Z\. Xue, J\. Wu, Y\. Gao, F\. Kong, L\. Zhu, M\. Chen, Z\. Liu, W\. Liu, Q\. Guo, W\. Huang, and P\. Luo \(2025\)DanceGRPO: unleashing grpo on visual generation\.ArXivabs/2505\.07818\.Cited by:[§4\.1](https://arxiv.org/html/2607.08056#S4.SS1.SSS0.Px1.p5.1)\.
- L\. Yang, Y\. Tian, B\. Li, X\. Zhang, K\. Shen, Y\. Tong, and M\. Wang \(2025\)MMaDA: multimodal large diffusion language models\.ArXivabs/2505\.15809\.External Links:[Link](https://api.semanticscholar.org/CorpusID:278782105)Cited by:[Appendix A](https://arxiv.org/html/2607.08056#A1.p1.9),[§1](https://arxiv.org/html/2607.08056#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.08056#S2.SS2.SSS0.Px1.p1.5),[§3\.2](https://arxiv.org/html/2607.08056#S3.SS2.SSS0.Px1.p1.5),[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px1.p1.1)\.
- X\. Yue, Y\. Ni, K\. Zhang, T\. Zheng, R\. Liu, G\. Zhang, S\. Stevens, D\. Jiang, W\. Ren, Y\. Sun, C\. Wei, B\. Yu, R\. Yuan, R\. Sun, M\. Yin, B\. Zheng, Z\. Yang, Y\. Liu, W\. Huang, H\. Sun, Y\. Su, and W\. Chen \(2024\)MMMU: a massive multi\-discipline multimodal understanding and reasoning benchmark for expert agi\.InProceedings of CVPR,Cited by:[2nd item](https://arxiv.org/html/2607.08056#S4.I2.i2.p1.1)\.
- S\. Zhao, D\. Gupta, Q\. Zheng, and A\. Grover \(2025\)D1: scaling reasoning in diffusion large language models via reinforcement learning\.ArXivabs/2504\.12216\.External Links:[Link](https://api.semanticscholar.org/CorpusID:277824207)Cited by:[§2\.2](https://arxiv.org/html/2607.08056#S2.SS2.SSS0.Px1.p1.4),[§3\.2](https://arxiv.org/html/2607.08056#S3.SS2.SSS0.Px1.p1.5)\.
- K\. Zheng, Y\. Chen, H\. Mao, M\. Liu, J\. Zhu, and Q\. Zhang \(2024\)Masked diffusion models are secretly time\-agnostic masked models and exploit inaccurate categorical sampling\.ArXivabs/2409\.02908\.External Links:[Link](https://api.semanticscholar.org/CorpusID:272397565)Cited by:[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px1.p1.1)\.
- L\. Zheng, J\. Yuan, L\. Yu, and L\. Kong \(2023\)A reparameterized discrete diffusion model for text generation\.ArXivabs/2302\.05737\.External Links:[Link](https://api.semanticscholar.org/CorpusID:256826865)Cited by:[§1](https://arxiv.org/html/2607.08056#S1.p2.2),[§3](https://arxiv.org/html/2607.08056#S3.p1.2),[§5](https://arxiv.org/html/2607.08056#S5.SS0.SSS0.Px2.p1.1)\.

## Appendix AImplementation Details

We build uponMMaDA\-8B\-MixCoT\[Yanget al\.,[2025](https://arxiv.org/html/2607.08056#bib.bib17)\]as our base model and optimize the control block via a GRPO\-style reinforcement learning objective\. For all experiments, we use the AdamW optimizer with a constant learning rate of5×10−65\\times 10^\{\-6\},β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999,ϵ=10−8\\epsilon=10^\{\-8\}, and no weight decay\. All models are trained for5,0005\{,\}000steps with a per\-GPU batch size of11and44gradient accumulation steps\. The clipping coefficient is set toϵclip=0\.2\\epsilon\_\{\\text\{clip\}\}=0\.2and the KL penalty coefficient toβ=0\\beta=0\.

For text\-to\-image generation \(GenEval\), experiments are conducted on2424NVIDIA A100 GPUs\. We sampleG=4G=4completions per prompt withR=6R=6repeat sampling rounds per training step \(i\.e\.,2424total samples per prompt\), using1616denoising steps and a generation length of1,0241\{,\}024tokens\. Training is conducted inbfloat16precision\.

For multimodal understanding \(VLMEvalKit\), experiments are conducted on88NVIDIA A100 GPUs\. We sampleG=4G=4completions per prompt withR=3R=3repeat sampling rounds \(i\.e\.,1212total samples per prompt\), using6464denoising steps and a generation length of256256tokens, trained infloat16precision\.