DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models

# DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models
Source: [https://arxiv.org/html/2610.12572](https://arxiv.org/html/2610.12572)
###### Abstract

Online reinforcement learning \(RL\) fine\-tunes diffusion models with black\-box rewards, but reverse\-process methods such as Flow\-GRPO are limited in efficiency by poor credit assignment, while forward\-process methods such as DiffusionNFT are efficient yet unstable without carefully tuned exponential\-moving\-average \(EMA\) off\-policy training\. We show that NFT\-style objectives containa hidden forward–reverse consistency forcethat joint positive–negative training cancels only partially, and that EMA stabilizes training throughrestorationrather than off\-policy rollouts\. We introduceDiffusionART\(DiffusionReinforcement withAggregation\-FirstRewardTilting\), which aggregates each prompt’s rewards into a shared reward\-improving direction for reverse\-process states, removing the consistency force by construction, and adds a trust\-region\-masked second update for drift\-adaptive restoration without EMA tuning\. It requires no differentiable reward model, remains effective with fewer samples per prompt, and supports pairwise reward models \(DiffusionART\-Pairwise\)\. Across image and video generation, DiffusionART trains efficiently and stably without per\-task tuning\. In particular, on HunyuanVideo\-13B with equally weighted visual and motion quality rewards, DiffusionART reaches the late\-stage mean reward levels of DiffusionNFT and Flow\-GRPO\-Fast with12\.02×12\.02\\timesand11\.72×11\.72\\timesfewer GPU hours, respectively, while achieving 28\.8% and 53\.0% higher mean training rewards\.

11footnotetext:Corresponding author\.![Refer to caption](https://arxiv.org/html/2610.12572v1/diffusionart_cover_base_model_20261008.png)Figure 1:Efficient learning and improved generations\.Left:Mean of raw VQ and MQ \(equal weights\)\.Right:Selected same\-prompt images and video frames before and after DiffusionART\.## 1Introduction

Visual diffusion and flow\-matching models111We use “diffusion” as a general term\. Our paper targets rectified\-flow with linear data–noise interpolation\[[Lipman et al\., 2022](https://arxiv.org/html/2610.12572#bib.bib2)\], which shares the core principles of diffusion\.are typically trained to fit the data distribution by regressing noise or velocity targets rather than to optimize downstream criteria such as visual quality, motion quality, or human preference, making reward\-guided fine\-tuning central to recent image and video generation tasks\[[Xu et al\., 2026a](https://arxiv.org/html/2610.12572#bib.bib23),[Gao et al\., 2025](https://arxiv.org/html/2610.12572#bib.bib24)\]\. We focus on online diffusion Reinforcement Learning \(RL\), which, unlike differentiable reward\-model optimization such as ReFL\[[Xu et al\., 2023](https://arxiv.org/html/2610.12572#bib.bib22)\]or offline preference learning such as Diffusion\-DPO\[[Wallace et al\., 2023](https://arxiv.org/html/2610.12572#bib.bib5)\], evaluates clean samples from the evolving generator and can use black\-box rewards\. Because these rewards arrive only after a complete output is generated, online diffusion RL faces a credit\-assignment problem: costly output\-level feedback must be translated into effective updates across many denoising states\[[Liu et al\., 2025a](https://arxiv.org/html/2610.12572#bib.bib3),[Zheng et al\., 2025](https://arxiv.org/html/2610.12572#bib.bib6)\]\.

Within this paradigm,*reverse\-process RL*\[[Black et al\., 2023](https://arxiv.org/html/2610.12572#bib.bib1)\], such as Flow\-GRPO and its variants\[[Liu et al\., 2025a](https://arxiv.org/html/2610.12572#bib.bib3),[Xue et al\., 2025](https://arxiv.org/html/2610.12572#bib.bib13),[He et al\., 2025](https://arxiv.org/html/2610.12572#bib.bib25),[Wang et al\., 2026](https://arxiv.org/html/2610.12572#bib.bib26)\], optimizes sampled denoising transitions via group\-based endpoint reward feedback\. It trains on states actually visited by the generator but requires ODE\-to\-SDE conversion for tractable likelihoods, and its policy gradient updates each selected transition along its own transition\-level sampling noise, weighted by a trajectory\-level advantage that this transition only partly determines, resulting in poor credit assignment and limiting training efficiency\.*Forward\-process RL*, such as DiffusionNFT\[[Zheng et al\., 2025](https://arxiv.org/html/2610.12572#bib.bib6)\], instead applies forward noising to the generated clean samples and regresses velocities with reward\-weighted positive and negative objectives\. It needs no transition likelihoods and is highly efficient, but it is unstable without two stabilizers: joint positive–negative training and an exponential moving average \(EMA\) rollout policy whose lag must be tuned for each task\[[Zheng et al\., 2025](https://arxiv.org/html/2610.12572#bib.bib6)\]\. Neither family is thus both efficient and stable without per\-task tuning\. In this paper, we focus on studying the instability of forward\-process RL and developing a scalable algorithm which is both stable and efficient\.

We first examine NFT\-style training, starting from a puzzle: its positive and negative branches share the same ideal reward\-improving direction, so either alone should suffice, yet each alone collapses \(Figure[2](https://arxiv.org/html/2610.12572#S3.F2)\(a\)\)\. We find that NFT’s targets, built from forward\-noised endpoints, implicitly assume that the model’s sampling velocity equals the ideal forward velocity induced by its own endpoints\. The two differ in practice, and the gap introduces a hidden non\-reward\-improving*forward–reverse consistency force*: the positive branch pulls each state’s velocity toward the ideal forward velocity and the negative branch pushes it away\. Joint training can cancel this force but only partially, since reward centering balances rewards across samples rather than at each state\. The residual can destabilize training and slow the reward optimization\. We further show that EMA stabilizes NFT through the*restoration force*of its lagged reference, not through off\-policy rollouts, but the EMA trick must be tuned per task\.

Driven by these insights, we introduceDiffusionART\(DiffusionReinforcement withAggregation\-FirstRewardTilting\)\. It borrows supervised velocity regression from forward\-process training but applies updates to states and actions during reverse\-process rollouts\. It first aggregates each prompt’s samples and rewards into a shared reward\-improving directionG⁡\(c\)G\(c\), then regresses each visited state’s action toward its guided target \(Section[4\.1](https://arxiv.org/html/2610.12572#S4.SS1)\)\. As this target never involves the forward velocity, the consistency force is removed by construction rather than by cross\-sample cancellation, and each state receives an explicit reward\-improving direction instead of a scalar advantage passed through likelihood ratios\. A second update with a trust\-region mask mechanism on each rollout adds a restoration force proportional to each state’s drift, replacing EMA with no schedule to tune \(Section[4\.2](https://arxiv.org/html/2610.12572#S4.SS2)\)\. We summarize our main contributions as follows:

Main Contributions 1\. Diagnosing the instability of NFT\-style training\.The NFT objective implicitly contains a harmful forward–reverse consistency force that joint training cancels only partially \(Section[3\.1](https://arxiv.org/html/2610.12572#S3.SS1)\)\.2\. Identifying restoration as the key stabilizer\.EMA stabilizes NFT through restoration rather than off\-policy data, but requires per\-task tuning \(Section[3\.2](https://arxiv.org/html/2610.12572#S3.SS2)\)\.3\. Aggregation\-first reward tilting with adaptive restoration\.DiffusionART removes the consistency force by construction, eliminating the need for EMA with drift\-adaptive restoration \(Section[4](https://arxiv.org/html/2610.12572#S4)\)\.4\. Efficient and stable training\.DiffusionART matches the strongest baseline’s efficiency on SD3\.5\-M without EMA or per\-task tuning\. Under VQ:MQ=1:1=1\{:\}1weights on HunyuanVideo\-13B, it reaches the late\-stage mean VQ and MQ rewards of DiffusionNFT and Flow\-GRPO\-Fast with12\.02×12\.02\\timesand11\.72×11\.72\\timesfewer GPU hours, while achieving 28\.8% and 53\.0% higher mean training rewards, respectively\. For image\-to\-video generation, it reaches MQ 6 with4\.8×4\.8\\timesfewer GPU hours than Flow\-GRPO\-Fast \(Section[5](https://arxiv.org/html/2610.12572#S5)\)\.

## 2Preliminaries

Diffusion RL\[[Xu et al\., 2026b](https://arxiv.org/html/2610.12572#bib.bib7)\]includes reverse\-process policy optimization, such as Flow\-GRPO\[[Liu et al\., 2025a](https://arxiv.org/html/2610.12572#bib.bib3)\], and forward\-process optimization, such as DiffusionNFT\[[Zheng et al\., 2025](https://arxiv.org/html/2610.12572#bib.bib6),[Zhu et al\., 2026](https://arxiv.org/html/2610.12572#bib.bib9),[Xue et al\., 2026](https://arxiv.org/html/2610.12572#bib.bib28)\]\. We focus on DiffusionNFT and defer Flow\-GRPO to Appendix[C\.1](https://arxiv.org/html/2610.12572#A3.SS1)\. At each iteration, for each promptcc, DiffusionNFT samplesKKclean images or videosx0x\_\{0\}from a frozen rollout policypold​\(x0∣c\)p\_\{\\mathrm\{old\}\}\(x\_\{0\}\\mid c\); we call these generated samples*endpoints*\. DiffusionNFT maps each raw rewardRi:=R⁡\(x0\(i\),c\)R\_\{i\}:=R\(x\_\{0\}^\{\(i\)\},c\)to a*goodness*weightri:=r⁡\(x0\(i\),c\)=12​\[1\+clip⁡\(\(Ri−R¯c\)/Zc,−1,1\)\]∈\[0,1\]r\_\{i\}:=r\(x\_\{0\}^\{\(i\)\},c\)=\\tfrac\{1\}\{2\}\[1\+\\operatorname\{clip\}\(\(R\_\{i\}\-\\bar\{R\}\_\{c\}\)/Z\_\{c\},\-1,1\)\]\\in\[0,1\], whereR¯c=K−1​∑iRi\\bar\{R\}\_\{c\}=K^\{\-1\}\\sum\_\{i\}R\_\{i\}is the prompt’s mean reward andZc\>0Z\_\{c\}\>0controls the scale; without clipping,∑i\(2​ri−1\)=0\\sum\_\{i\}\(2r\_\{i\}\-1\)=0\. Weightingpoldp\_\{\\mathrm\{old\}\}byrrand1−r1\-rinduces the positive and negative endpoint distributionsp\+​\(x0∣c\)∝r⁡\(x0,c\)​pold​\(x0∣c\)p\_\{\+\}\(x\_\{0\}\\mid c\)\\propto r\(x\_\{0\},c\)p\_\{\\mathrm\{old\}\}\(x\_\{0\}\\mid c\)andp−​\(x0∣c\)∝\[1−r⁡\(x0,c\)\]​pold​\(x0∣c\)p\_\{\-\}\(x\_\{0\}\\mid c\)\\propto\[1\-r\(x\_\{0\},c\)\]p\_\{\\mathrm\{old\}\}\(x\_\{0\}\\mid c\)\. DiffusionNFT is motivated by the result that the ideal velocities induced by forward noising the positive, original, and negative endpoint distributions are collinear\. Moving toward the positive field and away from the negative field therefore follows the same ideal improvement direction\.

We parameterize generation fromt=1t=1\(noise\) tot=0t=0\(clean samples\), so velocity points along the denoising direction\. For a forward statext=αt​x0\+σt​ϵx\_\{t\}=\\alpha\_\{t\}x\_\{0\}\+\\sigma\_\{t\}\\epsilon, letut=−α˙t​x0−σ˙t​ϵu\_\{t\}=\-\\dot\{\\alpha\}\_\{t\}x\_\{0\}\-\\dot\{\\sigma\}\_\{t\}\\epsilondenote its sample velocity target, where dots denote derivatives with respect tott; standard rectified\-flow interpolation\[[Lipman et al\., 2022](https://arxiv.org/html/2610.12572#bib.bib2),[Liu et al\., 2022](https://arxiv.org/html/2610.12572#bib.bib29)\]givesut=x0−ϵu\_\{t\}=x\_\{0\}\-\\epsilon\. Letvold:=vθold​\(xt,t,c\)v\_\{\\mathrm\{old\}\}:=v\_\{\\theta\_\{\\mathrm\{old\}\}\}\(x\_\{t\},t,c\)andvθ​\(xt,t,c\)v\_\{\\theta\}\(x\_\{t\},t,c\)be the velocities predicted by the rollout policypoldp\_\{\\mathrm\{old\}\}\(frozen within an iteration\) and the trainable policy, respectively, and let𝒟old\\mathcal\{D\}\_\{\\mathrm\{old\}\}be the training distribution of\(c,x0,r,t,ϵ\)\(c,x\_\{0\},r,t,\\epsilon\)withx0∼pold\(⋅∣c\)x\_\{0\}\\sim p\_\{\\mathrm\{old\}\}\(\\cdot\\mid c\)andϵ∼𝒩⁡\(0,I\)\\epsilon\\sim\\mathcal\{N\}\(0,I\)\. Forβ\>0\\beta\>0, DiffusionNFT constructsvθ,±=vold±β⁡\(vθ−vold\)v\_\{\\theta,\\pm\}=v\_\{\\mathrm\{old\}\}\\pm\\beta\(v\_\{\\theta\}\-v\_\{\\mathrm\{old\}\}\)and minimizesℒNFT=𝔼𝒟old​\[r​∥vθ,\+−ut∥22\+\(1−r\)​∥vθ,−−ut∥22\]\\mathcal\{L\}\_\{\\mathrm\{NFT\}\}=\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{\\mathrm\{old\}\}\}\[r\\lVert v\_\{\\theta,\+\}\-u\_\{t\}\\rVert\_\{2\}^\{2\}\+\(1\-r\)\\lVert v\_\{\\theta,\-\}\-u\_\{t\}\\rVert\_\{2\}^\{2\}\]\. Equivalently, leta=\(2​r−1\)/βa=\(2r\-1\)/\\betaand define the frozen targetvtgt=a​ut\+\(1−a\)​voldv\_\{\\mathrm\{tgt\}\}=au\_\{t\}\+\(1\-a\)v\_\{\\mathrm\{old\}\}\. Then

ℒNFT=β2​𝔼𝒟old​\[∥vθ−vtgt∥22\]\+const,\\mathcal\{L\}\_\{\\mathrm\{NFT\}\}=\\beta^\{2\}\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{\\mathrm\{old\}\}\}\\\!\\left\[\\lVert v\_\{\\theta\}\-v\_\{\\mathrm\{tgt\}\}\\rVert\_\{2\}^\{2\}\\right\]\+\\mathrm\{const\},\(1\)whereconst\\mathrm\{const\}is independent ofθ\\theta\. Notably, NFT retains avoldv\_\{\\mathrm\{old\}\}anchor invtgtv\_\{\\mathrm\{tgt\}\}\. At the on\-policy initializationvθ=voldv\_\{\\theta\}=v\_\{\\mathrm\{old\}\}, each sample’s output\-space descent direction is proportional to\(2​r−1\)​\(ut−vold\)\(2r\-1\)\(u\_\{t\}\-v\_\{\\mathrm\{old\}\}\), so high\- and low\-reward samples drive the same regression residual in opposite directions \(see Appendix[B\.2\.1](https://arxiv.org/html/2610.12572#A2.SS2.SSS1)for both derivations\)\. After optimizing the rollout batch, DiffusionNFT updates the rollout policy by exponential moving average \(EMA\):θold←η​θold\+\(1−η\)​θ\\theta\_\{\\mathrm\{old\}\}\\leftarrow\\eta\\theta\_\{\\mathrm\{old\}\}\+\(1\-\\eta\)\\theta\.

## 3Understanding NFT\-style Diffusion Reinforcement Learning

DiffusionNFT is highly efficient, but its stability relies on joint positive–negative training and an off\-policy EMA rollout policy\[[Zheng et al\., 2025](https://arxiv.org/html/2610.12572#bib.bib6)\]\. We examine why both are needed and where they fall short: joint training only partially cancels a hidden forward–reverse consistency force, while EMA provides restoration whose strength requires task\-specific tuning\. These limitations motivate the two components of DiffusionART in Section[4](https://arxiv.org/html/2610.12572#S4)\.

### 3\.1Joint Positive–Negative Training: Partial Cancellation of a Hidden Consistency Force

Figure 2:\(a\) Ablation results for positive and negative objectives\. \(b\) The positive and negative objectives partially cancel the consistency force\.We first remove EMA to isolate joint training, sovθ=voldv\_\{\\theta\}=v\_\{\\mathrm\{old\}\}at every on\-policy update\. Consistent with[Zheng et al\. \[2025\]](https://arxiv.org/html/2610.12572#bib.bib6), either branch alone collapses, whereas joint training learns effectively \(Figure[2](https://arxiv.org/html/2610.12572#S3.F2)\(a\)\)\. This is puzzling: the ideal positive and negative guidance directions are collinear, suggesting that either branch alone should suffice\.

To make this precise, forb∈\{\+,old,−\}b\\in\\\{\+,\\mathrm\{old\},\-\\\}, letPbP\_\{b\}denote the endpoint distributionpb​\(x0∣c\)p\_\{b\}\(x\_\{0\}\\mid c\), and letQbQ\_\{b\}denote the jointforwarddistribution of\(x0,ϵ,xt,ut\)\(x\_\{0\},\\epsilon,x\_\{t\},u\_\{t\}\)obtained by drawingx0∼Pbx\_\{0\}\\sim P\_\{b\}andϵ∼𝒩⁡\(0,I\)\\epsilon\\sim\\mathcal\{N\}\(0,I\)independently and formingxtx\_\{t\}andut=x0−ϵu\_\{t\}=x\_\{0\}\-\\epsilonby forward noising\. Withst=\(xt,t,c\)s\_\{t\}=\(x\_\{t\},t,c\), the*ideal forward velocity*induced byPbP\_\{b\}isvfwdb​\(st\):=𝔼Qb​\[ut∣st\]v\_\{\\mathrm\{fwd\}\}^\{b\}\(s\_\{t\}\):=\\mathbb\{E\}\_\{Q\_\{b\}\}\[u\_\{t\}\\mid s\_\{t\}\]\. We writevfwdoldv\_\{\\mathrm\{fwd\}\}^\{\\mathrm\{old\}\}forb=oldb=\\mathrm\{old\}, andvrvsold:=voldv\_\{\\mathrm\{rvs\}\}^\{\\mathrm\{old\}\}:=v\_\{\\mathrm\{old\}\}for the field that the reverse sampler actually uses to generate the endpoint distribution\.

###### Lemma 1\(Ideal NFT guidance is conditional reward covariance\)\.

At a fixed statests\_\{t\}, letQoldQ\_\{\\mathrm\{old\}\}denoteQbQ\_\{b\}withb=oldb=\\mathrm\{old\}, and letq:=𝔼Qold​\[r⁡\(x0,c\)∣st\]q:=\\mathbb\{E\}\_\{Q\_\{\\mathrm\{old\}\}\}\[r\(x\_\{0\},c\)\\mid s\_\{t\}\]andg:=CovQold\[r\(x0,c\),ut∣st\]g:=\\operatorname\{Cov\}\_\{Q\_\{\\mathrm\{old\}\}\}\[r\(x\_\{0\},c\),\\allowbreak u\_\{t\}\\mid s\_\{t\}\]\. If the tilted distributions and moments exist and0<q<10<q<1, then

q⁡\(vfwd\+−vfwdold\)\\displaystyle q\\bigl\(v\_\{\\mathrm\{fwd\}\}^\{\+\}\-v\_\{\\mathrm\{fwd\}\}^\{\\mathrm\{old\}\}\\bigr\)=\(1−q\)​\(vfwdold−vfwd−\)=g,\\displaystyle=\(1\-q\)\\bigl\(v\_\{\\mathrm\{fwd\}\}^\{\\mathrm\{old\}\}\-v\_\{\\mathrm\{fwd\}\}^\{\-\}\\bigr\)=g,\(2\)where all velocities are evaluated atsts\_\{t\}\. Hence the positive and negative branches point along the same reward\-tilt directiongg\.

Appendix[B\.1](https://arxiv.org/html/2610.12572#A2.SS1)provides the proof\. The ideal guidance of the two branches thus differs only in step size: under*forward–reverse compatibility*, i\.e\., if the reverse sampler satisfiedvrvsold=vfwdoldv\_\{\\mathrm\{rvs\}\}^\{\\mathrm\{old\}\}=v\_\{\\mathrm\{fwd\}\}^\{\\mathrm\{old\}\}, either branch alone would suffice\. In practice, however, the reverse sampling field generally differs from the ideal forward velocity induced by its own endpoints\. We denote this forward–reverse gap bye:=vfwdold​\(st\)−vrvsold​\(st\)e:=v\_\{\\mathrm\{fwd\}\}^\{\\mathrm\{old\}\}\(s\_\{t\}\)\-v\_\{\\mathrm\{rvs\}\}^\{\\mathrm\{old\}\}\(s\_\{t\}\)\. Proposition[1](https://arxiv.org/html/2610.12572#Thmproposition1)shows how it enters the NFT update\.

###### Proposition 1\(Practical forward\-process RL contains a forward–reverse consistency force\)\.

LetℒNFT​\(st\)\\mathcal\{L\}\_\{\\mathrm\{NFT\}\}\(s\_\{t\}\)denote Equation[1](https://arxiv.org/html/2610.12572#S2.E1)conditioned onsts\_\{t\},qqandggfollow the definitions of Lemma[1](https://arxiv.org/html/2610.12572#Thmlemma1)\. At the pure on\-policy update pointvθ=vrvsoldv\_\{\\theta\}=v\_\{\\mathrm\{rvs\}\}^\{\\mathrm\{old\}\}and forβ\>0\\beta\>0, its normalized output\-space force is

ℱNFT​\(st\):=−12​β​∇vθ​\(st\)ℒNFT​\(st\)\|vθ=vrvsold=2​g\+\(2​q−1\)​e\.\\mathcal\{F\}\_\{\\mathrm\{NFT\}\}\(s\_\{t\}\):=\-\\frac\{1\}\{2\\beta\}\\left\.\\nabla\_\{v\_\{\\theta\}\(s\_\{t\}\)\}\\mathcal\{L\}\_\{\\mathrm\{NFT\}\}\(s\_\{t\}\)\\right\|\_\{v\_\{\\theta\}=v\_\{\\mathrm\{rvs\}\}^\{\\mathrm\{old\}\}\}=2g\+\(2q\-1\)e\.\(3\)

Why both branches are needed\.Proposition[1](https://arxiv.org/html/2610.12572#Thmproposition1)decomposes the NFT force into reward\-improving guidance and a self\-consistency force weighted by the statewise reward imbalance2​q−12q\-1\. The individual branch forces areℱ\+=g\+q​e\\mathcal\{F\}\_\{\+\}=g\+qeandℱ−=g−\(1−q\)​e\\mathcal\{F\}\_\{\-\}=g\-\(1\-q\)e, which sum toℱNFT\\mathcal\{F\}\_\{\\mathrm\{NFT\}\}\(proof in Appendix[B\.2](https://arxiv.org/html/2610.12572#A2.SS2)\)\. Each branch alone thus carries an uncancelled consistency force: the positive branch pulls the velocity towardvfwdoldv\_\{\\mathrm\{fwd\}\}^\{\\mathrm\{old\}\}with weightqq, whereas the negative branch pushes it away with weight1−q1\-q\. Joint training therefore does more than double the guidance: the branches reinforce each other alongggbut oppose each other alongee, leaving\(2​q−1\)​e\(2q\-1\)e\. Without clipping, prompt\-level reward centering makesqqaverage to1/21/2across states, allowing partial cancellation after gradient aggregation \(Figure[2](https://arxiv.org/html/2610.12572#S3.F2)\(b\)\)\.

Why joint training is not enough\.For nonzero consistency gaps, cancellation is exact only ifq=1/2q=1/2at every state, which prompt\-level normalization does not guarantee:∑i\(2​ri−1\)≈0\\sum\_\{i\}\(2r\_\{i\}\-1\)\\approx 0neither balances each state nor implies∑i\(2​ri−1\)​ei≈0\\sum\_\{i\}\(2r\_\{i\}\-1\)e\_\{i\}\\approx 0, whereeie\_\{i\}denotes the one\-sample Monte Carlo estimate ofee\. States withq\>1/2q\>1/2are systematically pulled towardvfwdoldv\_\{\\mathrm\{fwd\}\}^\{\\mathrm\{old\}\}, while those withq<1/2q<1/2are pushed away\. Att=1t=1, the state contains no endpoint information, so without clippingq=1/2q=1/2and the residual vanishes\. As noise decreases, states reveal more endpoint quality, makingqqmore extreme and increasing the weight\|2​q−1\|\\lvert 2q\-1\\rverton the gap\. The residual dilutes reward\-improving guidance in the aggregate gradient and requires an additional stabilizer \(Section[3\.2](https://arxiv.org/html/2610.12572#S3.SS2)\)\.

Insight 1: Joint training cancels the consistency force only partially In NFT\-style forward\-process training, either branch alone carries an uncancelled forward–reverse consistency force that can destabilize training; joint training only partially cancels it across samples, not at each state\. A better design should remove this force statewise rather than rely on cross\-sample cancellation \(Section[4\.1](https://arxiv.org/html/2610.12572#S4.SS1)\)\.

### 3\.2Off\-Policy EMA: Stability through Restoration, at the Cost of Task\-Specific Tuning

![Refer to caption](https://arxiv.org/html/2610.12572v1/fig3_review_20261008_v2.png)Figure 3:\(a\) On\-policy DiffusionNFT without EMA \(β=1\\beta=1\): increasing number of micro\-batches per rollout from 1 to 2 results in15\.2×15\.2\\timesslower training\. The dashed curve is the one\-update run’s gradient norm, spiking as the reward collapses\. \(b\) EMA\-policy rollouts without the old\-policy anchor still collapse\. \(c\) At the second update, averaged over rollouts 0–17, restoration has a9\.4×9\.4\\timessmaller output\-force RMS but a4\.9×4\.9\\timeslarger parameter\-gradient norm than guidance\. \(d\) The second MSE update decomposes into guidance and restoration\.Even with joint training, pure on\-policy NFT collapses after a sharp rise in its pre\-clipping parameter\-gradient norm \(Figure[3](https://arxiv.org/html/2610.12572#S3.F3)\(a\)\)\. EMA may stabilize training through \(1\) a slower\-moving rollout distribution or \(2\) the lagged referencevoldv\_\{\\mathrm\{old\}\}retained in the loss \(in Eq\.[1](https://arxiv.org/html/2610.12572#S2.E1), thevtgtv\_\{\\mathrm\{tgt\}\}containsvoldv\_\{\\mathrm\{old\}\}, rather thanvθv\_\{\\theta\}\)\. We show that the latter, namely restoration toward this reference, is the key stabilizing mechanism\. Letd=vθ−voldd=v\_\{\\theta\}\-v\_\{\\mathrm\{old\}\}denote the current policy’s drift from the rollout old policy andh=ut−voldh=u\_\{t\}\-v\_\{\\mathrm\{old\}\}the sample training residual\.

###### Proposition 2\(EMA brings an explicit restoring term to DiffusionNFT\)\.

ℒNFT=𝔼𝒟old​\[−2​β​\(2​r−1\)​⟨d,h⟩\+β2​∥d∥22\]\+const,\\mathcal\{L\}\_\{\\mathrm\{NFT\}\}=\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{\\mathrm\{old\}\}\}\\\!\\left\[\-2\\beta\(2r\-1\)\\langle d,h\\rangle\+\\beta^\{2\}\\lVert d\\rVert\_\{2\}^\{2\}\\right\]\+\\mathrm\{const\},\(4\)

whereconst\\mathrm\{const\}is independent of the trainable parameters\.

Why EMA stabilizes\.Proposition[2](https://arxiv.org/html/2610.12572#Thmproposition2)separates the NFT loss into a linear term carrying both reward guidance and the consistency force, and a quadratic term restoringvθv\_\{\\theta\}towardvoldv\_\{\\mathrm\{old\}\}\(proof in Appendix[B\.2](https://arxiv.org/html/2610.12572#A2.SS2)\)\. Atd=0d=0, the restoration gradient vanishes\. Pure on\-policy NFT therefore receives no restoration, leaving the residual consistency force unopposed\. EMA maintains a nonzero drift from the lagged reference, allowing restoration to counteract this residual\.

We test these explanations with two ablations\. First, retaining EMA\-policy rollouts but removing the old\-policy anchor from the gradient \(i\.e\. replacingvoldv\_\{\\mathrm\{old\}\}byvθv\_\{\\theta\}invtgtv\_\{\\mathrm\{tgt\}\}of Eq[1](https://arxiv.org/html/2610.12572#S2.E1)\) still causes collapse \(Figure[3](https://arxiv.org/html/2610.12572#S3.F3)\(b\)\): off\-policy data alone does not stabilize NFT\. Second, without EMA, we split each rollout batch into two disjoint minibatches and update once on each, using the original objective withβ=1\\beta=1\. The first update starts atvθ\(0\)=voldv\_\{\\theta\}^\{\(0\)\}=v\_\{\\mathrm\{old\}\}, where restoration is zero\. After the model moves tovθ\(1\)v\_\{\\theta\}^\{\(1\)\}, the second update combines guidance with restoration towardvθ\(0\)v\_\{\\theta\}^\{\(0\)\}\(Figure[3](https://arxiv.org/html/2610.12572#S3.F3)\(d\)\)\. We surprisingly found that this two\-update run avoids collapse over the observed training window without EMA, but significantly slows training by 15\.2x \(Figure[3](https://arxiv.org/html/2610.12572#S3.F3)\(a\)\), supporting restoration as the key stabilizing mechanism at the cost of efficiency in NFT\-style forward\-process training\.

Why EMA is not enough\.EMA controls restoration indirectly through the reference policy’s lag: too little restoration risks collapse, while too much slows reward improvement\. Its schedule therefore requires task\-specific tuning\. DiffusionNFT’s results illustrate this sensitivity\[[Zheng et al\., 2025](https://arxiv.org/html/2610.12572#bib.bib6)\]: fully on\-policy updates collapse, nearly offline updates converge impractically slowly, and the default scheduleη=min⁡\(0\.001​i,0\.5\)\\eta=\\min\(0\.001i,0\.5\), whereiiis the iteration, is unstable on OCR, which requires a maximum retention of0\.9990\.999\. In our FLUX experiments, transferring NFT’s GenEval\-tuned EMA andβ\\betasettings also leads to unstable training and can cause collapse \(Figure[6](https://arxiv.org/html/2610.12572#S5.F6)\(a\)\)\. Such tuning requirement limits scalability\.

Two NFT updates per rollout provide restoration without EMA \(Figure[3](https://arxiv.org/html/2610.12572#S3.F3)\(d\)\), but significantly slow training \(Figure[3](https://arxiv.org/html/2610.12572#S3.F3)\(a\)\)\. Counterintuitively, averaged over rollouts 0–17, restoration has a9\.4×9\.4\\timessmaller output\-force RMS than guidance, yet its aggregated parameter\-gradient norm is4\.9×4\.9\\timeslarger \(Figure[3](https://arxiv.org/html/2610.12572#S3.F3)\(c\)\)\. This suggests that heterogeneous per\-sample guidance largely cancels across states, while restoration survives aggregation and dominates the update, slowing reward optimization\. We leave a detailed analysis to our future work\.

Insight 2: Restoration is crucial for stabilizing NFT\-style forward\-process training Restoration stabilizes NFT\-style training, but EMA requires task\-specific tuning, while two updates per rollout avoid EMA tuning at the cost of slower reward improvement\. A better design should adapt restoration to each state’s drift without task\-specific tuning or overwhelming reward guidance \(Section[4\.2](https://arxiv.org/html/2610.12572#S4.SS2)\)\.

Figure 4:Aggregation\-first guidance\.Left:NFT’s positive and negative branches introduce opposing consistency forces that may not fully cancel \(Section[3\.1](https://arxiv.org/html/2610.12572#S3.SS1)\)\. DiffusionART avoids these forces by applyingG⁡\(c\)G\(c\)directly at the reverse velocity\.Right:We aggregate each prompt’s endpoints and rewards into normalized guidanceG⁡\(c\)G\(c\)\(Equation[5](https://arxiv.org/html/2610.12572#S4.E5)\) and apply it to states visited along this rollout\.

## 4DiffusionART: Aggregation\-first Reward Tilting with Adaptive Restoration

DiffusionART \(Algorithm[1](https://arxiv.org/html/2610.12572#alg1)\) combines the advantages of forward\- and reverse\-process RL, while avoiding their limitations\. It applies aggregation\-first reward guidance to reverse\-rollout states \(section[4\.1](https://arxiv.org/html/2610.12572#S4.SS1)\) and uses a second MSE update for drift\-adaptive restoration \(section[4\.2](https://arxiv.org/html/2610.12572#S4.SS2)\), addressing the two failure modes identified in Section 3\. A trust\-region mechanism for an ODE solver further avoids excessive policy updates \(section[4\.3](https://arxiv.org/html/2610.12572#S4.SS3)\)\.

### 4\.1Aggregation\-First Guidance

The consistency force enters NFT because its regression target is built from the forward velocityvfwdoldv\_\{\\mathrm\{fwd\}\}^\{\\mathrm\{old\}\}, which differs from the reverse field byee\(Proposition[1](https://arxiv.org/html/2610.12572#Thmproposition1)\)\. We instead aggregate each prompt’s endpoints and rewards alone into guidance for reverse\-rollout states\. At each iteration, we exponentially tilt the current rollout distribution, rather than a fixed pretrained distribution\[[Potaptchik et al\., 2025](https://arxiv.org/html/2610.12572#bib.bib4),[Zhu et al\., 2026](https://arxiv.org/html/2610.12572#bib.bib9)\]:pτ​\(x0∣c\)∝pold​\(x0∣c\)​exp⁡\(τ​R​\(x0,c\)\)p\_\{\\tau\}\(x\_\{0\}\\mid c\)\\propto p\_\{\\mathrm\{old\}\}\(x\_\{0\}\\mid c\)\\exp\\bigl\(\\tau R\(x\_\{0\},c\)\\bigr\)\. Hereτ≥0\\tau\\geq 0controls the preference for higher\-reward endpoints, andτ=0\\tau=0recoverspoldp\_\{\\mathrm\{old\}\}\. The first\-order change in the endpoint mean is a covariance:

###### Lemma 2\(Reward tilting moves the endpoint mean along the reward–endpoint covariance\)\.

Under standard integrability conditions, the derivative of the tilted endpoint mean atτ=0\\tau=0equals the reward–endpoint covariance:∂∂τ​𝔼pτ​\[x0∣c\]\|τ=0=Covpold⁡\(R,x0∣c\)\\left\.\\frac\{\\partial\}\{\\partial\\tau\}\\mathbb\{E\}\_\{p\_\{\\tau\}\}\[x\_\{0\}\\mid c\]\\right\|\_\{\\tau=0\}=\\operatorname\{Cov\}\_\{p\_\{\\mathrm\{old\}\}\}\(R,x\_\{0\}\\mid c\)\.

Lemma[2](https://arxiv.org/html/2610.12572#Thmlemma2), proved in Appendix[B\.3](https://arxiv.org/html/2610.12572#A2.SS3), shows that up\-weighting higher\-reward endpoints shifts their mean alongCovpold⁡\(R,x0∣c\)\\operatorname\{Cov\}\_\{p\_\{\\mathrm\{old\}\}\}\(R,x\_\{0\}\\mid c\)\. Under rectified flow,ut=x0−ϵu\_\{t\}=x\_\{0\}\-\\epsilonwithϵ\\epsilonindependent ofx0x\_\{0\}, soCovQold⁡\(R,ut∣c\)=Covpold⁡\(R,x0∣c\)\\operatorname\{Cov\}\_\{Q\_\{\\mathrm\{old\}\}\}\(R,u\_\{t\}\\mid c\)=\\operatorname\{Cov\}\_\{p\_\{\\mathrm\{old\}\}\}\(R,x\_\{0\}\\mid c\)at every noise level\. This direction is the prompt\-level counterpart of ideal NFT guidancegg\(Lemma[1](https://arxiv.org/html/2610.12572#Thmlemma1)\): without clipping,rris affine inRR, and att=1t=1, when the forward state reveals no endpoint information,ggequals this covariance up to a positive factor\. We estimate the covariance from each prompt’sKKrollout endpoints and rewards, then normalize it to obtainG⁡\(c\)G\(c\):

Cov¯​\(R,x0∣c\)=1K​∑i=1K\(Ri−R¯c\)​\(x0\(i\)−x¯0,c\),G⁡\(c\)=Cov¯RMS⁡\(Cov¯\)\.\\boxed\{\\overline\{\\operatorname\{Cov\}\}\(R,x\_\{0\}\\mid c\)=\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{K\}\(R\_\{i\}\-\\bar\{R\}\_\{c\}\)\(x\_\{0\}^\{\(i\)\}\-\\bar\{x\}\_\{0,c\}\),\\qquad G\(c\)=\\frac\{\\overline\{\\operatorname\{Cov\}\}\}\{\\operatorname\{RMS\}\(\\overline\{\\operatorname\{Cov\}\}\)\}\}\.\(5\)HereR¯c\\bar\{R\}\_\{c\}andx¯0,c\\bar\{x\}\_\{0,c\}are the group means\. RMS normalization fixes the guidance magnitude across prompts and reward models, removing differences in reward scale and spread without a guidance\-strength coefficient\. We applyG⁡\(c\)G\(c\)to every rollout statests\_\{t\}of promptccand freeze the targetvtgtART​\(st\)=vθ\(0\)​\(st\)\+G⁡\(c\)v\_\{\\mathrm\{tgt\}\}^\{\\mathrm\{ART\}\}\(s\_\{t\}\)=v\_\{\\theta\}^\{\(0\)\}\(s\_\{t\}\)\+G\(c\), wherevθ\(0\)=voldv\_\{\\theta\}^\{\(0\)\}=v\_\{\\mathrm\{old\}\}is the prediction cached during rollout\. Atvθ=vθ\(0\)v\_\{\\theta\}=v\_\{\\theta\}^\{\(0\)\}, every state receives an output\-space force proportional toG⁡\(c\)G\(c\), with no NFT consistency term \(Equations[6](https://arxiv.org/html/2610.12572#S4.E6)and[3](https://arxiv.org/html/2610.12572#S3.E3); Insight[3\.1](https://arxiv.org/html/2610.12572#S3.SS1)\)\. Figure[4](https://arxiv.org/html/2610.12572#S3.F4)summarizes the construction\.

### 4\.2Two\-Update Restoration: Drift\-Adaptive Stability without EMA Tuning

Instead of tuning an EMA restoration schedule \(Section[3\.2](https://arxiv.org/html/2610.12572#S3.SS2)\), we split each rollout batch into two disjoint minibatches and apply one MSE update to each\. Letvθ\(0\)v\_\{\\theta\}^\{\(0\)\}andvθ\(1\)v\_\{\\theta\}^\{\(1\)\}denote the velocity fields before and after the first update\. Both updates use the frozen targetvtgtART=vθ\(0\)\+G⁡\(c\)v\_\{\\mathrm\{tgt\}\}^\{\\mathrm\{ART\}\}=v\_\{\\theta\}^\{\(0\)\}\+G\(c\)\. For each statests\_\{t\}, the loss isℒθ​\(st\)=w⁡\(st\)​MSE⁡\(vθ​\(st\),vtgtART​\(st\)\)\\mathcal\{L\}\_\{\\theta\}\(s\_\{t\}\)=w\(s\_\{t\}\)\\operatorname\{MSE\}\(v\_\{\\theta\}\(s\_\{t\}\),v\_\{\\mathrm\{tgt\}\}^\{\\mathrm\{ART\}\}\(s\_\{t\}\)\), whereMSE⁡\(a,b\)=D−1​∥a−b∥22\\operatorname\{MSE\}\(a,b\)=D^\{\-1\}\\lVert a\-b\\rVert\_\{2\}^\{2\},DDis the number of output coordinates, andwwcombines timestep weighting and masking\. Letd⁡\(st\)=vθ​\(st\)−vθ\(0\)​\(st\)d\(s\_\{t\}\)=v\_\{\\theta\}\(s\_\{t\}\)\-v\_\{\\theta\}^\{\(0\)\}\(s\_\{t\}\)denote the current displacement\. The loss and output gradient before each update are:

L^\(0\)=wMSE\(v\_θ^\(0\),v\_tgt^ART\) =w D∥G\(c\)∥\_2^2L^\(1\)=wMSE\(v\_θ^\(1\),v\_tgt^ART\) =w D∥G\(c\)\-d\(s\_t\)∥\_2^2\-∇\_v\_θL\_θ\|\_v\_θ=v\_θ^\(0\) =2w DG\(c\)\-∇\_v\_θL\_θ\|\_v\_θ=v\_θ^\(1\) =2w D\(G\(c\)\-d\(s\_t\)\)\.\{\{\\begin\{array\}\[\]\{@\{\}w\{c\}\{0\.43\\linewidth\}:w\{c\}\{0\.43\\linewidth\}@\{\}\}\\mathcal\{L\}^\{\(0\)\}=w\\operatorname\{MSE\}\(v\_\\theta^\{\(0\)\},v\_\{\\mathrm\{tgt\}\}^\{\\mathrm\{ART\}\}\) =\\dfrac\{w\}\{D\}\\lVert G\(c\)\\rVert\_2^2\}\\lx@intercol\\vrule\\lx@add@cssclass&\\mathcal\{L\}^\{\(1\)\}=w\\operatorname\{MSE\}\(v\_\\theta^\{\(1\)\},v\_\{\\mathrm\{tgt\}\}^\{\\mathrm\{ART\}\}\) =\\dfrac\{w\}\{D\}\\lVert G\(c\)\-d\(s\_t\)\\rVert\_2^2\\\\\[6\.0pt\] \-\\left\.\\nabla\_\{v\_\\theta\}\\mathcal\{L\}\_\\theta\\right\|\_\{v\_\\theta=v\_\\theta^\{\(0\)\}\} =\\dfrac\{2w\}\{D\}\\,G\(c\)\}\\lx@intercol\\vrule\\lx@add@cssclass&\-\\left\.\\nabla\_\{v\_\\theta\}\\mathcal\{L\}\_\\theta\\right\|\_\{v\_\\theta=v\_\\theta^\{\(1\)\}\} =\\dfrac\{2w\}\{D\}\\bigl\(G\(c\)\-d\(s\_t\)\\bigr\)\.\\end\{array\}\(6\)
The second update combines the original guidanceG⁡\(c\)G\(c\)with restoration−d⁡\(st\)\-d\(s\_\{t\}\)towardvθ\(0\)v\_\{\\theta\}^\{\(0\)\}\. Restoration grows with each state’s drift, counteracting large displacements while leaving guidance dominant when drift is small, without additional tuning\. In NFT, cross\-state cancellation weakens guidance, allowing restoration to dominate and slow training \(Section[3\.2](https://arxiv.org/html/2610.12572#S3.SS2)\)\. DiffusionART instead sharesG⁡\(c\)G\(c\)across each prompt’s states, reducing this cancellation and improving stability without sacrificing efficiency \(Insight[3\.2](https://arxiv.org/html/2610.12572#S3.SS2); Figure[6](https://arxiv.org/html/2610.12572#S5.F6)\)\.

### 4\.3Trust\-Region Masking for ODE Rollouts

Despite restoration, applying guidance gradient in both updates can produce an excessive policy change within a single rollout batch, potentially destabilizing training\. Inspired by PPO clipping\[[Schulman et al\., 2017](https://arxiv.org/html/2610.12572#bib.bib30)\], we constrain it with a trust\-region mask as a stabilizing trick\. However, ODE transitions are deterministic, providing no usable finite KL to measure policy change\. We address this by following Flow\-GRPO’s ODE\-to\-SDE formulation\[[Liu et al\., 2025a](https://arxiv.org/html/2610.12572#bib.bib3)\]to define reference Gaussian transitions for the current and old policies:qθ\(⋅∣st\)=𝒩\(μ\(st,vθ\(st\)\),σref2\(t\)I\)q\_\{\\theta\}\(\\cdot\\mid s\_\{t\}\)=\\mathcal\{N\}\(\\mu\(s\_\{t\},v\_\{\\theta\}\(s\_\{t\}\)\),\\sigma\_\{\\rm ref\}^\{2\}\(t\)I\)andqold\(⋅∣st\)=𝒩\(μ\(st,vθ\(0\)\(st\)\),σref2\(t\)I\)q\_\{\\rm old\}\(\\cdot\\mid s\_\{t\}\)=\\mathcal\{N\}\(\\mu\(s\_\{t\},v\_\{\\theta\}^\{\(0\)\}\(s\_\{t\}\)\),\\sigma\_\{\\rm ref\}^\{2\}\(t\)I\)\. Based on this, we propose a KL proxy to measure the drift:

m\(st\)=1DDKL\(qθ\(⋅∣st\)∥qold\(⋅∣st\)\)=‖μ⁡\(st,vθ​\(st\)\)−μ⁡\(st,vθ\(0\)​\(st\)\)‖222​D​σref2​\(t\)\.m\(s\_\{t\}\)=\\frac\{1\}\{D\}D\_\{\\rm KL\}\\\!\\left\(q\_\{\\theta\}\(\\cdot\\mid s\_\{t\}\)\\,\\\|\\,q\_\{\\rm old\}\(\\cdot\\mid s\_\{t\}\)\\right\)=\\frac\{\\\|\\mu\(s\_\{t\},v\_\{\\theta\}\(s\_\{t\}\)\)\-\\mu\(s\_\{t\},v\_\{\\theta\}^\{\(0\)\}\(s\_\{t\}\)\)\\\|\_\{2\}^\{2\}\}\{2D\\sigma\_\{\\rm ref\}^\{2\}\(t\)\}\.\(7\)The proxy uses velocity predictions and rollouts remain ODE trajectories\. Hereμ⁡\(st,v\)\\mu\(s\_\{t\},v\)is the mean of the next latent after one reference SDE step with velocityvv, andσref​\(t\)\\sigma\_\{\\rm ref\}\(t\)is the policy\-independent noise scale of that step \(formulas in Appendix[C\.2](https://arxiv.org/html/2610.12572#A3.SS2)\)\.

Inspired by Flow\-DPPO\[[Ping et al\., 2026](https://arxiv.org/html/2610.12572#bib.bib8)\], we mask excessive outward drift while retaining inward corrections\. We identify outward updates by a positive inner product between the displacementd⁡\(st\)d\(s\_\{t\}\)and the second update’s output\-space descent directionG⁡\(c\)−d⁡\(st\)G\(c\)\-d\(s\_\{t\}\):

w\(st\)=t\[1−𝟏\{m\(st\)≥δ\}1\{⟨d\(st\),G\(c\)−d\(st\)⟩\>0\}\]\.w\(s\_\{t\}\)=t\\left\[1\-\\mathbf\{1\}\\\{m\(s\_\{t\}\)\\geq\\delta\\\}\\,\\mathbf\{1\}\\\{\\langle d\(s\_\{t\}\),G\(c\)\-d\(s\_\{t\}\)\\rangle\>0\\\}\\right\]\.\(8\)The two conditions detect excessive drift and outward updates, respectively\. We fix the drift thresholdδ\\deltato10−510^\{\-5\}throughout and use the factorttto emphasize high\-noise states\.

Algorithm 1DiffusionART: one training iteration1:Sample with an ODE solver using

vθ\(0\)=vθv\_\{\\theta\}^\{\(0\)\}=v\_\{\\theta\}\.Rollout Stage

2:For each prompt

cc, collect

KKendpoints with rewards

RR, states

sts\_\{t\}, and predictions

vθ\(0\)​\(st\)v\_\{\\theta\}^\{\(0\)\}\(s\_\{t\}\)\.

3:Compute

G⁡\(c\)G\(c\)by Equation[5](https://arxiv.org/html/2610.12572#S4.E5)for each prompt

cc\.

4:Freeze

vtgtART​\(st\)=vθ\(0\)​\(st\)\+G⁡\(c\)v\_\{\\mathrm\{tgt\}\}^\{\\mathrm\{ART\}\}\(s\_\{t\}\)=v\_\{\\theta\}^\{\(0\)\}\(s\_\{t\}\)\+G\(c\)for each state

sts\_\{t\}of prompt

cc\.

5:

6:Split the samples into two equal\-sized disjoint minibatches

B1,B2B\_\{1\},B\_\{2\}\.Training Stage

7:For each

B∈\{B1,B2\}B\\in\\\{B\_\{1\},B\_\{2\}\\\}, recompute

w⁡\(st\)w\(s\_\{t\}\)and update

θ\\thetawith one step on

8:

ℒ⁡\(B,θ\)=1\|B\|​∑st∈Bw⁡\(st\)​MSE⁡\(vθ​\(st\),vtgtART​\(st\)\)\\mathcal\{L\}\(B;\\theta\)=\\dfrac\{1\}\{\|B\|\}\\sum\_\{s\_\{t\}\\in B\}w\(s\_\{t\}\)\\operatorname\{MSE\}\(v\_\{\\theta\}\(s\_\{t\}\),v\_\{\\mathrm\{tgt\}\}^\{\\mathrm\{ART\}\}\(s\_\{t\}\)\)\.

### 4\.4Extension to Pairwise Reward Models

Unlike math or code, visual outputs often lack objectively verifiable correctness criteria, making pairwise preferences a natural alternative to scalar scores for reward modeling\[[Kirstain et al\., 2023](https://arxiv.org/html/2610.12572#bib.bib12)\]\.DiffusionART\-Pairwiseis a variant of DiffusionART which naturally supports online pairwise preference feedback\[[Kirstain et al\., 2023](https://arxiv.org/html/2610.12572#bib.bib12)\]\. For each prompt, we divideKKendpoints intoK/2K/2disjoint pairs and assign winner/loser labels\(1,0\)\(1,0\), or\(1/2,1/2\)\(1/2,1/2\)for ties\. These labels replace scalar rewards in the otherwise unchanged covariance and training pipeline\. Appendix[C\.3](https://arxiv.org/html/2610.12572#A3.SS3)provides the covariance formulation and pseudocode\.

## 5Experiments

We study the efficiency, stability, and scalability of DiffusionART, progressing from text\-to\-image \(T2I\) generation to larger models and higher\-dimensional text\-to\-video \(T2V\) and image\-to\-video \(I2V\) generation\.

### 5\.1Text\-to\-Image Generation

Setup\.We compare DiffusionART with Flow\-GRPO, Flow\-GRPO\-Fast \(CPS\)\[[Liu et al\., 2025a](https://arxiv.org/html/2610.12572#bib.bib3),[Wang and Yu, 2025](https://arxiv.org/html/2610.12572#bib.bib27)\], DanceGRPO\[[Xue et al\., 2025](https://arxiv.org/html/2610.12572#bib.bib13)\], Flow\-DPPO\[[Ping et al\., 2026](https://arxiv.org/html/2610.12572#bib.bib8)\], and DiffusionNFT\[[Zheng et al\., 2025](https://arxiv.org/html/2610.12572#bib.bib6)\]with and without EMA, denoted DiffusionNFT \(EMA\) and DiffusionNFT \(on\-policy\)\. All methods train LoRA adapters\[[Hu et al\., 2021](https://arxiv.org/html/2610.12572#bib.bib14)\]with rankr=32r=32and scaling factorα=64\\alpha=64at512×512512\\times 512resolution, without CFG or reference\-policy KL regularization\. On SD3\.5\-M, we use a learning rate of3×10−43\\times 10^\{\-4\}and 10\-step rollouts\. DiffusionNFT uses the original per\-task hyperparameters\[[Zheng et al\., 2025](https://arxiv.org/html/2610.12572#bib.bib6)\]to maximize its baseline performance\.

Single\- and multi\-reward optimization\.We evaluate SD3\.5\-M\[[Stability AI, 2024](https://arxiv.org/html/2610.12572#bib.bib10)\]on GenEval\[[Ghosh et al\., 2023](https://arxiv.org/html/2610.12572#bib.bib11)\], OCR, and PickScore\[[Kirstain et al\., 2023](https://arxiv.org/html/2610.12572#bib.bib12)\], and further jointly optimize GenEval, PickScore, CLIPScore\[[Hessel et al\., 2021](https://arxiv.org/html/2610.12572#bib.bib16)\], and HPSv2\.1\[[Wu et al\., 2023](https://arxiv.org/html/2610.12572#bib.bib17)\]\. Figure[5](https://arxiv.org/html/2610.12572#S5.F5)\(a–d\) shows that DiffusionART behaves similarly under single\- and multi\-reward training\. Figure[5](https://arxiv.org/html/2610.12572#S5.F5)\(d\) plots the total reward, with components in Appendix[A\.4](https://arxiv.org/html/2610.12572#A1.SS4)\. Without per\-task EMA tuning, DiffusionART avoids the severe collapse of DiffusionNFT \(on\-policy\) and achieves training efficiency comparable to DiffusionNFT \(EMA\)\. It is also much faster than Flow\-GRPO, reaching Flow\-GRPO’s step\-500 reward with10\.0×10\.0\\times,5\.2×5\.2\\times, and9\.6×9\.6\\timesfewer training steps on GenEval, OCR, and PickScore, respectively\.

Figure 5:Training rewards: \(a–c\) single\- and \(d\) multi\-reward SD3\.5\-M\.Scaling to larger models\.We extend the same recipe to FLUX\.1\-dev \(12B\)\[[Black Forest Labs, 2024](https://arxiv.org/html/2610.12572#bib.bib15)\]on GenEval, using six\-step rollouts and its built\-in guidance embedding at scale 3\.5, without additional CFG\. For DiffusionNFT, we retainβ=1\\beta=1and rollout\-policy EMA decaymin⁡\(0\.001​n,0\.5\)\\min\(0\.001n,0\.5\)from the SD3\.5\-M GenEval baseline, wherenncounts successful training updates\. As shown in Figure[6](https://arxiv.org/html/2610.12572#S5.F6)\(a\), DiffusionART learns fastest among all methods, while DiffusionNFT exhibits pronounced training\-reward fluctuations because these hyperparameters are not retuned for FLUX\. Table[2](https://arxiv.org/html/2610.12572#A1.T2)in Appendix[A\.1](https://arxiv.org/html/2610.12572#A1.SS1)reports the GenEval evaluation scores of different methods at step 500\.

![Refer to caption](https://arxiv.org/html/2610.12572v1/arxiv_t2i_flux_restoration_fonts_20261009.png)Figure 6:\(a\) FLUX\.1\-dev on GenEval\. \(b\) Guidance and restoration ablation on OCR\.Ablation of guidance and restoration\.Figure[6](https://arxiv.org/html/2610.12572#S5.F6)\(b\) comparesART: 1 updateandART: 2 updates, two DiffusionART variants with the same covariance guidance, againstNFT: 2 updateswith a carefully tunedβ=0\.1\\beta=0\.1\. As explained in Appendix[B\.3](https://arxiv.org/html/2610.12572#A2.SS3),ART: 1 updatelearns quickly but gradually deteriorates\. The second update inART: 2 updatesintroduces restoration and provides long\-term stability\. In contrast,NFT: 2 updatesbriefly collapses to near zero before recovering\. These results highlight the importance of both guidance type and restoration\.

### 5\.2Text\-to\-Video Generation

Setup\.We fine\-tune all parameters of the CFG\-distilled HunyuanVideo\-13B model\[[Kong et al\., 2024](https://arxiv.org/html/2610.12572#bib.bib18)\]with a learning rate of10−510^\{\-5\}and no KL regularization\. Each group\-based rollout uses 32 prompts and 8 videos per prompt\. We generate 53\-frame videos at480×480480\\times 480and 8 FPS with 16 sampling steps\. VideoReward\[[Liu et al\., 2025b](https://arxiv.org/html/2610.12572#bib.bib19)\]scores videos at 4 FPS for training and evaluation\. All methods use matched TA:VQ:MQ weights of7:2:17\{:\}2\{:\}1to improve output quality while preserving prompt following\. Figure[7](https://arxiv.org/html/2610.12572#S5.F7)\(a–c\) compares all three rewards over 400 training steps; Appendix[A\.5](https://arxiv.org/html/2610.12572#A1.SS5)reports an additional VQ:MQ=1:1=1\{:\}1comparison\.

Performance\.As shown in Figure[7](https://arxiv.org/html/2610.12572#S5.F7), DiffusionART surpasses all baselines in both VQ and MQ while keeping TA comparable with the strongest baseline\. DiffusionART reaches training VQ of 6 and MQ of 3 after 29 and 46 steps, respectively; Flow\-GRPO\-Fast, DanceGRPO, and DiffusionNFT \(EMA\) do not reach these levels within their recorded trajectories\. For VQ of 5\.5, DiffusionART needs 24 steps and 63\.4 GPU hours, versus 273 steps and 1,401\.4 GPU hours for DiffusionNFT \(EMA\):11\.4×\\timesfewer training steps and 22\.1×\\timesfewer GPU hours\. It retains its gains through 400 steps and achieves evaluation VQ and MQ of 8\.970 and 4\.224, as reported in Table[3](https://arxiv.org/html/2610.12572#A1.T3)\. DiffusionART improves evaluation VQ and MQ by 31\.4% and 57\.4%, respectively, over the strongest baseline\.

Figure 7:Efficient reward improvement across video generation\.\(a–c\) T2V: DiffusionART improves VQ and MQ fastest while maintaining TA comparable to the strongest baseline\. \(d–e\) I2V: DiffusionART rapidly improves both rewards, whereas the baselines improve slowly or plateau\.Ablation of trust\-region masking\.We remove only the trust\-region mask described in Section[4\.3](https://arxiv.org/html/2610.12572#S4.SS3), retaining normalized covariance guidance, timestep weighting, and two disjoint MSE updates\. With VQ:MQ=1:1=1\{:\}1weights, Figure[8](https://arxiv.org/html/2610.12572#S5.F8)shows that the no\-mask run drops around step 200 before recovering, whereas the masked run improves steadily\. Masking helps limit excessive updates, serving as an additional stability safeguard; Appendix[A\.7](https://arxiv.org/html/2610.12572#A1.SS7)provides further diagnostics of gradients and velocity drift, and Appendix[A\.6](https://arxiv.org/html/2610.12572#A1.SS6)reports the trust\-region ablation on T2I tasks\.

Figure 8:T2V with and without trust\-region masking\.Table 1:VQ\-only evaluation and GPU hours at step 400\.

Pairwise feedback and ablation of group size\.The performance of DiffusionART with pairwise train reward is close to that obtained with scalar feedback, using scalar reward as the ground truth evaluation metric\. Meanwhile, DiffusionART is insensitive to the group sizeKK, and a smallerKKsubstantially reduces GPU hours\. Table[8](https://arxiv.org/html/2610.12572#S5.F8)shows that, under the same 400\-step budget with pairwise feedback,K=2K=2reaches a final VQ of 12\.235, no lower than 12\.092 forK=8K=8\. BecauseK=2K=2uses only a quarter of the rollout videos and training samples, it consumes only 260\.8 GPU hours over the 400 steps, 3\.4×\\timesfewer than the 883\.3 GPU hours ofK=8K=8\. One pair per prompt thus suffices for stable and compute\-efficient optimization in this setting\.

### 5\.3Image\-to\-Video Generation

Setup\.We fine\-tune all parameters of the CFG\- and step\-distilled Wan2\.1\-I2V\-14B model\[[Wan Team, 2025](https://arxiv.org/html/2610.12572#bib.bib20)\]released by LightX2V\[[LightX2V, 2025](https://arxiv.org/html/2610.12572#bib.bib21)\]\. Each rollout uses 32 prompts, FLUX\.1\-dev reference images, and 8 videos per prompt\. We generate 53\-frame videos at400×640400\\times 640and 8 FPS with four sampling steps\. All methods use VQ:MQ=1:1=1\{:\}1weights, a learning rate of10−510^\{\-5\}, and no KL regularization\. Reward\-model settings follow Section[5\.2](https://arxiv.org/html/2610.12572#S5.SS2)\.

Performance\.DiffusionART improves both VQ and MQ efficiently, whereas the baselines improve slowly or plateau \(Figure[7](https://arxiv.org/html/2610.12572#S5.F7)\(d–e\)\)\. It reaches MQ of 6 in 45\.8 GPU hours versus 220\.3 for Flow\-GRPO\-Fast, a4\.8×\\timesreduction\. DiffusionNFT plateaus early and remains near that plateau through 400 updates, which we attribute to the residual self\-consistency force diluting reward\-improving guidance after gradient aggregation\. Evaluation results are in Table[4](https://arxiv.org/html/2610.12572#A1.T4)\.

## 6Conclusion

We studied why NFT\-style forward\-process diffusion post\-training can be unstable and found that its objective can introduce an implicit forward–reverse self\-consistency force that is not reliably canceled across states, while stronger restoration can improve stability at the cost of slower reward optimization\. DiffusionART presents a simple but effective way to systematically cancel the unstable force by aggregating endpoint reward guidance before assigning it to reverse\-rollout states, and then using a second update with a trust\-region mechanism to provide drift\-adaptive restoration for stabilization\. Experiments on image and video generation show efficient, stable reward improvement, including with pairwise feedback and small rollout groups\. Beyond the algorithm, our work provides insights into forward\-process training, offering a clearer understanding of diffusion reinforcement learning and a path toward scalable diffusion post\-training for visual generation\.

## 7Acknowledgement

We thank Kaiwen Zheng and Huayu Chen for the insightful discussion about the intuition behind DiffusionNFT and the valuable suggestion for the experiment design\.

## References

- Black Forest LabsFLUX\.1 \[dev\]\.Note:[https://huggingface\.co/black\-forest\-labs/FLUX\.1\-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev)Cited by:[§5\.1](https://arxiv.org/html/2610.12572#S5.SS1.p3.1)\.
- Blacket al\.\(2023\)K\. Black, M\. Janner, Y\. Du, I\. Kostrikov, and S\. LevineTraining diffusion models with reinforcement learning\.arXiv preprint arXiv:2305\.13301\.Cited by:[§1](https://arxiv.org/html/2610.12572#S1.p2.1)\.
- Gaoet al\.\(2025\)Y\. Gao, H\. Guo, T\. Hoang, W\. Huang, L\. Jiang,et al\.Seedance 1\.0: exploring the boundaries of video generation models\.arXiv preprint arXiv:2506\.09113\.Cited by:[§1](https://arxiv.org/html/2610.12572#S1.p1.1)\.
- Ghoshet al\.\(2023\)D\. Ghosh, H\. Hajishirzi, and L\. SchmidtGenEval: an object\-focused framework for evaluating text\-to\-image alignment\.arXiv preprint arXiv:2310\.11513\.Cited by:[§5\.1](https://arxiv.org/html/2610.12572#S5.SS1.p2.1)\.
- Heet al\.\(2025\)X\. He, S\. Fu, Y\. Zhao, W\. Li, J\. Yang, D\. Yin, F\. Rao, and B\. ZhangTempFlow\-GRPO: when timing matters for GRPO in flow models\.arXiv preprint arXiv:2508\.04324\.Cited by:[§1](https://arxiv.org/html/2610.12572#S1.p2.1)\.
- Hesselet al\.\(2021\)J\. Hessel, A\. Holtzman, M\. Forbes, R\. Le Bras, and Y\. ChoiCLIPScore: a reference\-free evaluation metric for image captioning\.arXiv preprint arXiv:2104\.08718\.Cited by:[§5\.1](https://arxiv.org/html/2610.12572#S5.SS1.p2.1)\.
- Huet al\.\(2021\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li,et al\.LoRA: low\-rank adaptation of large language models\.arXiv preprint arXiv:2106\.09685\.Cited by:[§5\.1](https://arxiv.org/html/2610.12572#S5.SS1.p1.1)\.
- Kirstainet al\.\(2023\)Y\. Kirstain, A\. Polyak, U\. Singer, S\. Matiana, J\. Penna, and O\. LevyPick\-a\-pic: an open dataset of user preferences for text\-to\-image generation\.arXiv preprint arXiv:2305\.01569\.Cited by:[§4\.4](https://arxiv.org/html/2610.12572#S4.SS4.p1.1),[§5\.1](https://arxiv.org/html/2610.12572#S5.SS1.p2.1)\.
- Konget al\.\(2024\)W\. Kong, Q\. Tian, Z\. Zhang, R\. Min, Z\. Dai,et al\.HunyuanVideo: a systematic framework for large video generative models\.arXiv preprint arXiv:2412\.03603\.Cited by:[§5\.2](https://arxiv.org/html/2610.12572#S5.SS2.p1.1)\.
- LightX2V \(2025\)LightX2VWan2\.1\-I2V\-14B\-720P\-StepDistill\-CfgDistill\-Lightx2v\.Note:[https://huggingface\.co/lightx2v/Wan2\.1\-I2V\-14B\-720P\-StepDistill\-CfgDistill\-Lightx2v](https://huggingface.co/lightx2v/Wan2.1-I2V-14B-720P-StepDistill-CfgDistill-Lightx2v)Cited by:[§5\.3](https://arxiv.org/html/2610.12572#S5.SS3.p1.1)\.
- Lipmanet al\.\(2022\)Y\. Lipman, R\. T\. Q\. Chen, H\. Ben\-Hamu, M\. Nickel, and M\. LeFlow matching for generative modeling\.arXiv preprint arXiv:2210\.02747\.Cited by:[§2](https://arxiv.org/html/2610.12572#S2.p2.1),[footnote 1](https://arxiv.org/html/2610.12572#footnote1)\.
- Liuet al\.\(2025a\)J\. Liu, G\. Liu, J\. Liang, Y\. Li, J\. Liu, X\. Wang, P\. Wan, D\. Zhang, and W\. OuyangFlow\-GRPO: training flow matching models via online RL\.arXiv preprint arXiv:2505\.05470\.Cited by:[§C\.1](https://arxiv.org/html/2610.12572#A3.SS1.p1.1),[§C\.2](https://arxiv.org/html/2610.12572#A3.SS2.p1.2),[§1](https://arxiv.org/html/2610.12572#S1.p1.1),[§1](https://arxiv.org/html/2610.12572#S1.p2.1),[§2](https://arxiv.org/html/2610.12572#S2.p1.1),[§4\.3](https://arxiv.org/html/2610.12572#S4.SS3.p1.1),[§5\.1](https://arxiv.org/html/2610.12572#S5.SS1.p1.1)\.
- Liuet al\.\(2025b\)J\. Liu, G\. Liu, J\. Liang, Z\. Yuan, X\. Liu,et al\.Improving video generation with human feedback\.arXiv preprint arXiv:2501\.13918\.Cited by:[§5\.2](https://arxiv.org/html/2610.12572#S5.SS2.p1.1)\.
- Liuet al\.\(2022\)X\. Liu, C\. Gong, and Q\. LiuFlow straight and fast: learning to generate and transfer data with rectified flow\.External Links:2209\.03003,[Link](https://arxiv.org/abs/2209.03003)Cited by:[§2](https://arxiv.org/html/2610.12572#S2.p2.1)\.
- Pinget al\.\(2026\)B\. Ping, X\. Zhou, P\. Qi, M\. Luo, L\. Bo, and T\. PangFlow\-DPPO: divergence proximal policy optimization for flow matching models\.arXiv preprint arXiv:2606\.11025\.Cited by:[§4\.3](https://arxiv.org/html/2610.12572#S4.SS3.p2.1),[§5\.1](https://arxiv.org/html/2610.12572#S5.SS1.p1.1)\.
- Potaptchiket al\.\(2025\)P\. Potaptchik, C\. Lee, and M\. S\. AlbergoTilt matching for scalable sampling and fine\-tuning\.arXiv preprint arXiv:2512\.21829\.Cited by:[§4\.1](https://arxiv.org/html/2610.12572#S4.SS1.p1.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§4\.3](https://arxiv.org/html/2610.12572#S4.SS3.p1.1)\.
- Stability AI \(2024\)Stability AIStable diffusion 3\.5 medium\.Note:[https://huggingface\.co/stabilityai/stable\-diffusion\-3\.5\-medium](https://huggingface.co/stabilityai/stable-diffusion-3.5-medium)Cited by:[§5\.1](https://arxiv.org/html/2610.12572#S5.SS1.p2.1)\.
- Wallaceet al\.\(2023\)B\. Wallace, M\. Dang, R\. Rafailov, L\. Zhou, A\. Lou, S\. Purushwalkam, S\. Ermon, C\. Xiong, S\. Joty, and N\. NaikDiffusion model alignment using direct preference optimization\.arXiv preprint arXiv:2311\.12908\.Cited by:[§1](https://arxiv.org/html/2610.12572#S1.p1.1)\.
- Wan Team \(2025\)Wan TeamWan: open and advanced large\-scale video generative models\.arXiv preprint arXiv:2503\.20314\.Cited by:[§5\.3](https://arxiv.org/html/2610.12572#S5.SS3.p1.1)\.
- Wang and Yu \(2025\)F\. Wang and Z\. YuCoefficients\-preserving sampling for reinforcement learning with flow matching\.External Links:2509\.05952,[Link](https://arxiv.org/abs/2509.05952)Cited by:[§5\.1](https://arxiv.org/html/2610.12572#S5.SS1.p1.1)\.
- Wanget al\.\(2026\)J\. Wang, J\. Liang, J\. Liu, H\. Liu, G\. Liu, J\. Zheng, W\. Pang, A\. Ma, Z\. Xie, X\. Wang, M\. Wang, P\. Wan, and X\. LiangGRPO\-Guard: mitigating implicit over\-optimization in flow matching via regulated clipping\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 5988–5998\.Cited by:[§1](https://arxiv.org/html/2610.12572#S1.p2.1)\.
- Wuet al\.\(2023\)X\. Wu, Y\. Hao, K\. Sun, Y\. Chen, F\. Zhu, R\. Zhao, and H\. LiHuman preference score v2: a solid benchmark for evaluating human preferences of text\-to\-image synthesis\.arXiv preprint arXiv:2306\.09341\.Cited by:[§5\.1](https://arxiv.org/html/2610.12572#S5.SS1.p2.1)\.
- Xuet al\.\(2023\)J\. Xu, X\. Liu, Y\. Wu, Y\. Tong, Q\. Li, M\. Ding, J\. Tang, and Y\. DongImageReward: learning and evaluating human preferences for text\-to\-image generation\.arXiv preprint arXiv:2304\.05977\.Cited by:[§A\.5](https://arxiv.org/html/2610.12572#A1.SS5.p3.1),[§1](https://arxiv.org/html/2610.12572#S1.p1.1)\.
- Xuet al\.\(2026a\)Y\. Xu, K\. Gao, Y\. Chen, Y\. Chen, Z\. Tang,et al\.Qwen\-Image\-2\.0\-RL technical report\.arXiv preprint arXiv:2606\.27608\.Cited by:[§1](https://arxiv.org/html/2610.12572#S1.p1.1)\.
- Xuet al\.\(2026b\)Y\. Xu, Y\. Zhang, S\. Luo, L\. Wang, and D\. HeDesigning reinforcement learning for diffusion models: a unified path\-space view\.arXiv preprint arXiv:2608\.14430\.Cited by:[§2](https://arxiv.org/html/2610.12572#S2.p1.1)\.
- Xueet al\.\(2026\)S\. Xue, C\. Ge, S\. Zhang, Y\. Li, and Z\. MaAdvantage weighted matching: aligning rl with pretraining in diffusion models\.External Links:2509\.25050,[Link](https://arxiv.org/abs/2509.25050)Cited by:[§2](https://arxiv.org/html/2610.12572#S2.p1.1)\.
- Xueet al\.\(2025\)Z\. Xue, J\. Wu, Y\. Gao, F\. Kong, L\. Zhu,et al\.DanceGRPO: unleashing GRPO on visual generation\.arXiv preprint arXiv:2505\.07818\.Cited by:[§1](https://arxiv.org/html/2610.12572#S1.p2.1),[§5\.1](https://arxiv.org/html/2610.12572#S5.SS1.p1.1)\.
- Zhenget al\.\(2025\)K\. Zheng, H\. Chen, H\. Ye, H\. Wang, Q\. Zhang, K\. Jiang, H\. Su, S\. Ermon, J\. Zhu, and M\. LiuDiffusionNFT: online diffusion reinforcement with forward process\.arXiv preprint arXiv:2509\.16117\.Cited by:[§1](https://arxiv.org/html/2610.12572#S1.p1.1),[§1](https://arxiv.org/html/2610.12572#S1.p2.1),[§2](https://arxiv.org/html/2610.12572#S2.p1.1),[§3\.1](https://arxiv.org/html/2610.12572#S3.SS1.p1.1),[§3\.2](https://arxiv.org/html/2610.12572#S3.SS2.p5.1),[§3](https://arxiv.org/html/2610.12572#S3.p1.1),[§5\.1](https://arxiv.org/html/2610.12572#S5.SS1.p1.1)\.
- Zhuet al\.\(2026\)Y\. Zhu, X\. Wang, S\. Lathuiliere, and V\. KalogeitonDiffusion reinforcement learning via centered reward distillation\.arXiv preprint arXiv:2603\.14128\.Cited by:[§2](https://arxiv.org/html/2610.12572#S2.p1.1),[§4\.1](https://arxiv.org/html/2610.12572#S4.SS1.p1.1)\.

###### Appendix Contents

1. [1Introduction](https://arxiv.org/html/2610.12572#S1)
2. [2Preliminaries](https://arxiv.org/html/2610.12572#S2)
3. [3Understanding NFT\-style Diffusion Reinforcement Learning](https://arxiv.org/html/2610.12572#S3)1. [3\.1Joint Positive–Negative Training: Partial Cancellation of a Hidden Consistency Force](https://arxiv.org/html/2610.12572#S3.SS1) 2. [3\.2Off\-Policy EMA: Stability through Restoration, at the Cost of Task\-Specific Tuning](https://arxiv.org/html/2610.12572#S3.SS2)
4. [4DiffusionART: Aggregation\-first Reward Tilting with Adaptive Restoration](https://arxiv.org/html/2610.12572#S4)1. [4\.1Aggregation\-First Guidance](https://arxiv.org/html/2610.12572#S4.SS1) 2. [4\.2Two\-Update Restoration: Drift\-Adaptive Stability without EMA Tuning](https://arxiv.org/html/2610.12572#S4.SS2) 3. [4\.3Trust\-Region Masking for ODE Rollouts](https://arxiv.org/html/2610.12572#S4.SS3) 4. [4\.4Extension to Pairwise Reward Models](https://arxiv.org/html/2610.12572#S4.SS4)
5. [5Experiments](https://arxiv.org/html/2610.12572#S5)1. [5\.1Text\-to\-Image Generation](https://arxiv.org/html/2610.12572#S5.SS1) 2. [5\.2Text\-to\-Video Generation](https://arxiv.org/html/2610.12572#S5.SS2) 3. [5\.3Image\-to\-Video Generation](https://arxiv.org/html/2610.12572#S5.SS3)
6. [6Conclusion](https://arxiv.org/html/2610.12572#S6)
7. [7Acknowledgement](https://arxiv.org/html/2610.12572#S7)
8. [References](https://arxiv.org/html/2610.12572#bib)
9. [AAdditional Experimental Results](https://arxiv.org/html/2610.12572#A1)1. [A\.1Scaling to FLUX\.1\-dev](https://arxiv.org/html/2610.12572#A1.SS1) 2. [A\.2T2V Evaluation](https://arxiv.org/html/2610.12572#A1.SS2) 3. [A\.3I2V Evaluation](https://arxiv.org/html/2610.12572#A1.SS3) 4. [A\.4Multi\-Reward Optimization for T2I](https://arxiv.org/html/2610.12572#A1.SS4) 5. [A\.5Alternative Reward Weights for T2V](https://arxiv.org/html/2610.12572#A1.SS5) 6. [A\.6Trust\-Region Masking for Image Generation](https://arxiv.org/html/2610.12572#A1.SS6) 7. [A\.7Trust\-Region Masking Diagnostics](https://arxiv.org/html/2610.12572#A1.SS7)
10. [BProofs](https://arxiv.org/html/2610.12572#A2)1. [B\.1Conditional Covariance Identity for Ideal NFT](https://arxiv.org/html/2610.12572#A2.SS1) 2. [B\.2NFT Force and Restoration Decompositions](https://arxiv.org/html/2610.12572#A2.SS2)1. [B\.2\.1Equivalent MSE Objective and On\-Policy Descent Direction](https://arxiv.org/html/2610.12572#A2.SS2.SSS1) 2. [B\.2\.2Reward Guidance and Self\-Consistency Force](https://arxiv.org/html/2610.12572#A2.SS2.SSS2) 3. [B\.2\.3Guidance and Quadratic Restoration](https://arxiv.org/html/2610.12572#A2.SS2.SSS3) 3. [B\.3Covariance as the First\-Order Reward Tilt](https://arxiv.org/html/2610.12572#A2.SS3)
11. [CClarification of Methods and Experiments](https://arxiv.org/html/2610.12572#A3)1. [C\.1Reverse\-Process Policy Optimization](https://arxiv.org/html/2610.12572#A3.SS1) 2. [C\.2Reference Transitions for ODE Trust\-Region Masking](https://arxiv.org/html/2610.12572#A3.SS2) 3. [C\.3Pairwise Feedback: Identity and Implementation](https://arxiv.org/html/2610.12572#A3.SS3)
12. [DVisualization](https://arxiv.org/html/2610.12572#A4)

## Appendix AAdditional Experimental Results

### A\.1Scaling to FLUX\.1\-dev

Table 2:FLUX\.1\-dev GenEval evaluation\.
### A\.2T2V Evaluation

Table[3](https://arxiv.org/html/2610.12572#A1.T3)reports the latest evaluation scores under matched TA:VQ:MQ=7:2:1=7\{:\}2\{:\}1training weights\. DiffusionART improves evaluation VQ and MQ by 31\.4% and 57\.4%, respectively, over the strongest baseline\.

Table 3:T2V evaluation\.Matched TA:VQ:MQ=7:2:1=7\{:\}2\{:\}1\.
### A\.3I2V Evaluation

Table 4:I2V evaluation\.VQ:MQ=1:1=1\{:\}1\.
### A\.4Multi\-Reward Optimization for T2I

We jointly optimize GenEval, PickScore, CLIPScore, and HPSv2\.1 on a single SD3\.5\-M policy for 500 training steps\. Figure[9](https://arxiv.org/html/2610.12572#A1.F9)reports each reward separately, comparing how the methods balance compositional accuracy, image–text alignment, and human preference under a shared multi\-reward objective\.

Figure 9:Training reward curves for four jointly optimized objectives on SD3\.5\-M over 500 steps\. PickScore and CLIPScore are shown as unscaled model logits\.
### A\.5Alternative Reward Weights for T2V

Figure 10:T2V with matched VQ:MQ=1:1=1\{:\}1weights\.The left pair shows VQ and the right pair MQ, each versus training steps and GPU hours\.Figure[10](https://arxiv.org/html/2610.12572#A1.F10)compares all methods with VQ:MQ=1:1=1\{:\}1weights and no TA reward\. DiffusionART reaches a training VQ of 6 in 20 steps, compared with 50 for DiffusionNFT \(EMA\) and 427 for Flow\-GRPO\-Fast:2\.5×\\timesand 21\.4×\\timesfewer training steps\. It uses 44\.2 GPU hours, versus 250\.2 and 502\.5, giving5\.7×\\timesand 11\.4×\\timesGPU\-hour speedups\. DanceGRPO reaches this level after 821 steps and 1805\.5 GPU hours\. Using the mean raw VQ/MQ reward,\(VQ\+MQ\)/2\(\\mathrm\{VQ\}\+\\mathrm\{MQ\}\)/2, DiffusionART reaches the late\-stage reward levels of DiffusionNFT \(EMA\) and Flow\-GRPO\-Fast with12\.02×12\.02\\timesand11\.72×11\.72\\timesfewer GPU hours, respectively \(Figure[1](https://arxiv.org/html/2610.12572#S0.F1)\)\. Its mean reward over the final 50 training steps is 13\.018, compared with 10\.105 and 8\.510 for the two baselines, representing improvements of 28\.8% and 53\.0%\.

For MQ of 3, DiffusionART needs 19 steps and 42\.9 GPU hours, compared with 33 steps and 165\.6 GPU hours for NFT, and 117 steps and 149\.4 GPU hours for Fast\. This gives1\.7×\\times/6\.2×\\timesstep speedupsand3\.9×\\times/3\.5×\\timesGPU\-hour speedupsover NFT/Fast\. Thus, DiffusionART also learns efficiently when the objective focuses only on VQ and MQ\.

Direct optimization with a differentiable reward\.We originally followed the undifferentiable reward model setting \(i\.e\. we can only access the scalar reward value, not including the gradient of the reward model\)\. However, methods like ReFL\[[Xu et al\., 2023](https://arxiv.org/html/2610.12572#bib.bib22)\]utilize the gradient of a VLM reward model to help training\. We also compare DiffusionART with ReFL under matched VQ:MQ=1:1=1\{:\}1weights \(Figure[11](https://arxiv.org/html/2610.12572#A1.F11)\)\. ReFL uses 32 prompts and one video per prompt \(K=1K=1\), with gradients through the frozen reward model and VAE to the final denoising step; earlier steps are detached\. It is trained for 400 updates at a learning rate of10−510^\{\-5\}, without KL regularization or EMA\.

DiffusionART improves both rewards with fewer optimizer updates, while ReFL uses fewer samples per update \(K=1K=1versusK=8K=8of DiffusionART\) but requires reward\-model backpropagation; DiffusionART uses reward values alone\.

Figure 11:DiffusionART versus ReFL with matched VQ:MQ=1:1=1\{:\}1weights\.The left pair shows VQ and the right pair MQ, each versus training steps and GPU hours\.
### A\.6Trust\-Region Masking for Image Generation

We remove only the trust\-region mask, retaining normalized covariance guidance, timestep weighting, and two disjoint MSE updates\. On the three SD3\.5\-M benchmarks and FLUX\.1\-dev, training remains effective without masking and can improve slightly faster \(Figure[12](https://arxiv.org/html/2610.12572#A1.F12)\)\. On FLUX\.1\-dev, the no\-mask run reaches a training GenEval score of 0\.971 by step 180\. The benefit of masking is more pronounced for video generation \(Figure[8](https://arxiv.org/html/2610.12572#S5.F8)\); Appendix[A\.7](https://arxiv.org/html/2610.12572#A1.SS7)examines its stabilizing effect beyond reward curves\.

Figure 12:Image trust\-region mask ablation\.\(a–c\) SD3\.5\-M; \(d\) FLUX\.1\-dev\.
### A\.7Trust\-Region Masking Diagnostics

We further inspect the matched HunyuanVideo comparison with VQ:MQ weights of1:11\{:\}1\. Without masking, training exhibits larger late\-stage gradient and velocity\-drift spikes \(Figure[13](https://arxiv.org/html/2610.12572#A1.F13)\), complementing the reward curves in Figure[8](https://arxiv.org/html/2610.12572#S5.F8)\(a–b\)\.

Although the mask constrains only the second update of each rollout, its stabilizing effect extends to subsequent first updates\. Over the shared first 300 training steps, the maximum first\-update gradient norm is 1\.96 with masking, compared with 3\.33 without it\. The velocity drift measured before second updates, which reflects displacement caused by the first update, also shows much larger spikes without masking \(Figure[13](https://arxiv.org/html/2610.12572#A1.F13)\)\. With masking, the velocity drift score stays a healthy level\. These observations suggest that restricting second updates helps keep later first updates well behaved, rather than merely suppressing gradients at the masked update\.

Figure 13:Trust\-region diagnostics on HunyuanVideo\-13B\.\(a\) Parameter gradient norm before clipping\. \(b\) Scheduler\-reference\-weighted velocity drift energy, averaged over training states before each update\.

## Appendix BProofs

### B\.1Conditional Covariance Identity for Ideal NFT

Lemma[1](https://arxiv.org/html/2610.12572#Thmlemma1)\(restated\)\.At a fixed supported statests\_\{t\}, letQoldQ\_\{\\mathrm\{old\}\}denoteQbQ\_\{b\}withb=oldb=\\mathrm\{old\}, i\.e\., the joint law obtained by forward noising the rollout endpointsx0∼pold\(⋅∣c\)x\_\{0\}\\sim p\_\{\\mathrm\{old\}\}\(\\cdot\\mid c\), and letq:=𝔼Qold​\[r⁡\(x0,c\)∣st\]q:=\\mathbb\{E\}\_\{Q\_\{\\mathrm\{old\}\}\}\[r\(x\_\{0\},c\)\\mid s\_\{t\}\]andg:=CovQold\[r\(x0,c\),ut∣st\]g:=\\operatorname\{Cov\}\_\{Q\_\{\\mathrm\{old\}\}\}\[r\(x\_\{0\},c\),\\allowbreak u\_\{t\}\\mid s\_\{t\}\]\. If the tilted distributions and moments exist and0<q<10<q<1, then

q⁡\(vfwd\+−vfwdold\)\\displaystyle q\\bigl\(v\_\{\\mathrm\{fwd\}\}^\{\+\}\-v\_\{\\mathrm\{fwd\}\}^\{\\mathrm\{old\}\}\\bigr\)=\(1−q\)​\(vfwdold−vfwd−\)=g,\\displaystyle=\(1\-q\)\\bigl\(v\_\{\\mathrm\{fwd\}\}^\{\\mathrm\{old\}\}\-v\_\{\\mathrm\{fwd\}\}^\{\-\}\\bigr\)=g,where all fields are evaluated atsts\_\{t\}\. Hence the positive and negative branches point along the same reward\-tilt directiongg\.

Proof\.LetZ\+​\(c\)=𝔼Pold​\[r∣c\]\>0Z\_\{\+\}\(c\)=\\mathbb\{E\}\_\{P\_\{\\mathrm\{old\}\}\}\[r\\mid c\]\>0andZ−​\(c\)=𝔼Pold​\[1−r∣c\]\>0Z\_\{\-\}\(c\)=\\mathbb\{E\}\_\{P\_\{\\mathrm\{old\}\}\}\[1\-r\\mid c\]\>0normalizeP\+P\_\{\+\}andP−P\_\{\-\}\. SinceP±P\_\{\\pm\}reweight onlyx0x\_\{0\}, whileϵ\\epsilonis independent ofx0x\_\{0\}and\(xt,ut\)\(x\_\{t\},u\_\{t\}\)are functions of\(x0,ϵ\)\(x\_\{0\},\\epsilon\), the joint laws of\(x0,ϵ,xt,ut\)\(x\_\{0\},\\epsilon,x\_\{t\},u\_\{t\}\)satisfy

Q\+\(⋅∣t,c\)\\displaystyle Q\_\{\+\}\(\\cdot\\mid t,c\)=r⁡\(x0,c\)Z\+​\(c\)Qold\(⋅∣t,c\),\\displaystyle=\\frac\{r\(x\_\{0\},c\)\}\{Z\_\{\+\}\(c\)\}Q\_\{\\mathrm\{old\}\}\(\\cdot\\mid t,c\),\(9\)Q−\(⋅∣t,c\)\\displaystyle Q\_\{\-\}\(\\cdot\\mid t,c\)=1−r⁡\(x0,c\)Z−​\(c\)Qold\(⋅∣t,c\)\.\\displaystyle=\\frac\{1\-r\(x\_\{0\},c\)\}\{Z\_\{\-\}\(c\)\}Q\_\{\\mathrm\{old\}\}\(\\cdot\\mid t,c\)\.\(10\)Marginalizing ontoxtx\_\{t\}and usingq=𝔼Qold​\[r⁡\(x0,c\)∣st\]q=\\mathbb\{E\}\_\{Q\_\{\\mathrm\{old\}\}\}\[r\(x\_\{0\},c\)\\mid s\_\{t\}\]gives

Q\+​\(xt∣t,c\)\\displaystyle Q\_\{\+\}\(x\_\{t\}\\mid t,c\)=qZ\+​\(c\)​Qold​\(xt∣t,c\),\\displaystyle=\\frac\{q\}\{Z\_\{\+\}\(c\)\}Q\_\{\\mathrm\{old\}\}\(x\_\{t\}\\mid t,c\),\(11\)Q−​\(xt∣t,c\)\\displaystyle Q\_\{\-\}\(x\_\{t\}\\mid t,c\)=1−qZ−​\(c\)​Qold​\(xt∣t,c\)\.\\displaystyle=\\frac\{1\-q\}\{Z\_\{\-\}\(c\)\}Q\_\{\\mathrm\{old\}\}\(x\_\{t\}\\mid t,c\)\.\(12\)Bayes’ rule therefore yields

Q\+\(⋅∣st\)\\displaystyle Q\_\{\+\}\(\\cdot\\mid s\_\{t\}\)=r⁡\(x0,c\)qQold\(⋅∣st\),\\displaystyle=\\frac\{r\(x\_\{0\},c\)\}\{q\}Q\_\{\\mathrm\{old\}\}\(\\cdot\\mid s\_\{t\}\),\(13\)Q−\(⋅∣st\)\\displaystyle Q\_\{\-\}\(\\cdot\\mid s\_\{t\}\)=1−r⁡\(x0,c\)1−qQold\(⋅∣st\)\.\\displaystyle=\\frac\{1\-r\(x\_\{0\},c\)\}\{1\-q\}Q\_\{\\mathrm\{old\}\}\(\\cdot\\mid s\_\{t\}\)\.\(14\)Taking conditional expectations ofutu\_\{t\}gives

q​vfwd\+​\(st\)\\displaystyle qv\_\{\\mathrm\{fwd\}\}^\{\+\}\(s\_\{t\}\)=𝔼Qold​\[r​ut∣st\],\\displaystyle=\\mathbb\{E\}\_\{Q\_\{\\mathrm\{old\}\}\}\[ru\_\{t\}\\mid s\_\{t\}\],\(15\)\(1−q\)​vfwd−​\(st\)\\displaystyle\\bigl\(1\-q\\bigr\)v\_\{\\mathrm\{fwd\}\}^\{\-\}\(s\_\{t\}\)=𝔼Qold​\[\(1−r\)​ut∣st\]\.\\displaystyle=\\mathbb\{E\}\_\{Q\_\{\\mathrm\{old\}\}\}\[\(1\-r\)u\_\{t\}\\mid s\_\{t\}\]\.\(16\)Sincevfwdold​\(st\)=𝔼Qold​\[ut∣st\]v\_\{\\mathrm\{fwd\}\}^\{\\mathrm\{old\}\}\(s\_\{t\}\)=\\mathbb\{E\}\_\{Q\_\{\\mathrm\{old\}\}\}\[u\_\{t\}\\mid s\_\{t\}\],

q⁡\(vfwd\+​\(st\)−vfwdold​\(st\)\)\\displaystyle q\\bigl\(v\_\{\\mathrm\{fwd\}\}^\{\+\}\(s\_\{t\}\)\-v\_\{\\mathrm\{fwd\}\}^\{\\mathrm\{old\}\}\(s\_\{t\}\)\\bigr\)=𝔼Qold​\[r​ut∣st\]−q​𝔼Qold​\[ut∣st\]=g,\\displaystyle=\\mathbb\{E\}\_\{Q\_\{\\mathrm\{old\}\}\}\[ru\_\{t\}\\mid s\_\{t\}\]\-q\\mathbb\{E\}\_\{Q\_\{\\mathrm\{old\}\}\}\[u\_\{t\}\\mid s\_\{t\}\]=g,\(17\)\(1−q\)​\(vfwdold​\(st\)−vfwd−​\(st\)\)\\displaystyle\\bigl\(1\-q\\bigr\)\\bigl\(v\_\{\\mathrm\{fwd\}\}^\{\\mathrm\{old\}\}\(s\_\{t\}\)\-v\_\{\\mathrm\{fwd\}\}^\{\-\}\(s\_\{t\}\)\\bigr\)=\(1−q\)​𝔼Qold​\[ut∣st\]−𝔼Qold​\[\(1−r\)​ut∣st\]=g\.\\displaystyle=\\bigl\(1\-q\\bigr\)\\mathbb\{E\}\_\{Q\_\{\\mathrm\{old\}\}\}\[u\_\{t\}\\mid s\_\{t\}\]\-\\mathbb\{E\}\_\{Q\_\{\\mathrm\{old\}\}\}\[\(1\-r\)u\_\{t\}\\mid s\_\{t\}\]=g\.\(18\)This proves both the collinearity and the conditional covariance identity\.

### B\.2NFT Force and Restoration Decompositions

#### B\.2\.1Equivalent MSE Objective and On\-Policy Descent Direction

Preliminary claims \(restated\)\.Forβ\>0\\beta\>0, freeze𝒟old\\mathcal\{D\}\_\{\\mathrm\{old\}\}and treatrr,utu\_\{t\}, andvoldv\_\{\\mathrm\{old\}\}as independent ofθ\\theta\. Withvtgt=vold\+2​r−1β​\(ut−vold\)v\_\{\\mathrm\{tgt\}\}=v\_\{\\mathrm\{old\}\}\+\\frac\{2r\-1\}\{\\beta\}\(u\_\{t\}\-v\_\{\\mathrm\{old\}\}\), the NFT objective satisfies

ℒNFT=β2​𝔼𝒟old​\[∥vθ−vtgt∥22\]\+const,\\mathcal\{L\}\_\{\\mathrm\{NFT\}\}=\\beta^\{2\}\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{\\mathrm\{old\}\}\}\\\!\\left\[\\lVert v\_\{\\theta\}\-v\_\{\\mathrm\{tgt\}\}\\rVert\_\{2\}^\{2\}\\right\]\+\\mathrm\{const\},whereconst\\mathrm\{const\}is independent ofθ\\theta\. At the on\-policy initializationvθ=voldv\_\{\\theta\}=v\_\{\\mathrm\{old\}\}, the single\-sample output\-space descent direction is−∇vθℓNFT=2​β​\(2​r−1\)​\(ut−vold\)\-\\nabla\_\{v\_\{\\theta\}\}\\ell\_\{\\mathrm\{NFT\}\}=2\\beta\(2r\-1\)\(u\_\{t\}\-v\_\{\\mathrm\{old\}\}\)\.

###### Proof\.

LetℓNFT\\ell\_\{\\mathrm\{NFT\}\}denote the weighted positive–negative loss for one sample, soℒNFT=𝔼𝒟old​\[ℓNFT\]\\mathcal\{L\}\_\{\\mathrm\{NFT\}\}=\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{\\mathrm\{old\}\}\}\[\\ell\_\{\\mathrm\{NFT\}\}\]\. Writed=vθ−voldd=v\_\{\\theta\}\-v\_\{\\mathrm\{old\}\},h=ut−voldh=u\_\{t\}\-v\_\{\\mathrm\{old\}\}, andA=2​r−1A=2r\-1\. Expanding and completing the square gives

ℓNFT\\displaystyle\\ell\_\{\\mathrm\{NFT\}\}=r​∥β​d−h∥22\+\(1−r\)​∥−β​d−h∥22\\displaystyle=r\\lVert\\beta d\-h\\rVert\_\{2\}^\{2\}\+\(1\-r\)\\lVert\-\\beta d\-h\\rVert\_\{2\}^\{2\}=β2​∥d∥22−2​β​A​⟨d,h⟩\+∥h∥22\\displaystyle=\\beta^\{2\}\\lVert d\\rVert\_\{2\}^\{2\}\-2\\beta A\\langle d,h\\rangle\+\\lVert h\\rVert\_\{2\}^\{2\}=β2​‖d−Aβ​h‖22\+\(1−A2\)​∥h∥22\.\\displaystyle=\\beta^\{2\}\\left\\lVert d\-\\frac\{A\}\{\\beta\}h\\right\\rVert\_\{2\}^\{2\}\+\(1\-A^\{2\}\)\\lVert h\\rVert\_\{2\}^\{2\}\.Sincevtgt=vold\+\(A/β\)​hv\_\{\\mathrm\{tgt\}\}=v\_\{\\mathrm\{old\}\}\+\(A/\\beta\)h, the first term isβ2​∥vθ−vtgt∥22\\beta^\{2\}\\lVert v\_\{\\theta\}\-v\_\{\\mathrm\{tgt\}\}\\rVert\_\{2\}^\{2\}\. Using1−A2=4​r​\(1−r\)1\-A^\{2\}=4r\(1\-r\)and taking the expectation over the frozen𝒟old\\mathcal\{D\}\_\{\\mathrm\{old\}\}proves Equation[1](https://arxiv.org/html/2610.12572#S2.E1), withconst=𝔼𝒟old​\[4​r​\(1−r\)​∥ut−vold∥22\]\\mathrm\{const\}=\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{\\mathrm\{old\}\}\}\[4r\(1\-r\)\\lVert u\_\{t\}\-v\_\{\\mathrm\{old\}\}\\rVert\_\{2\}^\{2\}\]independent ofθ\\theta\.

Differentiating the same sample loss with respect tovθv\_\{\\theta\}yields

−∇vθℓNFT=2​β​A​h−2​β2​d\.\-\\nabla\_\{v\_\{\\theta\}\}\\ell\_\{\\mathrm\{NFT\}\}=2\\beta Ah\-2\\beta^\{2\}d\.At the on\-policy initializationvθ=voldv\_\{\\theta\}=v\_\{\\mathrm\{old\}\}, we haved=0d=0, so the output\-space descent direction is2​β​\(2​r−1\)​\(ut−vold\)2\\beta\(2r\-1\)\(u\_\{t\}\-v\_\{\\mathrm\{old\}\}\), proving the stated proportionality\. ∎

#### B\.2\.2Reward Guidance and Self\-Consistency Force

Proposition[1](https://arxiv.org/html/2610.12572#Thmproposition1)\(restated\)\.LetℒNFT​\(st\)\\mathcal\{L\}\_\{\\mathrm\{NFT\}\}\(s\_\{t\}\)denote Equation[1](https://arxiv.org/html/2610.12572#S2.E1)conditioned onsts\_\{t\},qqandggfollow Lemma[1](https://arxiv.org/html/2610.12572#Thmlemma1), ande=vfwdold−vrvsolde=v\_\{\\mathrm\{fwd\}\}^\{\\mathrm\{old\}\}\-v\_\{\\mathrm\{rvs\}\}^\{\\mathrm\{old\}\}\. At the pure on\-policy update pointvθ=vrvsoldv\_\{\\theta\}=v\_\{\\mathrm\{rvs\}\}^\{\\mathrm\{old\}\}and forβ\>0\\beta\>0, its normalized output\-space force is

ℱNFT​\(st\):=−12​β​∇vθ​\(st\)ℒNFT​\(st\)\|vθ=vrvsold=2​g\+\(2​q−1\)​e\.\\mathcal\{F\}\_\{\\mathrm\{NFT\}\}\(s\_\{t\}\):=\-\\frac\{1\}\{2\\beta\}\\left\.\\nabla\_\{v\_\{\\theta\}\(s\_\{t\}\)\}\\mathcal\{L\}\_\{\\mathrm\{NFT\}\}\(s\_\{t\}\)\\right\|\_\{v\_\{\\theta\}=v\_\{\\mathrm\{rvs\}\}^\{\\mathrm\{old\}\}\}=2g\+\(2q\-1\)e\.
Proof\.Fixsts\_\{t\}, and writed=vθ​\(st\)−vrvsold​\(st\)d=v\_\{\\theta\}\(s\_\{t\}\)\-v\_\{\\mathrm\{rvs\}\}^\{\\mathrm\{old\}\}\(s\_\{t\}\)andh=ut−vrvsold​\(st\)h=u\_\{t\}\-v\_\{\\mathrm\{rvs\}\}^\{\\mathrm\{old\}\}\(s\_\{t\}\)\. The conditional objective is

ℒNFT​\(st\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{NFT\}\}\(s\_\{t\}\)=𝔼Qold​\[r​∥β​d−h∥22\+\(1−r\)​∥−β​d−h∥22∣st\]\\displaystyle=\\mathbb\{E\}\_\{Q\_\{\\mathrm\{old\}\}\}\\\!\\left\[r\\lVert\\beta d\-h\\rVert\_\{2\}^\{2\}\+\(1\-r\)\\lVert\-\\beta d\-h\\rVert\_\{2\}^\{2\}\\mid s\_\{t\}\\right\]\(19\)=β2​∥d∥22−2​β​⟨d,𝔼Qold​\[\(2​r−1\)​h∣st\]⟩\+𝔼Qold​\[∥h∥22∣st\]\.\\displaystyle=\\beta^\{2\}\\lVert d\\rVert\_\{2\}^\{2\}\-2\\beta\\left\\langle d,\\mathbb\{E\}\_\{Q\_\{\\mathrm\{old\}\}\}\[\(2r\-1\)h\\mid s\_\{t\}\]\\right\\rangle\+\\mathbb\{E\}\_\{Q\_\{\\mathrm\{old\}\}\}\[\\lVert h\\rVert\_\{2\}^\{2\}\\mid s\_\{t\}\]\.\(20\)At the pure on\-policy update point,d=0d=0\. Therefore,

ℱNFT​\(st\)\\displaystyle\\mathcal\{F\}\_\{\\mathrm\{NFT\}\}\(s\_\{t\}\)=−12​β​∇dℒNFT​\(st\)\|d=0\\displaystyle=\-\\frac\{1\}\{2\\beta\}\\left\.\\nabla\_\{d\}\\mathcal\{L\}\_\{\\mathrm\{NFT\}\}\(s\_\{t\}\)\\right\|\_\{d=0\}\(21\)=𝔼Qold​\[\(2​r−1\)​h∣st\]\.\\displaystyle=\\mathbb\{E\}\_\{Q\_\{\\mathrm\{old\}\}\}\[\(2r\-1\)h\\mid s\_\{t\}\]\.\(22\)Because𝔼Qold​\[h∣st\]=vfwdold−vrvsold=e\\mathbb\{E\}\_\{Q\_\{\\mathrm\{old\}\}\}\[h\\mid s\_\{t\}\]=v\_\{\\mathrm\{fwd\}\}^\{\\mathrm\{old\}\}\-v\_\{\\mathrm\{rvs\}\}^\{\\mathrm\{old\}\}=eandCovQold⁡\(r,h∣st\)=CovQold⁡\(r,ut∣st\)=g\\operatorname\{Cov\}\_\{Q\_\{\\mathrm\{old\}\}\}\(r,h\\mid s\_\{t\}\)=\\operatorname\{Cov\}\_\{Q\_\{\\mathrm\{old\}\}\}\(r,u\_\{t\}\\mid s\_\{t\}\)=g,

𝔼Qold​\[\(2​r−1\)​h∣st\]\\displaystyle\\mathbb\{E\}\_\{Q\_\{\\mathrm\{old\}\}\}\[\(2r\-1\)h\\mid s\_\{t\}\]=2​CovQold​\(r,h∣st\)\+\(2​𝔼Qold​\[r∣st\]−1\)​𝔼Qold​\[h∣st\]\\displaystyle=2\\operatorname\{Cov\}\_\{Q\_\{\\mathrm\{old\}\}\}\(r,h\\mid s\_\{t\}\)\+\(2\\mathbb\{E\}\_\{Q\_\{\\mathrm\{old\}\}\}\[r\\mid s\_\{t\}\]\-1\)\\mathbb\{E\}\_\{Q\_\{\\mathrm\{old\}\}\}\[h\\mid s\_\{t\}\]\(23\)=2​g\+\(2​q−1\)​e,\\displaystyle=2g\+\(2q\-1\)e,\(24\)which proves Equation[3](https://arxiv.org/html/2610.12572#S3.E3)\.

#### B\.2\.3Guidance and Quadratic Restoration

Proposition[2](https://arxiv.org/html/2610.12572#Thmproposition2)\(restated\)\.Letd=vθ−voldd=v\_\{\\theta\}\-v\_\{\\mathrm\{old\}\}andh=ut−voldh=u\_\{t\}\-v\_\{\\mathrm\{old\}\}\.

ℒNFT=𝔼𝒟old​\[−2​β​\(2​r−1\)​⟨d,h⟩\+β2​∥d∥22\]\+const,\\mathcal\{L\}\_\{\\mathrm\{NFT\}\}=\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{\\mathrm\{old\}\}\}\\\!\\left\[\-2\\beta\(2r\-1\)\\langle d,h\\rangle\+\\beta^\{2\}\\lVert d\\rVert\_\{2\}^\{2\}\\right\]\+\\mathrm\{const\},whereconst\\mathrm\{const\}is independent of the trainable parameters\.

Proof\.Using the sameddandhh, the positive and negative residuals areβ​d−h\\beta d\-hand−β​d−h\-\\beta d\-h, respectively\. Expanding their weighted squared norms gives

r​∥β​d−h∥22\+\(1−r\)​∥−β​d−h∥22=β2​∥d∥22−2​β​\(2​r−1\)​⟨d,h⟩\+∥h∥22\.\\displaystyle r\\lVert\\beta d\-h\\rVert\_\{2\}^\{2\}\+\(1\-r\)\\lVert\-\\beta d\-h\\rVert\_\{2\}^\{2\}=\\beta^\{2\}\\lVert d\\rVert\_\{2\}^\{2\}\-2\\beta\(2r\-1\)\\langle d,h\\rangle\+\\lVert h\\rVert\_\{2\}^\{2\}\.\(25\)Taking𝔼𝒟old\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{\\mathrm\{old\}\}\}and absorbing the final term into the constant proves Equation[4](https://arxiv.org/html/2610.12572#S3.E4)\.

### B\.3Covariance as the First\-Order Reward Tilt

Lemma[2](https://arxiv.org/html/2610.12572#Thmlemma2)\(restated\)\.If the relevant moments exist and differentiation can be exchanged with expectation nearτ=0\\tau=0, then∂∂τ​𝔼pτ​\[x0∣c\]\|τ=0=Covpold⁡\(R,x0∣c\)\\left\.\\frac\{\\partial\}\{\\partial\\tau\}\\mathbb\{E\}\_\{p\_\{\\tau\}\}\[x\_\{0\}\\mid c\]\\right\|\_\{\\tau=0\}=\\operatorname\{Cov\}\_\{p\_\{\\mathrm\{old\}\}\}\(R,x\_\{0\}\\mid c\)\.

Proof\.Fixccand suppress it below\. WriteM⁡\(τ\)=𝔼pold​\[x0​eτ​R\]M\(\\tau\)=\\mathbb\{E\}\_\{p\_\{\\mathrm\{old\}\}\}\[x\_\{0\}e^\{\\tau R\}\]andZ⁡\(τ\)=𝔼pold​\[eτ​R\]Z\(\\tau\)=\\mathbb\{E\}\_\{p\_\{\\mathrm\{old\}\}\}\[e^\{\\tau R\}\]\. The normalized tilted density ispτ​\(x0\)=pold​\(x0\)​eτ​R​\(x0\)/Z⁡\(τ\)p\_\{\\tau\}\(x\_\{0\}\)=p\_\{\\mathrm\{old\}\}\(x\_\{0\}\)e^\{\\tau R\(x\_\{0\}\)\}/Z\(\\tau\), so its endpoint mean isM⁡\(τ\)/Z⁡\(τ\)M\(\\tau\)/Z\(\\tau\)\. Assume differentiation under these expectations is valid; a sufficient condition is𝔼⁡\[\(1\+∥x0∥\)​\(1\+\|R\|\)​ea​\|R\|\]<∞\\mathbb\{E\}\[\(1\+\\lVert x\_\{0\}\\rVert\)\(1\+\|R\|\)e^\{a\|R\|\}\]<\\inftyfor somea\>0a\>0\. SinceZ⁡\(0\)=1Z\(0\)=1, the quotient rule gives

∂∂τ​M⁡\(τ\)Z⁡\(τ\)\|0=𝔼⁡\[x0​R\]−𝔼⁡\[x0\]​𝔼​\[R\]=Cov⁡\(R,x0\)\.\\left\.\\frac\{\\partial\}\{\\partial\\tau\}\\frac\{M\(\\tau\)\}\{Z\(\\tau\)\}\\right\|\_\{0\}=\\mathbb\{E\}\[x\_\{0\}R\]\-\\mathbb\{E\}\[x\_\{0\}\]\\mathbb\{E\}\[R\]=\\operatorname\{Cov\}\(R,x\_\{0\}\)\.\(26\)
From shared guidance to statewise correction\.Lemma[2](https://arxiv.org/html/2610.12572#Thmlemma2)motivates a shared reward\-improving direction computed from theKKendpoints already sampled per prompt\. This construction avoids state\-conditional posterior sampling and excludes the NFT self\-consistency term by construction\. However, a prompt\-level direction need not match the posterior guidance at every state, and different states can respond differently to the same parameter update\. Without statewise correction, the resulting deviations can accumulate, providing a explanation for the gradual deterioration with a single update in Figure[6](https://arxiv.org/html/2610.12572#S5.F6)\. The second MSE update complements shared guidance with local feedback: its restoring force grows with each state’s displacement from the rollout prediction\. This controls drift without requiring posterior sampling, combining scalable guidance with statewise stabilization\.

## Appendix CClarification of Methods and Experiments

### C\.1Reverse\-Process Policy Optimization

FlowGRPO treats denoising as a finite\-horizon policy\. For each prompt, it samplesKKstochastic reverse trajectories, group\-normalizes their terminal rewards, and applies clipped PPO to every transition\[[Liu et al\., 2025a](https://arxiv.org/html/2610.12572#bib.bib3)\]\. This directly supervises sampled states but requires tractable transition densities and stored trajectories\.

For reverse trajectoryxT:0\(i\)∼Poldrev\(⋅∣c\)x\_\{T:0\}^\{\(i\)\}\\sim P\_\{\\mathrm\{old\}\}^\{\\mathrm\{rev\}\}\(\\cdot\\mid c\), FlowGRPO assigns the group\-normalized terminal advantageAiA\_\{i\}to every transition and maximizes

𝒥FG=𝔼i,k\[\\displaystyle\\mathcal\{J\}\_\{\\mathrm\{FG\}\}=\\mathbb\{E\}\_\{i,k\}\\\!\\Big\[min⁡\(ρi,k​Ai,clip⁡\(ρi,k,1−ϵclip,1\+ϵclip\)​Ai\)\\displaystyle\\min\\\!\\big\(\\rho\_\{i,k\}A\_\{i\},\\,\\operatorname\{clip\}\(\\rho\_\{i,k\},1\-\\epsilon\_\{\\mathrm\{clip\}\},1\+\\epsilon\_\{\\mathrm\{clip\}\}\)A\_\{i\}\\big\)\(27\)−λKLDKL\(pθ\(⋅∣xk\(i\),c\)∥pref\(⋅∣xk\(i\),c\)\)\]\.\\displaystyle\-\\lambda\_\{\\mathrm\{KL\}\}D\_\{\\mathrm\{KL\}\}\\\!\\big\(p\_\{\\theta\}\(\\cdot\\mid x\_\{k\}^\{\(i\)\},c\)\\,\\\|\\,p\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\_\{k\}^\{\(i\)\},c\)\\big\)\\Big\]\.whereρi,k=pθ​\(xk−1\(i\)∣xk\(i\),c\)/pold​\(xk−1\(i\)∣xk\(i\),c\)\\rho\_\{i,k\}=p\_\{\\theta\}\(x\_\{k\-1\}^\{\(i\)\}\\mid x\_\{k\}^\{\(i\)\},c\)/p\_\{\\mathrm\{old\}\}\(x\_\{k\-1\}^\{\(i\)\}\\mid x\_\{k\}^\{\(i\)\},c\)and𝔼i,k\\mathbb\{E\}\_\{i,k\}also averages over prompts and old\-policy trajectories\. Hereϵclip\>0\\epsilon\_\{\\mathrm\{clip\}\}\>0is the PPO clip radius,λKL≥0\\lambda\_\{\\mathrm\{KL\}\}\\geq 0weights the penalty, andprefp\_\{\\mathrm\{ref\}\}is a fixed reference transition\. This requires the stochastic reverse transition used for rollout to expose a tractable density\.

### C\.2Reference Transitions for ODE Trust\-Region Masking

We strictly follow Flow\-GRPO’s ODE\-to\-SDE construction\[[Liu et al\., 2025a](https://arxiv.org/html/2610.12572#bib.bib3)\]; we refer to that work for the derivation\. With reverse step sizeh=t−tnext\>0h=t\-t\_\{\\rm next\}\>0and statest=\(xt,t,c\)s\_\{t\}=\(x\_\{t\},t,c\), wherextx\_\{t\}is the current latent, the reference transition has mean and variance when1\>t\>01\>t\>0:

μ⁡\(st,v\)\\displaystyle\\mu\(s\_\{t\},v\)=xt\+h⁡\[\(1\+ν22\)​v−ν22​\(1−t\)​xt\],\\displaystyle=x\_\{t\}\+h\\left\[\\left\(1\+\\frac\{\\nu^\{2\}\}\{2\}\\right\)v\-\\frac\{\\nu^\{2\}\}\{2\(1\-t\)\}x\_\{t\}\\right\],\(28\)σref2​\(t\)\\displaystyle\\sigma\_\{\\rm ref\}^\{2\}\(t\)=ν2​t1−t​h\.\\displaystyle=\\nu^\{2\}\\frac\{t\}\{1\-t\}h\.Substitutingvθ​\(st\)v\_\{\\theta\}\(s\_\{t\}\)andvθ\(0\)​\(st\)v\_\{\\theta\}^\{\(0\)\}\(s\_\{t\}\)gives the current and old reference means, respectively, henceμ⁡\(st,vθ​\(st\)\)−μ⁡\(st,vθ\(0\)​\(st\)\)=h⁡\(1\+ν2/2\)​d​\(st\)\\mu\(s\_\{t\},v\_\{\\theta\}\(s\_\{t\}\)\)\-\\mu\(s\_\{t\},v\_\{\\theta\}^\{\(0\)\}\(s\_\{t\}\)\)=h\(1\+\\nu^\{2\}/2\)d\(s\_\{t\}\)\. Att=1t=1, we use the endpoint conventionμ⁡\(st,v\)=xt\+h​v\\mu\(s\_\{t\},v\)=x\_\{t\}\+hvandσref2​\(1\)=ν2\\sigma\_\{\\rm ref\}^\{2\}\(1\)=\\nu^\{2\}\.

We useν=0\.8\\nu=0\.8andδ=10−5\\delta=10^\{\-5\}without further tuning\. The mask is recomputed before each update and treated as constant during backpropagation\.

### C\.3Pairwise Feedback: Identity and Implementation

Pairwise covariance identity\.ForKKendpoints formingM=K/2M=K/2disjoint pairs, with pairwise labelsyi∈\{0,12,1\}y\_\{i\}\\in\\\{0,\\tfrac\{1\}\{2\},1\\\}replacingRiR\_\{i\}in Equation[5](https://arxiv.org/html/2610.12572#S4.E5), use population reduction1/K1/Kglobally and1/21/2within each pair\. The global covariance equals the average of theMMpair covariances:Cov¯=\(2​K\)−1​∑j=1j​non​\-​tiedM\(x0,jwin−x0,jlose\)\\overline\{\\operatorname\{Cov\}\}=\(2K\)^\{\-1\}\\sum\_\{\\begin\{subarray\}\{c\}j=1\\\\ j\\,\\mathrm\{non\\text\{\-\}tied\}\\end\{subarray\}\}^\{M\}\(x\_\{0,j\}^\{\\mathrm\{win\}\}\-x\_\{0,j\}^\{\\mathrm\{lose\}\}\)\. Equivalently, the coefficient is1/\(4​M\)1/\(4M\)\. Tied pairs contribute zero\. ForK=2K=2, the covariance reduces to one quarter of the winner\-minus\-loser difference: after normalization, the guidance simply points from the worse sample to the better one\. A tie yields zero guidance\.

Proof\.Every pair has label mean1/21/2, including ties, so the global label mean is also1/21/2\. Thus centering endpoints changes no cross term:

Cov¯\\displaystyle\\overline\{\\operatorname\{Cov\}\}=1K​∑i=1K\(yi−12\)​\(x0\(i\)−x¯0,c\)=1K​∑i=1K\(yi−12\)​x0\(i\)\\displaystyle=\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{K\}\(y\_\{i\}\-\\tfrac\{1\}\{2\}\)\(x\_\{0\}^\{\(i\)\}\-\\bar\{x\}\_\{0,c\}\)=\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{K\}\(y\_\{i\}\-\\tfrac\{1\}\{2\}\)x\_\{0\}^\{\(i\)\}\(29\)=12​K​∑j=1j​non​\-​tiedM\(x0,jwin−x0,jlose\)\.\\displaystyle=\\frac\{1\}\{2K\}\\sum\_\{\\begin\{subarray\}\{c\}j=1\\\\ j\\,\\mathrm\{non\\text\{\-\}tied\}\\end\{subarray\}\}^\{M\}\\bigl\(x\_\{0,j\}^\{\\mathrm\{win\}\}\-x\_\{0,j\}^\{\\mathrm\{lose\}\}\\bigr\)\.\(30\)A non\-tied pair’s covariance, computed with denominator two, is one quarter of its winner\-minus\-loser difference\. Averaging those pair covariances gives the same expression; tied pairs contribute zero\. The pairwise labels enter the same covariance operator as scalar rewards, establishing the stated specialization\. The coefficient12​K\\frac\{1\}\{2K\}doesn’t matter since the covariance will be normalized to form the guidance\.

Guidance assignment\.We compare prompt\-global and pair\-local guidance on GenEval, using 24 images per prompt to form 12 disjoint pairs\. Prompt\-global guidance aggregates pair covariances before normalization and applies the shared direction to all trajectories\. Pair\-local guidance normalizes each pair’s covariance separately and applies it only to that pair\. At step 400, prompt\-global guidance reaches a held\-out GenEval score of0\.91, versus0\.78for pair\-local guidance\. It also achieves higher training rewards \(Figure[C\.3](https://arxiv.org/html/2610.12572#A3.SS3)\), showing that the better practice is pooling the pairs of the same prompt together before computing the covariance to reduce the variance\.

Figure 14:Guidance assignment on GenEval\.
Algorithm 2DiffusionART\-Pairwise: one training iteration1:Sample with an ODE solver using

vθ\(0\)=vθv\_\{\\theta\}^\{\(0\)\}=v\_\{\\theta\}\.Rollout Stage

2:For each prompt

cc, collect an even number

KKof endpoints

x0\(i\)x\_\{0\}^\{\(i\)\}, states

sts\_\{t\}, and predictions

vθ\(0\)​\(st\)v\_\{\\theta\}^\{\(0\)\}\(s\_\{t\}\)\.

3:Form

M=K/2M=K/2disjoint pairs per prompt; query the pairwise reward model\.

4:Assign pairwise rewards

yi=1y\_\{i\}=1to winners,

yi=0y\_\{i\}=0to losers, and

yi=12y\_\{i\}=\\tfrac\{1\}\{2\}for ties\.

5:Compute covariance guidance by Equation[5](https://arxiv.org/html/2610.12572#S4.E5), replacing scalar rewards

RiR\_\{i\}with pairwise rewards

yiy\_\{i\}\.

6:Freeze

vtgt​\(st\)=vθ\(0\)​\(st\)\+G⁡\(c\)v\_\{\\mathrm\{tgt\}\}\(s\_\{t\}\)=v\_\{\\theta\}^\{\(0\)\}\(s\_\{t\}\)\+G\(c\)for each state\.

7:

8:Split the samples into two equal\-sized disjoint minibatches

B1,B2B\_\{1\},B\_\{2\}\.Training Stage

9:For each

B∈\{B1,B2\}B\\in\\\{B\_\{1\},B\_\{2\}\\\}, recompute

w⁡\(st\)w\(s\_\{t\}\)and update

θ\\thetawith one step on

10:

ℒ⁡\(B,θ\)=1\|B\|​∑st∈Bw⁡\(st\)​MSE⁡\(vθ​\(st\),vtgt​\(st\)\)\\mathcal\{L\}\(B;\\theta\)=\\dfrac\{1\}\{\|B\|\}\\sum\_\{s\_\{t\}\\in B\}w\(s\_\{t\}\)\\operatorname\{MSE\}\(v\_\{\\theta\}\(s\_\{t\}\),v\_\{\\mathrm\{tgt\}\}\(s\_\{t\}\)\)\.

## Appendix DVisualization

![Refer to caption](https://arxiv.org/html/2610.12572v1/diffusionart_t2v_selected24_p1_base_model_20261009.png)Figure 15:Text\-to\-video generation before and after DiffusionART\.Each pair uses the same prompt, seed, and sampling settings\. Four chronologically ordered frames are shown at identical frame positions within each pair: pretrained outputs on the left and DiffusionART outputs on the right\.![Refer to caption](https://arxiv.org/html/2610.12572v1/diffusionart_t2i_selected16_p1_base_model_20261009.png)Figure 16:Text\-to\-image generation with multi\-reward optimization\.Selected SD3\.5\-M outputs before and after DiffusionART, jointly optimized with GenEval, CLIPScore, PickScore, and HPSv2\. Each pair shows the same prompt, with pretrained outputs on the left and DiffusionART outputs on the right\.![Refer to caption](https://arxiv.org/html/2610.12572v1/diffusionart_t2i_selected16_p2_base_model_20261009.png)Figure 17:Additional text\-to\-image comparisons\.The top two rows use multi\-reward optimization; the bottom two use PickScore\. Each pair shows pretrained and DiffusionART outputs under the same prompt\.