DiffusionRVGE: Reinforcement Learning via Regression-Based Value Gradient Estimation

# Reinforcement Learning viaRegression-Based Value Gradient Estimation
Source: [https://arxiv.org/html/2610.22973](https://arxiv.org/html/2610.22973)
## DiffusionRVGE: Reinforcement Learning via Regression\-Based Value Gradient Estimation

Jiayuan WangAffiliation:Qiuzhen College, Tsinghua UniversityAffiliation:Beijing 100084, ChinaEmail:[jy\-wang25@mails\.tsinghua\.edu\.cn](mailto:)Tongyao Pang††thanks:Corresponding author\.Affiliation:Yau Mathematical Sciences CenterAffiliation:Shuangqing Complex, Tsinghua UniversityAffiliation:Beijing 100084, ChinaEmail:[typang@tsinghua\.edu\.cn](mailto:)

###### Abstract

Reinforcement learning improves diffusion models using rewards on generated images, but repeated denoising and gradient estimation make training expensive\. We introduce DiffusionRVGE, an efficient method that constructs useful update directions by regressing centered rewards on centered final denoised images\. It requires neither reward derivatives nor adjoint backpropagation through the sampling trajectory\. A unified first\-variation formulation identifies the value\-gradient target with or without a trajectory\-KL penalty\. A population analysis under Gaussian noising explains the regression geometry, while a subspace analysis motivates clean\-image coordinates\. The implementation adapts rollout bandwidth and query weights to reward continuity: local rollouts suit smooth rewards, whereas broader groups provide informative contrasts for discrete rewards\. Trained jointly on five rewards, one SD3\.5\-Medium model matches or exceeds DiffusionNFT’s joint model on seven of eight metrics, reaching GenEval0\.940\.94, OCR0\.920\.92, and PickScore23\.8223\.82at about one fifth of its estimated training compute\. In single\-reward training, DiffusionRVGE reaches matched reward thresholds2\.12\.1–6\.7×6\.7\\timesfaster than reproduced DiffusionNFT on identical hardware\. Ablations support the clean\-image estimator, direct direction transfer, and reward\-dependent rollout choices\. Code is available at[https://github\.com/jy\-wang\-tsinghua/DIFFUSIONRVGE](https://github.com/jy-wang-tsinghua/DIFFUSIONRVGE)\.

## 1Introduction

Diffusion and flow matching models generate images by transforming noise into data\([Ho et al\., 2020](https://arxiv.org/html/2610.22973#bib.bib9);[Song et al\., 2021](https://arxiv.org/html/2610.22973#bib.bib20);[Lipman et al\., 2023](https://arxiv.org/html/2610.22973#bib.bib13);[Esser et al\., 2024](https://arxiv.org/html/2610.22973#bib.bib5)\)\. Reinforcement learning \(RL\) adapts these generators to scalar rewards measuring preferences, text accuracy, or compositional correctness\. Methods including DDPO, Flow\-GRPO, and DiffusionNFT improve generation through reward\-based fine\-tuning\([Black et al\., 2023](https://arxiv.org/html/2610.22973#bib.bib1);[Liu et al\., 2025](https://arxiv.org/html/2610.22973#bib.bib14);[Zheng et al\., 2026](https://arxiv.org/html/2610.22973#bib.bib24)\)\. Training remains expensive: each update requires image generation and an estimate of how model changes affect the final reward\. Our aim is to obtain effective update directions at substantially lower training cost\.

We introduce DiffusionRVGE, which constructs reward\-informed update directions by regression on final denoised images\. To motivate the estimator, consider the Gaussian noising path

𝐗t=αt​𝐙\+σt​ϵ,𝐙∼ρ,ϵ∼𝒩⁡\(0,I\),\\mathbf\{X\}\_\{t\}=\\alpha\_\{t\}\\mathbf\{Z\}\+\\sigma\_\{t\}\\boldsymbol\{\\epsilon\},\\qquad\\mathbf\{Z\}\\sim\\rho,\\quad\\boldsymbol\{\\epsilon\}\\sim\\mathcal\{N\}\(0,I\),\(1\)where𝐙\\mathbf\{Z\}andϵ\\boldsymbol\{\\epsilon\}are independent,𝐙\\mathbf\{Z\}is a clean image or latent, and generation reverses the noising process\. LetP𝜽P\_\{\\boldsymbol\{\\theta\}\}denote the law of a stochastic denoising trajectory\. A common RL objective,𝔼P𝜽R\(𝐙\)−τKL\(P𝜽∥Pref\)\\mathbb\{E\}\_\{P\_\{\\boldsymbol\{\\theta\}\}\}R\(\\mathbf\{Z\}\)\-\\tau\\operatorname\{KL\}\(P\_\{\\boldsymbol\{\\theta\}\}\\\|P\_\{\\mathrm\{ref\}\}\), balances reward against deviation from a reference model\. Forτ\>0\\tau\>0, under an independent\-noise initialization, its optimal\-control formulation identifies a reward\-tilted terminal targetπ∗∝ρ​eR/τ\\pi^\{\*\}\\propto\\rho e^\{R/\\tau\}\. The policy first variation gives a common cross\-term plus quadratic\-penalty loss for everyτ≥0\\tau\\geq 0, including unregularized reward optimization\. This supplies a target for efficient approximation, rather than requiring differentiation through every denoising step\.

Our estimator fits centered reward differences to clean\-image contrasts using normalized ridge regression\. Under exact conditioning in the Gaussian noising model, the population normal equations connect clean\-image regression to a covariance\-preconditioned value gradient\. A subspace analysis characterizes how denoised coordinates improve directional estimation, and our alignment experiments demonstrate this advantage in the evaluated models\. We transfer the fitted directions directly to model updates without an adjoint backward pass\.

Regression uses reward values and benefits from informative variation across the rollout bank\. Smooth scores permit local perturbations; discrete scores can remain constant across these perturbations and benefit from independent\-seed groups\. We therefore adapt rollout bandwidth and query weights to reward continuity while retaining the same centered squared\-regression estimator\. This gives one practical method for both continuous preferences and discrete task scores\.

Our contributions are:

- •Efficient regression\-based updates\.Clean\-image regression constructs update directions without reward derivatives or adjoint backpropagation, guided by gradient matching and supported by directional analysis and empirical validation\.
- •Reward\-adaptive implementation\.Local trajectory banks and broader rollout groups share a centered regression formulation, with reward\-dependent query weights validated by ablations\.
- •Strong results at lower compute\.Our joint model matches or exceeds DiffusionNFT on seven of eight metrics at about one fifth of its estimated training compute\. Single\-reward runs reach matched thresholds2\.12\.1–6\.7×6\.7\\timesfaster on identical hardware\.

## 2Related work

#### RL for diffusion and flow models\.

DDPO and Flow\-GRPO optimize stochastic denoising policies\([Black et al\., 2023](https://arxiv.org/html/2610.22973#bib.bib1);[Liu et al\., 2025](https://arxiv.org/html/2610.22973#bib.bib14)\); DiffusionNFT uses positive and negative generations in forward\-process matching\([Zheng et al\., 2026](https://arxiv.org/html/2610.22973#bib.bib24)\)\. DPOK, Diffusion\-DPO, DanceGRPO, and Centered Reward Distillation provide other reward\- or preference\-based objectives\([Fan et al\., 2023](https://arxiv.org/html/2610.22973#bib.bib6);[Wallace et al\., 2023](https://arxiv.org/html/2610.22973#bib.bib21);[Xue et al\., 2025](https://arxiv.org/html/2610.22973#bib.bib23);[Zhu et al\., 2026](https://arxiv.org/html/2610.22973#bib.bib26)\)\. A path\-space perspective identifies value\-gradient estimation as a shared component\([Xu et al\., 2026](https://arxiv.org/html/2610.22973#bib.bib22)\)\. DRaFT differentiates rewards through sampling, whereas FDFO transfers paired reward differences into velocity updates\([Clark et al\., 2023](https://arxiv.org/html/2610.22973#bib.bib3);[McAllister et al\., 2026](https://arxiv.org/html/2610.22973#bib.bib16)\)\. Our joint regression accounts for correlations among clean\-image differences\. Temporal and stepwise methods change branching or credit assignment\([He et al\., 2025](https://arxiv.org/html/2610.22973#bib.bib8);[Savani et al\., 2026](https://arxiv.org/html/2610.22973#bib.bib19);[Zhou et al\., 2025](https://arxiv.org/html/2610.22973#bib.bib25);[Chen et al\., 2025](https://arxiv.org/html/2610.22973#bib.bib2)\); our bandwidth choices address reward continuity and discreteness\.

#### Regression and controlled dynamics\.

Adjoint Matching learns stochastic\-control targets through adjoint regression\([Domingo\-Enrich et al\., 2024](https://arxiv.org/html/2610.22973#bib.bib4)\)\. Reward Score Matching and path\-space analyses relate diffusion RL to reward\-conditioned score or value gradients\([Lee et al\., 2026](https://arxiv.org/html/2610.22973#bib.bib12);[Xu et al\., 2026](https://arxiv.org/html/2610.22973#bib.bib22)\)\. Our analysis distinguishes ordinary reward regression from the soft\-value oracle of trajectory\-KL control\. Dimension\-dependent derivative\-free estimation and subspace\-guided search\([Nesterov & Spokoiny, 2017](https://arxiv.org/html/2610.22973#bib.bib17);[Maheswaranathan et al\., 2019](https://arxiv.org/html/2610.22973#bib.bib15)\)motivate the clean\-coordinate comparison\.

## 3Gradient matching with clean\-image regression

We first identify the RL update target, then study the geometry of clean\-image regression\. Population and subspace analyses explain the role of preconditioning and the choice of coordinates; the corresponding proofs and technical conditions are in Appendices[A](https://arxiv.org/html/2610.22973#A1)and[B](https://arxiv.org/html/2610.22973#A2)\.

### 3\.1The gradient\-matching objective

Use generation times=T−ts=T\-tand a reference diffusion with drift𝐛ref\\mathbf\{b\}\_\{\\mathrm\{ref\}\}and noise coefficientγs\>0\\gamma\_\{s\}\>0\. A learned correction𝐮𝜽\\mathbf\{u\}\_\{\\boldsymbol\{\\theta\}\}defines

d𝐘s=\(𝐛ref\+𝐮𝜽\)\(𝐘s,s\)ds\+γsd𝐁s,Jτ=𝔼P𝜽R\(𝐘T\)−τKL\(P𝜽∥Pref\)\.\\mathrm\{d\}\\mathbf\{Y\}\_\{s\}=\(\\mathbf\{b\}\_\{\\mathrm\{ref\}\}\+\\mathbf\{u\}\_\{\\boldsymbol\{\\theta\}\}\)\(\\mathbf\{Y\}\_\{s\},s\)\\mathrm\{d\}s\+\\gamma\_\{s\}\\mathrm\{d\}\\mathbf\{B\}\_\{s\},\\qquad J\_\{\\tau\}=\\mathbb\{E\}\_\{P\_\{\\boldsymbol\{\\theta\}\}\}R\(\\mathbf\{Y\}\_\{T\}\)\-\\tau\\operatorname\{KL\}\(P\_\{\\boldsymbol\{\\theta\}\}\\\|P\_\{\\mathrm\{ref\}\}\)\.\(2\)Forτ\>0\\tau\>0, tilting reference paths byeR/τe^\{R/\\tau\}gives a lawQ∗Q^\{\*\}with terminal distributionπ∗∝eR/τ​ρ\\pi^\{\*\}\\propto e^\{R/\\tau\}\\rhoand

Jτ=τlogZτ−τKL\(P𝜽∥Q∗\),Zτ=𝔼ρeR/τ\.J\_\{\\tau\}=\\tau\\log Z\_\{\\tau\}\-\\tau\\operatorname\{KL\}\(P\_\{\\boldsymbol\{\\theta\}\}\\\|Q^\{\*\}\),\\qquad Z\_\{\\tau\}=\\mathbb\{E\}\_\{\\rho\}e^\{R/\\tau\}\.\(3\)Under the independent\-noise initialization, this identifies the desired terminal target\. Define the soft valueWt=τ​log⁡HtW\_\{t\}=\\tau\\log H\_\{t\}, whereHt​\(𝐱\)=𝔼ref​\[eR⁡\(𝐙\)/τ∣𝐗t=𝐱\]H\_\{t\}\(\\mathbf\{x\}\)=\\mathbb\{E\}\_\{\\mathrm\{ref\}\}\[e^\{R\(\\mathbf\{Z\}\)/\\tau\}\\mid\\mathbf\{X\}\_\{t\}=\\mathbf\{x\}\]\.

###### Theorem 1\(Gradient matching for trajectory\-KL RL\)\.

Forτ\>0\\tau\>0, reference and controlled processes with the same initial law and diffusion coefficient satisfy, under the regularity conditions in Appendix[A](https://arxiv.org/html/2610.22973#A1),

Cτ−Jτ=τ2𝔼P𝜽∫0T‖𝐮𝜽−𝐮s∗‖2γs2ds,𝐮s∗=γs2τ∇WT−s,C\_\{\\tau\}\-J\_\{\\tau\}=\\frac\{\\tau\}\{2\}\\mathbb\{E\}\_\{P\_\{\\boldsymbol\{\\theta\}\}\}\\int\_\{0\}^\{T\}\\frac\{\\\|\\mathbf\{u\}\_\{\\boldsymbol\{\\theta\}\}\-\\mathbf\{u\}\_\{s\}^\{\*\}\\\|^\{2\}\}\{\\gamma\_\{s\}^\{2\}\}\\mathrm\{d\}s,\\qquad\\mathbf\{u\}\_\{s\}^\{\*\}=\\frac\{\\gamma\_\{s\}^\{2\}\}\{\\tau\}\\nabla W\_\{T\-s\},\(4\)whereCτ=𝔼⁡\[WT​\(𝐘0\)\]C\_\{\\tau\}=\\mathbb\{E\}\[W\_\{T\}\(\\mathbf\{Y\}\_\{0\}\)\]is independent of𝛉\\boldsymbol\{\\theta\}\.

For a common local update valid for everyτ≥0\\tau\\geq 0, use the continuation value of the frozen collection policy𝜽old\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}:

Ft𝜽old,τ​\(𝐱\)=𝔼P𝜽old​\[R⁡\(𝐘T\)−τ2​∫T−tT‖𝐮𝜽old​\(𝐘s,s\)‖2γs2​𝑑s\|𝐘T−t=𝐱\]\.F\_\{t\}^\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\},\\tau\}\(\\mathbf\{x\}\)=\\mathbb\{E\}\_\{P\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}\}\\\!\\left\[R\(\\mathbf\{Y\}\_\{T\}\)\-\\frac\{\\tau\}\{2\}\\int\_\{T\-t\}^\{T\}\\frac\{\\\|\\mathbf\{u\}\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}\(\\mathbf\{Y\}\_\{s\},s\)\\\|^\{2\}\}\{\\gamma\_\{s\}^\{2\}\}\\mathrm\{d\}s\\,\\middle\|\\,\\mathbf\{Y\}\_\{T\-t\}=\\mathbf\{x\}\\right\]\.\(5\)Equivalently,Ft𝜽old,τ=Vt𝜽old−τ​Ct𝜽oldF\_\{t\}^\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\},\\tau\}=V\_\{t\}^\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}\-\\tau C\_\{t\}^\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}, whereVt𝜽oldV\_\{t\}^\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}is expected terminal reward andCt𝜽oldC\_\{t\}^\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}is expected future control energy\. ThusFt𝜽old,0=Vt𝜽oldF\_\{t\}^\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\},0\}=V\_\{t\}^\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}; at the optimal policy forτ\>0\\tau\>0,Ft∗,τ=WtF\_\{t\}^\{\*,\\tau\}=W\_\{t\}\. The same value definition covers both cases\.

For the velocity parameterization𝐛𝜽=At​𝐱−2​𝐯𝜽\\mathbf\{b\}\_\{\\boldsymbol\{\\theta\}\}=A\_\{t\}\\mathbf\{x\}\-2\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\}withAt=α˙t/αtA\_\{t\}=\\dot\{\\alpha\}\_\{t\}/\\alpha\_\{t\}, writeδ​𝐯𝜽=𝐯𝜽−𝐯ref\\delta\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\}=\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\}\-\\mathbf\{v\}\_\{\\mathrm\{ref\}\}andνt2=γT−t2\\nu\_\{t\}^\{2\}=\\gamma\_\{T\-t\}^\{2\}\. Then𝐮𝜽=−2​δ​𝐯𝜽\\mathbf\{u\}\_\{\\boldsymbol\{\\theta\}\}=\-2\\delta\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\}\. The first variation ofJτJ\_\{\\tau\}gives the detached loss

ℒGM,τ=𝔼μ𝜽old​\[wt​\(2​⟨δ​𝐯𝜽,sg⁡\[∇Ft𝜽old,τ\]⟩⏟value\-gradient update\+2​τνt2​‖δ​𝐯𝜽‖2⏟trajectory\-KL penalty\)\],τ≥0\.\\mathcal\{L\}\_\{\\mathrm\{GM\},\\tau\}=\\mathbb\{E\}\_\{\\mu\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}\}\\\!\\left\[w\_\{t\}\\left\(\\underbrace\{2\\langle\\delta\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\},\\operatorname\{sg\}\[\\nabla F\_\{t\}^\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\},\\tau\}\]\\rangle\}\_\{\\text\{value\-gradient update\}\}\+\\underbrace\{\\frac\{2\\tau\}\{\\nu\_\{t\}^\{2\}\}\\\|\\delta\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\}\\\|^\{2\}\}\_\{\\text\{trajectory\-KL penalty\}\}\\right\)\\right\],\\quad\\tau\\geq 0\.\(6\)With time\-sampling compensationwtw\_\{t\}, its gradient at𝜽=𝜽old\\boldsymbol\{\\theta\}=\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}equals−∇𝜽Jτ\|𝜽old\-\\nabla\_\{\\boldsymbol\{\\theta\}\}J\_\{\\tau\}\|\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}\(Appendix[A\.1](https://arxiv.org/html/2610.22973#A1.SS1)\)\. Heresg\\operatorname\{sg\}stops gradients\. The quadratic term is simply omitted atτ=0\\tau=0; the remaining cross term follows directly from the first variation\. This is a local gradient identity, whereas Theorem[1](https://arxiv.org/html/2610.22973#Thmtheorem1)characterizes the positive\-penalty optimum\. Local regression on query\-state perturbations estimates∇Ft𝜽old,τ\\nabla F\_\{t\}^\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\},\\tau\}from sampled rewards minus future control costs \(Appendix[A\.1](https://arxiv.org/html/2610.22973#A1.SS1)\); the latter are absent whenτ=0\\tau=0\.

### 3\.2Population interpretation and directional analysis

We study population regression under the exact conditional law induced by the Gaussian noising model in Eq\. \([1](https://arxiv.org/html/2610.22973#S1.E1)\)\. LetVt​\(𝐱\)=𝔼⁡\[R⁡\(𝐙\)∣𝐗t=𝐱\]V\_\{t\}\(\\mathbf\{x\}\)=\\mathbb\{E\}\[R\(\\mathbf\{Z\}\)\\mid\\mathbf\{X\}\_\{t\}=\\mathbf\{x\}\]be its ordinary value\. At the reference policy,𝐮𝜽old=0\\mathbf\{u\}\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}=0andFt𝜽old,τ=VtF\_\{t\}^\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\},\\tau\}=V\_\{t\}for everyτ≥0\\tau\\geq 0\. Write𝐦t=𝔼⁡\[𝐙∣𝐱\]\\mathbf\{m\}\_\{t\}=\\mathbb\{E\}\[\\mathbf\{Z\}\\mid\\mathbf\{x\}\],Σt=Cov⁡\(𝐙∣𝐱\)\\Sigma\_\{t\}=\\operatorname\{Cov\}\(\\mathbf\{Z\}\\mid\\mathbf\{x\}\),𝐜t=Cov⁡\(𝐙,R∣𝐱\)\\mathbf\{c\}\_\{t\}=\\operatorname\{Cov\}\(\\mathbf\{Z\},R\\mid\\mathbf\{x\}\), andκt=αt/σt2\\kappa\_\{t\}=\\alpha\_\{t\}/\\sigma\_\{t\}^\{2\}\. We fit centered rewards to centered clean images:

𝐠t=κt​arg​min𝐮⁡𝔼​\[\(R⁡\(𝐙\)−Vt−𝐮⊤​\(𝐙−𝐦t\)\)2∣𝐗t=𝐱\],\\mathbf\{g\}\_\{t\}=\\kappa\_\{t\}\\operatorname\*\{arg\\,min\}\_\{\\mathbf\{u\}\}\\mathbb\{E\}\\\!\\left\[\\big\(R\(\\mathbf\{Z\}\)\-V\_\{t\}\-\\mathbf\{u\}^\{\\top\}\(\\mathbf\{Z\}\-\\mathbf\{m\}\_\{t\}\)\\big\)^\{2\}\\mid\\mathbf\{X\}\_\{t\}=\\mathbf\{x\}\\right\],\(7\)choosing the minimum\-norm coefficient\.

###### Proposition 2\(Population geometry of clean\-image regression\)\.

Atαt,σt\>0\\alpha\_\{t\},\\sigma\_\{t\}\>0, for Gaussian corruption with finite conditional moments and justified differentiation under the conditional integral,

∇𝐱Vt=κt​𝐜t⏟value gradient,𝐠t=κt​Σt†​𝐜t=Σt†​∇𝐱Vt⏟preconditioned direction\.\\underbrace\{\\nabla\_\{\\mathbf\{x\}\}V\_\{t\}=\\kappa\_\{t\}\\mathbf\{c\}\_\{t\}\}\_\{\\text\{value gradient\}\},\\qquad\\underbrace\{\\mathbf\{g\}\_\{t\}=\\kappa\_\{t\}\\Sigma\_\{t\}^\{\\dagger\}\\mathbf\{c\}\_\{t\}=\\Sigma\_\{t\}^\{\\dagger\}\\nabla\_\{\\mathbf\{x\}\}V\_\{t\}\}\_\{\\text\{preconditioned direction\}\}\.\(8\)No derivative ofRRis needed\. The population direction has positive inner product with∇Vt\\nabla V\_\{t\}whenever the latter is nonzero\.

For an affine reward with slope𝐰\\mathbf\{w\}on the identifiable clean subspace,𝐜t=Σt​𝐰\\mathbf\{c\}\_\{t\}=\\Sigma\_\{t\}\\mathbf\{w\}while the population regression coefficient𝐠t/κt\\mathbf\{g\}\_\{t\}/\\kappa\_\{t\}recovers𝐰\\mathbf\{w\}\. Thus regression accounts for covariance scaling of affine reward slopes on that subspace\. Forτ\>0\\tau\>0, the optimal soft value has the analogous covariance formula witheR/τe^\{R/\\tau\}; its ordinary\-to\-soft\-value residual is discussed in Appendix[C](https://arxiv.org/html/2610.22973#A3)\.

#### Directional benefit of clean\-image coordinates\.

In an isotropic Gaussian clean\-subspace model of dimensionk<dk<dwith an affine reward, consider the samem≥2m\\geq 2Gaussian perturbations and mean\-centered observations\. Withn=m−1n=m\-1independent contrasts, the unregularized minimum\-norm fits satisfy

𝔼​cos2⁡\(𝐡^noisy,∇Vt\)=min⁡\(n/d,1\),𝔼​cos2⁡\(𝐠^clean,∇Vt\)=min⁡\(n/k,1\)\.\\mathbb\{E\}\\cos^\{2\}\(\\widehat\{\\mathbf\{h\}\}\_\{\\mathrm\{noisy\}\},\\nabla V\_\{t\}\)=\\min\(n/d,1\),\\qquad\\mathbb\{E\}\\cos^\{2\}\(\\widehat\{\\mathbf\{g\}\}\_\{\\mathrm\{clean\}\},\\nabla V\_\{t\}\)=\\min\(n/k,1\)\.\(9\)In this model, clean\-image regression identifies the useful direction from fewer observations\. The subspace comparison, proved in Appendix[B\.1](https://arxiv.org/html/2610.22973#A2.SS1), motivates the choice of clean\-image coordinates; Section[6\.1](https://arxiv.org/html/2610.22973#S6.SS1)evaluates their directional alignment in the trained model\. Ridge regularization additionally bounds coefficient sensitivity even for a singular design \(Appendix[B\.2](https://arxiv.org/html/2610.22973#A2.SS2)\)\. Directly using the fitted clean\-image direction avoids an adjoint backward pass; Section[6\.2](https://arxiv.org/html/2610.22973#S6.SS2)compares it with adjoint transport\.

A supplementary idealized analysis of Langevin\-corrected sampling under additional score\-error and mixing assumptions is provided in Appendix[C](https://arxiv.org/html/2610.22973#A3)\.

## 4Reward\-adaptive rollout and regression

DiffusionRVGE constructs update directions by regressing reward differences on clean\-image contrasts\. The rollout bank determines which contrasts enter the fit, and centered ridge regression combines them into a direction for each query\. We adapt rollout bandwidth to the reward’s variation and use reward\-dependent query weights to determine where the fitted directions are applied\.

#### Reward\-dependent rollout bandwidth\.

For smooth rewards, small\-noise trajectory banks capture local reward variation by reusing denoised completions along a stochastic rollout\. For discrete or nearly piecewise\-constant rewards, independent\-seed groups provide broader contrasts across reward plateaus\. Both constructions produce a bankℬi\\mathcal\{B\}\_\{i\}of clean images and rewards for queryii\. Appendix[F](https://arxiv.org/html/2610.22973#A6)specifies the observation sets and rollout schedules\.

#### Centered clean\-image regression\.

We form clean\-image and reward contrasts using rollout\-dependent centers\. Trajectory banks use the query’s own ODE completion and its reward; independent\-seed groups use leave\-one\-out group means \(Section[5\.1](https://arxiv.org/html/2610.22973#S5.SS1)\)\. Denoting these centers by𝐦^i\\widehat\{\\mathbf\{m\}\}\_\{i\}andr¯^i\\widehat\{\\bar\{r\}\}\_\{i\}, we define

𝐝i​j=𝐳j−𝐦^i,bi​j=rj−r¯^i,j∈ℬi\.\\mathbf\{d\}\_\{ij\}=\\mathbf\{z\}\_\{j\}\-\\widehat\{\\mathbf\{m\}\}\_\{i\},\\qquad b\_\{ij\}=r\_\{j\}\-\\widehat\{\\bar\{r\}\}\_\{i\},\\qquad j\\in\\mathcal\{B\}\_\{i\}\.\(10\)For each nonzero displacement, divide the feature and label by the same lengthℓi​j=‖𝐝i​j‖\\ell\_\{ij\}=\\\|\\mathbf\{d\}\_\{ij\}\\\|\. Stack𝐝i​j⊤/ℓi​j\\mathbf\{d\}\_\{ij\}^\{\\top\}/\\ell\_\{ij\}inQiQ\_\{i\}andbi​j/ℓi​jb\_\{ij\}/\\ell\_\{ij\}in𝐛i\\mathbf\{b\}\_\{i\}\. The coefficient and its scaled regression direction are

𝐮^i=arg​min𝐮⁡\{‖Qi​𝐮−𝐛i‖2\+λsum​‖𝐮‖2\},𝐠^i,t=κt​𝐮^i,\\widehat\{\\mathbf\{u\}\}\_\{i\}=\\operatorname\*\{arg\\,min\}\_\{\\mathbf\{u\}\}\\left\\\{\\\|Q\_\{i\}\\mathbf\{u\}\-\\mathbf\{b\}\_\{i\}\\\|^\{2\}\+\\lambda\_\{\\mathrm\{sum\}\}\\\|\\mathbf\{u\}\\\|^\{2\}\\right\\\},\\qquad\\widehat\{\\mathbf\{g\}\}\_\{i,t\}=\\kappa\_\{t\}\\widehat\{\\mathbf\{u\}\}\_\{i\},\(11\)withλsum=0\.3\\lambda\_\{\\mathrm\{sum\}\}=0\.3in the reported experiments\. A small solve in the number of observations gives

𝐮^i=Qi⊤​\(Qi​Qi⊤\+λsum​I\)−1​𝐛i\.\\widehat\{\\mathbf\{u\}\}\_\{i\}=Q\_\{i\}^\{\\top\}\(Q\_\{i\}Q\_\{i\}^\{\\top\}\+\\lambda\_\{\\mathrm\{sum\}\}I\)^\{\-1\}\\mathbf\{b\}\_\{i\}\.\(12\)Centers are computed before removing zero\-displacement rows; an empty fit returns zero\. Constant rewards also give zero direction\. Normalization and ridge stabilize the finite fit; we do not normalize the fitted direction again\.

#### Reward\-dependent query weights\.

Weightsaia\_\{i\}act on the policy update after fitting\. Trajectory banks useai=1a\_\{i\}=1; the GenEval recipe selects the six lowest\-reward queries per prompt\. All other clean images, including full\-score images, remain regression observations\. Section[6\.4](https://arxiv.org/html/2610.22973#S6.SS4)compares this rule with continuous saturation weights and other selections\. Query weighting does not change the ridge coefficient\.

#### Policy update without trajectory backpropagation\.

Following Eq\. \([6](https://arxiv.org/html/2610.22973#S3.E6)\), we insert the scaled regression directionχt​𝐠^i,t\\chi\_\{t\}\\widehat\{\\mathbf\{g\}\}\_\{i,t\}into the value\-gradient term, whereχt=σt2/αt2\\chi\_\{t\}=\\sigma\_\{t\}^\{2\}/\\alpha\_\{t\}^\{2\}\. With stopped query weights and a shared normalizationZa=𝔼μ𝜽old​\[ai\]\>0Z\_\{a\}=\\mathbb\{E\}\_\{\\mu\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}\}\[a\_\{i\}\]\>0, this yields

ℒτ=1sg⁡\[Za\]​𝔼μ𝜽old​\[ai​wt​\(2​χt​⟨δ​𝐯𝜽,sg⁡\[𝐠^i,t\]⟩\+2​τνt2​‖δ​𝐯𝜽‖2\)\]\.\\mathcal\{L\}\_\{\\tau\}=\\frac\{1\}\{\\operatorname\{sg\}\[Z\_\{a\}\]\}\\mathbb\{E\}\_\{\\mu\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}\}\\\!\\left\[a\_\{i\}w\_\{t\}\\left\(2\\chi\_\{t\}\\langle\\delta\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\},\\operatorname\{sg\}\[\\widehat\{\\mathbf\{g\}\}\_\{i,t\}\]\\rangle\+\\frac\{2\\tau\}\{\\nu\_\{t\}^\{2\}\}\\\|\\delta\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\}\\\|^\{2\}\\right\)\\right\]\.\(13\)Next, center the cross term at𝐯𝜽old\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}, the frozen policy that collected the current rollout bank\. Replacing𝐯ref\\mathbf\{v\}\_\{\\mathrm\{ref\}\}bysg⁡\[𝐯𝜽old\]\\operatorname\{sg\}\[\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}\]in this term changes the loss only by a parameter\-independent quantity, so the gradient is identical for every frozen batch\. Using𝐠^i,t=κt​𝐮^i\\widehat\{\\mathbf\{g\}\}\_\{i,t\}=\\kappa\_\{t\}\\widehat\{\\mathbf\{u\}\}\_\{i\}andωt=wt​χt​κt\\omega\_\{t\}=w\_\{t\}\\chi\_\{t\}\\kappa\_\{t\}yields the equivalent centered loss

ℒτctr=1sg⁡\[Za\]𝔼μ𝜽old\[ai\(\\displaystyle\\mathcal\{L\}\_\{\\tau\}^\{\\mathrm\{ctr\}\}=\\frac\{1\}\{\\operatorname\{sg\}\[Z\_\{a\}\]\}\\mathbb\{E\}\_\{\\mu\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}\}\\\!\\Big\[a\_\{i\}\\Big\(2​ωt​⟨𝐯𝜽−sg⁡\[𝐯𝜽old\],sg⁡\[𝐮^i\]⟩\\displaystyle 2\\omega\_\{t\}\\langle\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\}\-\\operatorname\{sg\}\[\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}\],\\operatorname\{sg\}\[\\widehat\{\\mathbf\{u\}\}\_\{i\}\]\\rangle\(14\)\+2​τ​wtνt2∥𝐯𝜽−𝐯ref∥2\)\]\.\\displaystyle\+\\frac\{2\\tau w\_\{t\}\}\{\\nu\_\{t\}^\{2\}\}\\\|\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\}\-\\mathbf\{v\}\_\{\\mathrm\{ref\}\}\\\|^\{2\}\\Big\)\\Big\]\.The centered cross term is zero samplewise at𝜽=𝜽old\\boldsymbol\{\\theta\}=\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}, removing loss fluctuations caused by the fixed\-reference offset at the collection policy\. Its gradient is preserved exactly; the quadratic penalty remains measured from𝐯ref\\mathbf\{v\}\_\{\\mathrm\{ref\}\}\. Since the gradients agree samplewise, their covariance is unchanged\. Only the velocity evaluation receives gradients: rollouts, rewards, and regression are detached, so no reward derivative or adjoint through denoising is required\. Atτ=0\\tau=0the quadratic term is omitted, including for ODE rollouts; positive trajectory KL requires a nonzero diffusion coefficient\. In practice we takeτ=0\\tau=0and set the time weightωt=t\\omega\_\{t\}=t, which we found to work best, normalized to unit mean over the sampling grid; Appendix[F](https://arxiv.org/html/2610.22973#A6)gives the discrete loss\.

## 5Main experiments

### 5\.1Setup and evaluation

We fine\-tune SD3\.5\-Medium\([Esser et al\., 2024](https://arxiv.org/html/2610.22973#bib.bib5)\)at512×512512\\times 512with rank\-3232, scale\-6464LoRA\([Hu et al\., 2021](https://arxiv.org/html/2610.22973#bib.bib10)\)and ten\-step rollouts\. PickScore\([Kirstain et al\., 2023](https://arxiv.org/html/2610.22973#bib.bib11)\)uses Pick\-a\-Pic training prompts\. DiffusionNFT and FDFO are reproduced from their official code on the same8×8\\timesA800\-40GB hardware\. Compute is wall time multiplied by eight\. Each rollout collection gives one parameter update through Algorithm[1](https://arxiv.org/html/2610.22973#alg1); Appendix[F](https://arxiv.org/html/2610.22973#A6)gives the configurations\. The two\-seed PickScore comparison is reported in Appendix[E\.4](https://arxiv.org/html/2610.22973#A5.SS4)\.

Both the single\-reward and five\-reward comparisons use evaluation protocols and datasets aligned with DiffusionNFT for direct comparison: GenEval\([Ghosh et al\., 2023](https://arxiv.org/html/2610.22973#bib.bib7)\)uses553553prompts with four images each; OCR uses1,0161\{,\}016evaluated prompts from the released OCR test split with PaddleOCR scoring; and DrawBench\([Saharia et al\., 2022](https://arxiv.org/html/2610.22973#bib.bib18)\)uses200200prompts with five images each for PickScore, ClipScore, HPSv2\.1, Aesthetic, ImageReward, and UnifiedReward\. All our models are trained and evaluated with CFG=1=1\. Evaluation uses a first\-order ODE sampler with4040steps, aligned with DiffusionNFT and Flow\-GRPO\. Published results are reported as released\.

Each recipe defines the image and reward centers in Eq\. \([10](https://arxiv.org/html/2610.22973#S4.E10)\) from its rollout construction\.

#### PickScore: samples along one trajectory\.

We collect24×2424\\times 24trajectory\-bank rollouts withη=0\.03\\eta=0\.03and fit each direction from clean completions along the same stochastic trajectory\. The query state’s own ODE completion and its reward serve as the image and reward centers\. All query trajectories receive actor updates\.

#### GenEval: samples across trajectories\.

We collectK=24K=24independent ODE trajectories for each of4848prompts and select the six lowest\-reward trajectories for actor updates\. Each selected query is fitted from the other2323trajectories, whose mean clean image and mean reward serve as the regression centers\. The query itself is excluded because its own reward selected it among the lowest six\. The fitted direction is reused along its trajectory\.

#### OCR\.

OCR combines graded credit for partially correct text with discrete recognition changes, placing it between smooth PickScore and plateaued GenEval\. Both sampling schemes are effective in the ablation in Section[6\.3](https://arxiv.org/html/2610.22973#S6.SS3)\. The main OCR\-only run uses8×128\\times 12trajectory\-bank rollouts withη=0\.5\\eta=0\.5; the group variant uses lowest\-six actor selection and also supplies the OCR quality\-floor run\.

### 5\.2Joint optimization of five rewards

We train one model on the five rewards of DiffusionNFT’s joint experiment—GenEval, OCR, PickScore, ClipScore, and HPSv2\.1—and also evaluate Aesthetic, ImageReward, and UnifiedReward\. Training alternates the two sampling schemes of Algorithm[1](https://arxiv.org/html/2610.22973#alg1): group updates on mixed GenEval, OCR, and Pick\-a\-Pic prompts, and trajectory\-bank updates on Pick\-a\-Pic prompts with the composite quality rewardq=PickScore/26\+ClipScore\+HPSv2​\.1q=\\mathrm\{PickScore\}/26\+\\mathrm\{ClipScore\}\+\\mathrm\{HPSv2\.1\}\. On GenEval and OCR prompts,qqenters through the group\-relative floor of Eq\. \([43](https://arxiv.org/html/2610.22973#A8.E43)\)\. Starting from SD3\.5\-Medium, the model uses1,0001\{,\}000updates and 404 GPU\-hours \(Appendix[F\.1](https://arxiv.org/html/2610.22973#A6.SS1)\)\.

Table[1](https://arxiv.org/html/2610.22973#S5.T1)gives the result\. Our model matches DiffusionNFT’s joint model on GenEval, exceeds it on OCR, PickScore, ClipScore, HPSv2\.1, ImageReward, and UnifiedReward, and trails only on Aesthetic \(5\.955\.95vs\.6\.016\.01\)\. DiffusionNFT does not report GPU\-hours for this model; measured per\-iteration runtime of its official code on our hardware implies at least2,0002\{,\}000GPU\-hours for the1,6001\{,\}600forty\-step iterations in its described stages \(its table lists1\.71\.7k\), roughly five times our budget\.

Table 1:Five\-reward comparison on the metric panel of DiffusionNFT Table 1\. Gray marks the rewards used to train each model\. Baseline rows are published results and retain their published inference protocols;†denotes official checkpoints\. DiffusionNFT lists1\.71\.7k iterations for its five\-stage joint model, whose described stages total1,6001\{,\}600\(Appendix[F\.1](https://arxiv.org/html/2610.22973#A6.SS1)\); ours uses1,0001\{,\}000updates \(404 GPU\-hours\)\. Bold marks the best value in each column\.
### 5\.3Single\-reward results and learning efficiency

Table[2](https://arxiv.org/html/2610.22973#S5.T2)reports all eight metrics for the single\-reward checkpoints and a separate OCR quality\-floor run\. The single\-reward models reach GenEval0\.980\.98, OCR0\.980\.98, and PickScore23\.9923\.99\. The quality\-floor run uses a composite training reward and is excluded from the single\-reward speed comparison\.

Table 2:Single\-reward results on the panel of DiffusionNFT Table 2, with a separate OCR quality\-floor run\. Gray marks the primary training reward; blue marks the three floor components \(PickScore, ClipScore, HPSv2\.1\)\. Published rows retain their original protocols\. Metric precision matches Table[1](https://arxiv.org/html/2610.22973#S5.T1): two decimals for GenEval, OCR, PickScore, Aesthetic, ImageReward, and UnifiedReward; three for ClipScore and HPSv2\.1\.Figure[1](https://arxiv.org/html/2610.22973#S5.F1)and Table[3](https://arxiv.org/html/2610.22973#S5.T3)compare single\-reward training time to matched reward thresholds, with each model optimized for the reward in its row\. Clean\-image regression is2\.12\.1–2\.6×2\.6\\timesfaster than DiffusionNFT on PickScore,6\.26\.2–6\.7×6\.7\\timeson OCR, and2\.62\.6–4\.5×4\.5\\timeson GenEval\.

FDFO\([McAllister et al\., 2026](https://arxiv.org/html/2610.22973#bib.bib16)\)does not evaluate discrete rewards such as OCR and GenEval, so Figure[1](https://arxiv.org/html/2610.22973#S5.F1)compares it on PickScore only\. Our reward\-adaptive rollouts address such discrete rewards directly \(Section[4](https://arxiv.org/html/2610.22973#S4)\)\. Even on the continuous PickScore reward, FDFO needs109\.3109\.3GPU\-hours to reach23\.2023\.20, versus14\.314\.3for our method, and stays below23\.5023\.50throughout its113113\-GPU\-hour run\.

Figure 1:Single\-reward learning on8×8\\timesA800 hardware: each curve optimizes the named reward\. All curves use the aligned DrawBench200×1200\\times 1, OCR1,0161\{,\}016, and official GenEval553×4553\\times 4datasets\. FDFO appears only in the PickScore panel because FDFO does not evaluate the non\-continuous OCR and GenEval rewards\.Table 3:Measured GPU\-hours for single\-reward training to shared reward thresholds in Figure[1](https://arxiv.org/html/2610.22973#S5.F1)\.#### Image quality beyond the trained reward\.

OCR\-only training reaches its scalar reward quickly but often replaces requested scenes with large text on a flat background\. After observing this reward hacking, we ran a separate, scheduled, group\-relative quality\-floor intervention\. It reaches OCR0\.970\.97and DrawBench PickScore22\.1722\.17after150150updates \(4141GPU\-hours\) while retaining scenes in the inspected samples \(Table[2](https://arxiv.org/html/2610.22973#S5.T2)\)\. Its training reward also uses PickScore, ClipScore, and HPSv2\.1\. Appendix[H\.1](https://arxiv.org/html/2610.22973#A8.SS1)gives the reward formulation, schedule, failed alternatives, and qualitative comparisons\.

At comparable trained\-reward scores, our GenEval and PickScore checkpoints are numerically higher than the published DiffusionNFT single\-reward rows on6/76/7auxiliary metrics each \(Table[2](https://arxiv.org/html/2610.22973#S5.T2)\)\. For the OCR quality\-floor checkpoint, GenEval, Aesthetic, ImageReward, and UnifiedReward did not enter its training reward; these scores and the inspected images complement the floor\-component scores\.

## 6Ablation Study

We test directional alignment, direction transport, reward contrasts, and actor weights\. Appendix[E](https://arxiv.org/html/2610.22973#A5)gives the detailed results\.

### 6\.1Clean\-image versus noisy\-state regression: directional alignment

We compare noisy\-state and clean\-image fits using identical PickScore perturbations and reward differences, with the gradient through the numerical ODE as a reference \(Appendix[E\.3](https://arxiv.org/html/2610.22973#A5.SS3)\)\. At a200200\-update checkpoint, direct clean\-image regression raises the time\-weighted mean signed cosine from0\.0090\.009to0\.1070\.107: a paired gain of0\.0980\.098\(95% prompt\-bootstrap CI\[0\.092,0\.106\]\[0\.092,0\.106\]\)\. The initial model shows the same trend \(gain0\.2040\.204\)\.

Figure 2:PickScore cosine with the real gradient through the numerical ODE\. Left and middle: noisy\-state and direct clean\-image regression\. Right: paired clean\-image\-minus\-noisy\-state cosine\. Shading denotes 95% confidence intervals from resampling prompts\.
### 6\.2Direct clean\-image directions versus adjoint transport

Keeping the PickScore recipe fixed, we replace each direct clean\-image direction by its numerical ODE suffix pullback along the sample’s own completion \(Appendix[D](https://arxiv.org/html/2610.22973#A4)\)\. We rescale the pullback to the original direction’s norm for every sample and step, so transport changes only the direction and not the per\-step update magnitudes\. At8181GPU\-hours \(6969updates\), norm\-matched transport reaches23\.1823\.18, versus23\.6523\.65for direct transfer at equal compute and23\.4823\.48at equal updates, and each transported update costs2\.1×2\.1\\timesas much \(Figure[3](https://arxiv.org/html/2610.22973#S6.F3); Appendix[E\.2](https://arxiv.org/html/2610.22973#A5.SS2)\)\.

Figure 3:PickScore training with direct clean\-image directions or their norm\-matched Jacobian pullback to𝐱t\\mathbf\{x\}\_\{t\}, shown against compute \(left\) and updates \(right\)\. Both arms use the same trajectory bank and actor settings\.
### 6\.3Rollout bandwidth depends on reward continuity

Figure 4:Rollout comparison across three rewards\.Training curves for trajectory\-bank and independent\-seed group regression\. Batch sizes differ in GenEval and OCR, so these panels compare practical recipes at equal compute\.#### PickScore: local contrasts outperform independent\-seed rollouts\.

Holding the24×2424\\times 24batch and optimizer fixed, we replace the small\-noise bank by ODE\-group regression, train all2424query trajectories, and use batch\-standard\-deviation reward scaling for the group arm\. At2020,5656, and112112GPU\-hours, the bank scores23\.4123\.41,23\.6323\.63, and23\.7823\.78, versus22\.8922\.89,23\.2123\.21, and23\.1823\.18for the group arm\. The latter remains0\.40\.4–0\.70\.7behind at the matched budgets from2020GPU\-hours onward\. This is consistent with the local approximation in Eq\. \([24](https://arxiv.org/html/2610.22973#A2.E24)\): independent\-seed rollouts give useful diversity but much longer clean\-image differences\.

#### GenEval: increasing local noise does not close the gap\.

We compare the48×2448\\times 24group recipe with all\-state SDE banks using48×648\\times 6trajectories atη∈\{0\.03,0\.3,0\.7\}\\eta\\in\\\{0\.03,0\.3,0\.7\\\}\. At2828GPU\-hours, group regression reaches0\.920\.92, versus0\.840\.84and0\.850\.85for the two larger noise levels;η=0\.03\\eta=0\.03collapses to about0\.300\.30by update4040\. Extendingη=0\.3\\eta=0\.3to7272GPU\-hours gives0\.760\.76, versus0\.940\.94for the group recipe\. The group recipe leads at both reported budgets\.

#### OCR\.

At update4040, the trajectory bank and ODE\-group both reach0\.970\.97, using6\.86\.8and12\.312\.3GPU\-hours, respectively\. Section[5\.3](https://arxiv.org/html/2610.22973#S5.SS3.SSS0.Px1)reports the image\-quality outcomes and the quality\-floor extension\.

### 6\.4Effect of reward\-adaptive weights on training

Keeping the GenEval group regression fixed, we vary the stopped actor weights in Eq\. \([42](https://arxiv.org/html/2610.22973#A6.E42)\)\. Table[4](https://arxiv.org/html/2610.22973#S6.T4)compares the resulting training scores at equal compute\. Selecting the lowest six rewards gives the best score at2828GPU\-hours; Appendix[E\.1](https://arxiv.org/html/2610.22973#A5.SS1)provides the full curves\.

Table 4:GenEval actor\-weight ablation\. Scores are interpolated within the recorded validation curves at equal GPU\-hours and rounded to two decimals; bold marks the best unrounded value in each column\. Only the selected query trajectories receive actor gradients; all sibling clean images remain regression observations\.

## 7Conclusion

DiffusionRVGE uses centered clean\-image regression to construct useful RL directions without adjoint backpropagation\. Gradient matching identifies the update target, while population and directional analyses motivate regression in clean\-image coordinates\. Reward\-adaptive rollouts and weights handle continuous and discrete scores within one regression method\. A joint model matches or exceeds DiffusionNFT on seven of eight metrics at about one fifth of its estimated training compute; single\-reward runs reach matched thresholds2\.12\.1–6\.7×6\.7\\timesfaster on identical hardware\.

### AI use statement

Generative AI tools assisted drafting, translation, literature organization, LaTeX preparation, and plotting saved diagnostics\. All reported numbers come from recorded evaluation files\.

### Reproducibility statement

Sections[3](https://arxiv.org/html/2610.22973#S3)and[4](https://arxiv.org/html/2610.22973#S4)define the population theory and evaluated estimator\. Appendices[A](https://arxiv.org/html/2610.22973#A1)and[B](https://arxiv.org/html/2610.22973#A2)give the control and regression proofs; Appendix[C](https://arxiv.org/html/2610.22973#A3)contains the supplementary conditional sampling analysis\. Appendices[E](https://arxiv.org/html/2610.22973#A5)–[G](https://arxiv.org/html/2610.22973#A7)provide ablations, implementation details, and evaluation protocols; Appendix[H](https://arxiv.org/html/2610.22973#A8)documents reward exploitation\. Our code is available at[https://github\.com/jy\-wang\-tsinghua/DIFFUSIONRVGE](https://github.com/jy-wang-tsinghua/DIFFUSIONRVGE), and checkpoints will be released\.

## References

- Black et al\. \(2023\)Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine\.Training diffusion models with reinforcement learning\.*arXiv preprint arXiv:2305\.13301*, 2023\.URL[https://arxiv\.org/abs/2305\.13301](https://arxiv.org/abs/2305.13301)\.
- Chen et al\. \(2025\)Kaijie Chen, Zhiyang Xu, Ying Shen, Zihao Lin, Yuguang Yao, and Lifu Huang\.SuperFlow: Training flow matching models with RL on the fly\.*arXiv preprint arXiv:2512\.17951*, 2025\.URL[https://arxiv\.org/abs/2512\.17951](https://arxiv.org/abs/2512.17951)\.
- Clark et al\. \(2023\)Kevin Clark, Paul Vicol, Kevin Swersky, and David J\. Fleet\.Directly fine\-tuning diffusion models on differentiable rewards\.*arXiv preprint arXiv:2309\.17400*, 2023\.URL[https://arxiv\.org/abs/2309\.17400](https://arxiv.org/abs/2309.17400)\.
- Domingo\-Enrich et al\. \(2024\)Carles Domingo\-Enrich, Michal Drozdzal, Brian Karrer, and Ricky T\. Q\. Chen\.Adjoint matching: Fine\-tuning flow and diffusion generative models with memoryless stochastic optimal control\.*arXiv preprint arXiv:2409\.08861*, 2024\.URL[https://arxiv\.org/abs/2409\.08861](https://arxiv.org/abs/2409.08861)\.
- Esser et al\. \(2024\)Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach\.Scaling rectified flow transformers for high\-resolution image synthesis\.*arXiv preprint arXiv:2403\.03206*, 2024\.URL[https://arxiv\.org/abs/2403\.03206](https://arxiv.org/abs/2403.03206)\.
- Fan et al\. \(2023\)Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee\.DPOK: Reinforcement learning for fine\-tuning text\-to\-image diffusion models\.*arXiv preprint arXiv:2305\.16381*, 2023\.URL[https://arxiv\.org/abs/2305\.16381](https://arxiv.org/abs/2305.16381)\.
- Ghosh et al\. \(2023\)Dhruba Ghosh, Hanna Hajishirzi, and Ludwig Schmidt\.GenEval: An object\-focused framework for evaluating text\-to\-image alignment\.*arXiv preprint arXiv:2310\.11513*, 2023\.URL[https://arxiv\.org/abs/2310\.11513](https://arxiv.org/abs/2310.11513)\.
- He et al\. \(2025\)Xiaoxuan He, Siming Fu, Yuke Zhao, Wanli Li, Jian Yang, Dacheng Yin, Fengyun Rao, and Bo Zhang\.TempFlow\-GRPO: When timing matters for GRPO in flow models\.*arXiv preprint arXiv:2508\.04324*, 2025\.URL[https://arxiv\.org/abs/2508\.04324](https://arxiv.org/abs/2508.04324)\.
- Ho et al\. \(2020\)Jonathan Ho, Ajay Jain, and Pieter Abbeel\.Denoising diffusion probabilistic models\.*arXiv preprint arXiv:2006\.11239*, 2020\.URL[https://arxiv\.org/abs/2006\.11239](https://arxiv.org/abs/2006.11239)\.
- Hu et al\. \(2021\)Edward J\. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\.LoRA: Low\-rank adaptation of large language models\.*arXiv preprint arXiv:2106\.09685*, 2021\.URL[https://arxiv\.org/abs/2106\.09685](https://arxiv.org/abs/2106.09685)\.
- Kirstain et al\. \(2023\)Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy\.Pick\-a\-pic: An open dataset of user preferences for text\-to\-image generation\.*arXiv preprint arXiv:2305\.01569*, 2023\.URL[https://arxiv\.org/abs/2305\.01569](https://arxiv.org/abs/2305.01569)\.
- Lee et al\. \(2026\)Jeongjae Lee, Jinho Chang, Jeongsol Kim, and Jong Chul Ye\.Reward score matching: Unifying reward\-based fine\-tuning for flow and diffusion models\.*arXiv preprint arXiv:2604\.17415*, 2026\.URL[https://arxiv\.org/abs/2604\.17415](https://arxiv.org/abs/2604.17415)\.
- Lipman et al\. \(2023\)Yaron Lipman, Ricky T\. Q\. Chen, Heli Ben\-Hamu, Maximilian Nickel, and Matt Le\.Flow matching for generative modeling\.*arXiv preprint arXiv:2210\.02747*, 2023\.URL[https://arxiv\.org/abs/2210\.02747](https://arxiv.org/abs/2210.02747)\.
- Liu et al\. \(2025\)Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang\.Flow\-GRPO: Training flow matching models via online RL\.*arXiv preprint arXiv:2505\.05470*, 2025\.URL[https://arxiv\.org/abs/2505\.05470](https://arxiv.org/abs/2505.05470)\.
- Maheswaranathan et al\. \(2019\)Niru Maheswaranathan, Luke Metz, George Tucker, Dami Choi, and Jascha Sohl\-Dickstein\.Guided evolutionary strategies: Augmenting random search with surrogate gradients\.In*International Conference on Machine Learning*, volume 97, pp\. 4264–4273, 2019\.URL[https://proceedings\.mlr\.press/v97/maheswaranathan19a\.html](https://proceedings.mlr.press/v97/maheswaranathan19a.html)\.
- McAllister et al\. \(2026\)David McAllister, Miika Aittala, Tero Karras, Janne Hellsten, Angjoo Kanazawa, Timo Aila, and Samuli Laine\.Finite difference flow optimization for RL post\-training of text\-to\-image models\.*arXiv preprint arXiv:2603\.12893*, 2026\.URL[https://arxiv\.org/abs/2603\.12893](https://arxiv.org/abs/2603.12893)\.
- Nesterov & Spokoiny \(2017\)Yurii Nesterov and Vladimir Spokoiny\.Random gradient\-free minimization of convex functions\.*Foundations of Computational Mathematics*, 17\(2\):527–566, 2017\.doi:10\.1007/s10208\-015\-9296\-2\.URL[https://doi\.org/10\.1007/s10208\-015\-9296\-2](https://doi.org/10.1007/s10208-015-9296-2)\.
- Saharia et al\. \(2022\)Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S\. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J\. Fleet, and Mohammad Norouzi\.Photorealistic text\-to\-image diffusion models with deep language understanding\.*arXiv preprint arXiv:2205\.11487*, 2022\.URL[https://arxiv\.org/abs/2205\.11487](https://arxiv.org/abs/2205.11487)\.
- Savani et al\. \(2026\)Yash Savani, Branislav Kveton, Yuchen Liu, Yilin Wang, Jing Shi, Subhojyoti Mukherjee, Nikos Vlassis, and Krishna Kumar Singh\.Stepwise credit assignment for GRPO on flow\-matching models\.In*IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2026\.URL[https://arxiv\.org/abs/2603\.28718](https://arxiv.org/abs/2603.28718)\.
- Song et al\. \(2021\)Yang Song, Jascha Sohl\-Dickstein, Diederik P\. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole\.Score\-based generative modeling through stochastic differential equations\.*arXiv preprint arXiv:2011\.13456*, 2021\.URL[https://arxiv\.org/abs/2011\.13456](https://arxiv.org/abs/2011.13456)\.
- Wallace et al\. \(2023\)Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik\.Diffusion model alignment using direct preference optimization\.*arXiv preprint arXiv:2311\.12908*, 2023\.URL[https://arxiv\.org/abs/2311\.12908](https://arxiv.org/abs/2311.12908)\.
- Xu et al\. \(2026\)Yixian Xu, Yuanrui Zhang, Shengjie Luo, Liwei Wang, and Di He\.Designing reinforcement learning for diffusion models: A unified path\-space view\.*arXiv preprint arXiv:2608\.14430*, 2026\.URL[https://arxiv\.org/abs/2608\.14430](https://arxiv.org/abs/2608.14430)\.
- Xue et al\. \(2025\)Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, and Ping Luo\.DanceGRPO: Unleashing GRPO on visual generation\.*arXiv preprint arXiv:2505\.07818*, 2025\.URL[https://arxiv\.org/abs/2505\.07818](https://arxiv.org/abs/2505.07818)\.
- Zheng et al\. \(2026\)Kaiwen Zheng, Huayu Chen, Haotian Ye, Haoxiang Wang, Qinsheng Zhang, Kai Jiang, Hang Su, Stefano Ermon, Jun Zhu, and Ming\-Yu Liu\.DiffusionNFT: Online diffusion reinforcement with forward process\.In*International Conference on Learning Representations*, 2026\.URL[https://arxiv\.org/abs/2509\.16117](https://arxiv.org/abs/2509.16117)\.
- Zhou et al\. \(2025\)Yujie Zhou, Pengyang Ling, Jiazi Bu, Yibin Wang, Yuhang Zang, Jiaqi Wang, Li Niu, and Guangtao Zhai\.G2\\mathrm\{G\}^\{2\}RPO: Granular GRPO for precise reward in flow models\.*arXiv preprint arXiv:2510\.01982*, 2025\.URL[https://arxiv\.org/abs/2510\.01982](https://arxiv.org/abs/2510.01982)\.
- Zhu et al\. \(2026\)Yuanzhi Zhu, Xi Wang, Stéphane Lathuilière, and Vicky Kalogeiton\.Diffusion reinforcement learning via centered reward distillation\.*arXiv preprint arXiv:2603\.14128*, 2026\.URL[https://arxiv\.org/abs/2603\.14128](https://arxiv.org/abs/2603.14128)\.

## Appendix AProof of the trajectory\-KL control identity

#### Assumptions and diffusion parameterization\.

Assume well\-posed reference and controlled diffusions with the same initial law and diffusion coefficient, finite expected reward and control energy, and Girsanov integrability\. The conditional exponential moment is positive and finite; its soft value is regular enough for Itô’s formula, with a zero\-mean martingale term\. Singular endpoints are treated by limits of truncated intervals with uniform integrability of the displayed terms\.

For the Gaussian noising path, setAt=α˙t/αtA\_\{t\}=\\dot\{\\alpha\}\_\{t\}/\\alpha\_\{t\}andνt2=∂tσt2−2​At​σt2\>0\\nu\_\{t\}^\{2\}=\\partial\_\{t\}\\sigma\_\{t\}^\{2\}\-2A\_\{t\}\\sigma\_\{t\}^\{2\}\>0on the interior\. The linear forward SDEd​𝐗t=At​𝐗t​d​t\+νt​d​𝐖t\\mathrm\{d\}\\mathbf\{X\}\_\{t\}=A\_\{t\}\\mathbf\{X\}\_\{t\}\\mathrm\{d\}t\+\\nu\_\{t\}\\mathrm\{d\}\\mathbf\{W\}\_\{t\}has the specified conditional Gaussian kernels\. Its time reversal, initialized at the forward terminal law, has drift

𝐛ref=−At𝐱\+νt2∇logpt=At𝐱−2𝐯ref,𝐯ref=At𝐱−12νt2∇logpt\.\\mathbf\{b\}\_\{\\mathrm\{ref\}\}=\-A\_\{t\}\\mathbf\{x\}\+\\nu\_\{t\}^\{2\}\\nabla\\log p\_\{t\}=A\_\{t\}\\mathbf\{x\}\-2\\mathbf\{v\}\_\{\\mathrm\{ref\}\},\\qquad\\mathbf\{v\}\_\{\\mathrm\{ref\}\}=A\_\{t\}\\mathbf\{x\}\-\\tfrac\{1\}\{2\}\\nu\_\{t\}^\{2\}\\nabla\\log p\_\{t\}\.Its conditional clean law is the Gaussian noising posterior\. The parameterization𝐛𝜽=At​𝐱−2​𝐯𝜽\\mathbf\{b\}\_\{\\boldsymbol\{\\theta\}\}=A\_\{t\}\\mathbf\{x\}\-2\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\}therefore gives𝐮𝜽=−2​δ​𝐯𝜽\\mathbf\{u\}\_\{\\boldsymbol\{\\theta\}\}=\-2\\delta\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\}andγs2=νT−s2\\gamma\_\{s\}^\{2\}=\\nu\_\{T\-s\}^\{2\}\.

#### Proof of Theorem[1](https://arxiv.org/html/2610.22973#Thmtheorem1)\.

Write𝒲s=WT−s\\mathcal\{W\}\_\{s\}=W\_\{T\-s\}andℋs=HT−s\\mathcal\{H\}\_\{s\}=H\_\{T\-s\}\. The reference backward equation forℋs=e𝒲s/τ\\mathcal\{H\}\_\{s\}=e^\{\\mathcal\{W\}\_\{s\}/\\tau\}yields

\(∂s\+ℒsref\)𝒲s\+γs22​τ∥∇𝒲s∥2=0,𝒲T=R,ℒsref=𝐛ref⋅∇\+12γs2Δ\.\(\\partial\_\{s\}\+\\mathcal\{L\}\_\{s\}^\{\\mathrm\{ref\}\}\)\\mathcal\{W\}\_\{s\}\+\\frac\{\\gamma\_\{s\}^\{2\}\}\{2\\tau\}\\\|\\nabla\\mathcal\{W\}\_\{s\}\\\|^\{2\}=0,\\qquad\\mathcal\{W\}\_\{T\}=R,\\qquad\\mathcal\{L\}\_\{s\}^\{\\mathrm\{ref\}\}=\\mathbf\{b\}\_\{\\mathrm\{ref\}\}\\cdot\\nabla\+\\tfrac\{1\}\{2\}\\gamma\_\{s\}^\{2\}\\Delta\.\(15\)Girsanov’s identity and Itô’s formula under the controlled process give

KL\(P𝜽∥Pref\)\\displaystyle\\operatorname\{KL\}\(P\_\{\\boldsymbol\{\\theta\}\}\\\|P\_\{\\mathrm\{ref\}\}\)=12​𝔼P𝜽​∫0Tγs−2​‖𝐮𝜽‖2​ds,\\displaystyle=\\frac\{1\}\{2\}\\mathbb\{E\}\_\{P\_\{\\boldsymbol\{\\theta\}\}\}\\int\_\{0\}^\{T\}\\gamma\_\{s\}^\{\-2\}\\\|\\mathbf\{u\}\_\{\\boldsymbol\{\\theta\}\}\\\|^\{2\}\\mathrm\{d\}s,\(16\)𝔼P𝜽​R​\(𝐘T\)−Cτ\\displaystyle\\mathbb\{E\}\_\{P\_\{\\boldsymbol\{\\theta\}\}\}R\(\\mathbf\{Y\}\_\{T\}\)\-C\_\{\\tau\}=𝔼P𝜽∫0T\[𝐮𝜽⊤∇𝒲s−γs22​τ∥∇𝒲s∥2\]ds\.\\displaystyle=\\mathbb\{E\}\_\{P\_\{\\boldsymbol\{\\theta\}\}\}\\int\_\{0\}^\{T\}\\left\[\\mathbf\{u\}\_\{\\boldsymbol\{\\theta\}\}^\{\\top\}\\nabla\\mathcal\{W\}\_\{s\}\-\\frac\{\\gamma\_\{s\}^\{2\}\}\{2\\tau\}\\\|\\nabla\\mathcal\{W\}\_\{s\}\\\|^\{2\}\\right\]\\mathrm\{d\}s\.Subtractingτ\\tautimes the first identity from the second and completing the square proves Eq\. \([4](https://arxiv.org/html/2610.22973#S3.E4)\)\. The maximizing correction is𝐮s∗=γs2∇𝒲s/τ\\mathbf\{u\}\_\{s\}^\{\*\}=\\gamma\_\{s\}^\{2\}\\nabla\\mathcal\{W\}\_\{s\}/\\tauwhen admissible\. Substituting𝐮𝜽=−2​δ​𝐯𝜽\\mathbf\{u\}\_\{\\boldsymbol\{\\theta\}\}=\-2\\delta\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\}proves Eq\. \([17](https://arxiv.org/html/2610.22973#A1.E17)\), including its positive residual sign\. Appendix[A\.1](https://arxiv.org/html/2610.22973#A1.SS1)gives the separate first\-variation argument for the frozen\-policy training loss\.

The equivalent velocity identity is

Cτ−Jτ=2​τ​𝔼P𝜽​∫0T∥δ𝐯𝜽\+γs2∇WT−s/\(2τ\)∥2γs2​𝑑s\.C\_\{\\tau\}\-J\_\{\\tau\}=2\\tau\\mathbb\{E\}\_\{P\_\{\\boldsymbol\{\\theta\}\}\}\\int\_\{0\}^\{T\}\\frac\{\\\|\\delta\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\}\+\\gamma\_\{s\}^\{2\}\\nabla W\_\{T\-s\}/\(2\\tau\)\\\|^\{2\}\}\{\\gamma\_\{s\}^\{2\}\}\\mathrm\{d\}s\.\(17\)

#### Reward tilt and initialization\.

The path measured​Q∗/d​Pref=eR⁡\(𝐘T\)/τ/Zτ\\mathrm\{d\}Q^\{\*\}/\\mathrm\{d\}P\_\{\\mathrm\{ref\}\}=e^\{R\(\\mathbf\{Y\}\_\{T\}\)/\\tau\}/Z\_\{\\tau\}has terminal lawπ∗∝eR/τ​ρ\\pi^\{\*\}\\propto e^\{R/\\tau\}\\rho\. Substitution into relative entropy proves Eq\. \([3](https://arxiv.org/html/2610.22973#S3.E3)\)\. Its initial law isμ∗​\(d​𝐱\)=HT​\(𝐱\)​μ​\(d​𝐱\)/Zτ\\mu^\{\*\}\(\\mathrm\{d\}\\mathbf\{x\}\)=H\_\{T\}\(\\mathbf\{x\}\)\\mu\(\\mathrm\{d\}\\mathbf\{x\}\)/Z\_\{\\tau\}\. The oracle drift started fromμ∗\\mu^\{\*\}realizesQ∗Q^\{\*\}\. With an arbitrary fixed initial lawμ\\mu, the same drift instead realizes the conditional tilteR⁡\(𝐘T\)/τ/HT​\(𝐘0\)e^\{R\(\\mathbf\{Y\}\_\{T\}\)/\\tau\}/H\_\{T\}\(\\mathbf\{Y\}\_\{0\}\)\. The two coincide whenHTH\_\{T\}is constant, as at the independent\-noise entrance in the main text\.

### A\.1A common first variation for all penalty strengths

In addition to the shared initial law and policy\-independent diffusion, assume thatFt𝜽old,τF\_\{t\}^\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\},\\tau\}isC1,2C^\{1,2\}on interior intervals, the following Itô terms are integrable, and differentiation of the objective and endpoint limits are justified\. Forτ\>0\\tau\>0assume Girsanov integrability and finite expected control energy; no exponential moment or soft\-value limit is needed for the first variation atτ=0\\tau=0\.

Writeℱs=FT−s𝜽old,τ\\mathcal\{F\}\_\{s\}=F\_\{T\-s\}^\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\},\\tau\}andc𝜽,s=τ​‖𝐮𝜽‖2/\(2​γs2\)c\_\{\\boldsymbol\{\\theta\},s\}=\\tau\\\|\\mathbf\{u\}\_\{\\boldsymbol\{\\theta\}\}\\\|^\{2\}/\(2\\gamma\_\{s\}^\{2\}\), takingc𝜽,s=0c\_\{\\boldsymbol\{\\theta\},s\}=0whenτ=0\\tau=0\. Its backward equation is

\(∂s\+ℒs𝜽old\)ℱs=c𝜽old,s,ℱT=R,ℒs𝜽old=𝐛𝜽old⋅∇\+12γs2Δ\.\(\\partial\_\{s\}\+\\mathcal\{L\}\_\{s\}^\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}\)\\mathcal\{F\}\_\{s\}=c\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\},s\},\\qquad\\mathcal\{F\}\_\{T\}=R,\\qquad\\mathcal\{L\}\_\{s\}^\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}=\\mathbf\{b\}\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}\\cdot\\nabla\+\\tfrac\{1\}\{2\}\\gamma\_\{s\}^\{2\}\\Delta\.Itô’s formula underP𝜽P\_\{\\boldsymbol\{\\theta\}\}gives the exact performance difference

Jτ​\(𝜽\)−Jτ​\(𝜽old\)=𝔼P𝜽​∫0T\[⟨𝐮𝜽−𝐮𝜽old,∇ℱs⟩−c𝜽,s\+c𝜽old,s\]​𝑑s\.J\_\{\\tau\}\(\\boldsymbol\{\\theta\}\)\-J\_\{\\tau\}\(\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\)=\\mathbb\{E\}\_\{P\_\{\\boldsymbol\{\\theta\}\}\}\\int\_\{0\}^\{T\}\\\!\\left\[\\langle\\mathbf\{u\}\_\{\\boldsymbol\{\\theta\}\}\-\\mathbf\{u\}\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\},\\nabla\\mathcal\{F\}\_\{s\}\\rangle\-c\_\{\\boldsymbol\{\\theta\},s\}\+c\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\},s\}\\right\]\\mathrm\{d\}s\.\(18\)The integrand vanishes at𝜽=𝜽old\\boldsymbol\{\\theta\}=\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}, so differentiating there leaves no first\-order occupancy term\. Consequently,

∇𝜽Jτ\|𝜽old=𝔼P𝜽old​∫0T\(∂𝜽𝐮𝜽\|𝜽old\)⊤​\(∇ℱs−τγs2​𝐮𝜽old\)​𝑑s\.\\left\.\\nabla\_\{\\boldsymbol\{\\theta\}\}J\_\{\\tau\}\\right\|\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}=\\mathbb\{E\}\_\{P\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}\}\\int\_\{0\}^\{T\}\(\\partial\_\{\\boldsymbol\{\\theta\}\}\\mathbf\{u\}\_\{\\boldsymbol\{\\theta\}\}\|\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}\)^\{\\top\}\\left\(\\nabla\\mathcal\{F\}\_\{s\}\-\\frac\{\\tau\}\{\\gamma\_\{s\}^\{2\}\}\\mathbf\{u\}\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}\\right\)\\mathrm\{d\}s\.\(19\)Substituting𝐮𝜽=−2​δ​𝐯𝜽\\mathbf\{u\}\_\{\\boldsymbol\{\\theta\}\}=\-2\\delta\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\}proves the gradient identity for Eq\. \([6](https://arxiv.org/html/2610.22973#S3.E6)\), withwt=1/ϱ⁡\(t\)w\_\{t\}=1/\\varrho\(t\)whentthas sampling densityϱ\\varrho\. At zero penalty the cost terms are omitted before evaluation\. For the reward\-only ODE with generation drift−𝐯𝜽\-\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\}, the same proof uses the transport equation and gives cross\-term factor11instead of22, assuming a differentiable continuation value\. Forτ\>0\\tau\>0,Ft𝜽old,τF\_\{t\}^\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\},\\tau\}agrees with the optimal soft valueWtW\_\{t\}at the oracle policy, linking the local update to Theorem[1](https://arxiv.org/html/2610.22973#Thmtheorem1)\.

#### Regression uses the same return definition for all penalties\.

Perturb a query byh​ϵh\\boldsymbol\{\\epsilon\}with𝔼​ϵ=0\\mathbb\{E\}\\boldsymbol\{\\epsilon\}=0andM=𝔼⁡\[ϵ​ϵ⊤\]≻0M=\\mathbb\{E\}\[\\boldsymbol\{\\epsilon\}\\boldsymbol\{\\epsilon\}^\{\\top\}\]\\succ 0\. Letℛτ\\mathcal\{R\}\_\{\\tau\}be the terminal reward minus the sampled future frozen\-policy control cost in Eq\. \([5](https://arxiv.org/html/2610.22973#S3.E5)\)\. Its conditional mean isFt𝜽old,τ​\(𝐱\+h​ϵ\)F\_\{t\}^\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\},\\tau\}\(\\mathbf\{x\}\+h\\boldsymbol\{\\epsilon\}\)\. With a bounded Hessian on the perturbed states, finite second moments ofℛτ\\mathcal\{R\}\_\{\\tau\}and𝔼​‖ϵ‖3<∞\\mathbb\{E\}\\\|\\boldsymbol\{\\epsilon\}\\\|^\{3\}<\\infty, the population regression satisfies

𝐚h,τ\\displaystyle\\mathbf\{a\}\_\{h,\\tau\}=arg​min𝐚⁡𝔼​\[\(ℛτ−𝔼​ℛτ−𝐚⊤​h​ϵ\)2\]\\displaystyle=\\operatorname\*\{arg\\,min\}\_\{\\mathbf\{a\}\}\\mathbb\{E\}\\\!\\left\[\(\\mathcal\{R\}\_\{\\tau\}\-\\mathbb\{E\}\\mathcal\{R\}\_\{\\tau\}\-\\mathbf\{a\}^\{\\top\}h\\boldsymbol\{\\epsilon\}\)^\{2\}\\right\]\(20\)=1h​M−1​𝔼​\[ϵ​Ft𝜽old,τ​\(𝐱\+h​ϵ\)\]=∇Ft𝜽old,τ​\(𝐱\)\+O⁡\(h\)\.\\displaystyle=\\frac\{1\}\{h\}M^\{\-1\}\\mathbb\{E\}\[\\boldsymbol\{\\epsilon\}F\_\{t\}^\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\},\\tau\}\(\\mathbf\{x\}\+h\\boldsymbol\{\\epsilon\}\)\]=\\nabla F\_\{t\}^\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\},\\tau\}\(\\mathbf\{x\}\)\+O\(h\)\.The normal equations give the middle equality; Taylor expansion gives the last\. Thus the same regression principle applies for each fixedτ≥0\\tau\\geq 0\. At zero penalty its labels are ordinary terminal rewards\. For a fixed bank, centered feature matrix, row normalization and ridge coefficient, linearity of the fit also gives𝐮^τ=𝐮^R−τ​𝐮^C\\widehat\{\\mathbf\{u\}\}\_\{\\tau\}=\\widehat\{\\mathbf\{u\}\}\_\{R\}\-\\tau\\widehat\{\\mathbf\{u\}\}\_\{C\}, where the two coefficients fit terminal rewards and sampled future control energies\.

## Appendix BCovariance preconditioning and regression stability

#### Covariance identities and proof of Proposition[2](https://arxiv.org/html/2610.22973#Thmtheorem2)\.

For finite conditional moments and differentiation under the conditional integral, Bayes’ rule gives

qt\(d𝐳∣𝐱\)∝e−∥𝐱−αt𝐳∥2/\(2σt2\)ρ\(d𝐳\),∇𝐱logqt\(𝐳∣𝐱\)=κt\(𝐳−𝐦t\)\.q\_\{t\}\(\\mathrm\{d\}\\mathbf\{z\}\\mid\\mathbf\{x\}\)\\propto e^\{\-\\\|\\mathbf\{x\}\-\\alpha\_\{t\}\\mathbf\{z\}\\\|^\{2\}/\(2\\sigma\_\{t\}^\{2\}\)\}\\rho\(\\mathrm\{d\}\\mathbf\{z\}\),\\qquad\\nabla\_\{\\mathbf\{x\}\}\\log q\_\{t\}\(\\mathbf\{z\}\\mid\\mathbf\{x\}\)=\\kappa\_\{t\}\(\\mathbf\{z\}\-\\mathbf\{m\}\_\{t\}\)\.The derivative is of the density relative toρ\\rho, so singular clean laws are allowed\. Consequently∇𝐱𝔼​\[f⁡\(𝐙\)∣𝐱\]=κt​Cov⁡\(𝐙,f⁡\(𝐙\)∣𝐱\)\\nabla\_\{\\mathbf\{x\}\}\\mathbb\{E\}\[f\(\\mathbf\{Z\}\)\\mid\\mathbf\{x\}\]=\\kappa\_\{t\}\\operatorname\{Cov\}\(\\mathbf\{Z\},f\(\\mathbf\{Z\}\)\\mid\\mathbf\{x\}\)\. Takingf=Rf=Rproves Eq\. \([8](https://arxiv.org/html/2610.22973#S3.E8)\); takingf=eR/τf=e^\{R/\\tau\}yields

∇𝐱Wt=τ​κt​Cov⁡\(𝐙,eR/τ∣𝐱\)Ht\.\\nabla\_\{\\mathbf\{x\}\}W\_\{t\}=\\tau\\kappa\_\{t\}\\frac\{\\operatorname\{Cov\}\(\\mathbf\{Z\},e^\{R/\\tau\}\\mid\\mathbf\{x\}\)\}\{H\_\{t\}\}\.\(21\)Neither identity differentiates the reward\. The normal equations in Eq\. \([7](https://arxiv.org/html/2610.22973#S3.E7)\) giveΣt​𝐠t=κt​𝐜t\\Sigma\_\{t\}\\mathbf\{g\}\_\{t\}=\\kappa\_\{t\}\\mathbf\{c\}\_\{t\}\. If𝐚∈ker⁡Σt\\mathbf\{a\}\\in\\ker\\Sigma\_\{t\}, then𝐚⊤​\(𝐙−𝐦t\)=0\\mathbf\{a\}^\{\\top\}\(\\mathbf\{Z\}\-\\mathbf\{m\}\_\{t\}\)=0almost surely, so𝐚⊤​𝐜t=0\\mathbf\{a\}^\{\\top\}\\mathbf\{c\}\_\{t\}=0\. Thus𝐜t∈Range⁡\(Σt\)\\mathbf\{c\}\_\{t\}\\in\\operatorname\{Range\}\(\\Sigma\_\{t\}\)and the minimum\-norm solution is𝐠t=κt​Σt†​𝐜t\\mathbf\{g\}\_\{t\}=\\kappa\_\{t\}\\Sigma\_\{t\}^\{\\dagger\}\\mathbf\{c\}\_\{t\}\. Its inner product with∇Vt\\nabla V\_\{t\}is\(∇Vt\)⊤Σt†∇Vt\>0\(\\nabla V\_\{t\}\)^\{\\top\}\\Sigma\_\{t\}^\{\\dagger\}\\nabla V\_\{t\}\>0for nonzero∇Vt\\nabla V\_\{t\}\.

For an affine reward with slope𝐰∈Range⁡\(Σt\)\\mathbf\{w\}\\in\\operatorname\{Range\}\(\\Sigma\_\{t\}\),𝐜t=Σt​𝐰\\mathbf\{c\}\_\{t\}=\\Sigma\_\{t\}\\mathbf\{w\}\. The maps𝐰↦∇Vt\\mathbf\{w\}\\mapsto\\nabla V\_\{t\}and𝐰↦𝐠t\\mathbf\{w\}\\mapsto\\mathbf\{g\}\_\{t\}therefore have condition numbersκ\+​\(Σt\)\\kappa\_\{\+\}\(\\Sigma\_\{t\}\)and11on that subspace, whereκ\+\\kappa\_\{\+\}uses the nonzero eigenvalues\.

### B\.1Directional advantage with mean\-centered observations

LetP=U​U⊤P=UU^\{\\top\}withU⊤​U=IkU^\{\\top\}U=I\_\{k\},1≤k<d1\\leq k<d, and𝐙∼𝒩⁡\(0,σdata2​P\)\\mathbf\{Z\}\\sim\\mathcal\{N\}\(0,\\sigma\_\{\\mathrm\{data\}\}^\{2\}P\)\. ForR⁡\(𝐳\)=b\+𝐰⊤​𝐳R\(\\mathbf\{z\}\)=b\+\\mathbf\{w\}^\{\\top\}\\mathbf\{z\},P​𝐰=𝐰≠0P\\mathbf\{w\}=\\mathbf\{w\}\\neq 0, Gaussian conditioning gives

𝐦t​\(𝐱\)=at​P​𝐱,Σt=λt​P,at=αt​σdata2αt2​σdata2\+σt2,λt=σdata2​σt2αt2​σdata2\+σt2\.\\mathbf\{m\}\_\{t\}\(\\mathbf\{x\}\)=a\_\{t\}P\\mathbf\{x\},\\quad\\Sigma\_\{t\}=\\lambda\_\{t\}P,\\quad a\_\{t\}=\\frac\{\\alpha\_\{t\}\\sigma\_\{\\mathrm\{data\}\}^\{2\}\}\{\\alpha\_\{t\}^\{2\}\\sigma\_\{\\mathrm\{data\}\}^\{2\}\+\\sigma\_\{t\}^\{2\}\},\\quad\\lambda\_\{t\}=\\frac\{\\sigma\_\{\\mathrm\{data\}\}^\{2\}\\sigma\_\{t\}^\{2\}\}\{\\alpha\_\{t\}^\{2\}\\sigma\_\{\\mathrm\{data\}\}^\{2\}\+\\sigma\_\{t\}^\{2\}\}\.\(22\)SinceWt=Vt\+λt​‖𝐰‖2/\(2​τ\)W\_\{t\}=V\_\{t\}\+\\lambda\_\{t\}\\\|\\mathbf\{w\}\\\|^\{2\}/\(2\\tau\), both gradients equalat​𝐰a\_\{t\}\\mathbf\{w\}, and𝐠t=κt​𝐰\\mathbf\{g\}\_\{t\}=\\kappa\_\{t\}\\mathbf\{w\}\. Drawm≥2m\\geq 2independent perturbations𝜹j∼𝒩⁡\(0,η2​Id\)\\boldsymbol\{\\delta\}\_\{j\}\\sim\\mathcal\{N\}\(0,\\eta^\{2\}I\_\{d\}\)and form a matrixDDwith rows𝜹j⊤\\boldsymbol\{\\delta\}\_\{j\}^\{\\top\}\. WriteC=Im−𝟏𝟏⊤/mC=I\_\{m\}\-\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}/mfor the centering matrix\. The centered noisy rows, centered denoised means, and centered labels are

Dc=C​D,Qc=at​Dc​P,𝐛c=at​Dc​𝐰=Qc​𝐰\.D\_\{c\}=CD,\\qquad Q\_\{c\}=a\_\{t\}D\_\{c\}P,\\qquad\\mathbf\{b\}\_\{c\}=a\_\{t\}D\_\{c\}\\mathbf\{w\}=Q\_\{c\}\\mathbf\{w\}\.The unnormalized minimum\-norm fits are

𝐡^noisy=Dc†​𝐛c=at​ΠS​𝐰,𝐠^clean=κt​Qc†​𝐛c=κt​ΠL​𝐰,\\widehat\{\\mathbf\{h\}\}\_\{\\mathrm\{noisy\}\}=D\_\{c\}^\{\\dagger\}\\mathbf\{b\}\_\{c\}=a\_\{t\}\\Pi\_\{S\}\\mathbf\{w\},\\qquad\\widehat\{\\mathbf\{g\}\}\_\{\\mathrm\{clean\}\}=\\kappa\_\{t\}Q\_\{c\}^\{\\dagger\}\\mathbf\{b\}\_\{c\}=\\kappa\_\{t\}\\Pi\_\{L\}\\mathbf\{w\},whereS=row⁡\(Dc\)S=\\operatorname\{row\}\(D\_\{c\}\)andL=row⁡\(Dc​P\)L=\\operatorname\{row\}\(D\_\{c\}P\)\. IfB∈ℝm×\(m−1\)B\\in\\mathbb\{R\}^\{m\\times\(m\-1\)\}is an orthonormal basis of𝟏⟂\\mathbf\{1\}^\{\\perp\}, thenC=B​B⊤C=BB^\{\\top\}andB⊤​DB^\{\\top\}Dhasm−1m\-1independent isotropic Gaussian rows\. HenceSShas the same distribution as the row space of an\(m−1\)×d\(m\-1\)\\times dGaussian matrix\. Rotational invariance gives

𝔼​ΠS=min⁡\(\(m−1\)/d,1\)​Id,𝔼​ΠL=min⁡\(\(m−1\)/k,1\)​P\.\\mathbb\{E\}\\Pi\_\{S\}=\\min\(\(m\-1\)/d,1\)I\_\{d\},\\qquad\\mathbb\{E\}\\Pi\_\{L\}=\\min\(\(m\-1\)/k,1\)P\.For either nonzero projection,cos2⁡\(ΠS​𝐰,𝐰\)=‖ΠS​𝐰‖2/‖𝐰‖2\\cos^\{2\}\(\\Pi\_\{S\}\\mathbf\{w\},\\mathbf\{w\}\)=\\\|\\Pi\_\{S\}\\mathbf\{w\}\\\|^\{2\}/\\\|\\mathbf\{w\}\\\|^\{2\}\. Taking expectations proves Eq\. \([9](https://arxiv.org/html/2610.22973#S3.E9)\), including positive alignment almost surely\.

### B\.2Ridge controls coefficient sensitivity

For the actual sum\-loss convention in Eq\. \([11](https://arxiv.org/html/2610.22973#S4.E11)\), letQ=U​diag⁡\(sj\)​V⊤Q=U\\operatorname\{diag\}\(s\_\{j\}\)V^\{\\top\}be a singular value decomposition\. Then

𝐮^=V​diag⁡\(sjsj2\+λsum\)​U⊤​𝐛,‖𝐮^‖≤‖𝐛‖2​λsum\.\\widehat\{\\mathbf\{u\}\}=V\\operatorname\{diag\}\\\!\\left\(\\frac\{s\_\{j\}\}\{s\_\{j\}^\{2\}\+\\lambda\_\{\\mathrm\{sum\}\}\}\\right\)U^\{\\top\}\\mathbf\{b\},\\qquad\\\|\\widehat\{\\mathbf\{u\}\}\\\|\\leq\\frac\{\\\|\\mathbf\{b\}\\\|\}\{2\\sqrt\{\\lambda\_\{\\mathrm\{sum\}\}\}\}\.\(23\)The bound follows froms/\(s2\+λ\)≤1/\(2​λ\)s/\(s^\{2\}\+\\lambda\)\\leq 1/\(2\\sqrt\{\\lambda\}\)\. For fixedQQ, perturbing the normalized labels byΔ​𝐛\\Delta\\mathbf\{b\}changes the coefficient by at most‖Δ​𝐛‖/\(2​λsum\)\\\|\\Delta\\mathbf\{b\}\\\|/\(2\\sqrt\{\\lambda\_\{\\mathrm\{sum\}\}\}\)\. No rank assumption is required; this includes the linear dependence introduced by mean centering\.

#### Local approximation and rollout radius\.

ForRRwith Hessian norm at mostLLon a convex neighborhood containing the bank, set𝐝j=𝐳j−𝐳¯\\mathbf\{d\}\_\{j\}=\\mathbf\{z\}\_\{j\}\-\\bar\{\\mathbf\{z\}\}and𝐰=∇R​\(𝐳¯\)\\mathbf\{w\}=\\nabla R\(\\bar\{\\mathbf\{z\}\}\)\. Taylor’s theorem and∑j𝐝j=0\\sum\_\{j\}\\mathbf\{d\}\_\{j\}=0give

rj−r¯=𝐰⊤​𝐝j\+ej−e¯,\|ej−e¯\|≤L2​\(‖𝐝j‖2\+1m​∑k‖𝐝k‖2\)\.r\_\{j\}\-\\bar\{r\}=\\mathbf\{w\}^\{\\top\}\\mathbf\{d\}\_\{j\}\+e\_\{j\}\-\\bar\{e\},\\qquad\|e\_\{j\}\-\\bar\{e\}\|\\leq\\frac\{L\}\{2\}\\left\(\\\|\\mathbf\{d\}\_\{j\}\\\|^\{2\}\+\\frac\{1\}\{m\}\\sum\_\{k\}\\\|\\mathbf\{d\}\_\{k\}\\\|^\{2\}\\right\)\.\(24\)This explains why smaller clean\-image displacements can improve a local fit for smooth rewards\.

## Appendix CSupplementary analysis of Langevin\-corrected sampling

This appendix provides a conditional consistency result for an idealized sampling dynamics atτ\>0\\tau\>0\. It assumes exact reference scores, ideal realization of the regression field, a vanishing score discrepancy, and a mixing inequality\. Under these assumptions, the corrected dynamics tracks the reward\-tilted target\.

#### Idealized score approximation\.

Forτ\>0\\tau\>0, to examine the structural effect of preconditioning, useχt=σt2/αt2\\chi\_\{t\}=\\sigma\_\{t\}^\{2\}/\\alpha\_\{t\}^\{2\}and replace∇Wt\\nabla W\_\{t\}byχt​𝐠t\\chi\_\{t\}\\mathbf\{g\}\_\{t\}\. In normalized coordinates𝐲=𝐱/αt\\mathbf\{y\}=\\mathbf\{x\}/\\alpha\_\{t\},χ=e−2​ℓ\\chi=e^\{\-2\\ell\}, define𝐆ℓ=αt​χt​𝐠t=Σℓ†​𝐜ℓ\\mathbf\{G\}\_\{\\ell\}=\\alpha\_\{t\}\\chi\_\{t\}\\mathbf\{g\}\_\{t\}=\\Sigma\_\{\\ell\}^\{\\dagger\}\\mathbf\{c\}\_\{\\ell\}\. Letpℓ=ρ∗𝒩⁡\(0,χ​I\)p\_\{\\ell\}=\\rho\*\\mathcal\{N\}\(0,\\chi I\)andπℓ∗=π∗∗𝒩⁡\(0,χ​I\)\\pi\_\{\\ell\}^\{\*\}=\\pi^\{\*\}\*\\mathcal\{N\}\(0,\\chi I\)\. The regression score is∇log⁡pℓ\+𝐆ℓ/τ\\nabla\\log p\_\{\\ell\}\+\\mathbf\{G\}\_\{\\ell\}/\\tau, with error

𝐝ℓ=𝐆ℓ−∇𝐲Wℓτ=\(I−Σℓ/χ\)​𝐆ℓ\+∇𝐲Vℓ−∇𝐲Wℓτ\.\\mathbf\{d\}\_\{\\ell\}=\\frac\{\\mathbf\{G\}\_\{\\ell\}\-\\nabla\_\{\\mathbf\{y\}\}W\_\{\\ell\}\}\{\\tau\}=\\frac\{\(I\-\\Sigma\_\{\\ell\}/\\chi\)\\mathbf\{G\}\_\{\\ell\}\+\\nabla\_\{\\mathbf\{y\}\}V\_\{\\ell\}\-\\nabla\_\{\\mathbf\{y\}\}W\_\{\\ell\}\}\{\\tau\}\.\(25\)The two terms isolate covariance geometry and the ordinary\-to\-soft\-value approximation\. The following conditional result states when this preconditioning preserves the terminal target\.

###### Theorem 3\(Conditional consistency of idealized corrected sampling\)\.

Assume an exact reference score, ideal realization of the regression field, and the regularity conditions stated below\. For a fixeda0\>0a\_\{0\}\>0, let

d​𝐘ℓ=\(χ\+a0\)​\(∇log⁡pℓ\+𝐆ℓτ\)​\(𝐘ℓ\)​d​ℓ\+2​a0​d​𝐁ℓ\.\\mathrm\{d\}\\mathbf\{Y\}\_\{\\ell\}=\(\\chi\+a\_\{0\}\)\\left\(\\nabla\\log p\_\{\\ell\}\+\\frac\{\\mathbf\{G\}\_\{\\ell\}\}\{\\tau\}\\right\)\(\\mathbf\{Y\}\_\{\\ell\}\)\\mathrm\{d\}\\ell\+\\sqrt\{2a\_\{0\}\}\\mathrm\{d\}\\mathbf\{B\}\_\{\\ell\}\.\(26\)Ifπℓ∗\\pi\_\{\\ell\}^\{\*\}satisfies a uniform log\-Sobolev inequality, the initial relative entropy is finite, and𝔼​‖𝐝ℓ​\(𝐘ℓ\)‖2→0\\mathbb\{E\}\\\|\\mathbf\{d\}\_\{\\ell\}\(\\mathbf\{Y\}\_\{\\ell\}\)\\\|^\{2\}\\to 0, then

KL\(μℓ∥πℓ∗\)→0,μℓ⇒π∗\(ℓ→∞\)\.\\operatorname\{KL\}\(\\mu\_\{\\ell\}\\\|\\pi\_\{\\ell\}^\{\*\}\)\\to 0,\\qquad\\mu\_\{\\ell\}\\Rightarrow\\pi^\{\*\}\\quad\(\\ell\\to\\infty\)\.\(27\)

#### Ideal control realization\.

The comparison isolates the regression approximation by assuming exact reference scores and

𝐯𝜽−𝐯ref=−νt2​χt2​τ​𝐠t,𝐬t𝜽=𝐬tref\+χt​𝐠tτ,𝐮~t=νt2​χtτ​𝐠t\.\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\}\-\\mathbf\{v\}\_\{\\mathrm\{ref\}\}=\-\\frac\{\\nu\_\{t\}^\{2\}\\chi\_\{t\}\}\{2\\tau\}\\mathbf\{g\}\_\{t\},\\qquad\\mathbf\{s\}\_\{t\}^\{\\boldsymbol\{\\theta\}\}=\\mathbf\{s\}\_\{t\}^\{\\mathrm\{ref\}\}\+\\frac\{\\chi\_\{t\}\\mathbf\{g\}\_\{t\}\}\{\\tau\},\\qquad\\widetilde\{\\mathbf\{u\}\}\_\{t\}=\\frac\{\\nu\_\{t\}^\{2\}\\chi\_\{t\}\}\{\\tau\}\\mathbf\{g\}\_\{t\}\.\(28\)Here𝐬tref=∇𝐱​log​pt\\mathbf\{s\}\_\{t\}^\{\\mathrm\{ref\}\}=\\nabla\_\{\\mathbf\{x\}\}\\log p\_\{t\}and𝐮t∗=νt2​∇𝐱Wt/τ\\mathbf\{u\}\_\{t\}^\{\*\}=\\nu\_\{t\}^\{2\}\\nabla\_\{\\mathbf\{x\}\}W\_\{t\}/\\tau\. In normalized coordinates𝐲=𝐱/αt\\mathbf\{y\}=\\mathbf\{x\}/\\alpha\_\{t\}, writePχP\_\{\\chi\}for Gaussian convolution of varianceχ\\chi\. ReindexVV,WW,Σ\\Sigmaand𝐜\\mathbf\{c\}at\(αt​𝐲,t\)\(\\alpha\_\{t\}\\mathbf\{y\},t\), and write𝐤ℓ=∇𝐲Wℓ/τ=∇log⁡πℓ∗−∇log⁡pℓ\\mathbf\{k\}\_\{\\ell\}=\\nabla\_\{\\mathbf\{y\}\}W\_\{\\ell\}/\\tau=\\nabla\\log\\pi\_\{\\ell\}^\{\*\}\-\\nabla\\log p\_\{\\ell\}\.

#### Scale and oracle control\.

Letχt=σt2/αt2\\chi\_\{t\}=\\sigma\_\{t\}^\{2\}/\\alpha\_\{t\}^\{2\},𝐲=𝐱/αt\\mathbf\{y\}=\\mathbf\{x\}/\\alpha\_\{t\}, andχ=e−2​ℓ\\chi=e^\{\-2\\ell\}\. Sinceαt​χt​κt=1\\alpha\_\{t\}\\chi\_\{t\}\\kappa\_\{t\}=1, the normalized regression field is𝐆ℓ=αt​χt​𝐠t=Σℓ†​𝐜ℓ\\mathbf\{G\}\_\{\\ell\}=\\alpha\_\{t\}\\chi\_\{t\}\\mathbf\{g\}\_\{t\}=\\Sigma\_\{\\ell\}^\{\\dagger\}\\mathbf\{c\}\_\{\\ell\}\. The control targets satisfy the exact identity

𝐮~t=χt​Σt†​𝐮t∗\+νt2​χtτ​Σt†​\(∇𝐱Vt−∇𝐱Wt\)\.\\widetilde\{\\mathbf\{u\}\}\_\{t\}=\\chi\_\{t\}\\Sigma\_\{t\}^\{\\dagger\}\\mathbf\{u\}\_\{t\}^\{\*\}\+\\frac\{\\nu\_\{t\}^\{2\}\\chi\_\{t\}\}\{\\tau\}\\Sigma\_\{t\}^\{\\dagger\}\(\\nabla\_\{\\mathbf\{x\}\}V\_\{t\}\-\\nabla\_\{\\mathbf\{x\}\}W\_\{t\}\)\.\(29\)Thus scale\-corrected preconditioning retains the ordinary/soft\-value residual\. Differentiating the conditional exponential moment gives

𝐤ℓ=Cov⁡\(𝐙,eR⁡\(𝐙\)/τ∣𝐲\)χ​𝔼​\[eR⁡\(𝐙\)/τ∣𝐲\],∇𝐲Vℓ=χ−1​Σℓ​𝐆ℓ\.\\mathbf\{k\}\_\{\\ell\}=\\frac\{\\operatorname\{Cov\}\(\\mathbf\{Z\},e^\{R\(\\mathbf\{Z\}\)/\\tau\}\\mid\\mathbf\{y\}\)\}\{\\chi\\,\\mathbb\{E\}\[e^\{R\(\\mathbf\{Z\}\)/\\tau\}\\mid\\mathbf\{y\}\]\},\\qquad\\nabla\_\{\\mathbf\{y\}\}V\_\{\\ell\}=\\chi^\{\-1\}\\Sigma\_\{\\ell\}\\mathbf\{G\}\_\{\\ell\}\.\(30\)These identities yield the two\-term discrepancy in Eq\. \([25](https://arxiv.org/html/2610.22973#A3.E25)\); neither residual is discarded\.

#### Uniform endpoint assumptions\.

On a stable covariance support with orthogonal projectorPP, assume, uniformly in𝐲\\mathbf\{y\}asχ↓0\\chi\\downarrow 0,

Range\(Σ\)=Range\(P\),Σ=χP\+O\(χ2\),𝐜=χ𝐰\+O\(χ2\),P𝐰=𝐰,Cov⁡\(𝐙,eR⁡\(𝐙\)/τ∣𝐲\)𝔼⁡\[eR⁡\(𝐙\)/τ∣𝐲\]=τ−1χ𝐰\+O\(χ2\),sup𝐲,ℓ∥𝐰∥<∞\.\\begin\{gathered\}\\operatorname\{Range\}\(\\Sigma\)=\\operatorname\{Range\}\(P\),\\qquad\\Sigma=\\chi P\+O\(\\chi^\{2\}\),\\qquad\\mathbf\{c\}=\\chi\\mathbf\{w\}\+O\(\\chi^\{2\}\),\\qquad P\\mathbf\{w\}=\\mathbf\{w\},\\\\ \\frac\{\\operatorname\{Cov\}\(\\mathbf\{Z\},e^\{R\(\\mathbf\{Z\}\)/\\tau\}\\mid\\mathbf\{y\}\)\}\{\\mathbb\{E\}\[e^\{R\(\\mathbf\{Z\}\)/\\tau\}\\mid\\mathbf\{y\}\]\}=\\tau^\{\-1\}\\chi\\mathbf\{w\}\+O\(\\chi^\{2\}\),\\qquad\\sup\_\{\\mathbf\{y\},\\ell\}\\\|\\mathbf\{w\}\\\|<\\infty\.\\end\{gathered\}\(31\)The support has fixed rank and the remainders are uniform in vector and operator norms\. These assumptions on the conditional covariance suffice for the mean\-square error criterion in Theorem[3](https://arxiv.org/html/2610.22973#Thmtheorem3); the noisy lawspℓ=Pχ​ρp\_\{\\ell\}=P\_\{\\chi\}\\rhoandπℓ∗=Pχ​π∗\\pi\_\{\\ell\}^\{\*\}=P\_\{\\chi\}\\pi^\{\*\}have ambient densities for everyχ\>0\\chi\>0, even whenρ\\rhois singular\. In the Gaussian\-subspace model of Eq\. \([22](https://arxiv.org/html/2610.22973#A2.E22)\), the covariance condition holds explicitly:

Σℓ=χ1\+χ/σdata2​P=χ​P−χ2σdata2​P\+O⁡\(χ3\),χ​Σℓ†=\(1\+χ/σdata2\)​P\.\\Sigma\_\{\\ell\}=\\frac\{\\chi\}\{1\+\\chi/\\sigma\_\{\\mathrm\{data\}\}^\{2\}\}P=\\chi P\-\\frac\{\\chi^\{2\}\}\{\\sigma\_\{\\mathrm\{data\}\}^\{2\}\}P\+O\(\\chi^\{3\}\),\\qquad\\chi\\Sigma\_\{\\ell\}^\{\\dagger\}=\(1\+\\chi/\\sigma\_\{\\mathrm\{data\}\}^\{2\}\)P\.On this support,λmin\+​\(Σ\)≥χ/2\\lambda\_\{\\min\}^\{\+\}\(\\Sigma\)\\geq\\chi/2for sufficiently smallχ\\chi\. Hence

𝐆ℓ−𝐰=Σ†​\(𝐜−Σ​𝐰\)=O⁡\(χ\),𝐤ℓ=τ−1​𝐰\+O⁡\(χ\),𝐝ℓ=O⁡\(χ\)\.\\mathbf\{G\}\_\{\\ell\}\-\\mathbf\{w\}=\\Sigma^\{\\dagger\}\(\\mathbf\{c\}\-\\Sigma\\mathbf\{w\}\)=O\(\\chi\),\\qquad\\mathbf\{k\}\_\{\\ell\}=\\tau^\{\-1\}\\mathbf\{w\}\+O\(\\chi\),\\qquad\\mathbf\{d\}\_\{\\ell\}=O\(\\chi\)\.This proves the scale\-consistency assertion\. In the original coordinates,

τνt2​‖𝐮~t−𝐮t∗‖=‖χt​𝐠t−∇𝐱Wt‖=ταt\|𝐝ℓ\|=O⁡\(χt\)\.\\frac\{\\tau\}\{\\nu\_\{t\}^\{2\}\}\\\|\\widetilde\{\\mathbf\{u\}\}\_\{t\}\-\\mathbf\{u\}\_\{t\}^\{\*\}\\\|=\\\|\\chi\_\{t\}\\mathbf\{g\}\_\{t\}\-\\nabla\_\{\\mathbf\{x\}\}W\_\{t\}\\\|=\\frac\{\\tau\}\{\\alpha\_\{t\}\}\\\|\\mathbf\{d\}\_\{\\ell\}\\\|=O\(\\chi\_\{t\}\)\.An unnormalized control\-drift bound additionally depends on the scheduleνt2\\nu\_\{t\}^\{2\}\.

#### Scope and regularity of the conditional guarantee\.

Assumeτ\>0\\tau\>0,0<Zτ=∫eR/τ​𝑑ρ<∞0<Z\_\{\\tau\}=\\int e^\{R/\\tau\}\\mathrm\{d\}\\rho<\\infty, well\-posed corrected dynamics, positive smooth noisy densities, and finite initial relative entropy at a finiteℓ0\\ell\_\{0\}\. Assume the differentiations and integrations by parts below are justified, and the score\-error energy is locally integrable\. Reference scores and the displayed guidance field are realized exactly\. Forℋℓ=KL\(μℓ∥πℓ∗\)\\mathcal\{H\}\_\{\\ell\}=\\operatorname\{KL\}\(\\mu\_\{\\ell\}\\\|\\pi\_\{\\ell\}^\{\*\}\)andℐℓ=𝔼μℓ​‖∇log⁡\(μℓ/πℓ∗\)‖2\\mathcal\{I\}\_\{\\ell\}=\\mathbb\{E\}\_\{\\mu\_\{\\ell\}\}\\\|\\nabla\\log\(\\mu\_\{\\ell\}/\\pi\_\{\\ell\}^\{\*\}\)\\\|^\{2\}, assumeℐℓ≥2​κ​ℋℓ\\mathcal\{I\}\_\{\\ell\}\\geq 2\\kappa\\mathcal\{H\}\_\{\\ell\}for someκ\>0\\kappa\>0independent ofℓ≥ℓ0\\ell\\geq\\ell\_\{0\}\.

#### Entropy proof\.

Gaussian convolution andχ=e−2​ℓ\\chi=e^\{\-2\\ell\}give∂ℓπℓ∗=−χ​Δ​πℓ∗\\partial\_\{\\ell\}\\pi\_\{\\ell\}^\{\*\}=\-\\chi\\Delta\\pi\_\{\\ell\}^\{\*\}\. Hence the oracle drift\(χ\+a0\)∇logπℓ∗\(\\chi\+a\_\{0\}\)\\nabla\\log\\pi\_\{\\ell\}^\{\*\}with diffusion2​a0​I\\sqrt\{2a\_\{0\}\}Iexactly transportsπℓ∗\\pi\_\{\\ell\}^\{\*\}\. The approximate drift differs by𝐃ℓ=\(χ\+a0\)​𝐝ℓ\\mathbf\{D\}\_\{\\ell\}=\(\\chi\+a\_\{0\}\)\\mathbf\{d\}\_\{\\ell\}\. Writingfℓ=log⁡\(μℓ/πℓ∗\)f\_\{\\ell\}=\\log\(\\mu\_\{\\ell\}/\\pi\_\{\\ell\}^\{\*\}\), the relative\-entropy identity and Young’s inequality imply

ℋℓ′=−a0​ℐℓ\+𝔼μℓ​⟨𝐃ℓ,∇fℓ⟩≤−a0​κ​ℋℓ\+εℓ,εℓ=\(χ\+a0\)22​a0​𝔼μℓ​‖𝐝ℓ‖2\.\\mathcal\{H\}\_\{\\ell\}^\{\\prime\}=\-a\_\{0\}\\mathcal\{I\}\_\{\\ell\}\+\\mathbb\{E\}\_\{\\mu\_\{\\ell\}\}\\langle\\mathbf\{D\}\_\{\\ell\},\\nabla f\_\{\\ell\}\\rangle\\leq\-a\_\{0\}\\kappa\\mathcal\{H\}\_\{\\ell\}\+\\varepsilon\_\{\\ell\},\\qquad\\varepsilon\_\{\\ell\}=\\frac\{\(\\chi\+a\_\{0\}\)^\{2\}\}\{2a\_\{0\}\}\\mathbb\{E\}\_\{\\mu\_\{\\ell\}\}\\\|\\mathbf\{d\}\_\{\\ell\}\\\|^\{2\}\.\(32\)Consequently,

ℋℓ≤e−a0​κ​\(ℓ−ℓ0\)​ℋℓ0\+∫ℓ0ℓe−a0​κ​\(ℓ−r\)​εr​𝑑r\.\\mathcal\{H\}\_\{\\ell\}\\leq e^\{\-a\_\{0\}\\kappa\(\\ell\-\\ell\_\{0\}\)\}\\mathcal\{H\}\_\{\\ell\_\{0\}\}\+\\int\_\{\\ell\_\{0\}\}^\{\\ell\}e^\{\-a\_\{0\}\\kappa\(\\ell\-r\)\}\\varepsilon\_\{r\}\\mathrm\{d\}r\.\(33\)Sinceεℓ→0\\varepsilon\_\{\\ell\}\\to 0, split the integral at a sufficiently large fixed time to see that both terms vanish\. This proves relative\-entropy tracking\. Pinsker’s inequality andπℓ∗⇒π∗\\pi\_\{\\ell\}^\{\*\}\\Rightarrow\\pi^\{\*\}then give the weak endpoint limit\. For a singular clean target, the endpoint conclusion is weak convergence, not total\-variation convergence to that singular measure\. More generally, the same proof needs only that the exponentially weighted error integral in Eq\. \([33](https://arxiv.org/html/2610.22973#A3.E33)\) tends to zero\.

#### Discrete rewards and a smooth reference\.

For bounded measurableRR, the positive\-noise covariance identity requires no reward derivative, and the convergence theorem applies to an explicitly specified bounded smooth referenceRεR\_\{\\varepsilon\}at fixedε\>0\\varepsilon\>0, whenever its assumptions hold\. IfRε→RR\_\{\\varepsilon\}\\to RinL1​\(ρ\)L^\{1\}\(\\rho\)with a common bound, then

π∗,ε​\(d​𝐳\)=eRε​\(𝐳\)/τ​ρ​\(d​𝐳\)∫eRε/τ​𝑑ρ⟶π∗​\(d​𝐳\)in total variation,\\pi^\{\*,\\varepsilon\}\(\\mathrm\{d\}\\mathbf\{z\}\)=\\frac\{e^\{R\_\{\\varepsilon\}\(\\mathbf\{z\}\)/\\tau\}\\rho\(\\mathrm\{d\}\\mathbf\{z\}\)\}\{\\int e^\{R\_\{\\varepsilon\}/\\tau\}\\mathrm\{d\}\\rho\}\\longrightarrow\\pi^\{\*\}\(\\mathrm\{d\}\\mathbf\{z\}\)\\quad\\text\{in total variation\},because the exponential is Lipschitz on the bounded reward range and the normalizers converge\.

## Appendix DNumerical adjoint used in the transport ablation

LetFtF\_\{t\}be a deterministic ODE completion with differentiable velocity\. Transporting a fitted clean\-image direction𝐛\\mathbf\{b\}to the current state requires the Jacobian productD​Ft⊤​𝐛DF\_\{t\}^\{\\top\}\\mathbf\{b\}\. For a smooth path, the continuous adjoint solves

𝝀˙s=−D𝐱​𝐯​\(𝐱s,s\)⊤​𝝀s,𝝀0=𝐛,𝝀t=D​Ft​\(𝐱t\)⊤​𝐛\.\\dot\{\\boldsymbol\{\\lambda\}\}\_\{s\}=\-D\_\{\\mathbf\{x\}\}\\mathbf\{v\}\(\\mathbf\{x\}\_\{s\},s\)^\{\\top\}\\boldsymbol\{\\lambda\}\_\{s\},\\qquad\\boldsymbol\{\\lambda\}\_\{0\}=\\mathbf\{b\},\\qquad\\boldsymbol\{\\lambda\}\_\{t\}=DF\_\{t\}\(\\mathbf\{x\}\_\{t\}\)^\{\\top\}\\mathbf\{b\}\.\(34\)Indeed, a state perturbation satisfies𝝃˙s=D𝐱​𝐯​𝝃s\\dot\{\\boldsymbol\{\\xi\}\}\_\{s\}=D\_\{\\mathbf\{x\}\}\\mathbf\{v\}\\,\\boldsymbol\{\\xi\}\_\{s\}, so𝝀s⊤​𝝃s\\boldsymbol\{\\lambda\}\_\{s\}^\{\\top\}\\boldsymbol\{\\xi\}\_\{s\}is constant\. For an Euler completion𝐳k\+1=𝐳k\+Δ​tk​𝐯​\(𝐳k,tk\)\\mathbf\{z\}\_\{k\+1\}=\\mathbf\{z\}\_\{k\}\+\\Delta t\_\{k\}\\mathbf\{v\}\(\\mathbf\{z\}\_\{k\},t\_\{k\}\), defineJk=I\+Δ​tk​D𝐱​𝐯​\(𝐳k,tk\)J\_\{k\}=I\+\\Delta t\_\{k\}D\_\{\\mathbf\{x\}\}\\mathbf\{v\}\(\\mathbf\{z\}\_\{k\},t\_\{k\}\)\. A velocity perturbation at stepqqgives

δR=Δtq⟨𝝀q\+1,δ𝐯q⟩,𝝀q\+1=Jq\+1⊤⋯JN−1⊤𝐛,𝐛=∇R\(𝐳N\)\.\\delta R=\\Delta t\_\{q\}\\langle\\boldsymbol\{\\lambda\}\_\{q\+1\},\\delta\\mathbf\{v\}\_\{q\}\\rangle,\\qquad\\boldsymbol\{\\lambda\}\_\{q\+1\}=J\_\{q\+1\}^\{\\top\}\\cdots J\_\{N\-1\}^\{\\top\}\\mathbf\{b\},\\qquad\\mathbf\{b\}=\\nabla R\(\\mathbf\{z\}\_\{N\}\)\.\(35\)The ablation replaces𝐛\\mathbf\{b\}by𝐮^q\\widehat\{\\mathbf\{u\}\}\_\{q\}, evaluates this suffix along the query’s own completion by vector–Jacobian products, and rescales it to the original coefficient’s norm\. An empty suffix is the identity\. This keeps per\-step update magnitudes fixed while testing the cost and benefit of transporting directions\. For a coefficient error𝐞\\mathbf\{e\}, the continuous pullback error is bounded by‖D​Ft‖op​‖𝐞‖\\\|DF\_\{t\}\\\|\_\{\\mathrm\{op\}\}\\\|\\mathbf\{e\}\\\|\. Direct transfer avoids these Jacobian products; its training performance is evaluated in Section[6\.2](https://arxiv.org/html/2610.22973#S6.SS2)\.

## Appendix EComplete reward and weighting ablations

Every training arm uses8×8\\timesA800\-40GB\. We show both compute and update count because the training choices change update cost\. Budget\-matched scores are linearly interpolated within the recorded curves\.

### E\.1Reward\-dependent actor weights

All six arms use the same mean\-centered ODE\-group clean\-image regression and48×2448\\times 24collected trajectories\. We change only the query weights: all2424\(ai=1a\_\{i\}=1\);ai=1−ria\_\{i\}=1\-r\_\{i\};ai=𝟏\{ri<1\}a\_\{i\}=\\mathbf\{1\}\\\{r\_\{i\}<1\\\}; random six; highest six; lowest six\. Random selection uses the first six independent\-seed rollouts, so selection is independent of reward\. The scalar functions consider all2424queries; zero\-weight queries need neither a fit for their own actor nor a backward pass\. Their clean images still serve as sibling observations\. The denominator is the sum of actor weights, as in Eq\. \([42](https://arxiv.org/html/2610.22973#A6.E42)\), not the number of candidates\.

Figure 5:GenEval query weights at a2828\-GPU\-hour budget, shown against compute and updates\.At2828GPU\-hours the two ceiling\-aware arms have completed approximately8181–8484updates, versus about110110for the lowest\-six arm\.

The saturation motivation also has a frozen\-policy check: at an early checkpoint, a one\-sided perturbation of3%3\\%of state RMS along the group fit changes the reward of failed samples by\+0\.064\+0\.064and successful samples by−0\.019\-0\.019, with random\-direction controls around±0\.003\\pm 0\.003\. During updates2828–6565, roughly74%74\\%of the all\-24 arm’s actor samples already have maximum reward\.

Figure 6:The all\-24 continuation\. The lowest\-six advantage persists beyond the initial budget but narrows near7272GPU\-hours\.Table 5:Longer GenEval comparison at fixed compute, using interpolation within the saved curves\.
### E\.2Norm\-matched adjoint transport of the clean\-image estimate

This arm keeps the PickScore24×2424\\times 24bank recipe and replaces each fitted clean\-image direction by the suffix pullback in Eq\. \([35](https://arxiv.org/html/2610.22973#A4.E35)\)\. The Jacobians are evaluated along each sample’s own Euler completion from𝐳q\+1=𝐱q\+Δ​tq​𝐯​\(𝐱q,tq\)\\mathbf\{z\}\_\{q\+1\}=\\mathbf\{x\}\_\{q\}\+\\Delta t\_\{q\}\\mathbf\{v\}\(\\mathbf\{x\}\_\{q\},t\_\{q\}\), the completion that produced its clean image, and applied by vector–Jacobian products without a dense Jacobian\. Each transported direction is then rescaled to the root\-mean\-square of the direction it replaces, separately for every sample and step, so the update keeps the recipe’s magnitudes across samples and times and changes only the orientation\. Over the6969updates, the transported direction has median cosine0\.180\.18with the clean\-image direction at the first step \(range0\.150\.15–0\.410\.41\), rising monotonically to0\.790\.79at step77and11at the last two\. The transport runs in the training precision \(bfloat16\); on eight probe samples its median cosine with a float32 reference is0\.710\.71,0\.950\.95,0\.970\.97,0\.990\.99, and1\.001\.00at steps00,22,44,66, and88\.

At1212,2424,3535,4747,5959,7070, and8181GPU\-hours \(updates10,20,…,6010,20,\\dots,60and6969\), the transported arm scores22\.2322\.23,22\.6822\.68,22\.9222\.92,22\.9622\.96,23\.0423\.04,23\.0923\.09, and23\.1823\.18\. At the same compute the direct arm scores22\.9722\.97,23\.4223\.42,23\.4323\.43,23\.6323\.63,23\.6623\.66,23\.7523\.75, and23\.6523\.65; at the same updates it scores22\.2122\.21,22\.9422\.94,23\.3623\.36,23\.4523\.45,23\.3123\.31,23\.4023\.40, and23\.4823\.48\. From update2020onward the per\-update deficit \(0\.260\.26–0\.490\.49\) exceeds the difference between the two training seeds at matched updates \(at most0\.140\.14through update7070; Appendix[E\.4](https://arxiv.org/html/2610.22973#A5.SS4)\)\.

### E\.3PickScore directional\-alignment probe

We compare noisy\-state and clean\-image directions from matched PickScore perturbations at the initial model and an update\-200200checkpoint\. We measure their cosine with the real reward gradient𝐠tqODE=∇𝐱\(R∘Ftq\)​\(𝐱q\)\\mathbf\{g\}\_\{t\_\{q\}\}^\{\\mathrm\{ODE\}\}=\\nabla\_\{\\mathbf\{x\}\}\(R\\circ F\_\{t\_\{q\}\}\)\(\\mathbf\{x\}\_\{q\}\), differentiated through the numerical ODE completion, VAE, and PickScore \(Figure[2](https://arxiv.org/html/2610.22973#S6.F2)\)\.

We average signed cosines within prompts with time weights proportional totqt\_\{q\}and use prompt\-bootstrap resampling for confidence intervals\. The clean\-image direction has higher cosine with the real gradient, although its advantage diminishes near the last query \(Figure[2](https://arxiv.org/html/2610.22973#S6.F2)\)\.

Using the actual trajectory bank, the direct estimate also has positive cosine with𝐠tqODE\\mathbf\{g\}\_\{t\_\{q\}\}^\{\\mathrm\{ODE\}\}at update200200\.

### E\.4PickScore training\-seed replication

We test the PickScore training recipe with multiple seeds\. Figure[7](https://arxiv.org/html/2610.22973#A5.F7)shows their learning curves\.

Figure 7:PickScore training across seeds, shown against GPU\-hours \(left\) and updates \(right\)\.

## Appendix FImplementation and reproducibility protocol

#### Reward\-adaptive rollouts, one clean\-image regression\.

Table[6](https://arxiv.org/html/2610.22973#A6.T6)gives the configurations for Algorithm[1](https://arxiv.org/html/2610.22973#alg1)\. All use SD3\.5\-Medium, the ten\-step shifted SD3 grid, CFG=1=1, and one parameter update per collection\. The reference rows and update normalization for the sampling schemes in Section[5\.1](https://arxiv.org/html/2610.22973#S5.SS1)are specified below\.

Table 6:Configurations at the reported checkpoints\. OCR floor uses Eq\. \([43](https://arxiv.org/html/2610.22973#A8.E43)\) withλfloor=10\\lambda\_\{\\mathrm\{floor\}\}=10and the margin schedulem=0→0\.03→0\.06m=0\\to 0\.03\\to 0\.06at updates4040and100100\.Algorithm 1One DiffusionRVGE update with centered clean\-image regression1:Collection parameters

𝜽old\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}, reward

RR, grid

t0\>⋯\>tN=0t\_\{0\}\>\\cdots\>t\_\{N\}=0, ridge

λsum\\lambda\_\{\\mathrm\{sum\}\}
2:Rollout and query\-weight rules \(Section[5\.1](https://arxiv.org/html/2610.22973#S5.SS1)\)

3:Sample prompts

cic\_\{i\}and collect query trajectories

\{𝐱i,q\}q=0N\\\{\\mathbf\{x\}\_\{i,q\}\\\}\_\{q=0\}^\{N\}\.

4:Complete rollout states to clean images

𝐳j\\mathbf\{z\}\_\{j\}and evaluate

rj=R⁡\(𝐳j,cj\)r\_\{j\}=R\(\\mathbf\{z\}\_\{j\},c\_\{j\}\)\.

5:Assign actor weights

aia\_\{i\}from the collected rewards; set

Za=∑iaiZ\_\{a\}=\\sum\_\{i\}a\_\{i\}\.

6:for allquery trajectories

iiwith

ai\>0a\_\{i\}\>0and times

q=0,…,N−1q=0,\\ldots,N\-1do

7:Form same\-prompt references

ℬi,q\\mathcal\{B\}\_\{i,q\}and set the rollout\-dependent image and reward centers

\(𝐦^i,q,r¯^i,q\)\(\\widehat\{\\mathbf\{m\}\}\_\{i,q\},\\widehat\{\\bar\{r\}\}\_\{i,q\}\)\.

8:for all

j∈ℬi,qj\\in\\mathcal\{B\}\_\{i,q\}do

9:

𝐝i​j←𝐳j−𝐦^i,q\\mathbf\{d\}\_\{ij\}\\leftarrow\\mathbf\{z\}\_\{j\}\-\\widehat\{\\mathbf\{m\}\}\_\{i,q\};

bi​j←rj−r¯^i,qb\_\{ij\}\\leftarrow r\_\{j\}\-\\widehat\{\\bar\{r\}\}\_\{i,q\}\.

10:endfor

11:Discard zero displacements; set

ℓi​j=‖𝐝i​j‖2\\ell\_\{ij\}=\\\|\\mathbf\{d\}\_\{ij\}\\\|\_\{2\}for each retained row\.

12:Stack

𝐝i​j⊤/ℓi​j\\mathbf\{d\}\_\{ij\}^\{\\top\}/\\ell\_\{ij\}into

Qi,qQ\_\{i,q\}and

bi​j/ℓi​jb\_\{ij\}/\\ell\_\{ij\}into

𝐛i,q\\mathbf\{b\}\_\{i,q\}\.

13:

𝐮^i,q←Qi,q⊤​\(Qi,q​Qi,q⊤\+λsum​I\)−1​𝐛i,q\\widehat\{\\mathbf\{u\}\}\_\{i,q\}\\leftarrow Q\_\{i,q\}^\{\\top\}\(Q\_\{i,q\}Q\_\{i,q\}^\{\\top\}\+\\lambda\_\{\\mathrm\{sum\}\}I\)^\{\-1\}\\mathbf\{b\}\_\{i,q\}; use

00for an empty fit\.

14:endfor

15:Set

ωq=tq/\(N−1​∑p=0N−1tp\)\\omega\_\{q\}=t\_\{q\}/\(N^\{\-1\}\\sum\_\{p=0\}^\{N\-1\}t\_\{p\}\); detach directions, weights, and

ZaZ\_\{a\}\.

16:if

Za\>0Z\_\{a\}\>0then

17:

δ​𝐯𝜽,i,q←𝐯𝜽​\(𝐱i,q,tq,ci\)−sg⁡\[𝐯𝜽old​\(𝐱i,q,tq,ci\)\]\\delta\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\},i,q\}\\leftarrow\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\}\(\\mathbf\{x\}\_\{i,q\},t\_\{q\},c\_\{i\}\)\-\\operatorname\{sg\}\[\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}\(\\mathbf\{x\}\_\{i,q\},t\_\{q\},c\_\{i\}\)\]\.

18:

ℒ←2N​Za∑i:ai\>0∑q=0N−1aiωq⟨δ𝐯𝜽,i,q,sg\[𝐮^i,q\]⟩\\mathcal\{L\}\\leftarrow\\dfrac\{2\}\{NZ\_\{a\}\}\\sum\_\{i:a\_\{i\}\>0\}\\sum\_\{q=0\}^\{N\-1\}a\_\{i\}\\omega\_\{q\}\\langle\\delta\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\},i,q\},\\operatorname\{sg\}\[\\widehat\{\\mathbf\{u\}\}\_\{i,q\}\]\\rangle\.

19:Starting from

𝜽=𝜽old\\boldsymbol\{\\theta\}=\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}, take one parameter update to minimize

ℒ\\mathcal\{L\}\.

20:endif

#### Exploration transition\.

WithΔ​tq=tq\+1−tq<0\\Delta t\_\{q\}=t\_\{q\+1\}\-t\_\{q\}<0and time\-dependent diffusion coefficientγq\\gamma\_\{q\}, its interior\-step discretization is

𝐱i,q\+1=𝐱i,q\+Δ​tq​\[𝐯𝜽old​\(𝐱i,q,tq,c\)\+γq22​tq​ϵ^i,q\]\+γq​−Δ​tq​𝝃i,q,ϵ^i,q=𝐱i,q\+\(1−tq\)𝐯𝜽old\(𝐱i,q,tq,c\),𝝃i,q∼𝒩\(0,I\)\.\\begin\{split\}\\mathbf\{x\}\_\{i,q\+1\}=\{\}&\\mathbf\{x\}\_\{i,q\}\+\\Delta t\_\{q\}\\left\[\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}\(\\mathbf\{x\}\_\{i,q\},t\_\{q\},c\)\+\\frac\{\\gamma\_\{q\}^\{2\}\}\{2t\_\{q\}\}\\widehat\{\\boldsymbol\{\\epsilon\}\}\_\{i,q\}\\right\]\+\\gamma\_\{q\}\\sqrt\{\-\\Delta t\_\{q\}\}\\,\\boldsymbol\{\\xi\}\_\{i,q\},\\\\ &\\widehat\{\\boldsymbol\{\\epsilon\}\}\_\{i,q\}=\\mathbf\{x\}\_\{i,q\}\+\(1\-t\_\{q\}\)\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}\(\\mathbf\{x\}\_\{i,q\},t\_\{q\},c\),\\qquad\\boldsymbol\{\\xi\}\_\{i,q\}\\sim\\mathcal\{N\}\(0,I\)\.\\end\{split\}\(36\)

#### Noise coefficient and boundaries\.

The configuration’s constant noise mode holds the radius parameterη\\etafixed, not the per\-transition standard deviation\. The recorded coefficient is

γq2=η2​tq1−t~q,t~q=\{t1,q=0,tq,q\>0\.\\gamma\_\{q\}^\{2\}=\\frac\{\\eta^\{2\}t\_\{q\}\}\{1\-\\widetilde\{t\}\_\{q\}\},\\qquad\\widetilde\{t\}\_\{q\}=\\begin\{cases\}t\_\{1\},&q=0,\\\\ t\_\{q\},&q\>0\.\\end\{cases\}\(37\)The factor−Δ​tq\\sqrt\{\-\\Delta t\_\{q\}\}makes the perturbation step\-dependent\. Using1−t11\-t\_\{1\}att0=1t\_\{0\}=1avoids the singular denominator\.

#### Regression rows and solve\.

Equations \([10](https://arxiv.org/html/2610.22973#S4.E10)\)–\([12](https://arxiv.org/html/2610.22973#S4.E12)\) state the fit for a generic query\. Hereiiindexes trajectories in a prompt group,qqindexes query time, andj∈ℬi,qj\\in\\mathcal\{B\}\_\{i,q\}indexes reference completions;\(𝐦^i,q,r¯^i,q\)\(\\widehat\{\\mathbf\{m\}\}\_\{i,q\},\\widehat\{\\bar\{r\}\}\_\{i,q\}\)denote the centers in Eq\. \([10](https://arxiv.org/html/2610.22973#S4.E10)\) at query timetqt\_\{q\}\. For each retained nonzero displacement, define

ϕi,q,j=𝐳j−𝐦^i,q‖𝐳j−𝐦^i,q‖2,b~i,q,j=rj−r¯^i,q‖𝐳j−𝐦^i,q‖2\.\\boldsymbol\{\\phi\}\_\{i,q,j\}=\\frac\{\\mathbf\{z\}\_\{j\}\-\\widehat\{\\mathbf\{m\}\}\_\{i,q\}\}\{\\\|\\mathbf\{z\}\_\{j\}\-\\widehat\{\\mathbf\{m\}\}\_\{i,q\}\\\|\_\{2\}\},\\qquad\\widetilde\{b\}\_\{i,q,j\}=\\frac\{r\_\{j\}\-\\widehat\{\\bar\{r\}\}\_\{i,q\}\}\{\\\|\\mathbf\{z\}\_\{j\}\-\\widehat\{\\mathbf\{m\}\}\_\{i,q\}\\\|\_\{2\}\}\.\(38\)Stackϕi,q,j⊤\\boldsymbol\{\\phi\}\_\{i,q,j\}^\{\\top\}intoQi,qQ\_\{i,q\}and the scalar targetsb~i,q,j\\widetilde\{b\}\_\{i,q,j\}into𝐛i,q\\mathbf\{b\}\_\{i,q\}\. The fitted coefficient is

𝐮^i,q=arg​min𝐮⁡\{‖Qi,q​𝐮−𝐛i,q‖22\+λsum​‖𝐮‖22\}=Qi,q⊤​\(Qi,q​Qi,q⊤\+λsum​I\)−1​𝐛i,q\.\\widehat\{\\mathbf\{u\}\}\_\{i,q\}=\\operatorname\*\{arg\\,min\}\_\{\\mathbf\{u\}\}\\left\\\{\\\|Q\_\{i,q\}\\mathbf\{u\}\-\\mathbf\{b\}\_\{i,q\}\\\|\_\{2\}^\{2\}\+\\lambda\_\{\\mathrm\{sum\}\}\\\|\\mathbf\{u\}\\\|\_\{2\}^\{2\}\\right\\\}=Q\_\{i,q\}^\{\\top\}\(Q\_\{i,q\}Q\_\{i,q\}^\{\\top\}\+\\lambda\_\{\\mathrm\{sum\}\}I\)^\{\-1\}\\mathbf\{b\}\_\{i,q\}\.\(39\)This is Eq\. \([12](https://arxiv.org/html/2610.22973#S4.E12)\) withℓi​j=‖𝐝i​j‖\\ell\_\{ij\}=\\\|\\mathbf\{d\}\_\{ij\}\\\|, before scaling the direction byκt\\kappa\_\{t\}; the sum\-loss ridge coefficient isλsum=0\.3\\lambda\_\{\\mathrm\{sum\}\}=0\.3\. The dual solve scales with the number of observations\. The fitted coefficient𝐮^\\widehat\{\\mathbf\{u\}\}summarizes the observed reward changes across the chosen clean\-image contrasts\.

#### Trajectory\-bank samples\.

For the trajectory bank, useℬi,q=\{\(i,m\):0≤m<8,m≠q\}\\mathcal\{B\}\_\{i,q\}=\\\{\(i,m\):0\\leq m<8,\\ m\\neq q\\\}on the ten\-step grid, with

𝐳i,q=Ftq𝜽old​\(𝐱i,q\),𝐦^i,q=𝐳i,q,r¯^i,q=R⁡\(𝐳i,q\)\.\\mathbf\{z\}\_\{i,q\}=F\_\{t\_\{q\}\}^\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}\(\\mathbf\{x\}\_\{i,q\}\),\\qquad\\widehat\{\\mathbf\{m\}\}\_\{i,q\}=\\mathbf\{z\}\_\{i,q\},\\qquad\\widehat\{\\bar\{r\}\}\_\{i,q\}=R\(\\mathbf\{z\}\_\{i,q\}\)\.\(40\)The numerators in Eq\. \([38](https://arxiv.org/html/2610.22973#A6.E38)\) become𝐳i,m−𝐳i,q\\mathbf\{z\}\_\{i,m\}\-\\mathbf\{z\}\_\{i,q\}andri,m−ri,qr\_\{i,m\}\-r\_\{i,q\}\. This head\-8 reference set avoids the shortest late\-stage chords\. All retained completions enter one regression with the same rule\.

#### Independent\-seed group samples\.

For independent\-seed groups, useℬi,q=\{j:j≠i\}\\mathcal\{B\}\_\{i,q\}=\\\{j:j\\neq i\\\}, with

𝐦^i,q=𝐳¯−i=1K−1​∑j≠i𝐳j,r¯^i,q=r¯−i=1K−1​∑j≠irj\.\\widehat\{\\mathbf\{m\}\}\_\{i,q\}=\\bar\{\\mathbf\{z\}\}\_\{\-i\}=\\frac\{1\}\{K\-1\}\\sum\_\{j\\neq i\}\\mathbf\{z\}\_\{j\},\\qquad\\widehat\{\\bar\{r\}\}\_\{i,q\}=\\bar\{r\}\_\{\-i\}=\\frac\{1\}\{K\-1\}\\sum\_\{j\\neq i\}r\_\{j\}\.\(41\)The raw rows satisfy∑j≠i𝐝i​j=0\\sum\_\{j\\neq i\}\\mathbf\{d\}\_\{ij\}=0and∑j≠ibi​j=0\\sum\_\{j\\neq i\}b\_\{ij\}=0\. Normalize these rows as in Eq\. \([38](https://arxiv.org/html/2610.22973#A6.E38)\), solve Eq\. \([39](https://arxiv.org/html/2610.22973#A6.E39)\), and reuse the resulting𝐮^i\\widehat\{\\mathbf\{u\}\}\_\{i\}at every query time on trajectoryii\.

#### Discrete velocity loss and normalization\.

The reported updates implement Eq\. \([14](https://arxiv.org/html/2610.22973#S4.E14)\) on theNN\-step grid\. ForBBcollected trajectories with stopped query weightsaia\_\{i\}andZa=∑i=1Bai\>0Z\_\{a\}=\\sum\_\{i=1\}^\{B\}a\_\{i\}\>0,

ℒlinear​\(𝜽,𝜽old\)=1N​sg⁡\[Za\]​∑i=1B∑q=0N−12​ωq​sg⁡\[ai\]​⟨𝐯𝜽​\(𝐱i,q,tq,ci\)−sg⁡\[𝐯𝜽old​\(𝐱i,q,tq,ci\)\],sg⁡\[𝐮^i,q\]⟩,\\mathcal\{L\}\_\{\\mathrm\{linear\}\}\(\\boldsymbol\{\\theta\};\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\)=\\frac\{1\}\{N\\operatorname\{sg\}\[Z\_\{a\}\]\}\\sum\_\{i=1\}^\{B\}\\sum\_\{q=0\}^\{N\-1\}2\\omega\_\{q\}\\operatorname\{sg\}\[a\_\{i\}\]\\left\\langle\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\}\(\\mathbf\{x\}\_\{i,q\},t\_\{q\},c\_\{i\}\)\-\\operatorname\{sg\}\[\\mathbf\{v\}\_\{\\boldsymbol\{\\theta\}\_\{\\mathrm\{old\}\}\}\(\\mathbf\{x\}\_\{i,q\},t\_\{q\},c\_\{i\}\)\],\\operatorname\{sg\}\[\\widehat\{\\mathbf\{u\}\}\_\{i,q\}\]\\right\\rangle,\(42\)and the loss is zero whenZa=0Z\_\{a\}=0\. Hereωq=tq/\(N−1​∑ptp\)\\omega\_\{q\}=t\_\{q\}/\(N^\{\-1\}\\sum\_\{p\}t\_\{p\}\)is the normalized empirical productωtq=wtq​χtq​κtq\\omega\_\{t\_\{q\}\}=w\_\{t\_\{q\}\}\\chi\_\{t\_\{q\}\}\\kappa\_\{t\_\{q\}\}from Section[4](https://arxiv.org/html/2610.22973#S4)\. Equation \([42](https://arxiv.org/html/2610.22973#A6.E42)\) is the discrete form of Eq\. \([14](https://arxiv.org/html/2610.22973#S4.E14)\), withttnormalized to unit mean over the grid and the average taken over queries withai\>0a\_\{i\}\>0\. We setai=1a\_\{i\}=1for trajectory banks andai=𝟏\{i∈Lowest6\(𝐫ci\)\}a\_\{i\}=\\mathbf\{1\}\\\{i\\in\\operatorname\{Lowest\}\_\{6\}\(\\mathbf\{r\}\_\{c\_\{i\}\}\)\\\}for GenEval, breaking ties randomly\. Both binary selection and the continuous weights compared in Section[6\.4](https://arxiv.org/html/2610.22973#S6.SS4)act on the stopped actor term after the fit\.

Both weights and denominator are stopped, and a zero denominator gives zero loss\. Setting allai=1a\_\{i\}=1recovers the unweighted batch average; binary lowest\-six weights recover the selected\-sample average exactly\. Zero\-weight queries require no actor backward pass\. Every reported run uses these unit\-mean linear time weights\. All\-equal rewards give a zero mean\-centered group direction even if rank selection retains six queries\. The stricter combination𝟏\{ri<1\}𝟏\{i∈Lowest6\}\\mathbf\{1\}\\\{r\_\{i\}<1\\\}\\mathbf\{1\}\\\{i\\in\\operatorname\{Lowest\}\_\{6\}\\\}is distinct from the ungated lowest\-six reference and is not silently substituted for it\.

### F\.1Five\-reward joint training

The joint model of Table[1](https://arxiv.org/html/2610.22973#S5.T1)alternates the two update types above on DiffusionNFT’s five training rewards\. Withq=PickScore/26\+ClipScore\+HPSv2​\.1q=\\mathrm\{PickScore\}/26\+\\mathrm\{ClipScore\}\+\\mathrm\{HPSv2\.1\}, Pick\-a\-Pic prompts receiver=10​qr=10q, and GenEval and OCR prompts receive their task reward with the floor of Eq\. \([43](https://arxiv.org/html/2610.22973#A8.E43)\),r=Rtask\+10​min⁡\(q−q¯c\+m,0\)r=R\_\{\\mathrm\{task\}\}\+10\\min\(q\-\\bar\{q\}\_\{c\}\+m,0\), wherem=0m=0before update700700andm=0\.03m=0\.03afterwards\. A group update draws4848prompts from the listed sources, rolls out2424ODE trajectories per prompt, and fits Algorithm[1](https://arxiv.org/html/2610.22973#alg1)with the centers in Eq\. \([41](https://arxiv.org/html/2610.22973#A6.E41)\)\. It trains the six lowest\-reward trajectories of each prompt, restricted on GenEval and OCR prompts to trajectories with task reward below0\.990\.99\. A bank update uses the PickScore configuration of Table[6](https://arxiv.org/html/2610.22973#A6.T6)with reward10​q10q\. The GenEval task reward is the partial\-credit score up to update450450and the mean of the partial\-credit and strict judgements afterwards\. Updates500500–600600use4040sampling steps with actor loss on every fourth state; all other stages use1010steps\. The learning rate for updates700700–850850is reached after a2020\-update warm\-up\. Table[7](https://arxiv.org/html/2610.22973#A6.T7)lists the stages\.

Table 7:Stages of the five\-reward joint model\. The prompt mix gives the fractions of Pick\-a\-Pic, GenEval, and OCR prompts per update; the first stage is the first training seed of the single\-reward PickScore run in Table[2](https://arxiv.org/html/2610.22973#S5.T2)\.#### Compute\.

The full joint model uses 404 GPU\-hours from SD3\.5\-Medium\. DiffusionNFT does not report the cost of its joint model\. Its described stages total800\+300\+200\+200\+100=1,600800\+300\+200\+200\+100=1\{,\}600iterations with4040sampling steps; its Table 1 lists1\.71\.7k, and we use the smaller count\. Timing its official code on our hardware and accounting for the described schedule imply at least2,0002\{,\}000GPU\-hours in total, or at least5×5\\timesour budget\.

## Appendix GMeasurement provenance

Our single\-reward and five\-reward evaluations use the same protocols and datasets as DiffusionNFT, enabling direct comparison \(Section[5\.1](https://arxiv.org/html/2610.22973#S5.SS1)\)\. Figure[1](https://arxiv.org/html/2610.22973#S5.F1)uses one image per DrawBench prompt for the PickScore curves, alongside the OCR and GenEval tests\. Sampling counts can differ between checkpoint evaluations and learning\-curve readouts, so their terminal values need not coincide\.

Each curve retains its recorded cumulative GPU time\. The source includes saved CSVs, a checksum manifest, and figure\-generation scripts\. Budget comparisons interpolate within observed runs; no missing result is extrapolated\.

## Appendix HReward exploitation

### H\.1Scheduled quality floor for OCR

OCR\-only training replaces requested scenes with large text on a flat background; on the main DrawBench evaluation, its PickScore is20\.3220\.32versus22\.3422\.34for the base model \(Table[2](https://arxiv.org/html/2610.22973#S5.T2)\)\. We counter this with a group\-relative quality floor, usingq=PickScore/26\+ClipScore\+HPSv2​\.1q=\\mathrm\{PickScore\}/26\+\\mathrm\{ClipScore\}\+\\mathrm\{HPSv2\.1\}and regressing

r=OCR\+λfloor​min⁡\(q−q¯c\+m,0\),λfloor=10,r=\\mathrm\{OCR\}\+\\lambda\_\{\\mathrm\{floor\}\}\\min\(q\-\\bar\{q\}\_\{c\}\+m,0\),\\qquad\\lambda\_\{\\mathrm\{floor\}\}=10,\(43\)whereq¯c\\bar\{q\}\_\{c\}is the prompt\-group mean\. A tight initial floor establishes scene content before relaxing the OCR tradeoff:m=0m=0until update4040,0\.030\.03until100100, then0\.060\.06\. After150150updates \(4141GPU\-hours\), this run reaches OCR0\.970\.97and DrawBench PickScore22\.1722\.17while retaining scenes in the inspected samples \(Figure[9](https://arxiv.org/html/2610.22973#A8.F9)\)\. The floor mitigates the observed flat\-page exploit\.

#### OCR layout comparison\.

We quantify templating by the mean cross\-prompt correlation of8×88\\times 8luminance thumbnails \(base0\.050\.05; a fixed layout approaches one\)\. At update4040, the trajectory\-bank and ODE\-group runs in Section[6\.3](https://arxiv.org/html/2610.22973#S6.SS3)have correlations0\.930\.93and0\.520\.52, respectively, and both produce flat, text\-dominated layouts\. The scheduled quality floor retains the requested scenes in the inspected samples, with a shared central\-sign composition \(correlation 0\.51\)\. Its DrawBench Aesthetic score is5\.645\.64, compared with4\.654\.65for the OCR\-only checkpoint \(Table[2](https://arxiv.org/html/2610.22973#S5.T2)\)\.

#### Why schedule the marginmm?

A high\-noise reference penalty prevents the template but also removes the OCR signal\. Training only failing trajectories still produces the flat\-page layout, and a linear PickScore term preserves content but stalls OCR near0\.850\.85\. A floor with a fixed reference \(PickScore\\mathrm\{PickScore\}two points above the base model\) preserves scenes and reaches0\.930\.93after120120updates but stops there, because once the model’s typical quality exceeds the reference the floor is silent; the relative floor of Eq\. \([43](https://arxiv.org/html/2610.22973#A8.E43)\) follows the model\. Starting from initialization with the loose marginm=0\.06m=0\.06andλfloor=10\\lambda\_\{\\mathrm\{floor\}\}=10reaches OCR0\.950\.95within4040updates, but on a cartoon\-signboard template whose PickScore stays at the base level, so the quality rewards do not see it\. The schedule starts with a tight margin to establish scene content, then relaxes it to improve OCR\. The scheduled floor reaches 0\.97 with scenes preserved; its remaining errors are mostly single glyphs in long strings, and the surviving style is a shared neon rendering of the text\.

![Refer to caption](https://arxiv.org/html/2610.22973v1/qualitative_single_reward.png)Figure 8:PickScore visual results comparing the DiffusionRVGE PickScore checkpoint with the public Flow\-GRPO PickScore checkpoint on DrawBench prompts\. Within each pair, models use the same prompt, seed, and 40\-step ODE sampler\.![Refer to caption](https://arxiv.org/html/2610.22973v1/qualitative_single_reward_ocr.png)Figure 9:OCR visual results comparing the DiffusionRVGE quality\-floor checkpoint with the public Flow\-GRPO Text checkpoint on OCR test prompts\. Within each pair, models use the same prompt, seed, and 40\-step ODE sampler; panel labels abbreviate the prompts\.![Refer to caption](https://arxiv.org/html/2610.22973v1/qualitative_single_reward_geneval.png)Figure 10:GenEval visual results comparing the DiffusionRVGE single\-reward GenEval checkpoint with the public Flow\-GRPO GenEval checkpoint on GenEval prompts\. Within each pair, models use the same prompt, seed, and 40\-step ODE sampler\.![Refer to caption](https://arxiv.org/html/2610.22973v1/qualitative_multireward.png)Figure 11:DrawBench visual results comparing the five\-reward joint DiffusionRVGE checkpoint with the public DiffusionNFT multi\-reward checkpoint\. Each pair uses the same full prompt and seed with 40\-step ODE sampling; panel labels abbreviate the prompts\.![Refer to caption](https://arxiv.org/html/2610.22973v1/qualitative_multireward_ocr.png)Figure 12:OCR test visual results comparing the five\-reward joint DiffusionRVGE checkpoint with the public DiffusionNFT multi\-reward checkpoint\. Each pair uses the same full prompt and seed with 40\-step ODE sampling; panel labels abbreviate the prompts\.