ReCAST: Reward Credit Assignment across Timesteps for Online Diffusion Reinforcement

# ReCAST: Reward Credit Assignment across Timesteps for Online Diffusion Reinforcement
Source: [https://arxiv.org/html/2609.13425](https://arxiv.org/html/2609.13425)
Yihang Chen††thanks:Equal contribution\.Yuanhao Ban11footnotemark:1Affiliation:University of California, Los AngelesAffiliation:Arena AIEmail:[banyh2000@cs\.ucla\.edu](mailto:)Kuei\-Chun KaoAffiliation:University of California, Los AngelesAffiliation:Arena AICho\-Jui HsiehAffiliation:University of California, Los AngelesAffiliation:Arena AI

###### Abstract

Training diffusion models with multiple rewards requires distinguishing*user preference*from*reward informativeness*\. User preference determines how much each reward should contribute to the overall objective; reward informativeness determines when its feedback is useful during denoising\. Some rewards can meaningfully evaluate a sample as soon as global structure emerges, but others become informative only when the sample is nearly clean\. To address both questions jointly, we propose ReCAST \(RewardCreditASsignment acrossTimesteps\), the first method, to our knowledge, for per\-reward, timestep\-dependent credit assignment in diffusion reward fine\-tuning\. ReCAST separates user preferences from temporal allocation through a reward\-by\-timestep weight matrix𝑾\\bm\{W\}, whose row sums match the user\-specified reward budgets𝝀\\bm\{\\lambda\}, while its column sums are equal, assigning the same total weight to each denoising step\. Under these marginal constraints, ReCAST allocates weight according to each reward’s informativeness, quantified by its Rényi discriminability gain at each step\. These gains telescope to the total discriminability between the reward\-induced positive policy and the current policy, providing a basis for temporal credit assignment\. We evaluate ReCAST by training SD3\.5\-Medium under two distinct four\-reward settings, each across five reward budgets𝝀\\bm\{\\lambda\}\. ReCAST improves the training rewards in one setting and matches them in the other, improves every held\-out judge in both, and is preferred by an independent LLM\-as\-a\-Judge\. Overall, these results show that ReCAST achieves improvements that generalize beyond the training rewards and support its core principle: assigning each reward greater weight at the denoising timesteps where its feedback is most informative\.

## 1Introduction

Reward\-based post\-training has become a standard way to align diffusion and flow models with human intent, via policy gradient over the denoising chain\[[1](https://arxiv.org/html/2609.13425#bib.bib16),[7](https://arxiv.org/html/2609.13425#bib.bib17)\], backpropagation through a differentiable reward\[[3](https://arxiv.org/html/2609.13425#bib.bib12),[29](https://arxiv.org/html/2609.13425#bib.bib11)\], preference optimization\[[37](https://arxiv.org/html/2609.13425#bib.bib13),[43](https://arxiv.org/html/2609.13425#bib.bib15)\], group\-relative objectives\[[24](https://arxiv.org/html/2609.13425#bib.bib1),[41](https://arxiv.org/html/2609.13425#bib.bib2)\], or negative\-aware fine\-tuning on the forward process\[[42](https://arxiv.org/html/2609.13425#bib.bib25)\]\. In practice, a single reward is rarely sufficient: standard training recipes mix prompt alignment\[[12](https://arxiv.org/html/2609.13425#bib.bib24)\], learned human preference\[[39](https://arxiv.org/html/2609.13425#bib.bib18),[18](https://arxiv.org/html/2609.13425#bib.bib8),[40](https://arxiv.org/html/2609.13425#bib.bib6)\], and rule\-based correctness\[[9](https://arxiv.org/html/2609.13425#bib.bib7)\]by linear scalarizationr⁡\(𝒙0,𝒄\)=∑iλi​ri​\(𝒙0,𝒄\)r\(\{\\bm\{x\}\}\_\{0\},\{\\bm\{c\}\}\)=\\sum\_\{i\}\\lambda\_\{i\}\\,r\_\{i\}\(\{\\bm\{x\}\}\_\{0\},\{\\bm\{c\}\}\)with hand\-setλi≥0\\lambda\_\{i\}\\geq 0,∑iλi=1\\sum\_\{i\}\\lambda\_\{i\}=1\. Over\-optimizing any single proxy can easily lead to reward hacking\[[8](https://arxiv.org/html/2609.13425#bib.bib32),[34](https://arxiv.org/html/2609.13425#bib.bib33)\]\.

However, static scalarization makes an implicit but imperfect assumption: once a reward is assigned an overall budgetλi\\lambda\_\{i\}, its relative influence remains the same across all diffusion timesteps\. Instead, we want a reward to have greater influence when it is more informative\. Multi\-reward alignment therefore involves two allocation problems: how much budget each reward receives, and when that budget should be spent\. More specifically, given a user\-specified reward budget𝝀\\bm\{\\lambda\}, can we allocate each reward’s weights across timesteps more efficiently?

Our key intuition is that each reward measures a different visual property, and those properties do not become visible at the same point in the denoising trajectory\. A reward for prompt alignment or rule\-based correctness\[[12](https://arxiv.org/html/2609.13425#bib.bib24),[9](https://arxiv.org/html/2609.13425#bib.bib7)\]can already grade a sample once coarse layout and composition exist, while a reward for aesthetic or fine local detail cannot say much until the sample is nearly clean\. Diffusion timesteps govern different aspects of generation by construction, with high\-noise steps fixing global composition and low\-noise steps refining local detail\[[2](https://arxiv.org/html/2609.13425#bib.bib30),[10](https://arxiv.org/html/2609.13425#bib.bib29)\]\. A static, timestep\-independentλi\\lambda\_\{i\}ignores this and forces every reward to contribute uniformly acrossttregardless of whether it can discriminate there\. Although timestep\-dependent treatment is well established for likelihood training\[[17](https://arxiv.org/html/2609.13425#bib.bib23),[15](https://arxiv.org/html/2609.13425#bib.bib28)\]and reward fine\-tuning\[[21](https://arxiv.org/html/2609.13425#bib.bib14),[20](https://arxiv.org/html/2609.13425#bib.bib4),[11](https://arxiv.org/html/2609.13425#bib.bib35)\], these methods apply a single reward\-agnostic schedule, not the per\-reward one we study\.

Credit assignment is widely analyzed for language\-model reasoning\[[36](https://arxiv.org/html/2609.13425#bib.bib41),[22](https://arxiv.org/html/2609.13425#bib.bib42),[16](https://arxiv.org/html/2609.13425#bib.bib43)\]but largely unexplored in the diffusion denoising chain\. We propose ReCAST \(RewardCreditASsignment acrossTimesteps\), the first per\-reward timestep\-dependent credit assignment method for diffusion reward fine\-tuning, to our knowledge\. More specifically, for rewardsr1,…,rmr\_\{1\},\\dots,r\_\{m\}andTTsteps, we replace the static convex weighting by an*automatically*designed weight matrix𝑾∈ℝ≥0m×T\\bm\{W\}\\in\\mathbb\{R\}\_\{\\geq 0\}^\{m\\times T\}, whose entryWi,tW\_\{i,t\}is the weight rewardiicarries at steptt, and define the reward at timestepttas

rt​\(𝒙0,𝒄\)=∑i=1mT⋅Wi,t​ri​\(𝒙0,𝒄\),∑tWi,t=λi⏟inter\-reward row budget,∑iWi,t=1/T⏟per\-step column budget\.\\boxed\{\\;r\_\{t\}\(\{\\bm\{x\}\}\_\{0\},\{\\bm\{c\}\}\)\\;=\\;\\sum\_\{i=1\}^\{m\}T\\cdot W\_\{i,t\}\\;r\_\{i\}\(\{\\bm\{x\}\}\_\{0\},\{\\bm\{c\}\}\),\\qquad\\underbrace\{\\textstyle\\sum\_\{t\}W\_\{i,t\}=\\lambda\_\{i\}\}\_\{\\begin\{subarray\}\{c\}\\text\{inter\-reward row budget\}\\end\{subarray\}\},\\qquad\\underbrace\{\\textstyle\\sum\_\{i\}W\_\{i,t\}=1/T\}\_\{\\begin\{subarray\}\{c\}\\text\{per\-step column budget\}\\end\{subarray\}\}\.\\;\}\(1\)The row marginalλi≥0\\lambda\_\{i\}\\geq 0\(∑iλi=1\\sum\_\{i\}\\lambda\_\{i\}=1\) is the inter\-reward budget set by the user, and states*how much*rewardiicounts\. The column marginal enforces that every step receives the same total weight1/T1/T, so no step is starved or over\-optimized\. Together the two marginals fix the totals but not the shape: how rowiispreads its budgetλi\\lambda\_\{i\}acrossttis precisely*when*rewardiicounts, and that is what we estimate rather than hand\-set\. We first measure each reward’s own per\-step gain in Rényi discriminability along the trajectory \(Sec\.[3\.2](https://arxiv.org/html/2609.13425#S3.SS2)–[3\.3](https://arxiv.org/html/2609.13425#S3.SS3)\), then build a kernel from these gains and project it onto the feasible set of Eq\. \([1](https://arxiv.org/html/2609.13425#S1.E1)\) to give the desired𝑾⋆\\bm\{W\}^\{\\star\}\(Sec\.[3\.4](https://arxiv.org/html/2609.13425#S3.SS4)\)\.

## 2Related work

#### RL in diffusion and flow models\.

Reward fine\-tuning is central to current text\-to\-image models\. DanceGRPO\[[41](https://arxiv.org/html/2609.13425#bib.bib2)\]extends group\-relative optimization to both diffusion and rectified\-flow image and video generators within a single framework; Flow\-GRPO\[[24](https://arxiv.org/html/2609.13425#bib.bib1)\]converts the flow\-matching ODE into an SDE so group sampling and its advantage estimator stay well defined; DiffusionNFT\[[42](https://arxiv.org/html/2609.13425#bib.bib25)\]regresses implicit positive/negative velocity fields with a forward\-only loss\. Our work is built upon DiffusionNFT due to its performance and efficiency\.

#### Combining multiple rewards\.

How multiple reward signals are combined has received wide attention in areas not limited to text\-to\-image tasks\. SafeRLHF\[[5](https://arxiv.org/html/2609.13425#bib.bib37)\]decouples helpfulness and harmlessness into separate reward/cost models balanced via a Lagrange multiplier rather than a fixed scalar; Personalized Soups\[[14](https://arxiv.org/html/2609.13425#bib.bib38)\]instead trains one policy per preference dimension and merges them post\-hoc, avoiding scalarization altogether, and Rewarded Soups\[[30](https://arxiv.org/html/2609.13425#bib.bib34)\]takes the same merging route for reward proxies to reach Pareto trade\-offs by interpolating weights\. ALaRM\[[19](https://arxiv.org/html/2609.13425#bib.bib39)\]organizes rewards hierarchically rather than flattening them into one sum, and large reasoning models combine a rule\-based outcome reward, a length penalty, and a language consistency reward within a single objective\. GDPO\[[25](https://arxiv.org/html/2609.13425#bib.bib36)\]shows that summing rewards before GRPO’s group normalization collapses into near\-identical advantages, and fixes this by normalizing per reward before recombining\. None of this work varies a reward’s weight across denoising timesteps: our contribution is orthogonal to how the mixture itself is formed\.

#### Timestep weighting in diffusion training\.

Existing works have adopted unequal treatment of denoising steps in diffusion training, such as ELBO\-consistent weightings\[[35](https://arxiv.org/html/2609.13425#bib.bib22),[17](https://arxiv.org/html/2609.13425#bib.bib23)\], perception\-prioritized and Min\-SNR schedules\[[2](https://arxiv.org/html/2609.13425#bib.bib30),[10](https://arxiv.org/html/2609.13425#bib.bib29)\], EDM weighting\[[15](https://arxiv.org/html/2609.13425#bib.bib28)\], and importance sampling oftt\[[28](https://arxiv.org/html/2609.13425#bib.bib31)\]\. On the reward side, step\-by\-step preference optimization\[[21](https://arxiv.org/html/2609.13425#bib.bib14)\]supervises every denoising step separately through a step\-aware preference model, while MixGRPO’s sliding ODE/SDE window\[[20](https://arxiv.org/html/2609.13425#bib.bib4)\]and TempFlow\-GRPO’s noise\-aware weighting\[[11](https://arxiv.org/html/2609.13425#bib.bib35)\]concentrate the reward signal on high\-noise steps\. All show that*when*a signal is applied matters, but each shares one heuristic curve across rewards; ours is per\-reward, from its own discriminability curve\.

## 3Method

This section builds the per\-step gain in reward discriminability, then turns those gains into the weight matrix𝑾\\bm\{W\}\. In Sec\.[3\.2](https://arxiv.org/html/2609.13425#S3.SS2)–[3\.3](https://arxiv.org/html/2609.13425#S3.SS3)we look at one reward, asking only how the per\-step gain is shaped acrosstt; Sec\.[3\.4](https://arxiv.org/html/2609.13425#S3.SS4)then puts the per\-reward shapes into a matrix that satisfies both marginals of Eq\. \([1](https://arxiv.org/html/2609.13425#S1.E1)\)\.

We work in the rectified\-flow setup and with the DiffusionNFT objective, and put the background in App\.[A\.1](https://arxiv.org/html/2609.13425#A1.SS1)\. Throughout this section we assumeri≥0r\_\{i\}\\geq 0\.

### 3\.1Setup and the density ratio

For each rewardrir\_\{i\},i∈\{1,…,m\}i\\in\\\{1,\\dots,m\\\}, letπi\+​\(𝒙0∣𝒄\)\\pi\_\{i\}^\{\+\}\(\{\\bm\{x\}\}\_\{0\}\\mid\{\\bm\{c\}\}\)be the positive policy induced by rewardiion the old policyπold\\pi^\{\\rm old\},

πi\+​\(𝒙0∣𝒄\):=ri​\(𝒙0,𝒄\)𝔼πold​\(𝒙0∣𝒄\)​\[ri​\(𝒙0,𝒄\)\]​πold​\(𝒙0∣𝒄\)\\pi\_\{i\}^\{\+\}\(\{\\bm\{x\}\}\_\{0\}\\mid\{\\bm\{c\}\}\):=\\frac\{r\_\{i\}\(\{\\bm\{x\}\}\_\{0\},\{\\bm\{c\}\}\)\}\{\\mathbb\{E\}\_\{\\pi^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{0\}\\mid\{\\bm\{c\}\}\)\}\[r\_\{i\}\(\{\\bm\{x\}\}\_\{0\},\{\\bm\{c\}\}\)\]\}\\,\\pi^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{0\}\\mid\{\\bm\{c\}\}\)\(2\)and letπi,t\+​\(𝒙t∣𝒄\)\\pi\_\{i,t\}^\{\+\}\(\{\\bm\{x\}\}\_\{t\}\\mid\{\\bm\{c\}\}\)andπtold​\(𝒙t∣𝒄\)\\pi\_\{t\}^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{t\}\\mid\{\\bm\{c\}\}\)be the marginals at diffusion timesteptt\. Define the density ratio ofrir\_\{i\},ρi,t​\(𝒙t\):=πi,t\+​\(𝒙t∣𝒄\)/πtold​\(𝒙t∣𝒄\)\\rho\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\):=\{\\pi\_\{i,t\}^\{\+\}\(\{\\bm\{x\}\}\_\{t\}\\mid\{\\bm\{c\}\}\)\}/\{\\pi\_\{t\}^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{t\}\\mid\{\\bm\{c\}\}\)\}\. Marginalizing the forward kernel and applying Bayes’ rule,πold​\(𝒙0∣𝒙t,𝒄\)=p⁡\(𝒙t∣𝒙0\)​πold​\(𝒙0∣𝒄\)/πtold​\(𝒙t∣𝒄\)\\pi^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{0\}\\mid\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\}\)=p\(\{\\bm\{x\}\}\_\{t\}\\mid\{\\bm\{x\}\}\_\{0\}\)\\pi^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{0\}\\mid\{\\bm\{c\}\}\)\\,/\\,\\pi\_\{t\}^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{t\}\\mid\{\\bm\{c\}\}\), gives the ratio at timestepttas a posterior expected reward normalized by the marginal one:

ρi,t​\(𝒙t\):=πi,t\+​\(𝒙t∣𝒄\)πtold​\(𝒙t∣𝒄\)=∫p⁡\(𝒙t∣𝒙0\)​πi\+​\(𝒙0∣𝒄\)​d​𝒙0πtold​\(𝒙t∣𝒄\)=𝔼πold​\(𝒙0∣𝒙t,𝒄\)​\[ri​\(𝒙0,𝒄\)\]𝔼πold​\(𝒙0∣𝒄\)​\[ri​\(𝒙0,𝒄\)\]\.\{\\;\\rho\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)\\;:=\\;\\frac\{\\pi\_\{i,t\}^\{\+\}\(\{\\bm\{x\}\}\_\{t\}\\mid\{\\bm\{c\}\}\)\}\{\\pi\_\{t\}^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{t\}\\mid\{\\bm\{c\}\}\)\}\\;=\\;\\frac\{\\int p\(\{\\bm\{x\}\}\_\{t\}\\mid\{\\bm\{x\}\}\_\{0\}\)\\,\\pi\_\{i\}^\{\+\}\(\{\\bm\{x\}\}\_\{0\}\\mid\{\\bm\{c\}\}\)\\,\{\\rm d\}\{\\bm\{x\}\}\_\{0\}\}\{\\pi\_\{t\}^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{t\}\\mid\{\\bm\{c\}\}\)\}\\;=\\;\\frac\{\\mathbb\{E\}\_\{\\pi^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{0\}\\mid\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\}\)\}\\\!\\big\[r\_\{i\}\(\{\\bm\{x\}\}\_\{0\},\{\\bm\{c\}\}\)\\big\]\}\{\\mathbb\{E\}\_\{\\pi^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{0\}\\mid\{\\bm\{c\}\}\)\}\\\!\\big\[r\_\{i\}\(\{\\bm\{x\}\}\_\{0\},\{\\bm\{c\}\}\)\\big\]\}\.\\;\}\(3\)Everything on the right\-hand side is a reward evaluation under the current policy, so the ratio is theoretically computable without ever directly sampling fromπi\+\\pi\_\{i\}^\{\+\}\. For simplicity, we writeμi,t​\(𝒙t\):=𝔼πold​\(𝒙0∣𝒙t,𝒄\)​\[ri\]\\mu\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\):=\\mathbb\{E\}\_\{\\pi^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{0\}\\mid\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\}\)\}\[r\_\{i\}\]for the conditional reward andZi:=𝔼πold​\(𝒙0∣𝒄\)​\[ri\]Z\_\{i\}:=\\mathbb\{E\}\_\{\\pi^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{0\}\\mid\{\\bm\{c\}\}\)\}\[r\_\{i\}\]for its marginal, soρi,t​\(𝒙t\)=μi,t​\(𝒙t\)/Zi\\rho\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)=\\mu\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)/Z\_\{i\}\.

### 3\.2Reward discriminability

We first ask one question: how distinguishable is the positive policy from the current one at a given timestep? We measure this by the Rényi divergence\[[31](https://arxiv.org/html/2609.13425#bib.bib3)\]of orderα\>1\\alpha\>1\. The*cumulative reward discriminability*at timestepttfor rewardiiis therefore

Di,t:=Dα\(πi,t\+∥πtold\)=1α−1log𝔼πi,t\+\[ρi,t\(𝒙t\)α−1\]\.D\_\{i,t\}\\;:=\\;D\_\{\\alpha\}\\\!\\big\(\\pi\_\{i,t\}^\{\+\}\\,\\big\\\|\\,\\pi\_\{t\}^\{\\mathrm\{old\}\}\\big\)\\;=\\;\\frac\{1\}\{\\alpha\-1\}\\log\\mathbb\{E\}\_\{\\pi\_\{i,t\}^\{\+\}\}\\\!\\Big\[\\rho\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)^\{\\alpha\-1\}\\Big\]\.\(4\)It vanishes exactly when the two marginals agree, grows as rewardiiseparates them, and is nondecreasing inα\\alpha\. Itsα→1\+\\alpha\\to 1^\{\+\}limit is the KL divergence between the same two marginals,

limα→1\+Di,t=KL\(πi,t\+∥πtold\)=𝔼πi,t\+\[logρi,t\(𝒙t\)\]\.\\lim\_\{\\alpha\\to 1^\{\+\}\}D\_\{i,t\}\\;=\\;\\mathrm\{KL\}\\big\(\\pi\_\{i,t\}^\{\+\}\\,\\big\\\|\\,\\pi\_\{t\}^\{\\mathrm\{old\}\}\\big\)\\;=\\;\\mathbb\{E\}\_\{\\pi\_\{i,t\}^\{\+\}\}\\\!\\big\[\\log\\rho\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)\\big\]\.\(5\)Sec\.[3\.3](https://arxiv.org/html/2609.13425#S3.SS3)shows why we choose Rényi instead of the KL divergence\.

At pure noise \(t=Tt=T\) the forward map hasσT=1\\sigma\_\{T\}=1and discards𝒙0\{\\bm\{x\}\}\_\{0\}entirely, soπi,T\+=πTold=𝒩⁡\(0,I\)\\pi\_\{i,T\}^\{\+\}=\\pi\_\{T\}^\{\\mathrm\{old\}\}=\\mathcal\{N\}\(0,I\)andDi,T=0D\_\{i,T\}=0for every reward\. At clean data the divergence reaches its maximumDi,0=Dα\(πi,0\+∥π0old\)D\_\{i,0\}=D\_\{\\alpha\}\(\\pi\_\{i,0\}^\{\+\}\\,\\\|\\,\\pi\_\{0\}^\{\\mathrm\{old\}\}\), the full data\-level discriminability rewardiiinduces\.

#### Discriminability gain\.

Cumulative discriminabilityDi,tD\_\{i,t\}contains all signal accumulated fromTTtott, so weighting by it would repeatedly count earlier gains\. Therefore, we use the per\-step*gain*

Δ​Di,t:=Di,t−1−Di,t,\{\\Delta D\_\{i,t\}:=D\_\{i,t\-1\}\-D\_\{i,t\},\}\(6\)which isolates the new discriminability contributed by rewardiiat transitiont→t−1t\\to t\-1\. The gains telescope along the denoising trajectory:

∑t=1TΔDi,t=Di,0−Di,T=Di,0=Dα\(πi,0\+∥π0old\)\.\{\\sum\_\{t=1\}^\{T\}\\Delta D\_\{i,t\}=D\_\{i,0\}\-D\_\{i,T\}=D\_\{i,0\}=D\_\{\\alpha\}\\\!\\big\(\\pi\_\{i,0\}^\{\+\}\\,\\big\\\|\\,\\pi\_\{0\}^\{\\mathrm\{old\}\}\\big\)\.\}\(7\)Thus,Δ​Di,t\\Delta D\_\{i,t\}decomposes the reward’s total data\-level discriminability across timesteps without double\-counting: rewards sensitive to global structure concentrate at largett, while those sensitive to fine details concentrate at smalltt\.

One basic property is thatΔ​Di,t≥0\\Delta D\_\{i,t\}\\geq 0, so at each denoising step \(t→t−1t\\to t\-1\) the discriminability does not decrease\. This ensures the soundness of our definition ofΔ​Di,t\\Delta D\_\{i,t\}\. We prove a general theorem\.

###### Theorem 3\.1\(Rényi dissipation under shared diffusion\)\.

Letπta\\pi\_\{t\}^\{a\}andπtb\\pi\_\{t\}^\{b\}be positive, sufficiently smooth probability densities onℝd\\mathbb\{R\}^\{d\}that both evolve under the same forward Fokker–Planck equation

∂tπt=−∇⋅\(𝒖\(𝒙,t\)πt\)\+g​\(t\)22Δπt,\\partial\_\{t\}\\pi\_\{t\}\\;=\\;\-\\nabla\\\!\\cdot\\\!\\big\(\{\\bm\{u\}\}\(\{\\bm\{x\}\},t\)\\,\\pi\_\{t\}\\big\)\\;\+\\;\\tfrac\{g\(t\)^\{2\}\}\{2\}\\,\\Delta\\pi\_\{t\},\(8\)for a shared drift𝐮\{\\bm\{u\}\}and spatially constant diffusion coefficientg⁡\(t\)g\(t\)\. Assume the displayed moments and derivatives are integrable and that the boundary terms in the integrations by parts vanish\. Letρ:=πa/πb\\rho:=\\pi^\{a\}/\\pi^\{b\}denote the density ratio, and define the*Rényi\-tilted distribution*of orderα\\alpha:

πt\(α\)​\(𝒙\):=ρ​\(𝒙\)α𝔼πtb​\[ρα\]​πtb​\(𝒙\)∝\(πta\)α​\(πtb\)1−α\.\\pi\_\{t\}^\{\(\\alpha\)\}\(\{\\bm\{x\}\}\)\\;:=\\;\\frac\{\\rho\(\{\\bm\{x\}\}\)^\{\\alpha\}\}\{\\mathbb\{E\}\_\{\\pi\_\{t\}^\{b\}\}\[\\rho^\{\\alpha\}\]\}\\,\\pi\_\{t\}^\{b\}\(\{\\bm\{x\}\}\)\\;\\propto\\;\\big\(\\pi\_\{t\}^\{a\}\\big\)^\{\\alpha\}\\,\\big\(\\pi\_\{t\}^\{b\}\\big\)^\{1\-\\alpha\}\.\(9\)This geometric family recoversπtb\\pi\_\{t\}^\{b\}atα=0\\alpha=0andπta\\pi\_\{t\}^\{a\}atα=1\\alpha=1; in the regime used here,α\>1\\alpha\>1, it extrapolates beyondπta\\pi\_\{t\}^\{a\}\. Then for everyα\>0\\alpha\>0withα≠1\\alpha\\neq 1,

dd​tDα\(πta∥πtb\)=−g​\(t\)22⋅α⋅𝔼πt\(α\)\[∥∇logπaπb∥2\],\{\\;\\frac\{\{\\rm d\}\}\{\{\\rm d\}t\}\\,D\_\{\\alpha\}\\\!\\big\(\\pi\_\{t\}^\{a\}\\,\\\|\\,\\pi\_\{t\}^\{b\}\\big\)\\;=\\;\-\\,\\frac\{g\(t\)^\{2\}\}\{2\}\\cdot\\alpha\\cdot\\mathbb\{E\}\_\{\\pi\_\{t\}^\{\(\\alpha\)\}\}\\\!\\big\[\\\|\\nabla\\log\\frac\{\\pi^\{a\}\}\{\\pi^\{b\}\}\\\|^\{2\}\\big\],\\;\}\(10\)where𝔼πt\(α\)​\[‖∇log⁡πaπb‖2\]\\mathbb\{E\}\_\{\\pi\_\{t\}^\{\(\\alpha\)\}\}\[\\\|\\nabla\\log\\frac\{\\pi^\{a\}\}\{\\pi^\{b\}\}\\\|^\{2\}\]is the Fisher information of the log\-density ratio measured under the tilted distributionπt\(α\)\\pi\_\{t\}^\{\(\\alpha\)\}\.

The proof of the theorem is deferred to App\.[A\.2](https://arxiv.org/html/2609.13425#A1.SS2)\. Bothπi,t\+\\pi\_\{i,t\}^\{\+\}andπtold\\pi\_\{t\}^\{\\mathrm\{old\}\}evolve according to Eq\. \([8](https://arxiv.org/html/2609.13425#S3.E8)\) with the same coefficients because the diffusion kernelp⁡\(𝒙t∣𝒙0\)p\(\{\\bm\{x\}\}\_\{t\}\\mid\{\\bm\{x\}\}\_\{0\}\)is independent of the policy\. Away from the singular endpointσt=1\\sigma\_\{t\}=1, the forward process𝒙t=\(1−σt\)​𝒙0\+σt​ϵ\{\\bm\{x\}\}\_\{t\}=\(1\-\\sigma\_\{t\}\)\{\\bm\{x\}\}\_\{0\}\+\\sigma\_\{t\}\{\\bm\{\\epsilon\}\}is the transition law of a linear SDE, and therefore satisfies Eq\. \([8](https://arxiv.org/html/2609.13425#S3.E8)\) with𝒖⁡\(𝒙,t\)=−σ˙t1−σt​𝒙,g​\(t\)2=2​σt​σ˙t1−σt\{\\bm\{u\}\}\(\{\\bm\{x\}\},t\)=\-\\frac\{\\dot\{\\sigma\}\_\{t\}\}\{1\-\\sigma\_\{t\}\}\{\\bm\{x\}\},\\qquad g\(t\)^\{2\}=\\frac\{2\\sigma\_\{t\}\\dot\{\\sigma\}\_\{t\}\}\{1\-\\sigma\_\{t\}\}\. Since Eq\. \([8](https://arxiv.org/html/2609.13425#S3.E8)\) is linear inπt\\pi\_\{t\}, and the reward tiltrir\_\{i\}acts only on the initial state𝒙0\{\\bm\{x\}\}\_\{0\}, the tilt changes only the initial distribution rather than the evolution coefficients\. Hence,πi,t\+\\pi\_\{i,t\}^\{\+\}andπtold\\pi\_\{t\}^\{\\mathrm\{old\}\}follow the same Fokker–Planck dynamics from different initial conditions\. Theorem[3\.1](https://arxiv.org/html/2609.13425#S3.Thmtheorem1)therefore applies withπa=πi,t\+\\pi^\{a\}=\\pi\_\{i,t\}^\{\+\},πb=πtold\\pi^\{b\}=\\pi\_\{t\}^\{\\mathrm\{old\}\}, andρ=ρi,t\\rho=\\rho\_\{i,t\}, yieldingdd​tDα\(πi,t\+∥πtold\)≤0\\frac\{\{\\rm d\}\}\{\{\\rm d\}t\}D\_\{\\alpha\}\\\!\\big\(\\pi\_\{i,t\}^\{\+\}\\,\\\|\\,\\pi\_\{t\}^\{\\mathrm\{old\}\}\\big\)\\leq 0\.

### 3\.3Estimation details

#### Why we useα\>1\\alpha\>1rather than the KL limit\.

The exact KL in Eq\. \([5](https://arxiv.org/html/2609.13425#S3.E5)\) is well defined under the usual absolute\-continuity and integrability conditions: states withρi,t=πi,t\+/πtold=0\\rho\_\{i,t\}=\\pi\_\{i,t\}^\{\+\}/\\pi^\{\\rm old\}\_\{t\}=0have zero probability underπi,t\+\\pi\_\{i,t\}^\{\+\}and therefore do not cause the expectation to diverge\. However, empirically in finite\-sample rollouts, the estimated conditional reward𝔼^πold​\(𝒙0∣𝒙t,𝒄\)​\[ri​\(𝒙0,𝒄\)\]\\hat\{\\mathbb\{E\}\}\_\{\\pi^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{0\}\\mid\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\}\)\}\\big\[r\_\{i\}\(\{\\bm\{x\}\}\_\{0\},\{\\bm\{c\}\}\)\\big\], and henceρ^i,t\\hat\{\\rho\}\_\{i,t\}, can be exactly zero, in which case averaginglog⁡ρ^i,t\\log\\hat\{\\rho\}\_\{i,t\}introduces−∞\-\\inftyvalues\. For example, consider a noisy state𝒙t\{\\bm\{x\}\}\_\{t\}that has already committed to a two\-object layout when the prompt requires only one\. Such an𝒙t\{\\bm\{x\}\}\_\{t\}will likely be denoised underπold\\pi^\{\\rm old\}into samples𝒙0\{\\bm\{x\}\}\_\{0\}that keep that structure, sori​\(𝒙0,𝒄\)=0r\_\{i\}\(\{\\bm\{x\}\}\_\{0\},\{\\bm\{c\}\}\)=0for multiple𝒙0\{\\bm\{x\}\}\_\{0\}\. Simply swapping the two marginals does not resolve this issue:KL\(πtold∥πi,t\+\)=−𝔼πtold\[logρi,t\(𝒙t\)\]\\mathrm\{KL\}\\big\(\\pi\_\{t\}^\{\\rm old\}\\,\\big\\\|\\,\\pi\_\{i,t\}^\{\+\}\\big\)=\-\\mathbb\{E\}\_\{\\pi\_\{t\}^\{\\rm old\}\}\\big\[\\log\\rho\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)\\big\]may also diverge for the same reason\.

This motivates using the Rényi divergence withα\>1\\alpha\>1\. In this case, the logarithm in Eq\. \([4](https://arxiv.org/html/2609.13425#S3.E4)\) is applied only after the empirical moment𝔼^πi,t\+​\[ρ^i,t​\(𝒙t\)α−1\]\\hat\{\\mathbb\{E\}\}\_\{\\pi\_\{i,t\}^\{\+\}\}\\big\[\\hat\{\\rho\}\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)^\{\\alpha\-1\}\\big\]\. Consequently, a zero conditional reward estimate, corresponding to a state𝒙t\{\\bm\{x\}\}\_\{t\}for whichρ^i,t​\(𝒙t\)=0\\hat\{\\rho\}\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)=0, contributes zero to the empirical moment rather than producing an infinite logarithmic term\. The estimate therefore remains finite whenever the empirical moment itself is positive, which is usually satisfied in practical estimation\. At the same time,DαD\_\{\\alpha\}retains a useful relation to the original objective: because Rényi divergence is nondecreasing in its order,DαD\_\{\\alpha\}upper\-bounds the KL forα\>1\\alpha\>1\. App\.[B\.2](https://arxiv.org/html/2609.13425#A2.SS2)examines how the resulting discriminability curve varies withα\\alpha\. We useα=2\\alpha=2in all training experiments for its simplicity: the exponent in Eq\. \([4](https://arxiv.org/html/2609.13425#S3.E4)\) is thenα−1=1\\alpha\-1=1\.

#### Importance sampling fromπ\+\\pi^\{\+\}to cover high\-reward regimes\.

Revisiting Eq\. \([4](https://arxiv.org/html/2609.13425#S3.E4)\), we find one problem: sampling𝒙t\{\\bm\{x\}\}\_\{t\}fromπi,t\+\\pi\_\{i,t\}^\{\+\}is impractical since we cannot directly sample from theπi\+\\pi\_\{i\}^\{\+\}of Eq\. \([2](https://arxiv.org/html/2609.13425#S3.E2)\)\. One seemingly plausible solution is to sample fromπold\\pi^\{\\rm old\}instead: sinceρi,t=πi,t\+/πtold\\rho\_\{i,t\}=\\pi\_\{i,t\}^\{\+\}/\\pi\_\{t\}^\{\\rm old\}, we have𝔼πtold​\[ρi,t​\(𝒙t\)α\]=𝔼πi,t\+​\[ρi,t​\(𝒙t\)α−1\]\\mathbb\{E\}\_\{\\pi\_\{t\}^\{\\mathrm\{old\}\}\}\\\!\\left\[\\rho\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)^\{\\alpha\}\\right\]\\;=\\;\\mathbb\{E\}\_\{\\pi\_\{i,t\}^\{\+\}\}\\\!\\left\[\\rho\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)^\{\\alpha\-1\}\\right\]\. This means we draw𝒙~0∼πold\(⋅∣𝒄\)\\widetilde\{\{\\bm\{x\}\}\}\_\{0\}\\sim\\pi^\{\\rm old\}\(\\cdot\\mid\{\\bm\{c\}\}\), forward\-noise it to𝒙t∼πtold\{\\bm\{x\}\}\_\{t\}\\sim\\pi^\{\\rm old\}\_\{t\}by𝒙t=\(1−σt\)​𝒙~0\+σt​ϵ\{\\bm\{x\}\}\_\{t\}=\(1\-\\sigma\_\{t\}\)\\widetilde\{\{\\bm\{x\}\}\}\_\{0\}\+\\sigma\_\{t\}\{\\bm\{\\epsilon\}\}, and roll out multiple𝒙0\{\\bm\{x\}\}\_\{0\}from𝒙t\{\\bm\{x\}\}\_\{t\}underπold\\pi^\{\\mathrm\{old\}\}\. Here,𝒙0\{\\bm\{x\}\}\_\{0\}is used to estimateμi,t​\(𝒙t\)\\mu\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)and𝒙~0\\widetilde\{\{\\bm\{x\}\}\}\_\{0\}is used to estimateZiZ\_\{i\}\. However, if the current policy \(πold\\pi\_\{\\rm old\}\) often produces low\-quality outputs, thenr⁡\(𝒙0,𝒄\)≈r⁡\(𝒙~0,𝒄\)r\(\{\\bm\{x\}\}\_\{0\},\{\\bm\{c\}\}\)\\approx r\(\\widetilde\{\{\\bm\{x\}\}\}\_\{0\},\{\\bm\{c\}\}\), soρi,t​\(𝒙t\)=μi,t​\(𝒙t\)/Zi≈1\\rho\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)=\\mu\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)/Z\_\{i\}\\approx 1andDi,t≈0D\_\{i,t\}\\approx 0: the high\-reward regions are unestimated\. The leftmost column of Fig\.[3](https://arxiv.org/html/2609.13425#A2.F3)shows exactly this: underπold\\pi^\{\\rm old\}the estimatedDi,tD\_\{i,t\}curves forClipScore,HPSv2, andPickScoreare close to00and significantly lower than theirπ\+\\pi^\{\+\}counterparts \(App\.[B\.3](https://arxiv.org/html/2609.13425#A2.SS3)\)\.

Therefore, we propose to use a single stronger external generatorπ\+\\pi^\{\+\}as a common proposal for all of theπi\+\\pi\_\{i\}^\{\+\}: we draw𝒙~0∼π\+\(⋅∣𝒄\)\\widetilde\{\{\\bm\{x\}\}\}\_\{0\}\\sim\\pi^\{\+\}\(\\cdot\\mid\{\\bm\{c\}\}\), forward\-noise it to𝒙t∼πt\+\{\\bm\{x\}\}\_\{t\}\\sim\\pi\_\{t\}^\{\+\}by𝒙t=\(1−σt\)​𝒙~0\+σt​ϵ\{\\bm\{x\}\}\_\{t\}=\(1\-\\sigma\_\{t\}\)\\widetilde\{\{\\bm\{x\}\}\}\_\{0\}\+\\sigma\_\{t\}\{\\bm\{\\epsilon\}\}, and roll out𝒙0\{\\bm\{x\}\}\_\{0\}from𝒙t\{\\bm\{x\}\}\_\{t\}underπold\\pi^\{\\mathrm\{old\}\}\. Since we assumeπ\+\\pi^\{\+\}generates higher\-quality outputs thanπold\\pi^\{\\rm old\},ρi,t​\(𝒙t\)=μi,t​\(𝒙t\)/Zi\\rho\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)=\\mu\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)/Z\_\{i\}will decrease to 1 asttincreases\. Because the ratioπi,t\+/πt\+\\pi\_\{i,t\}^\{\+\}/\\pi\_\{t\}^\{\+\}is unknown, this is not an exact importance\-sampling estimator ofDi,tD\_\{i,t\}but a surrogate log\-moment\. App\.[A\.3](https://arxiv.org/html/2609.13425#A1.SS3)analyzes the error the surrogate induces: whenKL\(π\+∥πi\+\)\\mathrm\{KL\}\(\\pi^\{\+\}\\\|\\pi\_\{i\}^\{\+\}\)is small, the error in the discriminability gain isO⁡\(KL\(π\+∥πi\+\)\)O\(\\sqrt\{\\mathrm\{KL\}\(\\pi^\{\+\}\\\|\\pi\_\{i\}^\{\+\}\)\}\)and propagates to the Sinkhorn weights\. Furthermore, replacingπ\+\\pi^\{\+\}withπold\\pi^\{\\mathrm\{old\}\}self\-rollouts yields far noisier curves in App\.[B\.3](https://arxiv.org/html/2609.13425#A2.SS3), while choosing a differentπ\+\\pi^\{\+\}, eitherGPT Image 1\.5orNano Banana Pro, leaves the estimated curves essentially unchanged in App\.[B\.4](https://arxiv.org/html/2609.13425#A2.SS4)\.

#### Denominator cancellation\.

Puttingρi,t​\(𝒙t\)=μi,t​\(𝒙t\)/Zi\\rho\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)=\\mu\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)/Z\_\{i\}into Eq\. \([4](https://arxiv.org/html/2609.13425#S3.E4)\), we haveDi,t=1α−1​log⁡𝔼πi,t\+​\[μi,t​\(𝒙t\)α−1\]−log⁡ZiD\_\{i,t\}=\\frac\{1\}\{\\alpha\-1\}\\log\\mathbb\{E\}\_\{\\pi\_\{i,t\}^\{\+\}\}\[\\mu\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)^\{\\alpha\-1\}\]\-\\log Z\_\{i\}\. Because only the*gain*Δ​Di,t=Di,t−1−Di,t\\Delta D\_\{i,t\}=D\_\{i,t\-1\}\-D\_\{i,t\}is needed,log⁡Zi\\log Z\_\{i\}cancels entirely:

Δ​Di,t=1α−1​\(log⁡𝔼πi,t−1\+​\[μi,t−1​\(𝒙t−1\)α−1\]−log⁡𝔼πi,t\+​\[μi,t​\(𝒙t\)α−1\]\)\.\\boxed\{\\,\\Delta D\_\{i,t\}=\\frac\{1\}\{\\alpha\-1\}\\Big\(\\log\\mathbb\{E\}\_\{\\pi\_\{i,t\-1\}^\{\+\}\}\\\!\\big\[\\mu\_\{i,t\-1\}\(\{\\bm\{x\}\}\_\{t\-1\}\)^\{\\alpha\-1\}\\big\]\-\\log\\mathbb\{E\}\_\{\\pi\_\{i,t\}^\{\+\}\}\\\!\\big\[\\mu\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)^\{\\alpha\-1\}\\big\]\\Big\)\.\\,\}\(11\)Replacing the unknown positive marginals by the common proposalπt\+\\pi\_\{t\}^\{\+\}leaves the quantity we actually compute,

Δ​D~i,t=1α−1​\(log⁡𝔼πt−1\+​\[μi,t−1​\(𝒙t−1\)α−1\]−log⁡𝔼πt\+​\[μi,t​\(𝒙t\)α−1\]\),\\widetilde\{\\Delta D\}\_\{i,t\}=\\frac\{1\}\{\\alpha\-1\}\\Big\(\\log\\mathbb\{E\}\_\{\\pi\_\{t\-1\}^\{\+\}\}\\\!\\big\[\\mu\_\{i,t\-1\}\(\{\\bm\{x\}\}\_\{t\-1\}\)^\{\\alpha\-1\}\\big\]\-\\log\\mathbb\{E\}\_\{\\pi\_\{t\}^\{\+\}\}\\\!\\big\[\\mu\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)^\{\\alpha\-1\}\\big\]\\Big\),\(12\)which coincides with Eq\. \([11](https://arxiv.org/html/2609.13425#S3.E11)\) only whenπt\+=πi,t\+\\pi\_\{t\}^\{\+\}=\\pi\_\{i,t\}^\{\+\}\. Onlyμi,t\\mu\_\{i,t\}has to be estimated, by rollouts underπold\\pi^\{\\mathrm\{old\}\}, andZiZ\_\{i\}is never formed\. Alg\.[1](https://arxiv.org/html/2609.13425#alg1)is the resulting estimator, in compact form; App\.[B\.1](https://arxiv.org/html/2609.13425#A2.SS1)writes out every index\. For readability in the empirical sections we write the outputΔ​D~i,t^\\widehat\{\\widetilde\{\\Delta D\}\_\{i,t\}\}simply asΔ​Di,t\\Delta D\_\{i,t\}, and identities involvingDαD\_\{\\alpha\}refer to the population quantity unless stated otherwise\.

Algorithm 1Per\-step Rényi discriminability gain proxiesΔ​D~i,t^\\widehat\{\\widetilde\{\\Delta D\}\_\{i,t\}\}fromπ\+\\pi^\{\+\}samples1:Prompt

𝒄\{\\bm\{c\}\}, policy

πold\\pi^\{\\mathrm\{old\}\}, stronger external generator

π\+\(⋅∣𝒄\)\\pi^\{\+\}\(\\cdot\\mid\{\\bm\{c\}\}\)used as a common proposal, rewards

r1:mr\_\{1:m\}, order

α\>0\\alpha\>0with

α≠1\\alpha\\neq 1, samples

NN, rollouts

KK, schedule

\{σt\}t=0T\\\{\\sigma\_\{t\}\\\}\_\{t=0\}^\{T\}
2:Per\-step gain proxies

Δ​D~i,t^\\widehat\{\\widetilde\{\\Delta D\}\_\{i,t\}\}for every reward

iiand step

tt
3:for

n=1,…,Nn=1,\\dots,Ndo

4:Draw

𝒙~0\(n\)∼π\+\(⋅∣𝒄\)\\widetilde\{\{\\bm\{x\}\}\}\_\{0\}^\{\(n\)\}\\\!\\sim\\pi^\{\+\}\(\\cdot\\mid\{\\bm\{c\}\}\); set

𝒙t\(n\)←\(1−σt\)​𝒙~0\(n\)\+σt​ϵt\{\\bm\{x\}\}\_\{t\}^\{\(n\)\}\\\!\\leftarrow\(1\-\\sigma\_\{t\}\)\\widetilde\{\{\\bm\{x\}\}\}\_\{0\}^\{\(n\)\}\\\!\+\\sigma\_\{t\}\{\\bm\{\\epsilon\}\}\_\{t\},

ϵt∼𝒩⁡\(0,I\)\{\\bm\{\\epsilon\}\}\_\{t\}\\\!\\sim\\\!\\mathcal\{N\}\(0,I\),

t=0,…,Tt=0,\\dots,T//𝐱t\(n\)∼πt\+\{\\bm\{x\}\}\_\{t\}^\{\(n\)\}\\\!\\sim\\pi\_\{t\}^\{\+\}

5:

μ^i,t\(n\)←1Kt​∑k≤Ktri​\(𝒙0\(n,t,k\),𝒄\)\\hat\{\\mu\}\_\{i,t\}^\{\(n\)\}\\leftarrow\\frac\{1\}\{K\_\{t\}\}\\sum\_\{k\\leq K\_\{t\}\}r\_\{i\}\(\{\\bm\{x\}\}\_\{0\}^\{\(n,t,k\)\},\{\\bm\{c\}\}\)from

KtK\_\{t\}rollouts

𝒙t\(n\)→𝒙0\(n,t,k\)\{\\bm\{x\}\}\_\{t\}^\{\(n\)\}\\\!\\to\\\!\{\\bm\{x\}\}\_\{0\}^\{\(n,t,k\)\}under

πold\\pi^\{\\mathrm\{old\}\}\(

K0=1K\_\{0\}\{=\}1, else

KK\)

6:endfor

7:

D~i,t^←1α−1​log⁡\(max⁡\{10−10,1N​∑n\[μ^i,t\(n\)\]α−1\}\)\\widehat\{\\widetilde\{D\}\_\{i,t\}\}\\leftarrow\\frac\{1\}\{\\alpha\-1\}\\log\\\!\\big\(\\max\\\{10^\{\-10\},\\frac\{1\}\{N\}\\sum\_\{n\}\[\\hat\{\\mu\}\_\{i,t\}^\{\(n\)\}\]^\{\\alpha\-1\}\\\}\\big\)for all

i,ti,t// numerical floor

8:return

Δ​D~i,t^←\[D~i,t−1^−D~i,t^\]\+\\widehat\{\\widetilde\{\\Delta D\}\_\{i,t\}\}\\leftarrow\\big\[\\widehat\{\\widetilde\{D\}\_\{i,t\-1\}\}\-\\widehat\{\\widetilde\{D\}\_\{i,t\}\}\\big\]\_\{\+\},

t=1,…,Tt=1,\\dots,T// Eq\. \([12](https://arxiv.org/html/2609.13425#S3.E12)\), clipped

### 3\.4Sinkhorn projection

The gainsΔ​Di,t\\Delta D\_\{i,t\}measure the utility of each reward at every step, but they do not yet constitute valid weights: they neither satisfy the budget𝝀\\bm\{\\lambda\}nor maintain equal per\-step total weights acrosstt\. We formulate the two requirements as marginal constraints and single out a unique matrix satisfying them by entropic projection\. We ablate the choice of equal per\-step total weights in Sec\.[4\.5](https://arxiv.org/html/2609.13425#S4.SS5)\.

We first define the*mean\-normalized gain*

Δ​D¯i,t:=Δ​Di,t1T​∑s=1TΔ​Di,s,so that1T​∑tΔ​D¯i,t=1​for every​i\.\\overline\{\\Delta D\}\_\{i,t\}\\;:=\\;\\frac\{\\Delta D\_\{i,t\}\}\{\\frac\{1\}\{T\}\\sum\_\{s=1\}^\{T\}\\Delta D\_\{i,s\}\},\\qquad\\text\{so that\}\\qquad\\frac\{1\}\{T\}\\sum\_\{t\}\\overline\{\\Delta D\}\_\{i,t\}\\;=\\;1\\;\\;\\text\{for every \}i\.\(13\)Because the gains telescope \(Eq\. \([7](https://arxiv.org/html/2609.13425#S3.E7)\)\), the denominator isDi,0/TD\_\{i,0\}/T, so the division removes the reward’s total discriminability and keeps only its shape inttto better capture the intra\-timestep relation\.

From the normalized gains we build the*affinity kernel*

Ki,t:=exp⁡\(Δ​D¯i,t\)\{\\;K\_\{i,t\}\\;:=\\;\\exp\\\!\\big\(\\overline\{\\Delta D\}\_\{i,t\}\\big\)\\;\}\(14\)which is largest at the steps where rewardiicontributes most of its discriminability and carries no scale parameter of its own, sinceΔ​D¯i,t\\overline\{\\Delta D\}\_\{i,t\}is already mean\-normalized\. Theorem[3\.1](https://arxiv.org/html/2609.13425#S3.Thmtheorem1)givesΔ​Di,t≥0\\Delta D\_\{i,t\}\\geq 0in population; in finite samples we clip, using\[Δ​Di,t\]\+\[\\Delta D\_\{i,t\}\]\_\{\+\}, so a step estimated as negative enters atKi,t=1K\_\{i,t\}=1, the smallest value the kernel takes\.

Recall the two marginal constraints from Eq\. \([1](https://arxiv.org/html/2609.13425#S1.E1)\)\. Writing𝟏m,𝟏T\{\\bm\{1\}\}\_\{m\},\{\\bm\{1\}\}\_\{T\}for all\-ones vectors, the feasible set is the*transportation polytope*

𝒰\(𝝀\):=\{𝑾∈ℝ≥0m×T:𝑾𝟏T=𝝀,𝑾⊤𝟏m=1T𝟏T\}\.\\mathcal\{U\}\(\\bm\{\\lambda\}\)\\;:=\\;\\Big\\\{\\,\\bm\{W\}\\in\\mathbb\{R\}\_\{\\geq 0\}^\{m\\times T\}\\;:\\;\\bm\{W\}\{\\bm\{1\}\}\_\{T\}=\\bm\{\\lambda\},\\;\\;\\bm\{W\}^\{\\top\}\{\\bm\{1\}\}\_\{m\}=\\tfrac\{1\}\{T\}\{\\bm\{1\}\}\_\{T\}\\,\\Big\\\}\.\(15\)The set is never empty, since the rank\-one matrixλi/T\\lambda\_\{i\}/Talways belongs to it\. Among its elements we want theKL\\mathrm\{KL\}projection of𝑲\\bm\{K\}onto the polytope,

𝑾⋆=arg​min𝐖∈𝒰⁡\(𝝀\)KL\(𝐖∥𝐊\)=arg​min𝐖∈𝒰⁡\(𝝀\)∑i,tWi,tlogWi,tKi,t−Wi,t\+Ki,t,\{\\;\\bm\{W\}^\{\\star\}\\;=\\;\\argmin\_\{\\bm\{W\}\\in\\mathcal\{U\}\(\\bm\{\\lambda\}\)\}\\;\\mathrm\{KL\}\\big\(\\bm\{W\}\\,\\\|\\,\\bm\{K\}\\big\)\\;=\\;\\argmin\_\{\\bm\{W\}\\in\\mathcal\{U\}\(\\bm\{\\lambda\}\)\}\\;\\sum\_\{i,t\}W\_\{i,t\}\\log\\frac\{W\_\{i,t\}\}\{K\_\{i,t\}\}\-W\_\{i,t\}\+K\_\{i,t\},\\;\}\(16\)which, substituting Eq\. \([14](https://arxiv.org/html/2609.13425#S3.E14)\), is an entropy\-regularized transport problem with costCi,t:=−Δ​D¯i,tC\_\{i,t\}:=\-\\overline\{\\Delta D\}\_\{i,t\}and unit regularization strength:

𝑾⋆=arg​min𝐖∈𝒰⁡\(𝝀\)∑i,t\[Ci,tWi,t\+Wi,tlogWi,t\]\.\\bm\{W\}^\{\\star\}\\;=\\;\\argmin\_\{\\bm\{W\}\\in\\mathcal\{U\}\(\\bm\{\\lambda\}\)\}\\;\\sum\_\{i,t\}\\Big\[\\,C\_\{i,t\}\\,W\_\{i,t\}\\;\+\\;W\_\{i,t\}\\log W\_\{i,t\}\\,\\Big\]\.\(17\)Since everyKi,t\>0K\_\{i,t\}\>0, the objective is strictly convex and the minimizer is unique\. Setting the Lagrangian gradient to zero gives the kernel rescaled by one factor per row and one per column,

Wi,t⋆=ai​Ki,t​bt=ai⋅exp⁡\(Δ​D¯i,t\)⋅bt,\{\\;W\_\{i,t\}^\{\\star\}\\;=\\;a\_\{i\}\\;K\_\{i,t\}\\;b\_\{t\}\\;=\\;a\_\{i\}\\cdot\\exp\\\!\\big\(\\overline\{\\Delta D\}\_\{i,t\}\\big\)\}\\cdot b\_\{t\},\(18\)whereaia\_\{i\}carries inter\-reward scaling, andbtb\_\{t\}the per\-step normalization\. Sinkhorn’s theorem\[[33](https://arxiv.org/html/2609.13425#bib.bib27),[4](https://arxiv.org/html/2609.13425#bib.bib26)\]guarantees that𝒂∈ℝ\>0m\{\\bm\{a\}\}\\in\\mathbb\{R\}^\{m\}\_\{\>0\}and𝒃∈ℝ\>0T\{\\bm\{b\}\}\\in\\mathbb\{R\}^\{T\}\_\{\>0\}exist and are unique up to the trivial rescaling\(𝒂,𝒃\)↦\(c​𝒂,𝒃/c\)\(\{\\bm\{a\}\},\{\\bm\{b\}\}\)\\mapsto\(c\\,\{\\bm\{a\}\},\{\\bm\{b\}\}/c\), and it finds them iteratively:

ai←λi∑tKi,t​bt,bt←1/T∑iKi,t​ai,a\_\{i\}\\;\\leftarrow\\;\\frac\{\\lambda\_\{i\}\}\{\\sum\_\{t\}K\_\{i,t\}\\,b\_\{t\}\},\\qquad\\qquad b\_\{t\}\\;\\leftarrow\\;\\frac\{1/T\}\{\\sum\_\{i\}K\_\{i,t\}\\,a\_\{i\}\},\(19\)which converges linearly; in the log domain it needs<100<100steps atm≤4m\\leq 4,T≤25T\\leq 25\(Alg\.[3](https://arxiv.org/html/2609.13425#alg3), App\.[C](https://arxiv.org/html/2609.13425#A3)\)\.

A reward whose gain is flat intthasΔ​D¯i,t≡1\\overline\{\\Delta D\}\_\{i,t\}\\equiv 1, since Eq\. \([13](https://arxiv.org/html/2609.13425#S3.E13)\) fixes the mean at11; if this holds for every reward, the kernel is rank one and the entropic optimum isWi,t⋆=λi/TW\_\{i,t\}^\{\\star\}=\\lambda\_\{i\}/T, so the static baseline is exactly the case of flat estimated curves \(App\.[C\.2](https://arxiv.org/html/2609.13425#A3.SS2)\)\.

### 3\.5Empirical gain curves on SD3\.5\-Medium

#### Setup\.

We instantiate Alg\.[1](https://arxiv.org/html/2609.13425#alg1)on SD3\.5\-Medium withα=2\\alpha\{=\}2,8080prompts from the training datasets,N=8N\{=\}8samples per prompt, andK=16K\{=\}16rollouts per𝒙t\{\\bm\{x\}\}\_\{t\}\. We approximateπ\+\\pi^\{\+\}byGPT Image 1\.5, VAE\-encode its images into latent space, forward\-noise to each step, and run batched rollouts underπold\\pi^\{\\mathrm\{old\}\}\. We plot seven rewards:ClipScore,HPSv2,PickScore,Aesthetic, andImageRewardjointly on this sharedT=10T\{=\}10grid, plusOCRandGenEvalon their own prompt sets at the nativeT=25T\{=\}25schedule they are later trained under\. Each reward’s architecture, checkpoint, and range are in Sec\.[4\.1](https://arxiv.org/html/2609.13425#S4.SS1)and App\.[E\.1](https://arxiv.org/html/2609.13425#A5.SS1); App\.[B\.2](https://arxiv.org/html/2609.13425#A2.SS2)conducts an ablation study onα\\alpha\.

#### Results\.

In Fig\.[1](https://arxiv.org/html/2609.13425#S3.F1), the estimated curves are strongly reward\-specific; the horizontal axis is the diffusion timestep, from the cleanest stept=1t\{=\}1\(left\) to the noisiestt=Tt\{=\}T\(right\)\.GenEval,OCR, andImageRewardpeak the most sharply, concentrating their gain toward the noisy end, while the per\-step gain of the others stays close to uniform\. The first two are rule\-based and, likeImageReward, can score a sample once its global structure has emerged\. The differences are large enough to matter: under uniform𝝀\\bm\{\\lambda\}, the aggregate demand∑iλi​Δ​D¯i,t\\sum\_\{i\}\\lambda\_\{i\}\\overline\{\\Delta D\}\_\{i,t\}varies by5\.4×5\.4\\timesacross the steps2≤t≤T2\\leq t\\leq T, and still by4\.5×4\.5\\timesonceGenEvalis excluded\. The right panel shows the Sinkhorn projection \(Eq\. \([19](https://arxiv.org/html/2609.13425#S3.E19)\)\) over the four rewards in theOCRsetting, correcting this imbalance by shifting each reward’s weight toward its own high\-gain steps while both marginals stay exact\.

Figure 1:Gain curves and the weight matrix they project to\.Left:per\-step gainΔ​Di,t\\Delta D\_\{i,t\}\.Center:mean\-normalized gainΔ​D¯i,t\\overline\{\\Delta D\}\_\{i,t\}\(Eq\. \([13](https://arxiv.org/html/2609.13425#S3.E13)\)\), the exponent of the affinity kernel \(Eq\. \([14](https://arxiv.org/html/2609.13425#S3.E14)\)\)\.Right:the Sinkhorn matrixWi,t⋆W\_\{i,t\}^\{\\star\}\(Eq\. \([18](https://arxiv.org/html/2609.13425#S3.E18)\)\) at uniformλi=1/m\\lambda\_\{i\}\{=\}1/m, solved over the four rewards that are trained together in theOCRsetting; both marginals hold exactly, every column summing to1/T1/Tand every row toλi\\lambda\_\{i\}\. In all panels the horizontal axis ist/Tt/T, running from the cleanest stept=1t\{=\}1\(left\) to the noisiestt=Tt\{=\}T\(right\)\.OCRandGenEval\(dashed\) are resampled onto the sharedT=10T\{=\}10grid from their nativeT=25T\{=\}25curves, so the right panel is illustrative: the training\-stage kernel is solved atT=25T\{=\}25onOCR’s own grid \(Sec\.[4\.1](https://arxiv.org/html/2609.13425#S4.SS1)\)\.

## 4Experiments

Sec\.[4\.1](https://arxiv.org/html/2609.13425#S4.SS1)introduces the setup\. Sec\.[4\.2](https://arxiv.org/html/2609.13425#S4.SS2)evaluates the training rewards on a held\-out split, and Sec\.[4\.3](https://arxiv.org/html/2609.13425#S4.SS3)the same checkpoints under held\-out judges, to show generalizable improvement\. Sec\.[4\.4](https://arxiv.org/html/2609.13425#S4.SS4)rescores those images with an independent LLM\-as\-a\-Judge, and Sec\.[4\.5](https://arxiv.org/html/2609.13425#S4.SS5)ablates the Sinkhorn projection\.

### 4\.1Training setup

#### Reward models\.

We evaluate ReCAST with nine reward models in two distinct roles\. Five are*training rewards*:ClipScore\[[12](https://arxiv.org/html/2609.13425#bib.bib24)\],HPSv2\[[39](https://arxiv.org/html/2609.13425#bib.bib18)\],PickScore\[[18](https://arxiv.org/html/2609.13425#bib.bib8)\],OCR, andGenEval\[[9](https://arxiv.org/html/2609.13425#bib.bib7)\]\. To evaluate generalization, four*held\-out judges*score the checkpoints:Aesthetic\[[32](https://arxiv.org/html/2609.13425#bib.bib9)\],ImageReward\[[40](https://arxiv.org/html/2609.13425#bib.bib6)\],HPSv3\[[27](https://arxiv.org/html/2609.13425#bib.bib19)\], andUnifiedReward\-2\[[38](https://arxiv.org/html/2609.13425#bib.bib10)\]\. App\.[E\.1](https://arxiv.org/html/2609.13425#A5.SS1)details architectures, checkpoints, and output ranges for all rewards\.

#### Training data and reward chain\.

Following DiffusionNFT\[[42](https://arxiv.org/html/2609.13425#bib.bib25)\], training runs in two stages\. Thewarmup stagejointly optimizesClipScore,HPSv2, andPickScorefor120120steps from the base model on Pick\-a\-Pic prompts \(25,43225\{,\}432train/2,0482\{,\}048test\); every later run resumes from its checkpoint\. Thetraining stage\-OCRthen adds anOCRreward on text\-rendering prompts \(19,65219\{,\}652train/1,0171\{,\}017test\) for6060further steps \(m=4m\{=\}4\)\. Thetraining stage\-GenEvalbranches from the same warmup parent on compositionalGenEvalprompts \(50,00050\{,\}000train/2,2112\{,\}211test\), swappingOCRfor a rule\-basedGenEvalreward on an identical schedule\.

#### Training configuration\.

Across all stages, we fine\-tune LoRA adapters of Stable Diffusion 3\.5\-Medium\. We use the DiffusionNFT objective withβ=0\.1\\beta\{=\}0\.1, rolling outT=25T\{=\}25sampling steps per iteration at an effective batch size of11521152\. We run the warmup stage at a balanced𝝀=\(1,1,1\)\\bm\{\\lambda\}\\\!=\\\!\(1,1,1\)on three rewards:ClipScore,HPSv2, andPickScore\. Then, we sweep each training\-stage setting’s row marginals over𝝀∈\{\(1,1,1,1\),\(1,1,1,2\),\(1,1,2,1\),\(1,2,1,1\),\(2,1,1,1\)\}\\bm\{\\lambda\}\\in\\\{\(1,1,1,1\),\(1,1,1,2\),\(1,1,2,1\),\(1,2,1,1\),\(2,1,1,1\)\\\}, the uniform budget together with each coordinate doubled in turn, where𝝀\\bm\{\\lambda\}is ordered \(ClipScore,HPSv2,PickScore,OCR/GenEval\)\. Every𝝀\\bm\{\\lambda\}is written unnormalized as an integer ratio between rewards, and is divided by its own sum before use to satisfy∑iλi=1\\sum\_\{i\}\\lambda\_\{i\}=1\. Each budget gives one matched pair of ReCAST against a static baseline, and both settings are repeated end\-to\-end under33independent seeds, giving1515matched \(budget, seed\) pairs per setting\. For our weights configuration, we use Alg\.[1](https://arxiv.org/html/2609.13425#alg1)withα=2\\alpha\{=\}2for the kernel and Alg\.[3](https://arxiv.org/html/2609.13425#alg3)for the Sinkhorn projection\. The training objective is detailed in App\.[A\.1](https://arxiv.org/html/2609.13425#A1.SS1)and the remaining hyperparameters in App\.[E\.2](https://arxiv.org/html/2609.13425#A5.SS2)\.

#### Evaluation protocol\.

We evaluate at three levels: thetraining rewardson a held\-out dataset \(Sec\.[4\.2](https://arxiv.org/html/2609.13425#S4.SS2)\);held\-out judge models\(Sec\.[4\.3](https://arxiv.org/html/2609.13425#S4.SS3)\); andLLM\-as\-a\-Judge\(Sec\.[4\.4](https://arxiv.org/html/2609.13425#S4.SS4)\)\. All three score each run’s final checkpoint at matched seeds,5122512^\{2\}resolution, CFG4\.54\.5, and4040steps; we report means over each training\-stage pair\-set and win rates over its matched pairs\.

### 4\.2Training rewards

Table 1:Training\-reward scores at the final checkpoint, reported as base / static / Ours, where*base*is the untuned SD3\.5\-Medium,*static*is static reweighting by𝝀\\bm\{\\lambda\}, and Aggregaterris the sum of the individual rewards weighted by the integer𝝀\\bm\{\\lambda\}\. Both settings were trained end\-to\-end \(warmup stage \+ training\-stage sweep, both methods\) under33seeds; entries are the mean of the three seed\-level means, standard deviation as a subscript, and*win*pools all1515matched \(budget, seed\) pairs\. Bold marks the better of static and Ours, and both when the two agree at the reported precision\. Per\-𝝀\\bm\{\\lambda\}results are in App\.[D](https://arxiv.org/html/2609.13425#A4)\.Tab\.[1](https://arxiv.org/html/2609.13425#S4.T1)reports each training reward on a held\-out dataset, averaged over the five budgets and three seed replicates, with*win*counting the matched pairs in which ReCAST beats static\. ReCAST raises the aggregate score from2\.8672\.867to2\.9202\.920on average \(\+0\.053±0\.049\+0\.053\_\{\\pm 0\.049\}across seeds\) and wins1111of the1515matched \(budget, seed\) pairs\. The gain is not obtained by sacrificing one objective for another: all four component rewards improve on average, withPickScorewinning all1515pairs andClipScoreandHPSv2winning1313and1212\. The targetOCRreward carries the largest seed variance \(\+0\.007±0\.046\+0\.007\_\{\\pm 0\.046\},9/159/15\)\. Adaptive timing therefore uses a fixed multi\-reward budget more efficiently, rather than merely changing the trade\-off encoded by𝝀\\bm\{\\lambda\}\. On the GenEval setting the two methods instead tie: every training reward shifts by less than one seed standard deviation \(aggregate−0\.003±0\.031\-0\.003\_\{\\pm 0\.031\}, targetGenEval\+0\.001±0\.017\+0\.001\_\{\\pm 0\.017\}\) and win rates are near chance \(55to99of1515\)\. That setting therefore acts as a matched\-training\-reward control, where Sec\.[4\.3](https://arxiv.org/html/2609.13425#S4.SS3)still finds ReCAST preferred by every held\-out judge\.

### 4\.3Held\-out judges

Table 2:Held\-out judge scores at the final checkpoint, reported as base / static / Ours\. Both settings cover55budgets×\\times33independent training seeds, and entries, subscripts,*win*, and the bolding rule are as in Tab\.[1](https://arxiv.org/html/2609.13425#S4.T1); theHPSv3row is at two decimals because its scale is an order of magnitude larger\. Per\-𝝀\\bm\{\\lambda\}results are in App\.[D](https://arxiv.org/html/2609.13425#A4)\.As shown in Tab\.[2](https://arxiv.org/html/2609.13425#S4.T2), on the OCR setting all four held\-out judges score ReCAST higher on average, winning1313,1212,1111, and1111of the1515matched \(budget, seed\) pairs\. For the GenEval setting, although the two methods are tied there on every reward being optimized \(Sec\.[4\.2](https://arxiv.org/html/2609.13425#S4.SS2)\), all four held\-out judges still favor ReCAST \(largest,HPSv3\+0\.56±0\.41\+0\.56\_\{\\pm 0\.41\}\), winning99or1010of the1515pairs\. A held\-out gain under an equal training reward is consistent with generalization rather than overfitting, and on the OCR setting the improved text rendering enhances rather than degrades broader image quality\. The two settings differ against the untuned base model, however: the OCR setting clears it on all four judges under both methods, whereas in the GenEval setting static falls below it onAesthetic,HPSv3, andUnifiedReward\-2\. Optimizing compositional correctness therefore costs generic visual quality, and ReCAST recovers that cost on every judge, fully onAestheticandUnifiedReward\-2and partly onHPSv3\.

### 4\.4LLM\-as\-a\-Judge

To assess general preference closer to real use, we use the MMRBv2 eval script\[[13](https://arxiv.org/html/2609.13425#bib.bib40)\]to compare the ReCAST checkpoints against static on a fixed1,0001\{,\}000\-prompt subset of the held\-out OCR set\. A multimodal LLM judge \(gemini\-3\.5\-flash, temperature00\) picks the better of the two images per prompt according to a fixed rubric, and every pair is judged twice with positions swapped to cancel position bias; we also report the rubric’s faithfulness and aesthetics criteria\. App\.[E\.3](https://arxiv.org/html/2609.13425#A5.SS3)gives the full prompt and the debiasing procedure\.

Table 3:MMRBv2 pairwise evaluation of ReCAST against static on the same subset as Tab\.[2](https://arxiv.org/html/2609.13425#S4.T2), position\-debiased over both presentation orders\. Every entry is ReCAST’s win rate, overall and under the two rubric criteria, with0\.50\.5indicating parity;𝝀\\bm\{\\lambda\}is ordered \(ClipScore,HPSv2,PickScore,OCR\)\.As shown in Tab\.[3](https://arxiv.org/html/2609.13425#S4.T3), ReCAST obtains a mean overall win rate of0\.5720\.572and is preferred at four of the five budgets, most strongly at\(1,1,1,1\)\(1,1,1,1\)and\(2,1,1,1\)\(2,1,1,1\)\(0\.6540\.654and0\.6330\.633\), where both faithfulness and aesthetics improve\. App\.[G](https://arxiv.org/html/2609.13425#A7)shows paired generations behind these judgments and their selection protocol\.

### 4\.5Ablation study: the Sinkhorn projection

We ablate the Sinkhorn projection of Sec\.[3\.4](https://arxiv.org/html/2609.13425#S3.SS4), which enforces a uniform total budget at each step, by training*no\-Sink Rényi*variants that keep the adaptive kernel but omit the column constraint:W⋆W^\{\\star\}becomes the row\-normalized kernelWi,t=λi​Ki,t/∑sKi,sW\_\{i,t\}=\\lambda\_\{i\}K\_\{i,t\}/\\sum\_\{s\}K\_\{i,s\}\. Every reward still spends exactly its budgetλi\\lambda\_\{i\}and the loss still consumesT⋅Wi,tT\\\!\\cdot\\\!W\_\{i,t\}, but∑iWi,t\\sum\_\{i\}W\_\{i,t\}is free, so each reward rides its own gain shape\. All other training choices match theOCRexperiment\.

Table 4:Aggregate training\-reward score for each𝝀=\(clip,hps,pick,ocr\)\\bm\{\\lambda\}=\(\\text\{clip\},\\text\{hps\},\\text\{pick\},\\text\{ocr\}\)budget in the training stage\-OCR experiment, comparing static weighting, the adaptive kernel alone \(no\-Sink Rényi\), and the full method\. Scores are on the held\-out OCR dataset at the last checkpoint \(step180180\), single matched training seed\.As shown in Tab\.[4](https://arxiv.org/html/2609.13425#S4.T4), no\-Sink Rényi beats static on four of five budgets, and the full method beats no\-Sink Rényi on four of five\. Averaged across budgets, the adaptive kernel accounts for\+0\.068\+0\.068of ReCAST’s total\+0\.099\+0\.099gain over static, and the Sinkhorn projection for the remaining\+0\.031\+0\.031\(this seed’s gap, not the three\-seed mean of Tab\.[1](https://arxiv.org/html/2609.13425#S4.T1); both parts of the decomposition share the seed\)\. Reward\-specific timing therefore identifies useful optimization steps, while the shared column budget keeps the resulting weights from concentrating on a few of them\.

## 5Conclusion

Multi\-reward diffusion fine\-tuning must decide both*how much*each reward matters and*when*it should act\. ReCAST separates these decisions: the budget𝝀\\bm\{\\lambda\}fixes each reward’s total contribution, and its Rényi discriminability curve distributes it across timesteps\.

On SD3\.5\-Medium, this temporal reallocation improves the OCR setting’s aggregate training objective across five budgets and three seeds, with all four training rewards increasing on average, and it transfers: every held\-out judge improves over static, including on the GenEval setting where the two methods tie on the training rewards, and the pairwise LLM judge prefers ReCAST at four of five budgets\.*When*a reward is applied can therefore matter as much as*how much*weight it receives\. We discuss the limitations of our work in App\.[F](https://arxiv.org/html/2609.13425#A6)\.

## References

- \[1\]K\. Black, M\. Janner, Y\. Du, I\. Kostrikov, and S\. Levine\(2023\)Training diffusion models with reinforcement learning\.arXiv preprint arXiv:2305\.13301\.Cited by:[§1](https://arxiv.org/html/2609.13425#S1.p1.1)\.
- \[2\]J\. Choi, J\. Lee, C\. Shin, S\. Kim, H\. Kim, and S\. Yoon\(2022\)Perception prioritized training of diffusion models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,Cited by:[§1](https://arxiv.org/html/2609.13425#S1.p3.1),[§2](https://arxiv.org/html/2609.13425#S2.SS0.SSS0.Px3.p1.1)\.
- \[3\]K\. Clark, P\. Vicol, K\. Swersky, and D\. J\. Fleet\(2023\)Directly fine\-tuning diffusion models on differentiable rewards\.arXiv preprint arXiv:2309\.17400\.Cited by:[§1](https://arxiv.org/html/2609.13425#S1.p1.1)\.
- \[4\]M\. Cuturi\(2013\)Sinkhorn distances: lightspeed computation of optimal transport\.InAdvances in Neural Information Processing Systems,Vol\.26\.Cited by:[§3\.4](https://arxiv.org/html/2609.13425#S3.SS4.p4.5)\.
- \[5\]J\. Dai, X\. Pan, R\. Sun, J\. Ji, X\. Xu, M\. Liu, Y\. Wang, and Y\. Yang\(2023\)Safe RLHF: safe reinforcement learning from human feedback\.arXiv preprint arXiv:2310\.12773\.Cited by:[§2](https://arxiv.org/html/2609.13425#S2.SS0.SSS0.Px2.p1.1)\.
- \[6\]P\. Esser, S\. Kulal, A\. Blattmann, R\. Entezari, J\. Müller, H\. Saini, Y\. Levi, D\. Lorenz, A\. Sauer, F\. Boesel,et al\.\(2024\)Scaling rectified flow transformers for high\-resolution image synthesis\.InForty\-first international conference on machine learning,Cited by:[§A\.1](https://arxiv.org/html/2609.13425#A1.SS1.SSS0.Px1.p1.1)\.
- \[7\]Y\. Fan, O\. Watkins, Y\. Du, H\. Liu, M\. Ryu, C\. Boutilier, P\. Abbeel, M\. Ghavamzadeh, K\. Lee, and K\. Lee\(2023\)Dpok: reinforcement learning for fine\-tuning text\-to\-image diffusion models\.Advances in Neural Information Processing Systems36,pp\. 79858–79885\.Cited by:[§1](https://arxiv.org/html/2609.13425#S1.p1.1)\.
- \[8\]L\. Gao, J\. Schulman, and J\. Hilton\(2023\)Scaling laws for reward model overoptimization\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2609.13425#S1.p1.1)\.
- \[9\]D\. Ghosh, H\. Hajishirzi, and L\. Schmidt\(2023\)Geneval: an object\-focused framework for evaluating text\-to\-image alignment\.Advances in Neural Information Processing Systems36,pp\. 52132–52152\.Cited by:[Table 11](https://arxiv.org/html/2609.13425#A5.T11.4.6.1.1.1),[§1](https://arxiv.org/html/2609.13425#S1.p1.1),[§1](https://arxiv.org/html/2609.13425#S1.p3.1),[§4\.1](https://arxiv.org/html/2609.13425#S4.SS1.SSS0.Px1.p1.1)\.
- \[10\]T\. Hang, S\. Gu, C\. Li, J\. Bao, D\. Chen, H\. Hu, X\. Geng, and B\. Guo\(2023\)Efficient diffusion training via min\-snr weighting strategy\.InProceedings of the IEEE/CVF International Conference on Computer Vision,Cited by:[§1](https://arxiv.org/html/2609.13425#S1.p3.1),[§2](https://arxiv.org/html/2609.13425#S2.SS0.SSS0.Px3.p1.1)\.
- \[11\]X\. He, S\. Fu, Y\. Zhao, W\. Li, J\. Yang, D\. Yin, F\. Rao, and B\. Zhang\(2025\)TempFlow\-grpo: when timing matters for grpo in flow models\.arXiv preprint arXiv:2508\.04324\.Cited by:[§1](https://arxiv.org/html/2609.13425#S1.p3.1),[§2](https://arxiv.org/html/2609.13425#S2.SS0.SSS0.Px3.p1.1)\.
- \[12\]J\. Hessel, A\. Holtzman, M\. Forbes, R\. L\. Bras, and Y\. Choi\(2021\)Clipscore: a reference\-free evaluation metric for image captioning\.arXiv preprint arXiv:2104\.08718\.Cited by:[Table 11](https://arxiv.org/html/2609.13425#A5.T11.4.2.1.1.1),[§1](https://arxiv.org/html/2609.13425#S1.p1.1),[§1](https://arxiv.org/html/2609.13425#S1.p3.1),[§4\.1](https://arxiv.org/html/2609.13425#S4.SS1.SSS0.Px1.p1.1)\.
- \[13\]Y\. Hu, R\. Askari\-Hemmat, M\. Hall, E\. Dinan, L\. Zettlemoyer, and M\. Ghazvininejad\(2025\)Multimodal RewardBench 2: evaluating omni reward models for interleaved text and image\.arXiv preprint arXiv:2512\.16899\.Cited by:[§E\.3](https://arxiv.org/html/2609.13425#A5.SS3.SSS0.Px1.p1.1),[§4\.4](https://arxiv.org/html/2609.13425#S4.SS4.p1.1)\.
- \[14\]J\. Jang, S\. Kim, B\. Y\. Lin, Y\. Wang, J\. Hessel, L\. Zettlemoyer, H\. Hajishirzi, Y\. Choi, and P\. Ammanabrolu\(2023\)Personalized soups: personalized large language model alignment via post\-hoc parameter merging\.arXiv preprint arXiv:2310\.11564\.Cited by:[§2](https://arxiv.org/html/2609.13425#S2.SS0.SSS0.Px2.p1.1)\.
- \[15\]T\. Karras, M\. Aittala, T\. Aila, and S\. Laine\(2022\)Elucidating the design space of diffusion\-based generative models\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§1](https://arxiv.org/html/2609.13425#S1.p3.1),[§2](https://arxiv.org/html/2609.13425#S2.SS0.SSS0.Px3.p1.1)\.
- \[16\]A\. Kazemnejad, M\. Aghajohari, E\. Portelance, A\. Sordoni, S\. Reddy, A\. Courville, and N\. Le Roux\(2024\)VinePPO: unlocking RL potential for LLM reasoning through refined credit assignment\.arXiv preprint arXiv:2410\.01679\.Cited by:[§1](https://arxiv.org/html/2609.13425#S1.p4.1)\.
- \[17\]D\. Kingma, T\. Salimans, B\. Poole, and J\. Ho\(2021\)Variational diffusion models\.Advances in Neural Information Processing Systems34,pp\. 21696–21707\.Cited by:[§1](https://arxiv.org/html/2609.13425#S1.p3.1),[§2](https://arxiv.org/html/2609.13425#S2.SS0.SSS0.Px3.p1.1)\.
- \[18\]Y\. Kirstain, A\. Polyak, U\. Singer, S\. Matiana, J\. Penna, and O\. Levy\(2023\)Pick\-a\-pic: an open dataset of user preferences for text\-to\-image generation\.Advances in Neural Information Processing Systems36,pp\. 36652–36663\.Cited by:[Table 11](https://arxiv.org/html/2609.13425#A5.T11.4.4.1.1.1),[§1](https://arxiv.org/html/2609.13425#S1.p1.1),[§4\.1](https://arxiv.org/html/2609.13425#S4.SS1.SSS0.Px1.p1.1)\.
- \[19\]Y\. Lai, S\. Wang, S\. Liu, X\. Huang, and Z\. Wei\(2024\)ALaRM: align language models via hierarchical rewards modeling\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 7817–7831\.Cited by:[§2](https://arxiv.org/html/2609.13425#S2.SS0.SSS0.Px2.p1.1)\.
- \[20\]J\. Li, Y\. Cui, T\. Huang, W\. Kong, Y\. Cheng, C\. Zeng, Y\. Ma, C\. Fan, M\. Yang, Z\. Zhong, and L\. Bo\(2025\)MixGRPO: unlocking flow\-based grpo efficiency with mixed ode\-sde\.arXiv preprint arXiv:2507\.21802\.Cited by:[§1](https://arxiv.org/html/2609.13425#S1.p3.1),[§2](https://arxiv.org/html/2609.13425#S2.SS0.SSS0.Px3.p1.1)\.
- \[21\]Z\. Liang, Y\. Yuan, S\. Gu, B\. Chen, T\. Hang, M\. Cheng, J\. Li, and L\. Zheng\(2025\)Aesthetic post\-training diffusion models from generic preferences with step\-by\-step preference optimization\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 13199–13208\.Cited by:[§1](https://arxiv.org/html/2609.13425#S1.p3.1),[§2](https://arxiv.org/html/2609.13425#S2.SS0.SSS0.Px3.p1.1)\.
- \[22\]H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe\(2024\)Let’s verify step by step\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.13425#S1.p4.1)\.
- \[23\]Y\. Lipman, R\. T\. Chen, H\. Ben\-Hamu, M\. Nickel, and M\. Le\(2022\)Flow matching for generative modeling\.arXiv preprint arXiv:2210\.02747\.Cited by:[§A\.1](https://arxiv.org/html/2609.13425#A1.SS1.SSS0.Px1.p1.1)\.
- \[24\]J\. Liu, G\. Liu, J\. Liang, Y\. Li, J\. Liu, X\. Wang, P\. Wan, D\. Zhang, and W\. Ouyang\(2025\)Flow\-grpo: training flow matching models via online rl\.arXiv preprint arXiv:2505\.05470\.Cited by:[§1](https://arxiv.org/html/2609.13425#S1.p1.1),[§2](https://arxiv.org/html/2609.13425#S2.SS0.SSS0.Px1.p1.1)\.
- \[25\]S\. Liu, X\. Dong, X\. Lu, S\. Diao, P\. Belcak, M\. Liu, M\. Chen, H\. Yin, Y\. F\. Wang, K\. Cheng, Y\. Choi, J\. Kautz, and P\. Molchanov\(2026\)GDPO: group reward\-decoupled normalization policy optimization for multi\-reward RL optimization\.arXiv preprint arXiv:2601\.05242\.Cited by:[§2](https://arxiv.org/html/2609.13425#S2.SS0.SSS0.Px2.p1.1)\.
- \[26\]X\. Liu, C\. Gong, and Q\. Liu\(2022\)Flow straight and fast: learning to generate and transfer data with rectified flow\.arXiv preprint arXiv:2209\.03003\.Cited by:[§A\.1](https://arxiv.org/html/2609.13425#A1.SS1.SSS0.Px1.p1.1)\.
- \[27\]Y\. Ma, Y\. Shui, X\. Wu, K\. Sun, and H\. Li\(2025\)HPSv3: towards wide\-spectrum human preference score\.arXiv preprint arXiv:2508\.03789\.Cited by:[Table 11](https://arxiv.org/html/2609.13425#A5.T11.4.9.1.1.1),[§4\.1](https://arxiv.org/html/2609.13425#S4.SS1.SSS0.Px1.p1.1)\.
- \[28\]A\. Q\. Nichol and P\. Dhariwal\(2021\)Improved denoising diffusion probabilistic models\.InInternational Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2609.13425#S2.SS0.SSS0.Px3.p1.1)\.
- \[29\]M\. Prabhudesai, A\. Goyal, D\. Pathak, and K\. Fragkiadaki\(2023\)Aligning text\-to\-image diffusion models with reward backpropagation\.arXiv preprint arXiv:2310\.03739\.Cited by:[§1](https://arxiv.org/html/2609.13425#S1.p1.1)\.
- \[30\]A\. Ramé, G\. Couairon, C\. Dancette, J\. Gaya, M\. Shukor, L\. Soulier, and M\. Cord\(2023\)Rewarded soups: towards pareto\-optimal alignment by interpolating weights fine\-tuned on diverse rewards\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§2](https://arxiv.org/html/2609.13425#S2.SS0.SSS0.Px2.p1.1)\.
- \[31\]A\. Rényi\(1961\)On measures of entropy and information\.InProceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics,pp\. 547–561\.Cited by:[§3\.2](https://arxiv.org/html/2609.13425#S3.SS2.p1.1)\.
- \[32\]C\. Schuhmann\(2022\)LAION\-aesthetics\.Note:[https://laion\.ai/blog/laion\-aesthetics/](https://laion.ai/blog/laion-aesthetics/)Cited by:[Table 11](https://arxiv.org/html/2609.13425#A5.T11.4.7.1.1.1),[§4\.1](https://arxiv.org/html/2609.13425#S4.SS1.SSS0.Px1.p1.1)\.
- \[33\]R\. Sinkhorn and P\. Knopp\(1967\)Concerning nonnegative matrices and doubly stochastic matrices\.Pacific Journal of Mathematics21\(2\),pp\. 343–348\.Cited by:[§3\.4](https://arxiv.org/html/2609.13425#S3.SS4.p4.5)\.
- \[34\]J\. Skalse, N\. Howe, D\. Krasheninnikov, and D\. Krueger\(2022\)Defining and characterizing reward gaming\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§1](https://arxiv.org/html/2609.13425#S1.p1.1)\.
- \[35\]Y\. Song, C\. Durkan, I\. Murray, and S\. Ermon\(2021\)Maximum likelihood training of score\-based diffusion models\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 1415–1428\.Cited by:[§2](https://arxiv.org/html/2609.13425#S2.SS0.SSS0.Px3.p1.1)\.
- \[36\]J\. Uesato, N\. Kushman, R\. Kumar, F\. Song, N\. Siegel, L\. Wang, A\. Creswell, G\. Irving, and I\. Higgins\(2022\)Solving math word problems with process\- and outcome\-based feedback\.arXiv preprint arXiv:2211\.14275\.Cited by:[§1](https://arxiv.org/html/2609.13425#S1.p4.1)\.
- \[37\]B\. Wallace, M\. Dang, R\. Rafailov, L\. Zhou, A\. Lou, S\. Purushwalkam, S\. Ermon, C\. Xiong, S\. Joty, and N\. Naik\(2024\)Diffusion model alignment using direct preference optimization\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 8228–8238\.Cited by:[§1](https://arxiv.org/html/2609.13425#S1.p1.1)\.
- \[38\]Y\. Wang, Y\. Zang, H\. Li, C\. Jin, and J\. Wang\(2025\)Unified reward model for multimodal understanding and generation\.arXiv preprint arXiv:2503\.05236\.Cited by:[Table 11](https://arxiv.org/html/2609.13425#A5.T11.4.10.1.1.1),[§4\.1](https://arxiv.org/html/2609.13425#S4.SS1.SSS0.Px1.p1.1)\.
- \[39\]X\. Wu, Y\. Hao, K\. Sun, Y\. Chen, F\. Zhu, R\. Zhao, and H\. Li\(2023\)Human preference score v2: a solid benchmark for evaluating human preferences of text\-to\-image synthesis\.arXiv preprint arXiv:2306\.09341\.Cited by:[Table 11](https://arxiv.org/html/2609.13425#A5.T11.4.3.1.1.1),[§1](https://arxiv.org/html/2609.13425#S1.p1.1),[§4\.1](https://arxiv.org/html/2609.13425#S4.SS1.SSS0.Px1.p1.1)\.
- \[40\]J\. Xu, X\. Liu, Y\. Wu, Y\. Tong, Q\. Li, M\. Ding, J\. Tang, and Y\. Dong\(2023\)Imagereward: learning and evaluating human preferences for text\-to\-image generation\.Advances in Neural Information Processing Systems36,pp\. 15903–15935\.Cited by:[Table 11](https://arxiv.org/html/2609.13425#A5.T11.4.8.1.1.1),[§1](https://arxiv.org/html/2609.13425#S1.p1.1),[§4\.1](https://arxiv.org/html/2609.13425#S4.SS1.SSS0.Px1.p1.1)\.
- \[41\]Z\. Xue, J\. Wu, Y\. Gao, F\. Kong, L\. Zhu, M\. Chen, Z\. Liu, W\. Liu, Q\. Guo, W\. Huang,et al\.\(2025\)DanceGRPO: unleashing grpo on visual generation\.arXiv preprint arXiv:2505\.07818\.Cited by:[§1](https://arxiv.org/html/2609.13425#S1.p1.1),[§2](https://arxiv.org/html/2609.13425#S2.SS0.SSS0.Px1.p1.1)\.
- \[42\]K\. Zheng, H\. Chen, H\. Ye, H\. Wang, Q\. Zhang, K\. Jiang, H\. Su, S\. Ermon, J\. Zhu, and M\. Liu\(2025\)DiffusionNFT: online diffusion reinforcement with forward process\.arXiv preprint arXiv:2509\.16117\.Cited by:[§A\.1](https://arxiv.org/html/2609.13425#A1.SS1.SSS0.Px2.p1.1),[Theorem A\.1](https://arxiv.org/html/2609.13425#A1.Thmtheorem1),[Theorem A\.2](https://arxiv.org/html/2609.13425#A1.Thmtheorem2),[§1](https://arxiv.org/html/2609.13425#S1.p1.1),[§2](https://arxiv.org/html/2609.13425#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.13425#S4.SS1.SSS0.Px2.p1.1)\.
- \[43\]H\. Zhu, T\. Xiao, and V\. G\. Honavar\(2025\)DSPO: direct score preference optimization for diffusion model alignment\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.13425#S1.p1.1)\.

## Appendix ASupplementary theory

In App\.[A\.1](https://arxiv.org/html/2609.13425#A1.SS1), we provide notations and basic properties on rectified flow matching and DiffusionNFT\. In App\.[A\.2](https://arxiv.org/html/2609.13425#A1.SS2), we prove Theorem[3\.1](https://arxiv.org/html/2609.13425#S3.Thmtheorem1)\. In App\.[A\.3](https://arxiv.org/html/2609.13425#A1.SS3), we bound the error induced by replacing each reward\-specific positive policyπi\+\\pi\_\{i\}^\{\+\}with the common surrogateπ\+\\pi^\{\+\}, and show that it isO⁡\(KL\(π\+∥πi\+\)\)O\(\\sqrt\{\\mathrm\{KL\}\(\\pi^\{\+\}\\\|\\pi\_\{i\}^\{\+\}\)\}\)on the discriminability gain and propagates to the Sinkhorn weights\.

### A\.1Background: rectified flow matching and DiffusionNFT

#### Rectified flow matching\.

All models in this paper are rectified\-flow generators\[[26](https://arxiv.org/html/2609.13425#bib.bib20),[23](https://arxiv.org/html/2609.13425#bib.bib21),[6](https://arxiv.org/html/2609.13425#bib.bib5)\]\. A clean sample𝒙0∼π\(⋅∣𝒄\)\{\\bm\{x\}\}\_\{0\}\\sim\\pi\(\\cdot\\mid\{\\bm\{c\}\}\)and noiseϵ∼𝒩⁡\(0,I\)\{\\bm\{\\epsilon\}\}\\sim\\mathcal\{N\}\(0,I\)are connected by the straight interpolation

𝒙t=αt​𝒙0\+σt​ϵ,αt=1−σt,σ0=0,σT=1,\{\\bm\{x\}\}\_\{t\}\\;=\\;\\alpha\_\{t\}\\,\{\\bm\{x\}\}\_\{0\}\+\\sigma\_\{t\}\\,\{\\bm\{\\epsilon\}\},\\qquad\\alpha\_\{t\}=1\-\\sigma\_\{t\},\\qquad\\sigma\_\{0\}=0,\\quad\\sigma\_\{T\}=1,\(20\)sot=0t\{=\}0is clean data andt=Tt\{=\}Tis pure noise\. Hereαt\\alpha\_\{t\}is the interpolation coefficient, distinct from the Rényi orderα\\alphaof Sec\.[3\.2](https://arxiv.org/html/2609.13425#S3.SS2)and from the mixture coefficientα⁡\(𝒙t\)\\alpha\(\{\\bm\{x\}\}\_\{t\}\)defined below\. Differentiating Eq\. \([20](https://arxiv.org/html/2609.13425#A1.E20)\) gives the target velocity𝒗=α˙t​𝒙0\+σ˙t​ϵ\{\\bm\{v\}\}=\\dot\{\\alpha\}\_\{t\}\\,\{\\bm\{x\}\}\_\{0\}\+\\dot\{\\sigma\}\_\{t\}\\,\{\\bm\{\\epsilon\}\}, which for the linear path is simply𝒗=ϵ−𝒙0\{\\bm\{v\}\}=\{\\bm\{\\epsilon\}\}\-\{\\bm\{x\}\}\_\{0\}\. A velocity model𝒗θ\{\\bm\{v\}\}\_\{\\theta\}is fit by regression on this target,

ℒFM​\(θ\)=𝔼𝒄,𝒙0,ϵ,t​\[‖𝒗θ​\(𝒙t,𝒄,t\)−𝒗‖22\],\\mathcal\{L\}\_\{\\mathrm\{FM\}\}\(\\theta\)\\;=\\;\\mathbb\{E\}\_\{\{\\bm\{c\}\},\\;\{\\bm\{x\}\}\_\{0\},\\;\{\\bm\{\\epsilon\}\},\\;t\}\\Big\[\\,\\big\\\|\{\\bm\{v\}\}\_\{\\theta\}\(\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\},t\)\-\{\\bm\{v\}\}\\big\\\|\_\{2\}^\{2\}\\,\\Big\],\(21\)and sampling integratesd​𝒙=𝒗θ​\(𝒙t,𝒄,t\)​d​σt\{\\rm d\}\{\\bm\{x\}\}=\{\\bm\{v\}\}\_\{\\theta\}\(\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\},t\)\\,\{\\rm d\}\\sigma\_\{t\}backward fromt=Tt\{=\}Ttot=0t\{=\}0over a discrete schedule\{σt\}t=0T\\\{\\sigma\_\{t\}\\\}\_\{t=0\}^\{T\}ofTTsteps\. Two features of Eq\. \([21](https://arxiv.org/html/2609.13425#A1.E21)\) matter for what follows: the expectation overttmakes every timestep a separate regression problem, so a per\-timestep coefficient can be attached to each without changing the estimator; and𝒗\{\\bm\{v\}\}depends on the data only through𝒙0\{\\bm\{x\}\}\_\{0\}, so tilting the data distribution by a reward moves the target while leaving the forward path Eq\. \([20](https://arxiv.org/html/2609.13425#A1.E20)\) untouched\. The latter is what lets Theorem[3\.1](https://arxiv.org/html/2609.13425#S3.Thmtheorem1)apply to the reward\-tilted marginals in Sec\.[3\.2](https://arxiv.org/html/2609.13425#S3.SS2)\.

#### DiffusionNFT\.

We briefly review the DiffusionNFT framework\[[42](https://arxiv.org/html/2609.13425#bib.bib25)\], which forms the basis of our multi\-reward training objective\. Theorems[A\.1](https://arxiv.org/html/2609.13425#A1.Thmtheorem1)and[A\.2](https://arxiv.org/html/2609.13425#A1.Thmtheorem2)below are from DiffusionNFT\[[42](https://arxiv.org/html/2609.13425#bib.bib25)\]; we restate them here, without proof, for the notation our objective builds on\.

#### Positive Policy\.

Letπold​\(𝒙0∣𝒄\)\\pi^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{0\}\\mid\{\\bm\{c\}\}\)denote the current, or old, policy andr⁡\(𝒙0,𝒄\):=p⁡\(𝒐=1∣𝒙0,𝒄\)r\(\{\\bm\{x\}\}\_\{0\},\{\\bm\{c\}\}\):=p\(\{\\bm\{o\}\}=1\\mid\{\\bm\{x\}\}\_\{0\},\{\\bm\{c\}\}\)the optimality probability\. The*positive policy*is defined as the old policy conditioned on optimality:

π\+​\(𝒙0∣𝒄\):=πold​\(𝒙0∣𝒐=1,𝒄\)=r⁡\(𝒙0,𝒄\)pπold​\(𝒐=1∣𝒄\)​πold​\(𝒙0∣𝒄\)\.\\pi^\{\+\}\(\{\\bm\{x\}\}\_\{0\}\\mid\{\\bm\{c\}\}\)\\;:=\\;\\pi^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{0\}\\mid\{\\bm\{o\}\}=1,\{\\bm\{c\}\}\)\\;=\\;\\frac\{r\(\{\\bm\{x\}\}\_\{0\},\{\\bm\{c\}\}\)\}\{p\_\{\\pi^\{\\mathrm\{old\}\}\}\(\{\\bm\{o\}\}=1\\mid\{\\bm\{c\}\}\)\}\\,\\pi^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{0\}\\mid\{\\bm\{c\}\}\)\.\(22\)With infinitely many samples fromπold\\pi^\{\\mathrm\{old\}\},π\+\\pi^\{\+\}is simply the distribution of the positive subset: the old policy restricted to its “optimal” outputs\.

#### The Coefficientα⁡\(𝒙t\)\\alpha\(\{\\bm\{x\}\}\_\{t\}\)\.

A key quantity linking the diffused positive marginal to the diffused old marginal is the scalar coefficientα⁡\(𝒙t\)∈\[0,1\]\\alpha\(\{\\bm\{x\}\}\_\{t\}\)\\in\[0,1\]:

α⁡\(𝒙t\):=πt\+​\(𝒙t∣𝒄\)πtold​\(𝒙t∣𝒄\)⋅𝔼πold​\(𝒙0∣𝒄\)​\[r⁡\(𝒙0,𝒄\)\]\.\\alpha\(\{\\bm\{x\}\}\_\{t\}\)\\;:=\\;\\frac\{\\pi\_\{t\}^\{\+\}\(\{\\bm\{x\}\}\_\{t\}\\mid\{\\bm\{c\}\}\)\}\{\\pi\_\{t\}^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{t\}\\mid\{\\bm\{c\}\}\)\}\\cdot\\mathbb\{E\}\_\{\\pi^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{0\}\\mid\{\\bm\{c\}\}\)\}\[r\(\{\\bm\{x\}\}\_\{0\},\{\\bm\{c\}\}\)\]\.\(23\)It comes out of a posterior decomposition: the old posterior splits into a mixture of the positive and negative posteriors, weighted byα⁡\(𝒙t\)\\alpha\(\{\\bm\{x\}\}\_\{t\}\)and1−α⁡\(𝒙t\)1\-\\alpha\(\{\\bm\{x\}\}\_\{t\}\)\.

###### Theorem A\.1\(Improvement direction,\[[42](https://arxiv.org/html/2609.13425#bib.bib25)\]\)\.

Let𝐯\+\{\\bm\{v\}\}^\{\+\},𝐯−\{\\bm\{v\}\}^\{\-\}, and𝐯old\{\\bm\{v\}\}^\{\\mathrm\{old\}\}be the velocity models for the policiesπ\+\\pi^\{\+\},π−\\pi^\{\-\}, andπold\\pi^\{\\mathrm\{old\}\}, respectively\. The directional differences between these models are proportional:

Δ:=\[1−α⁡\(𝒙t\)\]​\[𝒗old​\(𝒙t,𝒄,t\)−𝒗−​\(𝒙t,𝒄,t\)\]=α⁡\(𝒙t\)​\[𝒗\+​\(𝒙t,𝒄,t\)−𝒗old​\(𝒙t,𝒄,t\)\],\\Delta\\;:=\\;\\big\[1\-\\alpha\(\{\\bm\{x\}\}\_\{t\}\)\\big\]\\,\\big\[\{\\bm\{v\}\}^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\},t\)\-\{\\bm\{v\}\}^\{\-\}\(\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\},t\)\\big\]\\;=\\;\\alpha\(\{\\bm\{x\}\}\_\{t\}\)\\,\\big\[\{\\bm\{v\}\}^\{\+\}\(\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\},t\)\-\{\\bm\{v\}\}^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\},t\)\\big\],\(24\)where0≤α⁡\(𝐱t\)≤10\\leq\\alpha\(\{\\bm\{x\}\}\_\{t\}\)\\leq 1is defined in Eq\. \([23](https://arxiv.org/html/2609.13425#A1.E23)\)\.

SoΔ\\Deltais an improvement direction in velocity space: moving from𝒗old\{\\bm\{v\}\}^\{\\mathrm\{old\}\}toward𝒗\+\{\\bm\{v\}\}^\{\+\}is the same direction as moving away from𝒗−\{\\bm\{v\}\}^\{\-\}\.

###### Theorem A\.2\(Policy optimization,\[[42](https://arxiv.org/html/2609.13425#bib.bib25)\]\)\.

Consider the training objective

ℒ⁡\(θ\)=𝔼𝒄,πold​\(𝒙0∣𝒄\),ϵ,t​\[r​‖𝒗θ\+​\(𝒙t,𝒄,t\)−𝒗‖22\+\(1−r\)​‖𝒗θ−​\(𝒙t,𝒄,t\)−𝒗‖22\],\\mathcal\{L\}\(\\theta\)\\;=\\;\\mathbb\{E\}\_\{\{\\bm\{c\}\},\\,\\pi^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{0\}\\mid\{\\bm\{c\}\}\),\\,\{\\bm\{\\epsilon\}\},\\,t\}\\\!\\Big\[r\\,\\\|\{\\bm\{v\}\}\_\{\\theta\}^\{\+\}\(\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\},t\)\-\{\\bm\{v\}\}\\\|\_\{2\}^\{2\}\\;\+\\;\(1\-r\)\\,\\\|\{\\bm\{v\}\}\_\{\\theta\}^\{\-\}\(\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\},t\)\-\{\\bm\{v\}\}\\\|\_\{2\}^\{2\}\\Big\],\(25\)where𝐯=α˙t​𝐱0\+σ˙t​ϵ\{\\bm\{v\}\}=\\dot\{\\alpha\}\_\{t\}\\,\{\\bm\{x\}\}\_\{0\}\+\\dot\{\\sigma\}\_\{t\}\\,\{\\bm\{\\epsilon\}\}is the target velocity from the forward process, and the implicit positive and negative policies are parameterized as

𝒗θ\+​\(𝒙t,𝒄,t\)\\displaystyle\{\\bm\{v\}\}\_\{\\theta\}^\{\+\}\(\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\},t\):=\(1−β\)​𝒗old​\(𝒙t,𝒄,t\)\+β​𝒗θ​\(𝒙t,𝒄,t\),\\displaystyle\\;:=\\;\(1\-\\beta\)\\,\{\\bm\{v\}\}^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\},t\)\+\\beta\\,\{\\bm\{v\}\}\_\{\\theta\}\(\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\},t\),\(26\)𝒗θ−​\(𝒙t,𝒄,t\)\\displaystyle\{\\bm\{v\}\}\_\{\\theta\}^\{\-\}\(\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\},t\):=\(1\+β\)​𝒗old​\(𝒙t,𝒄,t\)−β​𝒗θ​\(𝒙t,𝒄,t\)\.\\displaystyle\\;:=\\;\(1\+\\beta\)\\,\{\\bm\{v\}\}^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\},t\)\-\\beta\\,\{\\bm\{v\}\}\_\{\\theta\}\(\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\},t\)\.\(27\)Given unlimited data and model capacity, the optimal solution satisfies

𝒗θ∗​\(𝒙t,𝒄,t\)=𝒗old​\(𝒙t,𝒄,t\)\+2β​Δ​\(𝒙t,𝒄,t\),\{\\bm\{v\}\}\_\{\\theta^\{\*\}\}\(\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\},t\)\\;=\\;\{\\bm\{v\}\}^\{\\mathrm\{old\}\}\(\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\},t\)\\;\+\\;\\frac\{2\}\{\\beta\}\\,\\Delta\(\{\\bm\{x\}\}\_\{t\},\{\\bm\{c\}\},t\),\(28\)whereΔ\\Deltais defined in Eq\. \([24](https://arxiv.org/html/2609.13425#A1.E24)\)\.

The optimum therefore sits along the reinforcement\-guidance directionΔ\\Delta, at strength2/β2/\\betarelative to𝒗old\{\\bm\{v\}\}^\{\\mathrm\{old\}\}, so the policy improves without any likelihood being estimated\.

### A\.2Proof of Theorem[3\.1](https://arxiv.org/html/2609.13425#S3.Thmtheorem1)

###### Proof\.

Letρ:=πa/πb\\rho:=\\pi^\{a\}/\\pi^\{b\}andMα:=𝔼πb​\[ρα\]=∫ρα​πb​𝑑𝒙=∫\(πa\)α​\(πb\)1−α​𝑑𝒙M\_\{\\alpha\}:=\\mathbb\{E\}\_\{\\pi^\{b\}\}\[\\rho^\{\\alpha\}\]=\\int\\rho^\{\\alpha\}\\,\\pi^\{b\}\\,\{\\rm d\}\{\\bm\{x\}\}=\\int\(\\pi^\{a\}\)^\{\\alpha\}\\,\(\\pi^\{b\}\)^\{1\-\\alpha\}\\,\{\\rm d\}\{\\bm\{x\}\}\. By definition,Dα=1α−1​log⁡MαD\_\{\\alpha\}=\\frac\{1\}\{\\alpha\-1\}\\log M\_\{\\alpha\}, so

dd​t​Dα=1α−1⋅M˙αMα\.\\frac\{\{\\rm d\}\}\{\{\\rm d\}t\}D\_\{\\alpha\}\\;=\\;\\frac\{1\}\{\\alpha\-1\}\\cdot\\frac\{\\dot\{M\}\_\{\\alpha\}\}\{M\_\{\\alpha\}\}\.It suffices to showM˙α=−g22α\(α−1\)∫ρα∥∇logρ∥2πbd𝒙\\dot\{M\}\_\{\\alpha\}=\-\\tfrac\{g^\{2\}\}\{2\}\\,\\alpha\(\\alpha\-1\)\\int\\rho^\{\\alpha\}\\\|\\nabla\\log\\rho\\\|^\{2\}\\,\\pi^\{b\}\\,\{\\rm d\}\{\\bm\{x\}\}\.

Step 1: DifferentiatingMαM\_\{\\alpha\}\.Sinceρ=πa/πb\\rho=\\pi^\{a\}/\\pi^\{b\}, we have∂tρ=\(∂tπa\)/πb−ρ⁡\(∂tπb\)/πb\\partial\_\{t\}\\rho=\(\\partial\_\{t\}\\pi^\{a\}\)/\\pi^\{b\}\-\\rho\\,\(\\partial\_\{t\}\\pi^\{b\}\)/\\pi^\{b\}, so

M˙α=∫\[α​ρα−1​\(∂tρ\)​πb\+ρα​∂tπb\]​𝑑𝒙=α​∫ρα−1​∂tπa​𝑑𝒙−\(α−1\)​∫ρα​∂tπb​𝑑𝒙\.\\dot\{M\}\_\{\\alpha\}\\;=\\;\\int\\big\[\\alpha\\,\\rho^\{\\alpha\-1\}\\,\(\\partial\_\{t\}\\rho\)\\,\\pi^\{b\}\+\\rho^\{\\alpha\}\\,\\partial\_\{t\}\\pi^\{b\}\\big\]\\,\{\\rm d\}\{\\bm\{x\}\}\\;=\\;\\alpha\\\!\\int\\rho^\{\\alpha\-1\}\\,\\partial\_\{t\}\\pi^\{a\}\\,\{\\rm d\}\{\\bm\{x\}\}\\;\-\\;\(\\alpha\{\-\}1\)\\\!\\int\\rho^\{\\alpha\}\\,\\partial\_\{t\}\\pi^\{b\}\\,\{\\rm d\}\{\\bm\{x\}\}\.
Step 2: Substituting the Fokker–Planck equation\.Bothπa\\pi^\{a\}andπb\\pi^\{b\}satisfy∂tπ=−∇⋅\(𝒖π\)\+g22Δπ\\partial\_\{t\}\\pi=\-\\nabla\\\!\\cdot\(\{\\bm\{u\}\}\\pi\)\+\\tfrac\{g^\{2\}\}\{2\}\\Delta\\pi, so

M˙α=α∫ρα−1\[−∇⋅\(𝒖πa\)\+g22Δπa\]d𝒙⏟=:Ia−\(α−1\)∫ρα\[−∇⋅\(𝒖πb\)\+g22Δπb\]d𝒙⏟=:Ib\.\\dot\{M\}\_\{\\alpha\}\\;=\\;\\underbrace\{\\alpha\\\!\\int\\rho^\{\\alpha\-1\}\\big\[\-\\nabla\\\!\\cdot\(\{\\bm\{u\}\}\\pi^\{a\}\)\+\\tfrac\{g^\{2\}\}\{2\}\\Delta\\pi^\{a\}\\big\]\\,\{\\rm d\}\{\\bm\{x\}\}\}\_\{=:\\,I\_\{a\}\}\\;\-\\;\\underbrace\{\(\\alpha\{\-\}1\)\\\!\\int\\rho^\{\\alpha\}\\big\[\-\\nabla\\\!\\cdot\(\{\\bm\{u\}\}\\pi^\{b\}\)\+\\tfrac\{g^\{2\}\}\{2\}\\Delta\\pi^\{b\}\\big\]\\,\{\\rm d\}\{\\bm\{x\}\}\}\_\{=:\\,I\_\{b\}\}\.
Step 3: Drift terms cancel\.For the drift part ofIaI\_\{a\}, integrate by parts \(boundary terms vanish by decay at infinity\):

−α∫ρα−1∇⋅\(𝒖πa\)d𝒙=α∫𝒖πa⋅∇\(ρα−1\)d𝒙=α\(α−1\)∫𝒖⋅∇ρρα−2πad𝒙\.\-\\alpha\\\!\\int\\rho^\{\\alpha\-1\}\\,\\nabla\\\!\\cdot\(\{\\bm\{u\}\}\\pi^\{a\}\)\\,\{\\rm d\}\{\\bm\{x\}\}\\;=\\;\\alpha\\\!\\int\{\\bm\{u\}\}\\pi^\{a\}\\\!\\cdot\\\!\\nabla\(\\rho^\{\\alpha\-1\}\)\\,\{\\rm d\}\{\\bm\{x\}\}\\;=\\;\\alpha\(\\alpha\{\-\}1\)\\\!\\int\{\\bm\{u\}\}\\\!\\cdot\\\!\\nabla\\rho\\;\\rho^\{\\alpha\-2\}\\pi^\{a\}\\,\{\\rm d\}\{\\bm\{x\}\}\.Usingπa=ρ​πb\\pi^\{a\}=\\rho\\pi^\{b\}, this equalsα\(α−1\)∫𝒖⋅∇ρρα−1πbd𝒙\\alpha\(\\alpha\{\-\}1\)\\int\{\\bm\{u\}\}\\\!\\cdot\\\!\\nabla\\rho\\;\\rho^\{\\alpha\-1\}\\pi^\{b\}\\,\{\\rm d\}\{\\bm\{x\}\}\. For the drift part ofIbI\_\{b\}:

\(α−1\)∫ρα∇⋅\(𝒖πb\)d𝒙=−\(α−1\)∫𝒖πb⋅∇\(ρα\)d𝒙=−α\(α−1\)∫𝒖⋅∇ρρα−1πbd𝒙\.\(\\alpha\{\-\}1\)\\\!\\int\\rho^\{\\alpha\}\\,\\nabla\\\!\\cdot\(\{\\bm\{u\}\}\\pi^\{b\}\)\\,\{\\rm d\}\{\\bm\{x\}\}\\;=\\;\-\(\\alpha\{\-\}1\)\\\!\\int\{\\bm\{u\}\}\\pi^\{b\}\\\!\\cdot\\\!\\nabla\(\\rho^\{\\alpha\}\)\\,\{\\rm d\}\{\\bm\{x\}\}\\;=\\;\-\\alpha\(\\alpha\{\-\}1\)\\\!\\int\{\\bm\{u\}\}\\\!\\cdot\\\!\\nabla\\rho\\;\\rho^\{\\alpha\-1\}\\pi^\{b\}\\,\{\\rm d\}\{\\bm\{x\}\}\.Hence the drift contributions fromIaI\_\{a\}andIbI\_\{b\}sum to zero\.

Step 4: Diffusion terms via integration by parts\.It remains to evaluate the diffusive parts\. For theIaI\_\{a\}diffusion term, apply Green’s first identity \(integration by parts twice\):

g22​α​∫ρα−1​Δ​πa​𝑑𝒙\\displaystyle\\tfrac\{g^\{2\}\}\{2\}\\,\\alpha\\\!\\int\\rho^\{\\alpha\-1\}\\,\\Delta\\pi^\{a\}\\,\{\\rm d\}\{\\bm\{x\}\}=−g22α∫∇\(ρα−1\)⋅∇πad𝒙\\displaystyle\\;=\\;\-\\tfrac\{g^\{2\}\}\{2\}\\,\\alpha\\\!\\int\\nabla\(\\rho^\{\\alpha\-1\}\)\\\!\\cdot\\\!\\nabla\\pi^\{a\}\\,\{\\rm d\}\{\\bm\{x\}\}=−g22α\(α−1\)∫ρα−2∇ρ⋅∇πad𝒙\.\\displaystyle\\;=\\;\-\\tfrac\{g^\{2\}\}\{2\}\\,\\alpha\(\\alpha\{\-\}1\)\\\!\\int\\rho^\{\\alpha\-2\}\\,\\nabla\\rho\\\!\\cdot\\\!\\nabla\\pi^\{a\}\\,\{\\rm d\}\{\\bm\{x\}\}\.Now use∇πa=∇\(ρπb\)=πb∇ρ\+ρ∇πb\\nabla\\pi^\{a\}=\\nabla\(\\rho\\pi^\{b\}\)=\\pi^\{b\}\\nabla\\rho\+\\rho\\nabla\\pi^\{b\}:

=−g22α\(α−1\)∫ρα−2\[πb∥∇ρ∥2\+ρ∇ρ⋅∇πb\]d𝒙\.=\-\\tfrac\{g^\{2\}\}\{2\}\\,\\alpha\(\\alpha\{\-\}1\)\\\!\\int\\rho^\{\\alpha\-2\}\\big\[\\pi^\{b\}\\\|\\nabla\\rho\\\|^\{2\}\+\\rho\\,\\nabla\\rho\\\!\\cdot\\\!\\nabla\\pi^\{b\}\\big\]\\,\{\\rm d\}\{\\bm\{x\}\}\.For theIbI\_\{b\}diffusion term:

−g22\(α−1\)∫ραΔπbd𝒙\\displaystyle\-\\tfrac\{g^\{2\}\}\{2\}\\,\(\\alpha\{\-\}1\)\\\!\\int\\rho^\{\\alpha\}\\,\\Delta\\pi^\{b\}\\,\{\\rm d\}\{\\bm\{x\}\}=g22​\(α−1\)​∫∇\(ρα\)⋅∇πb​𝑑𝒙\\displaystyle\\;=\\;\\tfrac\{g^\{2\}\}\{2\}\\,\(\\alpha\{\-\}1\)\\\!\\int\\nabla\(\\rho^\{\\alpha\}\)\\\!\\cdot\\\!\\nabla\\pi^\{b\}\\,\{\\rm d\}\{\\bm\{x\}\}=g22α\(α−1\)∫ρα−1∇ρ⋅∇πbd𝒙\.\\displaystyle\\;=\\;\\tfrac\{g^\{2\}\}\{2\}\\,\\alpha\(\\alpha\{\-\}1\)\\\!\\int\\rho^\{\\alpha\-1\}\\,\\nabla\\rho\\\!\\cdot\\\!\\nabla\\pi^\{b\}\\,\{\\rm d\}\{\\bm\{x\}\}\.Adding the two diffusive contributions, the∇ρ⋅∇πb\\nabla\\rho\\\!\\cdot\\\!\\nabla\\pi^\{b\}terms cancel:

−g22α\(α−1\)∫ρα−2ρ∇ρ⋅∇πbd𝒙\+g22α\(α−1\)∫ρα−1∇ρ⋅∇πbd𝒙=0,\-\\tfrac\{g^\{2\}\}\{2\}\\,\\alpha\(\\alpha\{\-\}1\)\\\!\\int\\rho^\{\\alpha\-2\}\\,\\rho\\,\\nabla\\rho\\\!\\cdot\\\!\\nabla\\pi^\{b\}\\,\{\\rm d\}\{\\bm\{x\}\}\\;\+\\;\\tfrac\{g^\{2\}\}\{2\}\\,\\alpha\(\\alpha\{\-\}1\)\\\!\\int\\rho^\{\\alpha\-1\}\\,\\nabla\\rho\\\!\\cdot\\\!\\nabla\\pi^\{b\}\\,\{\\rm d\}\{\\bm\{x\}\}\\;=\\;0,and we are left with

M˙α=−g​\(t\)22α\(α−1\)∫ρα−2∥∇ρ∥2πbd𝒙\.\\dot\{M\}\_\{\\alpha\}\\;=\\;\-\\tfrac\{g\(t\)^\{2\}\}\{2\}\\,\\alpha\(\\alpha\{\-\}1\)\\\!\\int\\rho^\{\\alpha\-2\}\\,\\\|\\nabla\\rho\\\|^\{2\}\\,\\pi^\{b\}\\,\{\\rm d\}\{\\bm\{x\}\}\.
Step 5: Rewrite in terms of∇log⁡ρ\\nabla\\log\\rho\.Since∇ρ=ρ∇logρ\\nabla\\rho=\\rho\\,\\nabla\\log\\rho, we haveρα−2​‖∇ρ‖2=ρα​‖∇log⁡ρ‖2\\rho^\{\\alpha\-2\}\\\|\\nabla\\rho\\\|^\{2\}=\\rho^\{\\alpha\}\\\|\\nabla\\log\\rho\\\|^\{2\}, giving

M˙α=−g​\(t\)22α\(α−1\)∫ρα∥∇logρ∥2πbd𝒙\.\\dot\{M\}\_\{\\alpha\}\\;=\\;\-\\tfrac\{g\(t\)^\{2\}\}\{2\}\\,\\alpha\(\\alpha\{\-\}1\)\\\!\\int\\rho^\{\\alpha\}\\,\\\|\\nabla\\log\\rho\\\|^\{2\}\\,\\pi^\{b\}\\,\{\\rm d\}\{\\bm\{x\}\}\.Sincedd​t​Dα=1α−1⋅M˙αMα\\frac\{\{\\rm d\}\}\{\{\\rm d\}t\}D\_\{\\alpha\}=\\frac\{1\}\{\\alpha\-1\}\\cdot\\frac\{\\dot\{M\}\_\{\\alpha\}\}\{M\_\{\\alpha\}\}, the factor\(α−1\)\(\\alpha\-1\)cancels, givingdd​tDα=−g​\(t\)22⋅α⋅∫ρα​‖∇log⁡ρ‖2​πb​𝑑𝒙∫ρα​πb​𝑑𝒙=−g​\(t\)22⋅α⋅𝔼π\(α\)\[∥∇logρ∥2\]\\frac\{\{\\rm d\}\}\{\{\\rm d\}t\}D\_\{\\alpha\}=\-\\frac\{g\(t\)^\{2\}\}\{2\}\\cdot\\alpha\\cdot\\frac\{\\int\\rho^\{\\alpha\}\\\|\\nabla\\log\\rho\\\|^\{2\}\\pi^\{b\}\\,\{\\rm d\}\{\\bm\{x\}\}\}\{\\int\\rho^\{\\alpha\}\\pi^\{b\}\\,\{\\rm d\}\{\\bm\{x\}\}\}=\-\\frac\{g\(t\)^\{2\}\}\{2\}\\cdot\\alpha\\cdot\\mathbb\{E\}\_\{\\pi^\{\(\\alpha\)\}\}\[\\\|\\nabla\\log\\rho\\\|^\{2\}\], which is Eq\. \([10](https://arxiv.org/html/2609.13425#S3.E10)\)\. ∎

### A\.3Error induced by the surrogate

We quantify the error introduced by replacing the reward\-specific positive policyπi\+\\pi\_\{i\}^\{\+\}with a common external positive policyπ\+\\pi^\{\+\}when estimating the reward\-specific temporal weights\. The main result shows that this approximation is stable whenπ\+\\pi^\{\+\}is close toπi\+\\pi\_\{i\}^\{\+\}in KL divergence\. Throughout this subsection we fix a prompt𝒄\{\\bm\{c\}\}, suppress it from the notation, and takeα\>1\\alpha\>1as in the main text\.

#### Bounding the discriminability error\.

Assume the reward is bounded,0≤ri​\(𝒙0\)≤Ri0\\leq r\_\{i\}\(\{\\bm\{x\}\}\_\{0\}\)\\leq R\_\{i\}withZi\>0Z\_\{i\}\>0, so that0≤ρi,t​\(𝒙t\)≤Mi:=Ri/Zi0\\leq\\rho\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)\\leq M\_\{i\}:=R\_\{i\}/Z\_\{i\}for everytt, and define the policy mismatchϵi:=KL\(π\+∥πi\+\)\\epsilon\_\{i\}:=\\mathrm\{KL\}\\big\(\\pi^\{\+\}\\,\\\|\\,\\pi\_\{i\}^\{\+\}\\big\)\. Becauseπt\+\\pi\_\{t\}^\{\+\}andπi,t\+\\pi\_\{i,t\}^\{\+\}are obtained by applying the same diffusion kernel, the data\-processing inequality givesKL\(πt\+∥πi,t\+\)≤ϵi\\mathrm\{KL\}\\big\(\\pi\_\{t\}^\{\+\}\\,\\\|\\,\\pi\_\{i,t\}^\{\+\}\\big\)\\leq\\epsilon\_\{i\}, and Pinsker’s inequality then givesTV⁡\(πt\+,πi,t\+\)≤ϵi/2\\mathrm\{TV\}\\big\(\\pi\_\{t\}^\{\+\},\\pi\_\{i,t\}^\{\+\}\\big\)\\leq\\sqrt\{\\epsilon\_\{i\}/2\}\. Sincefi,t:=ρi,tα−1f\_\{i,t\}:=\\rho\_\{i,t\}^\{\\alpha\-1\}takes values in\[0,Miα−1\]\[0,M\_\{i\}^\{\\alpha\-1\}\],

\|𝔼πt\+​\[fi,t\]−𝔼πi,t\+​\[fi,t\]\|≤Miα−1​TV​\(πt\+,πi,t\+\)≤Miα−1​ϵi2=:ηi\.\\Big\|\\mathbb\{E\}\_\{\\pi\_\{t\}^\{\+\}\}\[f\_\{i,t\}\]\-\\mathbb\{E\}\_\{\\pi\_\{i,t\}^\{\+\}\}\[f\_\{i,t\}\]\\Big\|\\;\\leq\\;M\_\{i\}^\{\\alpha\-1\}\\,\\mathrm\{TV\}\\big\(\\pi\_\{t\}^\{\+\},\\pi\_\{i,t\}^\{\+\}\\big\)\\;\\leq\\;M\_\{i\}^\{\\alpha\-1\}\\sqrt\{\\frac\{\\epsilon\_\{i\}\}\{2\}\}\\;=:\\;\\eta\_\{i\}\.\(29\)Moreover𝔼πi,t\+​\[fi,t\]=exp⁡\(\(α−1\)​Di,t\)≥1\\mathbb\{E\}\_\{\\pi\_\{i,t\}^\{\+\}\}\[f\_\{i,t\}\]=\\exp\\big\(\(\\alpha\-1\)D\_\{i,t\}\\big\)\\geq 1\. WriteD~i,t:=1α−1​log⁡𝔼πt\+​\[fi,t\]\\widetilde\{D\}\_\{i,t\}:=\\frac\{1\}\{\\alpha\-1\}\\log\\mathbb\{E\}\_\{\\pi\_\{t\}^\{\+\}\}\[f\_\{i,t\}\]for the surrogate log\-moment, whose differences give the surrogate gain of Eq\. \([12](https://arxiv.org/html/2609.13425#S3.E12)\)\. Wheneverηi<1\\eta\_\{i\}<1, Eq\. \([29](https://arxiv.org/html/2609.13425#A1.E29)\) yields

\|D~i,t−Di,t\|≤bi:=−log⁡\(1−ηi\)α−1=Miα−1α−1​ϵi2\+O⁡\(ϵi\),\\big\|\\widetilde\{D\}\_\{i,t\}\-D\_\{i,t\}\\big\|\\;\\leq\\;b\_\{i\}:=\\frac\{\-\\log\(1\-\\eta\_\{i\}\)\}\{\\alpha\-1\}\\;=\\;\\frac\{M\_\{i\}^\{\\alpha\-1\}\}\{\\alpha\-1\}\\sqrt\{\\frac\{\\epsilon\_\{i\}\}\{2\}\}\+O\(\\epsilon\_\{i\}\),\(30\)so the surrogate divergence is accurate toO⁡\(ϵi\)O\\big\(\\sqrt\{\\epsilon\_\{i\}\}\\big\)\.

#### Error in the per\-timestep gain\.

Applying Eq\. \([30](https://arxiv.org/html/2609.13425#A1.E30)\) at the two adjacent timesteps of the gainΔ​Di,t=Di,t−1−Di,t\\Delta D\_\{i,t\}=D\_\{i,t\-1\}\-D\_\{i,t\}and of its surrogateΔ​D~i,t=D~i,t−1−D~i,t\\widetilde\{\\Delta D\}\_\{i,t\}=\\widetilde\{D\}\_\{i,t\-1\}\-\\widetilde\{D\}\_\{i,t\}gives

\|Δ​D~i,t−Δ​Di,t\|≤2​bi≲2​Miα−1α−1​ϵi,\\big\|\\widetilde\{\\Delta D\}\_\{i,t\}\-\\Delta D\_\{i,t\}\\big\|\\;\\leq\\;2b\_\{i\}\\;\\lesssim\\;\\frac\{\\sqrt\{2\}\\,M\_\{i\}^\{\\alpha\-1\}\}\{\\alpha\-1\}\\sqrt\{\\epsilon\_\{i\}\},\(31\)where the final comparison holds in the small\-mismatch regime\.

#### Error after normalizing the temporal profile\.

The allocation uses the mean\-normalized gain of Eq\. \([13](https://arxiv.org/html/2609.13425#S3.E13)\),Δ​D¯i,t=T​Δ​Di,t/Gi\\overline\{\\Delta D\}\_\{i,t\}=T\\,\\Delta D\_\{i,t\}/G\_\{i\}withGi:=∑tΔ​Di,tG\_\{i\}:=\\sum\_\{t\}\\Delta D\_\{i,t\}, and its surrogateΔ​D¯~i,t:=T​Δ​D~i,t/G~i\\widetilde\{\\overline\{\\Delta D\}\}\_\{i,t\}:=T\\,\\widetilde\{\\Delta D\}\_\{i,t\}/\\widetilde\{G\}\_\{i\}\. Because the gains telescope,Gi=Di,0−Di,TG\_\{i\}=D\_\{i,0\}\-D\_\{i,T\}andG~i=D~i,0−D~i,T\\widetilde\{G\}\_\{i\}=\\widetilde\{D\}\_\{i,0\}\-\\widetilde\{D\}\_\{i,T\}, so\|G~i−Gi\|≤2​bi\|\\widetilde\{G\}\_\{i\}\-G\_\{i\}\|\\leq 2b\_\{i\}\. IfGi\>2​biG\_\{i\}\>2b\_\{i\}, then for everytt

\|Δ​D¯~i,t−Δ​D¯i,t\|≤T​\|Δ​D~i,t−Δ​Di,t\|G~i\+Δ​D¯i,t​\|Gi−G~i\|G~i≤4​T​biGi−2​bi=O⁡\(T​Miα−1\(α−1\)​Gi​ϵi\)\.\\Big\|\\widetilde\{\\overline\{\\Delta D\}\}\_\{i,t\}\-\\overline\{\\Delta D\}\_\{i,t\}\\Big\|\\leq T\\,\\frac\{\|\\widetilde\{\\Delta D\}\_\{i,t\}\-\\Delta D\_\{i,t\}\|\}\{\\widetilde\{G\}\_\{i\}\}\+\\overline\{\\Delta D\}\_\{i,t\}\\,\\frac\{\|G\_\{i\}\-\\widetilde\{G\}\_\{i\}\|\}\{\\widetilde\{G\}\_\{i\}\}\\leq\\frac\{4T\\,b\_\{i\}\}\{G\_\{i\}\-2b\_\{i\}\}=O\\\!\\left\(\\frac\{T\\,M\_\{i\}^\{\\alpha\-1\}\}\{\(\\alpha\-1\)\\,G\_\{i\}\}\\sqrt\{\\epsilon\_\{i\}\}\\right\)\.\(32\)

#### Propagation to Sinkhorn weights\.

Finally, consider the entropy\-regularized allocation, written with the cost notation of Eq\. \([17](https://arxiv.org/html/2609.13425#S3.E17)\),

𝑾\(𝑪\)=arg​min𝐖∈𝒰⁡\(𝝀,𝝂\)∑i,t\[Ci,tWi,t\+τWi,tlogWi,t\],Ci,t:=−Δ​D¯i,t,\\bm\{W\}\(\\bm\{C\}\)=\\argmin\_\{\\bm\{W\}\\in\\mathcal\{U\}\(\\bm\{\\lambda\},\\bm\{\\nu\}\)\}\\;\\sum\_\{i,t\}\\Big\[\\,C\_\{i,t\}\\,W\_\{i,t\}\+\\tau\\,W\_\{i,t\}\\log W\_\{i,t\}\\,\\Big\],\\qquad C\_\{i,t\}:=\-\\overline\{\\Delta D\}\_\{i,t\},\(33\)where𝒰⁡\(𝝀,𝝂\)\\mathcal\{U\}\(\\bm\{\\lambda\},\\bm\{\\nu\}\)fixes the reward marginals to𝝀\\bm\{\\lambda\}and the timestep marginals to𝝂\\bm\{\\nu\},τ\>0\\tau\>0is the Sinkhorn temperature, andB:=∑iλi=∑tνtB:=\\sum\_\{i\}\\lambda\_\{i\}=\\sum\_\{t\}\\nu\_\{t\}is the total transported mass; atνt≡1/T\\nu\_\{t\}\\equiv 1/Tandτ=1\\tau=1, Eq\. \([33](https://arxiv.org/html/2609.13425#A1.E33)\) is exactly Eq\. \([17](https://arxiv.org/html/2609.13425#S3.E17)\), with feasible set𝒰⁡\(𝝀\)\\mathcal\{U\}\(\\bm\{\\lambda\}\)of Eq\. \([15](https://arxiv.org/html/2609.13425#S3.E15)\) and solution𝑾⋆\\bm\{W\}^\{\\star\}\. Let𝑾⋆:=𝑾⁡\(𝑪\)\\bm\{W\}^\{\\star\}:=\\bm\{W\}\(\\bm\{C\}\)and𝑾~:=𝑾⁡\(𝑪~\)\\widetilde\{\\bm\{W\}\}:=\\bm\{W\}\(\\widetilde\{\\bm\{C\}\}\), where𝑪~:=\(−Δ​D¯~i,t\)i,t\\widetilde\{\\bm\{C\}\}:=\\big\(\-\\widetilde\{\\overline\{\\Delta D\}\}\_\{i,t\}\\big\)\_\{i,t\}is the cost built from the surrogate gains\. The entropic term isτ/B\\tau/B\-strongly convex with respect to theℓ1\\ell\_\{1\}norm on measures of massBB, so the optimality conditions for Eq\. \([33](https://arxiv.org/html/2609.13425#A1.E33)\) at𝑾⋆\\bm\{W\}^\{\\star\}and𝑾~\\widetilde\{\\bm\{W\}\}giveτB​‖𝑾~−𝑾⋆‖12≤⟨𝑪−𝑪~,𝑾~−𝑾⋆⟩\\frac\{\\tau\}\{B\}\\\|\\widetilde\{\\bm\{W\}\}\-\\bm\{W\}^\{\\star\}\\\|\_\{1\}^\{2\}\\leq\\langle\\bm\{C\}\-\\widetilde\{\\bm\{C\}\},\\,\\widetilde\{\\bm\{W\}\}\-\\bm\{W\}^\{\\star\}\\rangle, and Hölder’s inequality then yields the stability estimate‖𝑾~−𝑾⋆‖1≤Bτ​‖𝑪~−𝑪‖∞\\\|\\widetilde\{\\bm\{W\}\}\-\\bm\{W\}^\{\\star\}\\\|\_\{1\}\\leq\\frac\{B\}\{\\tau\}\\\|\\widetilde\{\\bm\{C\}\}\-\\bm\{C\}\\\|\_\{\\infty\}\. Since‖𝑪~−𝑪‖∞\\\|\\widetilde\{\\bm\{C\}\}\-\\bm\{C\}\\\|\_\{\\infty\}is bounded by Eq\. \([32](https://arxiv.org/html/2609.13425#A1.E32)\), the end\-to\-end bound for sufficiently small policy mismatch is

‖𝑾~−𝑾⋆‖1≤4​B​Tτ​maxi​biGi−2​bi=O⁡\(B​Tτ​maxi​Miα−1\(α−1\)​Gi​KL\(π\+∥πi\+\)\)\.\{\\\|\\widetilde\{\\bm\{W\}\}\-\\bm\{W\}^\{\\star\}\\\|\_\{1\}\\;\\leq\\;\\frac\{4BT\}\{\\tau\}\\max\_\{i\}\\frac\{b\_\{i\}\}\{G\_\{i\}\-2b\_\{i\}\}\\;=\\;O\\\!\\left\(\\frac\{BT\}\{\\tau\}\\max\_\{i\}\\frac\{M\_\{i\}^\{\\alpha\-1\}\}\{\(\\alpha\-1\)\\,G\_\{i\}\}\\sqrt\{\\mathrm\{KL\}\\big\(\\pi^\{\+\}\\\|\\pi\_\{i\}^\{\+\}\\big\)\}\\right\)\.\}\(34\)Eq\. \([34](https://arxiv.org/html/2609.13425#A1.E34)\) gives a direct interpretation of the approximation: a shared positive policy produces similar temporal weights whenever it is close to the reward\-specific positive policy, with the square root coming from Pinsker’s inequality\. Rewards with larger total discriminabilityGiG\_\{i\}are less sensitive to the approximation, while a smaller Sinkhorn temperatureτ\\taumakes the final allocation more sensitive to errors in the estimated temporal profile\.

## Appendix BGain estimation: algorithm,α\\alpha\-sensitivity, and curve details

### B\.1Algorithm

Alg\.[2](https://arxiv.org/html/2609.13425#alg2)writes out Alg\.[1](https://arxiv.org/html/2609.13425#alg1)in full, separated into its three phases, and implements the estimator of Sec\.[3\.3](https://arxiv.org/html/2609.13425#S3.SS3)\. Phase 1 forward\-noises cleanπ\+\\pi^\{\+\}samples to obtain𝒙t∼πt\+\{\\bm\{x\}\}\_\{t\}\\sim\\pi\_\{t\}^\{\+\}; Phase 2 rolls out underπold\\pi^\{\\mathrm\{old\}\}to estimate the unnormalized conditional reward at each𝒙t\{\\bm\{x\}\}\_\{t\}; Phase 3 differences the resulting log\-moments, at which point the marginal𝔼old​\[ri∣𝒄\]\\mathbb\{E\}^\{\\mathrm\{old\}\}\[r\_\{i\}\\mid\{\\bm\{c\}\}\]drops out and never has to be formed, leaving the surrogate gain of Eq\. \([12](https://arxiv.org/html/2609.13425#S3.E12)\)\.

Algorithm 2Estimation of per\-step Rényi discriminability gain proxiesΔ​D~i,t^\\widehat\{\\widetilde\{\\Delta D\}\_\{i,t\}\}fromπ\+\\pi^\{\+\}\(the expanded form of Alg\.[1](https://arxiv.org/html/2609.13425#alg1)\)1:Prompt

𝒄\{\\bm\{c\}\}, current policy

πold\\pi^\{\\mathrm\{old\}\}, external high\-reward generator approximating

π\+​\(𝒙0∣𝒄\)\\pi^\{\+\}\(\{\\bm\{x\}\}\_\{0\}\\mid\{\\bm\{c\}\}\), rewards

r1,…,rmr\_\{1\},\\dots,r\_\{m\}, Rényi order

α\>0\\alpha\>0with

α≠1\\alpha\\neq 1, number of

π\+\\pi^\{\+\}samples

NN, number of rollouts

KK, noise schedule

\{σt\}t=0T\\\{\\sigma\_\{t\}\\\}\_\{t=0\}^\{T\}
2:Per\-step gain proxies

Δ​D~i,t^\\widehat\{\\widetilde\{\\Delta D\}\_\{i,t\}\}for each reward

iiand timestep

tt
3:Phase 1: Samplext∼πt\+\{\\bm\{x\}\}\_\{t\}\\sim\\pi\_\{t\}^\{\+\}by forward\-noising cleanπ\+\\pi^\{\+\}samples

4:for

n=1,…,Nn=1,\\dots,Ndo

5:Draw clean sample

𝒙~0\(n\)∼π\+\(⋅∣𝒄\)\\widetilde\{\{\\bm\{x\}\}\}\_\{0\}^\{\(n\)\}\\sim\\pi^\{\+\}\(\\cdot\\mid\{\\bm\{c\}\}\)from the stronger external generator

6:foreach timestep

t∈\{0,1,…,T\}t\\in\\\{0,1,\\dots,T\\\}do

7:Sample

ϵt\(n\)∼𝒩⁡\(0,I\)\{\\bm\{\\epsilon\}\}\_\{t\}^\{\(n\)\}\\sim\\mathcal\{N\}\(0,I\)and set

𝒙t\(n\)←\(1−σt\)​𝒙~0\(n\)\+σt​ϵt\(n\)\{\\bm\{x\}\}\_\{t\}^\{\(n\)\}\\leftarrow\(1\-\\sigma\_\{t\}\)\\,\\widetilde\{\{\\bm\{x\}\}\}\_\{0\}^\{\(n\)\}\+\\sigma\_\{t\}\\,\{\\bm\{\\epsilon\}\}\_\{t\}^\{\(n\)\}//𝐱t\(n\)∼πt\+\{\\bm\{x\}\}\_\{t\}^\{\(n\)\}\\sim\\pi\_\{t\}^\{\+\}

8:endfor

9:endfor

10:Phase 2: Rollouts underπold\\pi^\{\\mathrm\{old\}\}from eachxt\(n\)\{\\bm\{x\}\}\_\{t\}^\{\(n\)\}to estimate the conditional reward

11:for

n=1,…,Nn=1,\\dots,Ndo

12:foreach timestep

t∈\{0,1,…,T\}t\\in\\\{0,1,\\dots,T\\\}do

13:if

t=0t=0then//𝐱~0\(n\)\\widetilde\{\{\\bm\{x\}\}\}\_\{0\}^\{\(n\)\}is already clean: no rollout, single deterministic evaluation

14:

𝒙~0\(n,0,1\)←𝒙~0\(n\)\\widetilde\{\\bm\{x\}\}\_\{0\}^\{\(n,0,1\)\}\\leftarrow\\widetilde\{\{\\bm\{x\}\}\}\_\{0\}^\{\(n\)\}; evaluate

ri​\(𝒙~0\(n,0,1\),𝒄\)r\_\{i\}\(\\widetilde\{\\bm\{x\}\}\_\{0\}^\{\(n,0,1\)\},\{\\bm\{c\}\}\)for all

ii
15:else

16:for

k=1,…,Kk=1,\\dots,Kdo

17:Run independent rollout from

𝒙t\(n\)\{\\bm\{x\}\}\_\{t\}^\{\(n\)\}to completion under

πold\\pi^\{\\mathrm\{old\}\}:

𝒙t\(n\)→𝒙0\(n,t,k\)\{\\bm\{x\}\}\_\{t\}^\{\(n\)\}\\to\{\\bm\{x\}\}\_\{0\}^\{\(n,t,k\)\}
18:Evaluate

ri​\(𝒙0\(n,t,k\),𝒄\)r\_\{i\}\(\{\\bm\{x\}\}\_\{0\}^\{\(n,t,k\)\},\{\\bm\{c\}\}\)for all rewards

ii
19:endfor

20:endif

21:endfor

22:endfor

23:Phase 3: Per\-step gains, with denominator cancellation

24:foreach reward

iido

25:foreach timestep

t∈\{0,1,…,T\}t\\in\\\{0,1,\\dots,T\\\}do

26:for

n=1,…,Nn=1,\\dots,Ndo

27:

μ^i,t\(n\)←1Kt​∑k=1Ktri​\(𝒙0\(n,t,k\),𝒄\)\\hat\{\\mu\}\_\{i,t\}^\{\(n\)\}\\leftarrow\\frac\{1\}\{K\_\{t\}\}\\sum\_\{k=1\}^\{K\_\{t\}\}r\_\{i\}\(\{\\bm\{x\}\}\_\{0\}^\{\(n,t,k\)\},\{\\bm\{c\}\}\), with

Kt=1K\_\{t\}=1if

t=0t=0else

KK// Conditional reward at𝐱t\(n\)\{\\bm\{x\}\}\_\{t\}^\{\(n\)\}

28:endfor

29:

D~i,t^←1α−1​log⁡\(max⁡\{10−10,1N​∑n=1N\[μ^i,t\(n\)\]α−1\}\)\\widehat\{\\widetilde\{D\}\_\{i,t\}\}\\leftarrow\\frac\{1\}\{\\alpha\-1\}\\log\\\!\\Big\(\\max\\\{10^\{\-10\},\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}\\big\[\\hat\{\\mu\}\_\{i,t\}^\{\(n\)\}\\big\]^\{\\alpha\-1\}\\\}\\Big\)// Surrogate log\-moment, with a numerical floor

30:endfor

31:for

t=1,…,Tt=1,\\dots,Tdo

32:

Δ​D~i,t^←\[D~i,t−1^−D~i,t^\]\+\\widehat\{\\widetilde\{\\Delta D\}\_\{i,t\}\}\\leftarrow\\big\[\\widehat\{\\widetilde\{D\}\_\{i,t\-1\}\}\-\\widehat\{\\widetilde\{D\}\_\{i,t\}\}\\big\]\_\{\+\}// Eq\. \([12](https://arxiv.org/html/2609.13425#S3.E12)\), clipped as in Sec\.[3\.4](https://arxiv.org/html/2609.13425#S3.SS4)

33:endfor

34:endfor

### B\.2α\\alpha\-sensitivity of the Rényi estimator

To analyze the influence ofα\\alpha, we sweepα∈\{0\.25,0\.5,0\.75,1\.5,2,4,8\}\\alpha\\in\\\{0\.25,\\,0\.5,\\,0\.75,\\,1\.5,\\,2,\\,4,\\,8\\\}and plot per\-step gain, mean\-normalized gain, and the Sinkhorn matrix they project to in Fig\.[2](https://arxiv.org/html/2609.13425#A2.F2)\. The sweep covers the five rewards estimated jointly on the sharedT=10T\{=\}10grid\. For0<α<10<\\alpha<1the exponent\(α−1\)\(\\alpha\-1\)is negative, so the log\-moment is dominated by the lower tail ofμ^i,t\\hat\{\\mu\}\_\{i,t\}rather than the upper one\. Of the two guarantees we rely on, only one needsα\>1\\alpha\>1: Theorem[3\.1](https://arxiv.org/html/2609.13425#S3.Thmtheorem1)givesΔ​Di,t≥0\\Delta D\_\{i,t\}\\geq 0for everyα\>0\\alpha\>0withα≠1\\alpha\\neq 1, whereasDαD\_\{\\alpha\}upper\-bounds the KL only above11\. Below11the gain therefore keeps its monotonicity but loses its link to the original objective, and we report these orders only to show what the estimator does outside its intended range\. Peak locations are stable through the middle of the range and move only at its ends:Aestheticpeaks at the clean end forα≤0\.75\\alpha\\leq 0\.75but near the noisy end forα≥1\.5\\alpha\\geq 1\.5, and atα=8\\alpha\{=\}8HPSv2andPickScorelose their own peaks closer to clean and collapse ontoImageReward’s att=9t\{=\}9\. Raisingα\\alphaalso localizes the gains:ImageReward’s normalized peak grows from1\.61×1\.61\\timesits own mean atα=0\.25\\alpha\{=\}0\.25to2\.12×2\.12\\timesatα=2\\alpha\{=\}2and3\.25×3\.25\\timesatα=8\\alpha\{=\}8\. The projected weights follow, and their excursions from uniform are mildest in the intermediate range, spanning0\.290\.29to1\.601\.60times1/\(m​T\)1/\(mT\)atα=2\\alpha\{=\}2against0\.240\.24to1\.891\.89atα=0\.25\\alpha\{=\}0\.25and0\.410\.41to2\.242\.24atα=8\\alpha\{=\}8, which is why we default toα=2\\alpha=2in the main text\.

Figure 2:α\\alpha\-sensitivity of theπ\+\\pi^\{\+\}\-sampling Rényi estimator on the SD3\.5\-Medium run of Sec\.[3\.5](https://arxiv.org/html/2609.13425#S3.SS5)\.Rows:α∈\{0\.25,0\.5,0\.75,1\.5,2,4,8\}\\alpha\\in\\\{0\.25,\\,0\.5,\\,0\.75,\\,1\.5,\\,2,\\,4,\\,8\\\}\.Columns:the same three quantities as Fig\.[1](https://arxiv.org/html/2609.13425#S3.F1), per\-step gainΔ​Di,t\\Delta D\_\{i,t\}\(left\), mean\-normalized gainΔ​D¯i,t\\overline\{\\Delta D\}\_\{i,t\}\(Eq\. \([13](https://arxiv.org/html/2609.13425#S3.E13)\), center\), and the Sinkhorn matrixWi,t⋆W\_\{i,t\}^\{\\star\}\(Eq\. \([18](https://arxiv.org/html/2609.13425#S3.E18)\)\) at uniformλi=1/m\\lambda\_\{i\}\{=\}1/m\(right\)\.
### B\.3π\+\\pi^\{\+\}vs\.πold\\pi^\{\\mathrm\{old\}\}self\-rollout gain curves

In this section, we validate the design ofπ\+\\pi^\{\+\}sampling\. To this end, we run the identical estimator and Sinkhorn recipe withπold\\pi^\{\\mathrm\{old\}\}self\-rollouts in place ofπ\+\\pi^\{\+\}samples, replacing the exponentα−1\\alpha\-1byα\\alphaso that both runs target the sameDi,tD\_\{i,t\}, since𝔼πtold​\[ρi,t​\(𝒙t\)α\]=𝔼πi,t\+​\[ρi,t​\(𝒙t\)α−1\]\\mathbb\{E\}\_\{\\pi\_\{t\}^\{\\mathrm\{old\}\}\}\[\\rho\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)^\{\\alpha\}\]=\\mathbb\{E\}\_\{\\pi\_\{i,t\}^\{\+\}\}\[\\rho\_\{i,t\}\(\{\\bm\{x\}\}\_\{t\}\)^\{\\alpha\-1\}\]\(Sec\.[3\.3](https://arxiv.org/html/2609.13425#S3.SS3)\), holding the training stage\-OCR prompt set, the four training rewards \(ClipScore,HPSv2,OCR,PickScore\),α=2\\alpha\{=\}2andT=25T\{=\}25fixed, so the two runs differ only in the sampling distribution\. As shown in Fig\.[3](https://arxiv.org/html/2609.13425#A2.F3), theπ\+\\pi^\{\+\}\-sampledΔ​D\\Delta Dcurves \(top\) are smooth and single\-peaked, with the peak location differing by reward, while theπold\\pi^\{\\mathrm\{old\}\}\-sampled curves are jagged and non\-monotonic step\-to\-step, with per\-step gain repeatedly spiking and collapsing back to the zero clip\. The leftmost column makes the same point quantitatively\. At the clean end theπold\\pi^\{\\mathrm\{old\}\}estimate givesDi,tD\_\{i,t\}of0\.0160\.016,0\.0200\.020, and0\.0010\.001forClipScore,HPSv2, andPickScore, against0\.2450\.245,0\.5010\.501, and0\.2090\.209underπ\+\\pi^\{\+\}: a1515–200×200\\timescollapse, and precisely theρi,t≈1\\rho\_\{i,t\}\\approx 1,Di,t≈0D\_\{i,t\}\\approx 0degeneracy anticipated in Sec\.[3\.3](https://arxiv.org/html/2609.13425#S3.SS3)\.OCRis the one reward that retains appreciable discriminability, but itsΔ​D\\Delta Dcurve is unstable intt\. The Sinkhorn matrices built from these noisy curves \(right column\) are correspondingly noisy themselves, oscillating step\-to\-step rather than smoothly redistributing weight toward each reward’s high\-gain region\. In short:π\+\\pi^\{\+\}sampling is what makes the estimated curve usable as a Sinkhorn kernel input\.

#### The jaggedness is not an artifact of the budget\.

A natural objection is that theπold\\pi^\{\\mathrm\{old\}\}curves are jagged only because the estimation budget is small\. The bottom row of Fig\.[3](https://arxiv.org/html/2609.13425#A2.F3)rules this out: it recomputes the sameπold\\pi^\{\\mathrm\{old\}\}run atN=32N\{=\}32andK=64K\{=\}64, a4×4\\timesincrease in both the samples per prompt and the rollouts per𝒙t\{\\bm\{x\}\}\_\{t\}and hence16×16\\timesthe rollout cost, so the twoπold\\pi^\{\\mathrm\{old\}\}rows differ in\(N,K\)\(N,K\)alone\. TheΔ​D\\Delta Dcurves stay just as jagged, so the shape the Sinkhorn kernel would consume is not yet stable\. The extra budget therefore almost does not improve smoothness\.π\+\\pi^\{\+\}sampling instead reaches a usable curve at1/161/16of the rollout cost of this row\.

Figure 3:Cumulative discriminabilityDi,tD\_\{i,t\}, per\-step gainΔ​Di,t\\Delta D\_\{i,t\}, mean\-normalized gainΔ​D¯i,t\\overline\{\\Delta D\}\_\{i,t\}, and Sinkhorn matrixWi,t⋆W\_\{i,t\}^\{\\star\}at uniformλi=1/m\\lambda\_\{i\}\{=\}1/m, left to right in the order they are derived, under the two sampling distributions\. The leftmost column is the estimator’s ownDi,tD\_\{i,t\}, shifted by the singlett\-independent constant that setsDi,T=0D\_\{i,T\}\{=\}0at pure noise, whereπi,T\+=πTold\\pi\_\{i,T\}^\{\+\}\{=\}\\pi\_\{T\}^\{\\mathrm\{old\}\}; it is not a cumulative sum of the clipped gains beside it\. Underπold\\pi^\{\\mathrm\{old\}\}almost all of the estimated discriminability sits onOCR, the other three rewards coming out nearly indiscriminable at every step, and theOCRcurve is not even monotone intt: it dips below zero, which Theorem[3\.1](https://arxiv.org/html/2609.13425#S3.Thmtheorem1)rules out in population\.Top:π\+\\pi^\{\+\}samples fromGPT Image 1\.5, at theN=8N\{=\}8,K=16K\{=\}16budget used throughout the paper\.Middle:πold\\pi^\{\\mathrm\{old\}\}self\-rollouts at that same budget\.Bottom:the sameπold\\pi^\{\\mathrm\{old\}\}run at the fullN=32N\{=\}32,K=64K\{=\}64estimation budget,16×16\\timesthe rollout cost\. All three rows are matched estimator runs on the training stage\-OCR curve: same four training rewards \(ClipScore,HPSv2,OCR,PickScore\), same prompt set, sameα=2\\alpha\{=\}2andT=25T\{=\}25; the top two differ only in the sampling distribution, the bottom two only in\(N,K\)\(N,K\)\. Bothπold\\pi^\{\\mathrm\{old\}\}rows are visibly noisier at every step than theπ\+\\pi^\{\+\}row, consistent with the higher\-varianceπold\\pi^\{\\mathrm\{old\}\}\-side estimator, and the16×16\\timesbudget does not smooth them\.

### B\.4Sensitivity to theπ\+\\pi^\{\+\}surrogate generator

The curves above approximateπ\+\\pi^\{\+\}by a single external generator,GPT Image 1\.5\. To test how much the resulting weights depend on the specificπ\+\\pi^\{\+\}, we regenerate the training stage\-OCRπ\+\\pi^\{\+\}image set with an independent generator,Nano Banana Pro\(Gemini 3 Pro Image\), same prompts andN=8N\{=\}8images per prompt, and rerun the identical estimator\. Tab\.[5](https://arxiv.org/html/2609.13425#A2.T5)compares the two per\-step curves reward\-by\-reward\. The curves are highly correlated \(Pearsonr∈\[0\.84,0\.92\]r\\in\[0\.84,0\.92\], Spearmanρ∈\[0\.82,0\.92\]\\rho\\in\[0\.82,0\.92\]\) and peak within00–22steps of each other\. Fig\.[4](https://arxiv.org/html/2609.13425#A2.F4)shows the two curves side by side: the per\-step gains, mean\-normalized gains, and the Sinkhorn matrices they project to are visually near\-interchangeable\. The gain curve, and hence the Sinkhorn weights built from it, is therefore largely a property of the reward and the trajectory, not of which strong external generator stands in forπ\+\\pi^\{\+\}\.

Table 5:Surrogate sensitivity of the training stage\-OCR gain curve:GPT Image 1\.5vs\.Nano Banana Proas theπ\+\\pi^\{\+\}generator, all other estimator settings identical\.rrandρ\\rhoare Pearson and Spearman correlations between the two per\-step curves over theT=25T\{=\}25steps\. “Peaktt” gives the argmax timestep ofΔ​Di,t\\Delta D\_\{i,t\}in the paper’s convention \(t=1t\{=\}1clean,t=Tt\{=\}Tpure noise\); “total gain” is the ratio of∑tΔ​Di,t\\sum\_\{t\}\\Delta D\_\{i,t\}underNano Banana Proto that underGPT Image 1\.5\.Figure 4:Per\-step gainΔ​Di,t\\Delta D\_\{i,t\}, mean\-normalized gainΔ​D¯i,t\\overline\{\\Delta D\}\_\{i,t\}, and Sinkhorn matrixWi,t⋆W\_\{i,t\}^\{\\star\}at uniformλi=1/m\\lambda\_\{i\}\{=\}1/m, for the training stage\-OCR curve under the twoπ\+\\pi^\{\+\}surrogate generators\.Top:GPT Image 1\.5\.Bottom:Nano Banana Pro\. The two rows produce near\-identical reward\-specific shapes and peak orderings \(quantified in Tab\.[5](https://arxiv.org/html/2609.13425#A2.T5)\)\.
### B\.5Closing the divergence toπ\+\\pi^\{\+\}over a training stage

In this section, we measure how much of the divergence the curve is built from survives training against it\. We rerun the identical training stage\-OCR estimator withπold\\pi^\{\\mathrm\{old\}\}set to checkpoint\-180180of the𝝀=\(1,1,1,1\)\\bm\{\\lambda\}\{=\}\(1,1,1,1\)ReCAST stage\-OCR run, keeping theGPT Image 1\.5π\+\\pi^\{\+\}samples and every other estimator setting fixed, so the policy the gains are measured at is the only thing that changes\.

By Eq\. \([7](https://arxiv.org/html/2609.13425#S3.E7)\), the total gain∑tΔ​Di,t\\sum\_\{t\}\\Delta D\_\{i,t\}is the data\-level divergenceDα\(πi,0\+∥π0old\)D\_\{\\alpha\}\\\!\\big\(\\pi\_\{i,0\}^\{\+\}\\,\\\|\\,\\pi\_\{0\}^\{\\mathrm\{old\}\}\\big\)still separating the policy from the reward\-induced positive distribution, so it reads directly as how much of rewardii’s gap is left to close\. Tab\.[6](https://arxiv.org/html/2609.13425#A2.T6)shows that by checkpoint\-180180almost none of it is: the total gain falls to0\.14×0\.14\\timesits base\-policy value forClipScoreand to0\.010\.01–0\.11×0\.11\\timesfor the other three rewards, and to0\.03×0\.03\\timessummed over all four\. Fig\.[5](https://arxiv.org/html/2609.13425#A2.F5)shows the same collapse inDi,tD\_\{i,t\}itself: on a shared vertical axis, the trained policy’s curves sit near zero across the whole trajectory\. Because Rényi divergence is nondecreasing inα\\alpha, theα=2\\alpha\{=\}2quantity we measure upper\-boundsKL\(πi,0\+∥π0old\)\\mathrm\{KL\}\\\!\\big\(\\pi\_\{i,0\}^\{\+\}\\,\\\|\\,\\pi\_\{0\}^\{\\mathrm\{old\}\}\\big\), so the KL gap to the positive distribution is closed at least as far\. Training under the reweighted objective therefore closes almost all of the gap the curve is built from, and it does so for all four rewards simultaneously rather than closing one reward’s gap at another’s expense\.

Table 6:Total discriminability gain∑tΔ​Di,t\\sum\_\{t\}\\Delta D\_\{i,t\}of the training stage\-OCR curve withπold\\pi^\{\\mathrm\{old\}\}at the untrained base policy vs\. at the trained checkpoint\-180180, with theGPT Image 1\.5π\+\\pi^\{\+\}samples and all other estimator settings identical\. By Eq\. \([7](https://arxiv.org/html/2609.13425#S3.E7)\) each entry is the data\-level divergence remaining between that policy and the reward’s positive distribution\.Figure 5:Cumulative discriminabilityDi,tD\_\{i,t\}of the training stage\-OCR curve at the two policies, withπ\+\\pi^\{\+\}set toGPT Image 1\.5and all other estimator settings identical\.Left:πold\\pi^\{\\mathrm\{old\}\}at the untrained base\.Right:πold\\pi^\{\\mathrm\{old\}\}at checkpoint\-180180of the𝝀=\(1,1,1,1\)\\bm\{\\lambda\}\{=\}\(1,1,1,1\)ReCAST stage\-OCR run, drawn on the same vertical axis; the inset repeats it at its own scale\. Each curve telescopes the per\-step gains up fromDi,T=0D\_\{i,T\}=0\(Eq\. \([7](https://arxiv.org/html/2609.13425#S3.E7)\)\), so its clean endpointDi,0D\_\{i,0\}is the full data\-level divergence between that policy and rewardii’s positive distribution\. Training leavesDi,tD\_\{i,t\}smaller at every timestep, for every reward\.

## Appendix CSinkhorn projection: solver and alternatives

### C\.1The log\-domain iteration

Alg\.[3](https://arxiv.org/html/2609.13425#alg3)is the solver of Eq\. \([16](https://arxiv.org/html/2609.13425#S3.E16)\)\. It normalizes the gains per reward, exponentiates them into the kernel, and alternates the two marginal updates of Eq\. \([19](https://arxiv.org/html/2609.13425#S3.E19)\) in the log domain, which keeps the scaling vectors stable when a kernel entry is tiny\. Convergence is linear and in practice takes fewer than100100iterations atm≤4m\\leq 4,T≤25T\\leq 25; the cost is negligible next to a single training step, and the matrix is computed once per𝝀\\bm\{\\lambda\}before training\.

Algorithm 3Sinkhorn construction of the reward\-by\-timestep weight matrix𝑾\\bm\{W\}1:Per\-step gains

\{Δ​Di,t\}\\\{\\Delta D\_\{i,t\}\\\}\(Alg\.[1](https://arxiv.org/html/2609.13425#alg1), Sec\.[3\.3](https://arxiv.org/html/2609.13425#S3.SS3)\), inter\-reward budget

𝝀∈Δm−1\\bm\{\\lambda\}\\in\\Delta^\{m\-1\}, tolerance

εtol\\varepsilon\_\{\\mathrm\{tol\}\}, max iterations

LL
2:weight matrix

𝑾∈𝒰⁡\(𝝀\)\\bm\{W\}\\in\\mathcal\{U\}\(\\bm\{\\lambda\}\), used in the loss as

T⋅Wi,tT\\\!\\cdot\\\!W\_\{i,t\}
3:foreach reward

iido

4:

Δ​D¯i,t←Δ​Di,t/\(1T​∑sΔ​Di,s\)\\overline\{\\Delta D\}\_\{i,t\}\\leftarrow\\Delta D\_\{i,t\}\\,/\\,\\big\(\\tfrac\{1\}\{T\}\\sum\_\{s\}\\Delta D\_\{i,s\}\\big\)// Eq\. \([13](https://arxiv.org/html/2609.13425#S3.E13)\): per\-reward mean\-normalize

5:

log⁡Ki,t←Δ​D¯i,t\\log K\_\{i,t\}\\leftarrow\\overline\{\\Delta D\}\_\{i,t\}// Eq\. \([14](https://arxiv.org/html/2609.13425#S3.E14)\);log⁡Ki,t←0\\log K\_\{i,t\}\\leftarrow 0if rewardiihas no gain estimate

6:endfor

7:

log⁡𝒂←𝟎m\\log\{\\bm\{a\}\}\\leftarrow\{\\bm\{0\}\}\_\{m\},

log⁡𝒃←𝟎T\\log\{\\bm\{b\}\}\\leftarrow\{\\bm\{0\}\}\_\{T\}
8:for

ℓ=1,…,L\\ell=1,\\dots,Ldo

9:

log⁡ai←log⁡λi−logsumexpt⁡\(log⁡Ki,t\+log⁡bt\)\\log a\_\{i\}\\leftarrow\\log\\lambda\_\{i\}\-\\operatorname\{logsumexp\}\_\{t\}\\\!\\big\(\\log K\_\{i,t\}\+\\log b\_\{t\}\\big\)// row marginals→λi\\to\\lambda\_\{i\}

10:

log⁡bt←−log⁡T−logsumexpi⁡\(log⁡Ki,t\+log⁡ai\)\\log b\_\{t\}\\leftarrow\-\\log T\-\\operatorname\{logsumexp\}\_\{i\}\\\!\\big\(\\log K\_\{i,t\}\+\\log a\_\{i\}\\big\)// column marginals→1/T\\to 1/T

11:

Wi,t←exp⁡\(log⁡ai\+log⁡Ki,t\+log⁡bt\)W\_\{i,t\}\\leftarrow\\exp\\\!\\big\(\\log a\_\{i\}\+\\log K\_\{i,t\}\+\\log b\_\{t\}\\big\)
12:break if

maxi⁡\|∑tWi,t−λi\|<εtol\\max\_\{i\}\\big\|\\sum\_\{t\}W\_\{i,t\}\-\\lambda\_\{i\}\\big\|<\\varepsilon\_\{\\mathrm\{tol\}\}and

maxt⁡\|∑iWi,t−1T\|<εtol\\max\_\{t\}\\big\|\\sum\_\{i\}W\_\{i,t\}\-\\tfrac\{1\}\{T\}\\big\|<\\varepsilon\_\{\\mathrm\{tol\}\}
13:endfor

14:return

𝑾\\bm\{W\}

### C\.2Boundary cases

Our construction sits between the alternatives one might have used instead, and each of them is instructive\.

#### A flat kernel: the static baseline\.

If no gain curve has any shape intt, thenΔ​D¯i,t≡1\\overline\{\\Delta D\}\_\{i,t\}\\equiv 1for every reward, soKi,t≡eK\_\{i,t\}\\equiv eis rank one and the entropic optimum over𝒰⁡\(𝝀\)\\mathcal\{U\}\(\\bm\{\\lambda\}\)isWi,t⋆=λi/TW\_\{i,t\}^\{\\star\}=\\lambda\_\{i\}/T, which is the standard static convex combination\.

#### No entropic term: hard assignment\.

Dropping the entropy from Eq\. \([17](https://arxiv.org/html/2609.13425#S3.E17)\) leaves an unregularized transport LP, whose solutions are vertices of𝒰⁡\(𝝀\)\\mathcal\{U\}\(\\bm\{\\lambda\}\): each timestep’s budget goes essentially to the single reward with the largest normalized gain there\. Committing that hard to argmax gains estimated from finite samples is brittle, so we keep the entropic term, which is what makes the solution the smooth, strictly positive matrix of Eq\. \([18](https://arxiv.org/html/2609.13425#S3.E18)\)\.

#### An uncurved reward\.

Because Eq\. \([13](https://arxiv.org/html/2609.13425#S3.E13)\) normalizes each reward to mean11, the kernel needs no scale or sharpness parameter, and Sec\.[3\.5](https://arxiv.org/html/2609.13425#S3.SS5)’s shapes give smooth, non\-degenerate matrices\. If a reward has no estimated curve at all, we setKi,t=1K\_\{i,t\}=1for everytt: it still receives its full row budgetλi\\lambda\_\{i\}, spread uniformly up to the sharedbtb\_\{t\}\.

## Appendix DPer\-𝝀\\bm\{\\lambda\}results

Tabs\.[1](https://arxiv.org/html/2609.13425#S4.T1)and[2](https://arxiv.org/html/2609.13425#S4.T2)pool the five budgets; this appendix resolves both of them per𝝀\\bm\{\\lambda\}, static vs\. ReCAST at matched𝝀\\bm\{\\lambda\}, every entry the mean over the33seed replicates with its standard deviation as a subscript\. Each budget also carries a Pareto verdict: Ours≻\\succstatic if the ReCAST mean is at least as high as the static mean on*every*reward in that table and strictly higher on at least one, static≻\\succOurs if static \(weakly\) dominates ReCAST in the same sense,*mixed*otherwise\. Every cell scores the run’s final checkpoint\.

### D\.1Training rewards

Tabs\.[7](https://arxiv.org/html/2609.13425#A4.T7)and[8](https://arxiv.org/html/2609.13425#A4.T8)give the per\-𝝀\\bm\{\\lambda\}training rewards behind Tab\.[1](https://arxiv.org/html/2609.13425#S4.T1)\. In training stage\-OCR \(Tab\.[7](https://arxiv.org/html/2609.13425#A4.T7)\) ReCAST Pareto\-dominates static at33of the55budgets, namely\(1,1,1,1\)\(1,1,1,1\),\(1,1,1,2\)\(1,1,1,2\), and\(1,1,2,1\)\(1,1,2,1\), and is dominated at none; the aggregate is higher at44of55\. The two mixed budgets fail the test on the same coordinate:\(1,2,1,1\)\(1,2,1,1\)raisesClipScore,HPSv2, andPickScorewhile concedingOCRby0\.0390\.039, and\(2,1,1,1\)\(2,1,1,1\)raisesClipScoreandPickScore, tiesHPSv2, and concedesOCRby0\.0260\.026\. This is the per\-budget form of the pattern in Tab\.[1](https://arxiv.org/html/2609.13425#S4.T1), where the target reward carries by far the largest seed variance of the four\.

Training stage\-GenEval \(Tab\.[8](https://arxiv.org/html/2609.13425#A4.T8)\) is a tie budget by budget as well as on average\. ReCAST dominates at no𝝀\\bm\{\\lambda\}, static dominates at none, and all55budgets are mixed; at every one of them the aggregate shifts by less than the seed standard deviation of either method there\. The targetGenEvalreward moves in both directions across budgets, up at33of55and down at\(1,1,1,2\)\(1,1,1,2\)and\(1,2,1,1\)\(1,2,1,1\), so no budget in this setting resolves the two methods\.

### D\.2Held\-out judges

Tabs\.[9](https://arxiv.org/html/2609.13425#A4.T9)and[10](https://arxiv.org/html/2609.13425#A4.T10)resolve Tab\.[2](https://arxiv.org/html/2609.13425#S4.T2)the same way\. In training stage\-OCR \(Tab\.[9](https://arxiv.org/html/2609.13425#A4.T9)\) ReCAST Pareto\-dominates static across all four judges at33of the55budgets,\(1,1,2,1\)\(1,1,2,1\),\(1,2,1,1\)\(1,2,1,1\), and\(2,1,1,1\)\(2,1,1,1\), and is dominated at none\.ImageReward,Aesthetic, andHPSv3improve at every budget without exception; the two mixed budgets,\(1,1,1,1\)\(1,1,1,1\)and\(1,1,1,2\)\(1,1,1,2\), concede onlyUnifiedReward\-2, by0\.0250\.025and0\.0170\.017\. The generalization gain of Sec\.[4\.3](https://arxiv.org/html/2609.13425#S4.SS3)is therefore not carried by a subset of the budgets\.

Training stage\-GenEval \(Tab\.[10](https://arxiv.org/html/2609.13425#A4.T10)\) is the setting whose training rewards tie, and its judge scores favor ReCAST at a majority of budgets: ReCAST dominates at\(1,1,1,2\)\(1,1,1,2\),\(1,1,2,1\)\(1,1,2,1\), and\(1,2,1,1\)\(1,2,1,1\), static dominates at none, and\(1,1,1,1\)\(1,1,1,1\)and\(2,1,1,1\)\(2,1,1,1\)are mixed\. The seed standard deviations here are up to an order of magnitude wider than in training stage\-OCR, wide enough that the individual budgets are not resolved by three replicates even where the means separate; the pooled comparison in Tab\.[2](https://arxiv.org/html/2609.13425#S4.T2), which averages the same runs, still favors ReCAST on all four judges\.

Table 7:Training stage\-OCR, per\-𝝀\\bm\{\\lambda\}training rewards on the held\-out dataset at each run’s final checkpoint, static vs\. ReCAST\. Every entry is the mean over the33seed replicates with its standard deviation as a subscript, Aggregaterris the𝝀\\bm\{\\lambda\}\-weighted sum as in Tab\.[1](https://arxiv.org/html/2609.13425#S4.T1), and bold marks the better method per cell, neither when the two agree at the reported precision\. Pareto is the verdict over the four training rewards\.Table 8:Training stage\-GenEval, per\-𝝀\\bm\{\\lambda\}training rewards on the held\-out dataset at each run’s final checkpoint, static vs\. ReCAST\. Entries, subscripts, bolding, and the Pareto verdict are as in Tab\.[7](https://arxiv.org/html/2609.13425#A4.T7)\.Table 9:Training stage\-OCR, per\-𝝀\\bm\{\\lambda\}held\-out judge scores at each run’s final checkpoint, static vs\. ReCAST\. Entries, subscripts, bolding, and the Pareto verdict are as in Tab\.[7](https://arxiv.org/html/2609.13425#A4.T7), now over the four judges; theHPSv3column is at two decimals because its scale is an order of magnitude larger\.Table 10:Training stage\-GenEval, per\-𝝀\\bm\{\\lambda\}held\-out judge scores at each run’s final checkpoint, static vs\. ReCAST\. Entries, subscripts, bolding, and the Pareto verdict are as in Tab\.[9](https://arxiv.org/html/2609.13425#A4.T9)\.

## Appendix EExperiment details

### E\.1Reward models

Tab\.[11](https://arxiv.org/html/2609.13425#A5.T11)documents every reward model used anywhere in the paper: its architecture, checkpoint or backbone, and native output range\. Two roles recur:*train*, entering the training objective through the weight matrix𝑾\\bm\{W\};*held\-out*, scoring checkpoints it never influenced\. Sec\.[3\.5](https://arxiv.org/html/2609.13425#S3.SS5)also estimates gain curves for two of the held\-out judges \(Aesthetic,ImageReward\), to show what the estimator produces on rewards of a different type; they enter no training kernel and no training loss\.

The training\-stage kernels are built fromOCR’s andGenEval’s own gain curves, estimated exactly as in Sec\.[3\.5](https://arxiv.org/html/2609.13425#S3.SS5)but on their own prompt sets and their ownT=25T\{=\}25schedule\. Fig\.[1](https://arxiv.org/html/2609.13425#S3.F1)also overlays both, linearly resampledT=25T\{=\}25onto theT=10T\{=\}10grid of the other five, purely for visual comparison\.

#### Positivity of the training rewards\.

The analysis of Sec\.[3\.2](https://arxiv.org/html/2609.13425#S3.SS2)assumesri≥0r\_\{i\}\\geq 0, so we state how each training reward is computed\. Writecos⁡\(𝒖,𝒗\)\\cos\(\\bm\{u\},\\bm\{v\}\)for the cosine similarity of an image embedding𝒖\\bm\{u\}and a text embedding𝒗\\bm\{v\}\.ClipScoreiscos\\cosbetween the CLIP ViT\-L/14 image and text embeddings\.HPSv2iscos\\cosbetween theℓ2\\ell\_\{2\}\-normalized image and text embeddings of the preference\-fine\-tuned ViT\-H/14, with no logit scale applied\.PickScoreis the preference head’scos\\cosrescaled by the model’s own learned logit scale and a fixed constant,\(es/26\)​cos\(e^\{s\}/26\)\\coswithes≈98\.9e^\{s\}\\approx 98\.9, so≈3\.80​cos\\approx 3\.80\\cos\.OCRis1−min⁡\{Lev⁡\(s^,s\),\|s\|\}/\|s\|1\-\\min\\\{\\mathrm\{Lev\}\(\\hat\{s\},s\),\\,\|s\|\\\}/\|s\|, wheressis the target string quoted in the prompt,s^\\hat\{s\}is the recognized text after lowercasing and removing whitespace,Lev\\mathrm\{Lev\}is the Levenshtein distance, and the distance is taken as00wheneverssoccurs as a substring ofs^\\hat\{s\}\.GenEvalis the fraction of the prompt’s object, count, color, and position clauses that the detector confirms\.

OCRandGenEvalare therefore non\-negative by construction, one a distance ratio capped at11and the other a fraction of satisfied clauses\. The remaining three are cosine similarities up to a positive scale, so they are sign\-indefinite in principle, with ranges\[−1,1\]\[\-1,1\],\[−1,1\]\[\-1,1\], and≈\[−3\.80,3\.80\]\\approx\[\-3\.80,3\.80\]\. We clamp the reward at00\. This clamping has no effect in estimation, because on our prompt sets they are positive throughout and with a wide margin: over every trained checkpoint we evaluate, the smallestClipScoreis0\.260\.26, the smallestHPSv2is0\.230\.23, and the smallestPickScoreis0\.770\.77, against a floor of00\. This reflects a property of CLIP\-style encoders rather than a coincidence, since image and text embeddings occupy separate cones and their cosines concentrate in a narrow positive band\. We apply a sigmoid toImageReward, whose raw output is a signed comparison scalar, when estimating its gain curve in Sec\.[3\.5](https://arxiv.org/html/2609.13425#S3.SS5)\.

Table 11:Reward models used in the paper\. “Range” is the reward’s output scale; several are unbounded in principle but concentrate in the interval shown in practice\.

### E\.2Training hyperparameters and compute

#### Architecture and optimization\.

All runs fine\-tune LoRA adapters \(rank3232, scale6464, Gaussian initialization\) inserted into the eight attention\-projection modules of the SD3\.5\-Medium joint\-attention transformer block; the transformer backbone itself is frozen\. The optimizer is AdamW with learning rate3×10−43\\times 10^\{\-4\},\(β1,β2\)=\(0\.9,0\.999\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.9,0\.999\),ϵ=10−8\\epsilon=10^\{\-8\}, weight decay10−410^\{\-4\}, and a decayed learning\-rate schedule \(decay\_type=1\) shared by every stage\. The DiffusionNFT coefficient isβ=0\.1\\beta=0\.1throughout, sampling usesT=25T=25steps, and the rollout batch is2424images/prompt on each of88GPUs with66gradient\-accumulation steps, for an effective batch of8×24×6=11528\\times 24\\times 6=1152\. Evaluation images are drawn at seed42\+42\{\+\}index, identical across the two methods of a pair\. Sinkhorn kernels are built once per𝝀\\bm\{\\lambda\}before training starts, atα=2\\alpha=2for every training reward\.

#### Compute\.

All training ran on single\-node8×8\\timesNVIDIA H200 allocations\. A warmup\-stage run costs≈68\\approx 68GPU\-hours, a training stage\-OCR run≈37\\approx 37, and a training stage\-GenEval run≈59\\approx 59\. The static and ReCAST methods of a pair cost the same: the projection is negligible next to sampling and reward scoring\.

#### Gain curve construction cost\.

Each curve usesN=8N\{=\}8π\+\\pi^\{\+\}samples per prompt, andK=16K\{=\}16πold\\pi^\{\\mathrm\{old\}\}rollouts from every intermediate𝒙t\{\\bm\{x\}\}\_\{t\}; every reward of a stage is scored on the same rollouts, so the cost is per curve rather than per reward, and App\.[F](https://arxiv.org/html/2609.13425#A6)reports what reproducing one curve from scratch costs\. Ignoring theGPT Image 1\.5API time, theP⋅N⋅T⋅KP\{\\cdot\}N\{\\cdot\}T\{\\cdot\}Krollouts of oneT=10T\{=\}10curve take≈15\\approx 15GPU\-hours, under half the≈37\\approx 37of a single training stage\-OCR run, and a training\-stage curve atT=25T\{=\}25takes about6×6\\timesthose1515GPU\-hours\. TheP⋅NP\{\\cdot\}Nqueries toGPT Image 1\.5cost around $35 in API fees\. Crucially, this is a one\-time cost per stage: each curve is estimated once at the untrained policy and then reused, unmodified, so its amortized cost per training run is well below the raw totals\.

### E\.3MMRBv2 judge: protocol, position debiasing, and prompt

#### Protocol\.

The pairwise judgments of Sec\.[4\.4](https://arxiv.org/html/2609.13425#S4.SS4)use the image\-generation judge from Meta’s MMRBv2\[[13](https://arxiv.org/html/2609.13425#bib.bib40)\]evaluation suite, served bygemini\-3\.5\-flashat temperature00\. For each of the five matched𝝀\\bm\{\\lambda\}budgets, we pair the static method and the ReCAST method image\-by\-image on the fixed1,0001\{,\}000\-prompt subset of the held\-out OCR dataset of Tab\.[2](https://arxiv.org/html/2609.13425#S4.T2)\. Each API call sends the rubric prompt below, the original text prompt, and the two images labeled\[RESPONSE A:\]and\[RESPONSE B:\]\. The judge returns a structured JSON verdict: step\-by\-step reasoning over seven criteria \(faithfulness to prompt, text rendering, input faithfulness, image consistency, text–image alignment, text quality, overall quality\), a scalar scores∈\{1,…,6\}s\\in\\\{1,\\dots,6\\\}wheres≥4s\\geq 4favors the image shown as A, a discretebetter\_response∈\{\\in\\\{A, B\}\\\}, and a self\-reported confidence\. We take the winner frombetter\_response; calls that still fail after88retries drop the pair\.

#### Position debiasing\.

LLM judges exhibit position bias: the response presented first is systematically favored\. Following MMRBv2’s forward\+reverse protocol, every pair is therefore judged*twice*: once with the static image shown as A and ReCAST as B \(forward\), and once with the images swapped \(reverse\)\. Each ordering credits one win to the method the judge picks, after mapping the shown\-side label back to the method’s identity\. The win rate overnnprompts is the fraction of the2​n2norderings won,

win⁡\(Ours\)=12​n​∑j=1n\#⁡\{orderings of pair​j​won by Ours\}\.\\mathrm\{win\}\(\\text\{Ours\{\}\}\)\\;=\\;\\frac\{1\}\{2n\}\\sum\_\{j=1\}^\{n\}\\\#\\\{\\text\{orderings of pair \}j\\text\{ won by Ours\{\}\}\\\}\.

#### Faithfulness and aesthetic sub\-scores\.

The judge is not asked for numeric per\-criterion scores; the*Faithfulness*and*Aesthetics*rows of Tab\.[3](https://arxiv.org/html/2609.13425#S4.T3)are extracted from its per\-criterion reasoning fieldsfaithfulness\_to\_promptandoverall\_quality, respectively\. For each ordering, the criterion’s reasoning text is mapped to a winner in\{\\\{A, B, undecided\}\\\}by matching declarative phrasings like “Response A is better…” and “the edge goes to B…”, with a text\-only LLM tie\-break when the patterns are ambiguous\. The sub\-score is the fraction of*decided*orderings won by ReCAST; orderings whose criterion text is undecided or marked not\-applicable are dropped from that criterion’s denominator\.

#### Judge prompt\.

The rubric prompt, verbatim from MMRBv2’s image\-generation judge:

```
You are an expert in multimodal quality analysis and generative AI evaluation. Your
role is to act as an objective judge for comparing two AI-generated responses to the
same prompt. You will evaluate which response is better based on a comprehensive
rubric.

**Important Guidelines:**
- Be completely impartial and avoid any position biases
- Ensure that the order in which the responses were presented does not influence
  your decision
- Do not allow the length of the responses to influence your evaluation
- Do not favor certain model names or types
- Be as objective as possible in your assessment
- Consider factors such as helpfulness, relevance, accuracy, depth, creativity, and
  level of detail

**Understanding the Content Structure:**
- **[ORIGINAL PROMPT TO MODEL:]**: This is the instruction given to both AI models
- **[INPUT IMAGE FROM PROMPT:]**: This is the source image provided to both models
  (if any)
- **[RESPONSE A:]**: The first model’s generated response (text and/or images)
- **[RESPONSE B:]**: The second model’s generated response (text and/or images)

Your evaluation must be based on a fine-grained rubric that covers the following
criteria. For each criterion, you must provide detailed step-by-step reasoning
comparing both responses. You will use a 1-6 scoring scale.

**Evaluation Criteria:**
1. **faithfulness_to_prompt:** Which response better adheres to the composition,
   objects, attributes, and spatial relationships described in the text prompt?

2. **text_rendering:** If either response contains rendered text, which one has
   better text quality (spelling, legibility, integration)? If no text is rendered,
   state "Not Applicable."

3. **input_faithfulness:** If an input image is provided, which response better
   respects and incorporates the key elements and style of that source image? If no
   input image is provided, state "Not Applicable."

4. **image_consistency:** If multiple images are generated, which response has
   better visual consistency between images (character appearance, scene details)?
   If no multiple images are provided, state "Not Applicable."

5. **text_image_alignment:** Which response has better alignment between text
   descriptions and visual content?

6. **text_quality:** If text was generated, which response has better linguistic
   quality (correctness, coherence, grammar, tone)?

7. **overall_quality:** Which response has better general technical and aesthetic
   quality, realism, coherence, and fewer visual artifacts or distortions?

**Scoring Rubric:**
- Score 6 (A is significantly better): Response A is significantly superior across
  most criteria
- Score 5 (A is marginally better): Response A is noticeably better across several
  criteria
- Score 4 (Unsure or A is negligibly better): Response A is slightly better or
  roughly equivalent
- Score 3 (Unsure or B is negligibly better): Response B is slightly better or
  roughly equivalent
- Score 2 (B is marginally better): Response B is noticeably better across several
  criteria
- Score 1 (B is significantly better): Response B is significantly superior across
  most criteria

**Confidence Assessment:**
After your evaluation, assess your confidence in this judgment on a scale of 0.0
to 1.0:

**CRITICAL**: Be EXTREMELY conservative with confidence scores. Most comparisons
should be in the 0.2-0.5 range.

- **Very High Confidence (0.8-1.0)**: ONLY for absolutely obvious cases where one
  response is dramatically better across ALL criteria with zero ambiguity. Use this
  extremely rarely (less than 10% of cases).
- **High Confidence (0.6-0.7)**: Clear differences but some uncertainty remains.
  Use sparingly (less than 20% of cases).
- **Medium Confidence (0.4-0.5)**: Noticeable differences but significant
  uncertainty. This should be your DEFAULT range.
- **Low Confidence (0.2-0.3)**: Very close comparison, difficult to distinguish.
  Responses are roughly equivalent or have conflicting strengths.
- **Very Low Confidence (0.0-0.1)**: Essentially indistinguishable responses or
  major conflicting strengths.

**IMPORTANT GUIDELINES**:
- DEFAULT to 0.3-0.5 range for most comparisons
- Only use 0.6+ when you are absolutely certain
- Consider: Could reasonable people disagree on this comparison?
- Consider: Are there any strengths in the "worse" response?
- Consider: How obvious would this be to a human evaluator?
- Remember: Quality assessment is inherently subjective

After your reasoning, you will provide a final numerical score, indicate which
response is better, and assess your confidence. You must always output your
response in the following structured JSON format:

{
    "reasoning": {
        "faithfulness_to_prompt": "YOUR REASONING HERE",
        "text_rendering": "YOUR REASONING HERE",
        "input_faithfulness": "YOUR REASONING HERE",
        "image_consistency": "YOUR REASONING HERE",
        "text_image_alignment": "YOUR REASONING HERE",
        "text_quality": "YOUR REASONING HERE",
        "overall_quality": "YOUR REASONING HERE",
        "comparison_summary": "YOUR OVERALL COMPARISON SUMMARY HERE"
    },
    "score": <int 1-6>,
    "better_response": "A" or "B",
    "confidence": <float 0.0-1.0>,
    "confidence_rationale": "YOUR CONFIDENCE ASSESSMENT REASONING HERE"
}
```

## Appendix FLimitations

#### Approximate positive\-policy sampling\.

In practice, we useGPT Image 1\.5, an external expert generator, as a surrogate forπ\+\\pi^\{\+\}for every reward\. Its distribution need not equal any reward\-specificπi\+\\pi\_\{i\}^\{\+\}; without an importance correction, the resulting gain curves are therefore biased estimates of the theoreticalΔ​Di,t\\Delta D\_\{i,t\}\. The empirical curves and ablations show that this proxy is useful, and swapping in an independent surrogate generator leaves the curve essentially unchanged \(App\.[B\.4](https://arxiv.org/html/2609.13425#A2.SS4)\), but neither removes this bias\. Future work could construct reward\-specific positive policies, estimate the required density correction, or update the curves on\-policy as training progresses; App\.[B\.5](https://arxiv.org/html/2609.13425#A2.SS5)re\-estimates the curve at a trained checkpoint\.

#### Reproducibility and API cost\.

Depending onGPT Image 1\.5also means depending on a proprietary, versioned, and priced API\. Even though it is a one\-time, per\-stage expense amortized over many downstream training runs, an independent group reproducing a single curve from scratch pays around $35 in API costs\. Releasing the estimated gain\-curve weight files themselves alongside the code would let downstream use of ReCAST skip re\-queryingGPT Image 1\.5entirely\.

## Appendix GQualitative examples

The two galleries below \(Figs\.[6](https://arxiv.org/html/2609.13425#A7.F6)and[7](https://arxiv.org/html/2609.13425#A7.F7)\) show paired generations from the static baseline and ReCAST for theOCRandGenEvalsettings\. We state explicitly how these examples were selected so that they are not mistaken for random or typical samples\. Every one of the2020pairs shown \(22per𝝀\\bm\{\\lambda\}budget, with55budgets per setting\) is a decisive, judge\-verified win for ReCAST, drawn from the pool of “high\-confidence” pairs identified by MMRBv2 pairwise\-judge runs following the protocol in Sec\.[4\.4](https://arxiv.org/html/2609.13425#S4.SS4)and App\.[E\.3](https://arxiv.org/html/2609.13425#A5.SS3), applied separately to each setting’s own held\-out prompts\. For theGenEvalsetting, this selection uses an analogous MMRBv2 run whose aggregate results are not reported in the main text, since Sec\.[4\.4](https://arxiv.org/html/2609.13425#S4.SS4)reports only theOCRsetting\. We call a win*decisive*when ReCAST wins under both presentation orderings with extreme scores and neither the faithfulness nor the aesthetic sub\-criterion contradicts the overall verdict\. Within each budget’s qualifying pool, the displayed pair was then selected by a human reviewer rather than sampled uniformly at random\. These galleries should therefore be interpreted as illustrating what ReCAST’s most decisive wins look like, rather than its average\-case behavior\. Representative aggregate results are reported in Tab\.[3](https://arxiv.org/html/2609.13425#S4.T3)for theOCRsetting and in Tabs\.[1](https://arxiv.org/html/2609.13425#S4.T1)and[2](https://arxiv.org/html/2609.13425#S4.T2), which pool all three seeds, for theGenEvalsetting\.

staticReCASTstaticReCAST![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/ocr/lam1_1_1_1_pair1_baseline.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/ocr/lam1_1_1_1_pair1_sinkhorn.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/ocr/lam1_1_1_1_pair2_baseline.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/ocr/lam1_1_1_1_pair2_sinkhorn.png)*A vibrant skateboard deck featuring the bold graphic ”SKATE OR DIE 4EVER” in dynamic, graffiti\-style lettering, set against a gradient background that shifts from deep blue at the nose to bright orange at the tail, with subtle scratch marks and wear to give it a well\-used, authentic look\.*𝝀=\(1,1,1,1\)\\bm\{\\lambda\}=\(1,1,1,1\)*A vibrant car dealership banner reads ”Electric Vehicles Here” in bold letters, showcasing a variety of sleek electric cars parked neatly in front of a modern showroom with large glass windows reflecting the sunny sky\.*𝝀=\(1,1,1,1\)\\bm\{\\lambda\}=\(1,1,1,1\)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/ocr/lam1_1_1_2_pair1_baseline.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/ocr/lam1_1_1_2_pair1_sinkhorn.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/ocr/lam1_1_1_2_pair2_baseline.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/ocr/lam1_1_1_2_pair2_sinkhorn.png)*A sleek, modern vampire fitness tracker displays ”0 Steps 10000 Bites” on its screen, resting on a dark, gothic desk surrounded by antique books and candles\. The scene is bathed in a dim, eerie light, highlighting the tracker’s eerie glow\.*𝝀=\(1,1,1,2\)\\bm\{\\lambda\}=\(1,1,1,2\)*A medieval knight stands proudly, holding a shield emblazoned with the motto ”Honor Above All” in intricate Old English font, set against a backdrop of a misty, ancient castle\.*𝝀=\(1,1,1,2\)\\bm\{\\lambda\}=\(1,1,1,2\)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/ocr/lam1_1_2_1_pair1_baseline.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/ocr/lam1_1_2_1_pair1_sinkhorn.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/ocr/lam1_1_2_1_pair2_baseline.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/ocr/lam1_1_2_1_pair2_sinkhorn.png)*In a cozy cat cafe, a menu board displays ”Purr Therapy 5 minute” among other offerings\. A fluffy gray cat sits nearby, looking relaxed and ready to offer its soothing presence to patrons\. The scene is warm and inviting, with soft lighting and comfortable seating\.*𝝀=\(1,1,2,1\)\\bm\{\\lambda\}=\(1,1,2,1\)*A vast desert landscape under a scorching sun, where a mirage forms the shimmering letters ”Water This Way” on the distant horizon, creating an illusion of hope in an otherwise barren and arid environment\.*𝝀=\(1,1,2,1\)\\bm\{\\lambda\}=\(1,1,2,1\)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/ocr/lam1_2_1_1_pair1_baseline.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/ocr/lam1_2_1_1_pair1_sinkhorn.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/ocr/lam1_2_1_1_pair2_baseline.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/ocr/lam1_2_1_1_pair2_sinkhorn.png)*A vibrant music festival wristband with ”Festival Access 2024” prominently displayed, featuring a colorful design with musical notes and festival logos, set against a backdrop of a bustling crowd and stage lights\.*𝝀=\(1,2,1,1\)\\bm\{\\lambda\}=\(1,2,1,1\)*A purple flower with a delicate crown on its head, featuring a speech bubble that says ”I am a purple flower”, set against a serene garden backdrop\.*𝝀=\(1,2,1,1\)\\bm\{\\lambda\}=\(1,2,1,1\)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/ocr/lam2_1_1_1_pair1_baseline.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/ocr/lam2_1_1_1_pair1_sinkhorn.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/ocr/lam2_1_1_1_pair2_baseline.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/ocr/lam2_1_1_1_pair2_sinkhorn.png)*A close\-up of a robot’s metallic chest, with a digital display prominently showing ”System Update In Progress”, surrounded by blinking lights and subtle wiring, set against a dimly lit, futuristic background\.*𝝀=\(2,1,1,1\)\\bm\{\\lambda\}=\(2,1,1,1\)*A realistic construction site with a warning sign that reads ”Bridge to Nowhere Ahead”, surrounded by a desolate landscape and half\-built structures, emphasizing the abandoned and eerie atmosphere\.*𝝀=\(2,1,1,1\)\\bm\{\\lambda\}=\(2,1,1,1\)Figure 6:Qualitative examples from theOCRsetting \(training stage\-OCR family, held\-out10001000\-prompt OCR dataset of Tab\.[2](https://arxiv.org/html/2609.13425#S4.T2)\): two prompts per𝝀\\bm\{\\lambda\}budget, static baseline \(left of each pair\) vs\. ReCAST \(right of each pair\)\. All1010pairs are a curated selection of decisive, judge\-verified ReCAST wins, not a random or representative sample; see App\.[G](https://arxiv.org/html/2609.13425#A7)for the selection methodology and Tab\.[3](https://arxiv.org/html/2609.13425#S4.T3)for this setting’s aggregate win rates\.staticReCASTstaticReCAST![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/geneval/lam1_1_1_1_pair1_baseline.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/geneval/lam1_1_1_1_pair1_sinkhorn.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/geneval/lam1_1_1_1_pair2_baseline.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/geneval/lam1_1_1_1_pair2_sinkhorn.png)*five mushrooms and a white truck*𝝀=\(1,1,1,1\)\\bm\{\\lambda\}=\(1,1,1,1\)*three metal mushrooms*𝝀=\(1,1,1,1\)\\bm\{\\lambda\}=\(1,1,1,1\)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/geneval/lam1_1_1_2_pair1_baseline.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/geneval/lam1_1_1_2_pair1_sinkhorn.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/geneval/lam1_1_1_2_pair2_baseline.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/geneval/lam1_1_1_2_pair2_sinkhorn.png)*four spotted birds*𝝀=\(1,1,1,2\)\\bm\{\\lambda\}=\(1,1,1,2\)*a dog and six bagels*𝝀=\(1,1,1,2\)\\bm\{\\lambda\}=\(1,1,1,2\)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/geneval/lam1_1_2_1_pair1_baseline.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/geneval/lam1_1_2_1_pair1_sinkhorn.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/geneval/lam1_1_2_1_pair2_baseline.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/geneval/lam1_1_2_1_pair2_sinkhorn.png)*five checkered dogs and a green giraffe*𝝀=\(1,1,2,1\)\\bm\{\\lambda\}=\(1,1,2,1\)*four red bears*𝝀=\(1,1,2,1\)\\bm\{\\lambda\}=\(1,1,2,1\)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/geneval/lam1_2_1_1_pair1_baseline.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/geneval/lam1_2_1_1_pair1_sinkhorn.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/geneval/lam1_2_1_1_pair2_baseline.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/geneval/lam1_2_1_1_pair2_sinkhorn.png)*three motorcycles and a brown giraffe*𝝀=\(1,2,1,1\)\\bm\{\\lambda\}=\(1,2,1,1\)*three metal zebras*𝝀=\(1,2,1,1\)\\bm\{\\lambda\}=\(1,2,1,1\)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/geneval/lam2_1_1_1_pair1_baseline.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/geneval/lam2_1_1_1_pair1_sinkhorn.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/geneval/lam2_1_1_1_pair2_baseline.png)![Refer to caption](https://arxiv.org/html/2609.13425v1/figures/appendix_gallery/geneval/lam2_1_1_1_pair2_sinkhorn.png)*four brown monkeys*𝝀=\(2,1,1,1\)\\bm\{\\lambda\}=\(2,1,1,1\)*a elephant and a purple kangaroo*𝝀=\(2,1,1,1\)\\bm\{\\lambda\}=\(2,1,1,1\)Figure 7:Qualitative examples from theGenEvalsetting \(training stage\-GenEval family, held\-outGenEvalprompts\): two prompts per𝝀\\bm\{\\lambda\}budget, static baseline \(left of each pair\) vs\. ReCAST \(right of each pair\)\. All1010pairs are a curated selection of decisive, judge\-verified ReCAST wins, not a random or representative sample; see App\.[G](https://arxiv.org/html/2609.13425#A7)for the selection methodology and Tabs\.[1](https://arxiv.org/html/2609.13425#S4.T1)and[2](https://arxiv.org/html/2609.13425#S4.T2)for this setting’s aggregate results; Sec\.[4\.4](https://arxiv.org/html/2609.13425#S4.SS4)reports the pairwise judge for theOCRsetting only\.