NormGuard: Reward-Preserving Norm Constraints in Flow-Matching Reinforcement Learning

# NormGuard: Reward-Preserving Norm Constraints in Flow-Matching Reinforcement Learning
Source: [https://arxiv.org/html/2606.27771](https://arxiv.org/html/2606.27771)
1\]The Hong Kong University of Science and Technology 2\]Kuaishou Technology 3\]University of Chinese Academy of Sciences

Lianyu Pang\*Cheng DaHuan YangChangqian YuKun GaiWenhan Luo†\[\[\[

###### Abstract

Reinforcement learning \(RL\) post\-training improves the reward alignment of flow\-based generators, but often degrades perceptual quality in ways that are not captured by the reward proxy\. We identify a simple structural signature of this drift: across three post\-training methods \(NFT, AWM, DPO\), RL fine\-tuning inflates the per\-step velocity norm‖vθ‖\\\|v\_\{\\theta\}\\\|by5%5\\%to15%15\\%relative to the reference\. A form of norm inflation has been studied in classifier\-free guidance \(CFG\), where rescaling the velocity back to a reference norm at inference time can mitigate the resulting artifacts\. However, this inference\-time correction does not transfer cleanly to RL: rescalingvθv\_\{\\theta\}to match‖vref‖\\\|v\_\{\\text\{ref\}\}\\\|at inference time neither improves reward nor fixes the quality degradation, because the inflation is co\-adapted into the model weights\. Furthermore, an adjoint sensitivity analysis shows that velocity magnitude rescaling carries no coherent first\-order reward signal at the batch level, indicating that suppressing norm inflation is unlikely to remove a consistently reward\-carrying component\. Since inference\-time renormalization fails while norm suppression carries no reward cost, training\-time intervention is the appropriate strategy\. Together, these findings motivateNormGuard, a hinge penalty that activates only when‖vθ‖\\\|v\_\{\\theta\}\\\|exceeds‖vref‖\\\|v\_\{\\text\{ref\}\}\\\|and composes additively with any velocity\-local base loss\. Across two base models, three post\-training methods, and two reward proxies,NormGuardconsistently improves MLLM\-judged image quality and forensic realism while preserving reward, with gains that amplify under few\-step inference and are not explained by early stopping\.

11footnotetext:Equal contribution\.†Corresponding author\.![Refer to caption](https://arxiv.org/html/2606.27771v1/x1.png)Figure 1:We proposeNormGuard, a simple norm\-budget regularizer that suppresses reward\-irrelevant norm inflation during RL training of flow\-matching models\.NormGuardreduces the over\-sharpening, color oversaturation, and unnatural lighting seen in the baseline, producing more photo\-realistic results\. Prompts see Appendix[10\.1](https://arxiv.org/html/2606.27771#S10.SS1)\.## 1Introduction

Reinforcement learning \(RL\) post\-training has become a standard tool for aligning flow\-based generative models\[ddpm,ddim,ldm,flow\-matching\]with human preferences\[PickScore,HPSv2\]\. Reward gains, however, are consistently accompanied by*reward over\-optimization*\[ScalingRewardOveroptimization,realgen,GARDO\]: perceptual quality degrades in ways that are not captured by the reward proxy, including over\-sharpening, color shift, unnatural lighting, and loss of fine texture\. The standard mitigations, such as early stopping and KL regularization, treat the post\-training drift as a single aggregate quantity\. Letvθv\_\{\\theta\}denote the fine\-tuned velocity andvrefv\_\{\\text\{ref\}\}the pre\-trained reference\. KL regularization typically takes the form of an MSE penalty‖vθ−vref‖2\\\|v\_\{\\theta\}\-v\_\{\\text\{ref\}\}\\\|^\{2\}\. These methods constrain how much the velocity deviates in total but do not distinguish which component of that deviation is associated with the artifact, leaving no basis for a targeted fix\.

![Refer to caption](https://arxiv.org/html/2606.27771v1/assets/idea.png)Figure 2:Motivation: RL post\-training inflates the per\-step velocity norm and produces visual artifacts \(left\)\. Unlike CFG, inference\-time renormalization fails for RL because the inflation is co\-adapted into the model weights \(middle\)\. An adjoint sensitivity analysis confirms that suppressing norm inflation carries no coherent first\-order reward signal, motivating our training\-time hinge penalty on excess velocity norm \(right\)\.Figure[2](https://arxiv.org/html/2606.27771#S1.F2)outlines the full motivation\. Without imposing any structural assumption on the drift, a simple starting point is to inspect the per\-step velocity norm‖vθ​\(xt,t\)‖\\\|v\_\{\\theta\}\(x\_\{t\},t\)\\\|and its ratio to the reference norm‖vref​\(xt,t\)‖\\\|v\_\{\\text\{ref\}\}\(x\_\{t\},t\)\\\|\. Take SD3\.5\-Medium with PickScore as an example: across NFT, AWM, and DPO, RL post\-training consistently inflates this norm by5%5\\%to15%15\\%relative to the reference, uniformly along the denoising trajectory \(Figure[3](https://arxiv.org/html/2606.27771#S3.F3)\)\.

A similar form of norm inflation has been well documented in inference\-time classifier\-free guidance \(CFG\)\[cfg\]\. For guidance scaleω\>1\\omega\>1, the CFG\-modified velocity has a substantially larger magnitude than the conditional velocity \(‖vCFG‖\>‖vcond‖\\\|v\_\{\\text\{CFG\}\}\\\|\>\\\|v\_\{\\text\{cond\}\}\\\|\), and this norm growth has been shown to drive sampling trajectories to overshoot beyond the learned data distribution\[cfg\-renorm\]\.The resulting artifacts, such as over\-sharpening and unnatural white balance, is also observed in RL fine\-tuned models\. As shown in Figure[2](https://arxiv.org/html/2606.27771#S1.F2)\(middle\),\[cfg\-renorm\]address the CFG case by rescaling the deployed velocity back to a reference norm at inference time while preserving its direction, and this simple fix successfully resolves the CFG artifacts\.

However, this CFG\-renormalization technique does not apply to the norm inflation induced by RL post\-training\. We scale the RL\-fine\-tuned velocity back to match‖vref‖\\\|v\_\{\\text\{ref\}\}\\\|at every step\. Reward is preserved after renormalization, yet the resulting images exhibit over\-sharpening and unnatural lighting \(Table[1](https://arxiv.org/html/2606.27771#S3.T1)and Figure[5](https://arxiv.org/html/2606.27771#S3.F5)\)\. The contrast with CFG is diagnostic: CFG inflation is an explicit inference\-time combination of two velocity heads, so rescaling removes it cleanly; RL inflation, by contrast, is trained into the model weights, and rescaling it back at inference distorts the co\-adapted dynamics\.

Then, we ask whether suppressing the inflation during training would disturb reward gains\. The first\-order reward change under a multiplicative velocity scalingvθ↦\(1\+ε\)​vθv\_\{\\theta\}\\mapsto\(1\+\\varepsilon\)v\_\{\\theta\}is governed by a per\-timestep norm\-scaling sensitivityS​\(t\)=vθ​\(xt,t\)⊤​a​\(t\)S\(t\)=v\_\{\\theta\}\(x\_\{t\},t\)^\{\\top\}a\(t\), wherea​\(t\)a\(t\)is the reward adjoint state\. On6,4006\{,\}400samples,S​\(t\)S\(t\)exhibits substantial per\-sample magnitude and sign heterogeneity across prompts, while its batch mean remains close to zero; the noise\-to\-signal ratioσ/\|μ\|\\sigma/\|\\mu\|ranges from3×3\\timesto100×100\\times\. Thus, at first order and at the batch level, velocity magnitude rescaling does not reveal a coherent reward signal, suggesting that suppressing norm inflation is unlikely to remove reward gains at training\.

The two findings point in the same direction\.1\)Inference\-time renormalization fails, so the intervention must happen at training time\.2\)Velocity magnitude rescaling does not reveal a coherent first\-order reward signal at the batch level, suggesting that training\-time norm suppression is unlikely to systematically reduce reward gains \. Based on these observations, we proposeNormGuard, a training\-time penalty on the velocity norm that activates only when‖vθ‖\\\|v\_\{\\theta\}\\\|exceeds‖vref‖\\\|v\_\{\\text\{ref\}\}\\\|, as shown in Figure[2](https://arxiv.org/html/2606.27771#S1.F2)\(right\)\. The objective sits alongside the existing velocity\-local post\-training loss and only adds a single scalar hyperparameter\.

We validate this regularizer across two base flow models, three post\-training methods \(NFT, AWM, DPO\), and two reward models\. Using both MLLM\-based image quality assessment and forensic realism detection, we find that the regularizer consistently improves visual quality and, in most settings, improves realism while preserving the majority of the reward gain\. Notably, the gains are particularly pronounced at few\-step inference, and the improvement cannot be explained by early stopping and is complementary to KL regularization\. These results suggest that the aggregate drift penalized by KL can be fruitfully decomposed: norm inflation drives perceptual artifacts while carrying little reward signal, whereas directional realignment accounts for most of the reward gain\. Distinguishing these two components offers a finer diagnostic for reward over\-optimization\.

Our contributions are threefold\.1\)We identify a consistent velocity\-norm inflation effect in RL post\-training for flow\-based generative models, which may cause visual artifacts\.2\)We show that a CFG\-style inference\-time renormalization does not transfer cleanly to RL\-trained models, while batch\-level first\-order sensitivity does not reveal a stable reward effect aligned with uniform norm scaling\.3\)We introduceNormGuard, a simple training\-time regularizer that suppresses excess velocity\-norm growth and improves perceptual quality across a range of models, post\-training methods, and reward functions\.

## 2Related Work

### 2\.1RL Post\-Training of Generative Models

Text\-to\-image generation has seen significant improvements in recent years\[ddpm,ldm,SD3,flux\-2\]\. Inspired by the success of RLHF in language models\[ppo,dpo,grpo\], a growing body of work has adapted these techniques to flow\-based generative models, including Diffusion\-DPO\[DiffusionDPO\], DDPO\[DDPO\], Flow\-GRPO\[Flow\-grpo\], Dance\-GRPO\[DanceGRPO\], Diffusion\-NFT\[NFT\], and AWM\[AWM\]\. All of these methods rely on a reward model such as PickScore\[PickScore\], GenEval\[GenEval\], or HPS\[HPSv2,HPSv3\]to provide the training signal\.

### 2\.2Reward Over\-optimization

A well\-documented pathology of RL fine\-tuning is reward over\-optimization\[ScalingRewardOveroptimization,ConfrontingRewardOveroptimization,DefiningRewardGaming\], where models exploit imperfections in the reward proxy at the expense of true quality\. In text\-to\-image generation, optimizing rewards such as PickScore or HPS can produce artifacts including over\-sharpening, color bias, and unnatural lighting\[realdpo,realgen\], so proxy reward may improve even as perceptual quality deteriorates\.

Several recent works address this issue from different angles\. GRPO\-Guard\[GRPO\-Guard\]stabilizes optimization with ratio normalization and gradient reweighting\. RSA\-FT\[rsa\-ft\]attributes reward hacking to sharp reward landscapes and mitigates it by flattening the landscape in image and parameter space\. RealGen\[realgen\]replaces human\-preference rewards with detector\-based rewards better aligned with photorealism\. RewardDance\[RewardDance\]reduces reward hacking by scaling up the reward model, Pref\-GRPO\[Pref\-GRPO\]replaces pointwise score maximization with pairwise preference fitting to avoid illusory advantages, and GARDO\[GARDO\]introduces gated adaptive regularization and diversity\-aware optimization to balance anti\-hacking and exploration\.

## 3Norm Inflation: Phenomenon and Diagnostics

This section sharpens the empirical motivation for our method\. We first document the norm inflation induced by RL post\-training and relate it to a similar phenomenon in classifier\-free guidance \(Section[3\.1](https://arxiv.org/html/2606.27771#S3.SS1)\)\. We then examine whether the corresponding inference\-time renormalization transfers to RL\-trained models, and whether suppressing the inflated norm exhibits a stable first\-order reward effect at the batch level \(Section[3\.2](https://arxiv.org/html/2606.27771#S3.SS2)\)\.

### 3\.1The Phenomenon: Norm Inflation and Its Visual Signature

![Refer to caption](https://arxiv.org/html/2606.27771v1/x2.png)
Figure 3:RL\-induced norm inflation\.Mean velocity norm across denoising timesteps for AWM, DPO, and NFT on SD3\.5\-Medium\. RL post\-training \(orange\) uniformly inflates the norm above the pretrained reference \(blue dashed\)\.NormGuardregularizer \(magenta\) suppresses the excess without sacrificing reward\.![Refer to caption](https://arxiv.org/html/2606.27771v1/x3.png)
Figure 4:Norm\-Scaling sensitivityS​\(t\)S\(t\)across denoising steps \(SD3\.5\-M \+ 6\.4K samples\)\. The noise\-to\-signal ratioσ/\|μ\|\\sigma/\|\\mu\|ranges from3×3\\timesto100×100\\times, indicating noisy signal along the norm inflation direction\.
##### Empirical observation\.

We measure the per\-step velocity norm‖vθ​\(xt,t\)‖\\\|v\_\{\\theta\}\(x\_\{t\},t\)\\\|before and after RL post\-training on SD3\.5\-Medium \+ PickScore, across three RL methods \(NFT\[NFT\], AWM\[AWM\]and DPO\[DiffusionDPO\]\)\. RL fine\-tuning consistently shifts the velocity norm distribution upward by5%5\\%to15%15\\%, uniformly along the denoising trajectory and across all configurations \(Figure[3](https://arxiv.org/html/2606.27771#S3.F3)\)\.

##### Connection to a known failure mode: CFG\-induced velocity inflation\.

A similar form of norm inflation is well documented in inference\-time classifier\-free guidance \(CFG\)\[cfg\]\. In CFG, the modified velocity is

v^CFG​\(xt,t\)=vuncond​\(xt,t\)\+ω​\(vcond​\(xt,t\)−vuncond​\(xt,t\)\),ω\>1,\\hat\{v\}\_\{\\text\{CFG\}\}\(x\_\{t\},t\)=v\_\{\\text\{uncond\}\}\(x\_\{t\},t\)\+\\omega\\big\(v\_\{\\text\{cond\}\}\(x\_\{t\},t\)\-v\_\{\\text\{uncond\}\}\(x\_\{t\},t\)\\big\),\\quad\\omega\>1,\(1\)whereω\\omegais the guidance scale\. STIV\[cfg\-renorm\]report that forω\>1\\omega\>1,v^CFG\\hat\{v\}\_\{\\text\{CFG\}\}shows substantially larger magnitude than the conditional velocity,‖v^CFG‖\>‖vcond‖\\\|\\hat\{v\}\_\{\\text\{CFG\}\}\\\|\>\\\|v\_\{\\text\{cond\}\}\\\|, especially in early integration stages\. This norm growth drives sampling trajectories to overshoot beyond the learned data distribution, resulting in artifacts\. Their fix,*CFG\-Renormalization*, rescalesv^CFG\\hat\{v\}\_\{\\text\{CFG\}\}to‖vcond‖\\\|v\_\{\\text\{cond\}\}\\\|at inference while preserving its direction:

v^CFG′​\(xt,t\):=‖vcond​\(xt,t\)‖‖v^CFG​\(xt,t\)‖⋅v^CFG​\(xt,t\)\.\\hat\{v\}\_\{\\text\{CFG\}\}^\{\\prime\}\(x\_\{t\},t\):=\\frac\{\\\|v\_\{\\text\{cond\}\}\(x\_\{t\},t\)\\\|\}\{\\\|\\hat\{v\}\_\{\\text\{CFG\}\}\(x\_\{t\},t\)\\\|\}\\cdot\\hat\{v\}\_\{\\text\{CFG\}\}\(x\_\{t\},t\)\.\(2\)
As a result, this inference\-time correction effectively mitigates the artifacts in the CFG setting\. The next subsection asks whether the same intervention can be transferred to inference time for an RL\-trained model\.

### 3\.2Can Inference\-Time Renormalization Transfer to RL?

Table 1:Inference\-time renormalization does not imporve reward\.Δ​R\\Delta Rdenotes the change in reward w/ and w/o velocity renormalization\.MetricConfig\.Bas\.RL\+RenormΔ​R\\Delta RPick\-ScoreSD3\.5 \(NFT\)0\.7680\.7680\.8690\.8690\.8690\.869\\cellcolortablegray\+0\.000\+0\.000SD3\.5 \(AWM\)0\.7680\.7680\.8690\.8690\.8650\.865\\cellcolortablegray−0\.004\-0\.004SD3\.5 \(DPO\)0\.7680\.7680\.8490\.8490\.8500\.850\\cellcolortablegray\+0\.001\+0\.001FLUX \(NFT\)0\.8020\.8020\.9050\.9050\.9040\.904\\cellcolortablegray−0\.001\-0\.001HPSv2SD3\.5 \(NFT\)0\.2890\.2890\.3080\.3080\.3000\.300\\cellcolortablegray−0\.008\-0\.008FLUX \(NFT\)0\.2710\.2710\.3150\.3150\.3130\.313\\cellcolortablegray−0\.002\-0\.002![Refer to caption](https://arxiv.org/html/2606.27771v1/x4.png)
Figure 5:Inference\-time renormalization introduces additional artifacts\.
We present two measurements\. Observation 1 shows that the inference\-time correction used in CFG does not transfer cleanly to the RL setting\. Observation 2 examines whether suppressing velocity magnitude exhibits a stable first\-order reward effect at the batch level\.

##### Observation 1: inference\-time renormalization does not transfer cleanly to RL\-trained models\.

We test whether the CFG\-renorm\-style inference fix transfers\. Given a reward\-fine\-tunedvθv\_\{\\theta\}, we apply the per\-step inference\-time norm scaling

vθ′​\(xt,t\):=‖vref​\(xt,t\)‖‖vθ​\(xt,t\)‖⋅vθ​\(xt,t\),v\_\{\\theta\}^\{\\prime\}\(x\_\{t\},t\):=\\frac\{\\\|v\_\{\\text\{ref\}\}\(x\_\{t\},t\)\\\|\}\{\\\|v\_\{\\theta\}\(x\_\{t\},t\)\\\|\}\\cdot v\_\{\\theta\}\(x\_\{t\},t\),\(3\)which preserves the direction ofvθv\_\{\\theta\}and rescales its magnitude to the reference norm\. Table[1](https://arxiv.org/html/2606.27771#S3.T1)and Figure[5](https://arxiv.org/html/2606.27771#S3.F5)report the result\. Reward does not improve after renormalization, and the sampled images exhibit over\-sharpening and unnatural lighting\. This differs from the CFG case, where norm inflation is introduced explicitly at inference time and can be removed by renormalizing the deployed velocity\. In the RL setting, the same intervention appears insufficient once the inflated norm has been absorbed by the fine\-tuned model, suggesting that a more effective intervention should be applied during training rather than only at inference time\.

##### Observation 2: Velocity magnitude rescaling does not exhibit a stable first\-order reward effect at the batch level\.

Independently of Observation 1, we ask whether suppressing the inflated norm would directly remove useful reward signal\. The first\-order reward change under a velocity\-space perturbationvθ↦vθ\+ε​ηv\_\{\\theta\}\\mapsto v\_\{\\theta\}\+\\varepsilon\\,\\etais

d​Rd​ε\|ε=0=∫01a​\(t\)⊤​η​\(xt,t\)​dt,\\frac\{\\mathrm\{d\}R\}\{\\mathrm\{d\}\\varepsilon\}\\Big\|\_\{\\varepsilon=0\}=\\int\_\{0\}^\{1\}a\(t\)^\{\\top\}\\eta\(x\_\{t\},t\)\\,\\mathrm\{d\}t,\(4\)where the adjoint statea​\(t\)a\(t\)satisfies−a˙​\(t\)=\[∇xtvθ\]⊤​a​\(t\)\-\\dot\{a\}\(t\)=\[\\nabla\_\{x\_\{t\}\}v\_\{\\theta\}\]^\{\\top\}a\(t\)with terminal conditiona​\(1\)=∇x1R​\(x1\)a\(1\)=\\nabla\_\{x\_\{1\}\}R\(x\_\{1\}\)\. Settingη=vθ\\eta=v\_\{\\theta\}\(i\.e\., a multiplicative magnitude scalingvθ↦\(1\+ε\)​vθv\_\{\\theta\}\\mapsto\(1\+\\varepsilon\)v\_\{\\theta\}\) yields the*norm\-scaling sensitivity*

S​\(t\):=vθ​\(xt,t\)⊤​a​\(t\),S\(t\):=v\_\{\\theta\}\(x\_\{t\},t\)^\{\\top\}a\(t\),\(5\)which measures the marginal first\-order reward effect of rescaling velocity magnitude at timett\.

We computeS​\(t\)S\(t\)on6,4006\{,\}400samples from SD3\.5\-Medium with PickScore and HPSv2\. Across all timesteps,SSexhibits substantial per\-sample magnitude and sign\-heterogeneous behavior across prompts, while its batch mean remains close to zero; the noise\-to\-signal ratioσ/\|μ\|\\sigma/\|\\mu\|ranges from3×3\\timesto100×100\\times\(Figure[4](https://arxiv.org/html/2606.27771#S3.F4)\)\. These measurements do not reveal a stable batch\-level first\-order reward signal aligned with uniform norm scaling\. This suggests that suppressing the inflated norm is unlikely to systematically remove a reward\-carrying direction\.

Taken together, these observations point to a training\-time solution\. Direct inference\-time renormalization is insufficient in the RL setting, while the sensitivity analysis does not reveal a stable batch\-level first\-order reward effect tied to uniform norm scaling\. This motivates the regularizer introduced next, which suppresses excess velocity\-norm growth during post\-training\.

## 4NormGuard

We now presentNormGuard\. We first identify a local output\-space structure shared by the post\-training losses considered in this paper \(Section[4\.1](https://arxiv.org/html/2606.27771#S4.SS1)\), and then introduce a norm\-budget regularizer that operates in the same space \(Section[4\.2](https://arxiv.org/html/2606.27771#S4.SS2)\)\.

### 4\.1Velocity\-Local Post\-Training Losses

We focus on post\-training objectives whose gradients act through local velocity residuals in the same output space as flow matching\. For a noisy statextx\_\{t\}at timestepttunder conditioncc, letvθ​\(xt,t,c\)v\_\{\\theta\}\(x\_\{t\},t,c\)denote the model velocity and letv~​\(xt,t,c\)\\tilde\{v\}\(x\_\{t\},t,c\)denote a local target velocity\. The following definition formalizes this shared structure\.

###### Definition 1\(Velocity\-local post\-training loss\)\.

A post\-training objectiveℒpost\\mathcal\{L\}\_\{\\text\{post\}\}is*velocity\-local*if its parameter gradient admits the form

∇θℒpost=𝔼​\[w​\(xt,t,c\)​∇θ‖vθ​\(xt,t,c\)−v~​\(xt,t,c\)‖22\],\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\text\{post\}\}=\\mathbb\{E\}\\\!\\left\[w\(x\_\{t\},t,c\)\\,\\nabla\_\{\\theta\}\\\|v\_\{\\theta\}\(x\_\{t\},t,c\)\-\\tilde\{v\}\(x\_\{t\},t,c\)\\\|\_\{2\}^\{2\}\\right\],\(6\)wherew​\(xt,t,c\)w\(x\_\{t\},t,c\)is a scalar weight that may depend on the sampled statextx\_\{t\}, timesteptt, conditioncc, or reward label\.This definition isolates whether the update is driven by per\-timestep residuals in the velocity output space\.

The three post\-training methods used in our experiments fit this template\.NFTuses reward\-weighted flow\-matching residuals to a target velocity\.AWMreduces, under the FM\-ELBO surrogate, to an advantage\-weighted flow\-matching residual with a KL term instantiated as‖vθ−vref‖22\\\|v\_\{\\theta\}\-v\_\{\\text\{ref\}\}\\\|\_\{2\}^\{2\}\.DPOinjects preference signal through sigmoid\-weighted local deviations from the reference velocity on winner–loser pairs\. By contrast,Flow\-GRPOis a non\-example: its gradient passes through a trajectory\-level likelihood ratio over reverse transitions, rather than a reweighting of single\-step velocity residuals\. Full derivations are deferred to Appendix[7](https://arxiv.org/html/2606.27771#S7)\.

### 4\.2NormGuard: Norm Regularizer

Our goal is to suppress excess velocity\-norm growth during post\-training while interfering as little as possible with directional updates\. We therefore add a one\-sided penalty that activates only when the current velocity norm exceeds the reference\-model norm\.

##### Regularizer\.

For each sampled\(xt,t,c\)\(x\_\{t\},t,c\), we define

ℛnorm​\(θ;xt,t,c\)=λ⋅max⁡\{0,‖vθ​\(xt,t,c\)‖22−‖vref​\(xt,t,c\)‖22‖vref​\(xt,t,c\)‖22\},\\mathcal\{R\}\_\{\\text\{norm\}\}\(\\theta;x\_\{t\},t,c\)=\\lambda\\cdot\\max\\\!\\left\\\{0,\\;\\frac\{\\\|v\_\{\\theta\}\(x\_\{t\},t,c\)\\\|\_\{2\}^\{2\}\-\\\|v\_\{\\text\{ref\}\}\(x\_\{t\},t,c\)\\\|\_\{2\}^\{2\}\}\{\\\|v\_\{\\text\{ref\}\}\(x\_\{t\},t,c\)\\\|\_\{2\}^\{2\}\}\\right\\\},\(7\)and optimize the batch\-averaged objective

ℒ=ℒbase\+𝔼​\[ℛnorm​\(θ;xt,t,c\)\],\\mathcal\{L\}=\\mathcal\{L\}\_\{\\text\{base\}\}\+\\mathbb\{E\}\\big\[\\mathcal\{R\}\_\{\\text\{norm\}\}\(\\theta;x\_\{t\},t,c\)\\big\],\(8\)whereλ\>0\\lambda\>0controls the regularization strength\. The hinge leaves updates unconstrained as long as‖vθ‖22≤‖vref‖22\\\|v\_\{\\theta\}\\\|\_\{2\}^\{2\}\\leq\\\|v\_\{\\text\{ref\}\}\\\|\_\{2\}^\{2\}, and penalizes only the excess norm beyond this reference budget\.

##### Compatibility with velocity\-local losses\.

The penalty acts in the same velocity output space as the base loss\. Whenever the hinge is active, its gradient is proportional toJθ⊤​vθJ\_\{\\theta\}^\{\\top\}v\_\{\\theta\}, so it modifies the update through the local velocity at the sampled state rather than through a separate parameter\-space constraint\. As a result,ℛnorm\\mathcal\{R\}\_\{\\text\{norm\}\}composes naturally with any velocity\-local base loss, including the NFT, AWM, and DPO objectives\.

## 5Experiments

Table 2:Training configuration\.Hyperparameters for all \(model, reward, method\) combinations\.λ\\lambdadenotes the velocity\-norm regularization strength, andrrdenotes the LoRA rank\. Full details are provided in Appendix[8](https://arxiv.org/html/2606.27771#S8)\.![[Uncaptioned image]](https://arxiv.org/html/2606.27771v1/x5.png)

Figure 6:Reward vs\. MLLM quality score\.Our regularizer improves quality with minimal reward change\.
### 5\.1Evaluation Benchmarks

##### Experimental Setup\.

We evaluate ourNormGuardregularizer across two base flow models \(SD3\.5\-Medium\[SD3\]and FLUX\.2\-klein\-base\-4B\[flux\-2\]\), three RL post\-training methods \(NFT\[NFT\], AWM\[AWM\], DPO\[DiffusionDPO\]\), and two reward models \(PickScore\[PickScore\], HPSv2\[HPSv2\]\)\. We conduct all experiments via Flow\-Factory\[Flow\-Factory\]on 8×\\timesGPUs\. Training prompts are drawn from the PickScore dataset, and test prompts are drawn from HPDv3\[HPSv3\]\. Training details are provided in Appendix[8](https://arxiv.org/html/2606.27771#S8)and summarized in Table[2](https://arxiv.org/html/2606.27771#S5.T2)\. For fair comparison, all hyperparameters other thanλ\\lambdafollow the defaults of Flow\-Factory\.

##### MLLM Image Quality Assessment\.

Following recent work on high\-quality RL\[realdpo,realgen\], we use multimodal LLMs as judges to assess image*quality*\. We employ both Qwen3\.5\-35B\-A3B\[qwen3\.5\]and GPT\-4\.1\[gpt4\.1\]to score generated images on six axes: physical plausibility, texture and material fidelity, edge and boundary coherence, color and tone consistency, semantic coherence, and artifact detection \(detailed prompts see Appendix[9](https://arxiv.org/html/2606.27771#S9)\)\. For each prompt, both methods generate an image and the judge provides a pairwise preference \(Win / Tie / Loss\) as well as an absolute score \(1∼101\\sim 10scale\)\. For each configuration, the rates are evaluated over33seeds with300300samples, where the average rates are reported\.

##### Forensic Realism Detection\.

Following RealGen\[realgen\], we additionally evaluate with Forensic\-Chat\[ForensicChat\], an AIGC detector that scores how closely a generated image resembles a real photograph versus a synthetic output\. This provides a complementary measure of*realism*independent of aesthetic preference, capturing artifacts such as unnatural lighting and over\-sharpening that reward models may overlook\.

### 5\.2Main Results

##### Qualitative Results\.

Figure[7](https://arxiv.org/html/2606.27771#S5.F7)shows representative samples from our method and the unregularized baseline across configurations\. The baseline images exhibit a range of quality issues, including texture degradation, edge blurring, and artifact introduction\. In contrast, our regularized models produce natural edges, more coherent textures, and fewer artifacts, suggesting that suppressing radial inflation is associated with tangible quality improvements in the generated images\.

![Refer to caption](https://arxiv.org/html/2606.27771v1/x6.png)Figure 7:Qualitative results\.We compare samples with and withoutNormGuardregularizer\. Prompts see Appendix[10\.2](https://arxiv.org/html/2606.27771#S10.SS2)\.
##### Image Quality and Realism\.

Table 3:Image Quality & Realism\.NormGuardimproves MLLM\-judged quality \(Qwen3\.5 \+ GPT\-4\.1\) and forensic realism \(RealScore\) while largely retaining reward gains\.Reward↑\\uparrowQwen3\.5 Win RateGPT\-4\.1 Win RateRealScore↑\\uparrowRewardModelMethodBas\.OursBas\.OursTieBas\.OursTieBas\.OursPickScoreSD3\.5\-MNFT0\.8690\.86635%\\cellcolortablegray64%1%30%\\cellcolortablegray67%2%0\.248\\cellcolortablegray0\.310SD3\.5\-MDPO0\.8490\.85128%\\cellcolortablegray68%4%20%\\cellcolortablegray47%33%0\.468\\cellcolortablegray0\.472SD3\.5\-MAWM0\.8690\.88028%\\cellcolortablegray72%0%23%\\cellcolortablegray73%4%0\.283\\cellcolortablegray0\.269FLUX\.2\-4BNFT0\.9050\.91038%\\cellcolortablegray47%15%46%\\cellcolortablegray51%3%0\.239\\cellcolortablegray0\.274FLUX\.2\-4BAWM0\.8860\.88641%\\cellcolortablegray59%0%38%\\cellcolortablegray55%7%0\.277\\cellcolortablegray0\.320HPSv2SD3\.5\-MNFT0\.3080\.30942%\\cellcolortablegray58%0%47%\\cellcolortablegray52%1%0\.218\\cellcolortablegray0\.299FLUX\.2\-4BNFT0\.3150\.31136%\\cellcolortablegray63%1%36%\\cellcolortablegray60%4%0\.240\\cellcolortablegray0\.274Table[3](https://arxiv.org/html/2606.27771#S5.T3)reports both MLLM\-judged pairwise win rates and forensic realism scores across all configurations\.1\)On image quality, our regularizer is preferred over the unregularized baseline by both Qwen3\.5\-35B and GPT\-4\.1 in all seven settings, spanning two base models, three fine\-tuning algorithms, and two reward signals\. The two judges agree on preference direction in every case, suggesting that the improvements reflect genuine quality gains rather than evaluator bias\.2\)On forensic realism, our method improves RealScore in six out of seven configurations\. Under AWM,NormGuardappears to trade a small amount of detector\-based realism for a larger gain in MLLM perceptual quality, suggesting that these axes are not perfectly aligned\.3\)On reward, our method largely preserves the gains of the unregularized baseline, with PickScore differences of−0\.003∼\+0\.011\-0\.003\\sim\+0\.011and HPSv2 differences of−0\.004∼\+0\.001\-0\.004\\sim\+0\.001\. This suggests that controlling radial inflation does not appear to materially hinder the directional changes that are more reward\-aligned in our measurements, consistent with our diagnosis\.

Figure[6](https://arxiv.org/html/2606.27771#S5.F6)visualizes the relationship between reward and quality across all configurations\. The arrows from baseline to ours are nearly vertical: MLLM quality improves substantially while PickScore remains largely unchanged\.

### 5\.3Ablations

Table 4:Few\-step ablation\(FLUX\.2\-4B, NFT, PickScore\)\. Our regularizer’s advantage grows as steps decrease\.
Table 5:Early stopping ablation\(FLUX\.2\-4B, NFT, PickScore\)\. Our method at iter 200 outperforms all earlier checkpoints\.

##### Few\-step robustness\.

Table[5](https://arxiv.org/html/2606.27771#S5.T5)reports quality metrics as inference steps are reduced from2828to44\. The advantage of our regularizer grows monotonically with fewer steps: the MLLM win\-rate gap widens from9%9\\%at2828steps to20%20\\%at44steps, while the baseline’s RealScore degrades sharply \(0\.2390\.239→\\to0\.1890\.189\) and ours remains comparatively stable\. This is consistent with our diagnosis: fewer steps mean larger step sizes in the ODE integrator, amplifying the effect of velocity\-norm inflation on discretization error\.

##### Not explained by early stopping\.

A potential confound is that our regularizer simply slows training, and the same quality could be obtained by stopping the baseline earlier\. Table[5](https://arxiv.org/html/2606.27771#S5.T5)rules this out: we compare baseline checkpoints at iterations160160,180180, and200200against our method at iteration200200\. Our regularized model achieves higher reward, higher RealScore, and higher MLLM score than*any*earlier baseline checkpoint\. The improvement is not accounted for by the early\-stopping baselines we test\.

##### Complementarity with KL regularization\.

Table 6:KL complementarity\(FLUX\.2\-4B, NFT, PickScore\)\. Our regularizer \(λ\\lambda\) improves RealScore at every KL strength \(βKL\\beta\_\{\\text\{KL\}\}\)\.![[Uncaptioned image]](https://arxiv.org/html/2606.27771v1/x7.png)

Figure 8:Training curve of PickScore evaluation corresponding to Table[6](https://arxiv.org/html/2606.27771#S5.T6)\.

![[Uncaptioned image]](https://arxiv.org/html/2606.27771v1/x8.png)

Figure 9:PickScore evaluation with different regularizer strengths \(λ\\lambda\)\.

Table[6](https://arxiv.org/html/2606.27771#S5.T6)shows that our regularizer is complementary to KL regularization rather than redundant with it\. At every level ofβKL\\beta\_\{\\text\{KL\}\}, adding our norm regularization \(λVN=1\.0\\lambda\_\{\\text\{VN\}\}=1\.0\) consistently improves RealScore\. Our regularizer targets excess radial growth specifically, whereas KL does not selectively suppress this norm inflation and may also constrain directional changes that appear more reward\-aligned in our measurements\. The two penalties appear to address distinct failure modes and combine additively in our measurements\.

##### Sensitivity toλ\\lambda\.

We vary the regularization strength overλ∈\{0,0\.1,1,10\}\\lambda\\in\\\{0,0\.1,1,10\\\}and track PickScore during training\. Nonzero regularization consistently outperformsλ=0\\lambda=0, suggesting that controlling radial growth is important\. Among the tested values, intermediate strengths work best:λ=1\\lambda=1achieves the highest score at200200iterations, whileλ=10\\lambda=10appears slightly over\-regularized early in training\. Overall, performance is stable across a broad range of nonzeroλ\\lambda, indicating that the method is not sensitive to precise tuning\.

## 6Conclusion

RL post\-training inflates the per\-step velocity norm by55to15%15\\%, producing the same artifact family as CFG over\-amplification\. Unlike CFG, however, RL inflation is trained into the weights: inference\-time renormalization fails because the network has co\-adapted to the inflated norm, while an adjoint sensitivity analysis shows that suppressing velocity magnitude carries no coherent first\-order reward signal at the batch level\. Because training\-time intervention is both necessary and safe, we proposeNormGuard, a one\-sided hinge penalty on excess velocity norm that composes additively with any velocity\-local base loss\. Across two base models, three post\-training methods, and two reward proxies,NormGuardconsistently improves perceptual quality and realism while preserving reward, with gains that amplify under few\-step inference and are not explained by early stopping or redundant with KL regularization\.

##### Limitations and future work\.

The dynamic origin of the radial inflation remains an open question\. Our reward\-neutrality claim concerns only first\-order signal at the batch level; per\-sample radial perturbations do affect reward, but their effects average to zero across prompts\. Our analysis applies to velocity\-local objectives \(NFT, AWM, DPO\); extending the norm\-budget perspective to trajectory\-level objectives such as Flow\-GRPO, where gradients flow through likelihood ratios over reverse transitions, is left for future investigation\.

## Acknowledgments

This work was supported by Kuaishou Technology and partially supported by the Beijing Natural Science Foundation under Grant No\. QY25188\.

## References

\\beginappendix

## 7Theoretical Analysis of Velocity\-Local Post\-Training Losses

This appendix provides the formal derivations underlying the velocity\-local property in Section[4\.1](https://arxiv.org/html/2606.27771#S4.SS1)\. We unify four representative RL post\-training methods \(Diffusion\-DPO, Diffusion\-NFT, AWM, Flow\-GRPO\) under a single notation and identify which of them admit a velocity\-residual gradient form\.

### 7\.1Notation

##### Shared symbols\.

- •xt=\(1−t\)​x0\+t​ϵx\_\{t\}=\(1\-t\)x\_\{0\}\+t\\,\\epsilonwithϵ∼𝒩​\(0,I\)\\epsilon\\sim\\mathcal\{N\}\(0,I\): noisy latent at timett\.
- •vθ​\(xt,t,c\)v\_\{\\theta\}\(x\_\{t\},t,c\): trainable velocity predictor\.
- •vtarget:=ϵ−x0v^\{\\text\{target\}\}:=\\epsilon\-x\_\{0\}: flow\-matching target\.
- •vrefv\_\{\\text\{ref\}\}: frozen reference velocity field \(the pretrained model\)\.
- •ω​\(t\)\\omega\(t\): a timestep\-dependent weighting\.
- •Jθ​\(xt,t,c\):=∇θvθ​\(xt,t,c\)J\_\{\\theta\}\(x\_\{t\},t,c\):=\\nabla\_\{\\theta\}v\_\{\\theta\}\(x\_\{t\},t,c\): per\-sample velocity Jacobian\.

The flow\-matching pretraining objective is

ℒFM​\(θ\)=𝔼x0,ϵ,t,c​\[ω​\(t\)​‖vθ​\(xt,t,c\)−vtarget‖22\]\.\\mathcal\{L\}\_\{\\text\{FM\}\}\(\\theta\)=\\mathbb\{E\}\_\{x\_\{0\},\\epsilon,t,c\}\\left\[\\omega\(t\)\\,\\\|v\_\{\\theta\}\(x\_\{t\},t,c\)\-v^\{\\text\{target\}\}\\\|\_\{2\}^\{2\}\\right\]\.\(9\)

### 7\.2Method Decompositions in Velocity Coordinates

We now express each method’s training signal in the coordinate system above and verify the velocity\-local property of Definition[1](https://arxiv.org/html/2606.27771#Thmdefinition1)\.

#### 7\.2\.1Diffusion\-DPO

Starting from the Bradley–Terry preference model and reparameterizing the reward through the diffusion likelihood, the practical Diffusion\-DPO loss can be rewritten in a per\-timestep comparison form between a winning samplex0wx\_\{0\}^\{w\}and a losing samplex0lx\_\{0\}^\{l\}:

ℒDPO\(θ\)=−𝔼t,ϵw,ϵllogσ\(−βTω\(λt\)\[\\displaystyle\\mathcal\{L\}\_\{\\text\{DPO\}\}\(\\theta\)=\-\\mathbb\{E\}\_\{t,\\epsilon^\{w\},\\epsilon^\{l\}\}\\log\\sigma\\Big\(\-\\beta T\\omega\(\\lambda\_\{t\}\)\\big\[‖vθ​\(xtw,t\)−vref​\(xtw,t\)‖22⏟winner deviation\\displaystyle\\underbrace\{\\\|v\_\{\\theta\}\(x\_\{t\}^\{w\},t\)\-v\_\{\\text\{ref\}\}\(x\_\{t\}^\{w\},t\)\\\|\_\{2\}^\{2\}\}\_\{\\text\{winner deviation\}\}\(10\)−\\displaystyle\-‖vθ​\(xtl,t\)−vref​\(xtl,t\)‖22⏟loser deviation\]\)\.\\displaystyle\\underbrace\{\\\|v\_\{\\theta\}\(x\_\{t\}^\{l\},t\)\-v\_\{\\text\{ref\}\}\(x\_\{t\}^\{l\},t\)\\\|\_\{2\}^\{2\}\}\_\{\\text\{loser deviation\}\}\\big\]\\Big\)\.The gradient is a sigmoid\-weighted sum of two velocity\-residual gradients with targetv~=vref\\tilde\{v\}=v\_\{\\text\{ref\}\}, hence velocity\-local\.

#### 7\.2\.2Diffusion\-NFT

NFT constructs implicit positive and negative policies from the same trainable parameters:

vθ\+​\(xt,c,t\)\\displaystyle v\_\{\\theta\}^\{\+\}\(x\_\{t\},c,t\)=\(1−β\)​vold​\(xt,c,t\)\+β​vθ​\(xt,c,t\),\\displaystyle=\(1\-\\beta\)\\,v^\{\\text\{old\}\}\(x\_\{t\},c,t\)\+\\beta\\,v\_\{\\theta\}\(x\_\{t\},c,t\),\(11\)vθ−​\(xt,c,t\)\\displaystyle v\_\{\\theta\}^\{\-\}\(x\_\{t\},c,t\)=\(1\+β\)​vold​\(xt,c,t\)−β​vθ​\(xt,c,t\)\.\\displaystyle=\(1\+\\beta\)\\,v^\{\\text\{old\}\}\(x\_\{t\},c,t\)\-\\beta\\,v\_\{\\theta\}\(x\_\{t\},c,t\)\.\(12\)The training objective is a supervised regression on the forward process:

ℒNFT​\(θ\)=𝔼c,x0,t​\[r​‖vθ\+​\(xt,c,t\)−vtarget‖22\+\(1−r\)​‖vθ−​\(xt,c,t\)−vtarget‖22\],\\mathcal\{L\}\_\{\\text\{NFT\}\}\(\\theta\)=\\mathbb\{E\}\_\{c,x\_\{0\},t\}\\Big\[r\\,\\\|v\_\{\\theta\}^\{\+\}\(x\_\{t\},c,t\)\-v^\{\\text\{target\}\}\\\|\_\{2\}^\{2\}\+\(1\-r\)\\,\\\|v\_\{\\theta\}^\{\-\}\(x\_\{t\},c,t\)\-v^\{\\text\{target\}\}\\\|\_\{2\}^\{2\}\\Big\],\(13\)wherer∈\{0,1\}r\\in\\\{0,1\\\}is a binary optimality label derived from the reward\. The gradient is a reward\-weighted FM residual tovtargetv^\{\\text\{target\}\}, hence velocity\-local\.

#### 7\.2\.3Advantage\-Weighted Matching \(AWM\)

AWM replaces the per\-step reverse\-transition likelihood used in DDPO with a sequence\-level policy whose likelihood is approximated through the flow\-matching ELBO\. The GRPO\-style objective is

𝒥AWM​\(θ\)=𝔼c,\{x0i\}i=1G∼πθold​1G​∑i=1G\(π^θ​\(x0i∣c\)π^θold​\(x0i∣c\)⋅Ai−β​DKL​\(π^θ∥π^ref\)\)\.\\mathcal\{J\}\_\{\\text\{AWM\}\}\(\\theta\)=\\mathbb\{E\}\_\{c,\\\{x\_\{0\}^\{i\}\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\theta\_\{\\text\{old\}\}\}\}\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\left\(\\frac\{\\hat\{\\pi\}\_\{\\theta\}\(x\_\{0\}^\{i\}\\mid c\)\}\{\\hat\{\\pi\}\_\{\\theta\_\{\\text\{old\}\}\}\(x\_\{0\}^\{i\}\\mid c\)\}\\cdot A\_\{i\}\-\\beta\\,D\_\{\\text\{KL\}\}\(\\hat\{\\pi\}\_\{\\theta\}\\,\\\|\\,\\hat\{\\pi\}\_\{\\text\{ref\}\}\)\\right\)\.\(14\)Under the FM\-ELBO surrogate, the likelihood ratio is estimated through the difference of two flow\-matching losses, and the per\-sample gradient simplifies to

∇θ𝒥AWM∝−∇θ𝔼t​\[ω​\(t\)​‖vθ​\(xt,t,c\)−vtarget‖22\]⋅Ai\.\\nabla\_\{\\theta\}\\mathcal\{J\}\_\{\\text\{AWM\}\}\\propto\-\\nabla\_\{\\theta\}\\,\\mathbb\{E\}\_\{t\}\\\!\\left\[\\omega\(t\)\\,\\\|v\_\{\\theta\}\(x\_\{t\},t,c\)\-v^\{\\text\{target\}\}\\\|\_\{2\}^\{2\}\\right\]\\cdot A\_\{i\}\.\(15\)The KL term is also instantiated as a velocity\-space MSE:

DKL∝ω​\(t\)​‖vθ​\(xt,t,c\)−vref​\(xt,t,c\)‖22\.D\_\{\\text\{KL\}\}\\propto\\omega\(t\)\\,\\\|v\_\{\\theta\}\(x\_\{t\},t,c\)\-v\_\{\\text\{ref\}\}\(x\_\{t\},t,c\)\\\|\_\{2\}^\{2\}\.\(16\)
AWM makes the structural symmetry between pretraining and RL explicit: at the optimization level, the policy update is the same per\-timestep FM residual, modulated only by an advantage weightAiA\_\{i\}\. Both the policy and KL terms are therefore velocity\-local\.

#### 7\.2\.4Flow\-GRPO

Flow\-GRPO formulates the denoising process as an MDP with statest=\(c,t,xt\)s\_\{t\}=\(c,t,x\_\{t\}\), actionat=xt−1a\_\{t\}=x\_\{t\-1\}, and policy equal to the reverse transitionπ​\(at∣st\)=pθ​\(xt−1∣xt,c\)\\pi\(a\_\{t\}\\mid s\_\{t\}\)=p\_\{\\theta\}\(x\_\{t\-1\}\\mid x\_\{t\},c\)\. The objective is a clipped policy gradient with importance ratio

rti​\(θ\)=pθ​\(xt−1i∣xti,c\)pθold​\(xt−1i∣xti,c\)\.r\_\{t\}^\{i\}\(\\theta\)=\\frac\{p\_\{\\theta\}\(x\_\{t\-1\}^\{i\}\\mid x\_\{t\}^\{i\},c\)\}\{p\_\{\\theta\_\{\\text\{old\}\}\}\(x\_\{t\-1\}^\{i\}\\mid x\_\{t\}^\{i\},c\)\}\.\(17\)
To enable stochastic exploration, Flow\-GRPO converts the deterministic ODE into an SDE and discretizes via Euler–Maruyama, yielding a Gaussian reverse transition

pθ​\(xt\+Δ​t∣xt,c\)=𝒩​\(x¯t\+Δ​t,θ,σt2​Δ​t⋅I\),p\_\{\\theta\}\(x\_\{t\+\\Delta t\}\\mid x\_\{t\},c\)=\\mathcal\{N\}\\\!\\left\(\\bar\{x\}\_\{t\+\\Delta t,\\theta\},\\;\\sigma\_\{t\}^\{2\}\\Delta t\\cdot I\\right\),\(18\)where the meanx¯t\+Δ​t,θ\\bar\{x\}\_\{t\+\\Delta t,\\theta\}depends onvθ​\(xt,t\)v\_\{\\theta\}\(x\_\{t\},t\)through the drift term\. Because the variance is shared betweenπθ\\pi\_\{\\theta\}andπθold\\pi\_\{\\theta\_\{\\text\{old\}\}\}, the importance ratio reduces to a function of the two Gaussian means:

rti​\(θ\)=exp⁡\(‖xt\+Δ​ti−x¯t\+Δ​t,old‖2−‖xt\+Δ​ti−x¯t\+Δ​t,θ‖22​σt2​Δ​t\)\.r\_\{t\}^\{i\}\(\\theta\)=\\exp\\\!\\left\(\\frac\{\\\|x\_\{t\+\\Delta t\}^\{i\}\-\\bar\{x\}\_\{t\+\\Delta t,\\text\{old\}\}\\\|^\{2\}\-\\\|x\_\{t\+\\Delta t\}^\{i\}\-\\bar\{x\}\_\{t\+\\Delta t,\\theta\}\\\|^\{2\}\}\{2\\sigma\_\{t\}^\{2\}\\Delta t\}\\right\)\.\(19\)The gradient flows through this trajectory\-level likelihood ratio rather than through a per\-timestep velocity residual;vθv\_\{\\theta\}enters only indirectly through the Gaussian transition mean\. Flow\-GRPO therefore violates Definition[1](https://arxiv.org/html/2606.27771#Thmdefinition1), and we treat Flow\-GRPO as out of scope for this work\.

### 7\.3Summary

Table 7:Classification of post\-training methods by velocity\-locality \(Definition[1](https://arxiv.org/html/2606.27771#Thmdefinition1)\)\.

## 8Training Details

Table[8](https://arxiv.org/html/2606.27771#S8.T8)provides the full training hyperparameters for each \(model, method, reward\) combination\. Table[9](https://arxiv.org/html/2606.27771#S8.T9)details method\-specific hyperparameters, and Table[10](https://arxiv.org/html/2606.27771#S8.T10)lists shared optimization settings\. For detailed implementation, see Flow\-Factory\[Flow\-Factory\]codes\.

Table 8:Main experiments\.Full hyperparameters for each \(model, method, reward\) combination\. HereTrain Stepsmeans the number of sampling steps during rollout, andGSmeans group size\.Table 9:Method\-specific hyperparameters\.Table 10:Shared optimization settings\.
## 9MLLM Evaluation Prompts

`Image Realism Judge Prompt`

`10 Prompts for Main\-Paper Figures This appendix lists the text prompts used to generate qualitative examples that appear in the main paper\. We group prompts by figure for ease of reference\. All the prompts are sampled from HPDv3 \[HPSv2\]\. 10\.1 Teaser Tropical coastline scene\. Bird over mountain vista\. Tree silhouette at sunset\. 10\.2 Main Results SD3\.5 \+ PickScore \+ NFT\. Person with smartphone, red\-lit urban night\. 3D sculpted book landscape\. Pumpkin patch in autumn\. SD3\.5 \+ PickScore \+ AWM\. Buddha statue with torii gate, twilight\. Woman on steps, urban mural background\. Graffiti wall, woman and alien character\. SD3\.5 \+ PickScore \+ AWM \+ 10 Steps\. Abstract painted landscape, stylized village\. Solitary figure on rocky mountain precipice\. Hanukkah gifts, snow globe with dreidel\. FLUX\.2 \+ PickScore \+ NFT\. Couple on skateboard, sepia road at dusk\. Metallic sculpture, Milky Way backdrop\. Man in modern chair, whiskey glass, airy interior\. FLUX\.2 \+ HPSv2 \+ NFT\. Camper at dusk, lantern and portable stove\. Couple on wet beach, ocean backdrop, golden hour\. Solitary figure on rocky precipice, misty peaks\.`