ReNFT: Repairing Mode Collapse in Reward Post-Training via Internal Probability-Mass Recalibration

# ReNFT: Repairing Mode Collapse in Reward Post-Training via Internal Probability-Mass Recalibration
Source: [https://arxiv.org/html/2609.00061](https://arxiv.org/html/2609.00061)
Chao WenHaowei WangRuoxin ChenDonghao LuoJiahui ZhanWenjian HuangShen ChenYiting WangTaiping YaoChengjie WangShouhong DingJianguo Zhang††thanks:Corresponding author\.

###### Abstract

Reward post\-training of diffusion generators inevitably concentrates probability mass on a few reward\-favored modes, a mode collapse that erases within\-prompt diversity\. Existing methods for mitigating collapse rely on external signals or interfaces, augmenting the reward with perceptual objectives, adjusting reference regularization, or modifying the text encoder, but none repairs an adapter that has already collapsed while preserving the acquired reward\. We observe that online post\-training primarily reallocates probability mass over capabilities inherited from pretraining rather than learning new visual content\. Collapse is therefore suppression, not deletion, and can be reversed from within the generator\. We proposeReNFT, which repairs a high\-reward, low\-diversity adapter through internal probability\-mass recalibration\. Unconditional probes first prioritize “anti\-hub” prompts where the prompt\-independent bias is easiest to expose\. Two policy\-dominated mixed routes then generate matched counterfactual proposals from the same prompt and initial noise, one probing the frozen base direction for suppressed alternatives and the other exposing the post\-trained unconditional tendency\. Reward ranking with an adaptive flipping guard assigns pull and push roles, and a joint\-and\-paired NFT update realizes the repair\. On PickScore and GenEval,ReNFTretains 98\.9% and 99\.0% of NFT’s reward while improving DreamSim\-Div by 58\.8% and 55\.0%, respectively, offering a complementary alternative to external interventions\.

1Southern University of Science and Technology

2Tencent Youtu Lab

## 1Introduction

Reward\-based post\-training has become a practical way to align diffusion and flow generators beyond pretraining\. Preference optimization and policy\-gradient methods, represented by DiffusionDPO and Flow\-GRPO, improve alignment by increasing the probability of reward\-preferred outputs\([Wallace et al\. 2024](https://arxiv.org/html/2609.00061#bib.bib32);[Liu et al\. 2025](https://arxiv.org/html/2609.00061#bib.bib20)\)\. DiffusionNFT \(Negative\-aware Fine\-Tuning, NFT\)\([Zheng et al\. 2026a](https://arxiv.org/html/2609.00061#bib.bib40)\)makes this alignment markedly more efficient: it replaces likelihood\-ratio policy gradients with forward\-process reconstruction, requires no classifier\-free guidance, and converges much faster\. The same feedback loop, however, inevitably concentrates probability mass; the more efficient the optimization, the more severe the concentration: whenever a narrow visual mode repeatedly receives high reward, different initial noises under the same prompt are mapped to similar outputs\. Reward differences among such semantically equivalent samples encode the preference of the reward model rather than genuine quality, so optimizing them only sharpens the preference\. The result is a high\-reward adapter whose within\-prompt structural and stylistic diversity has collapsed, most severely under the fastest optimizer, NFT\.

![Refer to caption](https://arxiv.org/html/2609.00061v1/figures/distribution1.png)Figure 1:Conceptual view of probability\-mass recalibration\.Pretraining spreads probability across the modes of each prompt, while unconditional behavior spans their marginal range\. NFT collapses each prompt onto a reward\-near mode and unconditional behavior onto a fixed mode, suppressing alternatives\.ReNFTrestores unconditional coverage and reward\-boundary modes while leaving low\-quality modes outside suppressed\. Gray dashed lines compare the prompts’ highest\-reward mode probabilities\.Existing methods for mitigating collapse mainly intervene at three positions\. Reward\-shaping approaches augment the reward with external perceptual signals\([Liu et al\. 2026a](https://arxiv.org/html/2609.00061#bib.bib18);[Tan et al\. 2026](https://arxiv.org/html/2609.00061#bib.bib29);[Liu et al\. 2026b](https://arxiv.org/html/2609.00061#bib.bib19)\), but can only reweight among samples the policy still generates\. Regularization\-based approaches align the policy toward the base model\([Liu et al\. 2026a](https://arxiv.org/html/2609.00061#bib.bib18);[He et al\. 2025](https://arxiv.org/html/2609.00061#bib.bib10)\), tying achievable reward to the base ceiling\. Interface\-level methods modify text representations\([Hu et al\. 2026](https://arxiv.org/html/2609.00061#bib.bib11);[Chen et al\. 2026](https://arxiv.org/html/2609.00061#bib.bib3)\), but depend on the architecture of the reward model\. We therefore ask:*can an already\-collapsed adapter be repaired using only distributions already inside the generator, without external diversity objectives or text\-encoder modification, and without sacrificing its acquired reward?*

Our answer begins with a distinction: pretraining learns from external images, whereas online post\-training supervises the generator with its own scored samples, so it reallocates probability mass over capabilities inherited from the base model, amplifying reward\-favored modes while starving alternatives\. Collapse is therefore suppression, not deletion: suppressed modes remain reachable through internal routes of the generator, and restoring their probability mass requires no new visual knowledge\. We refer to this operational repair view as*probability\-mass recalibration*\.

Recalibration first requires an internal signal that exposes where the mass has gone\. The post\-trained unconditional route provides such a readout: it shares the updated parameters but drops the prompt condition, exposing what the adapter injects without being asked\. This readout is diagnostic rather than a training target: the repair must act on conditional generations, and optimizing the unconditional direction directly offers no handle on per\-prompt structure\. We therefore use it in two indirect ways: locating prompts far from the exposed tendency, and constructing counterfactual samples inside ordinary conditional trajectories, where the tendency becomes testable by the original reward model\.

These observations motivateReNFT\(*Repair NFT*\), which repairs high\-reward, low\-diversity adapters through internal probability\-mass recalibration\. To expose the bias where it is most distinguishable, unconditional probes first prioritize*“anti\-hub”*prompts, those farthest from the unconditional*hub*, the shared mode toward which the unconditional distribution concentrates under reward hacking\. To obtain candidates without any external signal, two policy\-dominated mixed routes generate matched proposals from the same prompt and initial noise: one probes the frozen base direction for suppressed alternatives, while the other exposes the post\-trained unconditional tendency\. The original reward then ranks the pair into pull and mirrored push targets, and an adaptive flipping guard keeps at least half of the pull targets on the base\-probing route\. A native\-noise joint update then anchors the preference at the shared trajectory origin, and fresh\-noise paired updates propagate it across intermediate noise levels\. A frozen image–text encoder is used only to prioritize prompts; it never enters the reward, advantage, or repair loss\.

In summary, our contributions are as follows:

- •We formulate reward\-induced mode collapse as an internal probability\-mass reallocation problem and study the repair of a severely reward\-hacked, high\-reward adapter rather than diversity\-aware training from scratch\.
- •We identify the post\-trained unconditional route as a free internal readout of reward bias and exploit it in two ways: prioritizing*“anti\-hub”*prompts farthest from the exposed tendency, and injecting the tendency into conditional trajectories to turn a hidden bias into an explicit, reward\-verifiable candidate\.
- •We realize probability\-mass recalibration through two internal routes of the same generator: branched from the same prompt and initial noise, the frozen base route expands probability mass back toward suppressed alternatives, while the unconditional route exposes the learned bias in a mixed endpoint as a mirrored\-push candidate; pull and push follow the reward gap of each pair rather than route identity\. This pair construction is training\-only; the repaired adapter samples with the standard conditional forward pass, preserving 98\.9–99\.0% of NFT’s reward while improving DreamSim\-Div\([Fu et al\. 2023](https://arxiv.org/html/2609.00061#bib.bib7)\)by 58\.8% and 55\.0% on PickScore\([Kirstain et al\. 2023](https://arxiv.org/html/2609.00061#bib.bib13)\)and GenEval\([Ghosh, Hajishirzi, and Schmidt 2023](https://arxiv.org/html/2609.00061#bib.bib8)\), respectively\.

## 2Related Work

![Refer to caption](https://arxiv.org/html/2609.00061v1/figures/figure1_combined_dynamics.png)Figure 2:Training dynamics and ablations\.\(a\)\(b\): PickScore reward and DreamSim\-Div; \(c\)\(d\): GenEval reward and DreamSim\-Div\.ReNFTbranches from the hacked checkpointHH\(star\) and repairs for 50 steps; the dashed line marks the base generator\. \(e\)\(f\): mixed\-route pattern and anti\-hub ablations after 50 repair steps from the sameHH; the dashed line marks NFT atHH\.DiffusionDPO\([Wallace et al\. 2024](https://arxiv.org/html/2609.00061#bib.bib32)\)adapts preference optimization to reward post\-training of diffusion and flow generators; Flow\-GRPO\([Liu et al\. 2025](https://arxiv.org/html/2609.00061#bib.bib20)\), DenseGRPO\([Deng et al\. 2026](https://arxiv.org/html/2609.00061#bib.bib4)\), and AWM\([Xue et al\. 2026](https://arxiv.org/html/2609.00061#bib.bib37)\)improve group\-relative optimization, reward density, or stability; and DiffusionNFT\([Zheng et al\. 2026a](https://arxiv.org/html/2609.00061#bib.bib40)\)transfers supervision to forward\-process reconstruction;[Liu, He, and Li \(2026\)](https://arxiv.org/html/2609.00061#bib.bib21)survey this line\. Subsequent variants refine clipping, credit assignment, self\-correction, and distillation\([Ping et al\. 2026](https://arxiv.org/html/2609.00061#bib.bib23);[Tong et al\. 2026](https://arxiv.org/html/2609.00061#bib.bib30);[Qin et al\. 2026](https://arxiv.org/html/2609.00061#bib.bib24);[Li et al\. 2026b](https://arxiv.org/html/2609.00061#bib.bib15);[Fang et al\. 2026](https://arxiv.org/html/2609.00061#bib.bib6);[Go et al\. 2026](https://arxiv.org/html/2609.00061#bib.bib9);[Li et al\. 2025](https://arxiv.org/html/2609.00061#bib.bib17)\)\. Without a mechanism preserving broad support, repeatedly reinforcing self\-generated high\-reward samples concentrates probability mass on a few modes, i\.e\., mode collapse, as also observed for on\-policy, reverse\-divergence objectives\([Zheng et al\. 2026b](https://arxiv.org/html/2609.00061#bib.bib41)\)\. Interventions differ mainly in where they inject the diversity signal\.

Reward\-shaping approaches\.A first line augments the reward or advantage with perceptual diversity signals from external models\. DiverseGRPO\([Liu et al\. 2026a](https://arxiv.org/html/2609.00061#bib.bib18)\)clusters same\-prompt samples in CLIP space and adds an exploration bonus inversely proportional to cluster size\. PEC\([Tan et al\. 2026](https://arxiv.org/html/2609.00061#bib.bib29)\)replaces policy entropy with a perceptual\-entropy proxy from rollout states\. DRIFT\([Liu et al\. 2026b](https://arxiv.org/html/2609.00061#bib.bib19)\)combines reward\-concentrated rollout selection, prompt perturbation, and potential\-based shaping\. Related work replaces the reward signal itself with pairwise win rates, gated or adversarial advantages, or set\-level statistics\([Wang et al\. 2025](https://arxiv.org/html/2609.00061#bib.bib33);[Mao et al\. 2025](https://arxiv.org/html/2609.00061#bib.bib22);[Li et al\. 2026c](https://arxiv.org/html/2609.00061#bib.bib16)\)\.

Regularization\-based approaches\.A second line modifies reference regularization\. DiverseGRPO\([Liu et al\. 2026a](https://arxiv.org/html/2609.00061#bib.bib18)\)applies stronger KL regularization during early denoising steps and relaxes it later\. GARDO\([He et al\. 2025](https://arxiv.org/html/2609.00061#bib.bib10)\)gates penalization to high\-uncertainty samples, periodically updates the EMA reference, and amplifies rewards for high\-quality diverse samples\.

Interface\-level approaches\.A third line modifies the semantic interface through which prompts or rewards enter optimization\. D2\-Align\([Chen et al\. 2026](https://arxiv.org/html/2609.00061#bib.bib3)\)learns a directional correction in the text\-embedding space of a frozen reward model\. E2PO\([Hu et al\. 2026](https://arxiv.org/html/2609.00061#bib.bib11)\)perturbs content\-token embeddings on the generator side and anneals the perturbation during denoising\.

ReNFTis complementary to all three lines: its repair signal comes from components already inside the post\-trained generator \(a frozen encoder serves only to prioritize prompts\)\. A natural question is whether these methods can also*repair*an already\-collapsed adapter: reward\-shaping and interface\-level methods are designed for prevention during training, while KL\-style anchoring can restore suppressed modes but ties the attainable reward to the base model\.ReNFTrealizes this repair through two internal routes of the same generator: a frozen base route that probes suppressed alternatives and an unconditional route that exposes prompt\-independent bias\. Trajectory\-routing work\([Soboleva et al\. 2025](https://arxiv.org/html/2609.00061#bib.bib28);[Cao et al\. 2026](https://arxiv.org/html/2609.00061#bib.bib2);[Yin et al\. 2026](https://arxiv.org/html/2609.00061#bib.bib38);[Jin, Shi, and Gu 2026](https://arxiv.org/html/2609.00061#bib.bib12)\)likewise shows that route identity and timestep matter, but does not construct reward\-ranked, shared\-noise counterfactuals\.

## 3Preliminaries

![Refer to caption](https://arxiv.org/html/2609.00061v1/figures/pipeline1.png)Figure 3:ReNFT pipeline\.\(a\) Unconditional probes prioritize anti\-hub prompts; \(b\) two policy\-dominated mixed routes generate matched proposals from the same prompt and initial noise; \(c\) reward ranking and the adaptive flipping guard assign pull/push targets for joint\-and\-paired NFT repair\. The VAE decoder is omitted for clarity\.We introduce the flow\-matching notation and the NFT\-style forward\-process post\-training update that form the basis of our method\.

### 3\.1Flow\-Matching Generation

Letyydenote a text prompt,x0x\_\{0\}a clean latent, andε∼𝒩⁡\(0,I\)\\varepsilon\\sim\\mathcal\{N\}\(0,I\)the initial noise\. Under rectified\-flow interpolation,

xt=\(1−t\)​x0\+t​ε,t∈\[0,1\],x\_\{t\}=\(1\-t\)x\_\{0\}\+t\\varepsilon,\\quad t\\in\[0,1\],\(1\)the target transport velocity isv=ε−x0v=\\varepsilon\-x\_\{0\}, and the model predictsvθ​\(xt,t,y\)v\_\{\\theta\}\(x\_\{t\},t,y\)trained with the standard flow\-matching regression objective\. We denote three routes through this sampler:bb\(frozen base\),θ\\theta\(conditional policy\), anduu\(unconditional route of the current policy\); an EMA\-smoothed copy of the policy provides a stable reference, with its velocity writtenvoldv\_\{\\mathrm\{old\}\}\.

### 3\.2NFT\-Style Forward\-Process Post\-Training

Reward\-based post\-training reshapes the distribution of a pretrained generator using a reward modelR⁡\(x0,y\)R\(x\_\{0\},y\)\. DiffusionNFT\([Zheng et al\. 2026a](https://arxiv.org/html/2609.00061#bib.bib40)\)avoids likelihood\-ratio policy gradients by applying reward supervision through forward\-process regression\. Each generated endpointx0x\_\{0\}\(from a rollout ofNiN\_\{i\}ODE steps\) is re\-noised atNtN\_\{t\}training timesteps toxt=\(1−σt\)​x0\+σt​ξx\_\{t\}=\(1\-\\sigma\_\{t\}\)x\_\{0\}\+\\sigma\_\{t\}\\xi, with freshξ∼𝒩⁡\(0,I\)\\xi\\sim\\mathcal\{N\}\(0,I\)and noise levelσt=t\\sigma\_\{t\}\{=\}t, and the model regresses the target velocityv=ξ−x0v=\\xi\-x\_\{0\}\. High\-reward endpoints are reconstructed through the ordinary prediction

vθ\+​\(xt,t,y\)=vθ​\(xt,t,y\),v\_\{\\theta\}^\{\+\}\(x\_\{t\},t,y\)=v\_\{\\theta\}\(x\_\{t\},t,y\),\(2\)while low\-reward endpoints are pushed away through a mirrored prediction around the EMA reference\. With the official defaultβ=1\\beta=1in\(1\+β\)​vold−β​vθ\(1\+\\beta\)v\_\{\\mathrm\{old\}\}\-\\beta v\_\{\\theta\}, this reduces to

vθ−​\(xt,t,y\)=2​vold​\(xt,t,y\)−vθ​\(xt,t,y\)\.v\_\{\\theta\}^\{\-\}\(x\_\{t\},t,y\)=2\\,v\_\{\\mathrm\{old\}\}\(x\_\{t\},t,y\)\-v\_\{\\theta\}\(x\_\{t\},t,y\)\.\(3\)The combined objective over a rollout group is

ℒNFT=‖vθ\+​\(xt,t,y\)−v‖22⏟pull\+‖vθ−​\(xt,t,y\)−v‖22⏟push\.\\mathcal\{L\}\_\{\\mathrm\{NFT\}\}=\\underbrace\{\\left\\\|v\_\{\\theta\}^\{\+\}\(x\_\{t\},t,y\)\-v\\right\\\|\_\{2\}^\{2\}\}\_\{\\text\{pull\}\}\+\\underbrace\{\\left\\\|v\_\{\\theta\}^\{\-\}\(x\_\{t\},t,y\)\-v\\right\\\|\_\{2\}^\{2\}\}\_\{\\text\{push\}\}\.\(4\)This MSE formulation trains quickly without likelihood ratios or SDE rollouts, and the EMA reference invθ−v\_\{\\theta\}^\{\-\}anchors the update and stabilizes post\-training \(ablated in the appendix\)\. Equivalently, the loss can be written in endpoint space asℓ⁡\(x^0​\(v\),x0\)\\ell\(\\hat\{x\}\_\{0\}\(v\),x\_\{0\}\)withx^0​\(v\)=xt−σt​v\\hat\{x\}\_\{0\}\(v\)=x\_\{t\}\-\\sigma\_\{t\}v; our implementation uses this form with a stop\-gradient factor for loss\-order adjustment\. In our repair setting, the pull and push branches will operate on*different*endpoints rather than the samex0x\_\{0\}; only at the pure\-noise stepσt=1\\sigma\_\{t\}=1do the two trajectories share an identical state\.

Online post\-training supervises the model with its own sampled endpoints; when reward concentrates in a narrow mode, this acts as self\-distillation that sharpens it\.ReNFTretains the NFT parameterization but replaces this one\-sided supervision with a matched local comparison \(Section[4](https://arxiv.org/html/2609.00061#S4)\)\.

## 4Methodology

We address the repair of a high\-reward, low\-diversity adapter without adding an external diversity objective or modifying text representations of the generator\.ReNFTinstead constructs candidate endpoints from available routes of the generator and ranks them with the same reward used for post\-training\. Section[4\.1](https://arxiv.org/html/2609.00061#S4.SS1)first explains why an already\-collapsed adapter can be repaired through these internal routes\. Figure[3](https://arxiv.org/html/2609.00061#S3.F3)summarizes the ensuing pipeline: unconditional probes prioritize anti\-hub prompts where the prompt\-independent bias is easiest to expose \(Section[4\.2](https://arxiv.org/html/2609.00061#S4.SS2)\); two policy\-dominated mixed routes generate matched counterfactuals from the same prompt and initial noise \(Section[4\.3](https://arxiv.org/html/2609.00061#S4.SS3)\); and reward ranking assigns pull/push targets, optimized by native\-noise joint and fresh\-noise paired NFT updates \(Section[4\.4](https://arxiv.org/html/2609.00061#S4.SS4)\)\.

### 4\.1Repair View: Post\-Training Reweights an Inherited Distribution

We study a repair setting in which an adapterθH\\theta\_\{H\}already achieves high reward but maps different initial noises under the same prompt to a narrow set of outputs\. The frozen base generator andθH\\theta\_\{H\}share the same pretrained support; reward adaptation changes which parts of that support are easy to reach, but suppressed modes may remain accessible through routes closer to the base model\. Collapse is therefore probability\-mass compression rather than deletion, and repair should be possible from within\.

A coarse mode\-level view makes the feedback loop explicit\. Ifmmindexes a visual mode, reward\-only adaptation can be heuristically written as

pk\+1​\(m∣y\)∝pk​\(m∣y\)​exp⁡\(η​R¯k​\(m,y\)\),p\_\{k\+1\}\(m\\mid y\)\\propto p\_\{k\}\(m\\mid y\)\\exp\\\!\\left\(\\eta\\,\\bar\{R\}\_\{k\}\(m,y\)\\right\),\(5\)whereR¯k​\(m,y\)\\bar\{R\}\_\{k\}\(m,y\)is the average reward of sampled outputs in modemmandη\>0\\eta\>0scales the reward\-induced reweighting\. Equation \([5](https://arxiv.org/html/2609.00061#S4.E5)\) is an interpretation of this feedback, not the exact update law\. Figure[1](https://arxiv.org/html/2609.00061#S1.F1)illustrates the view: per\-prompt modes contract into narrow spikes as the loop reinforces the favored mode, then reopen during repair\. Conditional modes that had lost probability regain output mass, and the unconditional distribution range expands accordingly\.

This feedback also explains why re\-optimizing reward cannot repair collapse: among semantically equivalent samples, reward differences encode preference rather than quality, so the reweighting only sharpens that preference\. Repair instead needs reward gaps that reflect genuine quality differences\.ReNFTachieves this by constructing matched counterfactuals from different internal routes \(Sections[4\.2](https://arxiv.org/html/2609.00061#S4.SS2)–[4\.4](https://arxiv.org/html/2609.00061#S4.SS4)\), making the reward gap between paired endpoints informative rather than a restatement of the model bias\. We validate this reversibility in Section[5](https://arxiv.org/html/2609.00061#S5)\.

### 4\.2Reading Post\-Training Bias from the Unconditional Route

The unconditional route is the empty\-prompt velocityuθ​\(x,t\)=vθ​\(x,t,∅\)u\_\{\\theta\}\(x,t\)=v\_\{\\theta\}\(x,t,\\varnothing\)\. Although it sharesθ\\theta’s updated parameters, it has no separate reward target; its changes therefore expose prompt\-independent side effects of conditional post\-training: recurring structures, styles, or textures the adapter injects even when not requested, rather than directed optimization\. Unconditional samples from a fixed noise grid contract from a diverse base into a narrow hub after post\-training and progressively reopen during repair \(see the appendix for unconditional samples across training steps\)\. We use these tendencies as an internal bias probe, whose diagnostic validity is evaluated rather than assumed\.

We first use this readout to prioritize prompts on which the bias is easiest to expose, as shown in Figure[3](https://arxiv.org/html/2609.00061#S3.F3)a\. Offline, the prompt bank is encoded once by the SigLIP2 text encoder\([Tschannen et al\. 2025](https://arxiv.org/html/2609.00061#bib.bib31)\)and cached\. Online,MMempty\-prompt probes\{uj\}j=1M\\\{u\_\{j\}\\\}\_\{j=1\}^\{M\}are generated from the current policy and encoded by the SigLIP2 image encoder\. Each candidate promptyyis scored by

s⁡\(y\)=1M​∑j=1M⟨etext​\(y\),eimg​\(uj\)⟩,s\(y\)=\\frac\{1\}\{M\}\\sum\_\{j=1\}^\{M\}\\langle e\_\{\\mathrm\{text\}\}\(y\),e\_\{\\mathrm\{img\}\}\(u\_\{j\}\)\\rangle,\(6\)and the bottom\-KKprompts \(lowest mean similarity\) are selected as*anti\-hub prompts*, those farthest from the current unconditional tendency, where the exposed bias is most likely to yield distinguishable proposals\. The frozen encoder serves only prompt prioritization; it never enters the reward, advantage, or repair loss\. The same route also serves constructively in the mixed sampling of Section[4\.3](https://arxiv.org/html/2609.00061#S4.SS3), where it exposes the bias inside conditional trajectories for reward ranking\.

### 4\.3Constructing Internal Counterfactual Proposals

The frozen base route and the unconditional route play complementary roles\. A base step introduces a base\-like vector\-field direction and can propose alternatives that are sampled less often by the post\-trained policy; an unconditional step exposes the prompt\-independent bias identified in Section[4\.2](https://arxiv.org/html/2609.00061#S4.SS2)\. The conditional routeθ\\thetaoccupies most steps by design, limiting deviation from the high\-reward policy\. Crucially, neither route is hard\-coded as a pull or push branch: mixed routing generates nearby alternatives, and the reward decides their roles \(Figure[3](https://arxiv.org/html/2609.00061#S3.F3)b\)\.

![Refer to caption](https://arxiv.org/html/2609.00061v1/figures/figure4a_geneval_ocr_qual.jpg)Figure 4:Qualitative comparison on GenEval and OCR prompts\.Each prompt group shows four samples from the same method under different initial noises\.![Refer to caption](https://arxiv.org/html/2609.00061v1/figures/figure4b_pickscore_qual.jpg)Figure 5:Qualitative comparison on PickScore prompts\.Each prompt group shows four samples from the same method under different initial noises\.For a routing patternPP, letTP,yT\_\{P,y\}map initial noise to a terminal sample\. The final configuration repeats two three\-step blocks across the rollout,

PA=\(bθθ\)⋯,PB=\(θθu\)⋯\.P\_\{A\}=\(b\\theta\\theta\)\\cdots,\\qquad P\_\{B\}=\(\\theta\\theta u\)\\cdots\.\(7\)Both routes diverge from the same initial noiseε\\varepsilon, keeping two thirds of each route conditional and avoiding longb​bbboru​uuublocks that stray too far from the policy \(cf\. stage\-wise analyses\([Jin, Shi, and Gu 2026](https://arxiv.org/html/2609.00061#bib.bib12)\)\)\.

For a selected promptyyand shared initial noiseε\\varepsilon, the matched endpoints arex0A=TPA,y​\(ε\)x\_\{0\}^\{A\}=T\_\{P\_\{A\},y\}\(\\varepsilon\)andx0B=TPB,y​\(ε\)x\_\{0\}^\{B\}=T\_\{P\_\{B\},y\}\(\\varepsilon\)\. Sharing\(y,ε\)\(y,\\varepsilon\)makes routing the main changed factor: two pure\-θ\\thetatrajectories would coincide, while a direct base\-versus\-policy comparison is too one\-sided; the two mixed routes instead perturb the policy in different directions, and either may win for a particular\(y,ε\)\(y,\\varepsilon\)\. Trajectory\-level branching has likewise been exploited for alignment and credit assignment in TMPO and DenseGRPO\([Li et al\. 2026a](https://arxiv.org/html/2609.00061#bib.bib14);[Deng et al\. 2026](https://arxiv.org/html/2609.00061#bib.bib4)\); we instead branch two route mixtures from identical noise to construct a reward\-testable counterfactual pair\.

We rank the endpoints with the original reward \(Figure[3](https://arxiv.org/html/2609.00061#S3.F3)b, right\) and obtain\(x0H,x0L\)\(x\_\{0\}^\{H\},x\_\{0\}^\{L\}\); when rewards differ, the higher\-reward endpoint is always the pull target and the lower the push target, regardless of route\. The matched reward gap

Δ​R​\(y,ε\)=R⁡\(x0A,y\)−R⁡\(x0B,y\)\\Delta R\(y,\\varepsilon\)=R\(x\_\{0\}^\{A\},y\)\-R\(x\_\{0\}^\{B\},y\)\(8\)provides a local preference without claiming global superiority of either route\. For continuous rewards \(PickScore\), ties are rare; an*adaptive flipping*guard \(Figure[3](https://arxiv.org/html/2609.00061#S3.F3)c\) requires at least half of the pull targets to come from routeAA, flipping the lowest\-margin route\-BBwin when the ratio falls below0\.50\.5\. This guard keeps the coverage\-restoringb​θ​θb\\theta\\thetabranch dominant and prevents the hacking\-proneθ​θ​u\\theta\\theta ubranch from re\-hacking the pull distribution\. For rule\-based discrete rewards \(GenEval\), ties are frequent; we break them with a near 50/50 split that slightly favors routeAA, without activating the flipping guard\.

### 4\.4Reward\-Ranked Joint\-and\-Paired Repair

We realize the matched preference with the NFT mirror mechanism from Section[3\.2](https://arxiv.org/html/2609.00061#S3.SS2)\. The loss in v\-space is

ℒrepair=‖vθ−​\(xtL,t,y\)−vL‖22⏟mirrored push\+‖vθ\+​\(xtH,t,y\)−vH‖22⏟ordinary pull\.\\mathcal\{L\}\_\{\\mathrm\{repair\}\}=\\underbrace\{\\left\\\|v\_\{\\theta\}^\{\-\}\(x\_\{t\}^\{L\},t,y\)\-v^\{L\}\\right\\\|\_\{2\}^\{2\}\}\_\{\\text\{mirrored push\}\}\+\\underbrace\{\\left\\\|v\_\{\\theta\}^\{\+\}\(x\_\{t\}^\{H\},t,y\)\-v^\{H\}\\right\\\|\_\{2\}^\{2\}\}\_\{\\text\{ordinary pull\}\}\.\(9\)The mirror referencevθ−v\_\{\\theta\}^\{\-\}uses an EMA\-smoothed copy of the policy with constant decay, since repair starts from a late\-stage checkpoint\.

For each paired timestep, we sample a fresh perturbationξ\\xiand construct

xtH=\(1−σt\)​x0H\+σt​ξ,xtL=\(1−σt\)​x0L\+σt​ξ\.x\_\{t\}^\{H\}=\(1\-\\sigma\_\{t\}\)x\_\{0\}^\{H\}\+\\sigma\_\{t\}\\xi,\\quad x\_\{t\}^\{L\}=\(1\-\\sigma\_\{t\}\)x\_\{0\}^\{L\}\+\\sigma\_\{t\}\\xi\.\(10\)with targetsvH=ξ−x0Hv^\{H\}=\\xi\-x\_\{0\}^\{H\}andvL=ξ−x0Lv^\{L\}=\\xi\-x\_\{0\}^\{L\}\. The pull and push shareξ\\xi, treating corruption as a matched nuisance; samplingξ\\xiindependently ofε\\varepsilonavoids a per\-\(prompt, noise\) fitting shortcut\([Wen et al\. 2024](https://arxiv.org/html/2609.00061#bib.bib34)\)\.

Atσt=1\\sigma\_\{t\}=1, a freshξ\\xiwould break the coupling, so we use the native rollout noiseε\\varepsilonfor a*joint*update \(Figure[3](https://arxiv.org/html/2609.00061#S3.F3)c\), withxtH=xtL=εx\_\{t\}^\{H\}=x\_\{t\}^\{L\}=\\varepsilonand targetsvH=ε−x0Hv^\{H\}=\\varepsilon\-x\_\{0\}^\{H\},vL=ε−x0Lv^\{L\}=\\varepsilon\-x\_\{0\}^\{L\}\. This anchor matters because early denoising decisions strongly influence global structure\([Jin, Shi, and Gu 2026](https://arxiv.org/html/2609.00061#bib.bib12)\)\. Each repair step applies one joint update withε\\varepsilonatσt=1\\sigma\_\{t\}=1, followed by an even number of paired updates with freshξ\\xiatσt<1\\sigma\_\{t\}<1\(see the appendix for ablations on the joint\-noise choice and rollout length\)\. Mixed routes only construct training supervision; inference uses the standard conditional forward pass with no routing overhead\.

## 5Experiments

We evaluateReNFTon PickScore and GenEval: standard reward post\-training produces the collapsed checkpoint that motivates repair \(Section[5\.2](https://arxiv.org/html/2609.00061#S5.SS2)\),ReNFTrecovers diversity while retaining reward both quantitatively and qualitatively \(Sections[5\.2](https://arxiv.org/html/2609.00061#S5.SS2)and[5\.3](https://arxiv.org/html/2609.00061#S5.SS3)\), and controlled ablations identify the responsible design decisions \(Section[5\.4](https://arxiv.org/html/2609.00061#S5.SS4)\)\.

Table 1:Quantitative results on the PickScore test set\.Shaded: in\-domain training reward\. Parenthesized row: reference only, excluded from ranking\.Bold: best;underline: second best\.†See Section[5\.1](https://arxiv.org/html/2609.00061#S5.SS1)\.Table 2:Quantitative results on the GenEval test set\.Shaded: in\-domain training reward\. Parenthesized row: reference only, excluded from ranking\.Bold: best;underline: second best\.†See Section[5\.1](https://arxiv.org/html/2609.00061#S5.SS1)\.### 5\.1Experimental Setup

We use SD3\.5\-M\([Esser et al\. 2024](https://arxiv.org/html/2609.00061#bib.bib5)\)with LoRA adapters and train separate PickScore\([Kirstain et al\. 2023](https://arxiv.org/html/2609.00061#bib.bib13)\)and GenEval\([Ghosh, Hajishirzi, and Schmidt 2023](https://arxiv.org/html/2609.00061#bib.bib8)\)checkpoints\.ReNFTreloads the DiffusionNFT LoRA at step 700 \(PickScore\) or 250 \(GenEval\) and repairs for 50 steps with the default configuration in the appendix\. Model\-based rewards use all 1,024 PickScore test prompts; the rule\-based reward uses all 553 GenEval prompts; diversity uses 100 fixed prompts with 12 fixed\-noise samples each\. NFT, E2PO, andReNFTuse CFG=1; base and GRPO\-style checkpoints use CFG=4\.5, so cross\-CFG comparisons are end\-to\-end method comparisons, not controlled CFG ablations\.

We compare the untrained SD3\.5\-M reference, Flow\-GRPO\([Liu et al\. 2025](https://arxiv.org/html/2609.00061#bib.bib20)\), DiffusionNFT \(NFT\)\([Zheng et al\. 2026a](https://arxiv.org/html/2609.00061#bib.bib40)\), DiverseGRPO\([Liu et al\. 2026a](https://arxiv.org/html/2609.00061#bib.bib18)\), and E2PO\([Hu et al\. 2026](https://arxiv.org/html/2609.00061#bib.bib11)\)\. E2PO is not open\-sourced; we match its official training budget and reuse its paper\-reported reward values \(marked†\), omitting it from the diversity ranking\. Quality metrics are PickScore\-v1, Aesthetic\([Radford et al\. 2021](https://arxiv.org/html/2609.00061#bib.bib25);[Schuhmann et al\. 2022](https://arxiv.org/html/2609.00061#bib.bib26)\), ImageReward\-v1\.0\([Xu et al\. 2023](https://arxiv.org/html/2609.00061#bib.bib36)\), and HPSv2\.1\([Wu et al\. 2023](https://arxiv.org/html/2609.00061#bib.bib35)\)\. Diversity metrics are LPIPS\-Div\([Zhang et al\. 2018](https://arxiv.org/html/2609.00061#bib.bib39)\), DreamSim\-Div\([Fu et al\. 2023](https://arxiv.org/html/2609.00061#bib.bib7)\), and DINOv3\-Div\([Siméoni et al\. 2025](https://arxiv.org/html/2609.00061#bib.bib27)\), each computed as the mean pairwise distance among 12 samples per prompt\. Anti\-hub uses 24 probes and 1,000 candidates; further details are in the appendix\.

### 5\.2Quantitative Results

Reward post\-training produces the repair setting\.Tables[1](https://arxiv.org/html/2609.00061#S5.T1)and[2](https://arxiv.org/html/2609.00061#S5.T2)confirm the reward–diversity tension hypothesized in Section[4\.1](https://arxiv.org/html/2609.00061#S4.SS1): the untrained base \(shown in parentheses as a reference row, excluded from ranking\) has the largest within\-prompt diversity but the weakest task reward, while the continued DiffusionNFT run reaches the highest PickScore and GenEval values at steps 750 and 300, with DreamSim\-Div falling from 0\.242 to 0\.119 and from 0\.339 to 0\.149, respectively\.ReNFTinstead branches from the earlier NFT checkpointsHHat steps 700 and 250 under the same remaining 50\-step budget\. Among the compared post\-trained methods,ReNFTattains the highest diversity in all three metrics under both protocols\.

ReNFT recovers diversity while retaining reward\.Relative to NFT at the protocol endpoint,ReNFTretains 98\.9% of PickScore \(23\.26 vs\. 23\.51\) and 99\.0% of GenEval \(0\.937 vs\. 0\.946\), while improving DreamSim\-Div by 58\.8% \(0\.189 vs\. 0\.119\) and 55\.0% \(0\.231 vs\. 0\.149\), respectively\. The trade\-off is asymmetric: roughly 1% reward cost versus 55–59% diversity gain\.ReNFTranks second on GenEval reward and PickScore ImageReward; all three diversity metrics are best among compared post\-trained methods under both protocols\. This asymmetry reflects how susceptible each reward is to hacking: model\-based scores of PickScore depend heavily on exploiting the model preference, so an anti\-hacking method faces a harder reward landscape, whereas GenEval’s rule\-based scoring rewards correct color, position, and count through segmentation, leaving less room for hacking\-specific artifacts\.

Repair dynamics\.In Figure[2](https://arxiv.org/html/2609.00061#S2.F2)\(a\)–\(d\), under PickScore reward decreases with local oscillations toward the table value, while diversity rises rapidly, peaks around step 35, and decays mildly, remaining above every other post\-trained baseline at step 50\. The rise–peak–decay pattern reflects the dual\-route win ratio:b​θ​θb\\theta\\thetainitially dominatesθ​θ​u\\theta\\theta u, but as the routes converge the ratio oscillates near0\.50\.5; adaptive flipping maintains a minimumb​θ​θb\\theta\\thetapull share, yet the unconditional branch gradually re\-hacks, causing diversity to recede and reward to climb again along a hacking trajectory\. GenEval shows no turning point within 50 steps: its wider acceptable reward range produces more stable curves: reward drops less, diversity rises more, and the dual\-branch balance point is reached later\.

Comparison with diversity\-oriented baselines\.DiverseGRPO preserves more diversity than NFT but at a larger reward cost: 97\.8% of PickScore \(22\.99 vs\. 23\.51\) and 91\.9% of GenEval \(0\.869 vs\. 0\.946\), compared toReNFT’s 98\.9% and 99\.0%\. E2PO’s reused reward enters the ranking with provenance marked†; its interface\-level bias correction is complementary to our internal\-route approach\.

### 5\.3Qualitative Results

Figures[4](https://arxiv.org/html/2609.00061#S4.F4)and[5](https://arxiv.org/html/2609.00061#S4.F5)compare Base, NFT, DiverseGRPO, andReNFTon GenEval\+OCR and PickScore prompts, respectively, with a shared noise set\.

PickScore prompts\.NFT and DiverseGRPO exhibit recognizable hacking tendencies: a recurring color tone, densely packed elements, and a preference for single female subjects\. NFT’s late\-stage outputs respond primarily to prompt changes, becoming insensitive to the initial noise\.ReNFTrestores scene\- and style\-level variation across seeds without reverting to the base: dog statues vary in pose and setting, wedding scenes recover diverse compositions, and the emperor\-and\-robots prompt produces distinct arrangements\.

GenEval and OCR prompts\.NFT collapses counting\-related prompts onto pure white backgrounds, a hacking shortcut that simplifies segmentation and counting\.ReNFTremoves this bias: count, position, and color remain correct while backgrounds and styles vary\. The OCR prompts are out\-of\-distribution for our PickScore\-trained repair: NFT and DiverseGRPO retain some text\-rendering ability from OCR\-adjacent prompts, but hacking distorts style and layout\.ReNFTproduces more legible and varied text, approaching base\-level diversity with higher text fidelity\. These observations confirm the repair\-view hypothesis \(Section[4\.1](https://arxiv.org/html/2609.00061#S4.SS1)\): suppressed modes were compressed, not deleted\. A diagnostic view from unconditional \(CFG=0\) samples is provided in the appendix\.

### 5\.4Ablation Study

The ablations examine two design decisions in the main text: anti\-hub prompt prioritization and internal route construction\. All variants branch from the same checkpointHHunder the same 50\-step budget and checkpoint\-evaluation protocol\.

Anti\-hub prompt prioritization\.The controlled variant replaces anti\-hub prompts with random prompts from the same candidate pool, preserving the prompt count, sampling budget, and update rule\. The two variants track nearly identical reward trajectories, whereas diversity diverges sharply: without anti\-hub selection, diversity rises only briefly and then decays back below its starting level, while the full configuration sustains the recovery\. The PickScore training distribution is itself biased toward the hacking preferences of the reward model, soθ​θ​u\\theta\\theta umay not be substantially worse thanb​θ​θb\\theta\\thetafrom the start; without anti\-hub prompts that expose the bias most distinguishably, there is little diversity\-recovery momentum\.

Internal route construction\.Figure[2](https://arxiv.org/html/2609.00061#S2.F2)\(e,f\) compares\(b​θ,θ​u\)\(b\\theta,\\theta u\),\(b​θ​θ,θ​θ​u\)\(b\\theta\\theta,\\theta\\theta u\), and longer periodic variants under the same budget\. With pattern length 2, the base and unconditional content is too high:θ​u\\theta uconsistently produces low\-reward outputs, the win ratio stays near1\.01\.0, and saturation/contrast anomalies cause reward to collapse while diversity rises from degenerate outputs\. Longer blocks approach the NFT direction\. The three\-step pattern\(b​θ​θ,θ​θ​u\)\(b\\theta\\theta,\\theta\\theta u\)achieves the clearest diversity recovery with the least reward sacrifice\.

Appendix ablations support the default rollout length, joint\-noise choice, and EMA decay; sensitivity to the anti\-hub pool size and flipping threshold remains future work\.

## 6Conclusion

We studied reward\-induced mode collapse as internal probability\-mass reallocation, where post\-training concentrates mass on reward\-favored modes and suppressed alternatives remain accessible through internal routes of the generator\.ReNFToperationalizes this view by prioritizing anti\-hub prompts with unconditional probes, generating matched counterfactuals from the same prompt and noise through two policy\-dominated mixed routes, and assigning pull and push roles through reward ranking with an adaptive flipping guard, all realized through joint\-and\-paired NFT updates\. Across PickScore and GenEval,ReNFTretains 98\.9% and 99\.0% of NFT’s reward while improving DreamSim\-Div by 58\.8% and 55\.0%, confirming that collapsed adapters can be repaired from within without external diversity objectives, text\-encoder modification, or sacrificing the acquired reward\. This internal repair perspective opens a complementary direction to external interventions, extending naturally to other post\-training paradigms and backbones\.

## References

- Black Forest Labs \(2025\)Black Forest Labs\. 2025\.FLUX\.2: Frontier Visual Intelligence\.https://bfl\.ai/blog/flux\-2\.
- Cao et al\. \(2026\)Cao, Q\.; Chen, Y\.; Ma, C\.; and Yang, X\. 2026\.Dynamic Training\-Free Fusion of Subject and Style LoRAs\.https://qinglongcao\.xyz/TVML\-Diffusion\.github\.io/\.Manuscript\.
- Chen et al\. \(2026\)Chen, C\.; Hu, S\.; Zhu, J\.; Wu, M\.; Chen, J\.; Li, Y\.; Huang, N\.; Fang, C\.; Wu, J\.; Chu, X\.; and Li, X\. 2026\.Taming Preference Mode Collapse via Directional Decoupling Alignment in Diffusion Reinforcement Learning\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*\.
- Deng et al\. \(2026\)Deng, H\.; Yan, K\.; Mao, C\.; Wang, X\.; Liu, Y\.; Gao, C\.; and Sang, N\. 2026\.DenseGRPO: From Sparse to Dense Reward for Flow Matching Model Alignment\.In*International Conference on Learning Representations \(ICLR\)*\.
- Esser et al\. \(2024\)Esser, P\.; Kulal, S\.; Blattmann, A\.; Entezari, R\.; Müller, J\.; Saini, H\.; Levi, Y\.; Lorenz, D\.; Sauer, A\.; Boesel, F\.; et al\. 2024\.Scaling Rectified Flow Transformers for High\-Resolution Image Synthesis\.In*International Conference on Machine Learning \(ICML\)*, 12675–12702\.
- Fang et al\. \(2026\)Fang, Z\.; Huang, W\.; Zeng, Y\.; Zhao, Y\.; Chen, S\.; Feng, K\.; Lin, Y\.; Chen, L\.; Chen, Z\.; Cao, S\.; and Zhao, F\. 2026\.Flow\-OPD: On\-Policy Distillation for Flow Matching Models\.ArXiv preprint arXiv:2605\.08063\.
- Fu et al\. \(2023\)Fu, S\.; Tamir, N\.; Sundaram, S\.; Chai, L\.; Zhang, R\.; Dekel, T\.; and Isola, P\. 2023\.DreamSim: Learning New Dimensions of Human Visual Similarity Using Synthetic Data\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 50742–50768\.
- Ghosh, Hajishirzi, and Schmidt \(2023\)Ghosh, D\.; Hajishirzi, H\.; and Schmidt, L\. 2023\.GenEval: An Object\-Focused Framework for Evaluating Text\-to\-Image Alignment\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Go et al\. \(2026\)Go, H\.; Chung, H\.; Truong, P\.; Bhat, G\.; Mi, L\.; An, Z\.; Zhao, Z\.; Narnhofer, D\.; Belongie, S\.; Tombari, F\.; and Schindler, K\. 2026\.Stitched Value Model for Diffusion Alignment\.ArXiv preprint arXiv:2605\.19804\.
- He et al\. \(2025\)He, H\.; Ye, Y\.; Liu, J\.; Liang, J\.; Wang, Z\.; Yuan, Z\.; Wang, X\.; Mao, H\.; Wan, P\.; and Pan, L\. 2025\.GARDO: Reinforcing Diffusion Models without Reward Hacking\.ArXiv preprint arXiv:2512\.24138\.
- Hu et al\. \(2026\)Hu, S\.; Chen, C\.; Zhu, J\.; Wu, J\.; Chu, X\.; and Li, X\. 2026\.E2PO: Embedding\-perturbed Exploration Preference Optimization for Flow Models\.In*International Conference on Machine Learning \(ICML\)*\.
- Jin, Shi, and Gu \(2026\)Jin, C\.; Shi, Q\.; and Gu, Y\. 2026\.Stage\-wise Dynamics of Classifier\-Free Guidance in Diffusion Models\.In*International Conference on Learning Representations \(ICLR\)*\.
- Kirstain et al\. \(2023\)Kirstain, Y\.; Polyak, A\.; Singer, U\.; Matiana, S\.; Penna, J\.; and Levy, O\. 2023\.Pick\-a\-Pic: An Open Dataset of User Preferences for Text\-to\-Image Generation\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 36652–36663\.
- Li et al\. \(2026a\)Li, J\.; Zhu, C\.; Yi, N\.; Bao, Y\.; Sun, L\.; Lv, Q\.; Fang, X\.; Liu, D\.; Li, J\.; He, K\.; Zhou, B\.; and Ma, Z\. 2026a\.TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignment\.ArXiv preprint arXiv:2605\.10983\.
- Li et al\. \(2026b\)Li, Q\.; Yu, J\.; Jiang, K\.; Wei, Y\.; Xing, Z\.; Li, P\.; Chu, R\.; Zhang, S\.; Liu, Y\.; and Wu, Z\. 2026b\.DiffusionOPD: A Unified Perspective of On\-Policy Distillation in Diffusion Models\.ArXiv preprint arXiv:2605\.15055\.
- Li et al\. \(2026c\)Li, R\.; Xu, M\.; Gu, S\.; Qu, L\.; Feng, F\.; Hu, H\.; and Wang, W\. 2026c\.Optimizing Visual Generative Models via Distribution\-wise Rewards\.ArXiv preprint arXiv:2607\.02291\.
- Li et al\. \(2025\)Li, Z\.; Liu, Z\.; Zhang, Q\.; Lin, B\.; Wu, F\.; Yuan, S\.; Yan, Z\.; Ye, Y\.; Yu, W\.; Niu, Y\.; Wang, S\.; Cheng, X\.; and Yuan, L\. 2025\.Uniworld\-V2: Reinforce Image Editing with Diffusion Negative\-aware Finetuning and MLLM Implicit Feedback\.ArXiv preprint arXiv:2510\.16888\.
- Liu et al\. \(2026a\)Liu, H\.; Huang, H\.; Wang, J\.; Liu, C\.; Li, X\.; and Ji, X\. 2026a\.DiverseGRPO: Mitigating Mode Collapse in Image Generation via Diversity\-Aware GRPO\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*\.
- Liu et al\. \(2026b\)Liu, J\.; Li, H\.; Sun, Z\.; Chen, C\.; Bian, Y\.; Wang, B\.; Dong, D\.; Chen, C\.; and Wang, Z\. 2026b\.Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine\-Tuning for Versatile Image Generation\.ArXiv preprint arXiv:2601\.12401\.
- Liu et al\. \(2025\)Liu, J\.; Liu, G\.; Liang, J\.; Li, Y\.; Liu, J\.; Wang, X\.; Wan, P\.; Zhang, D\.; and Ouyang, W\. 2025\.Flow\-GRPO: Training Flow Matching Models via Online RL\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Liu, He, and Li \(2026\)Liu, Z\.; He, X\.; and Li, Y\. 2026\.Advances in GRPO for Generation Models: A Survey\.ArXiv preprint arXiv:2603\.06623\.
- Mao et al\. \(2025\)Mao, W\.; Chen, H\.; Yang, Z\.; and Shou, M\. Z\. 2025\.The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generation\.ArXiv preprint arXiv:2511\.20256\.
- Ping et al\. \(2026\)Ping, B\.; Zhou, X\.; Qi, P\.; Luo, M\.; Bo, L\.; and Pang, T\. 2026\.Flow\-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models\.ArXiv preprint arXiv:2606\.11025\.
- Qin et al\. \(2026\)Qin, Y\.; Wang, L\.; Fei, H\.; Zimmermann, R\.; Bo, L\.; Lu, Q\.; and Wang, C\. 2026\.SOAR: Self\-Correction for Optimal Alignment and Refinement in Diffusion Models\.ArXiv preprint arXiv:2604\.12617\.
- Radford et al\. \(2021\)Radford, A\.; Kim, J\. W\.; Hallacy, C\.; Ramesh, A\.; Goh, G\.; Agarwal, S\.; Sastry, G\.; Askell, A\.; Mishkin, P\.; Clark, J\.; Krueger, G\.; and Sutskever, I\. 2021\.Learning Transferable Visual Models From Natural Language Supervision\.In*International Conference on Machine Learning \(ICML\)*, 8748–8763\.
- Schuhmann et al\. \(2022\)Schuhmann, C\.; Beaumont, R\.; Vencu, R\.; Gordon, C\.; Wightman, R\.; Cherti, M\.; Coombes, T\.; Katta, A\.; Mullis, C\.; Wortsman, M\.; et al\. 2022\.LAION\-5B: An Open Large\-scale Dataset for Training Next Generation Image\-Text Models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Siméoni et al\. \(2025\)Siméoni, O\.; Vo, H\. V\.; Seitzer, M\.; Baldassarre, F\.; Oquab, M\.; Jose, C\.; Khalidov, V\.; Szafraniec, M\.; Yi, S\.; Ramamonjisoa, M\.; Massa, F\.; Haziza, D\.; Wehrstedt, L\.; Wang, J\.; Darcet, T\.; Moutakanni, T\.; Sentana, L\.; Roberts, C\.; Vedaldi, A\.; Tolan, J\.; Brandt, J\.; Couprie, C\.; Mairal, J\.; Jégou, H\.; Labatut, P\.; and Bojanowski, P\. 2025\.DINOv3\.*Transactions on Machine Learning Research \(TMLR\)*\.ArXiv:2508\.10104\.
- Soboleva et al\. \(2025\)Soboleva, V\.; Alanov, A\.; Kuznetsov, A\.; and Sobolev, K\. 2025\.T\-LoRA: Single Image Diffusion Model Customization Without Overfitting\.ArXiv preprint arXiv:2507\.05964\.
- Tan et al\. \(2026\)Tan, X\.; Liu, J\.; Gao, B\.\-B\.; Fan, Y\.; Jiang, X\.; Wang, C\.; Wang, H\.; and Zheng, F\. 2026\.When Policy Entropy Constraint Fails: Preserving Diversity in Flow\-based RLHF via Perceptual Entropy\.ArXiv preprint arXiv:2605\.12112\.
- Tong et al\. \(2026\)Tong, Y\.; Liu, M\.; Zhao, C\.; He, W\.; Zhang, S\.; Zhang, H\.; Zhang, P\.; Liu, J\.; Huang, J\.; Wang, J\.; Jiang, H\.; and Huang, P\. 2026\.Alleviating Sparse Rewards by Modeling Step\-Wise and Long\-Term Sampling Effects in Flow\-Based GRPO\.ArXiv preprint arXiv:2602\.06422\.
- Tschannen et al\. \(2025\)Tschannen, M\.; Gritsenko, A\.; Wang, X\.; Naeem, M\. F\.; Alabdulmohsin, I\.; Parthasarathy, N\.; Evans, T\.; Beyer, L\.; Xia, Y\.; Mustafa, B\.; Hénaff, O\.; Harmsen, J\.; Steiner, A\.; and Zhai, X\. 2025\.SigLIP 2: Multilingual Vision\-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features\.ArXiv preprint arXiv:2502\.14786\.
- Wallace et al\. \(2024\)Wallace, B\.; Dang, M\.; Rafailov, R\.; Zhou, L\.; Lou, A\.; Purushwalkam, S\.; Ermon, S\.; Xiong, C\.; Joty, S\.; and Naik, N\. 2024\.Diffusion Model Alignment Using Direct Preference Optimization\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 8228–8238\.
- Wang et al\. \(2025\)Wang, Y\.; Li, Z\.; Zang, Y\.; Zhou, Y\.; Bu, J\.; Wang, C\.; Lu, Q\.; Jin, C\.; and Wang, J\. 2025\.Pref\-GRPO: Pairwise Preference Reward\-based GRPO for Stable Text\-to\-Image Reinforcement Learning\.ArXiv preprint arXiv:2508\.20751\.
- Wen et al\. \(2024\)Wen, Y\.; Liu, Y\.; Chen, C\.; and Lyu, L\. 2024\.Detecting, Explaining, and Mitigating Memorization in Diffusion Models\.In*International Conference on Learning Representations \(ICLR\)*\.
- Wu et al\. \(2023\)Wu, X\.; Hao, Y\.; Sun, K\.; Chen, Y\.; Zhu, F\.; Zhao, R\.; and Li, H\. 2023\.Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text\-to\-Image Synthesis\.ArXiv preprint arXiv:2306\.09341\.
- Xu et al\. \(2023\)Xu, J\.; Liu, X\.; Wu, Y\.; Tong, Y\.; Li, Q\.; Ding, M\.; Tang, J\.; and Dong, Y\. 2023\.ImageReward: Learning and Evaluating Human Preferences for Text\-to\-Image Generation\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 56730–56748\.
- Xue et al\. \(2026\)Xue, S\.; Ge, C\.; Zhang, S\.; Li, Y\.; and Ma, Z\.\-M\. 2026\.Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models\.In*International Conference on Learning Representations \(ICLR\)*\.
- Yin et al\. \(2026\)Yin, B\.; Hu, X\.; Zhou, X\.; He, Y\.; Jiang, P\.\-T\.; Liao, Y\.; Zhu, J\.; Zhang, J\.; Tai, Y\.; and Yan, S\. 2026\.FeRA: Frequency\-Energy Constrained Routing for Effective Diffusion Adaptation Fine\-Tuning\.https://github\.com/YinBo0927/FeRA\.Manuscript\.
- Zhang et al\. \(2018\)Zhang, R\.; Isola, P\.; Efros, A\. A\.; Shechtman, E\.; and Wang, O\. 2018\.The Unreasonable Effectiveness of Deep Features as a Perceptual Metric\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 586–595\.
- Zheng et al\. \(2026a\)Zheng, K\.; Chen, H\.; Ye, H\.; Wang, H\.; Zhang, Q\.; Jiang, K\.; Su, H\.; Ermon, S\.; Zhu, J\.; and Liu, M\.\-Y\. 2026a\.DiffusionNFT: Online Diffusion Reinforcement with Forward Process\.In*International Conference on Learning Representations \(ICLR\)*\.
- Zheng et al\. \(2026b\)Zheng, K\.; He, G\.; Zhao, M\.; Zhang, J\.; Chen, H\.; Chen, J\.; Lin, C\.\-H\.; Liu, M\.\-Y\.; Zhu, J\.; and Ma, Q\. 2026b\.Causal\-rCM: A Unified Teacher\-Forcing and Self\-Forcing Open Recipe for Autoregressive Diffusion Distillation in Streaming Video Generation and Interactive World Models\.ArXiv preprint arXiv:2606\.25473\.

## Appendix AExperimental Details

Training configuration\.SD3\.5\-M\([Esser et al\. 2024](https://arxiv.org/html/2609.00061#bib.bib5)\)runs train LoRA adapters \(rank 32, alpha 64\) with learning rate10−410^\{\-4\}, 48 unique samples per epoch, and group size 24\. All runs use seed 42, the same seed as prior work\([Hu et al\. 2026](https://arxiv.org/html/2609.00061#bib.bib11)\)\. Repair uses off\-policy training with a constant EMA decay of0\.250\.25,Ni=36N\_\{i\}\{=\}36rollout steps, andNt=7N\_\{t\}\{=\}7training timesteps per update \(one native\-noise joint anchor and six fresh\-noise paired updates\)\. The minimum route\-AApull ratio is0\.50\.5\.

Prompt splits and tuning protocol\.PickScore\([Kirstain et al\. 2023](https://arxiv.org/html/2609.00061#bib.bib13)\)provides 15,471 training prompts and GenEval\([Ghosh, Hajishirzi, and Schmidt 2023](https://arxiv.org/html/2609.00061#bib.bib8)\)provides 33,199; evaluation uses the full held\-out test sets as in the main paper\. Anti\-hub candidate pools are drawn from the training bank, and all hyperparameter selection reported in Section[B](https://arxiv.org/html/2609.00061#A2)was based on evaluation trajectories of saved checkpoints on the training prompts \(not training\-metric logs\), with the held\-out prompts never used for tuning\.

Training budget and checkpoint choice\.The total budgets \(750 steps for PickScore, 300 for GenEval\) follow the official configuration of E2PO\([Hu et al\. 2026](https://arxiv.org/html/2609.00061#bib.bib11)\)\. All trained baselines share these budgets and are trained to the same endpoints\.ReNFTbranches from the NFT checkpoint 50 steps before the endpoint and uses the remaining 50 steps for repair, matching the continued\-training baselines step for step\. The 50\-step budget is motivated by the pull\-ratio analysis in Section[C\.1](https://arxiv.org/html/2609.00061#A3.SS1)\.

Evaluation settings\.On SD3\.5\-M, evaluation uses 512×\\times512 resolution with 40\-step ODE sampling\. NFT\-family checkpoints \(NFT, E2PO, andReNFT\) are evaluated at CFG==1 \(i\.e\., CFG\-free\), following the protocol of DiffusionNFT\([Zheng et al\. 2026a](https://arxiv.org/html/2609.00061#bib.bib40)\)\.

Diversity metrics\.For each held\-out prompt, we generate 12 samples from fixed initial noises\. LPIPS\([Zhang et al\. 2018](https://arxiv.org/html/2609.00061#bib.bib39)\)uses the AlexNet backbone with bicubic resizing to 256×\\times256; DreamSim\([Fu et al\. 2023](https://arxiv.org/html/2609.00061#bib.bib7)\)uses the official pretrained model and cosine distance between its embeddings; DINOv3\([Siméoni et al\. 2025](https://arxiv.org/html/2609.00061#bib.bib27)\)uses the ViT\-B/16 LVD\-1689M CLS embedding with Resize\(256\)–CenterCrop\(224\) preprocessing and cosine distance\. For every metric, we average all\(122\)=66\\binom\{12\}\{2\}=66pairwise distances within a prompt, then average the resulting prompt\-level scores over the 100 held\-out prompts\.

Quality metrics\.PickScore\([Kirstain et al\. 2023](https://arxiv.org/html/2609.00061#bib.bib13)\), Aesthetic\([Radford et al\. 2021](https://arxiv.org/html/2609.00061#bib.bib25);[Schuhmann et al\. 2022](https://arxiv.org/html/2609.00061#bib.bib26)\), ImageReward\([Xu et al\. 2023](https://arxiv.org/html/2609.00061#bib.bib36)\), and HPSv2\([Wu et al\. 2023](https://arxiv.org/html/2609.00061#bib.bib35)\)are computed as in the main paper\.

Algorithm summary\.Algorithm[1](https://arxiv.org/html/2609.00061#alg1)summarizes the repair loop; all symbols follow Sections 4\.2–4\.4 of the main paper, and concrete hyperparameter values are those given above\.

Algorithm 1Repair NFT \(ReNFT\)\.Require:Hacked checkpointθH\\theta\_\{H\}, frozen base routebb, reward modelR⁡\(⋅\)R\(\\cdot\), frozen SigLIP2 encodersetext,eimge\_\{\\mathrm\{text\}\},e\_\{\\mathrm\{img\}\}, training prompt bank𝒴\\mathcal\{Y\}, routing patternsPA,PBP\_\{A\},P\_\{B\}with rollout operator𝒯\\mathcal\{T\}, probe countMM, candidate numberCC, training timestepsNtN\_\{t\}, learning rateλ\\lambda, EMA decayη\\eta, minimum route\-AApull ratioτ\\tau\. Initialize:θ←θH\\theta\\leftarrow\\theta\_\{H\};θold←θH\\theta^\{\\mathrm\{old\}\}\\leftarrow\\theta\_\{H\}; batch bufferℬ←∅\\mathcal\{B\}\\leftarrow\\varnothing\.

1:foreach optimizer stepdo

2:Sample

MMunconditional probes

\{uj\}j=1M\\\{u\_\{j\}\\\}\_\{j=1\}^\{M\}// anti\-hub selection

3:Score

CCcandidate prompts by

s⁡\(y\)←1M​∑j=1M⟨etext​\(y\),eimg​\(uj\)⟩s\(y\)\\leftarrow\\frac\{1\}\{M\}\\sum\_\{j=1\}^\{M\}\\langle e\_\{\\mathrm\{text\}\}\(y\),e\_\{\\mathrm\{img\}\}\(u\_\{j\}\)\\rangle
4:Select anti\-hub set:

𝒴ah←BottomC,y∈𝒴s​\(y\)\\mathcal\{Y\}\_\{\\mathrm\{ah\}\}\\leftarrow\\operatorname\*\{Bottom\}\_\{C,\\,y\\in\\mathcal\{Y\}\}s\(y\)Eq\. \(6\)

5:for

y∈𝒴ahy\\in\\mathcal\{Y\}\_\{\\mathrm\{ah\}\}do

6:Draw

ε∼𝒩⁡\(0,I\)\\varepsilon\\sim\\mathcal\{N\}\(0,I\)// matched rollouts

7:Compute matched endpoints

x0A←𝒯PA,y​\(ε\)x\_\{0\}^\{A\}\\leftarrow\\mathcal\{T\}\_\{P\_\{A\},y\}\(\\varepsilon\),

x0B←𝒯PB,y​\(ε\)x\_\{0\}^\{B\}\\leftarrow\\mathcal\{T\}\_\{P\_\{B\},y\}\(\\varepsilon\)
8:Rank by reward:

Hy←arg⁡maxk∈\{A,B\}⁡R⁡\(x0k,y\)H\_\{y\}\\leftarrow\\arg\\max\_\{k\\in\\\{A,B\\\}\}R\(x\_\{0\}^\{k\},y\);

Ly←arg⁡mink∈\{A,B\}⁡R⁡\(x0k,y\)L\_\{y\}\\leftarrow\\arg\\min\_\{k\\in\\\{A,B\\\}\}R\(x\_\{0\}^\{k\},y\)Eq\. \(8\)

9:Collect

\(y,ε,x0Hy,x0Ly\)\(y,\\varepsilon,x\_\{0\}^\{H\_\{y\}\},x\_\{0\}^\{L\_\{y\}\}\)into

ℬ\\mathcal\{B\}
10:endfor

11:

ℬ←GuardFlip⁡\(ℬ,τ\)\\mathcal\{B\}\\leftarrow\\operatorname\{GuardFlip\}\(\\mathcal\{B\},\\tau\)// minimum route\-AApull share

12:for

\(y,ε,x0H,x0L\)∈ℬ\(y,\\varepsilon,x\_\{0\}^\{H\},x\_\{0\}^\{L\}\)\\in\\mathcal\{B\}do

13:for

t∈\{1,…,Nt\}t\\in\\\{1,\\ldots,N\_\{t\}\\\}do

14:

ξ←ε\\xi\\leftarrow\\varepsilonif

t=1t\{=\}1, else

ξ∼𝒩⁡\(0,I\)\\xi\\sim\\mathcal\{N\}\(0,I\)// self / fresh noise

15:Re\-noise high endpoint:

xtH←\(1−σt\)​x0H\+σt​ξx\_\{t\}^\{H\}\\leftarrow\(1\{\-\}\\sigma\_\{t\}\)x\_\{0\}^\{H\}\{\+\}\\sigma\_\{t\}\\xi
16:Re\-noise low endpoint:

xtL←\(1−σt\)​x0L\+σt​ξx\_\{t\}^\{L\}\\leftarrow\(1\{\-\}\\sigma\_\{t\}\)x\_\{0\}^\{L\}\{\+\}\\sigma\_\{t\}\\xi
17:Targets:

vH←ξ−x0Hv^\{H\}\\leftarrow\\xi\{\-\}x\_\{0\}^\{H\};

vL←ξ−x0Lv^\{L\}\\leftarrow\\xi\{\-\}x\_\{0\}^\{L\}Eq\. \(10\)

18:Implicit positive velocity:

vθ\+←vθ​\(xtH,t,y\)v\_\{\\theta\}^\{\+\}\\leftarrow v\_\{\\theta\}\(x\_\{t\}^\{H\},t,y\)Eq\. \(2\)

19:Implicit negative velocity:

vθ−←2​vold​\(xtL,t,y\)−vθ​\(xtL,t,y\)v\_\{\\theta\}^\{\-\}\\leftarrow 2\\,v\_\{\\mathrm\{old\}\}\(x\_\{t\}^\{L\},t,y\)\-v\_\{\\theta\}\(x\_\{t\}^\{L\},t,y\)Eq\. \(3\)

20:Positive loss:

ℒ\+←‖vθ\+−vH‖22\\mathcal\{L\}^\{\+\}\\leftarrow\\\|v\_\{\\theta\}^\{\+\}\-v^\{H\}\\\|\_\{2\}^\{2\}
21:Negative loss:

ℒ−←‖vθ−−vL‖22\\mathcal\{L\}^\{\-\}\\leftarrow\\\|v\_\{\\theta\}^\{\-\}\-v^\{L\}\\\|\_\{2\}^\{2\}
22:

θ←θ−λ​∇θ\(ℒ\+\+ℒ−\)\\theta\\leftarrow\\theta\-\\lambda\\nabla\_\{\\theta\}\(\\mathcal\{L\}^\{\+\}\{\+\}\\mathcal\{L\}^\{\-\}\)Eq\. \(9\)

23:endfor

24:endfor

25:

θold←η​θold\+\(1−η\)​θ\\theta^\{\\mathrm\{old\}\}\\leftarrow\\eta\\,\\theta^\{\\mathrm\{old\}\}\+\(1\{\-\}\\eta\)\\,\\theta// EMA update

26:endfor

Output:vθv\_\{\\theta\}

## Appendix BAdditional Ablations

All ablations in this section use the SD3\.5\-M backbone under the PickScore protocol, branching from the same hacked checkpointHHas the main experiments\.

Figure 6:Ablation of Anti\-Hub Candidate Pool Size on SD3\.5\-M \(PickScore\)\.The green trajectory \(1,000 candidates\) is identical to the default configuration in all other ablation figures\.Figure 7:Ablation of EMA Decay on SD3\.5\-M \(PickScore\)\.Larger decays preserve reward but barely recover diversity; smaller decays recover more diversity at larger reward cost, with on\-policy training \(decay00\) reaching the highest diversity but the lowest reward\. The0\.250\.25curve is Ours\.Figure 8:Ablation of Rollout Length and Joint\-Step Self\-Noise on SD3\.5\-M \(PickScore\)\.The 36\-step curve is Ours; all variants branch from the same checkpointHH\.### B\.1Ablation of Anti\-Hub Selection and Pool Size

Anti\-hub selection uses 24 unconditional probe generations to score each of 1,000 candidate prompts with SigLIP2\([Tschannen et al\. 2025](https://arxiv.org/html/2609.00061#bib.bib31)\)text–image embeddings, selecting the lowest\-mean\-similarity prompts for repair\. The on/off ablation in the main paper confirms that replacing anti\-hub\-selected prompts with random ones collapses diversity recovery while reward tracks closely, isolating prompt prioritization from prompt coverage\.

#### Candidate pool size\.

Figure[6](https://arxiv.org/html/2609.00061#A2.F6)varies the candidate pool size\{500,1000,2000,3000\}\\\{500,1000,2000,3000\\\}under the same 50\-step repair budget\. Pools of 1,000 or more candidates yield nearly identical reward and diversity trajectories, indicating that coverage saturates quickly; the smallest pool \(500\) retains reward best but recovers noticeably less diversity due to insufficient prompt coverage\. The default 1,000 candidates are sufficient\.

### B\.2Ablation of EMA Decay

We compare constant decays\{0,0\.1,0\.25,0\.5\}\\\{0,0\.1,0\.25,0\.5\\\}under the same hacked checkpoint, 50\-step repair budget, and checkpoint\-evaluation protocol, where00corresponds to on\-policy training without the off\-policy reference\. Large decay \(0\.50\.5\) slows the reference so much that diversity barely improves while reward instead rises slightly, reproducing an NFT\-like hacking direction\. Smaller decays move along the trade\-off in the opposite direction: decay0\.10\.1recovers substantially more diversity at a visible reward cost, and fully on\-policy training \(decay00\) reaches the highest diversity overall but the lowest reward, showing that without an off\-policy reference the mirrored push drifts too far from the current policy\. Decay0\.250\.25gives the best reward–diversity trade\-off and is used in all main experiments\.

### B\.3Ablation of Rollout Length and Joint\-Step Self\-Noise

Rollout length\.Figure[8](https://arxiv.org/html/2609.00061#A2.F8)varies the rollout length\. With 12 steps, reward drops quickly but diversity rises the highest, indicating that a short, coarse rollout leaves samples under\-converged toward the high\-reward mode\. The 24\- and 36\-step variants are close on both metrics, with the 36\-step rollout ending marginally higher on reward; since repair needs only about 50 optimizer steps, the extra sampling overhead is modest\. We use 36 steps in the main configuration\.

Self\-noise at the joint step\.At the joint \(first\) training timestep, ReNFT reuses the initial noise of the rolloutε\\varepsilonrather than drawing fresh noise\. Replacing this self\-noise with fresh random noise at the joint step keeps reward higher but markedly weakens diversity recovery, consistent with aggravated hacking: fresh noise decouples the pull and push endpoints from the shared counterfactual trajectory\. We retain self\-noise atσt=1\\sigma\_\{t\}\{=\}1\. Loading self\-sampled noise at every training timestep was unstable and diverged, so we exclude that variant from the controlled comparison\.

### B\.4Ablation of Route Position and Contiguity

The main paper uses fixed route patterns in which the base stepbboccupies the first position of each block and the unconditional stepuuoccupies the last\. This section explains the design rationale\.

#### Whybbat the block start\.

Early denoising steps have the strongest influence on global structure\([Jin, Shi, and Gu 2026](https://arxiv.org/html/2609.00061#bib.bib12)\): the vector field at high noise determines layout and composition before local details\. Placingbbat the block start injects a base\-like direction where it can most effectively steer toward suppressed structural alternatives\. A base step at the block end, by contrast, would leave no subsequentθ\\thetastep to correct base\-quality artifacts \(blur, incomplete objects\), which the reward ranking should filter out rather than recover\.

#### Whyuuat the block end\.

The unconditional route exposes prompt\-independent bias \(recurring textures, palettes, stylistic artifacts\)\. Placinguuat the block end lets it act on a nearly\-formed image where this bias is observable\. An earlieruuwould steer toward the unconditional hacking mode before the conditional route establishes prompt\-specific structure, yielding a pull ratio locked near1\.01\.0; the pattern\-length\-2 ablation in the main paper shows this leads to degenerate outputs\.

#### Why no consecutiveb​bbboru​uuublocks\.

The core principle is thatbbanduuserve to inject diversity or expose bias, but they must not dominate the trajectory\. Consecutive base steps \(b​bbb\) push the sampling manifold toward the base distribution for an extended interval, effectively reverting to base\-like sampling and losing the acquired reward\. Consecutive unconditional steps \(u​uuu\) amplify the prompt\-independent bias for an extended interval, strengthening the very hacking direction we aim to suppress\. In both cases the trajectory deviates too far from the current policy, destabilizing training and degrading both reward and diversity\.

#### Toward a minimal repeating pattern\.

The above considerations converge on a simple design: placebbat the start anduuat the end, avoid consecutiveb​bbboru​uuublocks, and keep the majority of steps on the conditional routeθ\\theta\. The most decoupled realization is to define the shortest pattern that satisfies these constraints and repeat it uniformly across the inference timesteps, which is exactly what the default\(b​θ​θ,θ​θ​u\)\(b\\theta\\theta,\\theta\\theta u\)configuration does\. A systematic ablation over alternative positions \(e\.g\.,θ​θ​b\\theta\\theta b,θ​u​θ\\theta u\\theta\) and consecutive blocks \(e\.g\.,b​b​θ​θbb\\theta\\theta\) would further validate these choices and is left to future work\.

## Appendix CRepair Dynamics and Diagnostics

### C\.1Dual\-Route Pull\-Ratio Dynamics

Figure 9:Dual\-Route Pull Ratios during SD3\.5\-M PickScore Repair\.\(a\) The raw route\-AA\(b​θ​θb\\theta\\theta, base\-probing\) pull ratio, which equals routeAA’s win rate in the per\-pair reward ranking before the adaptive flipping guard; \(b\) the guarded ratio after the guard enforces the≥0\.5\\geq 0\.5per\-group floor\. The vertical dotted line marks the reported 50\-step budget\.Figure[9](https://arxiv.org/html/2609.00061#A3.F9)reports the dual\-route pull ratios over 100 repair steps, extending beyond the reported 50\-step budget\. The raw route\-AApull ratio \(routeAA’s win rate before any flipping\) starts at0\.840\.84, drops below0\.50\.5around step2020, and settles in the0\.320\.32–0\.420\.42band from step5050onward, ending at0\.400\.40\. This trajectory matches the diversity peak near step3535\(Figure 2 of the main paper\): as repair progresses, the unconditional route increasingly wins the ranking, i\.e\., it gradually re\-hacks the reward model\. The guarded ratio holds within0\.520\.52–0\.620\.62from step1010onward and never breaches the0\.50\.5floor, with the gap to the raw curve widening as the guard flips more low\-margin pairs\. The same pull\-mass reasoning beyond step5050\(continued repair draws pull mass from guard\-flipped, lower\-margin pairs while diversity has already passed its peak\) directly motivates terminating repair at5050steps\.

### C\.2Unconditional Sampling Analysis

![Refer to caption](https://arxiv.org/html/2609.00061v1/figure8_uncond_display.png)Figure 10:Unconditional Sample Display across NFT Training and ReNFT Repair on SD3\.5\-M\.Empty\-prompt \(CFG=0\) samples from a fixed noise grid\. The base generator branches into PickScore \(top\) and GenEval \(bottom\) repair trajectories, shown at the hacked checkpointHHand repair steps 20 and 40\.Figure[10](https://arxiv.org/html/2609.00061#A3.F10)tracks the same fixed noise indices across training and repair on SD3\.5\-M\. The base generator spans unrelated subjects, scenes, styles, and palettes \(Figure 1 of the main paper\), whereas checkpointHHmaps the grid to protocol\-specific hubs\. Under PickScore, the nine samples concentrate on warm\-toned, highly detailed female portraits with similar framing and backgrounds; under GenEval, they concentrate on isolated, full\-body figures against plain backgrounds\. Thus the two rewards induce different visual signatures, but both sharply reduce unconditional diversity\. Repair progressively reopens the grid: step2020broadens appearance and context, and step4040produces more varied subjects, compositions, backgrounds, and palettes under the same noise indices\. The repaired grids remain between the narrowHHhubs and the wider base range rather than simply reverting to Base\. Because the same noise indices are shared across stages, each repaired sample partially retains the layout, style, and palette of the corresponding base sample \(e\.g\., a blue background in Base tends to stay blue throughHHand repair\), and the correspondence strengthens as diversity recovers: the more varied grids at step4040align more closely with Base than the collapsed grids atHH\. This same\-noise contraction\-and\-reopening pattern supports the hub\-reading use of the unconditional route \(Section 4\.2 of the main paper\) and the suppression\-not\-deletion hypothesis \(Section 4\.1 of the main paper\)\.

![Refer to caption](https://arxiv.org/html/2609.00061v1/figure12_flux_uncond.png)Figure 11:Unconditional Sample Display on FLUX\.2\-klein\-base\.Empty\-prompt samples from a fixed noise grid for the base model, the NFT checkpoint, andReNFT\(4B backbone on top, 9B at the bottom\)\.Figure[11](https://arxiv.org/html/2609.00061#A3.F11)repeats the diagnostic on the two FLUX\.2\-klein\-base backbones\. The unconditional distribution again contracts from a diverse base onto a narrow hub \(both backbones converge onto recurring portrait scenes after NFT\) and reopens under repair with varied subjects, styles, and rendering modes, confirming that the contraction\-and\-reopening pattern, and hence the hub\-reading use of the unconditional route, is not specific to SD3\.5\-M\.

## Appendix DExtended Comparisons

PickScore Test SetGenEval Test SetMethodPickScore↑\\uparrowAesthetic↑\\uparrowImageReward↑\\uparrowHPSv2↑\\uparrowLPIPS\-Div↑\\uparrowDreamSim\-Div↑\\uparrowDINO\-Div↑\\uparrowGenEval↑\\uparrowLPIPS\-Div↑\\uparrowDreamSim\-Div↑\\uparrowDINO\-Div↑\\uparrowSD3\.5\-M\(21\.73\)\(6\.026\)\(1\.06\)\(0\.295\)\(0\.599\)\(0\.242\)\(0\.251\)\(0\.654\)\(0\.687\)\(0\.339\)\(0\.379\)\+ DiffusionNFT23\.516\.5921\.440\.3270\.4300\.1190\.1120\.9460\.2920\.1490\.186\+ E2PO \(reproduced\)23\.176\.3121\.340\.3270\.4980\.1660\.1590\.9210\.4630\.2040\.258\+Ours23\.266\.3441\.410\.3230\.5650\.1890\.1820\.9370\.4960\.2310\.295

Table 3:Comparison with Locally Reproduced E2PO on SD3\.5\-M \(PickScore and GenEval Test Sets\)\.All metrics are computed under the same evaluation protocol\. Unlike the main paper, which reuses E2PO’s paper\-reported reward values, this table uses our local reproduction throughout\.Shaded: in\-domain training reward\.Table 4:Quantitative Results on Two FLUX\.2\-klein\-base Backbones \(PickScore Test Set\)\.Shaded: in\-domain training reward\. Parenthesized rows: reference only, excluded from comparison\.### D\.1Full Comparison with E2PO

E2PO\([Hu et al\. 2026](https://arxiv.org/html/2609.00061#bib.bib11)\)is not open\-sourced, so its published checkpoint and evaluation code are unavailable for direct comparison\. The main paper therefore reuses the paper\-reported reward values of E2PO and omits diversity metrics\. We have since reproduced E2PO locally, matching its official training budget and following its published configuration as closely as possible; Table[3](https://arxiv.org/html/2609.00061#A4.T3)reports the full comparison using these reproduced values throughout\. The reproduced reward \(23\.17 PickScore, 0\.921 GenEval\) is slightly lower than the paper\-reported values \(23\.38, 0\.932\) used in the main paper, likely due to implementation differences\. The main paper conservatively retains these stronger paper\-reported values; Table[3](https://arxiv.org/html/2609.00061#A4.T3)instead evaluates all methods under one protocol\.

Compared to the reproduced E2PO,ReNFTachieves higher diversity on all six diversity metrics across both protocols while also retaining more reward \(98\.9% vs\. 98\.6% PickScore, 99\.0% vs\. 97\.4% GenEval relative to the NFT endpoint\)\. The interface\-level approach does recover meaningful diversity, but at a larger reward cost and to a lesser extent than our internal\-route repair\.

### D\.2Cross\-Backbone Generalization

Table[4](https://arxiv.org/html/2609.00061#A4.T4)extends the evaluation to two FLUX\.2\-klein\-base\([Black Forest Labs 2025](https://arxiv.org/html/2609.00061#bib.bib1)\)backbones \(4B and 9B\) on the PickScore test set\. The NFT adapter of each backbone is trained for 200 steps and repaired for another 50 steps, with the same hyperparameters as on SD3\.5\-M except for the larger LoRA \(rank 64, alpha 128\); evaluation uses 20\-step sampling, with the base model at CFG 4 and the NFT andReNFTcheckpoints CFG\-free\. The collapse pattern reproduces on both backbones: DiffusionNFT raises PickScore by 2\.12 and 1\.31 points over the base model while DreamSim\-Div falls by 48% and 44%, respectively, matching the reward–diversity tension on SD3\.5\-M\.ReNFTtransfers without retuning: it retains 98\.9% \(4B\) and 98\.7% \(9B\) of NFT’s reward while improving DreamSim\-Div by 33% and 35%, with consistent gains on LPIPS\-Div and DINO\-Div as well\. The repair mechanism therefore generalizes beyond SD3\.5\-M: across a different backbone, LoRA capacity, training budget, and evaluation protocol, the same internal routes exist and the same 50\-step budget recalibrates them without retuning\.

## Appendix EExtended Qualitative Results

Figures[12](https://arxiv.org/html/2609.00061#A5.F12)and[13](https://arxiv.org/html/2609.00061#A5.F13)compare Base, NFT, andReNFTon five PickScore prompts per backbone, with four samples per method under different initial noises\. We discuss three representative prompts per backbone below; the shared*axolotl*prompt enables a cross\-backbone comparison at the end\.

FLUX\.2\-klein\-base\-4B\.On the*axolotl\-in\-the\-style\-of\-Minecraft*prompt, all three methods adopt a side\-view subject structure\. Base has clean, simple backgrounds; NFT fixes the structure across samples with visible noise artifacts;ReNFTmaintains the side\-view structure but with fuller backgrounds and textures than Base, without NFT’s artifacts\. On the*storefront\-with\-AAAI\-2027*prompt,ReNFTachieves higher text\-spelling accuracy than both Base and NFT, with diversity approaching Base and far exceeding NFT\. On the*squirrel\-gives\-an\-apple\-to\-a\-bird*prompt, Base shows high diversity in composition and style, though some samples have implausible object relationships;ReNFTproduces more structurally coherent scenes with style diversity markedly higher than NFT\.

FLUX\.2\-klein\-base\-9B\.On the same*axolotl\-in\-the\-style\-of\-Minecraft*prompt, the 9B base adopts a*front\-view*subject structure, distinct from the side view of the 4B base, and NFT correspondingly collapses to a different hub, withReNFTreopening within the 9B’s own range\. On the*fantasy\-pastel\-Wes\-Anderson\-pineapple\-character*prompt, Base already produces prompt\-relevant pineapple characters; both NFT andReNFTconverge on pineapple\-head with humanoid\-body forms, likely reflecting PickScore’s preference for human\-like subjects\.ReNFThas cleaner and more varied backgrounds compared to NFT’s dense, ornate settings\. On the*sheep\-holding\-a\-sign\-that\-says\-play\-chess*prompt, Base shows high diversity; NFT fixes to a narrow mode with only minor structural differences across samples;ReNFTmaintains multiple compositional styles and subject structures\.

Cross\-backbone axolotl comparison\.Under the same prompt, the two backbones’ Base, NFT, andReNFTsamples remain visually distinct: 4B and 9B each concentrate on a different high\-detail NFT hub and reopen toward different backbone\-specific ranges\. This provides qualitative evidence consistent with Section 4\.1 of the main paper: post\-training reweights the inherited distribution of each model rather than introducing a shared new mode \(Figure 1 of the main paper\)\. The backbone\-specific reopening is also consistent with the suppression\-not\-deletion hypothesis\.

Common NFT artifacts\.Beyond the per\-prompt structural collapse described above, the NFT columns in both grids share recurring visual symptoms: a warm, oversaturated color palette; densely textured or ornate backgrounds; and fine noise artifacts that give the images an over\-processed appearance\. These symptoms are consistent across both FLUX\.2\-klein\-base backbones and match the hacking artifacts observed in the SD3\.5\-M unconditional display \(Section 3\.2\), suggesting a shared reward\-hacking signature rather than a backbone\-specific artifact\. These visual artifacts \(over\-processed textures, unnatural detail, and palette fixation\) are not genuine quality improvements but shortcuts that overfit the preferences of the reward model, exploiting evaluation blind spots rather than producing subjectively higher\-quality images\.

Across the illustrated prompts,ReNFTis visibly more varied than NFT in composition, color, and rendering style, while Table[4](https://arxiv.org/html/2609.00061#A4.T4)shows that this recovery retains most of NFT’s reward\.

![Refer to caption](https://arxiv.org/html/2609.00061v1/figures/figure10_flux4b_cond.jpg)Figure 12:Qualitative Comparison on FLUX\.2\-klein\-base\-4B\.Each row is one PickScore prompt; each cell shows four samples from the same method under different initial noises\.![Refer to caption](https://arxiv.org/html/2609.00061v1/figures/figure11_flux9b_cond.jpg)Figure 13:Qualitative Comparison on FLUX\.2\-klein\-base\-9B\.Same layout as Figure[12](https://arxiv.org/html/2609.00061#A5.F12)\.