CoRe: Co-Evolving Reward Models for Mitigating Latent Reward Hacking in Video Diffusion Models
Summary
CoRe introduces a co-evolving latent reward framework that continually refits the reward model on the generator's current samples while anchoring to real-video preferences, preventing latent reward hacking and quality collapse in video diffusion models like Wan2.1-T2V-1.3B.
View Cached Full Text
Cached at: 09/30/26, 09:42 AM
# CoRe: Co-Evolving Reward Models for Mitigating Latent Reward Hacking in Video Diffusion Models Source: [https://arxiv.org/html/2609.36245](https://arxiv.org/html/2609.36245) Yujin HanAffiliation:The University of Hong KongEmail:[dzou@cs\.hku\.hk](mailto:[email protected])Feng WangAffiliation:Johns Hopkins UniversityJameson DongAffiliation:Johns Hopkins UniversityHins HuAffiliation:Cornell UniversityDifan Zou††thanks:Corresponding author\.Affiliation:The University of Hong Kong ###### Abstract Latent reward models \(LRMs\) enable efficient alignment of video diffusion models by scoring intermediate states directly in latent space\. However, we find that optimizing against a fixed latent reward rapidly leads to*latent reward hacking*: the predicted reward stays high while perceptual and motion quality deteriorate\. Our analysis identifies distributional escape as the central cause: within a few hundred updates, the generator moves beyond the reward model’s training support, where its scores no longer reflect video quality\. Based on this insight, we introduceCoRe, a co\-evolving reward framework that treats latent\-space alignment as a dynamic interaction between the generator and the reward model\. Rather than optimizing against a stationary proxy,CoRecontinually refits the reward model on the generator’s current samples while anchoring it to real\-video preferences, so the generator cannot gain reward by drifting away from the data\. On Wan2\.1\-T2V\-1\.3B, experiments show thatCoReconsistently improves generation quality over both the pretrained model and prior alignment methods, while avoiding the quality collapse of fixed\-reward optimization\. ## 1Introduction Reinforcement Learning \(RL\) commonly follows a two\-stage recipe: a reward model \(RM\) is first trained on human preference data, after which the generator is optimized against the learned reward\([Christiano et al\., 2017](https://arxiv.org/html/2609.36245#bib.bib1);[Ouyang et al\., 2022](https://arxiv.org/html/2609.36245#bib.bib2)\), either through direct reward backpropagation\([Xu et al\., 2023](https://arxiv.org/html/2609.36245#bib.bib3);[Clark et al\., 2024](https://arxiv.org/html/2609.36245#bib.bib4);[Prabhudesai et al\., 2023](https://arxiv.org/html/2609.36245#bib.bib5)\)or policy\-gradient\-style objectives\([Black et al\., 2024](https://arxiv.org/html/2609.36245#bib.bib7);[Fan et al\., 2023](https://arxiv.org/html/2609.36245#bib.bib8);[Liu et al\., 2026a](https://arxiv.org/html/2609.36245#bib.bib9);[Xue et al\., 2025](https://arxiv.org/html/2609.36245#bib.bib10)\)\. Repeatedly decoding video latents into pixel space is computationally expensive; to avoid this cost, recent work learns Latent Reward Models \(LRMs\) that operate directly on the generator’s latents, score intermediate states without decoding, and provide feedback throughout the denoising trajectory\([Zhang et al\., 2026](https://arxiv.org/html/2609.36245#bib.bib11);[Ding et al\., 2025](https://arxiv.org/html/2609.36245#bib.bib12);[Mi et al\., 2026](https://arxiv.org/html/2609.36245#bib.bib13)\)\. However, this efficiency does not come for free\. Optimizing a generator against a learned reward is well known to induce reward hacking, where the reward keeps increasing while the actual sample quality degrades\([Gao et al\., 2023](https://arxiv.org/html/2609.36245#bib.bib14);[Zhang et al\., 2024](https://arxiv.org/html/2609.36245#bib.bib15)\)\. The problem becomes particularly acute for LRMs: in a fixed setting, the reward model hacks quickly, resulting in video collapse \(Fig\.[1](https://arxiv.org/html/2609.36245#S1.F1)\)\. Existing latent reward methods mainly rely on regularization to suppress reward hacking\([Zhang et al\., 2026](https://arxiv.org/html/2609.36245#bib.bib11);[Mi et al\., 2026](https://arxiv.org/html/2609.36245#bib.bib13)\)\. This delays it but does not prevent it\([Fan et al\., 2023](https://arxiv.org/html/2609.36245#bib.bib8);[Xu et al\., 2023](https://arxiv.org/html/2609.36245#bib.bib3);[Mi et al\., 2026](https://arxiv.org/html/2609.36245#bib.bib13)\)\. We refer to this phenomenon as*latent reward hacking*\. As in conventional reward over\-optimization\([Gao et al\., 2023](https://arxiv.org/html/2609.36245#bib.bib14)\), its root cause is distribution shift: the generator drifts away from the data on which the reward model was trained, where the reward signals become unreliable\([Ackermann et al\., 2025](https://arxiv.org/html/2609.36245#bib.bib16)\)\. What distinguishes the latent setting is how fast and how completely this shift occurs\. Three properties of the latent interface contribute to this: \(i\) it is*continuous*, so small changes to individual latent channels can move the reward, unlike discrete token edits in language models; \(ii\) it is*unshielded*, since scoringxtx\_\{t\}directly removes the VAE decoder that would otherwise separate the generator’s representation from the reward input; and \(iii\) the gradient path is*short*: reward gradients reach the generator through a single denoising step and, with a shared backbone, through the very features the reward reads\. The failure is also hard to detect, because it takes place in noisy intermediate latents that are never decoded or inspected\. In Sec\.[4](https://arxiv.org/html/2609.36245#S4), the generator leaves the LRM support within a few hundred updates and becomes almost linearly separable from real data \(See Appendix Fig\.[7](https://arxiv.org/html/2609.36245#A1.F7)\)\. Figure 1:Latent reward hacking\.We fine\-tune Wan2\.1\-T2V\-1\.3B against a fixed latent reward model\.*Top:*frames generated, labeled with the image quality\. As the reward rises from0\.400\.40to0\.860\.86and stays well above its initial value, the video degrades and loses detail\.*Bottom:*mean latent reward and imaging quality over training\. The reward rises while imaging quality declines\.Prior work has begun to address latent reward hacking from two directions\. The first characterizes the phenomenon:[Gao et al\. \(2023\)](https://arxiv.org/html/2609.36245#bib.bib14)establishes scaling laws for reward\-model over\-optimization in RLHF, TDPO\-R\([Zhang et al\., 2024](https://arxiv.org/html/2609.36245#bib.bib15)\)shows that sustained reward optimization degrades diffusion models through temporal inductive and primacy biases, and RSA\-FT\([Kim and Sim, 2026](https://arxiv.org/html/2609.36245#bib.bib17)\)attributes reward hacking to sharp, non\-robust reward gradients\. The second mitigates it, either by making the reward model harder to exploit, through ensembles\([Coste et al\., 2024](https://arxiv.org/html/2609.36245#bib.bib18)\)or information bottlenecks\([Miao et al\., 2024](https://arxiv.org/html/2609.36245#bib.bib19)\), or by restricting how far the policy is optimized\([Moskovitz et al\., 2024](https://arxiv.org/html/2609.36245#bib.bib20)\)\. These methods, however, treat the reward model as fixed\. Methods that retrain the reward model as the policy changes do exist for language models\([Ackermann et al\., 2025](https://arxiv.org/html/2609.36245#bib.bib16);[Wolf et al\., 2025](https://arxiv.org/html/2609.36245#bib.bib21)\), but they rely on new preference labels or multiple rounds of retraining, which cannot keep pace with the escape in latent space, where it takes only a few hundred updates\. We therefore argue that the core issue of latent reward hacking is not that the reward model is inaccurate, but that it receives no supervision in the out\-of\-distribution regions that optimization itself creates\. This suggests a simple principle: the reward model should evolve with the generator \(Fig\.[2](https://arxiv.org/html/2609.36245#S1.F2)\)\. Based on this observation, we formulate latent\-space RL as a*co\-evolving reward process*and introduceCoRe, a latent reward framework that adapts as the video generator changes\. Rather than treating the LRM as a fixed objective,CoRecontinually refits its reward head on real videos and on the generator’s latest samples, making the generator score the reward model where it has recently been supervised\. This paper makes the following contributions: - •Diagnosis\.We identify*latent reward hacking*and trace it to*distributional escape*: within a few hundred updates, the generator leaves the support of a fixed latent reward model and collects in a region that receives uniformly high scores, where the reward no longer reflects video quality\. - •Co\-Evolving Method\.We propose a novel frameworkCoRe, which treats latent\-space alignment as a co\-evolving process: the reward model is continually refit on the generator’s current samples, a three\-way Bradley–Terry objective keeps it a measure of quality rather than realness, and a relative objective with a feature\-space anchor gives the generator a stable learning signal\. - •Empirical improvement\.On Wan2\.1\-T2V\-1\.3B,CoReavoids the collapse of fixed\-reward and unanchored online optimization and improves VBench and VBench\-2\.0 scores over the pretrained model and prior alignment methods\. \(a\) Fixed latent reward\(b\) Co\-evolving rewardGenerator optimizationFalse high rewardFixed RM supervision does not follow the generatorGenerator optimizationCo\-evolving RM supervision follows the generatorGenerated sampleRM\-supervised regionUnsupported reward region Figure 2:Latent reward hacking and co\-evolving supervision\.With a fixed reward model, generator optimization can move samples beyond the region covered by reward supervision and exploit unsupported high scores \(left\)\. With co\-evolution, reward supervision follows the generator distribution as it changes \(right\)\. ## 2Related Work Preference alignment for visual and video generation\.Aligning generative models with human preferences has become an important post\-training paradigm for diffusion and flow\-based models\. Early work on image generation adapted RL either by treating denoising as a sequential decision\-making problem, as in DDPO\([Black et al\., 2024](https://arxiv.org/html/2609.36245#bib.bib7)\), or by directly backpropagating differentiable reward gradients through the sampling trajectory, as in AlignProp\([Prabhudesai et al\., 2024](https://arxiv.org/html/2609.36245#bib.bib6)\)\. More recent work extended preference alignment to video generation\. VideoScore\([He et al\., 2024](https://arxiv.org/html/2609.36245#bib.bib22)\)learns a fine\-grained quality metric from large\-scale human ratings, while LiFT\([Wang et al\., 2024](https://arxiv.org/html/2609.36245#bib.bib23)\)trains a video critic from human scores and rationales and uses the resulting reward for reward\-weighted fine\-tuning\. VisionReward\([Xu et al\., 2026](https://arxiv.org/html/2609.36245#bib.bib24)\)models image and video preferences along interpretable dimensions\. VideoAlign\([Liu et al\., 2026b](https://arxiv.org/html/2609.36245#bib.bib25)\)introduces the multidimensional video reward model\. Latent\-space reward modeling\.Most visual reward models operate on decoded RGB images or videos\([Xu et al\., 2023](https://arxiv.org/html/2609.36245#bib.bib3);[Kirstain et al\., 2023](https://arxiv.org/html/2609.36245#bib.bib26);[Liu et al\., 2026b](https://arxiv.org/html/2609.36245#bib.bib25)\)\. Recent work has therefore begun to move reward evaluation into the generative latent space\. In particular, PRFL\([Mi et al\., 2026](https://arxiv.org/html/2609.36245#bib.bib13)\)observes that pretrained video generation models are themselves well suited for processing noisy video latents and constructs a process\-aware latent reward model that can score intermediate states at arbitrary denoising timesteps\. These results establish latent reward modeling as an effective and efficient alternative to pixel\-space reward feedback\. Reward over\-optimization and reward hacking\.Reward hacking is a general consequence of optimizing an imperfect proxy objective\.[Gao et al\. \(2023\)](https://arxiv.org/html/2609.36245#bib.bib14)demonstrate reward\-model over\-optimization in RLHF\. Conservative reward\-model optimization reduces reliance on idiosyncratic reward errors\([Coste et al\., 2024](https://arxiv.org/html/2609.36245#bib.bib18)\), while constrained RLHF limits optimization to regimes where the reward models remain useful proxies\([Moskovitz et al\., 2024](https://arxiv.org/html/2609.36245#bib.bib20)\)\. TDPO\-R\([Zhang et al\., 2024](https://arxiv.org/html/2609.36245#bib.bib15)\)attributes part of this behavior to temporal inductive bias in diffusion optimization, whereas RSA\-FT\([Kim and Sim, 2026](https://arxiv.org/html/2609.36245#bib.bib17)\)shows that the generator can exploit non\-robust reward gradients\. ## 3Preliminaries Flow\-matching generator\.We consider a conditional flow\-matching video generatorGθG\_\{\\theta\}\([Lipman et al\., 2022](https://arxiv.org/html/2609.36245#bib.bib37)\)\. Given a clean video latentx0x\_\{0\}, Gaussian noisex1∼𝒩\(0,I\)x\_\{1\}\\sim\\mathcal\{N\}\(0,I\), and a text conditioncc, the intermediate state is xt=\(1−t\)x0\+tx1,t∈\[0,1\]\.x\_\{t\}=\(1\-t\)x\_\{0\}\+tx\_\{1\},\\qquad t\\in\[0,1\]\.\(1\)The generator predicts the velocityvθ\(xt,t,c\)v\_\{\\theta\}\(x\_\{t\},t,c\)and is pretrained with ℒFM\(θ\)=𝔼x0,x1,t,c\[‖vθ\(xt,t,c\)−\(x1−x0\)‖22\]\.\\mathcal\{L\}\_\{\\mathrm\{FM\}\}\(\\theta\)=\\mathbb\{E\}\_\{x\_\{0\},x\_\{1\},t,c\}\\big\[\\\|v\_\{\\theta\}\(x\_\{t\},t,c\)\-\(x\_\{1\}\-x\_\{0\}\)\\\|\_\{2\}^\{2\}\\big\]\.\(2\)Generation follows the path in reverse, fromt=1t=1towardt=0t=0\. We report sampling timesteps on the 1000\-step scale below\. Shared\-backbone latent reward model\.Our latent reward model scores noisy video latents\. It shares the generator’s DiT backbone: a single\-query attention module pools its block\-8 features, and an MLP maps the pooled featurehθ,ϕ\(xt,t,c\)h\_\{\\theta,\\phi\}\(x\_\{t\},t,c\)to a pre\-sigmoid logitℓθ,ϕ\(xt,t,c\)∈ℝ\\ell\_\{\\theta,\\phi\}\(x\_\{t\},t,c\)\\in\\mathbb\{R\}\. We denote the sigmoid\-normalized score and theℓ2\\ell\_\{2\}\-normalized pooled feature by sθ,ϕ\(xt,t,c\)=σ\(ℓθ,ϕ\(xt,t,c\)\),zθ,ϕ\(xt,t,c\)=hθ,ϕ\(xt,t,c\)‖hθ,ϕ\(xt,t,c\)‖2\.s\_\{\\theta,\\phi\}\(x\_\{t\},t,c\)=\\sigma\\\!\\left\(\\ell\_\{\\theta,\\phi\}\(x\_\{t\},t,c\)\\right\),\\qquad z\_\{\\theta,\\phi\}\(x\_\{t\},t,c\)=\\frac\{h\_\{\\theta,\\phi\}\(x\_\{t\},t,c\)\}\{\\\|h\_\{\\theta,\\phi\}\(x\_\{t\},t,c\)\\\|\_\{2\}\}\.\(3\)Hereθ\\thetadenotes the shared DiT parameters andϕ\\phidenotes the pooling module and MLP parameters\. Reward\-model updates modify onlyϕ\\phi; generator updates modifyθ\\theta\. Matched latents at a shared timestep\.For a training promptcc, we roll out the generator from Gaussian noise without gradients, then take one differentiable denoising step to obtain a generated latentx^t\\hat\{x\}\_\{t\}\. We independently forward\-noise the real video latentx0x\_\{0\}to the same timesteptt, obtainingxtx\_\{t\}\. The reward model thus evaluates real and generated latents for the same prompt at the same noise level\. Gradients from the generator objective pass through the final denoising step, while the preceding rollout is detached\. ## 4Analysis of Latent Reward Hacking ### 4\.1Reward Models Rapidly Out\-of\-Distribution We study optimization against a fixed latent reward model under two settings, with and without a flow\-matching \(SFT\) anchor\. Following common LRM practice\([Mi et al\., 2026](https://arxiv.org/html/2609.36245#bib.bib13)\), we train the reward model on preference pairs with a Bradley–Terry loss and freeze it before optimizing the generator with a hinge loss on the reward model’s raw logit\. As shown in Appendix Fig\.[8](https://arxiv.org/html/2609.36245#A1.F8), without the anchor, the reward rises rapidly while the generated videos begin to blur around step 15, develop grid\-like artifacts by step 24, and collapse by step 100\. We further examine the anchored fixed\-reward run at selected checkpoints in Tab\.[1](https://arxiv.org/html/2609.36245#S4.T1)\. From step 0 to step 1000, the reported optimization reward increases from 0\.636 to 0\.742, while dynamic degree falls from 68\.06 to 27\.78 and imaging quality falls from 67\.91 to 64\.23\. Over the same checkpoints, generated\-feature spread contracts from 1\.16 to 0\.74 relative to real features\. At step 500, the reward model scores generated latents above real ones \(0\.753 vs\. 0\.554\)\. Moreover, we shows that the generated samples concentrate in a distinct region assigned high reward \(≈0\.9\\approx 0\.9\), while real samples span a wider range of scores \(See Appendix Fig\.[7](https://arxiv.org/html/2609.36245#A1.F7)\)\. Together with the decline in video quality in Tab\.[1](https://arxiv.org/html/2609.36245#S4.T1), this shows that the frozen reward head continues to assign high scores to the generator’s changed output distribution even when those scores no longer reflect video quality\. Table 1:Fixed latent reward optimization\. Reward is the frozen LRM mean sigmoid score\.sreals\_\{\\mathrm\{real\}\}andsgens\_\{\\mathrm\{gen\}\}are mean scores of 256 real and 256 generated latents, respectively, evaluated att=591t\{=\}591for each checkpoint\. Spread is the ratio of within\-generated to within\-real 10\-nearest\-neighbor distances in normalized reward features; lower values indicate more concentrated\.Bluemarks increasing generated scores;Redmarks declines in corresponding degree\. ### 4\.2Layer\-wise Reward Features We additionally study which generator features are most suitable for latent reward modeling\. We construct reward models using features from different DiT depths and evaluate held\-out preference accuracy, reward–quality correlation, OOD sensitivity, and optimization stability\. We report the full layer\-wise study in Appendix[B](https://arxiv.org/html/2609.36245#A2)\. Algorithm 1One iteration ofCoRe1:generator θ\\theta, reward head ϕ\\phi, EMA weights θema\\theta\_\{\\mathrm\{ema\}\} 2:for i=1i=1to nDn\_\{D\}do 3:sample \(c,x0\+,x0−\)\(c,x\_\{0\}^\{\+\},x\_\{0\}^\{\-\}\); roll out x^t\\hat\{x\}\_\{t\}without gradient; noise x0±→xt±x\_\{0\}^\{\\pm\}\\to x\_\{t\}^\{\\pm\} 4: ϕ←ϕ−ηD∇ϕℒR\(x^t,xt\+,xt−\)\\phi\\leftarrow\\phi\-\\eta\_\{D\}\\nabla\_\{\\phi\}\\mathcal\{L\}\_\{R\}\(\\hat\{x\}\_\{t\},x\_\{t\}^\{\+\},x\_\{t\}^\{\-\}\) 5:endfor 6:sample \(c,x0\+\)\(c,x\_\{0\}^\{\+\}\); roll out x^t\\hat\{x\}\_\{t\}with gradient through the last step; noise x0\+→xt\+x\_\{0\}^\{\+\}\\to x\_\{t\}^\{\+\} 7: θ←θ−ηG∇θℒG\(x^t,xt\+\)\\theta\\leftarrow\\theta\-\\eta\_\{G\}\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{G\}\(\\hat\{x\}\_\{t\},x\_\{t\}^\{\+\}\) 8: θema←βθema\+\(1−β\)θ\\theta\_\{\\mathrm\{ema\}\}\\leftarrow\\beta\\,\\theta\_\{\\mathrm\{ema\}\}\+\(1\-\\beta\)\\,\\theta ## 5Overview ofCoRe Figure 3:Overall framework ofCoRe\.### 5\.1Training Loop CoRealternates between fitting the reward head to the generator’s current outputs and updating the generator against the refreshed reward \(Fig\.[3](https://arxiv.org/html/2609.36245#S5.F3); Algorithm[1](https://arxiv.org/html/2609.36245#alg1)\)\. Each iteration uses a promptccwith its preferred and less\-preferred real videosx0\+x\_\{0\}^\{\+\}andx0−x\_\{0\}^\{\-\}\. We obtain generated and real latents at a shared timestept∈\[200,800\]t\\in\[200,800\], as described in Sec\.[3](https://arxiv.org/html/2609.36245#S3)\. *Reward\-head update\.*With the generator frozen, the headϕ\\phitakesnD=10n\_\{D\}\{=\}10steps onℒR\\mathcal\{L\}\_\{R\}\(Eq\.[5](https://arxiv.org/html/2609.36245#S5.E5)\), each on a fresh rollout, so that it is fit to the generator’s current distribution\. *Generator update\.*With the head frozen, the generator takes one step onℒG\\mathcal\{L\}\_\{G\}\(Eq\.[4](https://arxiv.org/html/2609.36245#S5.E4)\); gradients reachθ\\thetathrough the last denoising step and the shared backbone\. We evaluate an exponential moving average ofθ\\theta\. ### 5\.2CoRegenerator objective Section[4\.1](https://arxiv.org/html/2609.36245#S4.SS1)shows that a fixed reward is exploited as generated samples concentrate in a high\-scoring region of its feature space\. As shown in Fig\.[3](https://arxiv.org/html/2609.36245#S5.F3), we update the reward head on samples from the generator’s current distribution after each generator update\.CoRejointly updates the reward head and generator\. The generator is optimized through the last differentiable denoising step with ℒG=\\displaystyle\\mathcal\{L\}\_\{G\}=\{\}𝔼\[ReLU\(ℓ¯ϕ\(xt\)−ℓϕ\(x^t\)−δ\)\]\\displaystyle\\mathbb\{E\}\\\!\\left\[\\operatorname\{ReLU\}\\\!\\left\(\\bar\{\\ell\}\_\{\\phi\}\(x\_\{t\}\)\-\\ell\_\{\\phi\}\(\\hat\{x\}\_\{t\}\)\-\\delta\\right\)\\right\]\(4\)\+λk𝔼\[1−exp\(−‖z\(x^t\)−z¯\(xt\)‖222σ2\)\]\+λfmℒFM\(θ\),\\displaystyle\+\\lambda\_\{k\}\\,\\mathbb\{E\}\\\!\\left\[1\-\\exp\\\!\\left\(\-\\frac\{\\\|z\(\\hat\{x\}\_\{t\}\)\-\\bar\{z\}\(x\_\{t\}\)\\\|\_\{2\}^\{2\}\}\{2\\sigma^\{2\}\}\\right\)\\right\]\+\\lambda\_\{\\mathrm\{fm\}\}\\mathcal\{L\}\_\{\\mathrm\{FM\}\}\(\\theta\),whereℓϕ\\ell\_\{\\phi\}is the reward model’s pre\-sigmoid logit,xtx\_\{t\}is a forward\-noised real video matched to the generated samplex^t\\hat\{x\}\_\{t\}by prompt and timestep, and a bar denotes stop\-gradient\. The featurezzis theℓ2\\ell\_\{2\}\-normalized pooled representation fed to the reward head\. The first term uses the matched real logit as a relative target: withδ=0\\delta=0, its gradient vanishes once the generated logit reaches the real logit\. The kernel term encourages the generated and real pooled features to remain close, while the flow\-matching term anchors the shared DiT backbone on real data\. Table 2:Quantitative comparison on VBench and VBench\-2\.0\. All methods are built upon Wan2\.1\-T2V\-1\.3B; All scores are higher is better\. CW means chunk\-wise, FW means frame\-wise\. Best results are marked ingreen\.Table 3:VBench results for ablation study\. Avg\. is the mean of all dimensions\. Red shading \(–\) marks increasing hacking\.*Unanchored online RM*: co\-evolving reward with an absolute score target and no kernel anchor, trained from the pretrained model at two generator learning rates; red shading deepens with training as the collapse sets in\.*no EMA*/*EMA*: the same objective resumed at step 300 for 100 steps, without and with an EMA generator\.CoReshares the warm start of the naive runs and differs only in the generator objective\. Best inbold\.Table[3](https://arxiv.org/html/2609.36245#S5.T3) Table[3](https://arxiv.org/html/2609.36245#S5.T3) ### 5\.3CoRereward objective A discriminator trained only on \{real, generated\} pairs learns*realness*: it can satisfy its objective by keying on generator artifacts that are perceptually irrelevant, and the resulting reward gradient pushes the generator toward arbitrary real\-looking outputs rather than good ones\. We therefore introduce a third population:*negative reals*x−x^\{\-\}, genuine videos labeled low\-quality on the target attribute \(e\.g\., static or unstable motion\)\. The reward model is trained with a three\-way Bradley–Terry objective, ℒR=\\displaystyle\\mathcal\{L\}\_\{R\}=\{\}𝔼\[softplus\(ℓϕ\(x^t\)−ℓϕ\(xt\+\)\)\]\\displaystyle\\mathbb\{E\}\\\!\\left\[\\operatorname\{softplus\}\\\!\\left\(\\ell\_\{\\phi\}\(\\hat\{x\}\_\{t\}\)\-\\ell\_\{\\phi\}\(x\_\{t\}^\{\+\}\)\\right\)\\right\]\(5\)\+w𝔼\[softplus\(ℓϕ\(xt−\)−ℓϕ\(xt\+\)\)\],w=0\.5,\\displaystyle\+w\\,\\mathbb\{E\}\\\!\\left\[\\operatorname\{softplus\}\\\!\\left\(\\ell\_\{\\phi\}\(x\_\{t\}^\{\-\}\)\-\\ell\_\{\\phi\}\(x\_\{t\}^\{\+\}\)\\right\)\\right\],\\qquad w=0\.5,wherex\+x^\{\+\}denotes positive reals, and all three latents are evaluated at a shared timesteptt\. The second term forces the score function to order*within*the real manifold, so high reward cannot be achieved by realness alone; the negative\-real pair also remains informative even when the generator’s outputs become hard to distinguish from real data, which keeps the discriminator’s gradient alive precisely in the regime where a two\-way objective flatlines\. Crucially, the negative reals do not supervise the generator directly; they improve the reward model’s quality judgment and stabilize the generator only through a more reliable reward signal\. ### 5\.4Grounding Rewards inCoRe To learn a reliable quality ordering of preferred and dispreferred real videos while tracking the evolving generator distribution, the reward model must remain stable throughout co\-training\. We stabilize this loop with three inexpensive mechanisms\. Asymmetric updates\.The reward head is updatednD≥1n\_\{D\}\\geq 1times per generator step \(nD=5n\_\{D\}\{=\}5–1010in practice\), so that each generator update is taken against a freshly calibrated reward\. Because only the pooling head and MLP receive gradients, discriminator steps are cheap relative to generator rollouts\. Diffusion anchor\.The flow\-matching term in Eq\.[4](https://arxiv.org/html/2609.36245#S5.E4)\(weightλfm\\lambda\_\{\\mathrm\{fm\}\}\) regularizes the generator toward the data distribution, which serves double duty: it limits perceptual drift, and—in shared\-backbone mode—slows the drift of the reward model’s own feature extractor\. Kernel anchor\.Matching the reward of a preferred real video does not necessarily imply matching its quality, especially when the reward head has limited discriminative ability\. Inspired by drifting models\([Deng et al\., 2026](https://arxiv.org/html/2609.36245#bib.bib29)\), we complement the relative\-gap objective with a kernel anchor that pulls generated features toward those of paired preferred real videos in Eq\.[4](https://arxiv.org/html/2609.36245#S5.E4)\. The two terms provide complementary guidance: the relative gap constrains reward scores, while the kernel anchor directly encourages alignment with preferred real features\. This anchor reuses the pooled features already computed for the relative\-gap objective, requiring no additional forward pass\. ## 6Experiments 050%100%000\.10\.10\.20\.20\.30\.3\+0\.27\+0\.27training progresssgen−sreals\_\{\\mathrm\{gen\}\}\-s\_\{\\mathrm\{real\}\}\(a\) generated−\-real score along trainingFixed RMCoRe0\.20\.20\.40\.40\.60\.60\.80\.811005050100100150150P\(sgen\>sreal\)P\(s\_\{\\mathrm\{gen\}\}\>s\_\{\\mathrm\{real\}\}\)=0\.82=0\.82reward\-model score\# videos\(b\) Fixed RMreal \(train data\)generated0\.480\.480\.520\.520\.560\.560\.60\.600202040406060P\(sgen\>sreal\)P\(s\_\{\\mathrm\{gen\}\}\>s\_\{\\mathrm\{real\}\}\)=0\.45=0\.45reward\-model score\(c\)CoRereal \(train data\)generated Figure 4:Generated and real scores under fixed and co\-evolving rewards\.\(a\) Gap between the mean scores of generated and real latents over training; the horizontal axis shows normalized progress for each run\. Under the fixed reward, the gap grows, while underCoReit stays near zero\. \(b, c\) Histograms of reward\-model scores for real and generated latents at the end of training\.### 6\.1Experimental Setup Models and data\.We use Wan2\.1\-T2V\-1\.3B as the generator\. For training data, we use VideoDPO preference pairs, encoded once into 480p VAE latents together with their text embeddings\. The latent reward model takes features from DiT layers, pools them with a single\-query attention module, and maps the pooled feature to a logit with a three\-layer MLP\. Training\.All co\-evolving variants in Tab\.[3](https://arxiv.org/html/2609.36245#S5.T3)start from a shared warm\-started checkpoint; optimizer settings and the warm\-start schedule are given in Appendix[C](https://arxiv.org/html/2609.36245#A3)\. Evaluation\.For generation quality, we use VBench\([Huang et al\., 2024](https://arxiv.org/html/2609.36245#bib.bib27)\)and VBench\-2\.0\([Zheng et al\., 2025](https://arxiv.org/html/2609.36245#bib.bib28)\)\. VBench decomposes video quality into fine\-grained dimensions\. VBench\-2\.0 complements it by measuring intrinsic faithfulness, including human fidelity, controllability, physics, and commonsense\. We follow the standard protocol, generate832×480832\{\\times\}480, 81\-frame videos with 50 sampling steps, guidance scale 6\.0, shift 5\.0, and a fixed seed\.    \(a\)Fixed RM \(b\)Naive online RM \(c\)CoRe Figure 5:Reward\-model features of real and generated latents att=591t\{=\}591, visualized with t\-SNE and PCA\. Each panel shows: coloured by*domain*\(left; blue = real, red = generated\) and by the reward model’s score \(right; brighter = higher\)\. Under the fixed reward, generated latents form a separate cluster that receives uniformly high scores \(mean score0\.820\.82for generated vs\.0\.550\.55for real\); under ours, generated latents overlap with real data and their scores no longer depend on domain\. ### 6\.2Main Results CoReimproves generation quality across benchmarks\.Tab\.[3](https://arxiv.org/html/2609.36245#S5.T3)comparesCoRewith preference\-optimization and distillation methods built on the same Wan2\.1\-T2V\-1\.3B backbone\. On VBench,CoReimproves the pretrained model’s total score from83\.9683\.96to84\.9284\.92, the highest among all methods, and its semantic score from80\.1080\.10to82\.2082\.20, surpassing the strongest baseline, Self Forcing \(81\.2881\.28\), by nearly one point\. Its quality score \(85\.4085\.40\) is close to that of SIPO \(85\.7385\.73\), showing that the semantic gains do not come at the expense of visual quality\. On VBench\-2\.0,CoReobtains the best total score \(58\.0358\.03\) and the largest gain in controllability \(33\.8→38\.2933\.8\\rightarrow 38\.29\), exceeding every compared method by more than four points\. Its physics score \(61\.0361\.03\) improves over the pretrained model \(60\.060\.0\) but remains below TDM with FlashAttention\-2 \(63\.163\.1\)\. CoRestays stable where other rewards are hacked\.Tab\.[3](https://arxiv.org/html/2609.36245#S5.T3)tracks how each reward setting affects the generator over training\. Optimizing against a fixed latent reward lowers imaging quality from67\.9167\.91to60\.0160\.01without improving motion, and the VBench average drops below that of the pretrained model \(80\.6080\.60vs\.81\.3881\.38\)\. An online reward model trained without our anchors improves briefly and then collapses: at its peak, dynamic degree rises to 83\.72–89\.67 depending on the learning rate, but by the end of training it falls to 16\.67–19\.44, while subject consistency, background consistency, and smoothness increase\. These consistency gains are a symptom of hacking rather than improvement: the generator produces nearly static videos, which trivially score high on frame\-to\-frame consistency\. Resuming from step 300 with EMA only slows this decline \(dynamic degree58\.3358\.33at step 400\)\. In contrast,CoReremains stable at step 600, reaching dynamic degree86\.7286\.72together with the best aesthetic quality \(64\.4764\.47\) and the best VBench average \(84\.0584\.05\)\. Its main cost is subject consistency \(96\.35→92\.8996\.35\\rightarrow 92\.89\), which accompanies the added motion\. Fig\.[4](https://arxiv.org/html/2609.36245#S6.F4)and Fig\.[5](https://arxiv.org/html/2609.36245#S6.F5)show the same contrast in reward scores: under the fixed reward, generated latents score increasingly above real ones, whereas underCoRethe two remain indistinguishable\. ### 6\.3Main comparison Tab\.[3](https://arxiv.org/html/2609.36245#S5.T3)isolates the generator objective\. With an absolute score target, co\-evolution drifts toward static videos \(dynamic degree 68\.1 to 56\.9\), the same failure as the naive online reward\. The relative gap removes the drift: dynamic degree stays at the pretrained level \(69\.4\) and imaging quality drops by 2\.1 points instead of 3\.8 \(Appendix Tab\.[8](https://arxiv.org/html/2609.36245#A4.T8)\)\. The kernel anchor then raises dynamic degree by a further 12 to 15 points\. The fixed reward is unstable across checkpoints, while imaging quality reaches its minimum; the co\-evolving runs change smoothly\. Per\-checkpoint results for every run are in Appendix[D](https://arxiv.org/html/2609.36245#A4), Tab\.[7](https://arxiv.org/html/2609.36245#A4.T7)\. ### 6\.4Analysis of the Generator Objective The reward head separates real from generated latents only weakly, so its sigmoid output stays near 0\.55 for both\. The absolute target of 0\.9 is never met, and the reward loss pushes the generator at every step\. The relative hinge is zero whenever a generated latent scores as high as the matched real one, which happens on 44% of the generator steps; the mean reward loss is 0\.04 against 0\.33\. Kernel weight\.Loweringλk\\lambda\_\{k\}from 0\.05 to 0\.03 changes no dimension by more than 3 points\. Splitting the 72 videos by the VBench motion flag explains the trade\-off\. Subject consistency of static videos is unchanged \(0\.98 in all runs\)\. For moving videos it falls from 0\.95 to 0\.90, and the number of moving videos rises from 50 to about 60\. The kernel term matches each generated latent to the real video of the same prompt, so the generator copies the camera and object motion of that video; the larger motion lowers frame\-to\-frame subject similarity\. Imaging quality also drops on static videos \(70\.4 to 66\.2\), consistent with the generator moving toward the image statistics of the 480p training videos\. ### 6\.5Case Study Figure 6:Qualitative comparison\.We provide case studies on the generation task in VBench for qualitative analysis\. Fig\.[6](https://arxiv.org/html/2609.36245#S6.F6)shows four prompts generated by the pretrained model and byCoRe\. The left column corresponds to the gain in dynamic degree: the pretrained model renders the subject of a motion prompt but leaves it still, whileCoReproduces the described motion, whether it is the articulated motion of the horse’s head or the translation of the boat\. The right column corresponds to the gains in multiple objects and spatial relationship: the pretrained model omits one of the two requested objects or ignores the stated arrangement, andCoRerecovers both\. ## 7Discussion Multi\-timestep supervision\.We also explored supervising the reward at two denoising timesteps\. As shown in Table[4](https://arxiv.org/html/2609.36245#S7.T4), compared with single\-timestep training, two\-timestep supervision did not yield a significant improvement in reward reliability or resistance to reward hacking, while increasing the training time by approximately2\.3×2\.3\\times\. This suggests that the additional timestep provides limited benefit relative to its computational cost\. Training cost\.CoReis inexpensive to train: the full run, including the warm start, takes about 2 hours on 4 NVIDIA H100 GPUs \(about 8 GPU\-hours\)\. Tab\.[5](https://arxiv.org/html/2609.36245#S7.T5)lists the compute reported by prior video RL methods\. These settings differ in model size and resolution and are not directly comparable, but they indicate thatCoRecompute budget than online RL for video generation\. Table 4:Single\- vs\. two\-timestep reward supervision\. DanceGRPO\([Xue et al\., 2025](https://arxiv.org/html/2609.36245#bib.bib10)\); PRFL\([Mi et al\., 2026](https://arxiv.org/html/2609.36245#bib.bib13)\)\. Table 5:Reported training compute of video alignment methods\. ## 8Conclusion We introducedCoRe, which co\-evolves the reward model with the generator to mitigate this failure and stabilize optimization\. This work has the following limitations: \(i\) we conduct experiments on 1\.3B\-scale model; training larger models and developing finer\-grained probes of latent reward exploitation may reveal additional insights\. \(ii\) We use a limited set of datasets, and future work should testCoReon more challenging benchmarks\. ### AI use statement In this work, we used generative AI tools for language polishing, LaTeX formatting, and literature search\. We have reviewed all AI\-assisted work and take responsibility for the final content of this work, including text or artifacts produced with the aid of generative AI\. ### Reproducibility statement The training setup and hyperparameters are described in Sec\.[6](https://arxiv.org/html/2609.36245#S6); the fixed\-reward baseline and additional analyses are given in the appendix\. Code will be released as supplementary material\. ## References - Ackermannet al\.\(2025\)J\. Ackermann, T\. Ishida, and M\. SugiyamaOff\-policy corrected reward modeling for reinforcement learning from human feedback\.arXiv preprint arXiv:2507\.15507\.Cited by:[§1](https://arxiv.org/html/2609.36245#S1.p2.1),[§1](https://arxiv.org/html/2609.36245#S1.p4.1)\. - Blacket al\.\(2024\)K\. Black, M\. Janner, Y\. Du, I\. Kostrikov, and S\. LevineTraining diffusion models with reinforcement learning\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 4965–4987\.Cited by:[§1](https://arxiv.org/html/2609.36245#S1.p1.1),[§2](https://arxiv.org/html/2609.36245#S2.p1.1)\. - Christianoet al\.\(2017\)P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. AmodeiDeep reinforcement learning from human preferences\.Advances in neural information processing systems30\.Cited by:[§1](https://arxiv.org/html/2609.36245#S1.p1.1)\. - Clarket al\.\(2024\)K\. Clark, P\. Vicol, K\. Swersky, and D\. FleetDirectly fine\-tuning diffusion models on differentiable rewards\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 4793–4822\.Cited by:[§1](https://arxiv.org/html/2609.36245#S1.p1.1)\. - Costeet al\.\(2024\)T\. Coste, U\. Anwar, R\. Kirk, and D\. KruegerReward model ensembles help mitigate overoptimization\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 50905–50931\.Cited by:[§1](https://arxiv.org/html/2609.36245#S1.p4.1),[§2](https://arxiv.org/html/2609.36245#S2.p3.1)\. - Denget al\.\(2026\)M\. Deng, H\. Li, T\. Li, Y\. Du, and K\. HeGenerative modeling via drifting\.arXiv preprint arXiv:2602\.04770\.Cited by:[§5\.4](https://arxiv.org/html/2609.36245#S5.SS4.p4.1)\. - Dinget al\.\(2025\)Z\. Ding, C\. Jin, D\. Liu, H\. Zheng, K\. K\. Singh, Q\. Zhang, Y\. Kang, Z\. Lin, and Y\. LiuDollar: few\-step video generation via distillation and latent reward optimization\.In2025 IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 17961–17971\.Cited by:[§1](https://arxiv.org/html/2609.36245#S1.p1.1)\. - Fanet al\.\(2023\)Y\. Fan, O\. Watkins, Y\. Du, H\. Liu, M\. Ryu, C\. Boutilier, P\. Abbeel, M\. Ghavamzadeh, K\. Lee, and K\. LeeDpok: reinforcement learning for fine\-tuning text\-to\-image diffusion models\.Advances in neural information processing systems36,pp\. 79858–79885\.Cited by:[§1](https://arxiv.org/html/2609.36245#S1.p1.1),[§1](https://arxiv.org/html/2609.36245#S1.p2.1)\. - Gaoet al\.\(2023\)L\. Gao, J\. Schulman, and J\. HiltonScaling laws for reward model overoptimization\.InInternational conference on machine learning,pp\. 10835–10866\.Cited by:[§1](https://arxiv.org/html/2609.36245#S1.p2.1),[§1](https://arxiv.org/html/2609.36245#S1.p4.1),[§2](https://arxiv.org/html/2609.36245#S2.p3.1)\. - Guet al\.\(2026\)Y\. Gu, X\. Li, Y\. Hu, C\. Minqi, and B\. ZhuangBlade: block\-sparse attention meets step distillation for efficient video generation\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 56021–56045\.Cited by:[Table 3](https://arxiv.org/html/2609.36245#S5.T3.p1.1.1.1.11.1)\. - Heet al\.\(2024\)X\. He, D\. Jiang, G\. Zhang, M\. Ku, A\. Soni, S\. Siu, H\. Chen, A\. Chandra, Z\. Jiang, A\. Arulraj,et al\.Videoscore: building automatic metrics to simulate fine\-grained human feedback for video generation\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 2105–2123\.Cited by:[§2](https://arxiv.org/html/2609.36245#S2.p1.1)\. - Huanget al\.\(2026\)X\. Huang, Z\. Li, G\. He, M\. Zhou, and E\. ShechtmanSelf forcing: bridging the train\-test gap in autoregressive video diffusion\.Advances in Neural Information Processing Systems38,pp\. 167283–167308\.Cited by:[Table 3](https://arxiv.org/html/2609.36245#S5.T3.p1.1.1.1.8.1),[Table 3](https://arxiv.org/html/2609.36245#S5.T3.p1.1.1.1.9.1)\. - Huanget al\.\(2024\)Z\. Huang, Y\. He, J\. Yu, F\. Zhang, C\. Si, Y\. Jiang, Y\. Zhang, T\. Wu, Q\. Jin, N\. Chanpaisit,et al\.Vbench: comprehensive benchmark suite for video generative models\.In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 21807–21818\.Cited by:[§6\.1](https://arxiv.org/html/2609.36245#S6.SS1.p3.1)\. - Kim and Sim \(2026\)K\. Kim and B\. SimReward sharpness\-aware fine\-tuning for diffusion models\.arXiv preprint arXiv:2603\.21175\.Cited by:[§1](https://arxiv.org/html/2609.36245#S1.p4.1),[§2](https://arxiv.org/html/2609.36245#S2.p3.1)\. - Kirstainet al\.\(2023\)Y\. Kirstain, A\. Polyak, U\. Singer, S\. Matiana, J\. Penna, and O\. LevyPick\-a\-pic: an open dataset of user preferences for text\-to\-image generation\.Advances in neural information processing systems36,pp\. 36652–36663\.Cited by:[§2](https://arxiv.org/html/2609.36245#S2.p2.1)\. - Lipmanet al\.\(2022\)Y\. Lipman, R\. T\. Chen, H\. Ben\-Hamu, M\. Nickel, and M\. LeFlow matching for generative modeling\.arXiv preprint arXiv:2210\.02747\.Cited by:[§3](https://arxiv.org/html/2609.36245#S3.p1.1)\. - Liuet al\.\(2026a\)J\. Liu, G\. Liu, J\. Liang, Y\. Li, J\. Liu, X\. Wang, P\. Wan, D\. Zhang, and W\. OuyangFlow\-grpo: training flow matching models via online rl\.Advances in neural information processing systems38,pp\. 40783–40818\.Cited by:[§1](https://arxiv.org/html/2609.36245#S1.p1.1)\. - Liuet al\.\(2026b\)J\. Liu, G\. Liu, J\. Liang, Z\. Yuan, X\. Liu, M\. Zheng, X\. Wu, Q\. Wang, M\. Xia, X\. Wang,et al\.Improving video generation with human feedback\.Advances in Neural Information Processing Systems38,pp\. 82155–82192\.Cited by:[§2](https://arxiv.org/html/2609.36245#S2.p1.1),[§2](https://arxiv.org/html/2609.36245#S2.p2.1)\. - Loshchilov and Hutter \(2017\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.arXiv preprint arXiv:1711\.05101\.Cited by:[Appendix C](https://arxiv.org/html/2609.36245#A3.p1.1)\. - Luoet al\.\(2025\)Y\. Luo, T\. Hu, J\. Sun, Y\. Cai, and J\. TangLearning few\-step diffusion models by trajectory distribution matching\.In2025 IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 17719–17728\.Cited by:[Table 3](https://arxiv.org/html/2609.36245#S5.T3.p1.1.1.1.10.1)\. - Miet al\.\(2026\)X\. Mi, W\. Yu, J\. Lian, S\. Jie, R\. Zhong, Z\. Liu, G\. Zhang, Z\. Zhou, Z\. Xu, Y\. Zhou,et al\.Video generation models are good latent reward models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 4719–4728\.Cited by:[Appendix D](https://arxiv.org/html/2609.36245#A4.p1.1),[§1](https://arxiv.org/html/2609.36245#S1.p1.1),[§1](https://arxiv.org/html/2609.36245#S1.p2.1),[§2](https://arxiv.org/html/2609.36245#S2.p2.1),[§4\.1](https://arxiv.org/html/2609.36245#S4.SS1.p1.1),[Table 5](https://arxiv.org/html/2609.36245#S7.T5.2)\. - Miaoet al\.\(2024\)Y\. Miao, S\. Zhang, L\. Ding, R\. Bao, L\. Zhang, and D\. TaoInform: mitigating reward hacking in rlhf via information\-theoretic reward modeling\.Advances in Neural Information Processing Systems37,pp\. 134387–134429\.Cited by:[§1](https://arxiv.org/html/2609.36245#S1.p4.1)\. - Moskovitzet al\.\(2024\)T\. Moskovitz, A\. Singh, D\. Strouse, T\. Sandholm, R\. Salakhutdinov, A\. Dragan, and S\. McAleerConfronting reward model overoptimization with constrained rlhf\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 21998–22025\.Cited by:[§1](https://arxiv.org/html/2609.36245#S1.p4.1),[§2](https://arxiv.org/html/2609.36245#S2.p3.1)\. - Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2609.36245#S1.p1.1)\. - Prabhudesaiet al\.\(2023\)M\. Prabhudesai, A\. Goyal, D\. Pathak, and K\. FragkiadakiAligning text\-to\-image diffusion models with reward backpropagation\.Cited by:[§1](https://arxiv.org/html/2609.36245#S1.p1.1)\. - Prabhudesaiet al\.\(2024\)M\. Prabhudesai, R\. Mendonca, Z\. Qin, K\. Fragkiadaki, and D\. PathakVideo diffusion alignment via reward gradients\.arXiv preprint arXiv:2407\.08737\.Cited by:[§2](https://arxiv.org/html/2609.36245#S2.p1.1)\. - Wallaceet al\.\(2024\)B\. Wallace, M\. Dang, R\. Rafailov, L\. Zhou, A\. Lou, S\. Purushwalkam, S\. Ermon, C\. Xiong, S\. Joty, and N\. NaikDiffusion model alignment using direct preference optimization\.In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 8228–8238\.Cited by:[Table 3](https://arxiv.org/html/2609.36245#S5.T3.p1.1.1.1.4.1)\. - Wanet al\.\(2025\)T\. Wan, A\. Wang, B\. Ai, B\. Wen, C\. Mao, C\. Xie, D\. Chen, F\. Yu, H\. Zhao, J\. Yang,et al\.Wan: open and advanced large\-scale video generative models\.arXiv preprint arXiv:2503\.20314\.Cited by:[Table 3](https://arxiv.org/html/2609.36245#S5.T3.p1.1.1.1.3.1)\. - Wanget al\.\(2024\)Y\. Wang, Z\. Tan, J\. Wang, X\. Yang, C\. Jin, and H\. LiLift: leveraging human feedback for text\-to\-video model alignment\.arXiv preprint arXiv:2412\.04814\.Cited by:[§2](https://arxiv.org/html/2609.36245#S2.p1.1)\. - Wolfet al\.\(2025\)L\. Wolf, R\. Kirk, and M\. MusolesiReward model overoptimisation in iterated rlhf\.arXiv preprint arXiv:2505\.18126\.Cited by:[§1](https://arxiv.org/html/2609.36245#S1.p4.1)\. - Xuet al\.\(2026\)J\. Xu, Y\. Huang, J\. Cheng, Y\. Yang, J\. Xu, Y\. Wang, W\. Duan, S\. Yang, Q\. Jin, S\. Li,et al\.Visionreward: fine\-grained multi\-dimensional human preference learning for image and video generation\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 11269–11277\.Cited by:[§2](https://arxiv.org/html/2609.36245#S2.p1.1)\. - Xuet al\.\(2023\)J\. Xu, X\. Liu, Y\. Wu, Y\. Tong, Q\. Li, M\. Ding, J\. Tang, and Y\. DongImagereward: learning and evaluating human preferences for text\-to\-image generation\.Advances in Neural Information Processing Systems36,pp\. 15903–15935\.Cited by:[§1](https://arxiv.org/html/2609.36245#S1.p1.1),[§1](https://arxiv.org/html/2609.36245#S1.p2.1),[§2](https://arxiv.org/html/2609.36245#S2.p2.1)\. - Xueet al\.\(2025\)Z\. Xue, J\. Wu, Y\. Gao, F\. Kong, L\. Zhu, M\. Chen, Z\. Liu, W\. Liu, Q\. Guo, W\. Huang,et al\.Dancegrpo: unleashing grpo on visual generation\.arXiv preprint arXiv:2505\.07818\.Cited by:[§1](https://arxiv.org/html/2609.36245#S1.p1.1),[Table 5](https://arxiv.org/html/2609.36245#S7.T5.2)\. - Yanget al\.\(2025\)X\. Yang, M\. Yang, J\. Wang, Z\. Zhou, Z\. Tan, and H\. LiSIPO: stabilized and improved preference optimization for aligning diffusion models\.Cited by:[Table 3](https://arxiv.org/html/2609.36245#S5.T3.p1.1.1.1.5.1),[Table 3](https://arxiv.org/html/2609.36245#S5.T3.p1.1.1.1.6.1)\. - Yinet al\.\(2025\)T\. Yin, Q\. Zhang, R\. Zhang, W\. T\. Freeman, F\. Durand, E\. Shechtman, and X\. HuangFrom slow bidirectional to fast autoregressive video diffusion models\.In2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 22963–22974\.Cited by:[Table 3](https://arxiv.org/html/2609.36245#S5.T3.p1.1.1.1.7.1)\. - Zhanget al\.\(2026\)T\. Zhang, C\. Da, K\. Ding, H\. Yang, Y\. Li, T\. Gao, D\. Zhang, S\. Xiang, C\. Pan,et al\.Diffusion model as a noise\-aware latent reward model for step\-level preference optimization\.Advances in Neural Information Processing Systems38,pp\. 106202–106235\.Cited by:[§1](https://arxiv.org/html/2609.36245#S1.p1.1),[§1](https://arxiv.org/html/2609.36245#S1.p2.1)\. - Zhanget al\.\(2024\)Z\. Zhang, S\. Zhang, Y\. Zhan, Y\. Luo, Y\. Wen, and D\. TaoConfronting reward overoptimization for diffusion models: a perspective of inductive and primacy biases\.arXiv preprint arXiv:2402\.08552\.Cited by:[§1](https://arxiv.org/html/2609.36245#S1.p2.1),[§1](https://arxiv.org/html/2609.36245#S1.p4.1),[§2](https://arxiv.org/html/2609.36245#S2.p3.1)\. - Zhenget al\.\(2025\)D\. Zheng, Z\. Huang, H\. Liu, K\. Zou, Y\. He, F\. Zhang, L\. Gu, Y\. Zhang, J\. He, W\. Zheng,et al\.Vbench\-2\.0: advancing video generation benchmark suite for intrinsic faithfulness\.arXiv preprint arXiv:2503\.21755\.Cited by:[§6\.1](https://arxiv.org/html/2609.36245#S6.SS1.p3.1)\. ## Appendix AOOD Diagnostics in the Reward\-Feature Space Fig\.[7](https://arxiv.org/html/2609.36245#A1.F7)repeats the analysis of Fig\.[5](https://arxiv.org/html/2609.36245#S6.F5)on a larger set of samples from a fixed\-reward run, visualizing the features fed to the reward head\. The two domains are almost perfectly separable, and only1\.3%1\.3\\%of each sample’s nearest neighbors come from the other domain, so the separation is local as well as global\. Real samples span the full range of reward scores along an ordered direction, whereas generated samples extend beyond its high\-score end and receive uniformly high rewards, in a region where the reward model received no supervision\.  \(a\) PCA  \(b\) t\-SNE Figure 7:Real vs\. generated samples in the reward\-feature space under a fixed reward model, colored by reward score \(▲\\blacktrianglereal,∙\\bulletgenerated\)\. The two domains are almost perfectly separated, and generated samples concentrate in a high\-reward region \(yellow\) beyond the range covered by real samples\. \(a\) w/o flow\-matching anchor  \(b\) w/ flow\-matching anchor Figure 8:Training reward against a fixed latent reward model\.\(a\) Without regularization, the reward saturates near 0\.95 within about 20 updates, while the decoded videos blur by step 15, show grid\-like artifacts by step 24, and collapse by step 100\. \(b\) With the flow\-matching anchor, the reward no longer saturates but also shows no clear upward trend over 2\.5k updates\. A fixed reward thus leaves two outcomes: rapid exploitation, or little effective optimization\. ## Appendix BLayer\-wise Reward Features Tab\.[6](https://arxiv.org/html/2609.36245#A2.T6)compares reward models built on features from different DiT blocks\. Block 8 gives the highest held\-out preference accuracy\. Table 6:Reward models built on different DiT blocks\.Preference acc\.: held\-out accuracy on VideoDPO pairs\. Reward–quality corr\.: Pearson correlation between reward scores and VBench imaging quality on generated videos\. Stable steps: number of generator updates before dynamic degree falls below0\.30\.3\.DiT blockPreference acc\.↑\\uparrowReward–quality corr\.↑\\uparrowStable steps↑\\uparrow468\.20\.41420876\.40\.548501274\.10\.435101671\.60\.37290 ## Appendix CImplementation Details The generator is sampled on a 40\-step schedule\. We use AdamW\([Loshchilov and Hutter, 2017](https://arxiv.org/html/2609.36245#bib.bib38)\)with constant learning rates of2×10−62\\times 10^\{\-6\}\(generator\) and3×10−63\\times 10^\{\-6\}\(reward head\), a flow\-matching weightλfm=1\.5\\lambda\_\{\\mathrm\{fm\}\}\{=\}1\.5, and evaluate the EMA of the generator \(decay0\.990\.99\)\. All co\-evolving variants share the same warm start: starting from the pretrained model, the reward head is trained for 200 steps with the generator frozen, the generator is then trained for 100 steps with the absolute\-target objective, and the reward head is re\-fit for another 100 steps\. From this common checkpoint, each variant trains the generator for 100 further steps with the objective under study\. ## Appendix DAll Ablation Study Table 7:VBench scores for all runs\. SC = subject\_consistency, BC = background\_consistency, MS = motion\_smoothness, DD = dynamic\_degree, AQ = aesthetic\_quality, IQ = imaging\_quality\.Tab\.[7](https://arxiv.org/html/2609.36245#A4.T7)collects the six VBench quality dimensions for every run: subject consistency \(SC\), background consistency \(BC\), motion smoothness \(MS\), dynamic degree \(DD\), aesthetic quality \(AQ\), and imaging quality \(IQ\), each benchmark on VBench\. The runs are as follows\.*Fixed LRM*is the pipeline with a frozen latent reward model\([Mi et al\., 2026](https://arxiv.org/html/2609.36245#bib.bib13)\)\.*v2*and*w1p5*are co\-evolving runs started from the pretrained model with an absolute score target, differing only in the generator learning rate \(2×10−62\{\\times\}10^\{\-6\}and5×10−65\{\\times\}10^\{\-6\}\); the first 200 steps of v2 update the reward head alone, so its step\-200 checkpoint is the pretrained model\.*dyn300*resumes v2 at step 300 on VisionReward motion pairs instead of VideoDPO, with and without an EMA generator\.*oldw400ema*,*relgapema*and*kernelema*share one warm start: v2 at step 300 with the reward head re\-fit for a further 100 steps\. They then train the generator for 100 steps with the absolute target, the relative gap, and the relative gap plus kernel anchor respectively, and are evaluated with the EMA generator\. Two baselines bracket the results\. The pretrained model sets the reference for IQ \(0\.679\); the fixed reward leaves DD unchanged and lowers IQ by eight points\. The naive co\-evolving runs first raise DD \(0\.896 for v2 at step 300\) and then collapse it \(0\.194 at step 750; 0\.167 for w1p5 at step 450\)\. The rise in SC, BC and MS along the way reflects videos that stop moving, not an improvement\. Among the runs with the shared warm start, the absolute target continues the decline in DD \(0\.569\)\. The relative gap holds DD at the pretrained level \(0\.694\) with the best IQ of the block \(0\.659\)\. Table 8:Generator objective, evaluated at two checkpoints of the same 100\-step run\.
Similar Articles
RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling
RewardVerse presents a rubric-based framework for video reward modeling to mitigate scalar drift and provide stable evaluation criteria, enhancing reinforcement learning in video generation.
Stream-R1: Reliability-Perplexity Aware Reward Distillation for Streaming Video Generation
Stream-R1 introduces a reliability-perplexity aware reward distillation framework for streaming video generation that adaptively weights supervision to improve visual and motion quality without additional computational overhead.
Optimizing Visual Generative Models via Distribution-wise Rewards
This paper presents a reinforcement learning framework for visual generative models that uses distribution-wise rewards, with a subset-replace strategy for efficiency, improving image diversity and quality while addressing mode collapse and reward hacking.
Adaptive Reward Routing: Dynamic Multi-Reward Optimization for Joint Audio-Video Diffusion via Forward-Process RL
提出 Adaptive Reward Routing,在联合音频-视频扩散模型的前向过程强化学习(DiffusionNFT)中动态调整更新位置与奖励权重:利用双向跨注意力响应定位跨模态影响,并以偏好先验加残差修正的方式协调多个竞争奖励,从而提升模态质量、语义一致性与音视频同步。
ReCAST: Reward Credit Assignment across Timesteps for Online Diffusion Reinforcement
ReCAST proposes a method for per-reward, timestep-dependent credit assignment in diffusion model fine-tuning, separating user preferences from temporal allocation to improve alignment and informativeness.