RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling

Hugging Face Daily Papers Papers
reinforcement-learning video-generation reward-modeling rubric-evaluation scalar-drift policy-optimization

Summary

RewardVerse presents a rubric-based framework for video reward modeling to mitigate scalar drift and provide stable evaluation criteria, enhancing reinforcement learning in video generation.

Reinforcement learning (RL) is vital for optimizing video generation models, with a robust reward model (RM) serving as the cornerstone. However, existing video reward models often produce unstable scalar scores because they directly map complex, subjective video quality into a single score without explicit evaluation criteria. This leads to scalar drift, where the scoring scale collapses or shifts across different prompts, making the reward unreliable for RL. Drawing inspiration from professional human annotation engineering, we address this problem with RewardVerse, a rubric-based video reward framework that introduces a dynamic rubric as an intermediate representation between the evaluation query and the scorer. Instead of unconstrained direct scoring, RewardVerse first generates explicit evaluation criteria and then performs rubric-guided scoring, providing a stable semantic anchor that mitigates scalar drift. To efficiently optimize this collaborative pipeline, we propose Rubric-Guided Policy Optimization (RGPO), a two-stage training algorithm. RGPO first warms up the scorer using self-evolving seed rubrics and then jointly optimizes the rubric generator to produce query-adaptive evaluation criteria while continuously aligning the scorer with human ratings. Extensive experiments on the 16-dimensional EvalVerse benchmark and external datasets demonstrate that RewardVerse mitigates scalar drift, achieves state-of-the-art performance on both pointwise and pairwise evaluation, and provides a robust and interpretable reward signal for RL in video generation.
Original Article
View Cached Full Text

Cached at: 09/24/26, 03:37 AM

Paper page - RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling

Source: https://huggingface.co/papers/2609.22947

Abstract

Reinforcementlearning(RL)isvitalforoptimizingvideogenerationmodels,witharobustrewardmodel(RM)servingasthecornerstone.However,existingvideorewardmodelsoftenproduceunstablescalarscoresbecausetheydirectlymapcomplex,subjectivevideoqualityintoasinglescorewithoutexplicitevaluationcriteria.Thisleadstoscalardrift,wherethescoringscalecollapsesorshiftsacrossdifferentprompts,makingtherewardunreliableforRL.Drawinginspirationfromprofessionalhumanannotationengineering,weaddressthisproblemwithRewardVerse,arubric-basedvideorewardframeworkthatintroducesadynamicrubricasanintermediaterepresentationbetweentheevaluationqueryandthescorer.Insteadofunconstraineddirectscoring,RewardVersefirstgeneratesexplicitevaluationcriteriaandthenperformsrubric-guidedscoring,providingastablesemanticanchorthatmitigatesscalardrift.Toefficientlyoptimizethiscollaborativepipeline,weproposeRubric-GuidedPolicyOptimization(RGPO),atwo-stagetrainingalgorithm.RGPOfirstwarmsupthescorerusingself-evolvingseedrubricsandthenjointlyoptimizestherubricgeneratortoproducequery-adaptiveevaluationcriteriawhilecontinuouslyaligningthescorerwithhumanratings.Extensiveexperimentsonthe16-dimensionalEvalVersebenchmarkandexternaldatasetsdemonstratethatRewardVersemitigatesscalardrift,achievesstate-of-the-artperformanceonbothpointwiseandpairwiseevaluation,andprovidesarobustandinterpretablerewardsignalforRLinvideogeneration.

View arXiv pageView PDFProject pageGitHub3Add to collection

Get this paper in your agent:

hf papers read 2609\.22947

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.22947 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.22947 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.22947 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR

Hugging Face Daily Papers

This paper introduces POW3R, a policy-aware rubric reward framework for reinforcement learning with verifiable rewards (RLVR). It shows that static rubric aggregation misallocates learning signal, and POW3R achieves faster convergence and better performance across multiple settings.

Video Models Can Reason with Verifiable Rewards

Hugging Face Daily Papers

VideoRLVR optimizes video diffusion models for verifiable reasoning tasks using reinforcement learning with rule-based rewards, achieving better performance than supervised methods in constraint-satisfying video generation.

C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences

Hugging Face Daily Papers

C2 proposes a scalable rubric-augmented reward modeling framework that trains a cooperative rubric generator and critical verifier exclusively from binary preferences, eliminating the need for costly rubric annotations while achieving up to 6.5 point gains on RM-Bench.