The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement
Summary
The SAVE framework improves reward model training by using value functions to grade on-policy responses and update models through contrastive objectives, achieving outperforming results across six benchmarks.
View Cached Full Text
Cached at: 06/01/26, 07:18 AM
Paper page - The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement
Source: https://huggingface.co/papers/2605.30888
Abstract
SAVE framework improves reward model training by using value functions to grade on-policy responses and update models through contrastive objectives.
Building strongreward models(RMs) for language model alignment is bottlenecked by the cost and difficulty of acquiring diverse and reliable preference data from human annotation or judge models. It is dramatically worse as the policy evolves beyond the static RM training. Therefore, we propose SAVE (Self-supervised reward model improvement via Value-Anchored On-policy feedback), a framework that gradeson-policy responsesas feedback by using thevalue functionfor on-policy RM training. SAVE naturally converts the reward-gradedon-policy responsesinto supervision with a prompt-specific value head as an adaptive anchor. It computes RM advantages and filters ambiguous samples to update the RM via acontrastive objective. The effectiveness of SAVE for enhancing RM training is strongly validated through rigorous empirical evaluation across six diverse benchmarks. It achieves outperforming results across all datasets while maintaining consistent improvements across three RL algorithms (GRPO,RLOO,GSPO) and different policy backbones.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.30888
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.30888 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.30888 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.30888 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards
Proposes Correction-Oriented Policy Optimization (CIPO), an extension to RLVR that converts failed trajectories into correction-oriented supervision, improving reasoning and correction performance in LLMs across math and code benchmarks.
Reward Models Can Be Too Sensitive (22 minute read)
This paper argues that reward models in RL are often oversensitive, assigning different scores to equally good responses, and proposes a training-free discretization algorithm using Monte Carlo dropout to reduce oversensitivity, improving policy quality.
Verifier-Induced Support Reshaping in On-Policy Optimization
The paper introduces verifier-induced support reshaping, showing that on-policy RL with verifiable rewards can improve the current objective while making successful behaviors for later objectives too rare to sample. Experiments across math reasoning and instruction following demonstrate that endpoint improvements do not guarantee future trainability.
@svlevine: We can learn a model that provides shaped "process rewards" for robotic RL, that evolves automatically as the policy ge…
This work presents a model that learns shaped 'process rewards' for robotic reinforcement learning, which evolves automatically as the policy improves, enhancing performance on benchmarks and in real-world settings.
RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling
RewardVerse presents a rubric-based framework for video reward modeling to mitigate scalar drift and provide stable evaluation criteria, enhancing reinforcement learning in video generation.