Tag
CSPF proposes a constrained shared-private fusion method to integrate representations from multiple reward models for non-verifiable preference evaluation, outperforming baselines.
This paper argues that reward models in RL are often oversensitive, assigning different scores to equally good responses, and proposes a training-free discretization algorithm using Monte Carlo dropout to reduce oversensitivity, improving policy quality.
This paper introduces 'progress advantage', an implicit advantage function derived from reinforcement learning post-training that enables effective step-level scoring for LLM agents without requiring dedicated reward model training. It outperforms confidence-based baselines and trained reward models across multiple benchmarks and model families.
This paper identifies oversensitivity in continuous reward models for reinforcement learning, where equally good responses receive different scores, and proposes a discretization technique using Monte Carlo dropout to reduce this oversensitivity while maintaining discriminative ability, leading to better policies and less reward hacking.
Introduces SCPO, a novel reward model training algorithm that incorporates diverse cultural preferences in a balanced manner, achieving up to 7 points improvement and 280% data efficiency over baselines.
Proposes Demo2Reward, a test-time prompt optimization technique for VLM reward models using a few expert demonstrations, significantly reducing false positives and improving policy learning in robotics without additional model training.
The SAVE framework improves reward model training by using value functions to grade on-policy responses and update models through contrastive objectives, achieving outperforming results across six benchmarks.
This paper investigates preference instability in reward models for LLMs, where subtle input variations cause contradictory preference assignments. The authors propose two SAE-based mitigation strategies—SAE Feature Steering and SAE Residual Correction—to reduce incorrect preference assignments without retraining.
This paper introduces Test-Time Personalization (TTP), a framework that improves LLM personalization by scaling inference-time computation through candidate sampling and reward-based selection. It diagnoses failure modes in standard reward models and proposes a probabilistic personalized reward model to mitigate them.
OpenAI researchers empirically study how reward model overoptimization affects performance, establishing scaling laws that show the relationship between proxy reward optimization and ground truth performance varies by optimization method and scales predictably with model size.