@ishapuri101: It's never made sense to me that RL collapses all reward signals to a single scalar. Today, we fix that! Introducing Ve…
Summary
Introduces Vector Policy Optimization (VPO) to train models with vector-valued rewards instead of scalar rewards, enabling diverse answer sets for test-time search.
View Cached Full Text
Cached at: 05/23/26, 10:16 PM
It’s never made sense to me that RL collapses all reward signals to a single scalar. Today, we fix that!
Introducing Vector Policy Optimization: we train models to inherently optimize for the varied nature of a reward vector, creating diverse sets of answers ideal for test time search. Website and code coming soon!
Ryan Bahlous-Boldi (@RyanBoldi): Your RL post-training may be sabotaging your LLM’s test-time scaling!
Conventional RL pretends that you can collapse all reward signals upfront into a single scalar reward. We introduce Vector Policy Optimization (VPO), which natively maximizes vector-valued rewards,
Similar Articles
@RyanBoldi: Your RL post-training may be sabotaging your LLM’s test-time scaling! Conventional RL pretends that you can collapse al…
Introduces Vector Policy Optimization (VPO), a new RL method that handles vector-valued rewards to improve test-time scaling for LLMs, outperforming conventional scalar reward approaches.
Vector Policy Optimization: Training for Diversity Improves Test-Time Search
This paper introduces Vector Policy Optimization (VPO), a reinforcement learning algorithm that trains LLMs to produce diverse solutions by optimizing across multiple reward dimensions, significantly improving test-time search performance compared to scalar RL baselines.
RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling
RewardVerse presents a rubric-based framework for video reward modeling to mitigate scalar drift and provide stable evaluation criteria, enhancing reinforcement learning in video generation.
Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL
This paper proposes PRISM, a multi-reward RL framework that decomposes policy space rather than mixing rewards, improving multi-reward optimization and enabling inference-time controllability. Experiments on reasoning and alignment tasks show it outperforms existing baselines.
SLPO: Scaling Latent Reasoning via a Surrogate Policy
Introduces Surrogate Latent Policy Optimization (SLPO) to apply outcome-reward RL to autoregressive latent reasoners, enabling test-time scaling and variable-horizon policies that improve accuracy on harder instances.