@wenhaocha1: Really like this figure!
Summary
A developer discusses why tail reinforcement learning remains effective even when the initial policy lacks good coverage of target behaviors.
View Cached Full Text
Cached at: 09/08/26, 01:32 PM
Really like this figure!
Shrinivas Ramasubramanian (@stablegradients): We can throw more training compute at the problem and the objective optimizes a weighted combination of Best-of-k.
But what if the initial policy is really bad? These objectives don’t matter, right?
Wrong🚫
TailRL works even when the initial policy has poor coverage over good
Similar Articles
Reinforcement learning towards broadly and persistently beneficial models (22 minute read)
OpenAI researchers show that reinforcement learning on realistic scenarios targeting beneficial traits (honesty, transparency, corrigibility) produces broad improvements across dozens of alignment benchmarks, with gains generalizing beyond training domains and persisting under adversarial pressure.
Tail-Likelihood Reinforcement Learning
The paper proposes Tail-Likelihood Reinforcement Learning (TailRL), an optimization method that focuses on the upper tails of reward distributions to improve policy performance in generative tasks, demonstrated across various applications.
@svlevine: We can learn a model that provides shaped "process rewards" for robotic RL, that evolves automatically as the policy ge…
This work presents a model that learns shaped 'process rewards' for robotic reinforcement learning, which evolves automatically as the policy improves, enhancing performance on benchmarks and in real-world settings.
Reinforcement Learning Towards Broadly and Persistently Beneficial Models
This paper from OpenAI investigates whether reinforcement learning on beneficial behavior can produce broad and persistent alignment generalization beyond the training distribution. Using a dataset of realistic situations, they show that RL training on beneficial traits improves out-of-distribution alignment and persistence against adversarial attacks.
@mervenoyann: interesting talk by @willcb
This talk by Will Brown of Primordial AI discusses techniques for scaling Reinforcement Learning to complex, real-world tasks where rewards are not verifiable, using methods like anchoring, LLM judges, and simulation.