@mervenoyann: interesting talk by @willcb

interesting talk by @willcb https://t.co/5z1xlyWod2

TL;DR: This talk explores moving beyond Reinforcement Learning with Verifiable Rewards (RLVR) to tackle complex, real-world tasks where rewards are ambiguous, focusing on techniques like anchoring, LLM judges, and simulation to create scalable learning signals.

Beyond Verifiable Rewards: Scaling RL for the Real World

Will Brown of Primordial AI’s Applied Research team addresses a core challenge in modern AI: while Reinforcement Learning with Verifiable Rewards (RLVR) has become a powerful scaling tool, most practical, messy-world tasks lack clear, verifiable success criteria. This talk synthesizes the team’s research and tools aimed at extending RL to these more complex domains.

The RL Landscape and Primordial AI’s Tools

The speaker begins with a concise review of RL fundamentals. An agent (model) interacts within an environment composed of tasks, a runtime framework, and a reward mechanism. The goal is to update the model’s policy to maximize rewards using policy gradient algorithms like GRPO, REINFORCE, or SISPO.

Primordial AI has built a full-stack toolkit to support this pipeline, including:

• Large-scale GPU scheduling.
• The Primordial RL training framework.
• A composable environment module containing tasks, runtimes, and verifiers.
• Lab, a training platform centered on environments, offering managed training, evaluation, and inference.

A core mission is to democratize this process, lowering the barrier for researchers and teams to fine-tune open-source models for their specific needs using RL.

Defining the “Environment”

An environment is defined as the combination of tasks, a runtime framework, and a reward mechanism. It is a versatile concept applicable not just to RL but also to:

• Generating static data for Supervised Fine-Tuning (SFT).
• Running policy distillation.
• Prompt optimization (e.g., DSPy).
• Acting as a scientific platform for iterating on agents and runtimes.

The Challenge of Unverifiable Rewards

Verifiable rewards are straightforward in domains like math (checking a numerical answer) or code (running test cases). However, most real-world agent tasks—writing reports, handling bookings, customer service—lack a single, clear “correct” answer.

The difficulty in evaluation stems from:

• Scalability: Current benchmarks are labor-intensive to create and lack breadth.
• Open-Endedness: Tasks often have unbounded possibilities with no clear judgment standard.
• Reward Hacking: In RL, poorly defined proxy rewards can lead models to exploit loopholes instead of fulfilling the true intent.

The ultimate goal is continuous learning—deploying agents that improve autonomously through experience in complex environments, moving beyond low-level, impractical abstractions.

Building Learning Signals: Practical Techniques

To create reward signals for unverifiable tasks, the team has found several effective techniques:

Anchoring: Leveraging Source Material

This involves using a reliable source to create a supervision signal. For example, an agent’s performance can be compared with and without access to a “source material” (like a document or a human-generated trace). The measurable capability gap creates a learnable signal, effectively teaching the agent to depend on and utilize contextual information.

LLM Judges

Since LLMs are strong general-purpose reasoners, they can be deployed as “judges” to evaluate an agent’s behavior. By spending inference compute, they can judge the quality of actions, providing a scalable reward signal.

Extending Search

Search can be applied at multiple stages: to generate tasks, to design environments, and to formulate criteria for the judges themselves.

Using Production Data as Source Material

A particularly efficient source for anchoring, especially for continuous learning, is the agent’s own production trajectory. As an agent is deployed, it accumulates user prompts and interaction traces. This data represents the true task distribution. While initially unsupervised, it provides a realistic foundation for exploration and learning.

Inverting Problems for Free Supervision

The core idea is to work backward from a known-good state.

• For Documents: Sample a document, use an LLM to generate Q&A pairs, verify if answers are derivable from the document, and then remove the initial generation process. This reverse-engineers a task from its solution.
• For Code: Utilize real pull requests (diffs, descriptions, tests). By strategically deleting parts of code, you can create tasks that require the model to relearn and re-implement the functionality, working back from the completed state.
• For Complex Systems (Web Apps, Tools): Build simulators. By analyzing production trajectories and using techniques like test-time compute scaling and search, high-fidelity simulators can be learned. These simulators allow you to “implant the answer”—even if the real environment is unverifiable, the simulator provides full control and a verifiable endpoint for RL.

Scaling Judgment

As an agent executes tasks, errors inevitably occur. Looking back at the complete sequence of actions makes it easier to identify where things went wrong. This hindsight can be used to pinpoint error locations. Furthermore, having multiple models judge the same action can increase confidence; when consensus is reached on a flaw, it provides a stronger signal for correction.

Source: https://www.youtube.com/watch?v=AQv3qRCG6Gw