Introducing RLVP, Penalize the Path, Reward the Outcome: a paper I collaborated with Noah Shi from University of Washington published on ArXiv yesterday.

Most RL for agents rewards only the final outcome. But agents that act in the real world — making phone calls, resolving tickets — can fail even when they succeed: calling a user who declined contact, acting outside business hours, skipping authentication.

The outcome reward is blind to the path. In GRPO-style group-relative RL, advantage is within-group variance. A binary outcome reward goes to zero variance on all-fail groups (early training) AND all-success groups (late training).

A dense signal only helps if it supplies the variance the outcome lacks.

Our insight: Many real-world agentic environments are asymmetric verifiers: they can cheaply detect bad moves (rm -rf, calling before preconditions are met), but can’t certify progress.

The field has been spending verifiers on their weakest direction. Invert it: penalize the path, reward the outcome.

Result: near-zero constraint violations at equal task success, where outcome-only training breaks the rules on nearly every episode. On TerminalBench, ~6× fewer destructive actions at matched success, and the agent issues more productive commands, not fewer.

The same verifiable channel, paid as credit for verified progress, becomes a dense potential, turning dead all-fail rollouts into useful gradient and improving sample efficiency. But it’s reachability-gated: it only helps where partial progress actually happens (theorem proving , software repair ).

One trap: a penalty alone collapses into inaction: the optimal policy is to do nothing. So we need both sides: penalize the path, reward the outcome.

Paper: https://arxiv.org/abs/2607.07435 Code: https://github.com/19PINE-AI/rlvp Website: https://01.me/research/rlvp

RLVP: Penalize the Path, Reward the Outcome

Source: https://arxiv.org/html/2607.07435

Abstract

Agents acting on our behalf in the real world—placing phone calls, resolving support tickets—must learnonlinefrom costly, often irreversible interactions rather than cheap simulator steps. Two things follow. First, deployability depends on thepath, not only the outcome. An agent must respect outcome-neutral constraints such as not repeatedly calling an unresponsive user, respecting business hours, or completing required authentication—constraints that outcome-based rewards cannot express, since violating them frequently improves apparent success. Second, because each interaction is expensive, the agent must learn efficiently from very few examples.

Reinforcement learning from verifiable rewards (RLVR) is blind to both challenges: it optimizes solely on the outcome and wastes expensive rollouts on all-fail groups where group-relative advantage collapses to zero. Attempts to densify supervision by rewarding progress target the hard-to-verify direction. In contrast, real agentic environments can cheaply detect bad moves. Since group-relative advantage is equivalent to within-group variance, a dense signal helps only when it supplies variance the outcome lacks. A verifiablepenaltyon the path meets this condition reliably, while a progresspotentialhelps only where partial progress is reachable. The resulting recipe—penalize the path, reward the outcome—achieves high task success with near-zero violations, where outcome-only training violates constraints on nearly every episode. We provide four design rules for effective penalties, including avoidance of theinaction trapthat arises when a penalty is used in isolation.

Figure 1:Two verifiable path signals for agents that learn from costly, real-world rollouts.*(a) Penalize the path:rewarding the outcome while penalizing the path drives violation-free episodes from near zero to≈\approx100%—a deployability constraint outcome-only GRPO is blind to and never reaches—at no cost to task success (§3).(b) Reward progress where reachable:*a dense potential reaches competence in fewer costly rollouts than outcome-only training (§4). Mean over seeds; bands±\pm1 s.d.## 1 Introduction

Consider an agent that acts on behalf of a person in the real world: it calls a bank, navigates through the automatic voice menu, waits when put on hold, and eventually speaks with a representative to resolve a billing dispute. Every action is a real, often irreversible interaction—a genuine call rather than a resettable simulator step that can be replayed millions of times. A rapidly growing class of deployed agents operate in this high-stakes regime(Pine AI,2026b). Unlike the reversible, self-contained environments of math problems, coding benchmarks, and simulated games that shaped today’s foundation models, these agents must contend with partial observability, irreversible consequences, long feedback delays, and genuine uncertainty about the state of the external world. This regime differs from the reversible, high-throughput environments used to train today’s models in two fundamental ways.

First, deployability depends on thepaththe agent takes, not the outcome alone. A successful resolution is necessary but insufficient. An agent that reaches the right outcome by calling a user who has explicitly declined contact, phoning outside business hours, or bypassing a bank’s required authentication steps is not deployable—no matter how favorable the final result. Outcome-based rewards cannot express these constraints: violating them oftenincreasesthe apparent success rate, because shortcuts and rule-breaking tend to accelerate resolutions in the short term. A deployable agent must therefore satisfy both outcome and path constraints simultaneously.

Second, these agents must learnonlinefrom their own live interactions, and they must do so from very few examples. In math and code benchmarks, cheap simulators allow policies to generate millions of rollouts, making sample efficiency secondary—slow learning can be compensated by simply running more trials. In the real world, a phone call cannot be replayed. An agent that learns from a single interaction that a particular bank requires a customer’s date of birth and the last four digits of their account number must generalize that knowledge immediately to the next user. Here, sample efficiency is not merely an optimization concern; it is the difference between a system that meaningfully improves over time and one that cannot be deployed at all.

Reinforcement learning from verifiable rewards (RLVR)(DeepSeek-AI,2025; Lambertet al.,2024)is fundamentally blind to both challenges. It optimizes solely on the one signal the environment can evaluate cheaply—the final outcome—rendering the path invisible. This method also wastes its most expensive interactions on all-fail trajectories where group-relative advantage is zero and no learning occurs. The natural response is to seek denser rewards by creditingprogress(Lightmanet al.,2024; Wanget al.,2024; Setluret al.,2025; Yu and others,2024; Feng and others,2025). Yet in agentic settings, judging progress is precisely the hard problem the agent must solve. This leads either to a learned critic that the policy eventually fools(Cheng and others,2025; Kazemnejad and others,2024; Gaoet al.,2023)or to brittle, hand-crafted proxies.

Real agentic environments areasymmetric verifiers: they can cheaply and reliably detectbad moves(e.g., calling before preconditions are met or acting outside business hours), but cannot certify that an agent is making meaningful progress. The reliable dense signal they provide is therefore apenalty on the path, not a reward for progress. Contrary to folklore, penalties are not inherently problematic; the common collapse into inaction is a wiring error, not an inevitability. A penalty in isolation indeed fails—its optimal policy is to do nothing, falling into aninaction trap. However, when paired with an outcome reward—reward the desired result, penalize forbidden paths—the penalty effectively teaches precisely the outcome-neutral constraints required for deployment, leveraging the verifiable side of the environment.

Figure 2:Why the path channel creates gradient where the outcome cannot.Four rollouts of the same bug-fix task, all of which fail the hidden test (outcome is0for every trajectory).Left (outcome-only):All four rollouts receive identical reward equal to the group mean. The advantage is zero everywhere, so the policy update is dead.Right (with the verifiable path channel):The same rollouts receive different path scores—a bonus+β+\betafor running tests or making measurable progress, and a penalty−λ-\lambdafor destructive commands or editing test files. This creates within-group variance, producing a meaningful gradient even at0%0\%task success.The same asymmetry has a useful second edge. When the environmentcanverify a step of progress—a precondition satisfied, a subgoal reached—that signal can be used as a densepotentialto densify the otherwise sparse outcome reward. This is particularly valuable on the all-fail groups that otherwise waste rollouts early in training. A single principle underlies both mechanisms: group-relative advantage is simply within-group variance. The pure outcome signal is blind at both extremes—zero variance on all-failing groups early in training and on all-succeeding groups late in training. A dense auxiliary signal is useful only to the extent that it supplies variance the outcome lacks, and only in states the policy can actually reach. A verifiable penalty satisfies this by construction: bad actions are always detectable. A progress potential, by contrast, is reachability-gated—it helps only once the policy begins to enter states where partial success is possible. The penalty is the universally available half of the solution; the potential is the conditional half. Figure1previews both. We validate these ideas on tractable proxies for the target deployment setting, including system-administration and customer-service tasks, a shell benchmark, a theorem prover, and software repair. The claimed transfer is mechanistic: the same principles, not the same numerical results, are expected to apply in live production traffic.

Contributions.

• •A within-group-variance accountof when dense signals help in group-relative agentic RL. A dense signal is useful precisely when it supplies reachable within-group variance that the outcome reward lacks (§2). This view is symmetric—pure outcome-based RL is blind on both all-fail groups (early training) and all-success groups (late training), as shown in Figure2—and unifies the two complementary uses of a verifiable path channel: penalties for bad moves and credit for verified progress.
• •Penalize the path(§3). We introduce a per-action verifiable penalty, combined with the outcome reward and governed by four design principles (including mitigation of the inaction trap). This approach reliably teaches outcome-neutral deployment constraints. On a real benchmark it reduces harmful actions by nearly sixfold at equal success rate, driving violation rates from nearly every episode down to near zero.
• •Reward verified progress(§4). Using thesameverifiable channel for credit rather than punishment turns it into a densepotential. This directly addresses sample efficiency in online learning: it converts dead all-fail updates into useful gradient wherever partial progress is reachable, while remaining inert where it is not.

Figure 3:Outcome-only RL is blind at both ends of the success rate.Group-relative advantage equals within-group variance. A binary outcome signal drives this variance to zero both on all-fail groups (early in training) and all-success groups (late in training). A dense process signal is useful only insofar as it supplies variance in these regimes—and only in states the policy can actually reach. A verifiable penalty reliably provides this signal (bad moves are easy to detect), whereas a dense progress potential does not (partial success is rare on hard tasks).

2 The Verifiable Path Channel

Group-relative RL.

Modern agentic RL has largely moved away from learned value functions. Instead, it estimates advantages by comparing trajectorieswithin a groupsampled for the same prompt or task(Shaoet al.,2024; DeepSeek-AI,2025). For a given prompt, the method samples a group ofGGtrajectories, assigns each a reward, and computes its advantage as the deviation from the group mean (often normalized by the group standard deviation). There is no separate critic: the other trajectories in the same group serve as the baseline. This design is precisely why verifiable outcome rewards are so effective—a rule or test suite can score a trajectory directly—and it forms the foundation for all the techniques developed in this paper.

The advantage is a within-group variance.

Since the baseline is the group mean, a trajectory only contributes gradient when its rewarddiffersfrom the others in its group. When all rollouts receive identical rewards, every advantage is zero and the group yields no policy update. With a binary outcome reward, this degeneracy occurs in two dominant regimes (Figure3):all-failgroups, which prevail early in training on long-horizon tasks, andall-succeedgroups, which prevail once the task is nearly solved. Consequently, group-relative RL is blind at both extremes of the success rate. The community has only partially addressed this bydiscardingsuch groups rather than imbuing them with learning signal—for example, DAPO drops prompts that are uniformly correct or uniformly incorrect(Yuet al.,2025).

When a dense signal can help.

A dense process signal is useful exactly when it restores within-group variance that the outcome has lost. Consider the shaped reward of a trajectory in the form of the outcome plusβ\betatimes a process term,R=O+β​ΦR=O+\beta\,\Phi. The group’s reward variance—the quantity that drives the policy gradient—then decomposes as

VarG⁡(R)=VarG⁡(O)+β2​VarG⁡(Φ)+2​β​CovG⁡(O,Φ).\operatorname{Var}_{G}(R)\;=\;\operatorname{Var}_{G}(O)\;+\;\beta^{2}\,\operatorname{Var}_{G}(\Phi)\;+\;2\beta\,\operatorname{Cov}_{G}(O,\Phi).(1)On all-fail or all-success groups,VarG⁡(O)=0\operatorname{Var}_{G}(O)=0. Therefore,allusable gradient must come from the process term’s own within-group varianceVarG⁡(Φ)\operatorname{Var}_{G}(\Phi)(Figure2). This yields a simple, verifiable condition: a process signal helps only if it produces nonzero within-group variance, which occurs only when the policy reaches differing intermediate states across the trajectories in the group. Both conditions—finer granularity than the outcome and reachability—are satisfied by a single per-action verifiable channel, used in two complementary ways. The channel attaches a verifiable signal to the action that triggers it: a penalty−λ-\lambdafor a verifiedbadmove (e.g., a call before its precondition is met), a credit+β+\betafor a verifiedgoodmove (satisfying a precondition or reaching a subgoal). When used as apenalty, this signal almost always generates within-group variance early in training: some rollouts in the group take the bad action while others do not. Both cases are cheap to sample and verify, making the penalty thealways-reachablesource of gradient. When used as apotential—paying the same+β+\betacredit for verified progress—the signal provides variance only in states where the policy reaches differing levels of partial success. On hard tasks this is infrequent (Figure3, right), making the potential thereachability-gateduse. The remainder of this paper explores these two uses of one verifiable path channel: penalizing the path to achieve deployability (§3) and rewarding verified progress to improve sample efficiency (§4).

3 Penalizing the Path: Verifiable Constraints for Deployable Agents

This section develops the penalty use of the verifiable path channel. We present a two-channel approach that combines a per-action penalty with the standard outcome reward (§3.1). We then introduce a practical recipe that enables agents to achieve their tasks while reliably respecting outcome-neutral deployment constraints. This is validated on controlled proxies as well as a real agentic benchmark (§3.2). Finally, we articulate four design rules that distinguish a robust, effective penalty from one that collapses into an inaction trap (§3.3).

3.1 The two-channel method

Figure 4:The verifiable path channel in the real-world (phone-agent) setting.*Left (RLVR):*Outcome-only training receives only a terminal reward and remains blind to the path. Violations such as over-calling or acting without proper authentication go unpenalized.*Right (RLVP):*A deterministic per-action rule engine attaches an immediate verifiable signal to the responsible action—a penalty−λ-\lambdafor bad moves (e.g., over-calling) and a credit+β+\betafor good moves (e.g., successfully discharging a precondition)—while retaining the original outcome reward. The penalty use (§3) enforces path constraints for deployability. The same+β+\betacredit, when paid for verified progress, serves as the dense potential introduced in §4.#### Two channels.

We keep the outcome reward and introduce a second, per-action channel (Figure4). At each step, a deterministic rule engine—a pure predicate over the pre-action state and the action taken—evaluates the action. Aviolation(e.g., issuing a destructive command or calling before a required precondition is met) triggers a penalty−λ-\lambdaon the offending action’s tokens. Conversely, afulfillmentof a pending obligation (e.g., successfully performing the required precondition) triggers a credit+β+\beta. The two channels are normalized separately to prevent the sparse path signal from being diluted or overwhelmed by the outcome reward, and are then combined. “Penalize the path” is therefore literal: the penalty directly reduces the probability of the specific bad action in the precise context in which it occurs, while the outcome reward provides global trajectory-level credit for success. We call signals produced by this engineverifiable penalties. Importantly, the constraints we penalize areoutcome-neutral—violating or respecting them does not determine whether the overall task is solved. This is precisely the type of signal that a pure outcome reward can never supply. The complete recipe also includes pairing each penalty with its corresponding fulfillment credit, seeding a small number of scripted compliant demonstrations to ensure compliant behavior is reachable early in training, and annealing the path shaping once compliance saturates. Section3.3demonstrates that each of these components is essential.

3.2 The recipe attains the task and its constraints

Figure 5:Rewarding the outcome while penalizing the path satisfies both task success and constraints.Outcome-only training solves the task but violates outcome-neutral rules on nearly every episode. The two-channel method maintains high success while achieving near-100% violation-free episodes across tasks and model scales (mean ± std over 5 seeds).#### On controlled proxies.

We evaluate on a suite of diverse system-administration and customer-service tasks that serve as controllable proxies for the real deployment setting. These tasks encode outcome-neutral rules of §1 in miniature:verify a precondition before acting,do not repeat a request that was just refused, andread before you mutate. When trained with outcome-only rewards on held-out instances drawn from a large pool, agents learn to solve the tasks but violate these rules on nearly every episode. The constraints are invisible to the outcome reward, just as calling outside business hours is invisible to whether a billing dispute is ultimately resolved. Adding the penalty channel—reward the outcome, penalize the path—drives the violation rate to near zero while preserving high task success rate. This holds across five seeds at 4B and generalizes from 1.7B to 8B models (Figure5). Importantly, the agent does not achieve this by becoming passive: it takes roughly the same number of actions as before, but selects much cleaner ones. This result demonstrates the core benefit in its clearest form: a verifiable penalty teaches genuine deployable behavior—respecting outcome-neutral constraints—that a pure outcome reward cannot.

Table 1:Harm reduction at equal outcome (TerminalBench, Qwen3-4B, 5 seeds, mean±\pmstd).Task success is statistically equal and at the floor (4B rarely solves the benchmark; the small RLVP–outcome gap is within one standard deviation), yet the un-gameable harm penalty cuts harmful actions roughlysixfold. The policy is not going passive: it takes about three timesmoreproductive actions while violating less. Harm lives on an axis the terminal outcome cannot see.

It transfers to a real agentic benchmark.

While the synthetic domains allow precise control over the rules, we also test the same mechanism in a more realistic setting using TerminalBench. In this benchmark, each turn consists of a real shell command executed in a task-specific container, and the environment can flag genuinely destructive actions (e.g.,rm -rfor dropping a database) against a live filesystem—shortcuts that outcome-only training would otherwise exploit. On this benchmark, task success for smaller models is near the floor, making it an especially clean test of the harm-reduction axis.At equal task success rates, the verifiable harm penalty reduces destructive actions by a factor of six compared to outcome-only training, while the policy actually issuesmoreproductive commands overall (Table1, five seeds). The reduction is large relative to seed variance, with non-overlapping per-seed distributions, and is clearly not the result of passivity. A verifiable penalty bounds the path independently of task success, which is the exact property a deployable agent needs.

3.3 Design rules for the penalty

Figure 6:A design procedure for verifiable penalties.Before introducing a penalty, each rule can be checked against four criteria: (1) Is there a concrete, verifiableactionto penalize (rather than the absence of progress)? (2) Does the outcome reward remain the primary driver of task completion? (3) Is the penalty paired with a corresponding credit for the compliant action? (4) Is the compliant action reachable early in training, and is its target difficult to game? Only when all four conditions are satisfied does the penalty successfully bound the path without stalling the task.Figure 7:Each design element earns its place.On diverse held-out tasks, outcome-only training solves the task but exhibits high violation rates (low clean rate). Adding the verifiable penalty channelwithoutits complementary fulfillment credit and reachability seeding fails to reliably produce both clean and successful behavior across seeds. Only the full recipe—penalty + fulfillment + seeded compliant demonstrations + annealing—consistently attains high task successandnear-zero violations with low seed variance. The pure-penalty inaction trap (a penalty without any outcome reward, which collapses to zero success on every seed) is isolated separately in the ungamability sweep (AppendixB.1, Figure20). Results are means over five seeds; whiskers show one standard deviation.While a verifiable penalty is powerful, it is also easy to apply incorrectly. The variance account together with our ablations yield four design rules that distinguish the robust behavior shown above from policies that collapse into inaction (Figure6).

1. Penalize verifiableactions, not lack of progress.

A good penalty targets a specific, machine-checkable bad action—for example, executing a destructive command or issuing a call before its precondition is satisfied. It must not target the mereabsenceof progress (e.g., “penalize an unproductive step”), because the cheapest way to make no unproductive steps is to take no steps at all. Therefore: penalizecommission, never omission.

2. Keep the outcome reward as the task driver; never optimize a penalty in isolation.

A penalty must not become the main learning signal. In the all-fail regime, it is often theonlysignal with nonzero gradient, and its cheapest optimum is to stop acting entirely. The policy then drifts into a penalty-free but useless solution—theinaction trap. A controlled sweep confirms this empirically: a pure penalty (with no outcome reward) collapses task success to zero oneveryseed, whereas any penalty-free configuration continues to make progress (AppendixB.1, Figure20). The outcome reward is what compels the agent to keep acting toward task completion. The penalty merely shapeshowit does so. In short: reward the outcome; let the penalty steer.

3. Pair the penalty with a corresponding fulfillment credit.

A penalty only tells the policy whatnotto do. Pairing it with a fulfillment credit (+β+\beta) for the compliant action creates an active pull toward the desired behavior. This helps the policy move decisively away from the bad action rather than merely avoiding it. Ablations show that removing the fulfillment credit measurably slows learning and increases instability in acquiring compliant behavior (Figure7).

4. Make the compliant behavior reachable and its target un-gameable.

A fulfillment credit can only reinforce behavior that the policy actually samples. If the compliant path is never explored, the credit has no effect. To ensure reachability, we seed training with a small number of scripted compliant demonstrations. Once compliance saturates, the path shaping can be safely annealed away. Additionally, the penalized (and credited) target must be a concrete, verifiable action rather than a proxy. Using a learned judge of “compliance” simply moves the gaming problem into the judge itself (AppendixB).

Together, these four rules delineate a narrow but reliable regime for effective path penalties. The penalty must be verifiable and outcome-neutral, must accompany (rather than replace) the outcome reward, must be paired with a corresponding fulfillment credit, and must target reachable and un-gameable actions. The ablation in Figure7progressively builds from outcome-only training through partial versions of the penalty channel (without fulfillment credit or reachability seeding) to the full recipe. Compliant and successful behavior emerges robustly only when all four rules are satisfied. The pure-penalty inaction trap—where a penalty is used without any outcome reward—is isolated separately in the sweep shown in AppendixB.1.

4 Reward Verified Progress: Sample Efficiency Where Reachable

Figure 8:The aligned potential in action on a realminiF2Ftheorem.Onmathd_algebra_109, the Lean kernel verifies each tactic and the proof’s remaining obligations fall3→2→1→03\!\to\!2\!\to\!1\!\to\!0. The outcome reward issparse: it pays a single+1+1only when the proof is completed (■\blacksquare). The aligned potential isdense: every kernel-verified drop in obligations pays a+β+\betafulfillment credit (while an errored tactic in another rollout pays−λ-\lambda). On the all-fail groups early in training—where no rollout completes the proof—trajectories that make more progress receive higher path scores. This supplies the within-group variance, and therefore the gradient, that the pure outcome reward cannot provide.Figure 9:Dense potentials are reachability-gated.Left:On software repair, the finer-grained potential is unreachable. Across many rollouts, the policy makes zero partial progress, yielding zero within-group variance and thus no gradient.Right:Where the potentialisreachable, it eliminates the dead all-fail updates that dominate early training (0 vs.∼\sim16 of 40 iterations per seed). It supplies gradient precisely where the outcome reward is blind. The limiting factor is reachability, not fragility.The path channel’s credit has a second use. In §3the+β+\betacredit was paired with a penalty to reward compliant actions. Here, we pay thesamecredit for verifiableprogress—such as a reduction in remaining goals in a proof, an increase in the fraction of tests passed, or a precondition being satisfied. This turns the signal into a densepotential(Figure8) that directly addresses the second key challenge of real-world deployment: learning efficiently from very few costly interactions. By densifying the sparse outcome reward on all-fail groups (which otherwise waste expensive rollouts), the potential enables the policy to reach competence in fewer interactions. We study this approach in domains where partial progress is well-defined and cheap to evaluate—theorem proving and software repair. These serve as controllable proxies that allow extensive experimentation and multiple seeds, which would be infeasible in a live phone-agent setting. The within-group variance account applies directly: the credit supplies useful gradient only where the policy reaches intermediate states with differing progress levels. The determining factor isreachability. On software repair, for example, a natural potential is the fraction of hidden tests passed during an episode. For many capable models,everyrollout achieves zero test passes early in training, so the potential has zero within-group variance and provides no signal. Reachability can be diagnosed before training using a small number of rollouts from the base policy, and the benefit of adding a dense potential closely tracks this diagnostic. In short, reachability—not fragility—is the true gate.

Figure 10:The aligned-potential recipe on real theorems (miniF2F algebra), five seeds per configuration.Thin lines show individual seeds; thick lines show the mean over stable seeds;×\timesmarks a divergence-guard abort.(a)Controlled same-optimizer comparison at 4B scale: the aligned potential crosses the0.90.9threshold in4.4±0.54.4\pm 0.5iterations vs.7.0±0.77.0\pm 0.7for outcome-only. It is faster on every seed and never diverges, compared to outcome-only which loses one seed to entropy collapse.(b)At 30B scale, outcome-only exhibits a speed–reliability trade-off. Under Muon it diverges on three of five seeds; under AdamW it is stable but takes∼\sim3.6×\timeslonger (19.2±1.919.2\pm 1.9vs.5.45.4iterations). The aligned potential is both fastandstable across all seeds at both scales.Table 2:The aligned-potential matrix on miniF2F algebra(mean±\pmstd over stable seeds; divergence-guard aborts counted separately). The aligned potential is the only configuration that is simultaneously fast and reliable at both scales. The annealing ablation shows that it does not need annealing. The AdamW rows provide the stable-optimizer baseline for the outcome-only configuration.Where the potentialisreachable, it robustly supplies the gradient the outcome reward lacks. It eliminates the dead all-fail updates that dominate early training (0 vs.∼\sim16 of 40 iterations) and, when paired with a bounded optimizer, converts this signal into faster and more reliable learning. We quantify this benefit on real theorem proving using a matched five-seed matrix across two model scales (Figure10). The aligned potential delivers a consistent speed-up under a matched optimizer and a substantial reliability advantage under an aggressive optimizer. At 30B scale, outcome-only training faces a stark trade-off: it either diverges under the fast optimizer or trains roughly 3.6×\timesslower under its stable counterpart. In contrast, the aligned potential is both fast and stable across every seed at both scales. These gains are modest and seed-consistent. They are not the dramatic single-run speed-ups sometimes reported in the literature—our own single-run pilots showed such effects, but they disappeared under proper re-seeding. Protocol and divergence taxonomy are provided in AppendixA.

An aligned-potential recipe.

Simply adding a dense potential is not enough for robust improvement. Three ingredients are essential:(i) Alignment.The potential must beoutcome-instrumental—it should increase as the task is solved (e.g., a falling goal count or a rising fraction of tests passed). It is implemented as a token-attached fulfillment credit with its own normalizer. Genericstructuralproxies (such as read-before-write hygiene or tactic well-formedness) are misaligned: once the task is mastered, they tend to be over-optimized, imposing an artificial performance ceiling.(ii) Reachability.The potential must take on intermediate values that the policy actually samples. This can be diagnosed before training by measuringVarG⁡(Φ)\operatorname{Var}_{G}(\Phi)on rollouts from the base policy. As shown in Figure9, reachability is vacuous on software repair but present on theorem proving.(iii) A bounded optimizer.Aggressive optimizers easily drive the shaped policy into entropy collapse or gradient explosion. A more stable optimizer (we use Muon) is critical for reliable training. Annealing the potential after saturation, often considered mandatory, proves unnecessary in this setting. Because the potential is outcome-instrumental, there is no misaligned pressure to over-optimize it. The annealing ablation (Table2) shows no meaningful difference beyond noise.

5 Discussion

Verifier asymmetry as a design principle.

The central practical takeaway is a reallocation of effort. Verifiers are scarce and valuable resources. The field has been spending them primarily on the direction they are weakest at—certifying progress—while largely ignoring the direction they are strongest at—detecting bad moves. We should invert this: spend verifiable signals on penalties. Whenever there is an outcome-neutral constraint a deployed agent must respect, and a concrete verifiable bad action to penalize, a penalty paired with its corresponding fulfillment credit and combined with the outcome reward teaches the constraint robustly and at low cost. When the only available dense signal is a progress proxy, first diagnose reachability (e.g., by measuring within-group variance on base-policy rollouts). Where intermediate values are reachable, the potential meaningfully accelerates learning. Where they are not, it provides no signal, and using the outcome reward alone is preferable.

Figure 11:Verifiable agentic RL is often verifier-bound.In theorem-proving experiments the GPU sits near-idle at full memory utilization while the CPU-based proof kernel verifies each step. The same signal that makes a penalty cheap also moves the bottleneck away from the accelerator.

A systems note.

The same property that makes verifiable penalties cheap—a machine-checkable environment—often makes the verifier a CPU-bound process (a rule engine, a test runner, or a container). In our theorem-proving experiments, the accelerator generated a step in milliseconds but then waited seconds for the CPU-based proof kernel to verify it. This resulted in GPU utilization in the single digits despite full memory occupancy. This behavior is workload-dependent rather than fundamental, but in verifiable agentic domains where penalties are applicable, overall throughput is typically gated by environment parallelism. Effective systems design therefore focuses on co-locating many verifier workers per accelerator rather than simply scaling accelerators (Figure11).

From proxies to deployment.

The two properties we rely on—that each rollout is expensive and irreversible, and that the checkable signal concerns path constraints rather than progress judgments—are most pronounced in the real-world online setting that originally motivated this work: an agent improving on live phone calls cannot afford wasted rollouts and must respect operational rules for which it receives no direct reward. This same setting makes large-scale, controlled, multi-seed experimentation impractical, which is why we evaluated on tractable proxies carefully chosen to preserve the essential structure. The transfer we claim is therefore mechanistic rather than numerical: a verifiable penalty supplies within-group variance where the outcome reward cannot, and a verifiable potential does so wherever partial progress is reachable. Validating these ideas directly on a deployed online agent—where the true sample-efficiency gains are measured in real user interactions saved—is the natural and most important next step.

Limitations.

Our strongest real-task harm reduction result was obtained in a regime where task success is near the floor. While the underlying mechanism predicts that the benefit persists at higher success rates (since penalty variance remains reachable regardless of outcome), validating this on a highly capable model is an important next step. Additionally, our penalties were manually identified based on domain rules. Automating the discovery of penalizable constraints directly from an environment’s affordances remains an open challenge. Finally, the sweep isolating the inaction trap shows high variance at larger scales. We therefore report the qualitativesurvival patternrather than precise quantitative magnitudes.

6 Related Work

Group-relative RL and the zero-variance problem.

Modern agentic RL has largely converged on group-relative policy optimization (GRPO)(Shaoet al.,2024)and the rule-based R1-style methods(DeepSeek-AI,2025; Lambertet al.,2024). Because these approaches use a within-group baseline, advantage isdefinitionallyequivalent to within-group variance, which collapses to zero on groups that are uniformly successful or uniformly failed. Prior responses to this “advantage collapse” have either discarded the dead groups (e.g., DAPO’s dynamic sampling(Yuet al.,2025)) or attempted to re-weight them—for instance by removing the difficulty normalizer(Liuet al.,2025b), stabilizing the sequence-level ratio(Zhenget al.,2025), or injecting entropy-based signals(Leet al.,2025). Notably, DAPO discardsall-correctgroups in addition to all-wrong ones, implicitly acknowledging blindness atbothextremes of success rate. Our work makes this symmetry explicit. Rather than discarding blind groups, we ask what kind of dense signal can supply the missing within-group variance. A verifiable penalty does so reliably, while a potential does so only where partial progress is reachable. Concurrently, VeriGate(Agrawalet al.,2026)gates step-level supervision on a similar condition, applying it only when verifier rewards become degenerate.

Penalties and negative reinforcement.

Negative signals are powerful tools on their own. Training against negative examples can suppress undesirable generations and reallocate probability mass(Zhu and others,2025), while negative-enhanced updates have been shown to recover gradient from all-wrong groups(Nanet al.,2025). However, these approaches penalizewrong outcomes. In contrast, our verifiable penalty targets specific,outcome-neutralpath violations. This supplies a form of signal that no outcome-based reward—positive or negative—can ever provide. As emphasized in our design rules, such a penalty is effective only when paired with an outcome reward that continues to drive task completion (Design Rule 2).

Process reward models and their gameability.

Step-level supervision has been shown to outperform pure outcome supervision for verification(Lightmanet al.,2024), with labels obtainable via Monte-Carlo rollouts(Wanget al.,2024). Some works frame the useful signal as a measure of progressSetluret al.(2025), which is close to our variance view, but instead instantiated with alearnedprover. However, learned dense rewards are notoriously vulnerable under RL optimization pressure. They are prone to over-optimization and Goodhart’s Law effects(Gaoet al.,2023), and capable agents can exploit misspecifications more effectively(Panet al.,2022). Summation-form process reward models often lead to training collapse(Cheng and others,2025), while learned value functions provide unreliable credit(Kazemnejad and others,2024). Even DeepSeek-R1 ultimately abandoned process reward models, describing them as “inevitably” hackable(DeepSeek-AI,2025), despite an implicit process model existing within their outcome reward model(Yuanet al.,2024a). This motivates our use ofverifiable, non-learnedpenalties together with our fourth design rule (un-gameability).

Potential-based shaping: safe versus useful.

Potential-based shaping leaves the optimum unchanged foranypotential(Nget al.,1999)and is equivalent to value initialization(Wiewiora,2003). These results certify which dense signals aresafe, but say little about which ones are actuallyuseful. Our work is orthogonal. Among safe shapings, we identify those that are helpful: those that supply within-group variance where the outcome reward lacks it, and only where such variance is reachable. Concretely, a penalty on frequently sampled bad actions is helpful, whereas a potential based on rarely reached progress usually is not.

Process rewards in agentic RL, and the fragility of RL gains.

Prior work on agentic step- and turn-level credit includes hierarchical critics(Zhouet al.,2024), learned step rewards(Yu and others,2024; Wanget al.,2025), turn-level estimators(Weiet al.,2025a), and—closest to our setting—GiGPO(Feng and others,2025)and per-tool-call correctness(Qian and others,2025); seeZhang (2026)for a survey. More recent methods further densify rewards: HERO blends a binary verifier with a continuous RM score undervariance-awareweighting(Taoet al.,2025), implicit step rewards distill a step PRM from trajectory preferences(Liuet al.,2025a), and RLTR rewards tool-usecompleteness(Liet al.,2025). Among these, RLTR’s verifiable check is the reachable process credit we advocate. However, most learned dense rewards are fragile under optimization pressure. Naive per-turn rewards can degrade performance when discriminativeness and advantage direction misalign(Modecruaet al.,2026), and turn-level gains are highly sensitive to the choice of policy-gradient estimator(Wang and Ammanabrolu,2025). This fragility aligns with a broader literature on non-replicable RL gains, including high seed variance(Hochlehnert and others,2025), spurious correlations(Shao and others,2025), one-shot improvements that fail to replicate(Wang and others,2025), and cases where base models match RLVR performance at large sample counts(Yue and others,2025). Our result sharpens this picture: the benefit of a dense potential is gated by reachability and the use of a bounded optimizer, rather than being intrinsic to the signal. Most prior work assumes cheap, resettable simulators where sample efficiency is secondary. Our online, irreversible setting inverts this priority, making reachable potentials and extra-sample-free verifiable penalties first-order concerns. Verifiableoutcomerewards remain central to many agentic benchmarks(Yaoet al.,2024; Barreset al.,2025; Panet al.,2025; Weiet al.,2025b). Learned-judge rewards(Yuanet al.,2024b; Leeet al.,2023; Baiet al.,2022; Zhenget al.,2023)and demonstration mixing(Yan and others,2025)are complementary to our approach: we use the latter for reachability seeding while cautioning against the former due to gameability concerns.

7 Conclusion

RLVR has taught language agents using the one signal environments can evaluate cheaply—the final outcome. TThe natural instinct to add denser supervision has been to reward progress. However, real agentic environments are asymmetric verifiers: they can reliably detect mistakes, but they struggle to certify progress. The dense signal they actually provide is therefore a penalty on the path, not a reward for progress. When wired correctly—alongside the outcome reward, paired with a fulfillment credit, and targeted at reachable and un-gameable actions—a verifiable penalty teaches outcome-neutral constraints that are essential for deployability. A lone penalty collapses into inaction, while a dense potential helps precisely where partial progress is reachable. Our core contribution is not a new optimizer, but a reallocation of a scarce resource: the verifier. We should spend it on penalties that bound the path, and—where it can reliably certify progress—on potentials that convert wasted rollouts into useful learning. For agents that must learn from live, irreversible interactions, where every rollout is costly and every misstep is visible, this reallocation is fundamental: it is the difference between a system that can be safely deployed and meaningfully improved online, and one that cannot.

Acknowledgements

This paper was produced using Pine Copilot’s voice-directedwhisper codingworkflow(Pine AI,2026a), in which the authors specify, discuss, and review the work by voice while a coding agent—Claude Code with Claude Opus 4.8—carries out the planning, coding, experiments, and paper writing. We thank BSQL Networking for hosting the NVIDIA RTX PRO 6000 GPU.

References

• A. Agrawal, M. Liu, and F. Huang (2026)VeriGate: verifier-gated step-level supervision for GRPO.arXiv preprint arXiv:2605.30451.Cited by:§6.
• Y. Bai, S. Kadavath, S. Kundu, A. Askell,et al.(2022)Constitutional AI: harmlessness from AI feedback.arXiv preprint arXiv:2212.08073.Cited by:§B.3,§6.
• V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan (2025)τ2\tau^{2}-Bench: evaluating conversational agents in a dual-control environment.arXiv preprint arXiv:2506.07982.Cited by:§6.
• J. Chenget al.(2025)Stop summation: min-form credit assignment is all process reward model needs for reasoning.arXiv preprint arXiv:2504.15275.Cited by:§1,§6.
• DeepSeek-AI (2025)DeepSeek-r1: incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948.Cited by:§1,§2,§6,§6.
• L. Fenget al.(2025)Group-in-group policy optimization for llm agent training.arXiv preprint arXiv:2505.10978.Cited by:§A.2,§1,§6.
• L. Gao, J. Schulman, and J. Hilton (2023)Scaling laws for reward model overoptimization.InInternational Conference on Machine Learning (ICML),Cited by:§1,§6.
• A. Hochlehnertet al.(2025)A sober look at progress in language model reasoning: pitfalls and paths to reproducibility.arXiv preprint arXiv:2504.07086.Cited by:§6.
• D. Hong (2026)AI Coding: practice and exploration.Note:Keynote, ByteDance / Volcano Engine Force Conference, Beijing, June 2026.https://mp.weixin.qq.com/s/tJAinVKzZAqZrhiEvGU2Dg(accessed 2026-06-25)Cited by:§A.5.
• C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan (2024)SWE-bench: can language models resolve real-world GitHub issues?.InInternational Conference on Learning Representations (ICLR),Cited by:§A.1.
• A. Kazemnejadet al.(2024)VinePPO: unlocking RL potential for LLM reasoning through refined credit assignment.arXiv preprint arXiv:2410.01679.Cited by:§1,§6.
• N. Lambert, J. Morrison, V. Pyatkin,et al.(2024)Tülu 3: pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124.Cited by:§1,§6.
• T. V. Le, M. Jeon, K. Vu, V. Lai, and E. Yang (2025)No prompt left behind: exploiting zero-variance prompts in LLM reinforcement learning via entropy-guided advantage shaping.arXiv preprint arXiv:2509.21880.Cited by:§6.
• H. Lee, S. Phatale, H. Mansoor, K. Lu, T. Mesnard, C. Bishop, V. Carbune, and A. Rastogi (2023)RLAIF: scaling reinforcement learning from human feedback with ai feedback.arXiv preprint arXiv:2309.00267.Cited by:§B.3,§6.
• Z. Li, Y. Hu, and W. Wang (2025)Encouraging good processes without the need for good answers: reinforcement learning for LLM agent planning.arXiv preprint arXiv:2508.19598.Cited by:§6.
• H. Lightman, V. Kosaraju, Y. Burda,et al.(2024)Let’s verify step by step.International Conference on Learning Representations (ICLR).Cited by:§1,§6.
• X. Liu, K. Wang, Y. Wu, F. Huang, Y. Li, J. Zhang, and J. Jiao (2025a)Agentic reinforcement learning with implicit step rewards.arXiv preprint arXiv:2509.19199.Cited by:§6.
• Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin (2025b)Understanding r1-zero-like training: a critical perspective.arXiv preprint arXiv:2503.20783.Cited by:§6.
• W. Modecrua, K. Kaewtawee, K. Pachtrachai, and T. Kraisingkorn (2026)Multi-turn reinforcement learning for tool-calling agents with iterative reward calibration.arXiv preprint arXiv:2604.02869.Cited by:§6.
• G. Nan, S. Chen, J. Huang, M. Lu, Y. Li, X. Xu,et al.(2025)NGRPO: negative-enhanced group relative policy optimization.arXiv preprint arXiv:2509.18851.Cited by:§6.
• A. Y. Ng, D. Harada, and S. Russell (1999)Policy invariance under reward transformations: theory and application to reward shaping.InInternational Conference on Machine Learning (ICML),pp. 278–287.Cited by:§6.
• A. Pan, K. Bhatia, and J. Steinhardt (2022)The effects of reward misspecification: mapping and mitigating misaligned models.InInternational Conference on Learning Representations (ICLR),Cited by:§6.
• J. Pan, X. Wang, G. Neubig, N. Jaitly, H. Ji, A. Suhr, and Y. Zhang (2025)SWE-Gym: training software engineering agents and verifiers.InInternational Conference on Machine Learning (ICML),Cited by:§6.
• Pine AI (2026a)Pine AI: the most natural human-computer interface is your voice.Note:Blog postAccessed 2026-06-28External Links:LinkCited by:Acknowledgements.
• Pine AI (2026b)Pine AI.Note:https://www.19pine.aiAn AI assistant that completes real-world tasks—phone calls, customer service, account and subscription management—on a user’s behalf. Accessed 2026-07.Cited by:§1.
• C. Qianet al.(2025)ToolRL: reward is all tool learning needs.arXiv preprint arXiv:2504.13958.Cited by:§6.
• A. Setlur, C. Nagpal,et al.(2025)Rewarding progress: scaling automated process verifiers for LLM reasoning.InInternational Conference on Learning Representations (ICLR),Cited by:§1,§6.
• R. Shaoet al.(2025)Spurious rewards: rethinking training signals in RLVR.arXiv preprint arXiv:2506.10947.Cited by:§6.
• Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi,et al.(2024)DeepSeekMath: pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300.Cited by:§2,§6.
• L. Tao, I. Kulikov, S. Saha, T. Wang, J. Xu, Y. Li, J. Weston, and P. Yu (2025)Hybrid reinforcement: when reward is sparse, it’s better to be dense.arXiv preprint arXiv:2510.07242.Cited by:§6.
• Q. Team (2025)Qwen3 technical report.arXiv preprint arXiv:2505.09388.Cited by:Appendix A,§B.1.
• H. Wang, C. T. Leong, J. Wang, J. Wang, and W. Li (2025)SPA-RL: reinforcing LLM agents via stepwise progress attribution.arXiv preprint arXiv:2505.20732.Cited by:§6.
• P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui (2024)Math-shepherd: verify and reinforce LLMs step-by-step without human annotations.InAnnual Meeting of the Association for Computational Linguistics (ACL),Cited by:§1,§6.
• R. Wang and P. Ammanabrolu (2025)A practitioner’s guide to multi-turn agentic reinforcement learning.arXiv preprint arXiv:2510.01132.Cited by:§6.
• Y. Wanget al.(2025)Reinforcement learning for reasoning in large language models with one training example.arXiv preprint arXiv:2504.20571.Cited by:§6.
• Q. Wei, S. Zeng, C. Li, W. Brown, and M. Hong (2025a)Reinforcing multi-turn reasoning in LLM agents via turn-level reward design.arXiv preprint arXiv:2505.11821.Cited by:§6.
• Y. Wei, O. Duchenne, J. Copet, others, and S. I. Wang (2025b)SWE-RL: advancing LLM reasoning via reinforcement learning on open software evolution.arXiv preprint arXiv:2502.18449.Cited by:§6.
• E. Wiewiora (2003)Potential-based shaping and Q-value initialization are equivalent.Journal of Artificial Intelligence Research (JAIR)19,pp. 205–208.Cited by:§6.
• J. Yanet al.(2025)Learning to reason under off-policy guidance.arXiv preprint arXiv:2504.14945.Cited by:§6.
• J. Yang, K. L. Zhang,et al.(2025)SWE-smith: scaling data for software engineering agents.arXiv preprint arXiv:2504.21798.Cited by:§A.5.
• S. Yao, N. Shinn, P. Razavi, and K. Narasimhan (2024)τ\tau-bench: a benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045.Cited by:§B.2,§6.
• Q. Yu, Z. Zhang, R. Zhu, Y. Yuan,et al.(2025)DAPO: an open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476.Cited by:§A.2,§2,§6.
• Y. Yuet al.(2024)StepTool: enhancing multi-step tool usage in llms through step-grained reinforcement learning.arXiv preprint arXiv:2410.07745.Cited by:§A.2,§1,§6.
• L. Yuan, W. Li, H. Chen, G. Cui, N. Ding, K. Zhang, B. Zhou, Z. Liu, and H. Peng (2024a)Free process rewards without process labels.arXiv preprint arXiv:2412.01981.Cited by:§6.
• W. Yuan, R. Y. Pang, K. Cho, S. Sukhbaatar, J. Xu, and J. Weston (2024b)Self-rewarding language models.arXiv preprint arXiv:2401.10020.Cited by:§B.3,§6.
• Y. Yueet al.(2025)Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?.arXiv preprint arXiv:2504.13837.Cited by:§6.
• C. Zhang (2026)From reasoning to agentic: credit assignment in reinforcement learning for large language models.arXiv preprint arXiv:2604.09459.Cited by:§6.
• C. Zheng, S. Liu,et al.(2025)Group sequence policy optimization.arXiv preprint arXiv:2507.18071.Cited by:§6.
• K. Zheng, J. M. Han, and S. Polu (2022)MiniF2F: a cross-system benchmark for formal olympiad-level mathematics.International Conference on Learning Representations (ICLR).Cited by:§A.2,§B.1.
• L. Zheng, W. Chiang, Y. Sheng, S. Zhuang,et al.(2023)Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.arXiv preprint arXiv:2306.05685.Cited by:§B.3,§6.
• Y. Zhou, A. Zanette, J. Pan, S. Levine, and A. Kumar (2024)ArCHer: training language model agents via hierarchical multi-turn RL.InInternational Conference on Machine Learning (ICML),Cited by:§6.
• X. Zhuet al.(2025)The surprising effectiveness of negative reinforcement in LLM reasoning.arXiv preprint arXiv:2506.01347.Note:NeurIPS 2025Cited by:§6.

Appendix AFull Experiments on Dense Potentials

The main text treats the dense progresspotentialas the reachability-gated half of the process signal (§4). This appendix gives the full experiments. The picture is consistent with the variance account: where a potential’s intermediate values arereachableit supplies gradient the outcome lacks and accelerates training, and where they are unreachable it is exactly vacuous. Converting the extra gradient into a stable success gain requires a bounded optimizer—an aggressive update rule can destabilize the densely shaped policy, an optimization caveat we return to in §A.3. Unless noted, we train Qwen3[Team,2025]with our own GRPO, place any dense credit on the responsible tokens with a separate per-channel normalizer, and count efficiency inepisodes generatedso that resampling costs are visible.

A.1 Reachability is measurable before training, and it gates the benefit

The condition the variance account makes operative is reachability: a potentialΦ\Phihelps only where the policy realizes differing intermediate values across a group, i.e.VarG⁡(Φ)>0\operatorname{Var}_{G}(\Phi)>0. This can be checkedbeforespending training compute, from a handful of base-policy rollouts.

On SWE-bench software repair[Jimenezet al.,2024]the natural potential is the fraction of hiddenFAIL_TO_PASStests an episode makes pass. Two facts make it vacuous (Figure12). Structurally the finer potential barely exists: two-thirds of instances have a single failing test, soΦ\Phicollapses to the binary outcome, and only a third admit any partial progress. Empirically even that third is unreachable: across156156rollouts of a3030B policyeveryepisode scoresΦ=0\Phi=0, soVarG⁡(Φ)\operatorname{Var}_{G}(\Phi)is identically zero and aΦ\Phi-based reward centers out to no gradient. The cause is not gameability but unreachability—there is no realized intermediate state for any reward to act on.

Mapping the same no-training proxyVarG⁡(Φ)\operatorname{Var}_{G}(\Phi)across model capability (0.60.6–44B) and outcome sparsity (Figure13a) reproduces the structure the account predicts: it rises with capability and peaks in the sparse-but-reachable corner. The measured dense-reward benefit tracks it (Figure13b):≈0\approx\!0whereVarG⁡(Φ)≈0\operatorname{Var}_{G}(\Phi)\!\approx\!0(the SWE null; the too-weak small models) and large where it is high (the44B reachable chain, dense0.340.34vs. outcome0.010.01; real theorem proving, below). Benefit appears only past a reachability threshold. We could not fill the map’s interior with controlledtrainingpoints—small-scale synthetic training is too learning-rate-sensitive, most cells degenerating to zero for both configurations—so the benefit axis rests on the real-domain anchors plus one clean controlled point.

Figure 12:In software repair the finer potential is structurally rare and empirically unreachable.*(a)Of4848single-file-fix SWE-bench instances, two-thirds have a single failing test, so the test-fraction potential equals the binary outcome; only one-third admit a finerΦ\Phi.(b)For those, a3030B policy’s rollouts never realize it: all156156rollouts scoreΦ=0\Phi=0. The band of intermediate values exists but is empty—zero within-group variance, hence zero gradient.Figure 13:Reachability, measured cheaply, gates the dense-reward benefit.(a)VarG⁡(Φ)\operatorname{Var}_{G}(\Phi)on base-policy rollouts (no training) over model capability×\timesoutcome sparsity: it rises with capability and peaks in the sparse-but-reachable corner.(b)*The benefit of a dense reward (final-success gap, dense−-outcome) against that same probe: zero whereVarG⁡(Φ)≈0\operatorname{Var}_{G}(\Phi)\!\approx\!0and large where it is high. Benefit appears only past a reachability threshold.

A.2 Where the potential is reachable, it removes dead updates and accelerates

When the potential’s intermediate values are reachable and point the same way as success, a dense signal is genuinely useful. We instantiate it on a family ofNN-stage chainedfile-operations tasks (toolslist_dir,read_file,write_file,delete,run_tests,submit); success requires allNNstages, soNNtunes base difficulty (N=4N{=}4sits near8%8\%base success). The dense signal is a verifiable progress potential carried on a token-attached channel (a penalty when a call violates a rule—overwrite a file never read, submit an untested change—and a credit when a call fulfills a pending obligation).

Dead updates.

We count the fraction of iterations whose update isdead: an all-fail group with no reward spread (Figure14). Outcome-only GRPO is dead on65%65\%of iterations. DAPO[Yuet al.,2025], built to dodge all-fail groups by resampling, reaches54%54\%but pays a5.6×5.6\timesgeneration tax. The aligned potential cuts dead updates to8%8\%at no extra sampling, by producing gradient from the very failures the outcome reduces to zero. The cure is the credit scheme, not the budget—and it is contingent on reachability: on the silent-precondition gate (§A.4), where the pivotal action is never sampled, every reward-only method is dead almost always (outcome and DAPO100%100\%, the potential76%76\%), because a fulfillment credit can fire only on behavior the policy attempts.

Speed.

On held-out success against episodes generated (Figures15,16) outcome-only crawls, spending its budget on dead updates, then converges only at the end; the aligned potential rises immediately from the early all-fail groups. Reimplemented dense baselines (GiGPO-style step advantages[Feng and others,2025], StepTool-style per-call rewards[Yu and others,2024]) also outpace outcome-only—anyadmissibledense signal helps. The gain also holds on real theorems (miniF2F[Zhenget al.,2022]) at 4B and 30B, but only in the multi-seed form quantified in §A.3: a single-run pilot of ours showed a dramatic2.4×2.4\timesspeed-up thatflipped signwhen the identical configuration was re-run (rollout sampling is unseeded in the serving engine, and MoE training at this scale is bimodal), so we report only the five-seed matrix (Figure10, Table2): a∼\sim1.6×\timesspeed-up to mastery under a matched optimizer, consistent on every seed, plus the reliability advantage. Both configurations saturate—the claim is speed and reliability, not final success.

Figure 14:Dead updates across two regimes.Fraction of iterations with an all-fail, zero-spread group (Qwen3-4B, 3-seed mean).Chained task(bottleneck reachable): outcome-only65%65\%, DAPO54%54\%at a5.6×5.6\timesgeneration tax, the aligned potential8%8\%.Silent precondition gate(§A.4, pivotal action never sampled): outcome and DAPO dead oneveryiteration, the reward-only potential76%76\%; only a demonstration seed breaks the wall. The cure is areachable, admissiblesignal, not density.Figure 15:Sample efficiency on the calibrated hard task (N=4N{=}4).Held-out success vs. episodes generated. Outcome-only sits near base rate for most of training; the aligned potential climbs immediately from the all-fail groups; dense baselines also outpace outcome-only. Thexx-axis counts episodesgenerated, including DAPO’s resampling.Figure 16:Efficiency over three seeds.*Left:*episodes to the success threshold (lower better)—the potential is several times faster than GRPO and DAPO.*Right:*the mechanism—GRPO and DAPO burn most iterations on dead or stalled updates, and DAPO generates5.6×5.6\timesthe episodes it uses. Note the large GRPO error bar: the all-fail lottery on the outcome side, which the dense potential’s reachable within-group variance avoids.

A.3 Converting the gradient to success: protocol, and the matched matrix

The extra gradient a reachable potential supplies must still be turned into task success, and here the decisive factor is the optimizer, not the potential. Early runs read this as “fragility,” but the cause was two experimental artifacts. First, an aggressive, unbounded update rule can destabilize a densely shaped policy: at one fixed learning rate, a theorem-proving run anneals cleanly to full success (gradient norm<1.5<1.5, entropy decaying to zero) while an otherwise-identical sibling suffers a gradient blow-up (norm>109>10^{9}), its entropy explodes, and success collapses to zero and never recovers—the same signal, the same rate, decided by optimization stability. On the chained task the opposite failure appears: the policy collapses to a zero-entropy degenerate output, so every rollout is identical, the within-group variance is zero, and the update dies. Aboundedoptimizer (we use Muon, with a divergence guard that aborts on entropy collapse or gradient spikes) removes both failure modes. Second, per-iteration training success measured on a handful of freshly sampled tasks is too noisy to read a speed-up from—it swings between extremes iteration to iteration—so we evaluate on a fixed held-out task set. The result robust to all of this—the mechanism the paper relies on—is dead-update elimination: zero dead iterations against∼\sim16 of 40 for outcome-only, oneveryseed.

The matched matrix: five seeds, two scales.

With the bounded-optimizer protocol fixed (aligned goal-progress potential as a token-attached fulfillment credit, Muon, a divergence guard,3030iterations on miniF2F algebra), we ran the full matrix—aligned potential versus outcome-only, five seeds per configuration, at 4B and 30B, plus a three-seed anneal ablation and a 30B AdamW outcome baseline; all magnitudes are in Table2. Because rollout sampling in the serving engine is unseeded, “seeds” are i.i.d. draws of the whole run, so per-configuration consistency is the meaningful statistic. Three facts are seed-robust.(1) A modest, consistent speed-upunder a matched optimizer: the aligned potential reaches the threshold sooner and has higher area-under-curve on every seed-matched pair at 4B.(2) Reliability:it completes every run across both scales without a divergence abort, while outcome-only loses seeds to entropy collapse (4B) and gradient explosion (30B)—the dense, always-relevant gradient appears toregularizewhere the bursty sparse signal blows up under the same rate.(3) The outcome configuration’s dilemma at 30B:outcome-only is competitive under the fast optimizer only when it survives (it diverges on most seeds), and stable under its safe optimizer only at several times the potential’s time-to-mastery. The aligned potential requires no such choice.

Anatomy of the gain.

When the potential does win, ablating onN=4N{=}4(Figure17) attributes the speed to the token-attached channel: folding the same rule rewards into the scalar return doubles the episodes to threshold. Annealing the shaping off after saturation protects the final ceiling (unrelaxed pressure drifts the policy toward over-compliant, bloated episodes—a mild inaction-trap tendency). The fulfillment credit is the positive signal that escapes all-fail groups; removing it leaves more dead iterations and a lower ceiling. Finally, replacing every hand-written rule with three universal meta-rules derived only from per-tool category tags (observe/mutate/verify/terminal) and the environment’s error codes recovers the hand-engineered efficiency almost exactly: in well-structured environments the specification cost can be paid by metadata the tool schemas already expose.

Figure 17:Component attribution for the potential onN=4N{=}4(episodes to threshold, converged success annotated). The token-attached credit channel is the efficiency lever (folding into a scalar return doubles the cost); annealing protects the ceiling; the clean recipe is best. A per-step length cost, useful only on very long horizons, hurts here.

A.4 Reachability governs the ceiling, not just the speed

Sample efficiency is speed to a ceiling the chained tasks eventually reach. To ask whether a process reward can raise theceiling, we build a task that is hard todiscoverrather than hard to compute: asilent precondition gate, where to write a protected file the agent must first read an access-control file and request access; skip either and the write silently no-ops (reports success, does nothing). This is the bank-authentication problem of the introduction in controlled form—a required precondition an agent cannot stumble onto by trial and error and must be told—and it is where a purely reward-driven online learner would stall indefinitely. Calibration confirms the trap is total—the base model never reads the access file even when the rule is in its prompt. Then (Figure18)everyreward-only method (outcome, DAPO, the clean potential) fails to zero, because the fulfillment credit for reading the access file can only reward that behavior if the policy samples it, and it never does. A single rule-synthesized demonstration injected through the process channel breaks the wall to perfect, generalizing success. When the required behavior is never sampled, no on-policy reward can induce it; imitation must seed what reward then grows—the same reachability wall the SWE probe hit from the other side, and the R1-Zero (discoverable) versus R1 (demonstration cold-start) distinction in miniature.

Figure 18:A non-saturating ceiling test, and the discovery wall.*Left:*on the silent-gate task every reward-only method (outcome, DAPO, clean potential, even rules-in-prompt) converges to zero—the precondition is never sampled, so the fulfillment credit never fires.*Right:*a single synthesized demonstration through the process channel breaks the wall to perfect held-out success, a sharp phase transition.

A.5 Learning where the outcome is dead: real bug-fixing at zero success

The variance account predicts that when a group is all-failing—outcome variance zero, outcome gradient dead—a process signal is theonlything that can move the policy. We check this directly on real software repair, a domain where the path-vs-outcome gap is well documented: in a controlled industry study, AI coding agents reach high functional correctness yet score far lower—and almost at random—on reliability, maintainability, and not breaking previously working functionality[Hong,2026], the engineering discipline that decides whether a patch is deployable. Here an 8B agent is far below the task’s reach, which makes it an ideal test of the dead-gradient regime. We train on SWE-smith[Yanget al.,2025]instances (a repo container, a hiddenFAIL_TO_PASStest as the oracle), with the agent issuing bash commands to find and fix the bug; the process channel is the coding-discipline signal of §3(a fulfillment credit for running tests and making a verifiable edit, a penalty for re-running a failed command or editing without testing). At this scale the agent solves essentiallyzeroinstances under either signal, so outcome-only RL trains on an all-fail distribution throughout—the cleanest possible test of the dead-gradient regime. This is adifferentsignal than the progresspotentialof §4: on this same software-repair domain the test-fraction potential is vacuous (Figure9, left), because zero tests ever pass and so no intermediate value is reached. The penalized and fulfilled behaviors here—running a test, editing a file, re-running a failed command—are always reachable, so the penalty channel carries within-group variance exactly where the potential cannot. The two software-repair results are consistent: the reachability gate closes on the potential and stays open on the penalty.

The result is a sharp dissociation (Figure19, two seeds). Outcome-only leaves the trajectoryflat: productive actions per episode sit at≈1.5\approx\!1.5from the first iteration to the last, essentially identical across seeds (1.51.5and1.51.5)—the dead gradient changes nothing. The process reward instead drives a steadilyrisingtrajectory: productive actions climb over training to6.56.5and11.411.4(final-3, the two seeds) and test-runs to7.67.6and12.912.9, a four-to-tenfold increase, while the penalized behaviors fall (repeat-errors and untested edits toward zero). The agent learns to investigate, test, and edit like a disciplined engineer—from a signal available on every failed episode—exactly where the outcome reward is inert.

Two honest caveats scope the claim. First, the fulfillment creditrewardstest-running and productive edits, so part of their rise is by construction; the non-circular evidence is that thepenalizedbehaviors (which the fulfillment credit does not touch) also fall, and that the outcome baseline is perfectly flat—the effect is the process gradient, not the task. Second, this improved discipline didnotconvert to task success in our budget (both configurations remain at zero): 8B is simply too weak to close these bugs. The claim is therefore mechanistic and behavioral—the process channel produces learning, and better verifiable engineering discipline, precisely in the all-fail regime where outcome-only RL has no gradient at all—and a plausible precursor to success at capability, not a demonstration of success itself.

Figure 19:The process reward learns where the outcome is dead (SWE-smith, 8B,≈\approx0% solve).*(a)Productive actions per episode over training, two seeds: outcome-only is flat (the all-fail gradient is dead), while the process reward rises steadily.(b)*Final-3 trajectory quality (mean of two seeds): productive edits and test-runs increase several-fold under the process reward, while penalized behaviors (repeat-errors, untested edits) fall. Task success is zero for both configurations throughout—this is a behavioral/mechanistic result, not a success result.

Appendix BBoundaries: When the Penalized Target Is Not Verifiable

Design rule 4 (§3.3) requires the penalized (and fulfilled) target to be reachable andun-gameable. This appendix gives the experiments behind that rule: what happens when a penalty’s target is misaligned with the outcome, and when it is alearnedproxy rather than a verifiable predicate.

B.1 An un-gameability sweep: which signals survive

We construct five process signals for Lean theorem proving on miniF2F[Zhenget al.,2022], spanning aligned to misaligned, pre-register each one’s cheapest gaming policy, and train Qwen3-30B-A3B[Team,2025]with each (three seeds, everything else fixed; Figure20). Survival tracks admissibility. Thepenalty-freesignals reliably survive on every seed: a gameable “any-valid-tactic” credit, an aligned goal-progress fulfillment credit, and that creditgated on the eventual outcome(un-gameable by construction, the most consistent survivor). Thepure penalty—an errored-tactic penalty with no fulfillment credit and no outcome reward—reliablycollapses to essentially zero on every seed: its cheapest maximizer, avoid errors by not attempting, is precisely theinaction trapof design rule 2, and with no positive signal to escape it the policy dies every time. This is the paper’s cleanest evidence that a lone penalty cannot drive learning. The revealing case ispenalty-plus-fulfillment: across three seeds it is bimodal (collapses on two, rescued to perfect on the third), the largest variance of any configuration—adding a fulfillment credit to a misaligned penalty makes survivalpossiblebut not reliable. The robust reading: penalty-free progress signals survive, a pure penalty reliably collapses, and outcome-gating is the most consistent survivor. Precise magnitudes remain noisy at 30B; we report the survival pattern.

Figure 20:Which signals survive (Qwen3-30B, miniF2F, 3 seeds).(a)Terminal success (last-3-iteration mean) for five signals, mean±\pmstd with seed dots. The three penalty-free signals sit well above the dead zone on every seed; the pure penalty is reliably dead; the fulfillment++penalty configuration has by far the largest error bar.(b)That configuration isbimodal—collapsing on two seeds, rescued to1.01.0on one—whereas the pure penalty collapses on all three.

B.2 Task intent is not a verifiable rule

The hardest case for a penalty is a task whose difficulty is not procedural. Onτ\tau-bench-style airline customer service[Yaoet al.,2024]the reward turns onsemanticpolicy adherence (authorization, refund eligibility, confirmation) rather than structural ordering, so no verifiable rule is finer than the outcome: what is missing is intent, which is the reward itself. Three tiers make this concrete (Figure21).Genericstructural rules—the read-before-write hygiene that works on chained tasks—are now orthogonal to the reward and activelyharm: the cheapest way never to violate “confirm before calling” is to not place the risky calls at all, and the policy collapses into the inaction trap within a handful of iterations.Policy-derived proceduralrules (fetch the user id and look up the reservation before modifying it), outcome-instrumental by construction and paired with early annealing, remove the harm, because a correct modification requires the looked-up state so the fulfillment credit pulls toward the productive workflow.Verifiable semanticrules (do not modify a basic-economy reservation; do not use a payment method absent from the profile) prune failure-bound actions and lift the best iterations nearly to outcome-only’s peak. But neither aligned tierbeatsoutcome-only on average (Figure22): verifiable rules express policyvalidity, while the residual difficulty iswhichpermissible change this user wants—and a rule encoding that would just be the task specification.

Figure 21:The coverage gradient onτ\tau-bench airline.*Left:*training reward for four signals. Outcome-only learns the benchmark; generic rules orthogonal to the reward collapse into the inaction trap; policy-derived procedural and verifiable-semantic rules (with early annealing) do not.Right:mean reward per tier (bars) with peak (ticks). Misaligned rules harm; aligned rules remove the harm and nearly reach outcome-only’s peak, but neither beats it. The residual gap is intent.Figure 22:The boundary on one axis: are the rules outcome-instrumental?When the verifiable rules are aligned with what success requires (chained and gated tasks: the workflowisthe task), the process channel converts compliance into large outcome gains. When orthogonal to the reward (τ\tau-bench, generic rules), the same machinery optimizes compliance into a degenerate policy.

B.3 A learned critic relocates the hack

If verifiable rules cannot reach intent, why not let the policy judge itself, in the lineage of self-rewarding LMs and RLAIF[Yuanet al.,2024b, Leeet al.,2023, Baiet al.,2022, Zhenget al.,2023]? We use the critic as thesame modelas the policy (no larger judge), so any signal is genuine on-policy self-reflection. Across two axes—whether a verifiable rule is cheaply specifiable, and whether the on-policy critic can detect the violation (Figure23)—the learned critic is a poor training reward in every quadrant. As a detector, blind self-critique flags surface-evident violations but is essentially blind to stateful-bookkeeping norms (did it readthisfile before overwriting it) even when handed the rules, and the blind spot is structural (per-rule recall stays near zero as the critic scales). As a reward in the all-fail regime (Table4), a deterministic rule with identical credit structure is decisive on every seed while the self-critic—with perfect recallagainst the rule oracle—is inert; afrozencritic fails identically, isolating the cause as the critic’s∼\sim6% false-positive rate, not its non-stationarity. At the intent frontier (Table4) the self-critic is a usefulofflinedetector yet collapses as a training reward. Defining the penalized target by a learned proxy relocates the credit-assignment problem into the proxy, which the policy then games—the mechanism behind design rule 4. Verifiability is what makes the dense channel safe to optimize.

Figure 23:When a verifiable rule beats a same-model self-critic, and when neither trains.Axes: whether a verifiable rule is cheaply specifiable, and whether the on-policy critic can detect the violation. The learned critic is a poor training reward in every quadrant; its only robust use is offline diagnosis at the intent frontier.Table 3:A deterministic rule beats a same-model self-critic as a dense reward, in the all-fail regime (consumer-service domain, 1.7B, three seeds, late-training mean±\pmstd). Rule and critics share the identical penalty-only credit structure, and the critic additionally hasperfect recallagainst the rule oracle. The rule is decisive every seed, while the critics are inert. Thefrozen(stationary) critic fails like the live one, isolating the cause as the critic’s∼\sim6% false-positive imprecision rather than non-stationarity.Table 4:Self-critique is a usable intentdetectorbut a failed intentreward(τ\tau-bench airline, Qwen3-4B).*Left:*as a failure predictor on rule-clean (intent) failures the self-critic far exceeds the verifiable semantic rules (3 rollout seeds).*Right:*as the dense training reward the same self-critique collapses, while outcome-only and the rules both learn (training reward, early→\tolate, 20 iterations).

Appendix CSetting-by-Setting Summary

Table5collects the five settings studied across the paper against the three properties the variance account cares about—does a verifiable signal finer than the outcome exist, is it un-gameable, and are its intermediate values reachable—together with the observed outcome. Help appears only when a reachable, un-gameable signal is available; a penalty on always-sampled bad actions satisfies this on every row where it applies, while a progress potential does so only where partial progress is reachable.

Table 5:The five settings studied, summarized against the variance account.Each row is a domain. The three middle columns are the conditions the account cares about: does a verifiable signal finer than the outcome exist, is it un-gameable, and are its intermediate values reachable. “✓/✗” marks the condition that decides the row. A dense signal helps only when all three hold; this table organizes the settings rather than predicting them.