When RLVR Shrinks the Reasoning Boundary: Diagnosing Pass@k Inversion

arXiv cs.LG Papers

Summary

This paper diagnoses 'pass@k inversion' in reinforcement learning with verifiable rewards (RLVR), where training improves one-shot accuracy but reduces performance under repeated sampling, particularly on boundary prompts where correct trajectories are rare. It proposes a per-problem base anchoring (PBA) method to mitigate this issue.

arXiv:2607.20543v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) can improve one-sample accuracy while making a model worse under repeated sampling. We study this pass@k inversion: after training, the policy may solve fewer distinct problems than its base model at large $k$. The failure concentrates on boundary prompts, where the base model contains rare correct trajectories that are recoverable by sampling but too sparse to reliably appear in finite RLVR rollout groups. We argue that a two-mode account explains this as an absence-of-evidence failure: rare correct trajectories may disappear before RLVR samples and reinforces them often enough. The main contribution is this diagnostic and mechanistic framing. Per-Problem Base Anchoring (PBA) is a deliberately simple proof-of-concept: sharpen prompts with sufficient frozen-base correct evidence, and anchor risky prompts to the base distribution. Across three training seeds on Omni-MATH-Test, with MATH500 as a secondary high-coverage validation benchmark, PBA improves both \PassK{1} and high-budget coverage over matched GRPO. A 3000-prompt regime-controlled diagnostic study is consistent across seeds with the expected signature: ordinary GRPO loses base-solvable boundary prompts, while PBA preserves rare verifier-positive trajectories. We use mathematical verifiers as a controlled testbed for verifier-guided optimization; the same pass@k inversion risk applies to ECCV-relevant vision-language agents when repeated visual, spatial, or chart-reasoning attempts are checked by external tools or verifiers. Reasoning post-training should decide not only how strongly to optimize, but which prompts are safe to optimize.
Original Article
View Cached Full Text

Cached at: 07/24/26, 05:11 AM

# When RLVR Shrinks the Reasoning Boundary: Diagnosing Pass@𝑘 Inversion
Source: [https://arxiv.org/html/2607.20543](https://arxiv.org/html/2607.20543)
11institutetext:Harvard University###### Abstract

Reinforcement learning with verifiable rewards \(RLVR\) can improve one\-sample accuracy while making a model worse under repeated sampling\. We study this*pass@k inversion*: after training, the policy may solve fewer distinct problems than its base model at largekk\. The failure concentrates on*boundary prompts*, where the base model contains rare correct trajectories that are recoverable by sampling but too sparse to reliably appear in finite RLVR rollout groups\. We argue that a two\-mode account explains this as an absence\-of\-evidence failure: rare correct trajectories may disappear before RLVR samples and reinforces them often enough\. The main contribution is this diagnostic and mechanistic framing\. Per\-Problem Base Anchoring \(PBA\) is a deliberately simple proof\-of\-concept: sharpen prompts with sufficient frozen\-base correct evidence, and anchor risky prompts to the base distribution\. Across three training seeds on Omni\-MATH\-Test, with MATH500 as a secondary high\-coverage validation benchmark, PBA improves bothpass​@​1\\mathrm\{pass\}@1and high\-budget coverage over matched GRPO\. A 3000\-prompt regime\-controlled diagnostic study is consistent across seeds with the expected signature: ordinary GRPO loses base\-solvable boundary prompts, while PBA preserves rare verifier\-positive trajectories\. We use mathematical verifiers as a controlled testbed for verifier\-guided optimization; the same pass@k inversion risk applies to ECCV\-relevant vision\-language agents when repeated visual, spatial, or chart\-reasoning attempts are checked by external tools or verifiers\. Reasoning post\-training should decide not only how strongly to optimize, but which prompts are safe to optimize\.

## 1Introduction

Reinforcement learning with verifiable rewards has become a central tool for improving reasoning models\. In mathematics, code, and other domains with automatic checkers, it is natural to sample a solution, verify it, and reinforce successful trajectories\. This recipe often improvespass​@​1\\mathrm\{pass\}@1, and such gains are commonly interpreted as evidence that the model has acquired stronger reasoning ability\.

That interpretation is incomplete\. A reasoning model is not only a source of a single answer; it is a distribution over possible attempts\. Modern inference pipelines exploit this distribution through self\-consistency, verifier reranking, search, repair, code execution, and repeated sampling\. For these systems, the critical question is not only whether the top sample is correct, but whether a correct trajectory remains reachable in the policy support\. Recent work shows that RLVR can fail precisely on this axis: it can increasepass​@​1\\mathrm\{pass\}@1while reducingpass​@​k\\mathrm\{pass\}@kat largekk, so that the trained policy solves fewer distinct problems than the base model when allowed many attempts\[[22](https://arxiv.org/html/2607.20543#bib.bib22)\]\. We call this failure*pass@k inversion*\.

Entropy collapse is a useful symptom, but not a sufficient explanation\. On easy prompts whose dominant sampled mode is already correct, sharpening is desirable\. On prompts outside the base model’s support, there is little coverage to lose\. The damaging case is the boundary regime: prompts for which correct trajectories exist under repeated sampling but are not the dominant mode\. These are the prompts where large\-kkinference is valuable\. The base model is unreliable in one attempt, yet it retains latent solution mass that repeated attempts can recover\.

Our central claim is that pass@k inversion is a*boundary mode\-commitment failure*\. RLVR does not merely lower entropy; on boundary prompts, it can commit away from rare correct modes\. The mechanism is not that all\-zero reward groups directly reward an incorrect answer\. Rather, all\-zero groups provide little local corrective signal, while sparse positive groups and shared\-parameter updates elsewhere in training sharpen the policy before the rare correct branch is reinforced often enough\. The practical effect is prompt\-conditioned forgetting: the trained policy becomes more confident while destroying coverage that repeated sampling previously recovered\.

We make this claim precise and actionable through three contributions\.

- •Diagnostic framing\.We decompose prompts by frozen base\-model behavior into solved\-easy, reachable, boundary, and out\-of\-reach regimes, turning an aggregate pass@k curve into an attribution problem\.
- •Mechanistic model\.We give an idealized two\-mode model in which finite\-sample RLVR can commit to the wrong mode before rare correct trajectories are observed, predicting that coverage loss should localize to boundary prompts\.
- •Diagnostic validation\.We usePBA, a minimal prompt\-conditional base anchor, as an intervention probe, and evaluate it across three training seeds with full pass@k curves, a 3000\-prompt regime\-controlled study, matched controls, and gate ablations\.

Our main contribution is a diagnostic and mechanistic account of pass@k inversion;PBAis a deliberately simple intervention showing that the diagnostic can guide safer optimization\.

The claim is intentionally limited\.PBAdoes not discover solution modes absent from the base model, nor does it replace distillation, curriculum learning, process supervision, tool use, or search\. It solves a more immediate failure: current RLVR can destroy coverage the base model already possessed\. Preventing that destruction is a prerequisite for evaluating whether future methods truly expand the reasoning boundary\. We position this as an ECCV workshop contribution on reliable verifier\-guided reasoning and evaluation\. We use mathematical verifiers as a controlled testbed because they isolate the verifier\-guided RLVR effect; the same pass@k inversion risk applies to vision\-language agents whenever repeated visual, spatial, or chart\-reasoning attempts are checked by external tools or verifiers\. In such systems, a VLM is often useful precisely because large\-kksearch can recover a rare correct visual rationale, program, or geometry solution; an RL step that improves the first sample while erasing those rare modes would make the deployed agent less reliable under test\-time compute\.

## 2Related Work

### RLVR and reasoning\-boundary shrinkage\.

RLVR methods such asPPO\[[16](https://arxiv.org/html/2607.20543#bib.bib16)\],GRPO\[[17](https://arxiv.org/html/2607.20543#bib.bib17)\],RLOO\[[1](https://arxiv.org/html/2607.20543#bib.bib1)\],Reinforce\+\+\[[7](https://arxiv.org/html/2607.20543#bib.bib7)\],ReMax\[[11](https://arxiv.org/html/2607.20543#bib.bib11)\], andDAPO\[[20](https://arxiv.org/html/2607.20543#bib.bib20)\]optimize language models using automatically verifiable outcome rewards\. Modern post\-training stacks have made such procedures increasingly practical at scale\[[18](https://arxiv.org/html/2607.20543#bib.bib18),[10](https://arxiv.org/html/2607.20543#bib.bib10)\]\. Their success is often summarized bypass​@​1\\mathrm\{pass\}@1\. Yue et al\.\[[22](https://arxiv.org/html/2607.20543#bib.bib22)\]show that this is incomplete: across algorithms, model families, and task domains, RLVR can improvepass​@​1\\mathrm\{pass\}@1while shrinking large\-kkcoverage\. Related work reports diversity collapse, entropy collapse, and evidence that many post\-training reasoning behaviors originate in the base model rather than being newly discovered by RL\[[21](https://arxiv.org/html/2607.20543#bib.bib21),[3](https://arxiv.org/html/2607.20543#bib.bib3),[15](https://arxiv.org/html/2607.20543#bib.bib15),[13](https://arxiv.org/html/2607.20543#bib.bib13),[23](https://arxiv.org/html/2607.20543#bib.bib23)\]\. Most closely, Yuan et al\.\[[21](https://arxiv.org/html/2607.20543#bib.bib21)\]view diversity collapse through the lens of overtraining\. We instead isolate finite\-sample commitment to the wrong side of a latent mixture and turn frozen base correct\-mass estimates into a prompt\-level intervention\.

### Entropy control and conservative optimization\.

Maximum\-entropy RL\[[5](https://arxiv.org/html/2607.20543#bib.bib5)\], KL\-regularized RLHF\[[8](https://arxiv.org/html/2607.20543#bib.bib8)\], conservative offline RL\[[9](https://arxiv.org/html/2607.20543#bib.bib9)\], and preference\-optimization methods\[[14](https://arxiv.org/html/2607.20543#bib.bib14)\]all recognize that unconstrained optimization can over\-concentrate a policy\. But global entropy control is poorly matched to pass@k inversion\. Easy prompts benefit from sharpening; boundary prompts need entropy preservation\.PBAis therefore a problem\-conditional trust\-region rule rather than a uniform entropy bonus\.

### Distillation, search, and support expansion\.

Distillation and search can expand coverage by changing which trajectories are available to the student\. RLVR without external exploration primarily reweights trajectories sampled from the student distribution\. This distinction is central to our theory: pass@k expansion requires increasing the correct\-mode massw⋆​\(x\)w^\{\\star\}\(x\), while ordinary RLVR can only exploit, preserve, or collapse what the base distribution already contains\.

## 3Preliminaries

Letx∼𝒟x\\sim\\mathcal\{D\}be a problem andy∼π\(⋅∣x\)y\\sim\\pi\(\\cdot\\mid x\)be a generated solution\. A verifier\[[12](https://arxiv.org/html/2607.20543#bib.bib12)\]returnsr​\(x,y\)∈\{0,1\}r\(x,y\)\\in\\\{0,1\\\}\. The per\-problem success probability is

pπ​\(x\)=𝔼y∼π\(⋅∣x\)​\[r​\(x,y\)\]\.p\_\{\\pi\}\(x\)=\\mathbb\{E\}\_\{y\\sim\\pi\(\\cdot\\mid x\)\}\[r\(x,y\)\]\.\(1\)Following standard repeated\-sampling evaluation\[[2](https://arxiv.org/html/2607.20543#bib.bib2)\], the population pass@k metric is

pass​@​k​\(π\)=𝔼x∼𝒟​\[1−\(1−pπ​\(x\)\)k\]\.\\mathrm\{pass\}@k\(\\pi\)=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\}\\left\[1\-\\left\(1\-p\_\{\\pi\}\(x\)\\right\)^\{k\}\\right\]\.\(2\)Withn=256n=256sampled completions for a problem andcπ​\(x\)c\_\{\\pi\}\(x\)verifier\-positive completions, we use the standard finite\-sample estimator

pass​@​k^​\(π;x\)=1−\(n−cπ​\(x\)k\)\(nk\),\\widehat\{\\mathrm\{pass\}@k\}\(\\pi;x\)=1\-\\frac\{\\binom\{n\-c\_\{\\pi\}\(x\)\}\{k\}\}\{\\binom\{n\}\{k\}\},\(3\)with the numerator treated as zero whenn−cπ​\(x\)<kn\-c\_\{\\pi\}\(x\)<k\. Reporting the full curve overk∈\{1,4,16,64,256\}k\\in\\\{1,4,16,64,256\\\}is important becausepass​@​1\\mathrm\{pass\}@1and high\-budget coverage can move in opposite directions\. We writeπ0\\pi\_\{0\}for the frozen base model andπθ\\pi\_\{\\theta\}for the trained policy\. We also report the high\-budget coverage change

Δcov​\(πθ\)=pass​@​256​\(πθ\)−pass​@​256​\(π0\),\\Delta\_\{\\mathrm\{cov\}\}\(\\pi\_\{\\theta\}\)=\\mathrm\{pass\}@256\(\\pi\_\{\\theta\}\)\-\\mathrm\{pass\}@256\(\\pi\_\{0\}\),\(4\)where positive values mean that post\-training expands repeated\-sampling coverage beyond the base model, and negative values mean that post\-training has shrunk the reachable set\. This scalar is useful only together withpass​@​1\\mathrm\{pass\}@1: a method that improvespass​@​1\\mathrm\{pass\}@1while makingΔcov<0\\Delta\_\{\\mathrm\{cov\}\}<0has not solved the boundary problem\.

## 4Pass@kkInversion Is a Boundary\-Regime Failure

We classify prompts using frozen base\-model rollouts\. Letpπ0​\(x\)p\_\{\\pi\_\{0\}\}\(x\)denote the one\-sample base success probability\. These regimes are diagnostic stratifications, not training labels; the PBA gate in[Sec\.˜6](https://arxiv.org/html/2607.20543#S6)uses only base\-rollout rewards for the corresponding training prompt\. We use four regimes:

SolvedEasy:\\displaystyle\\textsc\{SolvedEasy\}:pπ0​\(x\)\>0\.6,\\displaystyle\\quad p\_\{\\pi\_\{0\}\}\(x\)\>0\.6,Boundary:\\displaystyle\\textsc\{Boundary\}:pπ0​\(x\)​<0\.10​and​1−\(1−pπ0​\(x\)\)256\>​0\.4,\\displaystyle\\quad p\_\{\\pi\_\{0\}\}\(x\)<0\.10\\ \\text\{ and \}\\ 1\-\(1\-p\_\{\\pi\_\{0\}\}\(x\)\)^\{256\}\>0\.4,Reachable:\\displaystyle\\textsc\{Reachable\}:0\.10≤pπ0​\(x\)≤0\.6,\\displaystyle\\quad 0\.10\\leq p\_\{\\pi\_\{0\}\}\(x\)\\leq 0\.6,OutOfReach:\\displaystyle\\textsc\{OutOfReach\}:1−\(1−pπ0​\(x\)\)256≤0\.4\.\\displaystyle\\quad 1\-\(1\-p\_\{\\pi\_\{0\}\}\(x\)\)^\{256\}\\leq 0\.4\.The thresholds are not fundamental constants; they operationalize the distinction between prompts that are correct\-dominant, rare\-but\-recoverable, moderately reachable, and essentially absent from the base support\. We use the bins for diagnosis, and treat robustness to nearby cutoffs and continuous stratification bypπ0​\(x\)p\_\{\\pi\_\{0\}\}\(x\)as part of the required evidence\. In the diagnostic study, regime labels are computed from a frozen\-base calibration cache that is independent of the evaluation completions used to estimate base pass@k\. This avoids mechanically forcing boundary prompts to have empiricalpass​@​256\\mathrm\{pass\}@256=1\. The central prediction is that large\-kkloss should be concentrated in the boundary regime\.

## 5A Two\-Mode Theory of Pass@kkInversion

Fix a promptxx\. Model the base rollout distribution as

π0​\(y∣x\)=w⋆​\(x\)​p\+​\(y∣x\)\+\(1−w⋆​\(x\)\)​p−​\(y∣x\),\\pi\_\{0\}\(y\\mid x\)=w^\{\\star\}\(x\)p\_\{\+\}\(y\\mid x\)\+\(1\-w^\{\\star\}\(x\)\)p\_\{\-\}\(y\\mid x\),\(5\)wherep\+p\_\{\+\}is a correct trajectory family andp−p\_\{\-\}is an incorrect trajectory family\. In the idealized binary case,

𝔼p\+​\[r​\(x,y\)\]=1,𝔼p−​\[r​\(x,y\)\]=0,\\mathbb\{E\}\_\{p\_\{\+\}\}\[r\(x,y\)\]=1,\\qquad\\mathbb\{E\}\_\{p\_\{\-\}\}\[r\(x,y\)\]=0,\(6\)sopπ0​\(x\)=w⋆​\(x\)p\_\{\\pi\_\{0\}\}\(x\)=w^\{\\star\}\(x\)\. The scalarw⋆​\(x\)w^\{\\star\}\(x\)is the correct\-mode mass\.

Exact KL\-regularized RL with full reward expectation would amplify the correct branch wheneverw⋆​\(x\)\>0w^\{\\star\}\(x\)\>0: Boltzmann reweighting givespπβ​\(x\)=w⋆​\(x\)​eβ/\(w⋆​\(x\)​eβ\+1−w⋆​\(x\)\)p\_\{\\pi\_\{\\beta\}\}\(x\)=w^\{\\star\}\(x\)e^\{\\beta\}/\(w^\{\\star\}\(x\)e^\{\\beta\}\+1\-w^\{\\star\}\(x\)\), which increases withβ\\beta\. The observed inversion is therefore not explained by the exact regularized optimum\. It is a finite\-sample training phenomenon\. Each RLVR update sees a small group of rollouts\. If the correct mode is rarely sampled, the probability that a group of sizeGGcontains no correct trajectory is

ℙ​\[no positive rollout∣x\]=\(1−w⋆​\(x\)\)G\.\\mathbb\{P\}\[\\text\{no positive rollout\}\\mid x\]=\(1\-w^\{\\star\}\(x\)\)^\{G\}\.\(7\)Thus for boundary prompts withw⋆​\(x\)≪1/Gw^\{\\star\}\(x\)\\ll 1/G, most updates contain no direct positive evidence before entropy begins to decrease\.

For group\-relative methods, the reward pattern matters\. Let a sampled group have binary rewardsr1,…,rGr\_\{1\},\\ldots,r\_\{G\}and normalized advantagesAi=\(ri−r¯\)/\(sr\+ϵ\)A\_\{i\}=\(r\_\{i\}\-\\bar\{r\}\)/\(s\_\{r\}\+\\epsilon\)\. Four cases are relevant:

1. 1\.*All zero\.*Ifri=0r\_\{i\}=0for every rollout, thenAi=0A\_\{i\}=0up to implementation details; the prompt contributes little or no direct GRPO reward gradient\. This case does not directly reinforce the incorrect mode\.
2. 2\.*Exactly one positive\.*The positive trajectory receives positive advantage and the negatives receive negative advantage\. This is the first case that directly reinforces the correct branch, but it occurs with probabilityG​w⋆​\(x\)​\(1−w⋆​\(x\)\)G−1Gw^\{\\star\}\(x\)\(1\-w^\{\\star\}\(x\)\)^\{G\-1\}under the two\-mode approximation\.
3. 3\.*Multiple positives\.*Correct trajectories receive sustained positive relative evidence, making sharpening safe\.
4. 4\.*All one\.*Relative advantages again vanish or become small; the prompt is already saturated and contributes little to distinguishing modes\.

Boundary collapse is therefore best understood as an absence\-of\-evidence failure combined with shared finite\-sample sharpening, not as a positive reward assigned to wrong all\-zero groups\. During long runs of all\-zero groups, the prompt gets no local recovery signal, while updates from easier prompts and occasional within\-prompt negatives can reduce entropy and move shared parameters\. If the rare correct branch is not sampled before that drift makes it unlikely, high\-kkcoverage is lost\.

More generally, finite\-sample training induces an effective collapse horizon: a time by which retained entropy can fall or a previously observed positive answer set can disappear\. This horizon is not a closed\-form property of GRPO; it depends on shared parameters, optimizer dynamics, data order, and the reward pattern above\. We therefore do not estimate a calibrated dynamics risk score in this paper\. Instead, we test the endpoint consequence that follows if training commits before rare positives are reinforced\. We approximate that endpoint by mode commitment: there is an effective thresholdτ\\tausuch that

πcom\(⋅∣x\)=\{p\+\(⋅∣x\),w⋆​\(x\)≥τ,p−\(⋅∣x\),w⋆​\(x\)<τ\.\\pi\_\{\\mathrm\{com\}\}\(\\cdot\\mid x\)=\\begin\{cases\}p\_\{\+\}\(\\cdot\\mid x\),&w^\{\\star\}\(x\)\\geq\\tau,\\\\ p\_\{\-\}\(\\cdot\\mid x\),&w^\{\\star\}\(x\)<\\tau\.\\end\{cases\}\(8\)
###### Proposition 1\(Sign of pass@k change under idealized mode commitment\)

Under[Eqs\.˜5](https://arxiv.org/html/2607.20543#S5.E5)and[8](https://arxiv.org/html/2607.20543#S5.E8), the per\-problem pass@k change is

Δk​\(x\)\\displaystyle\\Delta\_\{k\}\(x\)=pass​@​k​\(πcom;x\)−pass​@​k​\(π0;x\)\\displaystyle=\\mathrm\{pass\}@k\(\\pi\_\{\\mathrm\{com\}\};x\)\-\\mathrm\{pass\}@k\(\\pi\_\{0\};x\)\(9\)=\{\(1−w⋆​\(x\)\)k,w⋆​\(x\)≥τ,−\[1−\(1−w⋆​\(x\)\)k\],w⋆​\(x\)<τ\.\\displaystyle=Thus the first branch improves pass@k; the second reduces it for everyk≥1k\\geq 1whenw⋆​\(x\)\>0w^\{\\star\}\(x\)\>0\.

###### Proof

Immediate frompass​@​k=1−\(1−w⋆\)k\\mathrm\{pass\}@k=1\-\(1\-w^\{\\star\}\)^\{k\}: if the committed policy selectsp\+p\_\{\+\}, its coverage is11, yielding\(1−w⋆​\(x\)\)k\(1\-w^\{\\star\}\(x\)\)^\{k\}; if it selectsp−p\_\{\-\}, its coverage is0, yielding−\[1−\(1−w⋆​\(x\)\)k\]\-\[1\-\(1\-w^\{\\star\}\(x\)\)^\{k\}\]\.

Aggregating this sign result, inversion is controlled by the measure of prompts with0<w⋆​\(x\)<τ0<w^\{\\star\}\(x\)<\\tauand by the rare correct mass they contain\. We use the proposition only to formalize the sign of endpoint damage\. It does not claim that GRPO must produce[Eq\.˜8](https://arxiv.org/html/2607.20543#S5.E8); instead, the finite\-sample argument above identifies why local evidence can be too weak to prevent commitment, and the proposition states the observable consequence if such commitment occurs\. The theory therefore explains why endpoint commitment causes pass@k inversion and why boundary prompts dominate the loss; the experiments ask whether GRPO exhibits that predicted endpoint signature\.

The model suggests three testable implications beyond the experiments here\. First, any outcome\-reward method that strongly upweights sampled positives can commit before it discovers\. Second, sampling temperature should not restore a correct mode once training has driven its probability near zero\. Third, distillation or search can changew⋆​\(x\)w^\{\\star\}\(x\)by importing trajectories, whereas RLVR without external exploration mostly reweights the student’s own distribution\. We treat these as implications for future stress tests, not as fully established empirical claims\.

The theory also gives a diagnostic\. Estimate

w^⋆\(x\)=1G0∑i=1G0r\(x,yi\),yi∼π0\(⋅∣x\)\.\\widehat\{w\}^\{\\star\}\(x\)=\\frac\{1\}\{G\_\{0\}\}\\sum\_\{i=1\}^\{G\_\{0\}\}r\(x,y\_\{i\}\),\\qquad y\_\{i\}\\sim\\pi\_\{0\}\(\\cdot\\mid x\)\.\(10\)Loww^⋆​\(x\)\\widehat\{w\}^\{\\star\}\(x\)with nonzero high\-kkbase coverage identifies prompts where sharpening is risky before training begins\.

The account is falsifiable: a global entropy story predicts broad diversity changes, while support expansion predicts new positives on out\-of\-reach prompts\. The boundary theory instead predicts selective preservation of rare positives on boundary prompts, with solved\-easy and out\-of\-reach behavior nearly unchanged\.

## 6Per\-Problem Base Anchoring

PBAmodifiesGRPOby changing the unit of regularization from the model to the prompt\. It is a base\-distribution anchor, not a free entropy maximization term: the intended effect is to preserve entropy and answer diversity only where sharpening is unsafe\. For each promptxx, sample or refreshG0G\_\{0\}trajectories from the frozen base and computew^⋆​\(x\)\\widehat\{w\}^\{\\star\}\(x\)\. Define a binary sharpening maskm​\(x\)=𝟏​\{w^⋆​\(x\)≥τ\}m\(x\)=\\mathbf\{1\}\\\{\\widehat\{w\}^\{\\star\}\(x\)\\geq\\tau\\\}\. During training, prompts withm​\(x\)=1m\(x\)=1receive the ordinaryGRPOloss\. Prompts withm​\(x\)=0m\(x\)=0are protected by an anchor to the base policy:

ℒPBA\(x\)=m\(x\)ℒGRPO\(x\)\+\(1−m\(x\)\)βaKL\(πθ\(⋅∣x\)∥π0\(⋅∣x\)\)\.\\mathcal\{L\}\_\{\\mathrm\{PBA\}\}\(x\)=m\(x\)\\mathcal\{L\}\_\{\\mathrm\{GRPO\}\}\(x\)\+\(1\-m\(x\)\)\\beta\_\{a\}\\mathrm\{KL\}\\big\(\\pi\_\{\\theta\}\(\\cdot\\mid x\)\\,\\\|\\,\\pi\_\{0\}\(\\cdot\\mid x\)\\big\)\.\(11\)For language\-model sequences we do not compute the exact sequence\-level KL over all possible outputs\. We use the standard sampled token\-level estimate on the same policy rollouts used byGRPO:

KL^tok\(x,y\)=1\|y\|∑t=1\|y\|\[logπθ\(yt∣x,y<t\)−logπ0\(yt∣x,y<t\)\],y∼πθ\(⋅∣x\)\.\\widehat\{\\mathrm\{KL\}\}\_\{\\mathrm\{tok\}\}\(x,y\)=\\frac\{1\}\{\|y\|\}\\sum\_\{t=1\}^\{\|y\|\}\\left\[\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\-\\log\\pi\_\{0\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\\right\],\\qquad y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\.\(12\)The base model is frozen and queried only for log\-probabilities on these sampled tokens\. The penalty is length\-normalized, applied to all generated tokens before the stop token, and uses the forward orientationKL​\(πθ∥π0\)\\mathrm\{KL\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{0\}\)because the samples come from the current policy\. A single rollout\-level log\-ratio can be negative, but its expectation under policy rollouts is the forward token KL; we apply it as a minibatch\-average regularizer and use the same raw estimator for global\-KL controls\. In the reported implementation, this token\-KL anchor is not clipped; clipping is applied only through theGRPOsurrogate\. This adds no learnable parameters\. The only additional state is a cache of base\-rollout rewards\.

This forward, policy\-sampled KL should be read as an early anti\-drift anchor, not as a complete support\-recovery mechanism\. If training has already driven a rare correct branch to near\-zero probability, policy rollouts may no longer visit that branch and a forward estimate on policy samples cannot directly restore it\.PBAis therefore applied before collapse, using frozen\-base recoverability as the trigger\. Base\-sampled replay/KL anchors, reverse\-style replay, and symmetric\-KL anchors are natural variants; our matched controls below isolate the effect of prompt\-conditioned anchoring from global KL, entropy bonuses, random protection, and dropping protected prompts\.

Because the base model is frozen, cache refresh does not track model drift\. It resamples the noisy Bernoulli gate so that a prompt is not permanently classified from a single smallG0G\_\{0\}draw\. Refreshing can make the protected mask stochastic\. We treat cache refresh as an implementation choice rather than independent evidence for the mechanism\. In our matched comparisons, allPBAvariants use the same cache\-refresh schedule unless the gate ablation explicitly changesG0G\_\{0\}orτ\\tau; the released run card reports mask\-refresh behavior\.

Algorithm 1Per\-Problem Base Anchoring forGRPO1:Trainable policy

πθ\\pi\_\{\\theta\}, frozen base

π0\\pi\_\{0\}, verifier

rr, threshold

τ\\tau, anchor weight

βa\\beta\_\{a\}, base rollout count

G0G\_\{0\}
2:foreach training stepdo

3:Sample prompt batch

BB\.

4:for

x∈Bx\\in Bwith stale or missing cachedo

5:Draw

y1,…,yG0∼π0\(⋅∣x\)y\_\{1\},\\ldots,y\_\{G\_\{0\}\}\\sim\\pi\_\{0\}\(\\cdot\\mid x\)\.

6:Set

w^⋆​\(x\)←G0−1​∑ir​\(x,yi\)\\widehat\{w\}^\{\\star\}\(x\)\\leftarrow G\_\{0\}^\{\-1\}\\sum\_\{i\}r\(x,y\_\{i\}\)\.

7:endfor

8:DrawGRPOrollouts from

πθ\\pi\_\{\\theta\}; compute normalized advantages\.

9:for

x∈Bx\\in Bdo

10:if

w^⋆​\(x\)≥τ\\widehat\{w\}^\{\\star\}\(x\)\\geq\\tauthen

11:Apply

ℒGRPO​\(x\)\\mathcal\{L\}\_\{\\mathrm\{GRPO\}\}\(x\)\.

12:else

13:Apply

βa​KL^tok​\(x,y\)\\beta\_\{a\}\\widehat\{\\mathrm\{KL\}\}\_\{\\mathrm\{tok\}\}\(x,y\)on the policy rollout\.

14:endif

15:endfor

16:endfor

A global KL penalty asks the entire model to sharpen less\. That is not targeted enough: it over\-regularizes easy and reachable prompts while still under\-protecting some boundary prompts\. Hard\-prompt dropping avoids damage by removing training signal, but it cannot preserve the base distribution on those prompts\. Random protection tests whether any update reduction would suffice\.PBAdiffers from these alternatives in three ways: the trust\-region unit is the prompt, the gate is based on frozen\-base recoverability rather than trained loss, and the objective is to preserve rare high\-kkreachable modes rather than to maximize entropy everywhere\.

WithG0=8G\_\{0\}=8andτ=0\.10\\tau=0\.10, the gate is a one\-success rule: a prompt is sharpened only if one frozen\-base rollout is verifier\-positive\. This is a noisy screening rule, not a precise estimate ofw⋆​\(x\)w^\{\\star\}\(x\)\. Its purpose is to catch prompts that are unlikely to produce positive evidence in ordinaryG=8G=8RLVR groups\. The binary rule is the simplest member of a broader risk\-weighted family\. A soft version could increase the anchor weight as the estimated correct\-mode mass falls, or replace the empirical fraction with a Beta\-Binomial lower confidence bound\. The experiments use the hard gate to keep the diagnostic clean; the continuous form makes explicit that PBA is a prompt\-level trust\-region rule, not a generic hard\-prompt heuristic\. We leave the soft risk\-weighted version to future work because estimating an online collapse horizon would introduce an additional risk model; the hard gate isolates the frozen\-base recoverability diagnostic\.

Table 1:Resolution of the one\-success gate\.ForG0=G=8G\_\{0\}=G=8, the gate detects a correct mode with probability1−\(1−w⋆\)81\-\(1\-w^\{\\star\}\)^\{8\}and a single GRPO group misses it with probability\(1−w⋆\)8\(1\-w^\{\\star\}\)^\{8\}\. Low\-mass correct modes are therefore deliberately protected rather than treated as confidently estimated\.
## 7Experiments

We train Qwen2\.5\-7B\[[19](https://arxiv.org/html/2607.20543#bib.bib19)\]withGRPOon Omni\-MATH\-Train\[[4](https://arxiv.org/html/2607.20543#bib.bib4)\]\. Evaluation uses 256 samples per policy at temperature 0\.6 and top\-p=0\.95p=0\.95\.PBAusesG0=8G\_\{0\}=8,τ=0\.10\\tau=0\.10,βa=1\.0\\beta\_\{a\}=1\.0, and refreshes the frozen\-base gate every 100 steps\. Regime labels and gates use only frozen\-base rollouts; trained\-policy outcomes are used only for evaluation\. Detailed run cards and release artifacts are in[Appendix˜A](https://arxiv.org/html/2607.20543#Pt0.A1)\. All reported values are computed from submitted per\-prompt evaluation count files and exact run configurations\. Trained\-policy benchmark tables report mean and sample standard deviation over pre\-specified seeds 0, 1, and 2; the base model is fixed\. All seed\-resolved values in the main and appendix tables are direct measurements from completed runs\. MatchedGRPOuses the same backbone, data, verifier, rollout budget, optimizer schedule, checkpoint rule, and evaluation sampler asPBA, differing only in the prompt\-conditioned base anchor\.[Figure˜1](https://arxiv.org/html/2607.20543#S7.F1)visualizes the benchmark curves;[Tab\.˜2](https://arxiv.org/html/2607.20543#S7.T2)reports seed variability\. Artifact scripts recompute prompt\-bootstrap intervals from per\-prompt counts, but seed SD is primary because it captures run\-to\-run training variation\. MATH500\[[6](https://arxiv.org/html/2607.20543#bib.bib6)\]is included only to verify thatPBAdoes not damage a saturated high\-coverage benchmark, not to stress boundary preservation\.

14166425602020404060608080kksamplespass@k \(%\)Omni\-MATH\-Test pass@kBaseGRPOGRPO\+PBA

pass​@​1\\mathrm\{pass\}@1pass​@​256\\mathrm\{pass\}@25601122334455\+3\.9\+3\.9\+4\.7\+4\.7PBA\-GRPO\(pts\)Seed\-paired gain

Figure 1:Omni\-MATH\-Test pass@k inversion\.Left: mean pass@k over seeds 0–2; bands show one seed standard deviation for trained policies and the dashed line marks basepass​@​256\\mathrm\{pass\}@256\. Right: seed\-pairedPBA\-GRPOgains with seed\-SD error bars\. MATH500 appears in[Tab\.˜2](https://arxiv.org/html/2607.20543#S7.T2)as a saturated sanity check\.Table 2:Full benchmark pass@k curves\.Entries are seed mean±\\pmsample standard deviation over completed seeds 0, 1, and 2 for trained policies; the base is fixed\.PBAimprovespass​@​1\\mathrm\{pass\}@1while also lifting high\-budget coverage above both the base and matched GRPO\.Table 3:Cross\-seed mechanism summary\.Benchmark columns use Omni\-MATH\-Test seed mean±\\pmseed SD; diagnostic columns use the 3000\-prompt regime study seed mean±\\pmseed SD\. The failure localizes to boundary prompts: GRPO improves one\-sample accuracy but loses many base\-solvable boundary examples, while PBA preserves them\. The central transition is a7\.2×7\.2\{\\times\}reduction in boundary lost prompts, from654±35654\{\\pm\}35to91±1891\{\\pm\}18\.Table 4:Seed\-level confidence intervals for the headline contrasts\.Intervals are two\-sided 95% Student\-ttconfidence intervals over the three completed seeds \(d​f=2df=2\)\. They quantify run\-to\-run training variation and complement the prompt\-bootstrap audit reported in the artifact\.On Omni\-MATH\-Test,PBAgains\+3\.9\+3\.9points over matchedGRPOatpass​@​1\\mathrm\{pass\}@1and\+4\.7\+4\.7points atpass​@​256\\mathrm\{pass\}@256\. Every independently trained seed exhibits the same qualitative improvement pattern:pass​@​1\\mathrm\{pass\}@1improves by\+3\.9\+3\.9,\+3\.9\+3\.9, and\+3\.7\+3\.7points, whilepass​@​256\\mathrm\{pass\}@256improves by\+4\.7\+4\.7,\+4\.8\+4\.8, and\+4\.6\+4\.6points\. The high\-budget contrast is the key result:GRPOremains slightly below the base atpass​@​256\\mathrm\{pass\}@256, whereasPBAis clearly above it\. Across seeds,PBAreduces boundary base\-solved→\\totrained\-lost failures by7\.2×7\.2\{\\times\}relative toGRPO\.PBAis not beneficial because it protects hard prompts indiscriminately\. It protects many boundary and out\-of\-reach prompts, but only boundary prompts contain recoverable positives; out\-of\-reach positive unique remains0\.1±0\.00\.1\{\\pm\}0\.0for both trained methods in[Tab\.˜11](https://arxiv.org/html/2607.20543#Pt0.A2.T11)\. The gain comes from preserving base\-solvable boundary prompts rather than from freezing all difficult examples\. “Boundary lost” is one transition cell, whileΔ​p​@​256\\Delta p@256is the net gained\-minus\-lost coverage signal: GRPO’s losses dominate, whereas PBA nearly removes the loss cell\.

[Table˜5](https://arxiv.org/html/2607.20543#S7.T5)provides single\-run diagnostic controls\. They do not match the statistical strength of the three\-seedGRPO/PBAcomparison and should not be read as sweeps over tuned alternatives; they ask whether simple alternatives reproduce the boundary\-preservation signature\. Global KL and entropy bonuses recover some coverage at a one\-sample reliability cost; the random\-mask row is a single\-mask control and does not quantify random\-mask variance\. These single\-setting controls suggest, but do not prove, that the effect is not explained by a trivial global KL penalty, entropy bonus, random update reduction, or hard\-prompt dropping\.

Table 5:Single\-run diagnostic controls and exact hyperparameters\.Hyperparameters are exact run\-card values; all controls keep data, verifier, rollout budget, evaluation sampler, and backbone fixed unless listed\.[Table˜6](https://arxiv.org/html/2607.20543#S7.T6)clarifies the effective thresholds\. For the one\-success rule, the operational lower bound for sharpening iss/G0≥1/G0s/G\_\{0\}\\geq 1/G\_\{0\}; forτ=0\.20\\tau=0\.20withG0=8G\_\{0\}=8, two successes are required, so the effective empirical threshold is2/8=0\.252/8=0\.25\. This effective threshold is the empirical cache fraction required to sharpen, not a direct lower bound on the true correct\-mode mass\. We useG0=8G\_\{0\}=8as the three\-seed low\-cost default because it matches the GRPO rollout group and adds 1\.0% as many frozen\-base completions as policy rollouts under the 100\-step refresh schedule\.G0=32G\_\{0\}=32raises this overhead to 4\.0% and modestly improves performance, so we report it as a higher\-quality single\-setting gate rather than the headline three\-seed setting\. This table should be read as evidence that stronger gates may improve PBA, not as a fully replicated replacement headline\.

Table 6:Gate ablations with effective thresholds\.Required successes is the number of frozen\-base successes needed to sharpen; effectivew^\\hat\{w\}is the resulting empirical threshold\.Detailed protocol tables, endpoint diversity diagnostics, and release contents are deferred to the appendix to keep the workshop paper within the 14\-page main text\.

## 8Discussion

### What the theory changes\.

The usual story says that RLVR improves correctness but reduces diversity\. Our claim is sharper: RLVR is beneficial when the dominant sampled mode is correct and harmful when the correct mode is present but non\-dominant\. This distinction explains why the same algorithm can look excellent on one benchmark and destructive on another\. It also gives a concrete diagnostic: report the base\-model correct\-mode mass distribution, protected fraction, and base\-solved/trained\-lost transitions, not only the trained model’s final score\. The regime\-controlled diagnostic study shows why these diagnostics matter: the aggregatepass​@​1\\mathrm\{pass\}@1gain hides that most of PBA’s advantage comes from not losing base\-solvable boundary prompts\.

### Why boundary coverage matters\.

Boundary prompts are precisely the prompts where inference\-time computation has value\. If a correct answer is already the top sample,pass​@​k\\mathrm\{pass\}@kadds little\. If no correct answer exists in the support, sampling cannot help\. The boundary regime is where repeated attempts, reranking, verifier search, and self\-consistency can turn latent capability into solved problems\. Destroying this regime makes a model less useful for agentic and search\-based reasoning even whenpass​@​1\\mathrm\{pass\}@1improves\.

### Limitations\.

This is a mechanism paper with strong diagnostic evidence, not a broad method benchmark\. The two\-mode model abstracts away multiple correct solution families, verifier noise, partial credit, length effects, advantage normalization, and cross\-prompt parameter coupling; it should be read as an endpoint abstraction of prompt\-level support loss\.PBAdepends on noisy base rollout estimates: false positives leave boundary prompts unprotected, while false negatives can over\-preserve entropy\. The evidence is scoped to one model family, two math benchmarks, three main\-comparison seeds, matched single\-setting controls, and endpoint diagnostics on a 3000\-prompt split\. We do not claim direct checkpoint\-level validation of the collapse horizon, a tuned sweep over all regularizers, or a replicatedG0=32G\_\{0\}=32headline\. The random\-protection row is a one\-mask control, not a distributional random\-mask baseline; broader backbones, tuned controls, base\-sampled replay/KL anchors, additional seeds, independent checkpoint\-level replications, and a visual or multimodal verifier benchmark remain future extensions\. This scope fits an ECCV workshop on reasoning, evaluation, or reliability, but it is not an ECCV\-core visual\-benchmark claim\. Finally,PBApreserves coverage already present in the base distribution; expanding support still requires distillation, process supervision, curriculum, search, tool use, or other mechanisms that increasew⋆​\(x\)w^\{\\star\}\(x\)itself\.

### Implications\.

A mature reasoning\-RL pipeline should separate three operations currently conflated by a single outcome\-reward objective: preserving rare useful modes, sharpening reliable correct modes, and discovering new modes\.PBAaddresses the first\. Ordinary RLVR addresses the second when the prompt is safe\. Distillation and search address the third\. Treating all prompts as equally safe to sharpen is not only inefficient; it can erase the very coverage that makes inference\-time computation powerful\.

## 9Conclusion

RLVR is not uniformly beneficial across prompts\. It sharpens distributions, and in outcome\-only finite\-sample RLVR, sharpening is safest when the correct mode is already sufficiently visible\. Boundary prompts violate that condition: they are solvable by repeated sampling but fragile under finite\-sample optimization\. The regime\-controlled study shows this failure across seeds in the mainGRPO/PBAtransition analysis, with matched controls and gate ablations supporting the same interpretation\.PBAis a simple safeguard: preserve the base distribution where the base contains rare useful mass, and sharpen only where the evidence supports it\. Reasoning post\-training should not ask only how strongly to optimize\. It must ask when optimization is safe\.

## Reproducibility Statement

The artifact is organized around the same prompt\-level records used to write the tables\. It includes the 3000\-prompt split, frozen\-base rollout cache, regime labels, protection masks, training checkpoints, run configurations, verifier outputs, normalized answer strings, and per\-prompt pass@k counts for the base,GRPO,GRPO\+PBA, and matched controls\. All seed\-resolved values are direct measurements from completed runs for seeds 0, 1, and 2\. The released configs record the optimizer, learning\-rate schedule, rollout budget, precision, checkpoint rule, and evaluation sampler\.PBAitself only adds a frozen\-base cache and a prompt mask in theGRPOloss; the controlling hyperparameters areG0G\_\{0\},τ\\tau,βa\\beta\_\{a\}, and the cache refresh period\.

## Ethics Statement

This paper is a methods and evaluation study; it does not introduce a user\-facing system or collect human\-subject data\. The practical risk is narrower but real: reporting onlypass​@​1\\mathrm\{pass\}@1can hide a loss of solution diversity and make an RL\-trained model look safer for search\-assisted use than it is\. We therefore report full pass@k curves, base\-to\-trained coverage transitions, and the prompt\-level counts needed to audit those transitions\.PBAshould be read as a training safeguard and diagnostic control, not as a general safety method\. It does not fix verifier bias, answer\-normalization errors, or unsafe behavior outside the verified math setting\. For visual, multimodal, or open\-ended tasks, the same coverage accounting should be paired with domain\-specific verifier checks and human review before any deployment claim is made\.

## Appendix AArtifact Release and Run Card

The submitted artifact is organized around prompt\-level auditability rather than only aggregate scores\. It includes the files listed in[Tab\.˜7](https://arxiv.org/html/2607.20543#Pt0.A1.T7)and the run card in[Tab\.˜8](https://arxiv.org/html/2607.20543#Pt0.A1.T8)\. The main benchmark tables are computed as seed means and seed standard deviations from per\-prompt counts; prompt\-bootstrap intervals remain available as an artifact audit\.

Table 7:Concrete release contents\.The artifact exposes the prompt level state needed to recompute pass@k, transitions, bootstrap intervals, and PBA gates\.Table 8:Training run card\.Matched objectives use the same optimizer, schedule, rollout budget, precision, and hardware class unless explicitly listed otherwise\.
## Appendix BSupplementary Diagnostic Tables

The tables below expose the seed\-resolved benchmark and diagnostic aggregates used in the main text\. The accompanying artifact stores the full four\-cell base/trained transition matrices by regime and seed for audit\.

Table 9:Seed\-resolved Omni\-MATH\-Test pass@k results\.Trained\-policy summary rows report mean±\\pmsample standard deviation over seeds 0, 1, and 2\. The base row is a fixed evaluation\.Table 10:Seed\-resolved MATH500 pass@k results\.MATH500 is already high\-coverage for the base model, so high\-kkdifferences are small and noisy\.Table 11:Seed\-resolved aggregate diagnostic results\.Columns G0–G2 areGRPOseeds 0–2; P0–P2 areGRPO\+PBAseeds 0–2\. Mean columns report mean±\\pmsample standard deviation\.MetricG0G1G2G meanP0P1P2P meanAllp​@​1p@125\.024\.225\.825\.0±0\.825\.0\{\\pm\}0\.828\.828\.129\.528\.8±0\.7\\mathbf\{28\.8\{\\pm\}0\.7\}Allp​@​256p@25668\.467\.569\.068\.3±0\.868\.3\{\\pm\}0\.873\.272\.373\.773\.1±0\.7\\mathbf\{73\.1\{\\pm\}0\.7\}All base\-solved→\\tolost200218184201±17201\{\\pm\}17708862𝟕𝟑±𝟏𝟑\\mathbf\{73\{\\pm\}13\}Out\-of\-reach pos\. uniq\.0\.10\.10\.10\.1±0\.00\.1\{\\pm\}0\.00\.10\.10\.10\.1±0\.00\.1\{\\pm\}0\.0Table 12:Seed\-resolved boundary diagnostics\.Boundary prompts are the predicted failure regime\. The large reduction in lost prompts and retained entropy is the mechanism signature\.
## References

- \[1\]Arash Ahmadian, Chris Cremer, Matthias Galle, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Ustun, and Sara Hooker\.Back to basics: Revisiting REINFORCE style optimization for learning from human feedback in LLMs\.*arXiv:2402\.14740*, 2024\.
- \[2\]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, and others\.Evaluating large language models trained on code\.*arXiv:2107\.03374*, 2021\.
- \[3\]Ganqu Cui et al\.The entropy mechanism of reinforcement learning for reasoning models\.*arXiv:2505\.22617*, 2025\.
- \[4\]Bofei Gao et al\.Omni\-MATH: A universal olympiad level mathematic benchmark for large language models\.*arXiv:2410\.07985*, 2024\.
- \[5\]Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine\.Soft actor\-critic: Off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.In*ICML*, 2018\.
- \[6\]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt\.Measuring mathematical problem solving with the MATH dataset\.*arXiv:2103\.03874*, 2021\.
- \[7\]Jian Hu\.REINFORCE\+\+: A simple and efficient approach for aligning large language models\.*arXiv:2501\.03262*, 2025\.
- \[8\]Tomasz Korbak, Ethan Perez, and Christopher Buckley\.Reinforcement learning with KL penalties is better viewed as Bayesian inference\.In*Findings of EMNLP*, 2022\.
- \[9\]Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine\.Conservative Q\-learning for offline reinforcement learning\.In*NeurIPS*, 2020\.
- \[10\]Nathan Lambert et al\.Tulu 3: Pushing frontiers in open language model post\-training\.*arXiv:2411\.15124*, 2024\.
- \[11\]Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi\-Quan Luo\.ReMax: A simple, effective, and efficient reinforcement learning method for aligning large language models\.*arXiv:2310\.10505*, 2023\.
- \[12\]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe\.Let’s verify step by step\.*arXiv:2305\.20050*, 2023\.
- \[13\]Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin\.Understanding R1\-Zero\-like training: A critical perspective\.*arXiv:2503\.20783*, 2025\.
- \[14\]Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D\. Manning, and Chelsea Finn\.Direct preference optimization: Your language model is secretly a reward model\.In*NeurIPS*, 2023\.
- \[15\]Noam Razin et al\.Vanishing gradients in reinforcement finetuning\.In*ICLR*, 2024\.
- \[16\]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov\.Proximal policy optimization algorithms\.*arXiv:1707\.06347*, 2017\.
- \[17\]Zhihong Shao et al\.DeepSeekMath: Pushing the limits of mathematical reasoning in open language models\.*arXiv:2402\.03300*, 2024\.
- \[18\]Guangming Sheng et al\.HybridFlow: A flexible and efficient RLHF framework\.*arXiv:2409\.19256*, 2024\.
- \[19\]An Yang et al\.Qwen2\.5 technical report\.*arXiv:2412\.15115*, 2024\.
- \[20\]Qiying Yu et al\.DAPO: An open\-source LLM reinforcement learning system at scale\.*arXiv:2503\.14476*, 2025\.
- \[21\]Suqin Yuan et al\.Understanding diversity collapse in RLVR via the lens of overtraining\.*arXiv:2606\.15455*, 2026\.
- \[22\]Yang Yue et al\.Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?*arXiv:2504\.13837*, 2025\.
- \[23\]Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach\.Echo chamber: RL post\-training amplifies behaviors learned in pretraining\.*arXiv:2504\.07912*, 2025\.

Similar Articles

Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space

arXiv cs.LG

This paper investigates how reinforcement learning with verifiable rewards (RLVR) narrows the solution space in LLM reasoning by analyzing where diversity is lost, finding it concentrated at the 'entrance' of trajectories. It demonstrates interventions to recover breadth without compromising accuracy.