Demystifying Reinforcement Learning Post-Training of Language Models

# Demystifying Reinforcement Learning Post-Training of Language Models
Source: [https://arxiv.org/html/2608.24949](https://arxiv.org/html/2608.24949)
Saket GollapudiAffiliation:University of WashingtonSankar HarilalAffiliation:University of WashingtonMin JangAffiliation:University of WashingtonJacob MorrisonAffiliation:University of WashingtonAffiliation:Allen Institute for AISewoong OhAffiliation:University of WashingtonNatasha Jaques\*Equal contributionsAffiliation:University of Washington

###### Abstract

Reinforcement learning \(RL\) post\-training has emerged as a powerful framework for enhancing the capabilities of large language models \(LLMs\), enabling impressive reasoning, math, and coding capabilities\. Yet for many researchers and practitioners, the principles behind classical RL remain a ‘‘black box’’\. In this work, we deconstruct the RL post\-training algorithm, investigating each step to clarify what is actually happening beneath the surface\. By isolating the mechanics of RL with Verifiable Rewards in a controlled and simplified environment, we examine how RL outcomes are shaped by the base model’s prior distribution, the granularity of the reward signal, the diversity of the prompt distribution, and model scale\. We use the entropy of the policy’s output distribution as a lens to compare the distributions learned through pretraining, SFT, and RL post\-training, revealing how each stage shapes model certainty\. Our investigation sheds light on how these choices interact to affect post\-training success\. For example, we show that the effect of so\-called ‘spurious rewards’ depends on the prompt distribution used for post\-training\. We also provide insight into why the success of RL post\-training depends on whether the base model already places sufficient probability mass on the desired behavior, linking it to the classical concept of exploration in RL\. Ultimately, we provide this primer as a resource to those in the NLP community wishing to incorporate RL as a tool in their toolbox\.††Website:[https://minjang10\.github\.io/demystifying\-rl\-finetuning\-web/](https://minjang10.github.io/demystifying-rl-finetuning-web/) Code:[https://github\.com/sankarh\-1/demystifying\-rl\-finetuning](https://github.com/sankarh-1/demystifying-rl-finetuning)

## 1Introduction

Reinforcement Learning \(RL\) has rapidly ascended as a dominant framework for post\-training Large Language Models \(LLMs\), proving essential for aligning models with human intent, enhancing reasoning capabilities, and ensuring safety\([Ouyang et al\., 2022](https://arxiv.org/html/2608.24949#bib.bib7);[Guo et al\., 2025](https://arxiv.org/html/2608.24949#bib.bib8);[Hu et al\., 2024](https://arxiv.org/html/2608.24949#bib.bib1)\)\.

However, its rapid rise in popularity, combined with the fact that understanding RL post\-training for LLMs requires deep expertise in both classical RL and NLP, has led to several misunderstandings about its mechanisms and capabilities in literature\.

We offer this paper as a deep dive into the mechanics of RL post\-training\. Figure[1](https://arxiv.org/html/2608.24949#S1.F1)gives an overview of the post\-training algorithm\. At a high level, it is an iterative process that shapes an LLM’s output distribution using reward signals that might not be differentiable\. Given a prompt, responses are sampled from the model, scalar rewards are assigned based on their quality, and model parameters are adjusted to encourage higher\-reward responses\. Our goal is to study each component of this algorithm through carefully controlled experiments, showing how manipulating the base model distribution, prompt distribution, and reward function all induce changes in the success rate of post\-training a model in order to achieve a particular goal\.

![Refer to caption](https://arxiv.org/html/2608.24949v1/rl_overview.png)Figure 1:Overview of the RL Post\-Training Process\. Our experiments isolate the specific impact of the Base Distribution, the Prompt Distribution \(𝒟\\mathcal\{D\}\), and the Reward Function \(rr\)\. We exclude algorithmic variation from our scope, deferring to the extensive existing research on RL algorithmic choices and engineering techniques that enhance performance \(see e\.g\.\([Khatri et al\., 2025](https://arxiv.org/html/2608.24949#bib.bib20)\)\)Some of the conclusions we draw may seem obvious to researchers already well\-versed in RL\. However, we point out that several aspects of RL post\-training of LLMs make it distinct from classical RL, and require special attention\. First, RL post\-training does not need to explore from scratch to discover highly rewarding behaviors, and instead leverages strong prior knowledge\. The policy being learned is initialized with a base language model and is constrained to stay close to this base reference policy, in order to prevent language drift\([Jaques et al\., 2017](https://arxiv.org/html/2608.24949#bib.bib19)\)\. As we will show, the base model’s distribution exerts strong effects on post\-training outcomes\. Second, post\-training attempts to modify token\-level probabilities, typically using utterance or turn\-level rewards\. Optimizing behavior over a multi\-turn conversation could easily involve sequences10,000\+10,000\+actions, whereas past high\-profile successes in classical RL were limited to 250\-3000 actions[Silver et al\. \(2016\)](https://arxiv.org/html/2608.24949#bib.bib3);[Vinyals et al\. \(2019\)](https://arxiv.org/html/2608.24949#bib.bib4)\. Thus post\-training represents an application of RL at an unprecedented scale, sometimes exhibiting surprising and unintuitive results due to the complexities of the underlying language models[Jørgenvåg et al\. \(2026\)](https://arxiv.org/html/2608.24949#bib.bib12)\.

Because of the KL\-constraint, post\-training of LLMs can be understood as redistributing probability mass within the pre\-trained distribution\. To analyze these distributional shifts, we track the entropy of the policy throughout training, as well as the probability mass assigned to certain behaviors of interest, which allows us to directly compare how pretraining, SFT, and DPO each shape the model’s output distribution and how resilient each is to corruption\.

While prior work has studiedwhatRL learns through the redistribution of probability mass to base model behaviors[Zhao et al\. \(2025\)](https://arxiv.org/html/2608.24949#bib.bib11);[Rajani et al\. \(2025\)](https://arxiv.org/html/2608.24949#bib.bib32);[Yang et al\. \(2026\)](https://arxiv.org/html/2608.24949#bib.bib31);[Akgül et al\. \(2026\)](https://arxiv.org/html/2608.24949#bib.bib30), our goal is studywhyandwhenthis happens, by experimentally manipulating the base model distribution\.

To demonstrate these phenomena, we use a controlled sequence\-generation setting where probability shifts and task success can be measured exactly, utilizing toy RLVR tasks \(producing a target string, and solving a math problem\) to systematically vary the base model’s pre\-training distribution\. Using SFT, we increase the probability of the target string in the base model \(SFT\+\), and also negate the process to disrupt this knowledge from the base model \(SFT\-\)\. Across models of different scales, we analyze how these interventions affect post\-training under different reward signals, including sparse, dense, and random rewards, as well as varying prompt distributions, including on narrow reasoning datasets, and the broad pre\-training distribution used for Tulu\-3\([Lambert et al\., 2025](https://arxiv.org/html/2608.24949#bib.bib22)\)\. Although simplified, this setup enables precise analysis of how factors like the base model’s distribution affect outcomes, an analysis that would be impossible in full\-scale LLM post\-training\. Taken together, these results provide a clearer picture of the mechanisms behind RL post\-training\. We hope this paper will serve as a useful primer for those wishing to gain more expertise in this domain, or deepen their understanding of how to develop more effective post\-training pipelines\.

## 2Related Work

RL post\-training has proven highly effective for LLM reasoning and alignment\([Guo et al\., 2025](https://arxiv.org/html/2608.24949#bib.bib8);[Lambert et al\., 2025](https://arxiv.org/html/2608.24949#bib.bib22);[Ouyang et al\., 2022](https://arxiv.org/html/2608.24949#bib.bib7)\)\. However, recent works have surfaced seemingly contradictory findings and misconceptions that our experiments aim to clarify\.

For example,[Shao et al\. \(2025\)](https://arxiv.org/html/2608.24949#bib.bib10)showed a surprising result: RL with spurious \(random or incorrect\) rewards can still produce performance gains on math benchmarks for a certain base model family\. This has led some to suspect that many post\-training successes could be attributable to this ‘spurious’ phenomenon\. In contrast, our framework shows that this effect is governed by the prompt distribution: restricting training to a narrow prompt distribution can improve performance within that domain \(if the model already assigns high probability to the correct answer\), whereas training with random rewards over a broad prompt distribution increases policy entropy and degrades capabilities\.

Meanwhile, questions have arisen as to whether RL is capable of training the model to learn new behaviors, or whether it merely upweights correct behaviors already present in the base model\([Yue et al\., 2025](https://arxiv.org/html/2608.24949#bib.bib18)\)\. Excellent work has shown that offline methods like DPO, which operate on a fixed dataset, tend to concentrate probability mass on responses that already had high likelihood in the base model\([Ren and Sutherland, 2025](https://arxiv.org/html/2608.24949#bib.bib15)\)\. But is this true of online RL that explores to generate new data as it trains?

While prior work has studiedwhatRL learns through the redistribution of probability mass to base model behaviors[Zhao et al\. \(2025\)](https://arxiv.org/html/2608.24949#bib.bib11);[Rajani et al\. \(2025\)](https://arxiv.org/html/2608.24949#bib.bib32);[Yang et al\. \(2026\)](https://arxiv.org/html/2608.24949#bib.bib31);[Akgül et al\. \(2026\)](https://arxiv.org/html/2608.24949#bib.bib30), our goal is studywhyandwhenthis happens, by experimentally manipulating the base model distribution\. We find, unsurprisingly, that the probability of sampling a given response in the base model strongly affects how easily that response can be learned with post\-training, since the base model’s distribution affects what responses are sampled during post\-training, and thus the quality of exploration needed for RL\. In other words, behaviors with negligible initial probability are difficult to discover and reinforce, a perspective that aligns with theoretical frameworks such as the “Coverage Principle”\([Chen et al\., 2025](https://arxiv.org/html/2608.24949#bib.bib17)\)\.

However, we show that this limitation is strictly bound to settings with sparse rewards\.With a sufficiently dense and correct reward function, RL post\-training can succeed in teaching a model new behaviors that have negligible support in the base model\. While this result is well\-known in classical RL[Eysenbach et al\. \(2018\)](https://arxiv.org/html/2608.24949#bib.bib2), it has been heavily called into question in recent post\-training literature[Yue et al\. \(2025\)](https://arxiv.org/html/2608.24949#bib.bib18);[Shao et al\. \(2025\)](https://arxiv.org/html/2608.24949#bib.bib10);[Wu et al\. \(\)](https://arxiv.org/html/2608.24949#bib.bib16);[Zhao et al\. \(2025\)](https://arxiv.org/html/2608.24949#bib.bib11);[Wu et al\. \(2025\)](https://arxiv.org/html/2608.24949#bib.bib13)\. By systematically varying these components, our work reconciles these seemingly contradictory findings\. For a comprehensive literature review, see Appendix[B](https://arxiv.org/html/2608.24949#A2)\.

## 3Background: RL for Language Models

We formulate the text generation process as a finite horizon Markov decision process \(MDP\), defined as the tupleℳ=⟨𝒮,𝒜,𝒯,r,T⟩\\mathcal\{M\}=\\langle\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{T\},r,T\\rangle\. Here,𝒮\\mathcal\{S\}is the set of possible states,𝒜\\mathcal\{A\}is the set of possible actions,r:𝒮↦ℝr:\\mathcal\{S\}\\mapsto\\mathbb\{R\}is the reward function, andT∈ℤ\+T\\in\\mathbb\{Z\}\_\{\+\}is the finite planning horizon depth\. The transition function is𝒯:𝒮×𝒜×𝒮↦\[0,1\]\\mathcal\{T\}:\\mathcal\{S\}\\times\\mathcal\{A\}\\times\\mathcal\{S\}\\mapsto\[0,1\], which defines the probability of reaching statest\+1s\_\{t\+1\}given actionata\_\{t\}and statests\_\{t\}\. Note that for an LLM deployed in an interactive dialog with a human user,𝒯\\mathcal\{T\}depends on the human’s policy and response\. Therefore, to enable precise analysis, we restrict our focus to the RLVR setting, where we are attempting to learn an auto\-regressive LLM policy \(πθ\\pi\_\{\\theta\}\), and we map these definitions as follows:

- •State \(𝒮\\mathcal\{S\}\): All possible sequences of tokens up to lengthTT\(the promptxxplus the tokens ofyygenerated so far\)\.
- •Action \(𝒜\\mathcal\{A\}\): The model’s vocabulary \(every possible next token\)\.
- •Transition \(𝒯\\mathcal\{T\}\): A deterministic step that appends the next token to the current context, i\.e\.,st\+1←st⊕ats\_\{t\+1\}\\leftarrow s\_\{t\}\\oplus a\_\{t\}\(where⊕\\oplusis the concatenation operator\)\.
- •Horizon \(TT\): The model’s maximum token generation limit\.
- •Reward functionr⁡\(s\)r\(s\): A deterministic verifier applied to the generated sequence \(e\.g\., binary correctness for math\)\.

While this formulation aligns with the classical RL view of sequential decision\-making overℳ\\mathcal\{M\}, the structure of text generation induces a significantly simplified, deterministic system, with verifiable rewards assigned at the sequence level, rather than at intermediate steps\. From an RL perspective, this corresponds to the simplified setting of a contextual bandit; through an NLP lens, the problem is typically viewed as learning a probability distribution over complete sequences, where RL post\-training effectively reshapes this distribution in response to reward signals\.

Process Reward Models \(PRMs\):evaluate intermediate reasoning steps and award partial credit for logical milestones, enabling step\-level credit assignment\. As we will show, this can enable iteratively reshaping the model’s output distribution to explore behaviors that have low probability in the base model, and hence would not be explored or learned in a sparse reward setting\.

Policy learning:During post\-training, a language model acts as a stochastic policyπθ​\(y\|x\)\\pi\_\{\\theta\}\(y\|x\), parameterized by weightsθ\\theta\. The model generates a sequence of tokensy=\(y1,y2,…,yT\)y=\(y\_\{1\},y\_\{2\},\.\.\.,y\_\{T\}\)given a promptxxdrawn from a dataset𝒟\\mathcal\{D\}\. The objective is to maximize the expected rewardr⁡\(x,y\)r\(x,y\)while maintaining proximity to a reference policyπref\\pi\_\{\\text\{ref\}\}\(typically the initial SFT model\) to prevent reward hacking and degradation of language fluency\([Jaques et al\., 2017](https://arxiv.org/html/2608.24949#bib.bib19)\)\. Formally, the objective is:

U\(πθ\)=𝔼x∼𝒟,y∼πθ\[1cr\(x,y\)−DKL\(πθ∥πref\)\]U\(\\pi\_\{\\theta\}\)=\\mathbb\{E\}\_\{x\\sim\{\\cal D\},y\\sim\\pi\_\{\\theta\}\}\\Big\[\\frac\{1\}\{c\}r\(x,y\)\-D\_\{\\rm KL\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\rm ref\}\)\\Big\]where𝒟\{\\cal D\}is a dataset of prompts,DK​LD\_\{KL\}denotes the KL divergence,ccis a scaling constant, andπθ=πθ\(⋅\|x\)\\pi\_\{\\theta\}=\\pi\_\{\\theta\}\(\\cdot\|x\)andπref=πref\(⋅\|x\)\\pi\_\{\\rm ref\}=\\pi\_\{\\rm ref\}\(\\cdot\|x\)are conditioned on the promptxx\.

## 4The Mechanics of RL Post\-Training

The RL post\-training algorithm \(

Figure[1](https://arxiv.org/html/2608.24949#S1.F1)\) relies on three components: a base model, a reward function, and a curated prompt dataset\. To systematically investigate how each element impacts learning dynamics, we conduct two targeted experimental phases in a controlled NLP sandbox\. The first phase isolates the interaction between the base model distribution and the reward function by training the model to reliably generate a specific target stringτ\\tau\(Sections[4\.1](https://arxiv.org/html/2608.24949#S4.SS1)\-[4\.2](https://arxiv.org/html/2608.24949#S4.SS2)\)\. Phase 2 \(Section[4\.3](https://arxiv.org/html/2608.24949#S4.SS3)\) examines how the effect of training on spurious rewards is modulated by the prompt dataset\.

### 4\.1Experimental Design: Manipulating the Base Distribution

To isolate how a model’s learned base distribution shapes RL dynamics \(Figure[1](https://arxiv.org/html/2608.24949#S1.F1), Line 2\), we systematically manipulate the initial probability of a target behavior in the base model\. We evaluate this across two distinct settings: a controlled sequence\-generation sandbox and a combinatorial mathematical reasoning environment\. Across both tasks, we establish three distinct initialization variants to precisely measure how the reference policy’s initial coverage dictates RL success\. TheBasemodel is the unmodified base model \(we test OLMo 3, Qwen2\-7B, Qwen3\-1\.7B, Qwen2\.5\-7B\-Instruct\)\.

TheSFT\+model is the base model fine\-tuned on the correct target behavior to artificially inflate the prior probability of the desired sequence\. TheSFT\-model is the base model fine\-tuned on the opposite objective: to

maximizecross\-entropy loss on the target behavior to suppress the probability of sampling the target behavior to nearly 0%\.

Target string generation\.In our sequence\-generation sandbox, we focus on the target behavior of producing an exact stringτ\\tau, making it possible to precisely measure the probability of this behavior in a particular model, and trivially verify RL success\. We start with producing the movie quoteτ=\\tau=“life is like a box of chocolates you never know what youre gonna get”\. To obtain base models with different probabilities of producingτ\\tau, we use SFT\+/\- where the model is trained to produce the target quoteτ\\tau20% of the time, and on other random quotes sampled from the Cornell Movie\-Dialogs Corpus[Danescu\-Niculescu\-Mizil and Lee \(2011\)](https://arxiv.org/html/2608.24949#bib.bib33)on 80% of samples, to prevent catastrophic forgetting\.

RLVR reasoning\.To demonstrate that the principles of coverage extend to complex problem solving, we scale this setup to multi\-step mathematical reasoning using a problem from the AIME 2025 dataset \(Problem 4\):Find the number of ordered pairs\(x,y\)\(x,y\), where bothxxandyyare integers between−100\-100and100100inclusive, such that:12​x2−x​y−6​y2=012x^\{2\}\-xy\-6y^\{2\}=0\. We can then assess the base model’s probability of achieving the right answer \(117\) across several samples \(128\), and once again experimentally manipulate this probability using similar SFT\+/\- training on the correct answer\.

### 4\.2Reward Function: from Sparse to Dense

A key challenge in applying RLVR is the sparsity of reward signal used to update the policy \(Figure[1](https://arxiv.org/html/2608.24949#S1.F1), Line 6\)\. In verifiable tasks, rewards are typically binary, and applied only at the sequence level\. For our sequence\-generation task,rsparse\(y\)=𝟙\{τ⊆y\}r\_\{\\text\{sparse\}\}\(y\)=\\mathds\{1\}\\\{\\tau\\subseteq y\\\}, with a length penalty applied to prevent reward hacking via “rambling”\([Chevalier\-Boisvert et al\., 2023](https://arxiv.org/html/2608.24949#bib.bib14)\)\. For the math reasoning task, the sparse reward remains a strict binary verifier, assigning 1 if the final extracted answer exactly matches the ground truth, and 0 otherwise\.

Strict sparsity creates a severe exploration bottleneck: if the reference policyπθ\\pi\_\{\\theta\}assigns near\-zero probability to the target, the model may never produceτ\\tau, and thus receive a learning signal\. This mirrors the exploration challenge in systems like DeepSeek\-R1\([Guo et al\., 2025](https://arxiv.org/html/2608.24949#bib.bib8)\), where models receive zero reward unless a long chain\-of\-thought yields the exact correct answer\.

Dense rewards: Edit Distance\.Classical RL frequently trains randomly initialized policies to perform complex, novel tasks\([Christiano et al\., 2023](https://arxiv.org/html/2608.24949#bib.bib9);[Xie et al\., 2025](https://arxiv.org/html/2608.24949#bib.bib23);[Cheng et al\., 2023](https://arxiv.org/html/2608.24949#bib.bib24)\)\. Thus, we hypothesize that LLMs can similarly learn behaviors with vanishingly small base probabilities, if provided with dense rewards \(reward shaping\), which can mitigate exploration bottlenecks by providing feedback on incremental approximations\. Thus, we implement dense reward proxies for both tasks\. For sequence generation, we use Levenshtein distance as a proxy for dense feedback:

D⁡\(i,j\)=\{max⁡\(i,j\),if​i​or​j=0,min⁡\{D⁡\(i−1,j\)\+1,D⁡\(i,j−1\)\+1,D⁡\(i−1,j−1\)\+𝟙​\(yi≠τj\)\},otherwise\.D\(i,j\)=\\begin\{cases\}\\max\(i,j\),&\\text\{if \}i\\text\{ or \}j=0,\\\\\[4\.0pt\] \\min\\left\\\{\\begin\{aligned\} &D\(i\-1,j\)\+1,\\\\ &D\(i,j\-1\)\+1,\\\\ &D\(i\-1,j\-1\)\\\\ &\\qquad\+\\mathbb\{1\}\(y\_\{i\}\\neq\\tau\_\{j\}\)\\end\{aligned\}\\right\\\},&\\text\{otherwise\.\}\\end\{cases\}\(1\)
While it does not provide a per\-token reward along the trajectory, it offers intermediate gradient signals based on structural proximity to the target \(Refer to Appendix[C](https://arxiv.org/html/2608.24949#A3)for additional detail\)\.

Dense rewards: Process Reward Model\.For mathematical reasoning, we replace structural distance with a Process Reward Model \(PRM\) with further details in Appendix[D](https://arxiv.org/html/2608.24949#A4)\. Utilizing an LLM\-as\-a\-judge, the PRM evaluates intermediate reasoning steps, awarding partial credit for discovering any of five key valid sub\-components of the ground\-truth solution\. While PRMs have shown mixed results in prior literature\([Lightman et al\., 2023](https://arxiv.org/html/2608.24949#bib.bib25);[Setlur et al\., 2024](https://arxiv.org/html/2608.24949#bib.bib27);[Tiwari et al\., 2026](https://arxiv.org/html/2608.24949#bib.bib26);[Guo et al\., 2025](https://arxiv.org/html/2608.24949#bib.bib8)\), our idealized setup examining only a single problem enables us to construct an accurate, rich, and dense reward\. We can then test whether a model can be guided through a combinatorial space to construct a novel reasoning chain that has near\-zero support in its original base model’s distribution\.

### 4\.3The Role of Prompt Distributions

Recent work shows that RL post\-training with random rewards can improve reasoning capabilities, but only for the Qwen family of models\([Shao et al\., 2025](https://arxiv.org/html/2608.24949#bib.bib10)\)\. We hypothesize that this effect is fundamentally governed by not only the base model’s distribution, but also the prompt distribution𝒟\\mathcal\{D\}, which determines which states the model explores during RL training \(Figure[1](https://arxiv.org/html/2608.24949#S1.F1)\)\. For any prompt, highly probable responses under the base model are sampled more frequently and thus disproportionately rewarded by chance\. For the special case of training on a narrow distribution of prompts𝒟\\mathcal\{D\}\(e\.g\., exclusively math\), where the base model already has a high probability of sampling the correct answer, even an uninformative, random reward can further reinforce those answers, decreasing policy entropy and potentially increasing capabilities\. Conversely, training on a broad𝒟\\mathcal\{D\}uniformly rewards arbitrary behaviors, increasing overall policy entropy and inducing global unlearning\. In other words, if the reward equally incentivizes all behaviors, the policy may eventually converge to a near\-uniform token distribution, which would not comprise a capable model with good performance\.

To test these hypotheses, we train both Qwen2\.5\-7B\-Math and OLMo 3 base models with random rewards drawn uniformly from\[0,1\]\[0,1\]\. Two base models with contrasting priors let us separate the effect of𝒟\\mathcal\{D\}from the base model’s existing bias\. Qwen2\.5\-7B\-Math carries a strong math prior and reproduces the spurious\-rewards phenomenon; OLMo 3 carries no such prior and its evaluation scores do not artificially improve under random rewards\([Team et al\., 2025](https://arxiv.org/html/2608.24949#bib.bib5)\)\. For Qwen, the broad distribution is drawn from the WildChat dataset\([Zhao et al\., 2024](https://arxiv.org/html/2608.24949#bib.bib29)\)\(10k prompts\) and the narrow distribution is drawn from DeepScaleR\([Tan et al\., 2026](https://arxiv.org/html/2608.24949#bib.bib28)\)\(100 prompts\)\. For OLMo, the broad distribution condition conists of 10k mixed prompts from the OLMo 3 RLVR training mix and the narrow distribution condition consists of 100 math\-only prompts\. We also examine how post\-training starting from three OLMo 3 base models \(Base, SFT, and DPO\) affects results using identical random reward training\. Using three successive checkpoints allows us to test a secondary hypothesis: progressive fine\-tuning reduces the model’s initial output entropy, making it more resilient to the entropy\-increasing effects of random rewards\. To evaluate, we track the log\-likelihood and average per\-token entropy of both in\-distribution and held\-out prompts throughout training, alongside a suite of reasoning benchmarks\. Complete hardware, algorithmic, and evaluation sampling details are deferred to Appendix[F](https://arxiv.org/html/2608.24949#A6)\.

## 5Results: Unpacking Learning Dynamics

\(a\)Pre\-RL probabilities\.
\(b\)Post\-RL probabilities \(sparse rewards\)\.

Table 1:Probability of producing the target quote before and after RL post\-training, \(eval = 10,000 samples\)\.### 5\.1The Coverage Principle

The coverage principle proposes that pre\-training enables post\-training primarily by assigning non\-negligible probability mass to high\-quality data\([Chen et al\., 2025](https://arxiv.org/html/2608.24949#bib.bib17)\)\. To test this empirically, we manipulated base model priors through SFT\+ and SFT\- phases, resulting in models that exhibit differing probabilities of producing the target stringτ\\tau, which are shown in Table[1\(a\)](https://arxiv.org/html/2608.24949#S5.T1.st1)\.

When training these models utilizing a standard sparse reward, we indeed find that the base model’s probability of producing the desired behavior has a strong effect on the success of RL post\-training\. Models artificially injected with the target quote \(SFT\+\) demonstrated quick convergence and easily maximized the sparse reward\. Conversely, models lacking coverage completely failed to learn \(SFT\- and the unmodified Qwen3\-8B Base\)\. Because the target quote was absent from the reference policy, it was never generated during the RL phase, trapping the agent in a state of zero reward\. This failure provides empirical evidence to support the coverage principle: without initial probability mass, RL with sparse rewards does not succeed in learning a new behavior outside of the base model’s original distribution \(see Appendix[6](https://arxiv.org/html/2608.24949#A7.F6)for Qwen2\-7B learning curves\)\.

Furthermore, unmodified base models highlight the strict threshold of sparse RL\. While Qwen2\-7B \(≈3\.5%\\approx 3\.5\\%prior\) eventually converged after a 40\-step exploration plateau, smaller models with lower priors, like Qwen3\-1\.7B, \(≈0\.5%\\approx 0\.5\\%prior\) failed entirely\. With rewards so sparse, hitting the correct sequence by chance was statistically improbable under sparse optimization \(Figure[2\(a\)](https://arxiv.org/html/2608.24949#S5.F2.sf1)\)\.

We observe the same exploration bottleneck in our mathematical reasoning task \(Table[2](https://arxiv.org/html/2608.24949#S5.T2)\)\. Evaluating Qwen2\.5\-7B\-Instruct on AIME Problem 4 with a sparse reward reveals simlar behavior\. Models injected with the correct derivation \(SFT\+\) quickly converged \(∼\\sim85% exact match\), while the unmodified Base model stalled near 10%\. Because the complete reasoning chain was unlikely under the base distribution, it was rarely sampled during exploration, leaving the model with almost no positive learning signal\. This confirms that the Coverage Principle governs real\-world RLVR: without sufficient initial probability mass on complete reasoning trajectories, sparse optimization stalls\.

Table 2:Probability of correctly solving AIME Problem 4 before and after RL post\-training on Qwen2\.5\-7B\-Instruct \(eval = 128 samples\)\.
### 5\.2Dense Reward Shaping: Beyond Pass@k

![Refer to caption](https://arxiv.org/html/2608.24949v1/images/3_rl_curves_combined_raw_Qwen3-1.7B-Base.png)\(a\)Qwen3\-1\.7B, movie quote
![Refer to caption](https://arxiv.org/html/2608.24949v1/images/3_rl_curves_combined_raw_Qwen2.5-7B-Instruct.png)\(b\)Qwen2\.5\-7B\-Instruct, AIME

Figure 2:Overlay of sparse vs\. dense learning dynamics\. RL learning curves for\(a\)Qwen3\-1\.7B on the movie quote task;\(b\)Qwen2\.5\-7B\-Instruct on the AIME task\. Both figures show an example of how dense rewards can successfully guide the model to learn a behavior with very low probability in the base model, which it is not possible to learn with sparse rewards\.A prevailing assumption in recent alignment literature is that RL post\-training cannot formulate new behaviors, but rather strictly upweights the existing “pass@k” capabilities of the base model, acting merely as a selector rather than a driver of novel behavior\([Yue et al\., 2025](https://arxiv.org/html/2608.24949#bib.bib18)\)\. Our next experiment challenges this assumption, by testing if providing the model with sufficiently dense yet accurate rewards can enable learning new behaviors\.

Target string generation\.We first experiment in our sequence\-generation sandbox, by replacing the binary success metric with a dense Levenshtein distance reward to provide the model with intermediate gradient signals based on structural proximity to the target\. Figure[2\(a\)](https://arxiv.org/html/2608.24949#S5.F2.sf1)\) shows learning curves for the Qwen3\-1\.7B model trained with both sparse and dense rewards\. Despite the base model possessing a marginal≈0\.5%\\approx 0\.5\\%base probability of producingτ\\tau, with post\-training failing under standard sparse rewards, the dense signal successfully guided the model out of failure, pulling its final match rate to nearly 50%\. However, as in classical RL we find that reward shaping has its limits; the SFT\+ model trained with dense rewards underperforms the sparse variant in absolute string matching performance, likely because the Levenshtein distance does not sufficiently incentivize producing theexacttarget string over close approximations\.

RLVR reasoning\.We then assess if these results replicate for more complex reasoning tasks, testing whether dense rewards provided through a PRM can overcome base model exploration bottlenecks on an AIME mathematical problem\. As Figure[2\(b\)](https://arxiv.org/html/2608.24949#S5.F2.sf2)shows, when optimizing with standard sparse rewards on this task, the base reasoning model failed to meaningfully improve, flat\-lining at a match rate of approximately 10% \(see Table[2](https://arxiv.org/html/2608.24949#S5.T2)\)\. By contrast, introducing the dense reward radically altered the learning dynamics\. With exponentially increasing dense rewards for correctly reaching each step of the reasoning problem, the Base model’s exact match rate jumped to over 90%\. It even slightly outperformed the artificially inflated SFT\+ model, which plateaued around 85% under both reward types\. Qualitative analysis of the Base model’s generated trajectories reveals that the dense reward successfully guided the model to combine disparate, partial reasoning steps into novel, valid solution paths that differed from the original SFT\+ ground truth\. This shows that RLVR can achieve complex generalization, provided the reward signal is granular enough to credit intermediate logic\.

The SFT\- reasoning experiments also highlight an interesting nuance about partial learning\. While the heavily penalized SFT\- model achieved a 0% exact match rate under both sparse and dense rewards, tracking its trajectory revealed that under the dense reward, its average reward climbed steadily to approximately 0\.6 \(Figure[2\(b\)](https://arxiv.org/html/2608.24949#S5.F2.sf2)\)\. This indicates that even when a policy is actively suppressed from reaching the final correct answer, a dense reward allows it to re\-learn through iteratively building on intermediate reasoning milestones\.

Taken together, these results demonstrate that the limitations often attributed to RLVR—such as the inability to move beyond pass@k priors—are not inherent flaws of RL post\-training, but rather artifacts of the exploration bottleneck caused by sparse, binary reward functions\.

### 5\.3Prompt Distribution: Broad vs\. Narrow

Qwen family\.Figures[3](https://arxiv.org/html/2608.24949#S5.F3)show the effects of training on spurious \(random\) rewards, with both the narrow prompt distribution of[Shao et al\. \(2025\)](https://arxiv.org/html/2608.24949#bib.bib10), and the broad distribution\. The narrow distribution replicates the original results, showing that the capabilities of Qwen models can actually increase with random rewards\. However, when switching to a broad prompt distribution, we see minimal improvements on the MATH benchmark, and degrades performance on AMC\. This is supported by tracking the entropy through training \(Figure\. Under the narrow distribution, entropy remains low through training, while under the broad distribution, it jumps at the first step and remains high\.

![Refer to caption](https://arxiv.org/html/2608.24949v1/images/spurious_rewards_plot.png)Figure 3:Acc@1 on the MATH and AMC datasets after training on narrow and broad𝒟\\mathcal\{D\}; training on narrow𝒟\\mathcal\{D\}yields higher accuracy\.\(a\)Broad distribution: entropy\(b\)Broad distribution: capabilities\(c\)Narrow distribution
Figure 4:Effect of training OLMo models on spurious \(random\) rewards\)\.\(a\)Average entropy on broad𝒟\\mathcal\{D\}\(10k prompts\)\. Each training stage \(Base→\\toSFT→\\toDPO\) delays entropy increase onset, showing increasing resilience\.\(b\)Broad𝒟\\mathcal\{D\}SFT model—entropy spike at step∼\\sim400 coincides with collapse of all eval metrics \(GSM8K, MMLU, IFEval\), confirming global unlearning\.\(c\)Narrow𝒟\\mathcal\{D\}SFT model \(100 math prompts\)—entropy*decreases*, MMLU and IFEval are preserved, but GSM8K collapses, showing targeted corruption of the training domain\.OLMo family\.Figure[4](https://arxiv.org/html/2608.24949#S5.F4)shows the effect of post\-training the OLMo family of models \(base, SFT, and DPO\) on spurious rewards\. Under a broad prompt distribution𝒟\\mathcal\{D\}, we see that the entropy of the models increases \(Fig\.[4\(a\)](https://arxiv.org/html/2608.24949#S5.F4.sf1)\), and that this increase at step∼\\sim400 for the SFT model corresponds to a sharp decrease in capabilities, as measured with the GSM8K, MMLU, and IFEval benchmarks \(Fig\.[4\(b\)](https://arxiv.org/html/2608.24949#S5.F4.sf2)\)\. Notably, log likelihood decreases and entropy increases uniformly across all three domains \(math, code, instruction following\), with no generalization gap\. This indicates the unlearning is global: random rewards on a broad𝒟\\mathcal\{D\}effectively reward any response to any prompt, erasing the model’s learned distribution\.

Under a narrow𝒟\\mathcal\{D\}\(100 math\-only prompts\), the dynamics differ strikingly\. As shown in Figure[4\(c\)](https://arxiv.org/html/2608.24949#S5.F4.sf3), entropy*decreases*slightly across all domains, indicating the model’s distribution is tightening rather than dispersing\. However, this increased certainty does not mean improved performance: GSM8K drops from86%86\\%to∼\\sim32% by step 400, while MMLU \(∼\\sim65→\\to62\) and IFEval \(∼\\sim79→\\to77\) remain largely intact\. Damage is domain\-specific: random rewards corrupt the training domain while preserving capabilities in unsampled domains\.

The breadth of𝒟\\mathcal\{D\}, as well as the base model’s distribution, determine both the scope of degradation, and whether degradation occurs\. A broad𝒟\\mathcal\{D\}causes widespread unlearning\. A narrow𝒟\\mathcal\{D\}increases certainty about the responses the base model was already likely to produce, which can either lead to performance improvement or targeted corruption or improvement, depending on the base model’s initial accuracy\. Although a narrow𝒟\\mathcal\{D\}uses fewer examples with more repetition, this repetition is exactly what drives the policy to collapse toward its prior\. Neither setting creates new capabilities\. Qwen’s gains under a narrow𝒟\\mathcal\{D\}fit this explanation: the same targeted sharpening that harms GSM8K performance in OLMo 3 instead reinforces the correct math reasoning paths already present in Qwen\. This suggests that spurious rewards improve performance only when the prompt distribution is narrow and the base model is already strongly biased toward the target domain, a restricted setting that does not apply to most of RL post\-training\.

## 6Conclusion

In this work, we deconstructed the mechanics of RL post\-training for LLMs to explain how components of the post\-training algorithm—the base model distribution, prompt distribution, and reward structure—affect optimization outcomes\. By isolating the components of the RLVR pipeline within a controlled sequence\-generation sandbox, we explain, empirically verify, or provide a counterpoint to existing post\-training results in the literature\. In particular, we show that standard sparse RL is unable to surpass the base model’s initialization limits\. When a target behavior lacks sufficient baseline coverage, the model cannot sample it, making optimization untenable\. However, we show that this is not a hard and fast limitation of RL post\-training as a technique; a sufficiently dense and accurate reward function can enable learning new behaviors with vanishing support in the base model, a finding that contradicts a popular position in the literature[Yue et al\. \(2025\)](https://arxiv.org/html/2608.24949#bib.bib18);[Shao et al\. \(2025\)](https://arxiv.org/html/2608.24949#bib.bib10);[Wu et al\. \(\)](https://arxiv.org/html/2608.24949#bib.bib16);[Zhao et al\. \(2025\)](https://arxiv.org/html/2608.24949#bib.bib11);[Wu et al\. \(2025\)](https://arxiv.org/html/2608.24949#bib.bib13)\. Furthermore, we show the effect of “spurious rewards” depends entirely on the post\-training prompt distribution\. Training with random rewards on a broad prompt distribution leads to an increase in model entropy and a corresponding catastrophic decrease in capabilities\., By clarifying these interacting mechanics, we hope this work can serve as a resource to the NLP community and those looking to sharpen their understanding of the mechanics of RL post\-training, and help move the field beyond treating RL as a “black box” toward a more controlled and interpretable optimization process\.

## Acknowledgments

This research was supported by the UW\-Amazon Science Gift Hub, UW\-Tsukuba Amazon NVIDIA Cross Pacific AI Initiative \(XPAI\), Sony Research Award, Tinker Research Grants, Character\.AI, DoorDash, Open Philanthropy, Coefficient Giving, Toyota Research Institute, the Schmidt AI2050 Fellows program, and the NSF CISE RI program, award \#2550849\. This material is based upon work supported by the Defense Advanced Research Projects Agency and the Air Force Research Laboratory, contract number\(s\): FA8650\-23\-C\-7316\. Any opinions, findings and conclusions, or recommendations expressed in this material are those of the author\(s\) and do not necessarily reflect the views of AFRL or DARPA\. This work was supported by NSF grants CNS\-2112471, IIS\-2229876, CCF\-2505865, and DMS\-2502281\.

## References

- Akgület al\.\(2026\)Ö\. F\. Akgül, R\. Kannan, W\. Neiswanger, and V\. PrasannaRethinking rl for llm reasoning: it’s sparse policy selection, not capability learning\.arXiv preprint arXiv:2605\.06241\.Cited by:[§1](https://arxiv.org/html/2608.24949#S1.p6.1),[§2](https://arxiv.org/html/2608.24949#S2.p4.1)\.
- Baiet al\.\(2022\)Y\. Bai, S\. Kadavath, S\. Kundu, A\. Askell, J\. Kernion, A\. Jones, A\. Chen, A\. Goldie, A\. Mirhoseini, C\. McKinnon, C\. Chen, C\. Olsson, C\. Olah, D\. Hernandez, D\. Drain, D\. Ganguli, D\. Li, E\. Tran\-Johnson, E\. Perez, J\. Kerr, J\. Mueller, J\. Ladish, J\. Landau, K\. Ndousse, K\. Lukosuite, L\. Lovitt, M\. Sellitto, N\. Elhage, N\. Schiefer, N\. Mercado, N\. DasSarma, R\. Lasenby, R\. Larson, S\. Ringer, S\. Johnston, S\. Kravec, S\. E\. Showk, S\. Fort, T\. Lanham, T\. Telleen\-Lawton, T\. Conerly, T\. Henighan, T\. Hume, S\. R\. Bowman, Z\. Hatfield\-Dodds, B\. Mann, D\. Amodei, N\. Joseph, S\. McCandlish, T\. Brown, and J\. KaplanConstitutional ai: harmlessness from ai feedback\.External Links:2212\.08073,[Link](https://arxiv.org/abs/2212.08073)Cited by:[Appendix B](https://arxiv.org/html/2608.24949#A2.p1.1)\.
- Chenet al\.\(2025\)F\. Chen, A\. Huang, N\. Golowich, S\. Malladi, A\. Block, J\. T\. Ash, A\. Krishnamurthy, and D\. J\. FosterThe coverage principle: how pre\-training enables post\-training\.External Links:2510\.15020,[Link](https://arxiv.org/abs/2510.15020)Cited by:[Appendix B](https://arxiv.org/html/2608.24949#A2.p4.1),[§2](https://arxiv.org/html/2608.24949#S2.p4.1),[§5\.1](https://arxiv.org/html/2608.24949#S5.SS1.p1.1)\.
- Chenget al\.\(2023\)X\. Cheng, K\. Shi, A\. Agarwal, and D\. PathakExtreme parkour with legged robots\.External Links:2309\.14341,[Link](https://arxiv.org/abs/2309.14341)Cited by:[§4\.2](https://arxiv.org/html/2608.24949#S4.SS2.p3.1)\.
- Chevalier\-Boisvertet al\.\(2023\)M\. Chevalier\-Boisvert, B\. Dai, M\. Towers, R\. de Lazcano, L\. Willems, S\. Lahlou, S\. Pal, P\. S\. Castro, and J\. TerryMinigrid & miniworld: modular & customizable reinforcement learning environments for goal\-oriented tasks\.External Links:2306\.13831,[Link](https://arxiv.org/abs/2306.13831)Cited by:[§4\.2](https://arxiv.org/html/2608.24949#S4.SS2.p1.1)\.
- Christianoet al\.\(2023\)P\. Christiano, J\. Leike, T\. B\. Brown, M\. Martic, S\. Legg, and D\. AmodeiDeep reinforcement learning from human preferences\.External Links:1706\.03741,[Link](https://arxiv.org/abs/1706.03741)Cited by:[§4\.2](https://arxiv.org/html/2608.24949#S4.SS2.p3.1)\.
- Danescu\-Niculescu\-Mizil and Lee \(2011\)C\. Danescu\-Niculescu\-Mizil and L\. LeeChameleons in imagined conversations: a new approach to understanding coordination of linguistic style in dialogs\.InProceedings of the Workshop on Cognitive Modeling and Computational Linguistics, ACL 2011,Cited by:[§4\.1](https://arxiv.org/html/2608.24949#S4.SS1.p4.1)\.
- Eysenbachet al\.\(2018\)B\. Eysenbach, A\. Gupta, J\. Ibarz, and S\. LevineDiversity is all you need: learning skills without a reward function\.arXiv preprint arXiv:1802\.06070\.Cited by:[§2](https://arxiv.org/html/2608.24949#S2.p5.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi, X\. Zhang, X\. Yu, Y\. Wu, Z\. F\. Wu, Z\. Gou, Z\. Shao, Z\. Li, Z\. Gao, A\. Liu, B\. Xue, B\. Wang, B\. Wu, B\. Feng, C\. Lu, C\. Zhao, C\. Deng, C\. Ruan, D\. Dai, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Dai, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Xu, H\. Ding, H\. Gao, H\. Qu, H\. Li, J\. Guo, J\. Li, J\. Chen, J\. Yuan, J\. Tu, J\. Qiu, J\. Li, J\. L\. Cai, J\. Ni, J\. Liang, J\. Chen, K\. Dong, K\. Hu, K\. You, K\. Gao, K\. Guan, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Zhao, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, M\. Zhang, M\. Zhang, M\. Tang, M\. Zhou, M\. Li, M\. Wang, M\. Li, N\. Tian, P\. Huang, P\. Zhang, Q\. Wang, Q\. Chen, Q\. Du, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. J\. Chen, R\. L\. Jin, R\. Chen, S\. Lu, S\. Zhou, S\. Chen, S\. Ye, S\. Wang, S\. Yu, S\. Zhou, S\. Pan, S\. S\. Li, S\. Zhou, S\. Wu, T\. Yun, T\. Pei, T\. Sun, T\. Wang, W\. Zeng, W\. Liu, W\. Liang, W\. Gao, W\. Yu, W\. Zhang, W\. L\. Xiao, W\. An, X\. Liu, X\. Wang, X\. Chen, X\. Nie, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Yang, X\. Li, X\. Su, X\. Lin, X\. Q\. Li, X\. Jin, X\. Shen, X\. Chen, X\. Sun, X\. Wang, X\. Song, X\. Zhou, X\. Wang, X\. Shan, Y\. K\. Li, Y\. Q\. Wang, Y\. X\. Wei, Y\. Zhang, Y\. Xu, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Wang, Y\. Yu, Y\. Zhang, Y\. Shi, Y\. Xiong, Y\. He, Y\. Piao, Y\. Wang, Y\. Tan, Y\. Ma, Y\. Liu, Y\. Guo, Y\. Ou, Y\. Wang, Y\. Gong, Y\. Zou, Y\. He, Y\. Xiong, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Y\. X\. Zhu, Y\. Huang, Y\. Li, Y\. Zheng, Y\. Zhu, Y\. Ma, Y\. Tang, Y\. Zha, Y\. Yan, Z\. Z\. Ren, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Xu, Z\. Xie, Z\. Zhang, Z\. Hao, Z\. Ma, Z\. Yan, Z\. Wu, Z\. Gu, Z\. Zhu, Z\. Liu, Z\. Li, Z\. Xie, Z\. Song, Z\. Pan, Z\. Huang, Z\. Xu, Z\. Zhang, and Z\. ZhangDeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:ISSN 1476\-4687,[Link](http://dx.doi.org/10.1038/s41586-025-09422-z),[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[Appendix B](https://arxiv.org/html/2608.24949#A2.p1.1),[§1](https://arxiv.org/html/2608.24949#S1.p1.1),[§2](https://arxiv.org/html/2608.24949#S2.p1.1),[§4\.2](https://arxiv.org/html/2608.24949#S4.SS2.p2.1),[§4\.2](https://arxiv.org/html/2608.24949#S4.SS2.p5.1)\.
- Huet al\.\(2024\)J\. Hu, X\. Wu, Z\. Zhu, Xianyu, W\. Wang, D\. Zhang, and Y\. CaoOpenRLHF: an easy\-to\-use, scalable and high\-performance rlhf framework\.arXiv preprint arXiv:2405\.11143\.Cited by:[§1](https://arxiv.org/html/2608.24949#S1.p1.1)\.
- Jaqueset al\.\(2017\)N\. Jaques, S\. Gu, D\. Bahdanau, J\. M\. Hernández\-Lobato, R\. E\. Turner, and D\. EckSequence tutor: conservative fine\-tuning of sequence generation models with kl\-control\.External Links:1611\.02796,[Link](https://arxiv.org/abs/1611.02796)Cited by:[§1](https://arxiv.org/html/2608.24949#S1.p4.1),[§3](https://arxiv.org/html/2608.24949#S3.p4.1)\.
- Jørgenvåget al\.\(2026\)M\. Jørgenvåg, D\. Kaczér, L\. Ruttert, M\. Gülhan, L\. Flek, and F\. MaiReinforcement learning amplifies emergent misalignment from harmless rewards\.arXiv preprint arXiv:2605\.31328\.Cited by:[§1](https://arxiv.org/html/2608.24949#S1.p4.1)\.
- Khatriet al\.\(2025\)D\. Khatri, L\. Madaan, R\. Tiwari, R\. Bansal, S\. S\. Duvvuri, M\. Zaheer, I\. S\. Dhillon, D\. Brandfonbrener, and R\. AgarwalThe art of scaling reinforcement learning compute for llms\.External Links:2510\.13786,[Link](https://arxiv.org/abs/2510.13786)Cited by:[Figure 1](https://arxiv.org/html/2608.24949#S1.F1)\.
- Lambertet al\.\(2025\)N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda, A\. Liu, N\. Dziri, S\. Lyu, Y\. Gu, S\. Malik, V\. Graf, J\. D\. Hwang, J\. Yang, R\. L\. Bras, O\. Tafjord, C\. Wilhelm, L\. Soldaini, N\. A\. Smith, Y\. Wang, P\. Dasigi, and H\. HajishirziTulu 3: pushing frontiers in open language model post\-training\.External Links:2411\.15124,[Link](https://arxiv.org/abs/2411.15124)Cited by:[Appendix B](https://arxiv.org/html/2608.24949#A2.p1.1),[§1](https://arxiv.org/html/2608.24949#S1.p7.1),[§2](https://arxiv.org/html/2608.24949#S2.p1.1)\.
- Lightmanet al\.\(2023\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.External Links:2305\.20050,[Link](https://arxiv.org/abs/2305.20050)Cited by:[§4\.2](https://arxiv.org/html/2608.24949#S4.SS2.p5.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. LoweTraining language models to follow instructions with human feedback\.External Links:2203\.02155,[Link](https://arxiv.org/abs/2203.02155)Cited by:[Appendix B](https://arxiv.org/html/2608.24949#A2.p1.1),[§1](https://arxiv.org/html/2608.24949#S1.p1.1),[§2](https://arxiv.org/html/2608.24949#S2.p1.1)\.
- Rajaniet al\.\(2025\)N\. Rajani, A\. P\. Gema, S\. Goldfarb\-Tarrant, and I\. TitovScalpel vs\. hammer: grpo amplifies existing capabilities, sft replaces them\.arXiv preprint arXiv:2507\.10616\.Cited by:[§1](https://arxiv.org/html/2608.24949#S1.p6.1),[§2](https://arxiv.org/html/2608.24949#S2.p4.1)\.
- Ren and Sutherland \(2025\)Y\. Ren and D\. J\. SutherlandLearning dynamics of llm finetuning\.External Links:2407\.10490,[Link](https://arxiv.org/abs/2407.10490)Cited by:[Appendix B](https://arxiv.org/html/2608.24949#A2.p5.1),[§2](https://arxiv.org/html/2608.24949#S2.p3.1)\.
- Setluret al\.\(2024\)A\. Setlur, C\. Nagpal, A\. Fisch, X\. Geng, J\. Eisenstein, R\. Agarwal, A\. Agarwal, J\. Berant, and A\. KumarRewarding progress: scaling automated process verifiers for llm reasoning\.External Links:2410\.08146,[Link](https://arxiv.org/abs/2410.08146)Cited by:[§4\.2](https://arxiv.org/html/2608.24949#S4.SS2.p5.1)\.
- Shaoet al\.\(2025\)R\. Shao, S\. S\. Li, R\. Xin, S\. Geng, Y\. Wang, S\. Oh, S\. S\. Du, N\. Lambert, S\. Min, R\. Krishna, Y\. Tsvetkov, H\. Hajishirzi, P\. W\. Koh, and L\. ZettlemoyerSpurious rewards: rethinking training signals in rlvr\.External Links:2506\.10947,[Link](https://arxiv.org/abs/2506.10947)Cited by:[Appendix B](https://arxiv.org/html/2608.24949#A2.p2.1),[§2](https://arxiv.org/html/2608.24949#S2.p2.1),[§2](https://arxiv.org/html/2608.24949#S2.p5.1),[§4\.3](https://arxiv.org/html/2608.24949#S4.SS3.p1.1),[§5\.3](https://arxiv.org/html/2608.24949#S5.SS3.p1.1),[§6](https://arxiv.org/html/2608.24949#S6.p1.1)\.
- Silveret al\.\(2016\)D\. Silver, A\. Huang, C\. J\. Maddison, A\. Guez, L\. Sifre, G\. Van Den Driessche, J\. Schrittwieser, I\. Antonoglou, V\. Panneershelvam, M\. Lanctot,et al\.Mastering the game of go with deep neural networks and tree search\.nature529\(7587\),pp\. 484–489\.Cited by:[§1](https://arxiv.org/html/2608.24949#S1.p4.1)\.
- Stiennonet al\.\(2022\)N\. Stiennon, L\. Ouyang, J\. Wu, D\. M\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. ChristianoLearning to summarize from human feedback\.External Links:2009\.01325,[Link](https://arxiv.org/abs/2009.01325)Cited by:[Appendix B](https://arxiv.org/html/2608.24949#A2.p1.1)\.
- Tanet al\.\(2026\)S\. Tan, M\. Luo, J\. Wong, C\. Cai, X\. Shi, W\. Y\. Tang, M\. Roongta, T\. Zhang, L\. E\. Li, R\. A\. Popa, and I\. StoicaDeepScaleR: effective RL scaling of reasoning models via iterative context lengthening\.External Links:[Link](https://openreview.net/forum?id=I6GzDCne7U)Cited by:[§4\.3](https://arxiv.org/html/2608.24949#S4.SS3.p2.1)\.
- Teamet al\.\(2025\)O\. Team, A\. Ettinger, A\. Bertsch, B\. Kuehl, D\. Graham, D\. Heineman, D\. Groeneveld, F\. Brahman, F\. Timbers, H\. Ivison, J\. Morrison, J\. Poznanski, K\. Lo, L\. Soldaini, M\. Jordan, M\. Chen, M\. Noukhovitch, N\. Lambert, P\. Walsh, P\. Dasigi, R\. Berry, S\. Malik, S\. Shah, S\. Geng, S\. Arora, S\. Gupta, T\. Anderson, T\. Xiao, T\. Murray, T\. Romero, V\. Graf, A\. Asai, A\. Bhagia, A\. Wettig, A\. Liu, A\. Rangapur, C\. Anastasiades, C\. Huang, D\. Schwenk, H\. Trivedi, I\. Magnusson, J\. Lochner, J\. Liu, L\. Miranda, M\. Sap, M\. Morgan, M\. Schmitz, M\. Guerquin, M\. Wilson, R\. Huff, R\. L\. Bras, R\. Xin, R\. Shao, S\. Skjonsberg, S\. Z\. Shen, S\. S\. Li, T\. Wilde, V\. Pyatkin, W\. Merrill, Y\. Chang, Y\. Gu, Z\. Zeng, A\. Sabharwal, L\. Zettlemoyer, P\. W\. Koh, A\. Farhadi, N\. A\. Smith, and H\. HajishirziOlmo 3\.InTechnical Report, 2025,Cited by:[§4\.3](https://arxiv.org/html/2608.24949#S4.SS3.p2.1)\.
- Tiwariet al\.\(2026\)R\. Tiwari, A\. Tomar, U\. Bamba, M\. Maheswaran, H\. Yang, M\. W\. Mahoney, K\. Keutzer, and A\. GholamiReward under attack: analyzing the robustness and hackability of process reward models\.External Links:2603\.06621,[Link](https://arxiv.org/abs/2603.06621)Cited by:[§4\.2](https://arxiv.org/html/2608.24949#S4.SS2.p5.1)\.
- Vinyalset al\.\(2019\)O\. Vinyals, I\. Babuschkin, W\. M\. Czarnecki, M\. Mathieu, A\. Dudzik, J\. Chung, D\. H\. Choi, R\. Powell, T\. Ewalds, P\. Georgiev,et al\.Grandmaster level in starcraft ii using multi\-agent reinforcement learning\.nature575\(7782\),pp\. 350–354\.Cited by:[§1](https://arxiv.org/html/2608.24949#S1.p4.1)\.
- Wuet al\.\(2025\)F\. Wu, A\. Tu, W\. Xuan, H\. Qi, X\. Huang, Q\. Zeng, S\. Talaei, Y\. Xiao, P\. Xia, X\. Tang,et al\.Position: the hidden costs and measurement gaps of reinforcement learning with verifiable rewards\.arXiv preprint arXiv:2509\.21882\.Cited by:[§2](https://arxiv.org/html/2608.24949#S2.p5.1),[§6](https://arxiv.org/html/2608.24949#S6.p1.1)\.
- \[28\]F\. Wu, W\. Xuan, X\. Lu, M\. Liu, Y\. Dong, Z\. Harchaoui, and Y\. ChoiThe invisible leash: why rlvr may or may not escape its origin, 2026\.URL https://arxiv\. org/abs/2507\.14843\.Cited by:[§2](https://arxiv.org/html/2608.24949#S2.p5.1),[§6](https://arxiv.org/html/2608.24949#S6.p1.1)\.
- Xieet al\.\(2025\)W\. Xie, J\. Han, J\. Zheng, H\. Li, X\. Liu, J\. Shi, W\. Zhang, C\. Bai, and X\. LiKungfuBot: physics\-based humanoid whole\-body control for learning highly\-dynamic skills\.External Links:2506\.12851,[Link](https://arxiv.org/abs/2506.12851)Cited by:[§4\.2](https://arxiv.org/html/2608.24949#S4.SS2.p3.1)\.
- Yanget al\.\(2026\)W\. Yang, H\. Zang, J\. Zhang, W\. Shi, D\. Su, J\. Wang, X\. Cheng, and F\. SunBeyond reasoning: reinforcement learning unlocks parametric knowledge in llms\.arXiv preprint arXiv:2605\.07153\.Cited by:[§1](https://arxiv.org/html/2608.24949#S1.p6.1),[§2](https://arxiv.org/html/2608.24949#S2.p4.1)\.
- Yueet al\.\(2025\)Y\. Yue, Z\. Chen, R\. Lu, A\. Zhao, Z\. Wang, Y\. Yue, S\. Song, and G\. HuangDoes reinforcement learning really incentivize reasoning capacity in llms beyond the base model?\.External Links:2504\.13837,[Link](https://arxiv.org/abs/2504.13837)Cited by:[Appendix B](https://arxiv.org/html/2608.24949#A2.p3.1),[§2](https://arxiv.org/html/2608.24949#S2.p3.1),[§2](https://arxiv.org/html/2608.24949#S2.p5.1),[§5\.2](https://arxiv.org/html/2608.24949#S5.SS2.p1.1),[§6](https://arxiv.org/html/2608.24949#S6.p1.1)\.
- Zhaoet al\.\(2025\)R\. Zhao, A\. Meterez, S\. Kakade, C\. Pehlevan, S\. Jelassi, and E\. MalachEcho chamber: rl post\-training amplifies behaviors learned in pretraining\.arXiv preprint arXiv:2504\.07912\.Cited by:[§1](https://arxiv.org/html/2608.24949#S1.p6.1),[§2](https://arxiv.org/html/2608.24949#S2.p4.1),[§2](https://arxiv.org/html/2608.24949#S2.p5.1),[§6](https://arxiv.org/html/2608.24949#S6.p1.1)\.
- Zhaoet al\.\(2024\)W\. Zhao, X\. Ren, J\. Hessel, C\. Cardie, Y\. Choi, and Y\. DengWildChat: 1m chatgpt interaction logs in the wild\.External Links:2405\.01470,[Link](https://arxiv.org/abs/2405.01470)Cited by:[§4\.3](https://arxiv.org/html/2608.24949#S4.SS3.p2.1)\.

## Appendix AResource Details

### A\.1Computational Specifications

All experiments were run on either NVIDIA A100 80GB GPU or NVIDIA L40s 48GB\. The Math reasoning task, an NVIDIA A100 80GB GPU was used for the main experiments, while an NVIDIA L40s 48GB was used for the PRM LLM\-as\-a\-Judge\.

### A\.2LLM Usage

LLMs were used in the development of the code base/experiments, specifically modifying behavior to adapt to resource limitations, as well as simplify experiment pipeline for generating results\.

## Appendix BExpanded Related Work

Post\-training with RL has proven effective for a wide range of tasks, particularly for improving reasoning capabilities and aligning LLMs with human preferences\. For instance, RL with Verifiable Rewards \(RLVR\) has shown that models can achieve state\-of\-the\-art performance with smaller parameter counts, such as DeepSeek\-R1\([Guo et al\., 2025](https://arxiv.org/html/2608.24949#bib.bib8);[Lambert et al\., 2025](https://arxiv.org/html/2608.24949#bib.bib22)\), highlighting its role in developing efficient AI\. Through Reinforcement Learning from Human Feedback \(RLHF\), reward signals are learned from preference data, leading to improvements in instruction following and response quality\([Ouyang et al\., 2022](https://arxiv.org/html/2608.24949#bib.bib7);[Bai et al\., 2022](https://arxiv.org/html/2608.24949#bib.bib21);[Stiennon et al\., 2022](https://arxiv.org/html/2608.24949#bib.bib6)\)\. Across both settings, RL post\-training has shown to yield strong gains on reasoning and alignment tasks, making a mechanistic understanding of its success essential\.

Several recent works have surfaced seemingly contradictory findings about RL post\-training\.[Shao et al\. \(2025\)](https://arxiv.org/html/2608.24949#bib.bib10)demonstrate that RLVR can yield substantial gains in mathematical reasoning, even when reward signals are weak, random, or negatively correlated with the actual answer\. Notably, they find that spurious rewards still trigger systematic behavioral shifts, such as increased code reasoning frequency in Qwen2\.5\-Math, suggesting the base model’s pretraining dictates the form of emergent behavior, not the reward\. Moreover, they observe that these gains do not generalize consistently across different model families; for example, Qwen2\.5 and OLMo2 exhibit different behavioral trends to these spurious rewards\. These findings appear to challenge a core idea in RL: that improvements arise from optimizing correctly specified reward signals\. Our work addresses these questions by showing that the effect of random rewards depends critically on the prompt distribution\. A broad prompt distribution causes global entropy increase and capability degradation, while a narrow distribution causes targeted corruption of the training domain while preserving other capabilities\. The prompt distribution thus controls thescopeof degradation, not whether degradation occurs\.

Moreover, the original base model plays a central role in determining post\-training success\([Yue et al\., 2025](https://arxiv.org/html/2608.24949#bib.bib18)\)\. Measuring pass@k \(the probability of solving a problem inkkattempts\), the authors find that while RL\-trained models excel with small values ofkk, base models catch up askkincreases\. This suggests RLVR primarily improves sampling efficiency, refining existing behaviors rather than creating novel ones\. We build on this by comparing outcomes across models with different baseline capability distributions under controlled reward conditions, and show that a sufficiently shaped reward can overcome this problem, and teach new behaviors to models that previously did not generate them with high probability\.

[Chen et al\. \(2025\)](https://arxiv.org/html/2608.24949#bib.bib17)introduce the “Coverage Principle“ and argue that existing metrics \(e\.g\. cross\-entropy loss\) are often poor predictors of post\-training success, instead stating the coverage \(the total probability mass the pre\-trained model assigns to the set of high\-quality responses\) is the critical link\. This motivates our choice to treat this notion of coverage as an independent variable; by selecting base models with varying coverage over target tasks, we can test how it interacts with the effect of reward density, allowing us to disentangle these two sources of variation\.

[Ren and Sutherland \(2025\)](https://arxiv.org/html/2608.24949#bib.bib15)develop a framework to illustrate why post\-training often redistributes probability mass rather than creating new capabilities by decomposing changes in log\-probabilities after gradient updates\. While their results are restricted to Direct Preference Optimization \(DPO\), this perspective accounts for observed behavioral shifts such as hallucinations and highlights the complex dynamics underlying alignment and performance gains\. We adopt this framing for interpreting behavioral changes, as gains in the target task likely reflect probability mass redistribution rather than the emergence of new capabilities\.

## Appendix CReward Function Implementations \- Movie Quotes

In our sandbox experiments, we utilize two distinct reward functions to evaluate the impact of reward shaping on RL optimization: a sparse reward \(equipped with an anti\-rambling penalty\) and a dense Levenshtein\-based reward\. Letyydenote the model’s generated sequence andτ\\taudenote the exact target string\.

### C\.1Sparse Reward with Length Penalty

The sparse reward acts primarily as a binary success metric\. It assigns a base reward of 1 if the target stringτ\\tauis successfully generated, and 0 if the model fails to produce the target entirely\.

To prevent reward hacking by “rambling”, we use a length penalty termppto decrease the reward for any extra tokens generated other than the target string\. Letnnbe the maximum token generation limit andssbe the number of extra tokens generated\. Then,ppis defined as

p=\{0ifs=0snifs\>0p=\\begin\{cases\}0&\\text\{if $s=0$\}\\\\ \\frac\{s\}\{n\}&\\text\{if $s\>0$\}\\end\{cases\}\(2\)and the sparse reward function is defined as

rsparse​\(y,τ\)=\{max⁡\(0\.5,1−p\)if​τ∈y​and​\|y\|\>\|τ\|0otherwiser\_\{\\text\{sparse\}\}\(y,\\tau\)=\\begin\{cases\}\\max\(0\.5,1\-p\)&\\text\{if \}\\tau\\in y\\text\{ and \}\|y\|\>\|\\tau\|\\\\ 0&\\text\{otherwise\}\\end\{cases\}\(3\)\*\(Note:ppcan be defined as a fixed scalar penalty or a function of the excess length\(\|y\|−\|τ\|\)\(\|y\|\-\|\\tau\|\)\)\.\* This ensures the sparse optimization baseline actively discourages rambling and aligns only the precise target behavior\.

### C\.2Dense Reward \(Levenshtein Distance\)

To address the credit assignment problem for models lacking initial probability mass, we implement a dense reward that provides continuous, intermediate gradient signals based on structural proximity to the target\.

We utilize the standard Levenshtein edit distance, which calculates the minimum number of single\-character edits \(insertions, deletions, or substitutions\) required to transform the generated sequence into the target sequence\. Formally, letyyandτ\\taube strings of lengthsmmandnn, and letD⁡\(i,j\)D\(i,j\)denote the Levenshtein distance between the firstiicharacters ofyyandjjcharacters ofτ\\tau, whereyiy\_\{i\}andτj\\tau\_\{j\}are theii\-th andjj\-th characters\. The Levenshtein distance is defined for boundary cases asD⁡\(i,j\)=max⁡\(i,j\)D\(i,j\)=\\max\(i,j\)wheni=0i=0orj=0j=0\. Otherwise, it is calculated as:

D⁡\(i,j\)=min⁡\{D⁡\(i−1,j\)\+1D⁡\(i,j−1\)\+1D⁡\(i−1,j−1\)\+𝟙​\(yi≠τj\)D\(i,j\)=\\min\\begin\{cases\}D\(i\-1,j\)\+1\\\\ D\(i,j\-1\)\+1\\\\ D\(i\-1,j\-1\)\+\\mathbb\{1\}\(y\_\{i\}\\neq\\tau\_\{j\}\)\\end\{cases\}\(4\)
Letn=max⁡\(len​\(y\),len​\(τ\)\)n=\\max\(\\text\{len\}\(y\),\\text\{len\}\(\\tau\)\)andDDbe the Levenshtein distance betweenyyandτ\\tau\. Then, reward function is then defined as

rd​e​n​s​e=max⁡\(0,1−Dn\)r\_\{dense\}=\\max\(0,1\-\\frac\{D\}\{n\}\)\(5\)

## Appendix DExponential Process Reward Model \(PRM\) Implementation

For our math reasoning experiment, in order to scale our analysis from structural sequence\-matching \(Levenshtein distance\) to combinatorial mathematical reasoning, we replaced the string\-based dense reward with a Process Reward Model \(PRM\)\. Mathematical reasoning requires logical verification across diverse, valid solution paths \(e\.g\., direct factoring, quadratic formula, de\-homogenization\)\. To achieve this, we deployed a 32\-billion parameter instruction\-tuned model \(Qwen2\.5\-32B\-Instruct\) functioning as an LLM\-as\-a\-Judge\.

The PRM evaluates the generated trajectoryyyat the end of each rollout, classifying the strategy used and evaluating the presence of specific, sequential logical milestones\.

### D\.1Logical Milestones and Reward Curve

The AIME problem utilized in our experiments \(12​x2−x​y−6​y2=012x^\{2\}\-xy\-6y^\{2\}=0\) requires a multi\-step algebraic derivation\. We instruct the PRM to extract a boolean vectorM⁡\(y\)∈\{0,1\}5M\(y\)\\in\\\{0,1\\\}^\{5\}, representing the successful completion of the following five milestones:

- •m1m\_\{1\}: Initiated a correct algebraic path configuration \(e\.g\., setting up factorization\)\.
- •m2m\_\{2\}: Derived accurate base root relationships betweenxxandyy\.
- •m3m\_\{3\}: Evaluated the integer boundary constraints \(−100≤x,y≤100\-100\\leq x,y\\leq 100\) correctly\.
- •m4m\_\{4\}: Calculated the correct solution count for at least one case branch\.
- •m5m\_\{5\}: Identified and correctly subtracted the intersection overlap at the origin\(0,0\)\(0,0\)\.

A critical challenge in applying linear partial credit \(e\.g\.,\+0\.15\+0\.15per step\) is the introduction of local minima\. An optimizing policy may discover that completing three steps and halting yields a “good enough” reward, dampening the advantage gradient required to risk generating further tokens to reach the final answer\. To prevent this advantage compression, we map the milestones to a reward curve\. The step weights are defined asw=\[0\.05,0\.05,0\.10,0\.15,0\.25\]w=\[0\.05,0\.05,0\.10,0\.15,0\.25\]\. The base process reward is calculated as:

rp​r​o​c​e​s​s​\(y\)=∑i=15wi⋅mir\_\{process\}\(y\)=\\sum\_\{i=1\}^\{5\}w\_\{i\}\\cdot m\_\{i\}\(6\)This exponential scaling strictly enforces increasing marginal value, ensuring that advanced reasoning steps provide a stronger gradient pull than early exploratory steps\.

### D\.2Outcome Overrides and Defensive Penalties

To anchor the PRM to the absolute ground truth and prevent reward hacking, the final reward incorporates deterministic outcome overrides and active penalties\.

Letτ=117\\tau=117be the exact correct answer, andτn​e​a​r=118\\tau\_\{near\}=118represent the specific, known near\-miss where the model successfully completes the calculus but fails to account for the origin overlap\. Letextract​\(y\)\\text\{extract\}\(y\)be a deterministic parsing function that extracts the integer from the final`\\boxed\{\}`command\. The unpenalized rewardrb​a​s​er\_\{base\}is defined as:

rb​a​s​e​\(y\)=\{1\.0ifextract​\(y\)=τ0\.6ifextract​\(y\)=τn​e​a​rrp​r​o​c​e​s​s​\(y\)otherwiser\_\{base\}\(y\)=\\begin\{cases\}1\.0&\\text\{if \}\\text\{extract\}\(y\)=\\tau\\\\ 0\.6&\\text\{if \}\\text\{extract\}\(y\)=\\tau\_\{near\}\\\\ r\_\{process\}\(y\)&\\text\{otherwise\}\\end\{cases\}\(7\)
Finally, LLM policies optimizing against dense rewards frequently attempt to “farm” tokens by redundantly repeating early logical steps that are known to yield partial credit\. To counteract this, the PRM is instructed to flag cyclic reasoning with a boolean penalty markerml​o​o​pm\_\{loop\}\. If the model loops without reaching the final exact answer, a penaltypl​o​o​p=−0\.3p\_\{loop\}=\-0\.3is applied\. Furthermore, a strict formatting penaltypf​o​r​m​a​t=−0\.5p\_\{format\}=\-0\.5is applied if the trajectory fails to conclude with the`\\boxed\{\}`delimiter\.

The final dense reward is bounded at00and defined as:

rd​e​n​s​e​\(y\)=max⁡\(0,rb​a​s​e​\(y\)\+pl​o​o​p\+pf​o​r​m​a​t\)r\_\{dense\}\(y\)=\\max\\Big\(0,\\;r\_\{base\}\(y\)\+p\_\{loop\}\+p\_\{format\}\\Big\)\(8\)
By heavily penalizing redundancies and exponentially rewarding progress, this PRM structure successfully bridges the exploration gap for models with near\-zero initial priors\.

## Appendix ERLVR Algorithm

Algorithm 1RLVR Algorithm1:Input:base model

πr​e​f\\pi\_\{ref\}, prompt dataset

DD, reward function

r⁡\(x,y\)r\(x,y\)
2:Initialize:

πθ←πr​e​f\\pi\_\{\\theta\}\\leftarrow\\pi\_\{ref\}
3:fortrain stepsdo

4:Sample input:

x∼𝒟x\\sim\\mathcal\{D\}
5:Sample output:

y∼πθ​\(y∣x\)y\\sim\\pi\_\{\\theta\}\(y\\mid x\)
6:Get reward:

r←r⁡\(x,y\)r\\leftarrow r\(x,y\)
7:Store

⟨x,y,r⟩\\langle x,y,r\\ranglein buffer

8:ifTime to trainthen

9:Update model with an RL algorithm of choice to maximize:

𝔼buffer\[1cr\(x,y\)−DKL\(πθ∥πref\)\]\\mathbb\{E\}\_\{\{\\rm buffer\}\}\[\\frac\{1\}\{c\}r\(x,y\)\-D\_\{\\rm KL\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\rm ref\}\)\]
10:endif

11:endfor

12:return

πθ\\pi\_\{\\theta\}

## Appendix FPrompt Distribution Experimental Details

Training Details:For the broad\-distribution condition detailed in Section[4\.3](https://arxiv.org/html/2608.24949#S4.SS3), we use a production\-scale dataset comprising 10,000 prompts sampled from the OLMo 3 RLVR training mix, evenly split between mathematics, instruction following, and code\. We utilize the standard OlmoRL codebase with verifiable rewards replaced by random scalars drawn uniformly from\[0,1\]\[0,1\]\. To ensure fully on\-policy training, async steps are set to 0\. All experiments are distributed across three 8xH100 nodes, sampling 8 responses per prompt\. The narrow\-distribution condition utilizes the identical hardware and algorithmic setup, but is restricted to 100 math\-only prompts fine\-tuned directly from the SFT checkpoint\.

Evaluation Tracking:To precisely measure entropy and unlearning dynamics, we extract 200 evaluation prompts per domain \(100 that appeared during training, and 100 held\-out out\-of\-distribution prompts\)\. Prior to RL training, we sample a fixed response for each prompt from the base model\. Throughout the random reward fine\-tuning process, we evaluate the model every 50 training steps, measuring the log\-likelihood and average per\-token entropy assigned to these fixed baseline responses\.

## Appendix GModel Sizes

![Refer to caption](https://arxiv.org/html/2608.24949v1/images/1_sft_eval_bars_Qwen2.5-7B-Instruct.png)

![Refer to caption](https://arxiv.org/html/2608.24949v1/images/2a_rl_dense_reward_Qwen2.5-7B-Instruct.png)

![Refer to caption](https://arxiv.org/html/2608.24949v1/images/5_rl_eval_bars_Qwen2.5-7B-Instruct.png)

![Refer to caption](https://arxiv.org/html/2608.24949v1/images/2b_rl_sparse_reward_Qwen2.5-7B-Instruct.png)

Figure 5:RL post\-training dynamics and evaluation metrics for Qwen2\.5\-7B\-Instruct\.\(Top Left\)Prior to RL, the SFT\+ model shows a strong initial prior \(28\.8%\), and the Base model exhibits a small but non\-zero starting mass \(3\.5%\)\.\(Bottom Left\)Final RL evaluation reveals that both the SFT\+ and Base models successfully and perfectly acquire the target behavior \(\>98%\>98\\%match rates\) under both dense and sparse rewards, while SFT\- completely fails\.\(Right\)The raw reward curves highlight a delayed but successful optimization for the Base model\. While SFT\+ climbs immediately, the Base model requires roughly 40 steps to escape its initial plateau before rapidly converging to a 1\.0 reward\. The SFT\- model remains strictly suppressed\.![Refer to caption](https://arxiv.org/html/2608.24949v1/images/1_sft_eval_bars_Qwen2-7B.png)

![Refer to caption](https://arxiv.org/html/2608.24949v1/images/2a_rl_dense_reward_Qwen2-7B.png)

![Refer to caption](https://arxiv.org/html/2608.24949v1/images/5_rl_eval_bars_Qwen2-7B.png)

![Refer to caption](https://arxiv.org/html/2608.24949v1/images/2b_rl_sparse_reward_Qwen2-7B.png)

Figure 6:RL post\-training dynamics and evaluation metrics for Qwen2\-7B\.\(Top Left\)Prior to RL, the SFT\+ model shows a strong initial prior \(28\.8%\), and the Base model exhibits a small but non\-zero starting mass \(3\.5%\)\.\(Bottom Left\)Final RL evaluation reveals that both the SFT\+ and Base models successfully and perfectly acquire the target behavior \(\>98%\>98\\%match rates\) under both dense and sparse rewards, while SFT\- completely fails\.\(Right\)The raw reward curves highlight a delayed but successful optimization for the Base model\. While SFT\+ climbs immediately, the Base model requires roughly 40 steps to escape its initial plateau before rapidly converging to a 1\.0 reward\. The SFT\- model remains strictly suppressed\.![Refer to caption](https://arxiv.org/html/2608.24949v1/images/1_sft_eval_bars_Qwen2-1.5B.png)

![Refer to caption](https://arxiv.org/html/2608.24949v1/images/2a_rl_dense_reward_Qwen2-1.5B.png)

![Refer to caption](https://arxiv.org/html/2608.24949v1/images/5_rl_eval_bars_Qwen2-1.5B.png)

![Refer to caption](https://arxiv.org/html/2608.24949v1/images/2b_rl_sparse_reward_Qwen2-1.5B.png)

Figure 7:RL post\-training dynamics and evaluation metrics for Qwen2\-1\.5B\.\(Top Left\)Initial substring match performance prior to RL shows the SFT\+ model possesses only marginal probability mass for the target sequence \(8\.0%\), while Base \(0\.3%\) and SFT\- \(0\.0%\) exhibit near\-zero coverage\.\(Bottom Left\)Final RL evaluation demonstrates that only the SFT\+ model successfully acquires the target behavior \(achieving\>90%\>90\\%match rates under both reward regimes\)\. The Base model gains a slight benefit from dense rewards \(7\.0%\) but largely fails, alongside SFT\-\.\(Right\)Raw reward curves confirm these dynamics: the SFT\+ model successfully optimizes both signals, while the Base and SFT\- variants flatline, failing to escape their initial exploration plateaus\.![Refer to caption](https://arxiv.org/html/2608.24949v1/images/1_sft_eval_bars_Qwen3-1.7B-Base.png)

![Refer to caption](https://arxiv.org/html/2608.24949v1/images/2a_rl_dense_reward_Qwen3-1.7B-Base.png)

![Refer to caption](https://arxiv.org/html/2608.24949v1/images/5_rl_eval_bars_Qwen3-1.7B-Base.png)

![Refer to caption](https://arxiv.org/html/2608.24949v1/images/2b_rl_sparse_reward_Qwen3-1.7B-Base.png)

Figure 8:RL post\-training dynamics and evaluation metrics for Qwen3\-1\.7B\.\(Top Left\)Initial coverage is limited, with SFT\+ at 14\.0% and Base/SFT\- near zero\.\(Bottom Left\)Final evaluation demonstrates the specific power of dense rewards\. While the Base model only achieves a 10\.0% match rate under sparse rewards, the granular gradient of the dense reward allows it to reach 48\.8%\. The SFT\+ model succeeds broadly across both signals\.\(Right\)The reward curves explicitly show this dynamic: under dense rewards, the Base model successfully begins climbing \(reaching∼\\sim0\.5\), whereas under sparse rewards, it remains completely flat at 0\. SFT\- fails entirely across all regimes\.![Refer to caption](https://arxiv.org/html/2608.24949v1/images/1_sft_eval_bars_Qwen3-8B.png)

![Refer to caption](https://arxiv.org/html/2608.24949v1/images/2a_rl_dense_reward_Qwen3-8B.png)

![Refer to caption](https://arxiv.org/html/2608.24949v1/images/5_rl_eval_bars_Qwen3-8B.png)

![Refer to caption](https://arxiv.org/html/2608.24949v1/images/2b_rl_sparse_reward_Qwen3-8B.png)

Figure 9:RL post\-training dynamics and evaluation metrics for Qwen3\-8B\.\(Top Left\)The SFT\+ model starts with a highly accessible prior \(42\.8%\), while the Base and SFT\- models start with exactly 0\.0% probability mass\.\(Bottom Left\)Final evaluation shows a stark binary outcome: the SFT\+ model perfectly converges to a 100\.0% match rate under both sparse and dense rewards, while the Base and SFT\- models achieve absolute 0%\.\(Right\)The raw reward curves demonstrate rapid, smooth convergence for SFT\+ under both regimes\. In contrast, the Base and SFT\- models remain completely flat, unable to generate the target sequence a single time during exploration, confirming the strict bottleneck imposed by a 0% base prior\.