Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models

# Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models
Source: [https://arxiv.org/html/2607.02869](https://arxiv.org/html/2607.02869)
Anagha Radhakrishna Palandye New York University ap8884@nyu\.edu &Rebecca Glick New York University rmg9724@nyu\.edu &Osheen Kaul New York University ok684@nyu\.edu

###### Abstract

Reinforcement Learning with Verifiable Rewards \(RLVR\) has emerged as a promising paradigm for improving mathematical reasoning in language models\. Yet most RLVR work rewards only the final answer \(outcome\-based rewards\), leaving the impact of step\-level process supervision \(process rewards\) underexplored especially for small models that lack the capacity to self\-correct under sparse feedback\. We systematically compare five reward conditions applied to Qwen2\.5\-0\.5B fine\-tuned with Group Relative Policy Optimization \(GRPO\) on GSM8K: a no\-RL baseline,process\-only,outcome\-only, and three hybrid weightings \(λ∈\{0\.9,0\.5,0\.1\}\\lambda\\in\\\{0\.9,0\.5,0\.1\\\}process weight\)\. Process\-only supervision achieves 63\.73% test accuracy versus 53\.75% for outcome\-only, a nearly 10\-percentage point gap while yielding reasoning traces with higher step validity and lower deviation from ground\-truth chain length\. Hybrid rewards generally correlate positively with process weight, with one notable anomaly: the low\-process / high\-outcome configuration \(λ=0\.1\\lambda\{=\}0\.1\)*underperforms*pure outcome supervision, suggesting conflicting optimization signals\. Error analysis using GPT\-4o as a judge reveals distinct failure mode distributions: process models generate structurally inconsistent but arithmetically grounded traces, while outcome models produce concise but derivation\-error\-prone chains\. Our results demonstrate that reward granularity is a first\-order design decision for RLVR, with process\-level supervision substantially improving both accuracy and trace fidelity in small language models\.

Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models

Anagha Radhakrishna PalandyeNew York Universityap8884@nyu\.eduRebecca GlickNew York Universityrmg9724@nyu\.eduOsheen KaulNew York Universityok684@nyu\.edu

## 1Introduction

Improving the*reasoning ability*of language models remains an open challenge\. Even when models are prompted with Chain\-of\-Thought \(CoT\) instructions, an approach demonstrated byWeiet al\.\([2023](https://arxiv.org/html/2607.02869#bib.bib2)\)to improve transparency and interpretability, small LLMs frequently generate incorrect, inconsistent, or logically incoherent intermediate steps, even when reaching the correct final answer\. This recurring mismatch highlights a broader question in modern LLM research:

> *Are improvements in mathematical and symbolic reasoning driven by genuine gains in reasoning quality, or are models simply better at sampling plausible answers?*

A growing body of work attempts to address this through Reinforcement Learning with Verifiable Rewards \(RLVR\), where models receive reward signals derived from*mechanical correctness checks*rather than human preferences\. RLVR has proved especially promising in domains like math where solutions can be validated automatically\(Wenet al\.,[2025](https://arxiv.org/html/2607.02869#bib.bib3); Tanget al\.,[2025](https://arxiv.org/html/2607.02869#bib.bib4)\)\. The GSM8K dataset, with its stepwise arithmetic reasoning structure, is particularly suitable for evaluating these kinds of reward functions\.

A central but under\-explored dimension of RLVR is reward granularity\. Rewards can be given at the process level, where every intermediate step is evaluated and rewarded, or at the outcome level, where only the final predicted answer receives a reward\. We refer to these as process rewards \(process\-based feedback\) and outcome rewards \(outcome\-based feedback\)\. While outcome\-based rewards have been extensively studied in RLHF and RLVR contexts, less is concretely understood about how process, step\-level rewards shift a model’s reasoning behavior, or how the two types interact when combined\. This gap is especially important for small models that lack the ability to internally mitigate noisy or insufficient supervision\.

#### Research question

*Do process rewards improve the coherence and validity of reasoning traces, and does combining process \+ outcome rewards yield more stable and accurate reasoning than either alone?*

#### Contributions

We make the following contributions\. First, we provide a controlled comparison of five reward conditions for RLVR fine\-tuning of Qwen2\.5\-0\.5B on GSM8K using GRPO, covering process\-only, outcome\-only, and three hybrid weightings\. Second, we evaluate beyond final\-answer accuracy by assessing trace validity, chain\-length deviation, and process step ratio on a stratified 45\-question evaluation slice\. Third, we conduct an LLM\-as\-a\-judge error analysis \(GPT\-4o\) characterizing the failure mode distributions induced by each reward regime\. Finally, we identify a hybrid reward anomaly: the low\-process / high\-outcome configuration \(λ=0\.1\\lambda\{=\}0\.1\) underperforms pure outcome supervision, indicating conflicting rather than additive optimization dynamics\.

## 2Background and Prior Work

### 2\.1Chain\-of\-Thought Reasoning

Advances in LLM reasoning are moving beyond supervised fine\-tuning and RLHF\-style preference models toward verifiable reward frameworks, where a model is rewarded when its output is checked by an external tool\(Wenet al\.,[2025](https://arxiv.org/html/2607.02869#bib.bib3)\)\.Wenet al\.\([2025](https://arxiv.org/html/2607.02869#bib.bib3)\)show that purely final\-answer rewards can still influence reasoning traces, introducing CoT\-Pass@K, which evaluates whether any ofKKsampled chains of thought produce the correct answer*and*maintain a valid reasoning trajectory, thereby jointly measuring answer accuracy and trace validity\. Building on this verifiable framework,Tanget al\.\([2025](https://arxiv.org/html/2607.02869#bib.bib4)\)demonstrate how such verifiable rewards can be scaled using Jensen\-style lower bounds, replacing the intractable expectation over many sampled solutions with a tractable proxy objective that lower\-bounds the true expected reward\.

These developments intersect with a broader literature on CoT prompting\. CoT prompting\(Weiet al\.,[2023](https://arxiv.org/html/2607.02869#bib.bib2)\)encourages models to decompose problems into explicit intermediate steps\. Although this leads to better interpretability and often better accuracy, later work has shown that CoT outputs can be misleading or structurally unsound\(Turpinet al\.,[2023](https://arxiv.org/html/2607.02869#bib.bib11); Saparov and He,[2023](https://arxiv.org/html/2607.02869#bib.bib12)\)\. Small LLMs in particular tend to generate hallucinated steps or irrelevant justifications, commit arithmetic errors early and carry them through, and jump between steps without maintaining logical consistency\. This has motivated a shift toward evaluating reasoning quality independently of answer correctness, using dedicated reasoning metrics or LLM\-as\-a\-judge assessments of trace integrity\.

### 2\.2Process vs\. Outcome Supervision

Most reinforcement\-learning approaches for reasoning tasks, including RLHF, rely solely on outcome\-level rewards\. While this improves final\-answer accuracy\(Ziegleret al\.,[2020](https://arxiv.org/html/2607.02869#bib.bib10)\), it provides no guidance about the internal reasoning process\. Models can learn to exploit shortcuts such as memorized patterns, spurious correlations, and surface\-level cues unrelated to correct reasoning\. By contrast, process\-based supervision, explored in early form byLightmanet al\.\([2023](https://arxiv.org/html/2607.02869#bib.bib6)\)and expanded in more recent RLVR work, offers dense feedback at each stage of the reasoning chain\. This encourages explicit incremental computation, more faithful stepwise reasoning, and a stronger grounding in arithmetic operations\. However, process\-based feedback can also amplify instability, producing overly long or unnecessary reasoning chains if not paired with global constraints\. This highlights an important theme also present in RLVR research: the granularity of supervision shapes the coherence of model reasoning, with each option carrying distinct tradeoffs in stability and reasoning fidelity\.

### 2\.3Reinforcement Learning with Verifiable Rewards \(RLVR\)

RLVR differs from RLHF by allowing exact, programmatically computed rewards rather than relying on human judgment\. In mathematical domains, these rewards can directly verify solution quality by checking numeric equality between predicted and gold solutions, evaluating intermediate arithmetic, and penalizing invalid formats or logical inconsistencies\.

Weiet al\.\([2023](https://arxiv.org/html/2607.02869#bib.bib2)\)show that prompting for step\-by\-step rationales improves model performance, whileUesatoet al\.\([2022](https://arxiv.org/html/2607.02869#bib.bib5)\)compare outcome\-based and process\-based supervision for math word problems, finding that outcome\-only training is label\-efficient but often produces “flaky” reasoning, whereas process\-level supervision substantially reduces trace errors\.

Recent RLVR studies report improvements in both accuracy and reasoning structure\(Wenet al\.,[2025](https://arxiv.org/html/2607.02869#bib.bib3); Tanget al\.,[2025](https://arxiv.org/html/2607.02869#bib.bib4)\), yet they almost exclusively reward the final answer\. Consequently, the impact of introducing process, step\-level rewards remains an open question\.Samineniet al\.\([2025](https://arxiv.org/html/2607.02869#bib.bib7)\)take a step toward this direction by analyzing whether RLVR\-trained Qwen2\.5\-0\.5B models exhibit local coherence or merely achieve globally correct solutions on GSM8K, but their evaluation still centers on final\-answer rewards and does not vary reward granularity\. Our work directly addresses this gap\. Using the RLVR setup with Qwen2\.5\-0\.5B on GSM8K employed bySamineniet al\.\([2025](https://arxiv.org/html/2607.02869#bib.bib7)\), and drawing on insights fromUesatoet al\.\([2022](https://arxiv.org/html/2607.02869#bib.bib5)\), we systematically vary reward granularity\. We compare outcome\-only, process\-only, and variations of combined process \+ outcome reward regimes to study how verifiable reward design influences coherence and validity of generated reasoning traces in addition to final\-answer accuracy\.

## 3Dataset and Evaluation Strategy

We use GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.02869#bib.bib1)\), a widely adopted benchmark containing 8,792 grade\-school math word problems\. These tasks are designed to require stepwise arithmetic reasoning, making them ideal for evaluating changes in CoT behavior\.

Because reasoning characteristics differ across levels of problem difficulty, we curated a balanced 45\-question evaluation slice drawn from the 1,319\-question test set: 15 easy \(2–3 steps\), 15 medium \(4–5 steps\), and 15 hard \(6–11 steps\)\. Difficulty is defined by the ground\-truth number of steps in the solution, which provides a more objective stratification than surface\-level heuristics\. This balanced setup enables us to analyze whether reward structures affect reasoning differently across problem tiers, as summarized in Table[1](https://arxiv.org/html/2607.02869#S3.T1)\.

Steps in Ground TruthCount of GSM8K Questions232633704298517468874082092111Total1,319Table 1:Test set distribution by ground\-truth solution step count\.
## 4Model Setup and Reward Structures

### 4\.1Base Model and Baseline

All experiments use Qwen2\.5\-0\.5B\(Yanget al\.,[2024](https://arxiv.org/html/2607.02869#bib.bib9)\), a compact transformer model whose limited capacity makes reasoning failures easy to observe\. As a baseline, we evaluate the pre\-trained model in a direct\-answer \(no CoT\) setting, which relies on pattern\-based inference without explicit reasoning, achieving 33\.13% on the test set \(437/1,319\)\. The baseline reflects patterns observed in prior findings that CoT generation, without proper supervision, does*not*guarantee coherent reasoning\(Saparov and He,[2023](https://arxiv.org/html/2607.02869#bib.bib12)\)\.

### 4\.2Training Setup and Implementation

All RLVR experiments were conducted using Group Relative Policy Optimization \(GRPO\)\(Shaoet al\.,[2024](https://arxiv.org/html/2607.02869#bib.bib8)\), a variant of Proximal Policy Optimization \(PPO\) designed for language model fine\-tuning\. We utilized the VERL \(Volcano Engine Reinforcement Learning\) library, which provides efficient implementations of RL algorithms for large\-scale language model training\. Hyperparameters are held constant across all reward conditions: batch size 64, mini\-batch size 8, learning rate1×10−61\\times 10^\{\-6\}, KL coefficient 0\.001, 5 epochs \(580 steps\), temperature 0\.6, and 5 rollouts per prompt using vLLM\. Maximum sequence lengths are 512 tokens \(prompt\) and 1,024 tokens \(response\)\. All runs use a single A100 80GB GPU\. Following VERL’s evaluation protocol, test set accuracy is reported asval\-core/math/reward/mean@1, evaluated on the GSM8K test split after each training epoch\. Full hyperparameter details are provided in Appendix Table[4](https://arxiv.org/html/2607.02869#A1.T4)\.

### 4\.3Reward Structures

Figure[1](https://arxiv.org/html/2607.02869#S4.F1)illustrates the end\-to\-end training pipeline\. We evaluate three forms of reward supervision\.

![Refer to caption](https://arxiv.org/html/2607.02869v1/figures/RLVR_Pipeline.png)Figure 1:Overview of the RLVR training pipeline\. The base model Qwen2\.5\-0\.5B generates rollouts via vLLM, which are evaluated by the reward function \(process, outcome, or hybrid\) and used to update the policy via GRPO within the VERL framework\.#### Process rewards \(process supervision\)

Process rewards evaluate each intermediate step of the CoT\. In our implementation, we extract the ground\-truth reasoning steps from GSM8K solutions which include step\-by\-step calculations and compare the model’s generated steps against these references\. A step receives a reward of 1 if it matches the expected intermediate calculation \(within numerical tolerance of1×10−51\\times 10^\{\-5\}\), and 0 otherwise\. The total process reward is the proportion of correct steps:

Rprocess=correct stepstotal stepsR\_\{\\text\{process\}\}=\\frac\{\\text\{correct steps\}\}\{\\text\{total steps\}\}\(1\)To discourage excessive verbosity, we apply a penalty when the generated chain exceeds1\.5×1\.5\\timesthe ground\-truth step count\. This step\-level supervision encourages the model to break computations into finer\-grained components, maintain stepwise numeric correctness, output interpretable reasoning that mirrors ground\-truth solutions, and explicitly show variable definitions and intermediate calculations\. However, because each correct step is independently rewarded, the model can become incentivized to overproduce steps or to explore unnecessarily detailed paths, aligning with concerns raised byLightmanet al\.\([2023](https://arxiv.org/html/2607.02869#bib.bib6)\)regarding stepwise supervision in RL settings\.

#### Outcome rewards \(outcome supervision\)

Outcome rewards evaluate only the final numerical answer\. We extract the final answer from the model’s response \(supporting formats like\\boxed\{answer\},\#\#\#\# answer, and natural language patterns\) and compare it to the ground\-truth solution\. The reward is binary:

Routcome=\{1if​\|a^−a∗\|<ϵ0otherwise,ϵ=10−5R\_\{\\text\{outcome\}\}=\\begin\{cases\}1&\\text\{if \}\|\\hat\{a\}\-a^\{\*\}\|<\\epsilon\\\\ 0&\\text\{otherwise\}\\end\{cases\},\\quad\\epsilon=10^\{\-5\}\(2\)wherea^\\hat\{a\}is the model’s predicted answer anda∗a^\{\*\}is the ground\-truth answer\.

This outcome\-level supervision pushes the model to reach correct outcomes efficiently, skip intermediate noise when possible, rely on internal shortcuts or pattern\-matching behavior, and optimize for answer correctness regardless of reasoning path\. Outcome\-only rewards can improve overall accuracy, but they often leave the reasoning chain underdetermined, brittle, or logically inconsistent – an issue documented inWenet al\.\([2025](https://arxiv.org/html/2607.02869#bib.bib3)\)\.

#### Hybrid rewards

For hybrid reward configurations, we implemented weighted combinations of process and outcome rewards:

R=λ⋅Rprocess\+\(1−λ\)⋅RoutcomeR=\\lambda\\cdot R\_\{\\text\{process\}\}\+\(1\-\\lambda\)\\cdot R\_\{\\text\{outcome\}\}\(3\)whereλ\\lambdais the process reward weight\. We evaluate three mixtures:*0\.9 process \+ 0\.1 outcome*\(heavily process\-focused with light outcome guidance\),*0\.5 process \+ 0\.5 outcome*\(balanced supervision of both process and outcome\), and*0\.1 process \+ 0\.9 outcome*\(primarily outcome\-focused with minimal process supervision\)\. The hypothesis, followingTanget al\.\([2025](https://arxiv.org/html/2607.02869#bib.bib4)\), is that hybrid signals can encourage coherent intermediate reasoning without sacrificing final\-answer correctness\.

## 5Quantitative Results

### 5\.1Final\-Answer Accuracy

Process rewards achieved the largest gains in accuracy, outperforming outcome\-only supervision and significantly improving over both baselines, as shown in Table[2](https://arxiv.org/html/2607.02869#S5.T2), supporting the view that dense reward signals provide more informative guidance for small models than sparse outcome\-level feedback\.

Table 2:Test set accuracy and improvement over baseline across all reward regimes on GSM8K\. Process\-only achieves the highest accuracy \(63\.73%\), with Process \(0\.9\) \+ Outcome \(0\.1\) as the best hybrid configuration \(61\.10%\)\.Results are reported using final\-answer accuracy on the full 1,319\-question test set\. All hybrid weightings improve over baseline, though none match process\-only’s peak performance\.

### 5\.2Reasoning Fidelity and Stability

Beyond final\-answer accuracy, we analyze how reasoning differs between regimes on the 45\-question evaluation slice using four metrics\.

#### CoT\-Pass@1

This measures whether the model’s single sampled reasoning trace leads to a correct final answer, capturing end\-to\-end reasoning success without sampling multiple trajectories\.

#### Trace Validity

This assesses whether the reasoning chain is logically well\-formed and free of structural inconsistencies such as contradictions, invalid transformations, or unsupported jumps – a trace can be invalid even if the final answer is correct\.

#### Chain Length Deviation

This quantifies how much the model’s reasoning length differs from the ground\-truth number of steps, where large positive deviations indicate unnecessary or hallucinated steps and negative deviations signal skipped or missing reasoning\.

#### Process Step Ratio

This computes the proportion of intermediate steps that receive a positive process reward under the RLVR framework, serving as a proxy for how often the model produces steps that align with ground\-truth intermediate reasoning\.

Table[3](https://arxiv.org/html/2607.02869#S5.T3)summarizes these metrics across four conditions\. On the full test set, Process Only achieves the highest CoT\-Pass@1 \(0\.64\), substantially outperforming the base model \(0\.33\) and confirming that process reward training provides meaningful gains in end\-to\-end reasoning success\.

Table 3:Reasoning fidelity metrics for four conditions\. \*CoT\-Pass@1 is reported on the full 1,319\-question test set; Trace Validity, Chain Length Deviation, and Process Step Ratio are computed on the 45\-question stratified evaluation slice\.Bold= best result among RLVR\-trained conditions\. Higher is better for CoT\-Pass@1, Trace Validity, and Process Step Ratio; lower is better for Chain Length Deviation\.The Process \(0\.9\) \+ Outcome \(0\.1\) configuration achieves the highest Trace Validity \(0\.60\) and Process Step Ratio \(0\.84\)\. Process Only exhibits the largest Chain Length Deviation \(6\.49\), reflecting its tendency toward verbose reasoning chains\. Outcome\-only produces more concise traces \(deviation 3\.64\) but scores lower on step alignment\.

### 5\.3Qualitative Behavior Analysis

Qualitative examination of model outputs highlights the behavioral signature of each reward regime\. Process\-only models typically produce detailed arithmetic decompositions\. They often solve problems correctly, but their CoT traces can become overly verbose, sometimes duplicating steps or drifting off\-path before returning to the correct computation\. Outcome\-only models tend to “jump” more aggressively between reasoning points, producing short chains that may omit key logical transitions\. These models occasionally reach correct answers through opportunistic reasoning but lack consistency and robustness\. Hybrid models, especially the Process \(0\.9\) \+ Outcome \(0\.1\) configuration, exhibit the most human\-like reasoning flow\. Their chains are neither overexpanded nor underexplained, and they maintain continuity between steps while avoiding unnecessary detours\.

Figure[2](https://arxiv.org/html/2607.02869#S5.F2)illustrates the reasoning traces of our four regimes applied to the same geometry word problem\. The*Base*and*Outcome*regimes incorrectly calculate the area \(300 sq ft\), while the*Process*and*Process\+Outcome*regimes correctly identify the need for perimeter \(70 ft\)\. The process regime mitigates reasoning errors by forcing explicit variable definition \(length/width\) prior to calculation, whereas the base regime rushes to a formulaic application \(L×WL\\times W\)\.

![Refer to caption](https://arxiv.org/html/2607.02869v1/figures/LLM_Reasoning_Traces.png)Figure 2:Example reasoning chains across four reward regimes for an easy GSM8K geometry problem, illustrating the qualitative differences in reasoning structure across the four regimes\.
### 5\.4LLM\-as\-a\-Judge Error Analysis

We use GPT\-4o to annotate common reasoning error types: incorrect derivation, calculator error, missing step, contradiction, formatting issues, or no error\. The error distribution reveals clear distinctions\. Process\-only models frequently introduce structural inconsistencies and contradictions, likely due to extended chain length, suggesting that token\-level rewards can encourage locally plausible but globally inconsistent reasoning\. Outcome\-only models show derivation and arithmetic errors, reflecting weak intermediate grounding and a tendency to optimize for outcome\-level correctness at the expense of intermediate steps\. The base model generates relatively coherent traces but still accumulates rule\- and derivation\-related errors\. Hybrid models exhibit fewer catastrophic errors and the most even error distribution\. Taken together, these patterns illustrate how different reward schemes meaningfully shape the reasoning behaviors of Qwen2\.5\-0\.5B, as shown in Figure[3](https://arxiv.org/html/2607.02869#S5.F3), corroborating claims in RLVR literature that reward structure strongly shapes not just accuracy but the*failure modes*of model reasoning\.

![Refer to caption](https://arxiv.org/html/2607.02869v1/figures/llm_judge_error_distribution.png)Figure 3:Error type distribution across reward regimes annotated by GPT\-4o on the 45\-question evaluation slice, broken down by error type\.

## 6Discussion

Integrating our analyses, a clear picture emerges\. Process rewards strengthen stepwise correctness and improve accuracy but can destabilize reasoning trajectories\. Outcome rewards focus the model on final outcomes but leave reasoning opaque and error\-prone\. Hybrid rewards effectively blend both signals, producing coherent reasoning with strong outcome performance\.

## 7Limitations

Several limitations qualify our findings\. All reported results are from single training runs without multiple seeds, and we do not report variance estimates or confidence intervals\. The observed accuracy differences, including the approximately 10\-point gap between process\-only and outcome\-only supervision, should therefore be interpreted as indicative rather than statistically confirmed\. Step\-level verification is easier in math than in open\-ended domains\. Qwen2\.5\-0\.5B’s small size makes it sensitive to reward noise\. Our 45\-problem evaluation slice, though balanced, remains relatively small; further, LLM\-as\-a\-Judge findings must be taken with a grain of salt as they may not reflect human evaluation\.

## 8Future Work

Promising extensions include applying process/outcome reward structures to harder datasets such as MATH, SVAMP, and AQuA\-RAT; studying multi\-hop reasoning via datasets like HotpotQA, where verifying intermediate reasoning is more complex; experimenting with partial\-credit reward schemes or learned verifiers to provide richer supervisory signals; and scaling to larger Qwen or Llama models to assess whether hybrid rewards provide increasing returns with model capacity\. Beyond accuracy, future work should also investigate whether reward granularity shapes reasoning behavior differently across problem domains, model architectures, and training regimes, with the goal of establishing more generalizable design principles for process supervision in RLVR\.

## 9Conclusion

This work demonstrates that reward granularity \(process vs\. outcome rewards\) fundamentally shapes the structure, coherence, and correctness of CoT reasoning in small language models\. To develop models that not only answer correctly, but also reason reliably, it is essential to supervise both the process and the outcome of reasoning\. The nearly 10\-point accuracy gap between process\-only and outcome\-only supervision underscores that reward granularity is not a minor implementation detail, but a primary lever for reasoning quality in small models\. While our results are specific to GSM8K and a 0\.5B\-parameter model, they motivate broader investigation into reward granularity as models and tasks scale\. Code and evaluation data will be released upon publication\.

## Acknowledgments

Training was conducted on Lightning\.ai A100 GPU instances\. We thank the VERL and vLLM open\-source communities\.

## References

- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.External Links:2110\.14168,[Link](https://arxiv.org/abs/2110.14168)Cited by:[§3](https://arxiv.org/html/2607.02869#S3.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2023\)Let’s verify step by step\.External Links:2305\.20050,[Link](https://arxiv.org/abs/2305.20050)Cited by:[§2\.2](https://arxiv.org/html/2607.02869#S2.SS2.p1.1),[§4\.3](https://arxiv.org/html/2607.02869#S4.SS3.SSS0.Px1.p1.2)\.
- Local coherence or global validity? investigating rlvr traces in math domains\.External Links:2510\.18176,[Link](https://arxiv.org/abs/2510.18176)Cited by:[§2\.3](https://arxiv.org/html/2607.02869#S2.SS3.p3.1)\.
- A\. Saparov and H\. He \(2023\)Language models are greedy reasoners: a systematic formal analysis of chain\-of\-thought\.External Links:2210\.01240,[Link](https://arxiv.org/abs/2210.01240)Cited by:[§2\.1](https://arxiv.org/html/2607.02869#S2.SS1.p2.1),[§4\.1](https://arxiv.org/html/2607.02869#S4.SS1.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.External Links:2402\.03300,[Link](https://arxiv.org/abs/2402.03300)Cited by:[§4\.2](https://arxiv.org/html/2607.02869#S4.SS2.p1.1)\.
- Y\. Tang, S\. Wang, L\. Madaan, and R\. Munos \(2025\)Beyond verifiable rewards: scaling reinforcement learning for language models to unverifiable data\.External Links:2503\.19618,[Link](https://arxiv.org/abs/2503.19618)Cited by:[§1](https://arxiv.org/html/2607.02869#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.02869#S2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2607.02869#S2.SS3.p3.1),[§4\.3](https://arxiv.org/html/2607.02869#S4.SS3.SSS0.Px3.p1.1)\.
- M\. Turpin, J\. Michael, E\. Perez, and S\. R\. Bowman \(2023\)Language models don’t always say what they think: unfaithful explanations in chain\-of\-thought prompting\.External Links:2305\.04388,[Link](https://arxiv.org/abs/2305.04388)Cited by:[§2\.1](https://arxiv.org/html/2607.02869#S2.SS1.p2.1)\.
- J\. Uesato, N\. Kushman, R\. Kumar, F\. Song, N\. Siegel, L\. Wang, A\. Creswell, G\. Irving, and I\. Higgins \(2022\)Solving math word problems with process\- and outcome\-based feedback\.External Links:2211\.14275,[Link](https://arxiv.org/abs/2211.14275)Cited by:[§2\.3](https://arxiv.org/html/2607.02869#S2.SS3.p2.1),[§2\.3](https://arxiv.org/html/2607.02869#S2.SS3.p3.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. Le, and D\. Zhou \(2023\)Chain\-of\-thought prompting elicits reasoning in large language models\.External Links:2201\.11903,[Link](https://arxiv.org/abs/2201.11903)Cited by:[§1](https://arxiv.org/html/2607.02869#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.02869#S2.SS1.p2.1),[§2\.3](https://arxiv.org/html/2607.02869#S2.SS3.p2.1)\.
- X\. Wen, Z\. Liu, S\. Zheng, S\. Ye, Z\. Wu, Y\. Wang, Z\. Xu, X\. Liang, J\. Li, Z\. Miao, J\. Bian, and M\. Yang \(2025\)Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms\.External Links:2506\.14245,[Link](https://arxiv.org/abs/2506.14245)Cited by:[§1](https://arxiv.org/html/2607.02869#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.02869#S2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2607.02869#S2.SS3.p3.1),[§4\.3](https://arxiv.org/html/2607.02869#S4.SS3.SSS0.Px2.p2.1)\.
- A\. Yang, B\. Yang, B\. Hui, B\. Zheng, B\. Yu, C\. Zhou, C\. Li, C\. Li, D\. Liu, F\. Huang, G\. Dong, H\. Wei, H\. Lin, J\. Tang, J\. Wang, J\. Yang, J\. Tu, J\. Zhang, J\. Ma, J\. Yang, J\. Xu, J\. Zhou, J\. Bai, J\. He, J\. Lin, K\. Dang, K\. Lu, K\. Chen, K\. Yang, M\. Li, M\. Xue, N\. Ni, P\. Zhang, P\. Wang, R\. Peng, R\. Men, R\. Gao, R\. Lin, S\. Wang, S\. Bai, S\. Tan, T\. Zhu, T\. Li, T\. Liu, W\. Ge, X\. Deng, X\. Zhou, X\. Ren, X\. Zhang, X\. Wei, X\. Ren, X\. Liu, Y\. Fan, Y\. Yao, Y\. Zhang, Y\. Wan, Y\. Chu, Y\. Liu, Z\. Cui, Z\. Zhang, Z\. Guo, and Z\. Fan \(2024\)Qwen2 technical report\.External Links:2407\.10671,[Link](https://arxiv.org/abs/2407.10671)Cited by:[§4\.1](https://arxiv.org/html/2607.02869#S4.SS1.p1.1)\.
- D\. M\. Ziegler, N\. Stiennon, J\. Wu, T\. B\. Brown, A\. Radford, D\. Amodei, P\. Christiano, and G\. Irving \(2020\)Fine\-tuning language models from human preferences\.External Links:1909\.08593,[Link](https://arxiv.org/abs/1909.08593)Cited by:[§2\.2](https://arxiv.org/html/2607.02869#S2.SS2.p1.1)\.

## Appendix AAdditional Figures

![Refer to caption](https://arxiv.org/html/2607.02869v1/figures/training_curves_upd.png)Figure 4:Training curves for all five reward conditions on GSM8K\. Process\-only converges to the highest validation accuracy \(63\.73%\), while Process \(0\.1\) \+ Outcome \(0\.9\) consistently underperforms all other conditions including pure outcome supervision, illustrating the hybrid reward anomaly identified in Section[5\.1](https://arxiv.org/html/2607.02869#S5.SS1)\.Table 4:GRPO training hyperparameters on GSM8K\.