Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards

# Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards
Source: [https://arxiv.org/html/2605.14539](https://arxiv.org/html/2605.14539)
Mengjie Ren1,2,Jie Lou3,Boxi Cao1,Xueru Wen1,2,Hongyu Lin1,Xianpei Han1,Le Sun1,Xing Yu3,Yaojie Lu1

1Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences 3Xiaohongshu Inc

###### Abstract

Reinforcement Learning with Verifiable Rewards \(RLVR\) has emerged as an effective paradigm for improving the reasoning capabilities of large language models\. However, RLVR training is often hindered by sparse binary rewards and weak credit assignment, resulting in ambiguous optimization signals and underutilization of the useful information embedded in failed trajectories\. To address this challenge, we proposeCorrection\-OrIentedPolicyOptimization\(CIPO\), a simple and effective extension to RLVR that converts on\-policy failed trajectories into correction\-oriented supervision, without relying on any external signals\. By jointly optimizing correction samples derived from the model’s own failed attempts together with the standard RLVR objective, CIPO improves learning effectiveness while explicitly enhancing the model’s ability to correct its own errors\. Extensive experiments across 11 benchmarks spanning mathematical reasoning and code generation demonstrate that CIPO consistently and significantly outperforms strong baselines in both reasoning and correction performance\. Moreover, CIPO yields stronger pass@K gains, indicating that it improves the model’s intrinsic reasoning capacity rather than merely redistributing probability mass over existing correct answers\.

## 1Introduction

Reinforcement Learning with Verifiable Rewards \(RLVR\) has emerged as a core paradigm for enhancing the reasoning capabilities of large language models \(LLMs\), with notable success in mathematical reasoning and code generation\(OpenAIet al\.,[2024](https://arxiv.org/html/2605.14539#bib.bib61); Guoet al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib58); Teamet al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib60)\)\. By leveraging automatically verifiable reward signals from on\-policy rollouts, RLVR enables scalable training without requiring additional human annotations\.

![Refer to caption](https://arxiv.org/html/2605.14539v1/figs/rhead_2.png)Figure 1:Comparison of how standard RLVR and CIPO exploit failed trajectories\. CIPO provides more directional and informative learning signals\.Despite the success, existing RLVR algorithms such as Group Relative Policy Optimization \(GRPO\)\(Shaoet al\.,[2024](https://arxiv.org/html/2605.14539#bib.bib54)\)are fundamentally built upon a reinforce–suppress paradigm, where successful trajectories are reinforced while failed ones are uniformly penalized, regardless of their logical proximity to the ground truth\(Hübotteret al\.,[2026](https://arxiv.org/html/2605.14539#bib.bib9)\)\. Due to the binary and sparse nature of verifiable rewards, training signals often provide ambiguous optimization guidance and fail to capture the heterogeneous nature of failures, particularly in long\-horizon reasoning\. As illustrated in Figure[1](https://arxiv.org/html/2605.14539#S1.F1)\(a\), failed rollouts may arise from fundamentally different error modes, ranging from critical logical flaws and intermediate inconsistencies to minor final\-step miscalculations\. By treating all failures as identical negative signals, existing approaches merely suppress the likelihood of entire trajectories, without offering explicit guidance on how specific errors can be corrected\(Yueet al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib53)\)\. Moreover, failed trajectories often contain partially correct reasoning steps that constitute valuable learning signals\. Discarding such intermediate structures not only wastes useful supervision but may also hinder effective exploration, ultimately leading to suboptimal generalization\(Huet al\.,[2026](https://arxiv.org/html/2605.14539#bib.bib25); Yueet al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib53); Hübotteret al\.,[2026](https://arxiv.org/html/2605.14539#bib.bib9)\)\.

Previous studies have sought to address these challenges through the integration of additional process reward models\(Cuiet al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib30); Wanget al\.,[2024](https://arxiv.org/html/2605.14539#bib.bib29)\)or LLM\-based critics\(Xieet al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib28)\)\. Nevertheless, these methods are often hampered by the costs of additional manual labeling and computation resources, while the limited capacity of the auxiliary models can introduce noise and undermine generalizability\(Wenet al\.,[2024](https://arxiv.org/html/2605.14539#bib.bib27); Gaoet al\.,[2023](https://arxiv.org/html/2605.14539#bib.bib26)\)\. More recently, such as SDPO\(Hübotteret al\.,[2026](https://arxiv.org/html/2605.14539#bib.bib9)\), leverages environmental feedback or self\-generated trajectories to construct a feedback\-conditioned teacher and derive fine\-grained supervision from distributional discrepancies\. However, these methods rely on reliable feedback signals and reflective capabilities that are often limited in weaker models\. Moreover, its generalization has been criticized for suppressing epistemic uncertainty, thereby undermining robust reasoning\(Kimet al\.,[2026](https://arxiv.org/html/2605.14539#bib.bib7)\)\. Consequently, there is an urgent need for a task\-agnostic solution that addresses these challenges without requiring additional external supervision signals\.

To this end, we proposeCorrection\-OrIentedPolicyOptimization\(CIPO\), a systematic extension within the RLVR paradigm without requiring any external information\. The core idea of CIPO is to transform on\-policy failed trajectories from mere objects of penalty into exploitable supervisory signals\. Specifically, in figure[2](https://arxiv.org/html/2605.14539#S3.F2), during each policy update, we construct correction pairs from failed trajectories by conditioning the model on the original prompt together with its own erroneous output, and then sampling refined solutions\. This correction objective is then jointly optimized with the standard GRPO objective\. Since all correction samples are derived from the model’s own on\-policy failures without additional human annotation, CIPO ensures strict consistency between the training and inference distributions\. Furthermore, to prevent policy degradation caused by naively incorporating all failed trajectories into training, we integrate an adaptive mechanism that dynamically balances the proportion of successful versus failed trajectories, along with risk\-aversion reward shaping\. Moreover, we design a rollout preference strategy based on on\-policy sampling accuracy to ensure a sustained and informative training signal\. These designs enable CIPO to effectively exploit the information contained in failed samples while preserving the original advantages of RLVR\.

Intuitively, as illustrated in Figure[1](https://arxiv.org/html/2605.14539#S1.F1),CIPOimproves RLVR from two complementary perspectives\. First, the correction objective provides learning signals with stronger directionality\. Crucially, this process differentiates failure modes by sampling in the local neighborhood of erroneous trajectories: a “near\-miss” attempt \(e\.g\., simple final\-step calculation errors\) has a much higher probability of yielding correct solutions during refinement sampling than a fundamentally flawed one\. By naturally leveraging these varying rectification probabilities,CIPOextracts richer, denser signals from failures, reducing gradient ambiguity\. Second,CIPOexplicitly trains the model’s correction capability, generating correct solutions conditioned on its own erroneous attempts\. This enables our trained model not only to improve its reasoning ability but also to acquire stronger error\-correction skills, thereby extending its practical applicability to scenarios such as debugging and refinement\.

We conduct extensive experiments across 11 representative benchmarks spanning mathematical reasoning and code generation\. Results show thatCIPOconsistently improves both reasoning and error\-correction performance over strong baselines\. For correction, Seed\-Coder\-8B\(Seedet al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib52)\)trained withCIPOachieves a 7\.63% gain on DebugBench\(Tianet al\.,[2024](https://arxiv.org/html/2605.14539#bib.bib51)\), reaching performance comparable to Claude\-4\-sonnet\(Anthropic,[2025](https://arxiv.org/html/2605.14539#bib.bib24)\)and surpassing GRPO\. For reasoning, Qwen\-3\-4B\(Yanget al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib50)\)trained withCIPOimproves average accuracy by 17\.56% across six mathematical benchmarks, outperforming GRPO by 4\.55%\. Additionally,CIPOyields higher pass@K, suggesting that it goes beyond simple probability concentration, thereby enhancing intrinsic reasoning\(Yueet al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib53)\)\. In summary, our contributions are:

- •We revisit the role of failed trajectories in RLVR and investigate how they can be transformed from sparse negative feedback into useful correction\-oriented supervision\.
- •We proposeCIPO, a correction\-oriented extension for RLVR that constructs correction samples from on\-policy failed trajectories without additional annotations\.
- •Extensive experiments across 11 benchmarks demonstrate thatCIPOconsistently outperforms strong baselines in both reasoning and correction tasks, with further gains in pass@K metrics indicating genuine expansion of reasoning capabilities rather than probability redistribution\.

## 2Preliminaries

In this section, we briefly introduce RLVR and review GRPO, a representative algorithm in this paradigm\.

### 2\.1Reinforcement Learning with Verifiable Rewards

RLVR is a paradigm tailored for LLM reasoning tasks where the validity of generated outputs can be automatically verified—for instance, checking the final answer in mathematical reasoning or functional execution in code generation\.

Given a promptx∼𝒟x\\sim\\mathcal\{D\}, a policyπθ\\pi\_\{\\theta\}generates a rolloutyyautoregressively and receives a binary rewardR​\(x,y\)∈\{0,1\}R\(x,y\)\\in\\\{0,1\\\}\. The objective of RLVR is to maximize the expected reward:

maxθ⁡𝔼x∼𝒟,y∼πθ\(⋅\|x\)​\[R​\(x,y\)\]\.\\max\_\{\\theta\}\\;\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\},\\,y\\sim\\pi\_\{\\theta\}\(\\cdot\|x\)\}\\big\[R\(x,y\)\\big\]\.Due to the sparse and sequence\-level nature of verifiable rewards, policy optimization in RLVR typically relies on sampling\-based gradient estimators\.

### 2\.2Group Relative Policy Optimization

GRPO is designed to stabilize training under sparse binary rewards without requiring a value model\. For each promptxx, GRPO samples a group ofNNtrajectories\{yi\}i=1N\\\{y\_\{i\}\\\}\_\{i=1\}^\{N\}from the current policy and evaluates their rewards\{ri\}i=1N\\\{r\_\{i\}\\\}\_\{i=1\}^\{N\}\. GRPO computes a normalized relative advantage within each group:

Ai=ri−μrσr,μr=1N​∑j=1Nrj,A\_\{i\}=\\frac\{r\_\{i\}\-\\mu\_\{r\}\}\{\\sigma\_\{r\}\},\\quad\\mu\_\{r\}=\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}r\_\{j\},whereσr\\sigma\_\{r\}denotes the standard deviation of rewards in the group\. The policy is updated by reinforcing trajectories with positive advantages and suppressing those with negative advantages\.

Under this formulation, successful trajectories are reinforced relative to the group mean\. However, failed trajectories receive uniformly negative advantages whenever successful trajectories exist in the group, regardless of their specific error modes or potential partial correctness\.

## 3Correction\-Oriented Policy Optimization

![Refer to caption](https://arxiv.org/html/2605.14539v1/figs/framework0401.jpg)Figure 2:The overall framework ofCIPO\. First, we generate rollouts for the curated data via the policy model and verify their correctness\. Subsequently, we construct replayed samples using a template governed by an adaptive mechanism, which dynamically adjusts the ratio of successful to failed rollouts in the replay\. We then generate and verify rollouts for this replayed data\. Finally, we perform RL on the rollouts from both the replayed and original samples\.To address the aforementioned limitations of current RLVR methods, we proposeCIPO, which transforms on\-policy failed trajectories from mere objects of penalty into exploitable supervisory signals\. In this section, we first introduce the overall procedure ofCIPO\(§[3\.1](https://arxiv.org/html/2605.14539#S3.SS1)\), then describe two key strategies designed to enhance training stability and efficiency: adaptive replay with risk\-averse shaping \(§[3\.2](https://arxiv.org/html/2605.14539#S3.SS2)\) and difficulty\-aware trajectory preference \(§[3\.3](https://arxiv.org/html/2605.14539#S3.SS3)\)\. The core algorithm is outlined in Appendix[A](https://arxiv.org/html/2605.14539#A1)\.

### 3\.1Overall Procedure

The overall framework ofCIPO, illustrated in Figure[2](https://arxiv.org/html/2605.14539#S3.F2), extends standard RLVR by establishing an iterative cycle of generation andcorrection\-oriented replay\. At each training steptt, we optimize the policyπθ\\pi\_\{\\theta\}using two data streams: \(1\)Base Stream: Standard on\-policy rolloutsyi\{y\_\{i\}\}generated from original queriesx∼𝒟x\\sim\\mathcal\{D\}; \(2\)Correction Stream: Refinement rolloutsyi′\{y^\{\\prime\}\_\{i\}\}generated by conditioning the policy on the original query and a previous trajectoryyy\(i\.e\., promptsxrep=Concat​\(x,y\)x\_\{\\text\{rep\}\}=\\text\{Concat\}\(x,y\); the concatenation template is detailed in Appendix[A\.3](https://arxiv.org/html/2605.14539#A1.SS3)\)\.

From Suppression to Directional Guidance\.Standard RLVR methods \(e\.g\., GRPO\) inefficiently treat all failures with uniform negative suppression, providing no information onhowto improve\.CIPOtransforms these failures into informative anchors\. By successfully refining a specific erroryf​a​i​ly\_\{fail\}into a correct solutiony′y^\{\\prime\}, the model establishes a distinct gradient path connecting the failure mode to the goal state as shown in Figure[1](https://arxiv.org/html/2605.14539#S1.F1)\(b\)\. This converts ambiguous suppression signals into precise directional guidance\. However, indiscriminately training on all failed trajectories introduces severe distribution shift and learning inefficiencies\. To mitigate these risks, we introduce two main strategic mechanisms\.

### 3\.2Adaptive Replay with Risk\-Averse Shaping

To prevent policy degradation caused by naively incorporating all failed trajectories into training, we propose two complementary mechanisms for stable and efficient learning:*adaptive replay ratio*, which dynamically adjusts the mixture of successful and failed trajectories, and*risk\-averse reward shaping*, which explicitly penalizes capability regressions\.

Adaptive Replay Ratio\.To balance learning from failed trajectories with retaining previously acquired capabilities, we maintain a dynamic replay ratioρt∈\[ρmin,ρmax\]\\rho\_\{t\}\\in\[\\rho\_\{\\min\},\\rho\_\{\\max\}\]for mixing successful and failed trajectories\. This ratio is adjusted according to the model’s recent retention performance on recycled successful samples: when performance degrades or continues to decline, we increase the replay fraction of successful trajectories; when performance remains stable and high, we allow more emphasis on failed trajectories\. This yields a simple feedback\-based replay mechanism, with the full update rule deferred to Appendix[2](https://arxiv.org/html/2605.14539#alg2)\.

Risk\-Averse Reward Shaping\.Inspired by risk\-sensitive reinforcement learning\(Mihatsch and Neuneier,[2002](https://arxiv.org/html/2605.14539#bib.bib21)\), we introduce an asymmetric penalty mechanism to impose a stronger constraint against capability regressions\. Although adaptive mixing can adjust the correctness distribution of replayed rollouts, it does not directly penalize the following failure mode: the model is conditioned on a correct trajectory yet generates an incorrect response\. To mitigate this issue, we impose an additional penalty on “correct→\\rightarrowincorrect” transitions:

Rrisk​\(x,y,y′\)=R​\(x,y′\)−λrisk⋅𝕀​\[R​\(x,y\)=1∧R​\(x,y′\)=0\]R\_\{\\text\{risk\}\}\(x,y,y^\{\\prime\}\)=R\(x,y^\{\\prime\}\)\-\\lambda\_\{\\text\{risk\}\}\\cdot\\mathbb\{I\}\[R\(x,y\)=1\\land R\(x,y^\{\\prime\}\)=0\]\(1\)whereyydenotes the conditioning trajectory andy′y^\{\\prime\}is the new response\. This penalty is activated when the conditioning trajectory is correct but the new response is incorrect\. In this way, the objective explicitly suppresses capability regressions, prioritizing the preservation of existing correct behaviors while still enabling the acquisition of new ones\.

The combination of adaptive replay and risk\-averse reward shaping creates a self\-regulating training system: the adaptive controller manages the curriculum at a macro level by adjusting trajectory composition, while the shaped reward provides micro\-level guidance by penalizing individual regressions\. Together, these mechanisms enable stable learning from failure while preserving the model’s ability to reproduce correct solutions when conditioned on them\.

### 3\.3Difficulty\-Aware Trajectories Preference

To improve learning efficiency, we propose a Difficulty\-aware Trajectories Preference mechanism that prioritizes replaying prompts with moderate pass rates, thereby ensuring the model focuses on the effective learning window\. Previous studies\(Yuet al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib55); Cuiet al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib30); Liet al\.,[2025a](https://arxiv.org/html/2605.14539#bib.bib57); Chenet al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib3)\)indicate that prompts that are consistently solved \(too easy\) or consistently failed \(too hard\) may hinder the learning process or contribute zero gradient signals\. Replaying such samples wastes computational resources\.

Specifically, we target the medium\-difficulty regime\. We define the set of prioritized prompts𝒳med\\mathcal\{X\}\_\{\\text\{med\}\}as:

𝒳med=\{x∈𝒟∣δlow≤P^​\(x\)≤δhigh\}\\mathcal\{X\}\_\{\\text\{med\}\}=\\\{x\\in\\mathcal\{D\}\\mid\\delta\_\{\\text\{low\}\}\\leq\\hat\{P\}\(x\)\\leq\\delta\_\{\\text\{high\}\}\\\}\(2\)whereP^​\(x\)\\hat\{P\}\(x\)represents the empirical pass rate, andδlow,δhigh\\delta\_\{\\text\{low\}\},\\delta\_\{\\text\{high\}\}are thresholds\. When insufficient medium\-difficulty prompts are available, we adopt a fallback strategy that samples from the full distribution𝒟\\mathcal\{D\}\(see Algorithm[2](https://arxiv.org/html/2605.14539#alg2)in Appendix[A](https://arxiv.org/html/2605.14539#A1)\)\.

### 3\.4Training Objective

The joint objective combines base and correction rollouts:

𝒥CIPO​\(θ\)\\displaystyle\\mathcal\{J\}\_\{\\text\{CIPO\}\}\(\\theta\)=𝔼x∼𝒟​\[1m​∑i=1mA\(i\)​log⁡πθ​\(y\(i\)∣x\)\]\+λ​𝔼\(x,yc,rc\)∼𝒳rec​\[1n​∑i=1nA′⁣\(i\)​log⁡πθ​\(y′⁣\(i\)∣x,yc\)\]\\displaystyle=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\}\\Bigg\[\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}A^\{\(i\)\}\\log\\pi\_\{\\theta\}\\\!\\big\(y^\{\(i\)\}\\mid x\\big\)\\Bigg\]\+\\lambda\\mathbb\{E\}\_\{\(x,y\_\{c\},r\_\{c\}\)\\sim\\mathcal\{X\}\_\{\\text\{rec\}\}\}\\Bigg\[\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}A^\{\\prime\(i\)\}\\log\\pi\_\{\\theta\}\\\!\\big\(y^\{\\prime\(i\)\}\\mid x,y\_\{c\}\\big\)\\Bigg\]\(3\)where advantages are computed separately within each group, and correction rewards incorporate risk\-averse shaping\.mmandnndenote the numbers of sampled responses for base and correction rollouts, respectively, whileλ\>0\\lambda\>0controls the relative importance of correction rollouts\. The core algorithm is summarized in Algorithm[1](https://arxiv.org/html/2605.14539#alg1)\.

## 4Experiments

### 4\.1Setup

Training DatasetFor mathematical reasoning, following previous works\(Liet al\.,[2025b](https://arxiv.org/html/2605.14539#bib.bib40)\), we utilize the DeepScalerR\(Anonymous,[2025](https://arxiv.org/html/2605.14539#bib.bib41)\), which consists of approximately 40,000 unique mathematics problem\-answer pairs\. For code generation, we curate verifiable prompts from AM\-DeepSeek\-Distilled\-40M\(Tianet al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib39)\)with a primary focus on Python code generation and obtain approximately 370,000 unique items that can be verified by our sandbox server\(Bytedance\-Seed\-Foundation\-Code\-Teamet al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib38)\)\.

Baselines and VariantsCIPOis orthogonal to existing open\-source RL training recipes and can be integrated with various base algorithms\. In this work, we instantiateCIPOon top of GRPO and compare against vanilla GRPO under different training budgets as the baseline\. We also compare with PRIME\(Cuiet al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib30)\)under RLOO\(Ahmadianet al\.,[2024](https://arxiv.org/html/2605.14539#bib.bib8)\), which adheres to its official implementation\. Additionally, to isolate the contribution of online replay, we report an offline variant that only replays trajectories collected at initialization rather than continuously during training\.

ImplementationWe use the instruct mode of Qwen3\-4B\(Yanget al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib50)\)for math experiments and Seed\-Coder\-8B\(Seedet al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib52)\)for code experiments\. We implement our RL training pipeline with theverlframework\(Shenget al\.,[2024](https://arxiv.org/html/2605.14539#bib.bib37)\)\. Each batch contains 128 questions, and we generate 8 responses per question during rollout\. For rollout sampling, we use temperature=1\.0=1\.0, top\-pp=1\.0=1\.0, and a maximum of 4096 tokens\. We set the learning rate to1×10−61\\times 10^\{\-6\}and KL loss coefficient to1×10−41\\times 10^\{\-4\}\. All models are trained for 500 steps, and we report results on the final step111PRIME exhibits early training instability and fail to maintain stable optimization up to 500 steps\. We therefore report their best\-performing checkpoints for a fair comparison\.\. ForCIPO, we setλ=1\\lambda=1, the correction batch size to 128 and the number of correction rollouts to 8\.

BenchmarksWe evaluate our method on diverse reasoning benchmarks with a maximum generation length of 8192 tokens\.Math\.We evaluate on AIME24/25\(Zhang and Math\-AI,[2024](https://arxiv.org/html/2605.14539#bib.bib48),[2025](https://arxiv.org/html/2605.14539#bib.bib47)\), AMC23, MATH500\(Lightmanet al\.,[2023a](https://arxiv.org/html/2605.14539#bib.bib49)\), Minerva\(Lewkowyczet al\.,[2022](https://arxiv.org/html/2605.14539#bib.bib42)\), and OlympiadBench\(Heet al\.,[2024](https://arxiv.org/html/2605.14539#bib.bib43)\)\. For math datasets with fewer than 100 problems, we use temperature sampling \(temperature=0\.7\) with 32 samples per problem and report pass@1\. For larger datasets, we use greedy decoding\.Coding\.We evaluate on LiveCodeBench v6 \(2024\.8–2025\.5\)\(Jainet al\.,[2024](https://arxiv.org/html/2605.14539#bib.bib46)\), and LeetCode problems collected by DebugBench\(Tianet al\.,[2024](https://arxiv.org/html/2605.14539#bib.bib51)\), with unit tests from\(Xiaet al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib4)\)due to the unavailability of automated official submission\. Following the official setting, we run LiveCodeBench 10 times with temperature=0\.2, and use greedy decoding for LeetCode\.Correction\.We evaluate on CriticBench\(Linet al\.,[2024](https://arxiv.org/html/2605.14539#bib.bib44)\)under three completeness settings using greedy decoding\. For DebugBench\(Tianet al\.,[2024](https://arxiv.org/html/2605.14539#bib.bib51)\), we use temperature=0\.2 and run 8 times\.

Table 1:Main results on mathematical reasoning and code generation benchmarks\. Mathematical reasoning is evaluated on Qwen3\-4B, and code generation on Seed\-Coder\-8B\.Table 2:Pass@K results of competition\-level mathematical reasoning on Qwen3\-4B and code generation on Seed\-Coder\-8B\.
![Refer to caption](https://arxiv.org/html/2605.14539v1/x1.png)
Figure 3:Pass@8 training dynamics on LiveCodeBench v6\.

### 4\.2Main Results

Table 3:Qwen3\-4B trained withCIPOdemonstrate improved correction and critique capabilities on CriticBench in\-domain, with effective generalization to out\-of\-domain tasks\. Comm\.: Commonsense; Symb\.: Symbolic; Algo\.: Algorithmic\.Table 4:Seed\-Coder\-8B trained withCIPOachieves consistent improvements in code debugging performance on DebugBench under different completeness settings\.CIPOyields significant improvements in model reasoning performance\.To validate the effectiveness of CIPO in reasoning tasks, we conduct a systematic comparison between CIPO and the strong baseline GRPO on mathematical reasoning and code generation benchmarks\. As shown in Table[1](https://arxiv.org/html/2605.14539#S4.T1), CIPO consistently outperforms GRPO across all reasoning tasks\. Specifically, our method achieves an overall accuracy of 64\.38% on mathematical reasoning, surpassing GRPO by 4\.55%, with even larger gains on the more challenging AIME24 and AIME25 datasets, while also delivering stable improvements in code generation\. Notably, under matched computational budgets, CIPO still outperforms GRPO \(BS=256\) by 4\.72%, which further confirms that the observed gains primarily stem from algorithmic design rather than increased computational resources\.

CIPOsuccessfully expands the model’s intrinsic reasoning capabilities, which vanilla GRPO struggles to achieve\.To validate the advantage of CIPO in expanding intrinsic reasoning ability, we evaluate the pass@32 metric on competition\-style mathematical benchmarks and analyze the training dynamics on code generation tasks\. The results demonstrate that CIPO genuinely expands the model’s intrinsic capacity rather than merely reshuffling solutions via sampling\. Specifically, under a fixed budget of 32 samples, CIPO outperforms vanilla GRPO by 6\.12% on mathematical tasks\. Furthermore, on code generation, CIPO maintains a robust, monotonic upward trajectory throughout training, effectively preventing the performance saturation and fluctuation observed in the GRPO baseline, which further indicates that CIPO continuously explores diverse solutions to substantially enhance reasoning capacity\.

CIPOsubstantially enhances the model’s correction ability\.To validate CIPO’s effectiveness in error correction, we evaluate it on CriticBench and DebugBench\. As shown in Table[3](https://arxiv.org/html/2605.14539#S4.T3)and Table[4](https://arxiv.org/html/2605.14539#S4.T4), CIPO consistently improves error detection and rectification, significantly outperforming GRPO\. Specifically, on CriticBench \(Math\), CIPO boosts the correction rate by 7\.74%, surpassing GRPO by 4\.67%\. On DebugBench, CIPO achieves a 4\.20% gain, outperforming GRPO \(\+2\.53%\) and even surpassing Qwen2\.5\-72B\-Instruct while matching Claude\-Sonnet\-4\(Anthropic,[2025](https://arxiv.org/html/2605.14539#bib.bib24)\), with consistent gains across all settings\. These results demonstrate that CIPO effectively enhances the model’s ability to repair errors\.

The error correction capabilities acquired throughCIPOtraining demonstrate robust cross\-scenario generalization to diverse reasoning tasks\.To assess the transferability of the learned capabilities, we evaluate the math\-trained model on out\-of\-domain tasks\. The results indicate that, despite being trained solely on mathematical data, CIPO generalizes effectively to unseen scenarios\. Specifically, as shown in Table[3](https://arxiv.org/html/2605.14539#S4.T3), the model achieves substantial correction gains in symbolic and algorithmic reasoning\. Furthermore, CIPO enhances critique performance across all out\-of\-domain categories, which further suggests that it fosters general critique and correction capabilities rather than task\-specific overfitting, enabling effective transfer to new reasoning environments\.

Table 5:Ablation study of key components ofCIPOon Qwen3\-4B\. Each row shows performance when one component is removed\. “w/o on\-policy replay” corresponds to the offline variant \(CIPO\-Offline\) described in Section[4](https://arxiv.org/html/2605.14539#S4)\.
### 4\.3Ablation Study

We conduct ablation studies to isolate the contributions ofCIPO’s design choices: \(i\) the effect of on\-policy correction versus offline replay, and \(ii\) the necessity of each proposed strategy: adaptive\-control, risk\-aversion reward shaping and difficulty\-aware preference\.

On\-policy replay drives substantial gains over offline replay\.To disentangle the benefit of on\-policy correction from merely adding offline data, we compareCIPOagainst a “step\-0 replay” variant that utilizes offline trajectories\. As shown in Table[1](https://arxiv.org/html/2605.14539#S4.T1),CIPOsignificantly outperforms the variant relying solely on offline data, establishing on\-policy replay as a key driver of performance improvement\. Specifically,CIPOoutperforms the offline variant by an additional 3\.91% with consistent gains across all benchmarks, confirming that the improvements stem from the on\-policy mechanism’s ability to dynamically correct errors and implicit credit assignment within the current policy distribution, rather than simple data augmentation\.

Adaptive\-control dynamic replaying balances exploration and exploitation\.To demonstrate the importance of adaptively regulating the ratio of successful to failed trajectories in balancing exploration and exploitation, we compare the adaptive\-control strategy against a fixed 1:1 replay ratio\. The results show that the fixed ratio strategy lags behind the full CIPO model by 4\.19%\. Specifically, the adaptive mechanism dynamically balances the proportion of successful versus failed trajectories—leveraging failed trajectories for correction learning in early stages while adjusting the proportion later to prevent policy degradation\. This ensures a sustained and informative training signal, effectively exploiting the information in failed trajectories without the risk of over\-correction associated with fixed ratios\.

Risk\-averse reward shaping prevents over\-correction and capability regression\.To validate the necessity of risk\-averse reward shaping for model stability, we conduct an ablation study by removing this component\. The results reveal that this removal causes the most severe performance degradation, with overall accuracy plummeting by 6\.97%\. Specifically, performance drops are significant across all datasets \(ranging from \-3\.20% to \-10\.73%\), indicating that the asymmetric reward mechanism effectively balances preventing overfitting with avoiding capability regression, thereby maintaining robust reasoning capabilities while exploring correction strategies\.

Difficulty\-aware preference improves training efficiency\.To verify the role of difficulty\-aware preference in optimizing training efficiency, we remove this component and observe model performance\. The results show that removing this component leads to consistent performance declines relative to full CIPO across all benchmarks\. This validates the “zone of proximal development” theory: by prioritizing samples at the edge of the model’s capability, the strategy avoids inefficient computation on overly simple or difficult samples\.

## 5Related Works

Process SupervisionTo address the ambiguous optimization signals caused by sparse binary rewards in RLVR, prior work introduces process supervision to provide denser feedback by evaluating intermediate steps\(Lightmanet al\.,[2023b](https://arxiv.org/html/2605.14539#bib.bib17); Uesatoet al\.,[2022](https://arxiv.org/html/2605.14539#bib.bib16)\)\. A common approach trains a process reward model \(PRM\) to assess step\-level correctness for reinforcement learning or search guidance\(Wanget al\.,[2024](https://arxiv.org/html/2605.14539#bib.bib29); Luoet al\.,[2024](https://arxiv.org/html/2605.14539#bib.bib15)\)\. Other methods employ LLM critics for corrective feedback\(Xieet al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib28); Shinnet al\.,[2023](https://arxiv.org/html/2605.14539#bib.bib32)\)or leverage environment feedback to guide credit assignment via teacher model distributions\(Hübotteret al\.,[2026](https://arxiv.org/html/2605.14539#bib.bib9)\)\. Recent work such as SDPO further constructs feedback\-conditioned teachers from environmental feedback or self\-generated trajectories\. However, these approaches typically rely on additional annotations or incur substantial computational overhead, and auxiliary models may introduce biases that harm generalization\(Gaoet al\.,[2023](https://arxiv.org/html/2605.14539#bib.bib26); Wenet al\.,[2024](https://arxiv.org/html/2605.14539#bib.bib27); Kimet al\.,[2026](https://arxiv.org/html/2605.14539#bib.bib7)\)\. In contrast, CIPO derives directional signals directly from failed trajectories through correction sampling, eliminating the need for external models or human annotations while preserving the simplicity of the RLVR paradigm\.

Learning from FailureFailed trajectories often contain richer signals than uniform suppression can capture\. In reinforcement learning, hindsight experience replay \(HER\)\(Andrychowiczet al\.,[2017](https://arxiv.org/html/2605.14539#bib.bib20)\)alleviates sparse rewards by replacing intended goals with achieved outcomes\. This idea has been extended to LLMs by converting failed interactions into training data via post\-hoc rewriting, often using external rewriters to correct invalid responses\(Zhanget al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib19)\)\. Another line of work studies correction\-oriented supervised fine\-tuning \(SFT\), where models are trained on critique\-and\-revise data to improve reasoning\(Anet al\.,[2023](https://arxiv.org/html/2605.14539#bib.bib2); Zhenget al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib18); Wanget al\.,[2025](https://arxiv.org/html/2605.14539#bib.bib14)\)\. In contrast,CIPOremoves the need for relabeling or external rewriters, and integrates correction learning directly into an on\-policy RLVR framework, avoiding the distribution shift issues of offline methods\.

Self\-refinementRecent work improves self\-refinement in language models via reinforcement learning, often using multi\-turn generation where models iteratively refine their own outputs\(Kumaret al\.,[2024](https://arxiv.org/html/2605.14539#bib.bib6)\)\. These approaches treat failed trajectories as context for subsequent attempts and primarily optimize refined outputs, sometimes at the cost of first\-pass performance\. Other methods incorporate external feedback to guide correction\(Gehringet al\.,[2024](https://arxiv.org/html/2605.14539#bib.bib5); Chenet al\.,[2023](https://arxiv.org/html/2605.14539#bib.bib1)\)\. However, most operate on a single refinement trajectory per failure, without exploring nearby alternatives\. In contrast,CIPOsamples around failed trajectories and integrates the resulting signals directly into policy optimization, enabling more informative supervision without requiring multi\-turn inference at test time\.

## 6Conclusion

We introduce CIPO, which transforms on\-policy failed trajectories into exploitable supervisory signals for RLVR training\. The key insight is that correction naturally differentiates failure modes: near\-miss attempts are more likely to yield correct solutions during refinement, enabling richer learning signals from failures\. Experiments across 11 benchmarks demonstrate that CIPO achieves significant improvements on both reasoning and correction tasks, with pass@K gains indicating genuine expansion of intrinsic reasoning capabilities\.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning\. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here\.

## References

- \[1\]\(2024\)Back to basics: revisiting reinforce\-style optimization for learning from human feedback in llms\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 12248–12267\.Cited by:[§4\.1](https://arxiv.org/html/2605.14539#S4.SS1.p2.1)\.
- \[2\]S\. An, Z\. Ma, Z\. Lin, N\. Zheng, J\. Lou, and W\. Chen\(2023\)Learning from mistakes makes llm better reasoner\.arXiv preprint arXiv:2310\.20689\.Cited by:[§5](https://arxiv.org/html/2605.14539#S5.p2.1)\.
- \[3\]M\. Andrychowicz, F\. Wolski, A\. Ray, J\. Schneider, R\. Fong, P\. Welinder, B\. McGrew, J\. Tobin, O\. Pieter Abbeel, and W\. Zaremba\(2017\)Hindsight experience replay\.Advances in neural information processing systems30\.Cited by:[§5](https://arxiv.org/html/2605.14539#S5.p2.1)\.
- \[4\]Anonymous\(2025\)DeepScaleR: effective RL scaling of reasoning models via iterative context lengthening\.InSubmitted to The Fourteenth International Conference on Learning Representations,Note:under reviewExternal Links:[Link](https://openreview.net/forum?id=I6GzDCne7U)Cited by:[§4\.1](https://arxiv.org/html/2605.14539#S4.SS1.p1.1)\.
- \[5\]Anthropic\(2025\)Claude 4\.Note:[https://www\.anthropic\.com/news/claude\-4](https://www.anthropic.com/news/claude-4)Accessed: 2026\-01\-29External Links:[Link](https://www.anthropic.com/news/claude-4)Cited by:[§1](https://arxiv.org/html/2605.14539#S1.p6.1),[§4\.2](https://arxiv.org/html/2605.14539#S4.SS2.p3.1)\.
- \[6\]Bytedance\-Seed\-Foundation\-Code\-Team, :, Y\. Cheng, J\. Chen, J\. Chen, L\. Chen, L\. Chen, W\. Chen, Z\. Chen, S\. Geng, A\. Li, B\. Li, B\. Li, L\. Li, B\. Liu, J\. Liu, K\. Liu, Q\. Liu, S\. Liu, S\. Liu, T\. Liu, T\. Liu, Y\. Liu, R\. Long, J\. Mai, G\. Ning, Z\. Y\. Peng, K\. Shen, J\. Su, J\. Su, T\. Sun, Y\. Sun, Y\. Tao, G\. Wang, S\. Wang, X\. Wang, Y\. Wang, Z\. Wang, J\. Xia, L\. Xiang, X\. Xiao, Y\. Xiao, C\. Xi, S\. Xin, J\. Xu, S\. Xu, H\. Yang, J\. Yang, Y\. Yang, J\. Yuan, J\. Zhang, Y\. Zhang, Y\. Zhang, S\. Zheng, H\. Zhu, and M\. Zhu\(2025\)FullStack bench: evaluating llms as full stack coders\.External Links:2412\.00535,[Link](https://arxiv.org/abs/2412.00535)Cited by:[§4\.1](https://arxiv.org/html/2605.14539#S4.SS1.p1.1)\.
- \[7\]X\. Chen, J\. Lu, M\. Kim, D\. Zhang, J\. Tang, A\. Piché, N\. Gontier, Y\. Bengio, and E\. Kamalloo\(2025\)Self\-evolving curriculum for llm reasoning\.arXiv preprint arXiv:2505\.14970\.Cited by:[§3\.3](https://arxiv.org/html/2605.14539#S3.SS3.p1.1)\.
- \[8\]X\. Chen, M\. Lin, N\. Schärli, and D\. Zhou\(2023\)Teaching large language models to self\-debug\.arXiv preprint arXiv:2304\.05128\.Cited by:[§5](https://arxiv.org/html/2605.14539#S5.p3.1)\.
- \[9\]G\. Cui, L\. Yuan, Z\. Wang, H\. Wang, W\. Li, B\. He, Y\. Fan, T\. Yu, Q\. Xu, W\. Chen,et al\.\(2025\)Process reinforcement through implicit rewards\.arXiv preprint arXiv:2502\.01456\.Cited by:[§1](https://arxiv.org/html/2605.14539#S1.p3.1),[§3\.3](https://arxiv.org/html/2605.14539#S3.SS3.p1.1),[§4\.1](https://arxiv.org/html/2605.14539#S4.SS1.p2.1)\.
- \[10\]L\. Gao, J\. Schulman, and J\. Hilton\(2023\)Scaling laws for reward model overoptimization\.InInternational Conference on Machine Learning,pp\. 10835–10866\.Cited by:[§1](https://arxiv.org/html/2605.14539#S1.p3.1),[§5](https://arxiv.org/html/2605.14539#S5.p1.1)\.
- \[11\]J\. Gehring, K\. Zheng, J\. Copet, V\. Mella, Q\. Carbonneaux, T\. Cohen, and G\. Synnaeve\(2024\)Rlef: grounding code llms in execution feedback with reinforcement learning\.arXiv preprint arXiv:2410\.02089\.Cited by:[§5](https://arxiv.org/html/2605.14539#S5.p3.1)\.
- \[12\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi, X\. Zhang, X\. Yu, Y\. Wu, Z\. F\. Wu, Z\. Gou, Z\. Shao, Z\. Li, Z\. Gao, A\. Liu, B\. Xue, B\. Wang, B\. Wu, B\. Feng, C\. Lu, C\. Zhao, C\. Deng, C\. Ruan, D\. Dai, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Dai, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Xu, H\. Ding, H\. Gao, H\. Qu, H\. Li, J\. Guo, J\. Li, J\. Chen, J\. Yuan, J\. Tu, J\. Qiu, J\. Li, J\. L\. Cai, J\. Ni, J\. Liang, J\. Chen, K\. Dong, K\. Hu, K\. You, K\. Gao, K\. Guan, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Zhao, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, M\. Zhang, M\. Zhang, M\. Tang, M\. Zhou, M\. Li, M\. Wang, M\. Li, N\. Tian, P\. Huang, P\. Zhang, Q\. Wang, Q\. Chen, Q\. Du, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. J\. Chen, R\. L\. Jin, R\. Chen, S\. Lu, S\. Zhou, S\. Chen, S\. Ye, S\. Wang, S\. Yu, S\. Zhou, S\. Pan, S\. S\. Li, S\. Zhou, S\. Wu, T\. Yun, T\. Pei, T\. Sun, T\. Wang, W\. Zeng, W\. Liu, W\. Liang, W\. Gao, W\. Yu, W\. Zhang, W\. L\. Xiao, W\. An, X\. Liu, X\. Wang, X\. Chen, X\. Nie, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Yang, X\. Li, X\. Su, X\. Lin, X\. Q\. Li, X\. Jin, X\. Shen, X\. Chen, X\. Sun, X\. Wang, X\. Song, X\. Zhou, X\. Wang, X\. Shan, Y\. K\. Li, Y\. Q\. Wang, Y\. X\. Wei, Y\. Zhang, Y\. Xu, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Wang, Y\. Yu, Y\. Zhang, Y\. Shi, Y\. Xiong, Y\. He, Y\. Piao, Y\. Wang, Y\. Tan, Y\. Ma, Y\. Liu, Y\. Guo, Y\. Ou, Y\. Wang, Y\. Gong, Y\. Zou, Y\. He, Y\. Xiong, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Y\. X\. Zhu, Y\. Huang, Y\. Li, Y\. Zheng, Y\. Zhu, Y\. Ma, Y\. Tang, Y\. Zha, Y\. Yan, Z\. Z\. Ren, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Xu, Z\. Xie, Z\. Zhang, Z\. Hao, Z\. Ma, Z\. Yan, Z\. Wu, Z\. Gu, Z\. Zhu, Z\. Liu, Z\. Li, Z\. Xie, Z\. Song, Z\. Pan, Z\. Huang, Z\. Xu, Z\. Zhang, and Z\. Zhang\(2025\-09\)DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:ISSN 1476\-4687,[Link](http://dx.doi.org/10.1038/s41586-025-09422-z),[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[§1](https://arxiv.org/html/2605.14539#S1.p1.1)\.
- \[13\]C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang, J\. Liu, L\. Qi, Z\. Liu, and M\. Sun\(2024\-08\)OlympiadBench: a challenging benchmark for promoting AGI with olympiad\-level bilingual multimodal scientific problems\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 3828–3850\.External Links:[Link](https://aclanthology.org/2024.acl-long.211/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.211)Cited by:[§4\.1](https://arxiv.org/html/2605.14539#S4.SS1.p4.1)\.
- \[14\]Z\. Hu, Y\. Wang, Y\. He, J\. Wu, Y\. Zhao, S\. Ng, C\. Breazeal, A\. T\. Luu, H\. W\. Park, and B\. Hooi\(2026\)Rewarding the rare: uniqueness\-aware rl for creative problem solving in llms\.arXiv preprint arXiv:2601\.08763\.Cited by:[§1](https://arxiv.org/html/2605.14539#S1.p2.1)\.
- \[15\]J\. Hübotter, F\. Lübeck, L\. Behric, A\. Baumann, M\. Bagatella, D\. Marta, I\. Hakimi, I\. Shenfeld, T\. K\. Buening, C\. Guestrin,et al\.\(2026\)Reinforcement learning via self\-distillation\.arXiv preprint arXiv:2601\.20802\.Cited by:[§1](https://arxiv.org/html/2605.14539#S1.p2.1),[§1](https://arxiv.org/html/2605.14539#S1.p3.1),[§5](https://arxiv.org/html/2605.14539#S5.p1.1)\.
- \[16\]N\. Jain, K\. Han, A\. G\. W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica\(2024\)LiveCodeBench: holistic and contamination free evaluation of large language models for code\.arXiv preprint\.Cited by:[§4\.1](https://arxiv.org/html/2605.14539#S4.SS1.p4.1)\.
- \[17\]J\. Kim, X\. Luo, M\. Kim, S\. Lee, D\. Kim, J\. Jeon, D\. Li, and Y\. Yang\(2026\)Why does self\-distillation \(sometimes\) degrade the reasoning capability of llms?\.arXiv preprint arXiv:2603\.24472\.Cited by:[§1](https://arxiv.org/html/2605.14539#S1.p3.1),[§5](https://arxiv.org/html/2605.14539#S5.p1.1)\.
- \[18\]A\. Kumar, V\. Zhuang, R\. Agarwal, Y\. Su, J\. D\. Co\-Reyes, A\. Singh, K\. Baumli, S\. Iqbal, C\. Bishop, R\. Roelofs,et al\.\(2024\)Training language models to self\-correct via reinforcement learning\.arXiv preprint arXiv:2409\.12917\.Cited by:[§5](https://arxiv.org/html/2605.14539#S5.p3.1)\.
- \[19\]A\. Lewkowycz, A\. Andreassen, D\. Dohan, E\. Dyer, H\. Michalewski, V\. Ramasesh, A\. Slone, C\. Anil, I\. Schlag, T\. Gutman\-Solo,et al\.\(2022\)Solving quantitative reasoning problems with language models\.Advances in neural information processing systems35,pp\. 3843–3857\.Cited by:[§4\.1](https://arxiv.org/html/2605.14539#S4.SS1.p4.1)\.
- \[20\]J\. Li, H\. Lin, H\. Lu, K\. Wen, Z\. Yang, J\. Gao, Y\. Wu, and J\. Zhang\(2025\)QuestA: expanding reasoning capacity in llms via question augmentation\.External Links:2507\.13266,[Link](https://arxiv.org/abs/2507.13266)Cited by:[§3\.3](https://arxiv.org/html/2605.14539#S3.SS3.p1.1)\.
- \[21\]T\. Li, Y\. Zhang, P\. Yu, S\. Saha, D\. Khashabi, J\. Weston, J\. Lanchantin, and T\. Wang\(2025\)Jointly reinforcing diversity and quality in language model generations\.External Links:2509\.02534,[Link](https://arxiv.org/abs/2509.02534)Cited by:[§4\.1](https://arxiv.org/html/2605.14539#S4.SS1.p1.1)\.
- \[22\]H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe\(2023\)Let’s verify step by step\.arXiv preprint arXiv:2305\.20050\.Cited by:[§4\.1](https://arxiv.org/html/2605.14539#S4.SS1.p4.1)\.
- \[23\]H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe\(2023\)Let’s verify step by step\.InThe Twelfth International Conference on Learning Representations,Cited by:[§5](https://arxiv.org/html/2605.14539#S5.p1.1)\.
- \[24\]Z\. Lin, Z\. Gou, T\. Liang, R\. Luo, H\. Liu, and Y\. Yang\(2024\-08\)CriticBench: benchmarking LLMs for critique\-correct reasoning\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 1552–1587\.External Links:[Link](https://aclanthology.org/2024.findings-acl.91/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.91)Cited by:[§4\.1](https://arxiv.org/html/2605.14539#S4.SS1.p4.1)\.
- \[25\]L\. Luo, Y\. Liu, R\. Liu, S\. Phatale, M\. Guo, H\. Lara, Y\. Li, L\. Shu, Y\. Zhu, L\. Meng,et al\.\(2024\)Improve mathematical reasoning in language models by automated process supervision\.arXiv preprint arXiv:2406\.06592\.Cited by:[§5](https://arxiv.org/html/2605.14539#S5.p1.1)\.
- \[26\]O\. Mihatsch and R\. Neuneier\(2002\)Risk\-sensitive reinforcement learning\.Machine learning49\(2\),pp\. 267–290\.Cited by:[§3\.2](https://arxiv.org/html/2605.14539#S3.SS2.p3.1)\.
- \[27\]OpenAI, :, A\. Jaech, A\. Kalai, A\. Lerer, A\. Richardson, A\. El\-Kishky, A\. Low, A\. Helyar, A\. Madry, A\. Beutel, A\. Carney, A\. Iftimie, A\. Karpenko, A\. T\. Passos, A\. Neitz, A\. Prokofiev, A\. Wei, A\. Tam, A\. Bennett, A\. Kumar, A\. Saraiva, A\. Vallone, A\. Duberstein, A\. Kondrich, A\. Mishchenko, A\. Applebaum, A\. Jiang, A\. Nair, B\. Zoph, B\. Ghorbani, B\. Rossen, B\. Sokolowsky, B\. Barak, B\. McGrew, B\. Minaiev, B\. Hao, B\. Baker, B\. Houghton, B\. McKinzie, B\. Eastman, C\. Lugaresi, C\. Bassin, C\. Hudson, C\. M\. Li, C\. de Bourcy, C\. Voss, C\. Shen, C\. Zhang, C\. Koch, C\. Orsinger, C\. Hesse, C\. Fischer, C\. Chan, D\. Roberts, D\. Kappler, D\. Levy, D\. Selsam, D\. Dohan, D\. Farhi, D\. Mely, D\. Robinson, D\. Tsipras, D\. Li, D\. Oprica, E\. Freeman, E\. Zhang, E\. Wong, E\. Proehl, E\. Cheung, E\. Mitchell, E\. Wallace, E\. Ritter, E\. Mays, F\. Wang, F\. P\. Such, F\. Raso, F\. Leoni, F\. Tsimpourlas, F\. Song, F\. von Lohmann, F\. Sulit, G\. Salmon, G\. Parascandolo, G\. Chabot, G\. Zhao, G\. Brockman, G\. Leclerc, H\. Salman, H\. Bao, H\. Sheng, H\. Andrin, H\. Bagherinezhad, H\. Ren, H\. Lightman, H\. W\. Chung, I\. Kivlichan, I\. O’Connell, I\. Osband, I\. C\. Gilaberte, I\. Akkaya, I\. Kostrikov, I\. Sutskever, I\. Kofman, J\. Pachocki, J\. Lennon, J\. Wei, J\. Harb, J\. Twore, J\. Feng, J\. Yu, J\. Weng, J\. Tang, J\. Yu, J\. Q\. Candela, J\. Palermo, J\. Parish, J\. Heidecke, J\. Hallman, J\. Rizzo, J\. Gordon, J\. Uesato, J\. Ward, J\. Huizinga, J\. Wang, K\. Chen, K\. Xiao, K\. Singhal, K\. Nguyen, K\. Cobbe, K\. Shi, K\. Wood, K\. Rimbach, K\. Gu\-Lemberg, K\. Liu, K\. Lu, K\. Stone, K\. Yu, L\. Ahmad, L\. Yang, L\. Liu, L\. Maksin, L\. Ho, L\. Fedus, L\. Weng, L\. Li, L\. McCallum, L\. Held, L\. Kuhn, L\. Kondraciuk, L\. Kaiser, L\. Metz, M\. Boyd, M\. Trebacz, M\. Joglekar, M\. Chen, M\. Tintor, M\. Meyer, M\. Jones, M\. Kaufer, M\. Schwarzer, M\. Shah, M\. Yatbaz, M\. Y\. Guan, M\. Xu, M\. Yan, M\. Glaese, M\. Chen, M\. Lampe, M\. Malek, M\. Wang, M\. Fradin, M\. McClay, M\. Pavlov, M\. Wang, M\. Wang, M\. Murati, M\. Bavarian, M\. Rohaninejad, N\. McAleese, N\. Chowdhury, N\. Chowdhury, N\. Ryder, N\. Tezak, N\. Brown, O\. Nachum, O\. Boiko, O\. Murk, O\. Watkins, P\. Chao, P\. Ashbourne, P\. Izmailov, P\. Zhokhov, R\. Dias, R\. Arora, R\. Lin, R\. G\. Lopes, R\. Gaon, R\. Miyara, R\. Leike, R\. Hwang, R\. Garg, R\. Brown, R\. James, R\. Shu, R\. Cheu, R\. Greene, S\. Jain, S\. Altman, S\. Toizer, S\. Toyer, S\. Miserendino, S\. Agarwal, S\. Hernandez, S\. Baker, S\. McKinney, S\. Yan, S\. Zhao, S\. Hu, S\. Santurkar, S\. R\. Chaudhuri, S\. Zhang, S\. Fu, S\. Papay, S\. Lin, S\. Balaji, S\. Sanjeev, S\. Sidor, T\. Broda, A\. Clark, T\. Wang, T\. Gordon, T\. Sanders, T\. Patwardhan, T\. Sottiaux, T\. Degry, T\. Dimson, T\. Zheng, T\. Garipov, T\. Stasi, T\. Bansal, T\. Creech, T\. Peterson, T\. Eloundou, V\. Qi, V\. Kosaraju, V\. Monaco, V\. Pong, V\. Fomenko, W\. Zheng, W\. Zhou, W\. McCabe, W\. Zaremba, Y\. Dubois, Y\. Lu, Y\. Chen, Y\. Cha, Y\. Bai, Y\. He, Y\. Zhang, Y\. Wang, Z\. Shao, and Z\. Li\(2024\)OpenAI o1 system card\.External Links:2412\.16720,[Link](https://arxiv.org/abs/2412.16720)Cited by:[§1](https://arxiv.org/html/2605.14539#S1.p1.1)\.
- \[28\]B\. Seed, Y\. Zhang, J\. Su, Y\. Sun, C\. Xi, X\. Xiao, S\. Zheng, A\. Zhang, K\. Liu, D\. Zan, T\. Sun, J\. Zhu, S\. Xin, D\. Huang, Y\. Bai, L\. Dong, C\. Li, J\. Chen, H\. Zhou, Y\. Huang, G\. Ning, X\. Song, J\. Chen, S\. Liu, K\. Shen, L\. Xiang, and Y\. Wu\(2025\)Seed\-coder: let the code model curate data for itself\.External Links:2506\.03524,[Link](https://arxiv.org/abs/2506.03524)Cited by:[§1](https://arxiv.org/html/2605.14539#S1.p6.1),[§4\.1](https://arxiv.org/html/2605.14539#S4.SS1.p3.6)\.
- \[29\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo\(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.External Links:2402\.03300,[Link](https://arxiv.org/abs/2402.03300)Cited by:[§1](https://arxiv.org/html/2605.14539#S1.p2.1)\.
- \[30\]G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu\(2024\)HybridFlow: a flexible and efficient rlhf framework\.arXiv preprint arXiv: 2409\.19256\.Cited by:[§4\.1](https://arxiv.org/html/2605.14539#S4.SS1.p3.6)\.
- \[31\]N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao\(2023\)Reflexion: language agents with verbal reinforcement learning\.Advances in Neural Information Processing Systems36,pp\. 8634–8652\.Cited by:[§5](https://arxiv.org/html/2605.14539#S5.p1.1)\.
- \[32\]K\. Team, A\. Du, B\. Gao, B\. Xing, C\. Jiang, C\. Chen, C\. Li, C\. Xiao, C\. Du, C\. Liao, C\. Tang, C\. Wang, D\. Zhang, E\. Yuan, E\. Lu, F\. Tang, F\. Sung, G\. Wei, G\. Lai, H\. Guo, H\. Zhu, H\. Ding, H\. Hu, H\. Yang, H\. Zhang, H\. Yao, H\. Zhao, H\. Lu, H\. Li, H\. Yu, H\. Gao, H\. Zheng, H\. Yuan, J\. Chen, J\. Guo, J\. Su, J\. Wang, J\. Zhao, J\. Zhang, J\. Liu, J\. Yan, J\. Wu, L\. Shi, L\. Ye, L\. Yu, M\. Dong, N\. Zhang, N\. Ma, Q\. Pan, Q\. Gong, S\. Liu, S\. Ma, S\. Wei, S\. Cao, S\. Huang, T\. Jiang, W\. Gao, W\. Xiong, W\. He, W\. Huang, W\. Xu, W\. Wu, W\. He, X\. Wei, X\. Jia, X\. Wu, X\. Xu, X\. Zu, X\. Zhou, X\. Pan, Y\. Charles, Y\. Li, Y\. Hu, Y\. Liu, Y\. Chen, Y\. Wang, Y\. Liu, Y\. Qin, Y\. Liu, Y\. Yang, Y\. Bao, Y\. Du, Y\. Wu, Y\. Wang, Z\. Zhou, Z\. Wang, Z\. Li, Z\. Zhu, Z\. Zhang, Z\. Wang, Z\. Yang, Z\. Huang, Z\. Huang, Z\. Xu, Z\. Yang, and Z\. Lin\(2025\)Kimi k1\.5: scaling reinforcement learning with llms\.External Links:2501\.12599,[Link](https://arxiv.org/abs/2501.12599)Cited by:[§1](https://arxiv.org/html/2605.14539#S1.p1.1)\.
- \[33\]R\. Tian, Y\. Ye, Y\. Qin, X\. Cong, Y\. Lin, Y\. Pan, Y\. Wu, H\. Haotian, L\. Weichuan, Z\. Liu, and M\. Sun\(2024\-08\)DebugBench: evaluating debugging capability of large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 4173–4198\.External Links:[Link](https://aclanthology.org/2024.findings-acl.247/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.247)Cited by:[§1](https://arxiv.org/html/2605.14539#S1.p6.1),[§4\.1](https://arxiv.org/html/2605.14539#S4.SS1.p4.1)\.
- \[34\]X\. Tian, S\. Zhao, H\. Wang, S\. Chen, Y\. Peng, Y\. Ji, H\. Zhao, and X\. Li\(2025\)DeepDistill: enhancing llm reasoning capabilities via large\-scale difficulty\-graded data training\.External Links:2504\.17565,[Link](https://arxiv.org/abs/2504.17565)Cited by:[§4\.1](https://arxiv.org/html/2605.14539#S4.SS1.p1.1)\.
- \[35\]J\. Uesato, N\. Kushman, R\. Kumar, F\. Song, N\. Siegel, L\. Wang, A\. Creswell, G\. Irving, and I\. Higgins\(2022\)Solving math word problems with process\-and outcome\-based feedback\.arXiv preprint arXiv:2211\.14275\.Cited by:[§5](https://arxiv.org/html/2605.14539#S5.p1.1)\.
- \[36\]P\. Wang, L\. Li, Z\. Shao, R\. Xu, D\. Dai, Y\. Li, D\. Chen, Y\. Wu, and Z\. Sui\(2024\)Math\-shepherd: verify and reinforce llms step\-by\-step without human annotations\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9426–9439\.Cited by:[§1](https://arxiv.org/html/2605.14539#S1.p3.1),[§5](https://arxiv.org/html/2605.14539#S5.p1.1)\.
- \[37\]Y\. Wang, X\. Yue, and W\. Chen\(2025\)Critique fine\-tuning: learning to critique is more effective than learning to imitate\.arXiv preprint arXiv:2501\.17703\.Cited by:[§5](https://arxiv.org/html/2605.14539#S5.p2.1)\.
- \[38\]X\. Wen, J\. Lou, Y\. Lu, H\. Lin, X\. Yu, X\. Lu, B\. He, X\. Han, D\. Zhang, and L\. Sun\(2024\)Rethinking reward model evaluation: are we barking up the wrong tree?\.arXiv preprint arXiv:2410\.05584\.Cited by:[§1](https://arxiv.org/html/2605.14539#S1.p3.1),[§5](https://arxiv.org/html/2605.14539#S5.p1.1)\.
- \[39\]Y\. Xia, W\. Shen, Y\. Wang, J\. K\. Liu, H\. Sun, S\. Wu, J\. Hu, and X\. Xu\(2025\)Leetcodedataset: a temporal dataset for robust evaluation and efficient training of code llms\.arXiv preprint arXiv:2504\.14655\.Cited by:[§4\.1](https://arxiv.org/html/2605.14539#S4.SS1.p4.1)\.
- \[40\]G\. Xie, Y\. Shi, H\. Tian, T\. Yao, and X\. Zhang\(2025\)CAPO: towards enhancing llm reasoning through generative credit assignment\.arXiv preprint arXiv:2508\.02298\.Cited by:[§1](https://arxiv.org/html/2605.14539#S1.p3.1),[§5](https://arxiv.org/html/2605.14539#S5.p1.1)\.
- \[41\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu\(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§1](https://arxiv.org/html/2605.14539#S1.p6.1),[§4\.1](https://arxiv.org/html/2605.14539#S4.SS1.p3.6)\.
- \[42\]Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, M\. Qiao, Y\. Wu, and M\. Wang\(2025\)DAPO: an open\-source llm reinforcement learning system at scale\.External Links:2503\.14476,[Link](https://arxiv.org/abs/2503.14476)Cited by:[§3\.3](https://arxiv.org/html/2605.14539#S3.SS3.p1.1)\.
- \[43\]Y\. Yue, Z\. Chen, R\. Lu, A\. Zhao, Z\. Wang, Y\. Yue, S\. Song, and G\. Huang\(2025\)Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?\.External Links:2504\.13837,[Link](https://arxiv.org/abs/2504.13837)Cited by:[§1](https://arxiv.org/html/2605.14539#S1.p2.1),[§1](https://arxiv.org/html/2605.14539#S1.p6.1)\.
- \[44\]K\. Zhang, Q\. Yao, S\. Liu, W\. Zhang, M\. Cen, Y\. Zhou, W\. Fang, Y\. Zhao, B\. Lai, and M\. Song\(2025\)Replay failures as successes: sample\-efficient reinforcement learning for instruction following\.arXiv preprint arXiv:2512\.23457\.Cited by:[§5](https://arxiv.org/html/2605.14539#S5.p2.1)\.
- \[45\]Y\. Zhang and T\. Math\-AI\(2024\)American invitational mathematics examination \(aime\) 2024\.Cited by:[§4\.1](https://arxiv.org/html/2605.14539#S4.SS1.p4.1)\.
- \[46\]Y\. Zhang and T\. Math\-AI\(2025\)American invitational mathematics examination \(aime\) 2025\.Cited by:[§4\.1](https://arxiv.org/html/2605.14539#S4.SS1.p4.1)\.
- \[47\]X\. Zheng, J\. Lou, B\. Cao, X\. Wen, Y\. Ji, H\. Lin, Y\. Lu, X\. Han, D\. Zhang, and L\. Sun\(2025\)Critic\-cot: boosting the reasoning abilities of large language model via chain\-of\-thought critic\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 1768–1806\.Cited by:[§5](https://arxiv.org/html/2605.14539#S5.p2.1)\.

## Appendix ADetails about Method

### A\.1Algorithm of CIPO

Algorithm[1](https://arxiv.org/html/2605.14539#alg1)outlines the core workflow of CIPO\. In practice, to improve training efficiency, correction rollouts are based on the previous step, enabling parallel sampling instead of sequentially waiting for the base rollout to complete and be rewarded\.

Algorithm 1Correction\-Oriented Policy Optimization1:Initial policy

πθ\\pi\_\{\\theta\}, prompt set

𝒟\\mathcal\{D\}, reward function

R​\(⋅\)R\(\\cdot\)
2:Hyperparams:group size

GG, replay fraction

γ\\gamma, difficulty range

\[δlow,δhigh\]\[\\delta\_\{\\text\{low\}\},\\delta\_\{\\text\{high\}\}\], risk penalty

λrisk\\lambda\_\{\\text\{risk\}\}, target reward

R∗R^\{\*\}
3:Initialize

ρ←ρ0\\rho\\leftarrow\\rho\_\{0\},

c←0c\\leftarrow 0,

Rprev←NoneR\_\{\\text\{prev\}\}\\leftarrow\\textbf\{None\}
4:fortraining step

t=1,2,…t=1,2,\\ldotsdo

5:// Base rollouts

6:Sample prompts

\{xj\}j=1B\\\{x\_\{j\}\\\}\_\{j=1\}^\{B\}from

𝒟\\mathcal\{D\}; for each

xx, sample

GGresponses and compute rewards

7:Collect base trajectories

ℬbase=\{\(x,yi,ri\)\}\\mathcal\{B\}\_\{\\text\{base\}\}=\\\{\(x,y\_\{i\},r\_\{i\}\)\\\}
8:// Correction rollouts

9:

𝒮replay←RolloutReplay​\(ℬbase,\[δlow,δhigh\],⌊γ​B⌋,ρ\)\\mathcal\{S\}\_\{\\text\{replay\}\}\\leftarrow\\textsc\{RolloutReplay\}\(\\mathcal\{B\}\_\{\\text\{base\}\},\[\\delta\_\{\\text\{low\}\},\\delta\_\{\\text\{high\}\}\],\\lfloor\\gamma B\\rfloor,\\rho\)as in Algorithm[2](https://arxiv.org/html/2605.14539#alg2)

10:foreach

\(x,yc,rc\)∈𝒮replay\(x,y\_\{c\},r\_\{c\}\)\\in\\mathcal\{S\}\_\{\\text\{replay\}\}do

11:Construct

x′←Augment​\(x,yc\)x^\{\\prime\}\\leftarrow\\textsc\{Augment\}\(x,y\_\{c\}\); sample

GGresponses

\{yi′\}i=1G\\\{y^\{\\prime\}\_\{i\}\\\}\_\{i=1\}^\{G\}
12:Compute shaped rewards

r~i←Rrisk​\(x,yc,yi′\)\\tilde\{r\}\_\{i\}\\leftarrow R\_\{\\text\{risk\}\}\(x,y\_\{c\},y^\{\\prime\}\_\{i\}\)via Eq\.[1](https://arxiv.org/html/2605.14539#S3.E1)

13:endfor

14:Collect correction trajectories

ℬcor=\{\(x′,yi′,r~i\)\}\\mathcal\{B\}\_\{\\text\{cor\}\}=\\\{\(x^\{\\prime\},y^\{\\prime\}\_\{i\},\\tilde\{r\}\_\{i\}\)\\\}
15:// Policy update

16:Update

πθ\\pi\_\{\\theta\}on

ℬbase∪ℬcor\\mathcal\{B\}\_\{\\text\{base\}\}\\cup\\mathcal\{B\}\_\{\\text\{cor\}\}
17:// Adaptive ratio update

18:

𝒮\+←\{\(x′,yi′,r~i\)∈ℬcor:rc=1\}\\mathcal\{S\}\_\{\+\}\\leftarrow\\\{\(x^\{\\prime\},y^\{\\prime\}\_\{i\},\\tilde\{r\}\_\{i\}\)\\in\\mathcal\{B\}\_\{\\text\{cor\}\}:r\_\{c\}=1\\\}
19:// corrections conditioned on successful trajectories

20:

Rt←1\|𝒮\+\|​∑\(x′,y′,r~\)∈𝒮\+r~R\_\{t\}\\leftarrow\\frac\{1\}\{\|\\mathcal\{S\}\_\{\+\}\|\}\\sum\_\{\(x^\{\\prime\},y^\{\\prime\},\\tilde\{r\}\)\\in\\mathcal\{S\}\_\{\+\}\}\\tilde\{r\}
21:

ρ,c←UpdateRatio​\(ρ,Rt,Rprev,c;R∗\)\\rho,c\\leftarrow\\textsc\{UpdateRatio\}\(\\rho,R\_\{t\},R\_\{\\text\{prev\}\},c;R^\{\*\}\)as in Algorithm[3](https://arxiv.org/html/2605.14539#alg3)

22:

Rprev←RtR\_\{\\text\{prev\}\}\\leftarrow R\_\{t\}
23:endfor

24:Return:

πθ\\pi\_\{\\theta\}

### A\.2Algorithm of RolloutReplay and UpdateRatio

This section presents detailed descriptions of the two sub\-algorithms employed in Algorithm[1](https://arxiv.org/html/2605.14539#alg1)\.

RolloutReplay\(Algorithm[2](https://arxiv.org/html/2605.14539#alg2)\) selects trajectories for correction rollouts\. It first prioritizes prompts within the medium\-difficulty range\[δlow,δhigh\]\[\\delta\_\{\\text\{low\}\},\\delta\_\{\\text\{high\}\}\]based on their empirical pass rates, then falls back to remaining prompts if needed\. The selected trajectories are split into successful and failed groups, with the ratioρ\\rhocontrolling their mixture\. The detailed hyperparameter settings are provided in Appendix[B](https://arxiv.org/html/2605.14539#A2)\.

Algorithm 2RolloutReplay\(ℬ\\mathcal\{B\},\[δlow,δhigh\]\[\\delta\_\{\\text\{low\}\},\\delta\_\{\\text\{high\}\}\],NN,ρ\\rho\)1:Input:base rollouts

ℬ=\{\(x,y,r\)\}\\mathcal\{B\}=\\\{\(x,y,r\)\\\}, difficulty range

\[δlow,δhigh\]\[\\delta\_\{\\text\{low\}\},\\delta\_\{\\text\{high\}\}\], target size

NN, positive ratio

ρ\\rho
2:Compute per\-prompt pass rate

P^​\(x\)=1\|\{\(x,y,r\)∈ℬ\}\|​∑\(x,y,r\)∈ℬr\\hat\{P\}\(x\)=\\frac\{1\}\{\|\\\{\(x,y,r\)\\in\\mathcal\{B\}\\\}\|\}\\sum\_\{\(x,y,r\)\\in\\mathcal\{B\}\}r
3:

ℬmed←\{\(x,y,r\)∈ℬ:δlow≤P^​\(x\)≤δhigh\}\\mathcal\{B\}\_\{\\text\{med\}\}\\leftarrow\\\{\(x,y,r\)\\in\\mathcal\{B\}:\\delta\_\{\\text\{low\}\}\\leq\\hat\{P\}\(x\)\\leq\\delta\_\{\\text\{high\}\}\\\}// medium\-difficulty

4:

ℬ′←Shuffle​\(ℬmed\)⊕Shuffle​\(ℬ∖ℬmed\)\\mathcal\{B\}^\{\\prime\}\\leftarrow\\textsc\{Shuffle\}\(\\mathcal\{B\}\_\{\\text\{med\}\}\)\\oplus\\textsc\{Shuffle\}\(\\mathcal\{B\}\\setminus\\mathcal\{B\}\_\{\\text\{med\}\}\)// prioritize medium, fallback to rest

5:Split

ℬ′\\mathcal\{B\}^\{\\prime\}into

ℬ\+=\{\(x,y,r\):r=1\}\\mathcal\{B\}\_\{\+\}=\\\{\(x,y,r\):r=1\\\}and

ℬ−=\{\(x,y,r\):r=0\}\\mathcal\{B\}\_\{\-\}=\\\{\(x,y,r\):r=0\\\}
6:

N\+←min⁡\(⌊ρ​N⌋,\|ℬ\+\|\)N\_\{\+\}\\leftarrow\\min\(\\lfloor\\rho N\\rfloor,\|\\mathcal\{B\}\_\{\+\}\|\),

N−←min⁡\(N−N\+,\|ℬ−\|\)N\_\{\-\}\\leftarrow\\min\(N\-N\_\{\+\},\|\\mathcal\{B\}\_\{\-\}\|\)
7:Backfill from

ℬ\+\\mathcal\{B\}\_\{\+\}if

N\+\+N−<NN\_\{\+\}\+N\_\{\-\}<N
8:Return:

ℬ\+\[1:N\+\]∪ℬ−\[1:N−\]\\mathcal\{B\}\_\{\+\}\[1\\\!:\\\!N\_\{\+\}\]\\cup\\mathcal\{B\}\_\{\-\}\[1\\\!:\\\!N\_\{\-\}\]

UpdateRatio\.As shown in Algorithm[3](https://arxiv.org/html/2605.14539#alg3), we adapt the replay ratioρt∈\[ρmin,ρmax\]\\rho\_\{t\}\\in\[\\rho\_\{\\min\},\\rho\_\{\\max\}\]according to the model’s retention performance on replayed successful samples\. LetRtR\_\{t\}denote the average shaped reward on corrections replayed from successful trajectories at iterationtt, and letR∗R^\{\*\}denote the target retention level\. The replay ratio is updated using three signals: the current performance gap to the target, the recent performance deterioration relative to the previous step, and a capped consecutive\-underperformance term:

ρt\+1=clip⁡\[ρt​\(1\+w1​\(R∗−Rt\)\+w2​max⁡\(0,Rt−1−Rt\)\+w3​min⁡\(ct,3\)\),ρmin,ρmax\]\.\\rho\_\{t\+1\}=\\operatorname\{clip\}\\\!\\left\[\\rho\_\{t\}\\left\(1\+w\_\{1\}\(R^\{\*\}\-R\_\{t\}\)\+w\_\{2\}\\max\(0,R\_\{t\-1\}\-R\_\{t\}\)\+w\_\{3\}\\min\(c\_\{t\},3\)\\right\),\\,\\rho\_\{\\min\},\\rho\_\{\\max\}\\right\]\.\(4\)Here,ctc\_\{t\}denotes the consecutive\-underperformance counter, which is incremented whenRt<R∗R\_\{t\}<R^\{\*\}and reset otherwise\. The first term increases the replay fraction of successful trajectories when the current retention performance falls below the target\. The second term further increases this fraction when retention performance deteriorates relative to the previous iteration\. The third term makes the update more conservative under persistent underperformance, while capping its contribution to avoid dominating the overall update\. Detailed hyperparameter settings are provided in Appendix[B](https://arxiv.org/html/2605.14539#A2)\.

Algorithm 3UpdateRatio\(ρt\\rho\_\{t\},RtR\_\{t\},Rt−1R\_\{t\-1\},ct−1c\_\{t\-1\};R∗R^\{\*\}\)1:Input:current replay ratio

ρt\\rho\_\{t\}, current average shaped reward

RtR\_\{t\}, previous average shaped reward

Rt−1R\_\{t\-1\}, previous underperformance counter

ct−1c\_\{t\-1\}, target reward

R∗R^\{\*\}
2:Hyperparameters:

w1,w2,w3w\_\{1\},w\_\{2\},w\_\{3\},

ρmin,ρmax\\rho\_\{\\min\},\\rho\_\{\\max\}
3:if

Rt<R∗R\_\{t\}<R^\{\*\}then

4:

ct←ct−1\+1c\_\{t\}\\leftarrow c\_\{t\-1\}\+1
5:else

6:

ct←0c\_\{t\}\\leftarrow 0
7:endif

8:

f1←R∗−Rtf\_\{1\}\\leftarrow R^\{\*\}\-R\_\{t\}
9:if

Rt−1R\_\{t\-1\}is availablethen

10:

f2←max⁡\(0,Rt−1−Rt\)f\_\{2\}\\leftarrow\\max\(0,R\_\{t\-1\}\-R\_\{t\}\)
11:else

12:

f2←0f\_\{2\}\\leftarrow 0
13:endif

14:

f3←min⁡\(ct,3\)f\_\{3\}\\leftarrow\\min\(c\_\{t\},3\)
15:

ρt\+1←Clip​\(ρt⋅\(1\+w1​f1\+w2​f2\+w3​f3\),ρmin,ρmax\)\\rho\_\{t\+1\}\\leftarrow\\textsc\{Clip\}\\\!\\left\(\\rho\_\{t\}\\cdot\(1\+w\_\{1\}f\_\{1\}\+w\_\{2\}f\_\{2\}\+w\_\{3\}f\_\{3\}\),\\,\\rho\_\{\\min\},\\,\\rho\_\{\\max\}\\right\)
16:return

ρt\+1,ct\\rho\_\{t\+1\},c\_\{t\}

### A\.3Correction Prompt Construction\.

After selecting a subset of rolloutsℛ\\mathcal\{R\}\(containing both correct and incorrect candidates\), we constructcorrection promptsthat condition on the original problem and the model’s previous attempt\. Specifically, for a selected rollout\(x,ycand,r\)\(x,y\_\{\\text\{cand\}\},r\), we create:

\{Original Prompt x\} Below is a candidate solution from a large language model \(correctness unknown\): <candidate\_solution\> \{y\_cand\} </candidate\_solution\> Please refer to this solution and provide your solution\.

Crucially, we inform the model that the candidate’scorrectness is unknown, avoiding explicit labeling that would leak reward information and undermine the RL objective\.

This construction transforms a potentially failed response into a structured learning signal: the model learns torecognize and fix its own mistakes\(whenycandy\_\{\\text\{cand\}\}is incorrect\) orreinforce correct reasoning patterns\(whenycandy\_\{\\text\{cand\}\}is correct\), rather than simply avoiding failures through likelihood suppression\.

## Appendix BHyperparameters

Table[6](https://arxiv.org/html/2605.14539#A2.T6)details the full hyperparameters configuration for CIPO\.

Table 6:Hyperparameter configuration of CIPO\.