Internalizing Outcome Supervision into Process Supervision: A New Paradigm for Reinforcement Learning for Reasoning

# Internalizing Outcome Supervision into Process Supervision: A New Paradigm for Reasoning Reinforcement Learning
Source: [https://arxiv.org/html/2605.05226](https://arxiv.org/html/2605.05226)
Fei Ding1Yongkang Zhang1Runhao Liu1Yuhao Liao2Zijian Zeng2Sibo Wang2Huiming Yang21Alibaba Group2Tsinghua University

###### Abstract

A central challenge in reasoning reinforcement learning lies not merely in the sparsity of outcome supervision, but in converting end\-of\-sequence outcome feedback into fine\-grained learning signals that can act on intermediate reasoning steps\. Existing approaches either perform sequence\-level optimization with outcome rewards, which precludes precise credit assignment, or rely on externally constructed process supervision at substantial cost and limited scalability\. We propose a new perspective: reasoning reinforcement learning as the problem of*internalizing outcome supervision into process supervision*\. Building on this perspective, we introduce IOP, a framework that enables the model to automatically distill process\-level learning signals by identifying, repairing, and reusing failed reasoning trajectories—achieving finer\-grained policy optimization under outcome supervision alone\. We further formalize this idea as a training paradigm in which the model continuously generates and refines internal process supervision throughout reinforcement learning, offering a new pathway for fine\-grained credit assignment that is distinct from exogenous process supervision\.

\(a\) Outcome Supervisionz1z\_\{1\}z2z\_\{2\}z3z\_\{3\}z4z\_\{4\}z5z\_\{5\}z6z\_\{6\}z7z\_\{7\}×\\boldsymbol\{\\times\}∇−\\nabla^\{\\\!\-\}∇−\\nabla^\{\\\!\-\}∇−\\nabla^\{\\\!\-\}∇−\\nabla^\{\\\!\-\}∇−\\nabla^\{\\\!\-\}∇−\\nabla^\{\\\!\-\}∇−\\nabla^\{\\\!\-\}Uniform penalty onalltokensInternalize\(b\) IOP: Internalized Process SupervisionFailedy1y\_\{1\}y2y\_\{2\}y3y\_\{3\}y4y\_\{4\}y5y\_\{5\}y6y\_\{6\}y7y\_\{7\}×\\boldsymbol\{\\times\}Repairedy1\{y\}\_\{1\}y2\{y\}\_\{2\}y3\{y\}\_\{3\}y~4\\tilde\{y\}\_\{4\}y~5\\tilde\{y\}\_\{5\}y6\{y\}\_\{6\}y~7\\tilde\{y\}\_\{7\}✓\\boldsymbol\{\\checkmark\}Gradient–––∇\\nabla∇\\nabla–∇\\nablaTargeted gradientonlyat error tokens

\+6\.9%avg accuracy⋅\\cdot2\.3×\\timessample efficiency⋅\\cdotNo external annotations

Figure 1:IOP turns outcome feedback into process\-level supervision by fixing failed trajectories and penalizing only the erroneous tokens\.## 1Introduction

Recent advances in large language model \(LLM\) reasoning—spanning mathematics, code generation, and complex logical tasks—have been substantially driven by reinforcement learning \(RL\)\. Outcome\-based RL has proven effective for improving performance on challenging reasoning benchmarks\(Havrilla et al\.,[2024](https://arxiv.org/html/2605.05226#bib.bib4); Shao et al\.,[2024](https://arxiv.org/html/2605.05226#bib.bib22); DeepSeek\-AI et al\.,[2025](https://arxiv.org/html/2605.05226#bib.bib3); OpenAI,[2024](https://arxiv.org/html/2605.05226#bib.bib18)\), with methods such as RLVR and GRPO directly optimizing final\-answer correctness through verifiable outcome rewards\(Shao et al\.,[2024](https://arxiv.org/html/2605.05226#bib.bib22); Wen et al\.,[2025](https://arxiv.org/html/2605.05226#bib.bib25)\)\. However, as reasoning chains grow longer, the limitations of outcome\-level supervision become increasingly apparent: a failed trajectory is rarely wrong in its entirety—it typically contains many correct intermediate steps alongside a few critical errors\. Uniformly rewarding or penalizing the entire trajectory based solely on the final outcome conflates correct local reasoning with the actual sources of failure\. The fundamental difficulty in reasoning RL, therefore, is not merely that outcome supervision is sparse, but that it lacks the granularity required for effective credit assignment to intermediate reasoning steps\.

To address this, researchers have introduced process supervision that explicitly models intermediate reasoning steps\. Seminal work demonstrated that step\-by\-step verification substantially improves complex mathematical reasoning compared to outcome supervision alone\(Lightman et al\.,[2023](https://arxiv.org/html/2605.05226#bib.bib10)\)\. Subsequent efforts explored automated construction of process supervision—including automatic intermediate\-step sampling, training of process reward models \(PRMs\), and search\- or verifier\-based identification of critical error steps\(Luo et al\.,[2024](https://arxiv.org/html/2605.05226#bib.bib12); Zhang et al\.,[2025](https://arxiv.org/html/2605.05226#bib.bib29); Yang et al\.,[2025](https://arxiv.org/html/2605.05226#bib.bib27)\)\. More recent work further integrates process signals directly into RL through error\-region penalties, prefix\-level supervision, or localized process optimization\(Liu et al\.,[2026](https://arxiv.org/html/2605.05226#bib.bib11); Liang et al\.,[2026](https://arxiv.org/html/2605.05226#bib.bib9); Nie et al\.,[2026](https://arxiv.org/html/2605.05226#bib.bib16)\)\. Collectively, these studies establish that process supervision is essential for effective credit assignment in long\-chain reasoning\.

Yet existing process supervision methods predominantly rely on externally provided signals—whether human step annotations, separately trained PRMs, explicitly constructed verifiers, or search\-derived pseudo\-labels—all of which constitute*exogenous*process supervision\. While effective, such methods incur substantial construction and maintenance costs and are difficult to scale as the policy model improves\. In other words, prior work has established*that*process supervision matters, but has not adequately addressed a more fundamental question:*can the model generate its own process supervision during reinforcement learning, given only outcome\-level feedback?*

This paper departs from precisely this question and proposes a new perspective: reasoning RL as the problem of*internalizing outcome supervision into process supervision*\. The key observation is that failed trajectories are not uniformly negative samples but contain structured information that can be corrected and reused\. If the model can generate a repaired version of a failed trajectory and distill critical changes from the difference between the two, outcome supervision can be converted into process supervision that acts on intermediate steps\. This perspective resonates with recent work on self\-correction and reflective reasoning, which shows that models can perform self\-verification and correction during inference, and that this capability can be strengthened via online RL\(Kumar et al\.,[2024](https://arxiv.org/html/2605.05226#bib.bib7); Ma et al\.,[2025](https://arxiv.org/html/2605.05226#bib.bib13); Xiong et al\.,[2025](https://arxiv.org/html/2605.05226#bib.bib26); Lee et al\.,[2025](https://arxiv.org/html/2605.05226#bib.bib8)\)\. We extend this self\-correction capability from inference time to training time, thereby proposing a new training paradigm: under outcome supervision alone, the model continuously generates and refines internal process supervision to achieve finer\-grained policy optimization\.

##### Contributions\.

- •We reframe reasoning RL as the problem of internalizing outcome supervision into process supervision, and establish a corresponding training paradigm in which the model automatically distills process\-level signals through failure repair during RL\.
- •We propose the IOP framework and its GSPO instantiation IOP\-GSPO, which converts sequence\-level rewards into token\-level gating signals via audit gating, minimum\-edit repair, and verification\-based adaptive truncation\.
- •Experiments demonstrate a synergistic reinforcement loop between policy and repair capabilities; IOP\-GSPO consistently outperforms GSPO \(\+4\.9–6\.9%\) and exogenous process supervision methods across three reasoning benchmarks, with approximately2\.3×2\.3\\timessample efficiency\.

## 2Related Work

##### Outcome\-supervised reasoning RL\.

Mainstream open\-source reasoning RL methods rely on outcome supervision for sequence\-level optimization, including GRPO\(Shao et al\.,[2024](https://arxiv.org/html/2605.05226#bib.bib22)\), GSPO\(Zheng et al\.,[2025a](https://arxiv.org/html/2605.05226#bib.bib30)\), DPO\(Rafailov et al\.,[2023](https://arxiv.org/html/2605.05226#bib.bib21)\), and the broader RLVR line\(Havrilla et al\.,[2024](https://arxiv.org/html/2605.05226#bib.bib4); Wen et al\.,[2025](https://arxiv.org/html/2605.05226#bib.bib25); DeepSeek\-AI et al\.,[2025](https://arxiv.org/html/2605.05226#bib.bib3); OpenAI,[2024](https://arxiv.org/html/2605.05226#bib.bib18)\)\. These methods achieve strong results on math, code, and complex reasoning tasks, yet supervision operates on entire trajectories, leaving local errors in long\-chain reasoning unattributable to specific intermediate steps\.

##### Explicit process supervision and process reward models\.

A complementary line of work provides explicit process supervision through intermediate\-step verification, scoring, or training of process reward models \(PRMs\)\.Lightman et al\. \([2023](https://arxiv.org/html/2605.05226#bib.bib10)\)demonstrated the value of step\-level supervision for complex mathematical reasoning;Wang et al\. \([2023](https://arxiv.org/html/2605.05226#bib.bib24)\); Luo et al\. \([2024](https://arxiv.org/html/2605.05226#bib.bib12)\)further explored automated process label construction, PRM training, and scaling of process data via automated supervision\. Recent studies note that reflection and self\-correction complicate first\-error localization\(Zhang et al\.,[2025](https://arxiv.org/html/2605.05226#bib.bib29); Yang et al\.,[2025](https://arxiv.org/html/2605.05226#bib.bib27)\), and generative verifiers with data\-efficient process supervision modeling have gained attention\(Khalifa et al\.,[2025](https://arxiv.org/html/2605.05226#bib.bib6)\)\. This line establishes the importance of process supervision for fine\-grained attribution, but the signal sources remain limited to human annotations, external PRMs, or static labels\.

##### Process\-supervised RL and fine\-grained attribution\.

More recent work integrates process supervision directly into RL rather than using it solely as an independent verifier—for example, through prefix separation, error\-region penalties, or attention\-guided updates to improve policy update precision\(Liu et al\.,[2026](https://arxiv.org/html/2605.05226#bib.bib11); Yao et al\.,[2026](https://arxiv.org/html/2605.05226#bib.bib28); Pronesti et al\.,[2026](https://arxiv.org/html/2605.05226#bib.bib20); Liang et al\.,[2026](https://arxiv.org/html/2605.05226#bib.bib9); Nie et al\.,[2026](https://arxiv.org/html/2605.05226#bib.bib16)\)\. These methods demonstrate that process\-level signals can directly improve local update quality, yet they still rely on explicit error boundaries, step labels, or independent PRMs—the signal source remains fundamentally exogenous\.

##### Self\-correction, self\-verification, and repair\-based reasoning\.

Another body of work investigates whether models can improve reasoning through correction and verification, including Self\-Refine\(Madaan et al\.,[2023](https://arxiv.org/html/2605.05226#bib.bib14)\), Reflexion\(Shinn et al\.,[2023](https://arxiv.org/html/2605.05226#bib.bib23)\), SCoRe\(Kumar et al\.,[2024](https://arxiv.org/html/2605.05226#bib.bib7)\), S2R\(Ma et al\.,[2025](https://arxiv.org/html/2605.05226#bib.bib13)\), and self\-rewarding correction\(Xiong et al\.,[2025](https://arxiv.org/html/2605.05226#bib.bib26)\)\. These methods show that models can generate valuable internal feedback to improve output quality\. However, they primarily target inference\-time correction and do not systematically address how to organize correction differences as training\-time process supervision—IOP fills precisely this gap\.

##### Relationship to this work\.

The above lines respectively establish the effectiveness of outcome\-supervised RL, exogenous process supervision, and self\-correction\. This paper addresses an orthogonal dimension: organizing the model’s own correction capability into continuously improving internal process supervision during RL training, enabling fine\-grained credit assignment without external step annotations or independent PRMs\. IOP is complementary to any of the above lines—for instance, an exogenous PRM could initialize the repair reference or filter low\-quality repairs\.

## 3Method

\(a\) Existing: Outcome Supervisionz1z\_\{1\}z2z\_\{2\}z3z\_\{3\}z4z\_\{4\}z5z\_\{5\}z6z\_\{6\}z7z\_\{7\}z8z\_\{8\}r=0r\\\!=\\\!0Uniform negative gradient∇−\\nabla^\{\-\}cannot distinguish correct from incorrect tokensInternalizesequence\-level→\\totoken\-level\(b\) New Perspective: Internalize Outcome into Process SupervisionFailedyyy1y\_\{1\}y2y\_\{2\}y3y\_\{3\}y4y\_\{4\}y5y\_\{5\}y6y\_\{6\}y7y\_\{7\}y8y\_\{8\}r=0r\\\!=\\\!0Repairy~\\tilde\{y\}y1\{y\}\_\{1\}y2\{y\}\_\{2\}y3\{y\}\_\{3\}y~4\\tilde\{y\}\_\{4\}y~5\\tilde\{y\}\_\{5\}y6\{y\}\_\{6\}y7\{y\}\_\{7\}y~8\\tilde\{y\}\_\{8\}r=1r\\\!=\\\!1AlignGategtg\_\{t\}00011001Gradients only at diff positionsinternalized token\-level process supervision\(c\)NewPolicy Modeπθ\\pi\_\{\\theta\}Split Cor/Err𝒢cor/𝒢err\\mathcal\{G\}\_\{\\text\{cor\}\}/\\mathcal\{G\}\_\{\\text\{err\}\}Repair Modeρθ\\rho\_\{\\theta\}Audit Gateℳaudit\\mathcal\{M\}\_\{\\text\{audit\}\}ℬrep\\mathcal\{B\}\_\{\\text\{rep\}\}Align\+Truncgtg\_\{t\}ℬpol\\mathcal\{B\}\_\{\\text\{pol\}\}Joint Updateθ\\theta𝒥IOP=𝒥GSPO\+λ​𝒥rep\\mathcal\{J\}\_\{\\text\{IOP\}\}\\\!=\\\!\\mathcal\{J\}\_\{\\text\{GSPO\}\}\\\!\+\\\!\\lambda\\mathcal\{J\}\_\{\\text\{rep\}\}G′G^\{\\prime\}y\+ay\{\+\}a\(y,y~\)\(y,\\tilde\{y\}\)gtg\_\{t\}Synergistic reinforcement: Policy↑\\uparrow→\\rightarrowReference quality↑\\uparrow→\\rightarrowRepair precision↑\\uparrow→\\rightarrowGating focus↑\\uparrow→\\rightarrowPolicy↑\\uparrow

Figure 2:Internalizing outcome supervision into process supervision\. Existing paradigm→\\toNew perspective→\\toNew paradigm\.##### Overview\.

We refer to the proposed method asIOP\(*Internalizing Outcome Supervision into Process Supervision*\), which comprises three core steps: \(i\) selecting repairable failed trajectories, \(ii\) generating internal process supervision through a repair mode, and \(iii\) feeding process signals back into policy updates via truncation gating\. The gating mechanism is decoupled from the outer optimizer and can be embedded into any group\-comparison algorithm such as GRPO, GSPO, or RLOO; we instantiate IOP with GSPO\(Zheng et al\.,[2025a](https://arxiv.org/html/2605.05226#bib.bib30)\)as the outer optimizer, yieldingIOP\-GSPO\.

A single model with shared parametersθ\\thetaserves two roles: apolicy modeπθ​\(𝐲∣x\)\\pi\_\{\\theta\}\(\\mathbf\{y\}\\mid x\)that generates reasoning trajectories𝐲=\(y1,…,yT\)\\mathbf\{y\}=\(y\_\{1\},\\ldots,y\_\{T\}\)and receives binary scoresr​\(x,𝐲\)∈\{0,1\}r\(x,\\mathbf\{y\}\)\\\!\\in\\\!\\\{0,1\\\}; and arepair modeρθ​\(𝐲~∣x,𝐲,a\)\\rho\_\{\\theta\}\(\\tilde\{\\mathbf\{y\}\}\\mid x,\\mathbf\{y\},a\)that produces a repair𝐲~\\tilde\{\\mathbf\{y\}\}conditioned on a failed trajectory𝐲\\mathbf\{y\}and a referenceaa\. Binary rewards lack discrimination at the sequence level, but the repair–alignment pipeline converts them into token\-level difference signals\. Parameter sharing avoids extra model overhead and ensures that repair capability improves in tandem with the policy: a stronger policy produces more concentrated errors, yielding more precise repairs and more focused gating\.

### 3\.1IOP\-GSPO: Cold Start and Joint Optimization

IOP proceeds in two stages: Stage 1 endows the model with basic repair capability through cold\-start SFT, and Stage 2 continuously converts failed trajectories into process supervision and feeds it back to the policy via joint RL\.

#### 3\.1\.1Stage 1: Acquiring Repair Capability via SFT

We first construct a cold\-start repair dataset of 500 examples

𝒟rep0=\{\(x,𝐲,a,𝐲∗\)\}\\mathcal\{D\}\_\{\\text\{rep\}\}^\{0\}=\\\{\(x,\\mathbf\{y\},a,\\mathbf\{y\}^\{\*\}\)\\\}and fine\-tune the repair modeρθ\\rho\_\{\\theta\}with SFT\. The input is the concatenation of a repair instruction and\(x,𝐲,a\)\(x,\\mathbf\{y\},a\)\(prompt: “Please correct the following erroneous reasoning with reference to the correct trajectory”\), and the output is𝐲∗\\mathbf\{y\}^\{\*\}:

ℒSFT​\(θ\)=−𝔼\(x,𝐲,a,𝐲∗\)∼𝒟rep0​\[∑t=1T∗log⁡ρθ​\(yt∗∣x,𝐲,a,𝐲<t∗\)\]\.\\mathcal\{L\}\_\{\\text\{SFT\}\}\(\\theta\)=\-\\mathbb\{E\}\_\{\(x,\\mathbf\{y\},a,\\mathbf\{y\}^\{\*\}\)\\sim\\mathcal\{D\}\_\{\\text\{rep\}\}^\{0\}\}\\Bigg\[\\sum\_\{t=1\}^\{T^\{\*\}\}\\log\\rho\_\{\\theta\}\(y\_\{t\}^\{\*\}\\mid x,\\mathbf\{y\},a,\\mathbf\{y\}^\{\*\}\_\{<t\}\)\\Bigg\]\.\(1\)The dataset is constructed by sampling from the base model, filtering incorrect–correct pairs, and manually writing repair trajectories\. 500 examples suffice to learn the local correction format; subsequent RL continuously improves repair quality and coverage \(38\.4%→\\to73\.1%, Table[3](https://arxiv.org/html/2605.05226#A1.T3)\)\.

#### 3\.1\.2Stage 2: Joint RL Optimization of Policy and Repair

Each step executes: sampling→\\tocorrect/incorrect partition→\\torepair→\\todual\-batch construction→\\tojoint update\. Failed trajectories are simultaneously written toℬrep\\mathcal\{B\}\_\{\\text\{rep\}\}\(for repair capability\) andℬpol\\mathcal\{B\}\_\{\\text\{pol\}\}\(for localized policy updates\), so that both capabilities are jointly updated at every step\.

##### Sampling the initial group\.

For each promptxx, we sampleG′G^\{\\prime\}candidate trajectories from the old policyπθold\\pi\_\{\\theta\_\{\\text\{old\}\}\}:

𝒢′\(x\)=\{yi\}i=1G′,yi∼πθold\(⋅∣x\),\\mathcal\{G\}^\{\\prime\}\(x\)=\\\{y\_\{i\}\\\}\_\{i=1\}^\{G^\{\\prime\}\},\\qquad y\_\{i\}\\sim\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\cdot\\mid x\),\(2\)

##### Correct/incorrect partition\.

Trajectories are partitioned by reward thresholdτr\\tau\_\{r\}\(with binary rewards,τr=0\.5\\tau\_\{r\}\\\!=\\\!0\.5\):

𝒢cor​\(x\)=\{y∈𝒢′​\(x\)∣r​\(x,y\)≥τr\},𝒢err​\(x\)=\{y∈𝒢′​\(x\)∣r​\(x,y\)<τr\}\.\\mathcal\{G\}\_\{\\text\{cor\}\}\(x\)=\\\{y\\in\\mathcal\{G\}^\{\\prime\}\(x\)\\mid r\(x,y\)\\geq\\tau\_\{r\}\\\},\\qquad\\mathcal\{G\}\_\{\\text\{err\}\}\(x\)=\\\{y\\in\\mathcal\{G\}^\{\\prime\}\(x\)\\mid r\(x,y\)<\\tau\_\{r\}\\\}\.\(3\)

##### Reference trajectory\.

If the error set is empty, the prompt is skipped\. If the correct set is empty, the prompt is deferred until the model’s capability improves\. When both sets are non\-empty, a reference is sampled uniformlya∼Unif​\(𝒢cor​\(x\)\)a\\sim\\mathrm\{Unif\}\(\\mathcal\{G\}\_\{\\text\{cor\}\}\(x\)\)to avoid bias toward any particular solution\.

##### Design choice: correct samples serve only as reference anchors\.

𝒢cor​\(x\)\\mathcal\{G\}\_\{\\text\{cor\}\}\(x\)provides only the reference anchoraaand does not participate in policy gradient computation\. Including correct samples concentrates probability mass on a few trajectories and induces entropy collapse \(HMMT25 drops 6\.1%, Table[2](https://arxiv.org/html/2605.05226#S4.T2)\)\.

##### Repairing failed trajectories\.

For eachyi∈𝒢err​\(x\)y\_\{i\}\\\!\\in\\\!\\mathcal\{G\}\_\{\\text\{err\}\}\(x\),GrepG\_\{\\text\{rep\}\}candidate repairs are generated usingaaas the anchor:

𝒢~rep\(x,yi,a\)=\{y~i\(j\)\}j=1Grep,y~i\(j\)∼ρθ\(⋅∣x,yi,a\),\\widetilde\{\\mathcal\{G\}\}\_\{\\text\{rep\}\}\(x,y\_\{i\},a\)=\\\{\\tilde\{y\}\_\{i\}^\{\(j\)\}\\\}\_\{j=1\}^\{G\_\{\\text\{rep\}\}\},\\qquad\\tilde\{y\}\_\{i\}^\{\(j\)\}\\sim\\rho\_\{\\theta\}\(\\cdot\\mid x,y\_\{i\},a\),\(4\)wherertask​\(x,y~\)∈\{0,1\}r\_\{\\text\{task\}\}\(x,\\tilde\{y\}\)\\\!\\in\\\!\\\{0,1\\\}denotes the correctness judgment\.

##### Reward hacking audit gate\.

Each candidate repair is subject to an external audit gatehi\(j\)∈\{0,1\}h\_\{i\}^\{\(j\)\}\\\!\\in\\\!\\\{0,1\\\}: the original failed trajectoryyiy\_\{i\}, candidate repairy~i\(j\)\\tilde\{y\}\_\{i\}^\{\(j\)\}, and reference trajectoryaaare submitted to an independent audit modelℳaudit\\mathcal\{M\}\_\{\\text\{audit\}\}\(we use OpenAI’s open\-source gpt\-oss\-120b\(OpenAI,[2025](https://arxiv.org/html/2605.05226#bib.bib19)\)\), avoiding circular reasoning from parameter sharing:

hi\(j\)=ℳaudit​\(x,yi,y~i\(j\),a\)∈\{0,1\},h\_\{i\}^\{\(j\)\}=\\mathcal\{M\}\_\{\\text\{audit\}\}\(x,y\_\{i\},\\tilde\{y\}\_\{i\}^\{\(j\)\},a\)\\in\\\{0,1\\\},\(5\)wherehi\(j\)=1h\_\{i\}^\{\(j\)\}\\\!=\\\!1indicates passing the audit \(no reward hacking\) andhi\(j\)=0h\_\{i\}^\{\(j\)\}\\\!=\\\!0indicates detected reward hacking\.

Candidates are jointly scored by the audit gate, correctness, and normalized edit distanceΔ¯edit\\bar\{\\Delta\}\_\{\\text\{edit\}\}, favoring repairs that pass the audit, are correct, and involve minimal edits:

srep​\(x,yi,a,y~i\(j\)\)=hi\(j\)⋅\(rtask​\(x,y~i\(j\)\)−λedit​Δ¯edit​\(yi,y~i\(j\)\)\)\.s\_\{\\text\{rep\}\}\(x,y\_\{i\},a,\\tilde\{y\}\_\{i\}^\{\(j\)\}\)=h\_\{i\}^\{\(j\)\}\\cdot\\Big\(r\_\{\\text\{task\}\}\(x,\\tilde\{y\}\_\{i\}^\{\(j\)\}\)\-\\lambda\_\{\\text\{edit\}\}\\,\\bar\{\\Delta\}\_\{\\text\{edit\}\}\(y\_\{i\},\\tilde\{y\}\_\{i\}^\{\(j\)\}\)\\Big\)\.\(6\)

##### Selecting the best repair\.

The highest\-scoring candidate is selected \(ties broken in favor ofrtask=1r\_\{\\text\{task\}\}\\\!=\\\!1\):

y~i=arg⁡maxy~i\(j\)∈𝒢~rep​\(x,yi,a\)⁡srep​\(x,yi,a,y~i\(j\)\)\.\\tilde\{y\}\_\{i\}=\\arg\\max\_\{\\tilde\{y\}\_\{i\}^\{\(j\)\}\\in\\widetilde\{\\mathcal\{G\}\}\_\{\\text\{rep\}\}\(x,y\_\{i\},a\)\}s\_\{\\text\{rep\}\}\(x,y\_\{i\},a,\\tilde\{y\}\_\{i\}^\{\(j\)\}\)\.\(7\)If the best repair hasrtask=0r\_\{\\text\{task\}\}\\\!=\\\!0, the prompt is deferred; otherwise, the repair candidate group is written toℬrep\\mathcal\{B\}\_\{\\text\{rep\}\}, and the best repair paired with the original error forms a two\-element tuple written toℬpol\\mathcal\{B\}\_\{\\text\{pol\}\}\. The pairwise structure \(\|𝒢i\|=2\|\\mathcal\{G\}\_\{i\}\|\\\!=\\\!2\) maximizes the discriminability of difference masks: token\-level comparison directly localizes the key positions that change the answer:

𝒢i​\(x\)=\{yi,y~i\},ℬpol​\(x\)=\{𝒢i​\(x\)∣yi∈𝒢err​\(x\)\},\|𝒢i​\(x\)\|=2\.\\mathcal\{G\}\_\{i\}\(x\)=\\\{y\_\{i\},\\tilde\{y\}\_\{i\}\\\},\\quad\\mathcal\{B\}\_\{\\text\{pol\}\}\(x\)=\\\{\\mathcal\{G\}\_\{i\}\(x\)\\mid y\_\{i\}\\in\\mathcal\{G\}\_\{\\text\{err\}\}\(x\)\\\},\\quad\|\\mathcal\{G\}\_\{i\}\(x\)\|=2\.\(8\)The corresponding repair mini\-batch is

ℬrep​\(x\)=\{\(x,yi,a,𝒢~rep​\(x,yi,a\)\)∣yi∈𝒢err​\(x\)\}\.\\mathcal\{B\}\_\{\\text\{rep\}\}\(x\)=\\\{\(x,y\_\{i\},a,\\widetilde\{\\mathcal\{G\}\}\_\{\\text\{rep\}\}\(x,y\_\{i\},a\)\)\\mid y\_\{i\}\\in\\mathcal\{G\}\_\{\\text\{err\}\}\(x\)\\\}\.

#### 3\.1\.3Process Supervision Signal Generation: Alignment and Truncation

The alignment operator𝒜\\mathcal\{A\}computes bilateral difference masks\(𝐦,𝐦′\)=𝒜​\(𝐲,𝐲~\)\(\\mathbf\{m\},\\mathbf\{m\}^\{\\prime\}\)=\\mathcal\{A\}\(\\mathbf\{y\},\\tilde\{\\mathbf\{y\}\}\)offline along the token\-level Levenshtein path, where𝐦∈\{0,1\}T\\mathbf\{m\}\\\!\\in\\\!\\\{0,1\\\}^\{T\}and𝐦′∈\{0,1\}T~\\mathbf\{m\}^\{\\prime\}\\\!\\in\\\!\\\{0,1\\\}^\{\\tilde\{T\}\}mark substitution, deletion, or insertion positions, respectively\.

##### Truncated difference mask\.

Early errors induce cascade drift—a single local error triggers global token\-level divergence downstream—and full\-sequence updates propagate negative feedback to unrelated positions\. To mitigate this, we retain only the earliestKKdifference positions and construct atruncated mask:

mt\(K\)=\{1,t∈𝒮K​\(𝐲,𝐲~\),0,otherwise,m\_\{t\}^\{\(K\)\}=\\begin\{cases\}1,&t\\in\\mathcal\{S\}\_\{K\}\(\\mathbf\{y\},\\tilde\{\\mathbf\{y\}\}\),\\\\ 0,&\\text\{otherwise\},\\end\{cases\}\(9\)withmt′⁣\(K\)m\_\{t\}^\{\\prime\(K\)\}defined analogously for the repair side\.𝒮K\\mathcal\{S\}\_\{K\}denotes the firstKKdifference tokens selected in positional order\.

##### Verification\-based adaptive truncation\.

A fixedKKmay be insufficient to complete the repair\. We therefore construct agrafted trajectory: the tokens at positions𝒮K\\mathcal\{S\}\_\{K\}in𝐲\\mathbf\{y\}are replaced with the corresponding tokens from𝐲~\\tilde\{\\mathbf\{y\}\}, and generation continues fromtKmax=max⁡\(𝒮K\)t\_\{K\}^\{\\max\}\\\!=\\\!\\max\(\\mathcal\{S\}\_\{K\}\):

y^t\(K\)=\{y~t,t∈𝒮K,yt,t≤tKmax,t∉𝒮K,∼πθ\(⋅∣x,y^<t\(K\)\),t\>tKmax\.\\hat\{y\}\_\{t\}^\{\(K\)\}=\\begin\{cases\}\\tilde\{y\}\_\{t\},&t\\in\\mathcal\{S\}\_\{K\},\\\\ y\_\{t\},&t\\leq t\_\{K\}^\{\\max\},\\;t\\notin\\mathcal\{S\}\_\{K\},\\\\ \\sim\\pi\_\{\\theta\}\(\\cdot\\mid x,\\hat\{y\}\_\{<t\}^\{\(K\)\}\),&t\>t\_\{K\}^\{\\max\}\.\\end\{cases\}\(10\)Ifrtask​\(x,𝐲^\(K\)\)=1r\_\{\\text\{task\}\}\(x,\\hat\{\\mathbf\{y\}\}^\{\(K\)\}\)\\\!=\\\!1, theKKedits are verified as sufficient; otherwise the window is expanded to2​K2K; if this also fails, the full mask is used:

K∗=\{K,rtask​\(x,𝐲^\(K\)\)=1,2​K,rtask​\(x,𝐲^\(K\)\)=0∧rtask​\(x,𝐲^\(2​K\)\)=1,\|𝒮∞\|,otherwise\.K^\{\*\}=\\begin\{cases\}K,&r\_\{\\text\{task\}\}\(x,\\hat\{\\mathbf\{y\}\}^\{\(K\)\}\)=1,\\\\ 2K,&r\_\{\\text\{task\}\}\(x,\\hat\{\\mathbf\{y\}\}^\{\(K\)\}\)=0\\;\\wedge\\;r\_\{\\text\{task\}\}\(x,\\hat\{\\mathbf\{y\}\}^\{\(2K\)\}\)=1,\\\\ \|\\mathcal\{S\}\_\{\\infty\}\|,&\\text\{otherwise\}\.\\end\{cases\}\(11\)Empirically, approximately 72% of samples pass atKK, 19% fall back to2​K2K, and 9% use the full mask \(Table[2](https://arxiv.org/html/2605.05226#S4.T2)\)\.

The resulting token\-level gating is:

gt∈\[0,1\],gt=mt\(K∗\)orgt=mt′⁣\(K∗\),g\_\{t\}\\in\[0,1\],\\qquad g\_\{t\}=m\_\{t\}^\{\(K^\{\*\}\)\}\\ \\ \\text\{or\}\\ \\ g\_\{t\}=m\_\{t\}^\{\\prime\(K^\{\*\}\)\},\(12\)

### 3\.2Gated Policy Optimization and Joint Training

For each𝒢i​\(x\)\\mathcal\{G\}\_\{i\}\(x\)inℬpol​\(x\)\\mathcal\{B\}\_\{\\text\{pol\}\}\(x\), we write

𝒢i​\(x\)=\{yi,y~i\},\\mathcal\{G\}\_\{i\}\(x\)=\\\{y\_\{i\},\\tilde\{y\}\_\{i\}\\\},and letz∈𝒢i​\(x\)z\\in\\mathcal\{G\}\_\{i\}\(x\)denote either trajectory in the pair, with the sequence\-level advantageA^i​\(z\)\\widehat\{A\}\_\{i\}\(z\)computed via within\-pair normalization\.

##### Bilateral difference gating\.

For the pair\{yi,y~i\}\\\{y\_\{i\},\\tilde\{y\}\_\{i\}\\\}, the truncated alignment operator𝒜K\\mathcal\{A\}\_\{K\}simultaneously produces error\-side and repair\-side gating:

\(mi,1:\|yi\|\(K\),mi,1:\|y~i\|′⁣\(K\)\)=𝒜K​\(yi,y~i\),\\big\(m\_\{i,1:\|y\_\{i\}\|\}^\{\(K\)\},\\ m\_\{i,1:\|\\tilde\{y\}\_\{i\}\|\}^\{\\prime\(K\)\}\\big\)=\\mathcal\{A\}\_\{K\}\(y\_\{i\},\\tilde\{y\}\_\{i\}\),\(13\)with the corresponding token\-level gating

gz,t​\(x\)=\{mi,t\(K\),z=yi,mi,t′⁣\(K\),z=y~i\.g\_\{z,t\}\(x\)=\\begin\{cases\}m\_\{i,t\}^\{\(K\)\},&z=y\_\{i\},\\\\\[3\.0pt\] m\_\{i,t\}^\{\\prime\(K\)\},&z=\\tilde\{y\}\_\{i\}\.\\end\{cases\}\(14\)
GSPO uses a sequence\-level likelihood ratio; IOP introduces gating on top, restricting the sequence\-level ratio to only those tokens at difference positions:

wzgate​\(θ\)=exp⁡\(1∑t′=1Tzgz,t′​∑t=1Tzgz,t​log⁡πθ​\(zt∣x,z<t\)πθold​\(zt∣x,z<t\)\)\.w\_\{z\}^\{\\text\{gate\}\}\(\\theta\)=\\exp\\\!\\Bigg\(\\frac\{1\}\{\\sum\_\{t^\{\\prime\}=1\}^\{T\_\{z\}\}g\_\{z,t^\{\\prime\}\}\}\\sum\_\{t=1\}^\{T\_\{z\}\}g\_\{z,t\}\\,\\log\\frac\{\\pi\_\{\\theta\}\(z\_\{t\}\\mid x,z\_\{<t\}\)\}\{\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(z\_\{t\}\\mid x,z\_\{<t\}\)\}\\Bigg\)\.\(15\)
Following the token\-level importance ratio of GSPO\-token\(Zheng et al\.,[2025a](https://arxiv.org/html/2605.05226#bib.bib30)\):

sz,t​\(θ\)=sg​\[wzgate​\(θ\)\]⋅πθ​\(zt∣x,z<t\)sg​\[πθ​\(zt∣x,z<t\)\],s\_\{z,t\}\(\\theta\)=\\mathrm\{sg\}\\\!\\big\[w\_\{z\}^\{\\text\{gate\}\}\(\\theta\)\\big\]\\cdot\\frac\{\\pi\_\{\\theta\}\(z\_\{t\}\\mid x,z\_\{<t\}\)\}\{\\mathrm\{sg\}\\\!\\big\[\\pi\_\{\\theta\}\(z\_\{t\}\\mid x,z\_\{<t\}\)\\big\]\},\(16\)wheresg​\[⋅\]\\mathrm\{sg\}\[\\cdot\]denotes the stop\-gradient operator that takes the numerical value while blocking gradient flow\.sz,ts\_\{z,t\}is numerically equal towzgatew\_\{z\}^\{\\text\{gate\}\}, but the gradient propagates only through the log\-probability at positiontt\. The gated token\-level clipped objective is averaged over active positions:

ℒzclip=1∑t′=1Tzgz,t′​∑t=1Tzgz,t​min⁡\(sz,t​\(θ\)​A^i​\(z\),clip​\(sz,t​\(θ\),1−ϵ,1\+ϵ\)​A^i​\(z\)\)\.\\mathcal\{L\}\_\{z\}^\{\\text\{clip\}\}=\\frac\{1\}\{\\sum\_\{t^\{\\prime\}=1\}^\{T\_\{z\}\}g\_\{z,t^\{\\prime\}\}\}\\sum\_\{t=1\}^\{T\_\{z\}\}g\_\{z,t\}\\,\\min\\Big\(s\_\{z,t\}\(\\theta\)\\,\\widehat\{A\}\_\{i\}\(z\),\\;\\mathrm\{clip\}\\big\(s\_\{z,t\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\\big\)\\,\\widehat\{A\}\_\{i\}\(z\)\\Big\)\.\(17\)Dividing by∑t′gz,t′\\sum\_\{t^\{\\prime\}\}g\_\{z,t^\{\\prime\}\}\(the actual number of active tokens\) ensures that the gradient scale is invariant to mask sparsity, serving the same role as the1/\|yi\|1/\|y\_\{i\}\|normalization in GSPO\. The IOP\-GSPO policy objective is

𝒥IOP\-GSPO​\(θ;ℬpol\)\\displaystyle\\mathcal\{J\}\_\{\\text\{IOP\-GSPO\}\}\(\\theta;\\mathcal\{B\}\_\{\\text\{pol\}\}\)=𝔼x∼𝒟,𝒢i​\(x\)∼ℬpol​\(x\)​\[12​∑z∈𝒢i​\(x\)ℒzclip\]−βkl​𝒦​\(θ\),\\displaystyle=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\},\\,\\mathcal\{G\}\_\{i\}\(x\)\\sim\\mathcal\{B\}\_\{\\text\{pol\}\}\(x\)\}\\Bigg\[\\frac\{1\}\{2\}\\sum\_\{z\\in\\mathcal\{G\}\_\{i\}\(x\)\}\\mathcal\{L\}\_\{z\}^\{\\text\{clip\}\}\\Bigg\]\-\\beta\_\{\\text\{kl\}\}\\,\\mathcal\{K\}\(\\theta\),\(18\)

##### Repair mini\-batch objective\.

For each\(x,yi,a,𝒢~rep\)\(x,y\_\{i\},a,\\widetilde\{\\mathcal\{G\}\}\_\{\\text\{rep\}\}\)inℬrep​\(x\)\\mathcal\{B\}\_\{\\text\{rep\}\}\(x\), the candidate group

𝒢~rep​\(x,yi,a\)=\{y~i\(j\)\}j=1Grep\\widetilde\{\\mathcal\{G\}\}\_\{\\text\{rep\}\}\(x,y\_\{i\},a\)=\\\{\\tilde\{y\}\_\{i\}^\{\(j\)\}\\\}\_\{j=1\}^\{G\_\{\\text\{rep\}\}\}is optimized in GSPO form\. LetA^irep​\(y~i\(j\)\)\\widehat\{A\}^\{\\text\{rep\}\}\_\{i\}\(\\tilde\{y\}\_\{i\}^\{\(j\)\}\)denote the within\-group advantage normalized bysreps\_\{\\text\{rep\}\}\. The repair sequence\-level likelihood ratio is

ui\(j\)​\(θ\)=exp⁡\(1Ty~i\(j\)​∑t=1Ty~i\(j\)log⁡ρθ​\(y~i,t\(j\)∣x,yi,a,y~i,<t\(j\)\)ρθold​\(y~i,t\(j\)∣x,yi,a,y~i,<t\(j\)\)\)\.u\_\{i\}^\{\(j\)\}\(\\theta\)=\\exp\\\!\\Bigg\(\\frac\{1\}\{T\_\{\\tilde\{y\}\_\{i\}^\{\(j\)\}\}\}\\sum\_\{t=1\}^\{T\_\{\\tilde\{y\}\_\{i\}^\{\(j\)\}\}\}\\log\\frac\{\\rho\_\{\\theta\}\(\\tilde\{y\}\_\{i,t\}^\{\(j\)\}\\mid x,y\_\{i\},a,\\tilde\{y\}\_\{i,<t\}^\{\(j\)\}\)\}\{\\rho\_\{\\theta\_\{\\text\{old\}\}\}\(\\tilde\{y\}\_\{i,t\}^\{\(j\)\}\\mid x,y\_\{i\},a,\\tilde\{y\}\_\{i,<t\}^\{\(j\)\}\)\}\\Bigg\)\.\(19\)The repair sequence\-level clipped term is

ℒirep,\(j\)=min⁡\(ui\(j\)​\(θ\)​A^irep​\(y~i\(j\)\),clip​\(ui\(j\)​\(θ\),1−ϵ,1\+ϵ\)​A^irep​\(y~i\(j\)\)\)\.\\mathcal\{L\}\_\{i\}^\{\\text\{rep\},\(j\)\}=\\min\\Big\(u\_\{i\}^\{\(j\)\}\(\\theta\)\\,\\widehat\{A\}^\{\\text\{rep\}\}\_\{i\}\(\\tilde\{y\}\_\{i\}^\{\(j\)\}\),\\;\\mathrm\{clip\}\\big\(u\_\{i\}^\{\(j\)\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\\big\)\\,\\widehat\{A\}^\{\\text\{rep\}\}\_\{i\}\(\\tilde\{y\}\_\{i\}^\{\(j\)\}\)\\Big\)\.\(20\)The repair objective optimizes overGrepG\_\{\\text\{rep\}\}candidates in GSPO form, steering the repair mode toward high\-scoring candidates:

𝒥rep​\(θ;ℬrep\)\\displaystyle\\mathcal\{J\}\_\{\\text\{rep\}\}\(\\theta;\\mathcal\{B\}\_\{\\text\{rep\}\}\)=𝔼\(x,yi,a,𝒢~rep\)∼ℬrep​\[1Grep​∑j=1Grepℒirep,\(j\)\]−βkl​𝒦​\(θ\),\\displaystyle=\\mathbb\{E\}\_\{\(x,y\_\{i\},a,\\widetilde\{\\mathcal\{G\}\}\_\{\\text\{rep\}\}\)\\sim\\mathcal\{B\}\_\{\\text\{rep\}\}\}\\Bigg\[\\frac\{1\}\{G\_\{\\text\{rep\}\}\}\\sum\_\{j=1\}^\{G\_\{\\text\{rep\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{rep\},\(j\)\}\\Bigg\]\-\\beta\_\{\\text\{kl\}\}\\,\\mathcal\{K\}\(\\theta\),\(21\)
Each step jointly updates the shared parametersθ\\theta:

𝒥IOP​\(θ;ℬpol,ℬrep\)=𝒥IOP\-GSPO​\(θ;ℬpol\)\+λrep​𝒥rep​\(θ;ℬrep\),\\mathcal\{J\}\_\{\\text\{IOP\}\}\(\\theta;\\mathcal\{B\}\_\{\\text\{pol\}\},\\mathcal\{B\}\_\{\\text\{rep\}\}\)=\\mathcal\{J\}\_\{\\text\{IOP\-GSPO\}\}\(\\theta;\\mathcal\{B\}\_\{\\text\{pol\}\}\)\+\\lambda\_\{\\text\{rep\}\}\\,\\mathcal\{J\}\_\{\\text\{rep\}\}\(\\theta;\\mathcal\{B\}\_\{\\text\{rep\}\}\),\(22\)whereλrep\\lambda\_\{\\text\{rep\}\}controls the repair weight \(sensitivity analysis in Appendix[A\.4](https://arxiv.org/html/2605.05226#A1.SS4)\) and𝒦​\(θ\)\\mathcal\{K\}\(\\theta\)is the token\-level KL regularizer against the reference policy\. The complete single\-step procedure is given in Appendix Algorithm 1\.

## 4Experiments

### 4\.1Experimental Setup

We compare IOP\-GSPO with GSPO under a compute\-matched setting \(identical total token budget\)\. The evaluation metric is Acc avg@32 \(average accuracy over 32 independent samples\)\. Base models are Qwen3\-32B \(dense\) and Qwen3\-Next\-80B\-A3B\-Thinking \(MoE, 3B active parameters\), covering both dense and MoE architectures\. Training data is a mixed subset of DeepMath\-103K\(Zheng et al\.,[2025b](https://arxiv.org/html/2605.05226#bib.bib31)\)\(decontaminated mathematical reasoning dataset\) and OpenCodeReasoning\(Dikkala et al\.,[2025](https://arxiv.org/html/2605.05226#bib.bib17)\)\(decontaminated code reasoning dataset\); cold\-start data is constructed as described in §[3\.1\.1](https://arxiv.org/html/2605.05226#S3.SS1.SSS1)\. Evaluation benchmarks span three reasoning categories: AIME25Mathematical Association of America \([2024](https://arxiv.org/html/2605.05226#bib.bib15)\)\(competition mathematics\), HMMT25Balunović et al\. \([2025](https://arxiv.org/html/2605.05226#bib.bib1)\)\(long\-chain mathematical reasoning\), and LiveCodeBench v6Jain et al\. \([2024](https://arxiv.org/html/2605.05226#bib.bib5)\)\(code reasoning\)\. Full hyperparameters are provided in Appendix Table[4](https://arxiv.org/html/2605.05226#A1.T4)\.

### 4\.2Main Results

Table 1:Main results under compute\-matched setting \(mean±\\pm95% bootstrap CI over 5 seeds\)\. IOP\-GSPO consistently outperforms GSPO across all three benchmarks and both architectures \(dense and MoE\); all confidence intervals are non\-overlapping, indicating statistically significant differences\.Table[1](https://arxiv.org/html/2605.05226#S4.T1)shows that IOP\-GSPO consistently outperforms GSPO across all configurations \(Qwen3\-32B average \+6\.9%, Qwen3\-Next average \+4\.9%\)\. Improvements are most pronounced on long\-chain reasoning \(HMMT25: \+7\.8%/\+6\.8%\), where localized gating has the greatest advantage\. Additionally, under a step\-matched comparison, IOP\-GSPO reaches the final performance of GSPO at 800 steps in only approximately 350 steps, yielding approximately2\.3×2\.3\\timessample efficiency \(Appendix[A\.7](https://arxiv.org/html/2605.05226#A1.SS7)\)\.

### 4\.3Ablation Analysis and Method Comparison

Table 2:Ablation and method comparison on Qwen3\-32B \(Acc avg@32, %\)\. Top: all six design ablations cause significant degradation; middle: IOP\-GSPO outperforms PRM rerank, PRIME, and V\-PPO without external annotations; bottom: increased sampling alone does not explain the gains\.Table[2](https://arxiv.org/html/2605.05226#S4.T2)validates the necessity of each of the six design components on Qwen3\-32B\. The most impactful iscorrect sample exclusion: including correct samples causes probability mass to concentrate too rapidly, inducing entropy collapse mid\-training \(HMMT25 drops 6\.1%\)\. Removing theminimum\-edit constraintdegrades repair into wholesale rewriting; disablingtruncationallows cascade drift to propagate negative feedback to unrelated positions; removinggatingreduces the method to sequence\-level advantage weighting; removing theaudit gate\(h≡1h\\\!\\equiv\\\!1\) allows approximately 8% of reward\-hacking repairs into training, introducing spurious process supervision signals \(AIME25 drops 2\.3%\)\. Replacingverification\-based adaptive truncationwith fixed\-KKtruncation \(without graft verification\) causes a 1\.4% drop on AIME25—approximately 28% of samples actually require a larger truncation window, and fixedKKintroduces insufficient training signals for these cases\.

The middle section shows that IOP\-GSPO outperforms GSPO \+ PRM rerank, PRIME, and V\-PPO without any external annotations\. The bottom section rules out the “more sampling is sufficient” hypothesis: compute\-matchedG′=48G^\{\\prime\}\\\!=\\\!48and rejection sampling both fall well short of IOP\-GSPO, confirming that the gains stem from signal quality rather than sampling volume\.

##### Training dynamics\.

Appendix Table[3](https://arxiv.org/html/2605.05226#A1.T3)shows that repair success rate rises from 38\.4% to 73\.1%, policy accuracy from 52\.1% to 83\.5%, and active token ratio decreases from 24\.6% to 9\.8%\. The three metrics evolve in concert: improved policy produces better references that in turn improve repair quality, and more precise repairs improve local update signals\. The steadily declining active token ratio indicates that gating automatically refines over training—direct evidence that outcome supervision is being internalized into process supervision\. Hyperparameter sensitivity and computational overhead are discussed in Appendix[A\.4](https://arxiv.org/html/2605.05226#A1.SS4)–[A\.7](https://arxiv.org/html/2605.05226#A1.SS7)\.

## 5Conclusion

We propose a new paradigm—internalizing outcome supervision into process supervision—and realize it through the IOP framework and its instantiation IOP\-GSPO\. The core idea is to have the same model generate minimum\-edit repairs for failed trajectories, then convert outcome\-level feedback into token\-level gating signals via truncated alignment and audit gating\. Experiments show that IOP\-GSPO consistently outperforms GSPO \(\+4\.9–6\.9%\) and exogenous process supervision methods across three reasoning benchmarks, with policy and repair capabilities forming a synergistic reinforcement loop\. IOP demonstrates a self\-improvement pathway: supervision quality improves in tandem with policy capability, providing an effective and low\-cost alternative for fine\-grained credit assignment under outcome feedback alone\. Future directions include extension to multi\-turn agent tasks\.

## 6Limitations

IOP has the following limitations\. \(1\) Minimum edit distance does not equate to causal minimality—paraphrasing or equivalent transformations may cause the difference mask to deviate from the true error source; the method should therefore be viewed as a practical approximation rather than strict causal localization\. \(2\) The method requires at least one correct trajectory per prompt as a reference anchor, and the repair mode must reliably produce high\-quality candidates; both conditions are difficult to satisfy when the base model is too weak\. \(3\) Experiments are limited to math and code reasoning benchmarks \(up to 32B dense parameters\) and do not cover larger\-scale models, long\-context reasoning, tool use, or multi\-turn agent scenarios\.

## Reproducibility Statement

Code will be released via an anonymous repository\.

## References

- Balunović et al\. \(2025\)Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev\.MathArena: Evaluating LLMs on uncontaminated math competitions\.*arXiv preprint arXiv:2505\.23281*, 2025\.
- Cui et al\. \(2025\)Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al\.Process reinforcement through implicit rewards\.*arXiv preprint arXiv:2502\.01456*, 2025\.
- DeepSeek\-AI et al\. \(2025\)DeepSeek\-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al\.DeepSeek\-R1: Incentivizing reasoning capability in LLMs via reinforcement learning\.*arXiv preprint arXiv:2501\.12948*, 2025\.
- Havrilla et al\. \(2024\)Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi\-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu\.Teaching large language models to reason with reinforcement learning\.*arXiv preprint arXiv:2403\.04642*, 2024\.
- Jain et al\. \(2024\)Naman Jain, King Han, Alex Gu, Wen\-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar\-Lezama, Koushik Sen, and Ion Stoica\.LiveCodeBench: Holistic and contamination free evaluation of large language models for code\.*arXiv preprint arXiv:2403\.07974*, 2024\.
- Khalifa et al\. \(2025\)Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng, Moontae Lee, Honglak Lee, and Lu Wang\.Process reward models that think\.*arXiv preprint arXiv:2504\.16828*, 2025\.
- Kumar et al\. \(2024\)Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D\. Co\-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M\. Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksandra Faust\.Training language models to self\-correct via reinforcement learning\.*arXiv preprint arXiv:2409\.12917*, 2024\.
- Lee et al\. \(2025\)Hyunseok Lee, Seunghyuk Oh, Jaehyung Kim, Jinwoo Shin, and Jihoon Tack\.ReVISE: Learning to refine at test\-time via intrinsic self\-verification\.*arXiv preprint arXiv:2502\.14565*, 2025\.
- Liang et al\. \(2026\)Qiao Liang, Yuke Zhu, Chao Ge, Lei Yang, Ying Shen, Bo Zheng, and Sheng Guo\.Learning from the irrecoverable: Error\-localized policy optimization for tool\-integrated LLM reasoning\.*arXiv preprint arXiv:2602\.09598*, 2026\.
- Lightman et al\. \(2023\)Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe\.Let’s verify step by step\.*arXiv preprint arXiv:2305\.20050*, 2023\.
- Liu et al\. \(2026\)Haolin Liu, Dian Yu, Sidi Lu, Yujun Zhou, Rui Liu, Zhenwen Liang, Haitao Mi, Chen\-Yu Wei, and Dong Yu\.Save the good prefix: Precise error penalization via process\-supervised RL to enhance LLM reasoning\.*arXiv preprint arXiv:2601\.18984*, 2026\.
- Luo et al\. \(2024\)Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi\.Improve mathematical reasoning in language models by automated process supervision\.*arXiv preprint arXiv:2406\.06592*, 2024\.
- Ma et al\. \(2025\)Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li\.S2R: Teaching LLMs to self\-verify and self\-correct via reinforcement learning\.*arXiv preprint arXiv:2502\.12853*, 2025\.
- Madaan et al\. \(2023\)Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al\.Self\-Refine: Iterative refinement with self\-feedback\.*arXiv preprint arXiv:2303\.17651*, 2023\.
- Mathematical Association of America \(2024\)Mathematical Association of America\.2024\-25 AIME thresholds are available\.[https://maa\.org/aime\-thresholds\-are\-available/](https://maa.org/aime-thresholds-are-available/), 2024\.
- Nie et al\. \(2026\)Shuaiyi Nie, Siyu Ding, Wenyuan Zhang, Linhao Yu, Tianmeng Yang, Yao Chen, Tingwen Liu, Weichong Yin, Yu Sun, and Hua Wu\.ATTNPO: Attention\-guided process supervision for efficient reasoning\.*arXiv preprint arXiv:2602\.09953*, 2026\.
- Dikkala et al\. \(2025\)Nishanth Dikkala, Jiayi Shi, Naman Jain, Shaikh Quader Hossain, Niklas Muennighoff, Yuntian Tao, Jonathan Tow, Hailey Wang, Guowei Shen, Tushar Jain, et al\.OpenCodeReasoning: Advancing data distillation for competitive coding\.*arXiv preprint arXiv:2504\.01943*, 2025\.
- OpenAI \(2024\)OpenAI\.Learning to reason with LLMs\.[https://openai\.com/index/learning\-to\-reason\-with\-llms/](https://openai.com/index/learning-to-reason-with-llms/), 2024\.
- OpenAI \(2025\)OpenAI\.Introducing gpt\-oss\.*OpenAI Technical Report*, 2025\.
- Pronesti et al\. \(2026\)Massimiliano Pronesti, Anya Belz, and Yufang Hou\.Beyond outcome verification: Verifiable process reward models for structured reasoning\.*arXiv preprint arXiv:2601\.17223*, 2026\.
- Rafailov et al\. \(2023\)Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D\. Manning, and Chelsea Finn\.Direct preference optimization: Your language model is secretly a reward model\.*arXiv preprint arXiv:2305\.18290*, 2023\.
- Shao et al\. \(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y\. K\. Li, Y\. Wu, et al\.DeepSeekMath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.
- Shinn et al\. \(2023\)Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao\.Reflexion: Language agents with verbal reinforcement learning\.*arXiv preprint arXiv:2303\.11366*, 2023\.
- Wang et al\. \(2023\)Peiyi Wang, Lei Li, Zhihong Shao, R\. X\. Xu, Damai Dai, Yifei Li, Deli Chen, Y\. Wu, and Zhifang Sui\.Math\-Shepherd: Verify and reinforce LLMs step\-by\-step without human annotations\.*arXiv preprint arXiv:2312\.08935*, 2023\.
- Wen et al\. \(2025\)Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, et al\.Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base LLMs\.*arXiv preprint arXiv:2506\.14245*, 2025\.
- Xiong et al\. \(2025\)Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang\.Self\-rewarding correction for mathematical reasoning\.*arXiv preprint arXiv:2502\.19613*, 2025\.
- Yang et al\. \(2025\)Zhaohui Yang, Chenghua He, Xiaowen Shi, Linjing Li, Qiyue Yin, Shihong Deng, and Daxin Jiang\.Beyond the first error: Process reward models for reflective mathematical reasoning\.*arXiv preprint arXiv:2505\.14391*, 2025\.
- Yao et al\. \(2026\)Jiarui Yao, Ruida Wang, and Tong Zhang\.PRL: Process reward learning improves LLMs’ reasoning ability and broadens the reasoning boundary\.*arXiv preprint arXiv:2601\.10201*, 2026\.
- Zhang et al\. \(2025\)Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin\.The lessons of developing process reward models in mathematical reasoning\.*arXiv preprint arXiv:2501\.07301*, 2025\.
- Zheng et al\. \(2025a\)Chujie Zheng, Shixuan Liu, Mingze Li, Xiong\-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin\.Group sequence policy optimization\.*arXiv preprint arXiv:2507\.18071*, 2025\.
- Zheng et al\. \(2025b\)Chujie Zheng, Jie Zhou, Zhoufan Meng, Yilun Fan, and Junyang Lin\.DeepMath\-103K: A large\-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning\.*arXiv preprint arXiv:2504\.11456*, 2025\.

## Appendix AAppendix

### A\.1Algorithm

Algorithm 1IOP\-GSPO Stage 2 single\-step procedure: sampling→\\topartition→\\torepair→\\togating→\\tojoint update1:Input:prompt set

𝒟\\mathcal\{D\}, shared\-parameter model

θ\\theta\(with policy mode and repair mode\)

2:foreach prompt

x∈𝒟x\\in\\mathcal\{D\}do

3:Sample initial group

𝒢′​\(x\)=\{yi\}i=1G′\\mathcal\{G\}^\{\\prime\}\(x\)=\\\{y\_\{i\}\\\}\_\{i=1\}^\{G^\{\\prime\}\}, where

yi∼πθ\(⋅∣x\)y\_\{i\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)
4:Partition

𝒢′​\(x\)\\mathcal\{G\}^\{\\prime\}\(x\)into

𝒢cor​\(x\)\\mathcal\{G\}\_\{\\text\{cor\}\}\(x\)and

𝒢err​\(x\)\\mathcal\{G\}\_\{\\text\{err\}\}\(x\)by reward threshold

5:if

𝒢cor​\(x\)=∅\\mathcal\{G\}\_\{\\text\{cor\}\}\(x\)=\\emptysetor

𝒢err​\(x\)=∅\\mathcal\{G\}\_\{\\text\{err\}\}\(x\)=\\emptysetthen

6:Skip the IOP pipeline for this prompt

7:else

8:Sample reference trajectory

aafrom

𝒢cor​\(x\)\\mathcal\{G\}\_\{\\text\{cor\}\}\(x\)
9:foreach

yi∈𝒢err​\(x\)y\_\{i\}\\in\\mathcal\{G\}\_\{\\text\{err\}\}\(x\)do

10:Generate repair candidate group

𝒢~rep​\(x,yi,a\)=\{𝐲~\(j\)\}j=1Grep\\widetilde\{\\mathcal\{G\}\}\_\{\\text\{rep\}\}\(x,y\_\{i\},a\)=\\\{\\tilde\{\\mathbf\{y\}\}^\{\(j\)\}\\\}\_\{j=1\}^\{G\_\{\\text\{rep\}\}\}
11:Score by

sreps\_\{\\text\{rep\}\}and select best repair

y~i\\tilde\{y\}\_\{i\}
12:Write

\(x,yi,a,𝒢~rep​\(x,yi,a\)\)\(x,y\_\{i\},a,\\widetilde\{\\mathcal\{G\}\}\_\{\\text\{rep\}\}\(x,y\_\{i\},a\)\)to repair mini\-batch

ℬrep\\mathcal\{B\}\_\{\\text\{rep\}\}
13:If

rtask​\(x,y~i\)=1r\_\{\\text\{task\}\}\(x,\\tilde\{y\}\_\{i\}\)\\\!=\\\!1: align and truncate to obtain mask, construct pair

𝒢i​\(x\)=\{yi,y~i\}\\mathcal\{G\}\_\{i\}\(x\)\\\!=\\\!\\\{y\_\{i\},\\tilde\{y\}\_\{i\}\\\}
14:endfor

15:Write all pairs to policy mini\-batch

ℬpol​\(x\)=\{𝒢i​\(x\)\}\\mathcal\{B\}\_\{\\text\{pol\}\}\(x\)=\\\{\\mathcal\{G\}\_\{i\}\(x\)\\\}
16:Compute IOP\-GSPO policy objective from

ℬpol​\(x\)\\mathcal\{B\}\_\{\\text\{pol\}\}\(x\)
17:Compute standard GSPO objective for repair mode from

ℬrep​\(x\)\\mathcal\{B\}\_\{\\text\{rep\}\}\(x\)
18:Jointly update shared parameters

θ\\theta
19:endif

20:endfor

### A\.2Training Dynamics

Table 3:Qwen3\-32B training dynamics: synergistic reinforcement between policy and repair\. Repair success rate \(\+34\.7%\) and policy accuracy \(\+31\.4%\) rise in tandem, while active token ratio steadily decreases \(24\.6%→\\to9\.8%\), indicating that gating signals automatically refine over training\.The post\-cold\-start repair success rate of only 38\.4% confirms the necessity of Stage 2 joint RL\. Repair success rate improves most rapidly during Steps 0–200 \(\+13\.3%\), coinciding with the period of fastest policy accuracy improvement, directly corroborating synergistic reinforcement\. The slowdown in later stages reflects natural convergence: as policy accuracy improves, the number of available failed trajectories decreases, correspondingly reducing repair opportunities\.

### A\.3Training Hyperparameters

HyperparameterQwen3\-32BQwen3\-NextInitial sampling group sizeG′G^\{\\prime\}1616Repair candidatesGrepG\_\{\\text\{rep\}\}44Truncation lengthKK5050Edit distance weightλedit\\lambda\_\{\\text\{edit\}\}\(min\. threshold 0\.05\)0\.30\.3Repair objective weightλrep\\lambda\_\{\\text\{rep\}\}0\.20\.2KL penaltyβkl\\beta\_\{\\text\{kl\}\}0\.0020\.002Reward thresholdτr\\tau\_\{r\}0\.50\.5Learning rate \(AdamW, 20\-step warmup\)1×10−61\\times 10^\{\-6\}5×10−75\\times 10^\{\-7\}Prompt batch size6464Cold\-start SFT data size500500Maximum sequence length3276832768Table 4:Complete IOP\-GSPO training hyperparameters\. The two architectures differ only in learning rate, indicating robustness to model scale and architecture type\.
### A\.4Sensitivity of Repair Objective Weightλrep\\lambda\_\{\\text\{rep\}\}

Table 5:Sensitivity of repair objective weightλrep\\lambda\_\{\\text\{rep\}\}\(Qwen3\-32B, Acc avg@32, %\)\. All values inλrep∈\[0\.1,0\.3\]\\lambda\_\{\\text\{rep\}\}\\\!\\in\\\!\[0\.1,0\.3\]outperform the GSPO baseline\.Whenλrep\\lambda\_\{\\text\{rep\}\}is too small, repair training is insufficient \(at 0\.1, AIME25 reaches only 81\.4%\); when too large, repair gradients dominate the shared parameters and degrade policy performance\.λrep=0\.2\\lambda\_\{\\text\{rep\}\}\\\!=\\\!0\.2achieves the best balance between policy and repair\.

### A\.5Sensitivity of Truncation LengthKK

Table 6:Effect of truncation lengthKKon IOP\-GSPO performance \(Qwen3\-32B, Acc avg@32, %\)\.
### A\.6Effect of Repair Candidate CountGrepG\_\{\\text\{rep\}\}

Table 7:Effect of repair candidate countGrepG\_\{\\text\{rep\}\}on task performance and repair success rate \(Qwen3\-32B, Acc avg@32, %\)\.IncreasingGrepG\_\{\\text\{rep\}\}from 2 to 4 raises repair success rate from 59\.6% to 73\.1%, with task performance improving correspondingly; further increase to 8 yields diminishing returns while linearly increasing cost\.Grep=4G\_\{\\text\{rep\}\}\\\!=\\\!4provides the best balance between performance and efficiency\.

### A\.7Computational Overhead

The additional overhead of IOP\-GSPO primarily comes from repair sampling\. As training progresses, the error rate decreases from approximately 48% initially to approximately 17% later \(Table[3](https://arxiv.org/html/2605.05226#A1.T3)\); the training\-weighted average error rate is approximately 26%, generating approximately 17 additional repair candidates per input \(≈\\approx4\.2 failed trajectories×Grep=4\\times\\,G\_\{\\text\{rep\}\}\\\!=\\\!4\), resulting in approximately2\.2×2\.2\\timestoken throughput increase after accounting for audit gating and graft verification\. The GSPO baseline’s training steps are proportionally increased to consume the same total token budget \(compute\-matched setting\)\. Under step\-matched comparison, IOP\-GSPO reaches GSPO’s final 800\-step performance in only approximately 350 steps \(AIME25: 76\.8% vs\. 76\.6%\), yielding approximately2\.3×2\.3\\timessample efficiency\. Qwen3\-32B is trained on 8×\\timesA100 80 GB \(IOP\-GSPO approximately 48 vs\. GSPO approximately 46 GPU hours\); Qwen3\-Next is trained on 16×\\timesA100 80 GB \(approximately 72 vs\. 68 GPU hours\)\. At both scales, the additional GPU hour overhead is<<6%, while performance improvement reaches 3\.7–7\.8%, representing a favorable cost\-performance trade\-off\.

### A\.8Inference Settings

Both Qwen3\-32B and Qwen3\-Next use Temperature = 0\.6, TopP = 0\.95, TopK = 20, MinP = 0\. All methods are compared under identical decoding settings to ensure that differences arise solely from the training method\. All evaluations use 32 independent samples to compute avg@32\.