Inducing Process Supervision from Outcome-Only Reinforcement Learning

Hugging Face Daily Papers Papers
process-reward-models reinforcement-learning llm-reasoning chain-of-thought step-level-feedback math-benchmarks

Summary

The paper introduces TIPS (Thinking-Induced Process Supervision), an outcome-only reinforcement learning framework that trains generative process reward models using only outcome-level rewards, achieving 85.2 F1 on ProcessBench with just 3.2K outcome-labeled trajectories and outperforming existing trained PRMs and prompt-only judges like GPT-5.4-Instruct.

Process reward models (PRMs) have become a key component for LLMs, as their step-level feedback supports both post-training and test-time reasoning. However, training strong PRMs remains costly: human step annotation is difficult to scale, while Monte Carlo estimation is computationally expensive and can drift from the intrinsic correctness of steps. To get effective PRMs at low cost, we introduce TIPS (Thinking-Induced Process Supervision), an outcome-only reinforcement learning (RL) framework for training generative PRMs. In TIPS, the model generates a chain-of-thought (CoT) followed by step-level labels and an outcome label. The reward depends solely on whether the predicted outcome matches the ground truth, and the resulting group-relative advantage is used to optimize the entire generated response. Intuitively, when checking intermediate steps helps determine the outcome, more accurate checks can lead to better outcome judgments and higher rewards. Outcome-only RL can therefore reinforce step-level verification without explicit process supervision. We validate the effectiveness of TIPS across math and agent benchmarks and four backbone families. Notably, TIPS-Qwen3-4B-Thinking-2507 reaches 85.2 F1 on ProcessBench with only 3.2K outcome-labeled trajectories, surpassing all evaluated trained PRMs and strong prompt-only judges such as GPT-5.4-Instruct and Claude-4.7-Opus, while still trailing o1-mini. Code and data are available at https://github.com/RUCBM/TIPS.
Original Article
View Cached Full Text

Cached at: 10/12/26, 06:55 AM

Paper page - Inducing Process Supervision from Outcome-Only Reinforcement Learning

Source: https://huggingface.co/papers/2609.36641

Abstract

Processrewardmodels(PRMs)havebecomeakeycomponentforLLMs,astheirstep-levelfeedbacksupportsbothpost-trainingandtest-timereasoning.However,trainingstrongPRMsremainscostly:humanstepannotationisdifficulttoscale,whileMonteCarloestimationiscomputationallyexpensiveandcandriftfromtheintrinsiccorrectnessofsteps.TogeteffectivePRMsatlowcost,weintroduceTIPS(Thinking-InducedProcessSupervision),anoutcome-onlyreinforcementlearning(RL)frameworkfortraininggenerativePRMs.InTIPS,themodelgeneratesachain-of-thought(CoT)followedbystep-levellabelsandanoutcomelabel.Therewarddependssolelyonwhetherthepredictedoutcomematchesthegroundtruth,andtheresultinggroup-relativeadvantageisusedtooptimizetheentiregeneratedresponse.Intuitively,whencheckingintermediatestepshelpsdeterminetheoutcome,moreaccuratecheckscanleadtobetteroutcomejudgmentsandhigherrewards.Outcome-onlyRLcanthereforereinforcestep-levelverificationwithoutexplicitprocesssupervision.WevalidatetheeffectivenessofTIPSacrossmathandagentbenchmarksandfourbackbonefamilies.Notably,TIPS-Qwen3-4B-Thinking-2507reaches85.2F1onProcessBenchwithonly3.2Koutcome-labeledtrajectories,surpassingallevaluatedtrainedPRMsandstrongprompt-onlyjudgessuchasGPT-5.4-InstructandClaude-4.7-Opus,whilestilltrailingo1-mini.Codeanddataareavailableathttps://github.com/RUCBM/TIPS.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2609\.36641

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.36641 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.36641 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.36641 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Unsupervised Process Reward Models

Hugging Face Daily Papers

This paper proposes unsupervised Process Reward Models (uPRM) that eliminate the need for human annotations by using LLM next-token probabilities to identify erroneous reasoning steps, achieving up to 15% accuracy improvements over LLM-as-a-Judge and performing comparably to supervised PRMs as verifiers and reward signals.

Process Reward Informed Tree Rollout for Effective Multi-Turn RL

arXiv cs.CL

Proposes PaTR, a process-reward-guided adaptive tree rollout framework for multi-turn reinforcement learning in LLM agents. It selectively branches from promising states and prunes dead-end paths, achieving up to +5.0 on SWE-Bench and +9.3 on FrozenLake under the same training budget.

Improving mathematical reasoning with process supervision

OpenAI Blog

OpenAI demonstrates that process supervision—rewarding intermediate reasoning steps rather than just final answers—improves mathematical reasoning while reducing alignment costs. This approach produces more interpretable, human-aligned reasoning without sacrificing model performance.