Unsupervised Process Reward Models
Summary
This paper proposes unsupervised Process Reward Models (uPRM) that eliminate the need for human annotations by using LLM next-token probabilities to identify erroneous reasoning steps, achieving up to 15% accuracy improvements over LLM-as-a-Judge and performing comparably to supervised PRMs as verifiers and reward signals.
View Cached Full Text
Cached at: 05/22/26, 10:19 AM
Paper page - Unsupervised Process Reward Models
Source: https://huggingface.co/papers/2605.10158
Abstract
Unsupervised reward models eliminate the need for human annotations in training by leveraging language model next-token probabilities to identify erroneous reasoning steps and improve policy optimization in reinforcement learning.
Process Reward Models(PRMs) are a powerful mechanism for steering largelanguage modelreasoning by providing fine-grained, step-level supervision. However, this effectiveness comes at a significant cost: PRMs require expert annotations for every reasoning step, making them costly and difficult to scale. Here, we propose a method for training unsupervised PRMs (uPRM) that requires no human supervision, neither at the level of step-by-step annotations nor through ground-truth verification of final answers. The key idea behind our approach is to define a scoring function, derived from LLMnext-token probabilities, that jointly assesses candidate positions offirst erroneous stepsacross a batch ofreasoning trajectories. We demonstrate the effectiveness of uPRM across diverse scenarios: (i) uPRM achieves up to 15% absolute accuracy improvements over the LLM-as-a-Judge in identifyingfirst erroneous stepson theProcessBenchdataset; (ii) as a verifier fortest-time scaling, uPRM performs comparably to supervised PRMs and outperforms the majority voting baseline by up to 6.9%, and (iii) when used as a reward signal inreinforcement learning, uPRM enables more robustpolicy optimizationthroughout training compared to a supervised PRM trained using ground-truth labels. Overall, our results open a path toward scalable reward modeling for complex reasoning tasks.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.10158
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.10158 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.10158 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.10158 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design
This paper systematically investigates reward function design for reinforcement learning to improve the quality of LLM-generated BPMN process models, finding that equal reward weighting outperforms targeted weighting and that design choices interact with model architecture in non-trivial ways.
Process Rewards with Learned Reliability
BetaPRM is a process reward model that predicts both a step-level success probability and the reliability of that prediction using a Beta belief from Monte Carlo continuations, enabling adaptive computation allocation that reduces token usage by up to 33.57% while improving accuracy.
SCI-PRM: A Tool Aware Process Reward Model for Scientific Reasoning Verification
SCI-PRM introduces a tool-aware Process Reward Model for scientific reasoning, trained on the SCIPRM70K dataset featuring 'Chain-of-Tool' trajectories that interleave reasoning with scientific tool execution. It enables effective test-time scaling and serves as a dense reward signal in reinforcement learning, outperforming proprietary models like GPT-5-Mini on tool-calling steps across scientific benchmarks.
The Weakest Link Tells It All: Outcome-Supervised Process Reward Modeling via Learnable Credit Assignment
This paper proposes Outcome-Supervised Process Reward Modeling via Learnable Credit Assignment (LCA), a framework that jointly learns credit assignment and reward modeling under a weakest-link principle, formulated as a Multiple Instance Learning problem with Softmax-Weighted-Sum pooling. Experiments show it outperforms existing outcome-supervised PRMs across multiple tasks.
ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Reward
ProcessThinker introduces a practical post-training pipeline that provides step-level process rewards without training an explicit process reward model. It uses rollout-based rewards to give dense credit assignment for multi-step reasoning in multimodal LLMs, consistently improving performance on video benchmarks.