Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States

Hugging Face Daily Papers Papers
reinforcement-learning value-estimation policy-optimization internal-states variance-reduction reasoning-models

Summary

This paper introduces POISE, a method for stable policy optimization in large reasoning models by estimating baselines using the model's own internal states, reducing computational overhead compared to PPO and GRPO.

Reinforcement learning with verifiable rewards (RLVR) for Large Reasoning Models hinges on baseline estimation for variance reduction, but existing approaches pay a heavy price: PPO requires a policy-model scale critic, while GRPO needs multiple rollouts per prompt to keep its empirical group mean stable. We introduce Policy Optimization with Internal State Value Estimation), which obtains a baseline at negligible cost by using the policy model's internal signals already computed during the policy forward pass. A lightweight probe predicts the expected verifiable reward from the hidden states of the prompt and generated trajectory, as well as token-entropy statistics, and is trained online alongside the policy. To preserve gradient unbiasedness despite using trajectory-conditioned features, we introduce a cross-rollout construction that predicts each rollout's value from an independent rollout's internal states. Because POISE estimates prompt value using only a single rollout, it enables higher prompt diversity for a fixed compute budget during training. This reduces gradient variance for more stable learning and also eliminates the compute overhead of sampling costs for detecting zero-advantage prompts. On Qwen3-4B and DeepSeek-R1-Distill-Qwen-1.5B across math reasoning benchmarks, POISE matches DAPO while requiring less compute. Moreover, its value estimator shows similar performance to a separate LLM-scale value model and generalizes to various verifiable tasks. By leveraging the model's own internal representations, POISE enables more stable and efficient policy optimization.
Original Article
View Cached Full Text

Cached at: 05/13/26, 12:14 PM

Paper page - Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor’s Internal States

Source: https://huggingface.co/papers/2605.07579

Abstract

POISE enables stable and efficient policy optimization for large reasoning models by estimating baselines using internal model signals, reducing computational overhead while maintaining performance comparable to existing methods.

Reinforcement learning with verifiable rewards(RLVR) for Large Reasoning Models hinges on baseline estimation forvariance reduction, but existing approaches pay a heavy price:PPOrequires apolicy-model scale critic, whileGRPOneeds multiple rollouts per prompt to keep itsempirical group meanstable. We introducePolicy OptimizationwithInternal State Value Estimation), which obtains a baseline at negligible cost by using the policy model’s internal signals already computed during the policy forward pass. Alightweight probepredicts the expected verifiable reward from the hidden states of the prompt and generated trajectory, as well as token-entropy statistics, and is trained online alongside the policy. To preservegradient unbiasednessdespite usingtrajectory-conditioned features, we introduce across-rollout constructionthat predicts each rollout’s value from an independent rollout’s internal states. Because POISE estimates prompt value using only a single rollout, it enables higher prompt diversity for a fixed compute budget during training. This reduces gradient variance for more stable learning and also eliminates the compute overhead of sampling costs for detecting zero-advantage prompts. On Qwen3-4B and DeepSeek-R1-Distill-Qwen-1.5B across math reasoning benchmarks, POISE matchesDAPOwhile requiring less compute. Moreover, its value estimator shows similar performance to a separateLLM-scale value modeland generalizes to various verifiable tasks. By leveraging the model’s own internal representations, POISE enables more stable and efficientpolicy optimization.

View arXiv pageView PDFProject pageGitHubAdd to collection

Get this paper in your agent:

hf papers read 2605\.07579

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.07579 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.07579 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.07579 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Towards Robust Reinforcement Learning for Small-Scale Language Model Agents

Hugging Face Daily Papers

This paper systematically investigates failure modes in reinforcement learning for small language models (70-500M parameters) using PPO, identifies silent LoRA freezing, numerical overflow, and catastrophic policy collapse, and proposes a robust system with merge-and-reinitialize adapters, float32 precision, and a safety mechanism. The approach converges stably and outperforms baselines with less data.

Best Practice Critic Optimization

Hugging Face Daily Papers

BPCO introduces a stable recipe for critic-based reinforcement learning in language models, combining bounded value predictions and Monte Carlo targets to match group-based methods with single-response sampling.

Value-Gradient Hypothesis of RL for LLMs

arXiv cs.LG

This paper introduces the value-gradient hypothesis to explain why critic-free RL methods like PPO and GRPO work well for LLMs, showing that the actor backward pass carries a value-gradient-like signal. It derives a predictive criterion for when RL is most effective along the pretraining trajectory.