Best Practice Critic Optimization
Summary
BPCO introduces a stable recipe for critic-based reinforcement learning in language models, combining bounded value predictions and Monte Carlo targets to match group-based methods with single-response sampling.
View Cached Full Text
Cached at: 08/26/26, 03:15 AM
Paper page - Best Practice Critic Optimization
Source: https://huggingface.co/papers/2608.23566
Abstract
BPCO stabilizes critic-based reinforcement learning for language models by combining bounded value predictions, Monte Carlo targets, and adaptive advantage estimation, matching group-based methods with single-response sampling.
Group-based reinforcement learning methods such asGRPOfor large language models avoid training a critic by sampling multiple responses for each prompt. A reliable critic could instead estimatetoken-level advantagesfrom one response, but standard critic-based training recipes are often unstable. We study this instability and develop **Best PracticeCritic Optimization(BPCO)**, a recipe that combinesDPPO,value predictionsbounded to the reward range,Monte Carlo value targets, unnormalized policy advantages, and length-adaptivegeneralized advantage estimation. Because the critic is used only during training, BPCO can also condition it on reward-defining information, such as a reference answer or grading rubric, that is hidden from the policy. Controlled experiments isolate the effect of each design choice. Across mathematical reasoning tasks with models ranging from 1.5B parameters to 30B-A3B mixtures of experts, BPCO improves a strong critic-based baseline consistently, and matches or exceeds a group-based baseline while sampling one response per prompt. The same recipe also improves learning withrubric-based rewards. These results show that a carefully designed critic provides a reliable alternative to group-relative advantage estimation. Code is available at https://github.com/QPHutu/golden_critic.
View arXiv pageView PDFGitHubAdd to collection
Get this paper in your agent:
hf papers read 2608\.23566
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.23566 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.23566 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.23566 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Contrastive Branch Policy Optimization
CBPO introduces a contrastive branch policy optimization method for fine-grained credit assignment in reinforcement learning with verifiable rewards, enhancing language model performance in tool-integrated reasoning tasks across multiple benchmarks.
EasyPPO: Stabilizing the Critic Is Key
EasyPPO stabilizes Proximal Policy Optimization for large language models by addressing critic instability through actor-only overlong filtering and noise-normalized regression, achieving performance gains over baseline methods.
Bellman Policy Optimization
Bellman Policy Optimization (BPO) is a critic-free reinforcement learning method that reformulates Policy Mirror Descent using the Bellman equation for autoregressive generation with terminal rewards, improving mathematical reasoning in large language models.
Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States
This paper introduces POISE, a method for stable policy optimization in large reasoning models by estimating baselines using the model's own internal states, reducing computational overhead compared to PPO and GRPO.
BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards
BV-Blend is a critic-free reinforcement learning framework that combines prompt-local on-policy statistics with historical moments from semantic clusters to stabilize advantage estimation, improving training stability and performance for aligning large language models with verifiable rewards.