Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening

Hugging Face Daily Papers Papers
reinforcement-learning ppo value-flattening critic-learning ai-research sparse-supervision

Summary

The paper identifies Value Flattening as a failure mode in PPO critic learning for LLMs and introduces SP3O, a sparse supervision method, to mitigate it, showing consistent improvements in experiments.

In reinforcement learning for large language models, Proximal Policy Optimization (PPO) commonly uses a critic to estimate state values and reduce the variance of policy updates. However, we uncover a systematic failure mode in PPO critics, which we call Value Flattening: state values, estimated from multiple Monte Carlo continuations, change sharply across intermediate states while critic predictions remain comparatively flat. We further observe this phenomenon in a controlled FrozenLake environment and find that it becomes more pronounced as the state space grows. Our theoretical and empirical analyses relate Value Flattening to an implicit variance penalty in the critic loss and redundant updates from temporally correlated states with similar gradients. Motivated by these findings, we introduce SParse Proximal Policy Optimization (SP^3O), which applies the value loss to only a few well-separated states in each response to mitigate both effects. Experiments on Qwen3-Base show that SP^3O with only three states supervised per response can mitigate Value Flattening and consistently improve the learned policy across model sizes and evaluation suites. Together, our results identify Value Flattening as an important yet overlooked failure mode of critic learning in standard PPO and show that a simple sparse supervision strategy can mitigate it.
Original Article
View Cached Full Text

Cached at: 09/17/26, 06:52 AM

Paper page - Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening

Source: https://huggingface.co/papers/2609.18708 Published on Sep 16

·

Submitted byhttps://huggingface.co/ramiroluo

Luoon Sep 17

Authors:

,

,

,

,

,

,

,

,

,

,

Abstract

Inreinforcementlearningforlargelanguagemodels,ProximalPolicyOptimization(PPO)commonlyusesacritictoestimatestatevaluesandreducethevarianceofpolicyupdates.However,weuncoverasystematicfailuremodeinPPOcritics,whichwecallValueFlattening:statevalues,estimatedfrommultipleMonteCarlocontinuations,changesharplyacrossintermediatestateswhilecriticpredictionsremaincomparativelyflat.WefurtherobservethisphenomenoninacontrolledFrozenLakeenvironmentandfindthatitbecomesmorepronouncedasthestatespacegrows.OurtheoreticalandempiricalanalysesrelateValueFlatteningtoanimplicitvariancepenaltyinthecriticlossandredundantupdatesfromtemporallycorrelatedstateswithsimilargradients.Motivatedbythesefindings,weintroduceSParseProximalPolicyOptimization(SP^3O),whichappliesthevaluelosstoonlyafewwell-separatedstatesineachresponsetomitigatebotheffects.ExperimentsonQwen3-BaseshowthatSP^3OwithonlythreestatessupervisedperresponsecanmitigateValueFlatteningandconsistentlyimprovethelearnedpolicyacrossmodelsizesandevaluationsuites.Together,ourresultsidentifyValueFlatteningasanimportantyetoverlookedfailuremodeofcriticlearninginstandardPPOandshowthatasimplesparsesupervisionstrategycanmitigateit.

View arXiv pageView PDFProject pageGitHubAdd to collection

Get this paper in your agent:

hf papers read 2609\.18708

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.18708 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.18708 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.18708 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

KLip-PPO: A per-sample KL perspective on PPO-Clip

arXiv cs.LG

This paper shows that the gradient of the clipped surrogate in Proximal Policy Optimization (PPO) is exactly reproduced by a per-sample Kullback-Leibler penalty with a variable coefficient, revealing structural features of the clipped surrogate and suggesting new design directions.

Value-Gradient Hypothesis of RL for LLMs

arXiv cs.LG

This paper introduces the value-gradient hypothesis to explain why critic-free RL methods like PPO and GRPO work well for LLMs, showing that the actor backward pass carries a value-gradient-like signal. It derives a predictive criterion for when RL is most effective along the pretraining trajectory.