Tag
The paper identifies Value Flattening as a failure mode in PPO critic learning for LLMs and introduces SP3O, a sparse supervision method, to mitigate it, showing consistent improvements in experiments.