Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex

Hugging Face Daily Papers Papers
reinforcement-learning large-language-models policy-gradient training-stability post-training verifiable-rewards

Summary

This paper introduces Listwise Policy Optimization (LPO), a method for RLVR that explicitly handles target projection via divergence minimization on the response simplex to improve training stability and performance in LLMs.

Reinforcement learning with verifiable rewards (RLVR) has become a standard approach for large language models (LLMs) post-training to incentivize reasoning capacity. Among existing recipes, group-based policy gradient is prevalent, which samples a group of responses per prompt and updates the policy via group-relative advantage signals. This work reveals that these optimization strategies share a common geometric structure: each implicitly defines a target distribution on the response simplex and projects toward it via first-order approximation. Building on this insight, we propose Listwise Policy Optimization (LPO) to explicitly conduct the target-projection, which demystifies the implicit target by restricting the proximal RL objective to the response simplex, and then projects the policy via exact divergence minimization. This framework provides (i) monotonic improvement on the listwise objective with bounded, zero-sum, and self-correcting projection gradients, and (ii) flexibility in divergence selection with distinct structural properties through the decoupled projection step. On diverse reasoning tasks and LLM backbones, LPO consistently improves training performance over typical policy gradient baselines under matched targets, while intrinsically preserving optimization stability and response diversity.
Original Article
View Cached Full Text

Cached at: 05/11/26, 02:43 AM

Paper page - Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex

Source: https://huggingface.co/papers/2605.06139 Published on May 7

·

Submitted byhttps://huggingface.co/yunqu

Yun Quon May 11

#1 Paper of the day Authors:

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Group-based policy gradient methods in reinforcement learning with verifiable rewards share a common geometric structure, leading to the development of Listwise Policy Optimization that explicitly handles target projection through divergence minimization for improved training performance and stability.

Reinforcement learning with verifiable rewards(RLVR) has become a standard approach for large language models (LLMs) post-training to incentivize reasoning capacity. Among existing recipes, group-basedpolicy gradientis prevalent, which samples a group of responses per prompt and updates the policy via group-relative advantage signals. This work reveals that these optimization strategies share a common geometric structure: each implicitly defines atarget distributionon theresponse simplexand projects toward it viafirst-order approximation. Building on this insight, we proposeListwise Policy Optimization(LPO) to explicitly conduct the target-projection, which demystifies the implicit target by restricting theproximal RL objectiveto theresponse simplex, and then projects the policy via exactdivergence minimization. This framework provides (i)monotonic improvementon the listwise objective with bounded, zero-sum, and self-correctingprojection gradients, and (ii) flexibility in divergence selection with distinct structural properties through the decoupled projection step. On diverse reasoning tasks and LLM backbones, LPO consistently improves training performance over typicalpolicy gradientbaselines under matched targets, while intrinsically preserving optimization stability and response diversity.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2605\.06139

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.06139 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.06139 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.06139 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVR

arXiv cs.AI

The paper proposes OM-GRPO, a label-free RLVR framework that decouples reward estimation from policy optimization by masking gradients on answer spans, and introduces Contrast-Augmented Reward to refine reward estimation without extra rollouts. It consistently outperforms existing label-free methods and matches supervised ground-truth reward training across reasoning benchmarks.

Gradient Extrapolation-Based Policy Optimization

arXiv cs.LG

The article introduces Gradient Extrapolation-Based Policy Optimization (GXPO), a method that approximates multi-step lookahead in RL training for LLMs using only three backward passes. It demonstrates improved reasoning performance on math benchmarks over standard GRPO while maintaining fixed active-phase costs.

Parameter Exploration for RLVR via Variational Learning

Hugging Face Daily Papers

This paper introduces Perturbed Parameter Policy Optimization (3PO), a family of parameter-space exploration methods for LLM reinforcement learning, showing consistent improvements over GRPO on math and code tasks.