PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic Tasks

arXiv cs.AI Papers
reinforcement-learning llm policy-optimization multi-turn agentic-tasks credit-assignment

Summary

PGPO proposes potential-guided policy optimization for multi-turn agentic tasks, enabling finer-grained credit assignment in LLM post-training and showing strong results on ALFWorld and WebShop benchmarks.

arXiv:2609.02236v1 Announce Type: new Abstract: Group-based reinforcement learning (RL) has become an effective paradigm for LLM post-training, but in multi-turn agentic tasks with sparse terminal rewards, it often provides coarse credit for intermediate actions. To obtain more fine-grained credit assignment, recent work such as GiGPO introduces step-level advantages for intermediate actions. However, these step-level signals still rely on the final outcome of each individual trajectory. As a result, actions within failed trajectories can remain poorly differentiated, so effective actions can receive the same unfavorable credit as erroneous ones. In this work, we propose Potential-Guided Policy Optimization (PGPO) for multi-turn agentic tasks. PGPO estimates empirical state potentials from anchor-state-group return statistics within each rollout group. It then derives action advantages from potential differences between adjacent states, enabling cross-trajectory credit propagation. This provides finer-grained step-level credit assignment, especially within failed trajectories. Experiments on ALFWorld and WebShop show strong overall performance relative to recent group-based RL methods. Further analysis provides evidence that PGPO yields more informative failure-side credit signals with negligible training overhead.
Original Article
View Cached Full Text

Cached at: 09/03/26, 06:03 AM 

# PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic Tasks
Source: [https://arxiv.org/abs/2609.02236](https://arxiv.org/abs/2609.02236)
[View PDF](https://arxiv.org/pdf/2609.02236)

> Abstract:Group\-based reinforcement learning \(RL\) has become an effective paradigm for LLM post\-training, but in multi\-turn agentic tasks with sparse terminal rewards, it often provides coarse credit for intermediate actions\. To obtain more fine\-grained credit assignment, recent work such as GiGPO introduces step\-level advantages for intermediate actions\. However, these step\-level signals still rely on the final outcome of each individual trajectory\. As a result, actions within failed trajectories can remain poorly differentiated, so effective actions can receive the same unfavorable credit as erroneous ones\. In this work, we propose Potential\-Guided Policy Optimization \(PGPO\) for multi\-turn agentic tasks\. PGPO estimates empirical state potentials from anchor\-state\-group return statistics within each rollout group\. It then derives action advantages from potential differences between adjacent states, enabling cross\-trajectory credit propagation\. This provides finer\-grained step\-level credit assignment, especially within failed trajectories\. Experiments on ALFWorld and WebShop show strong overall performance relative to recent group\-based RL methods\. Further analysis provides evidence that PGPO yields more informative failure\-side credit signals with negligible training overhead\.

## Submission history

From: Yuyao Zheng \[[view email](https://arxiv.org/show-email/0022beab/2609.02236)\] **\[v1\]**Wed, 2 Sep 2026 07:44:16 UTC \(2,881 KB\)

Similar Articles