Contextual Credit Policy Optimization

# Contextual Credit Policy Optimization
Source: [https://arxiv.org/html/2610.19103](https://arxiv.org/html/2610.19103)
Huiwen Han1,2,†, Ziyao Zeng3, Zihao Zhu2, Sisong Bei4, Hao Wang2, Hongyu Xu5, Yan Han6, Degui Zhi3, Zhiwen Fan2Affiliation:1Stanford University,2Texas A&M University,3Yale University,4Amazon,5Meta AI,6MicrosoftAffiliation:†Project lead

###### Abstract

Group\-based reinforcement learning enables large language model agents to learn from sparse outcome rewards in long\-horizon tasks\. Recent methods derive step\-level credit by comparing observation\-matched experiences across trajectories\. However, identical observations can occur in different contexts, and matched peers are not equally informative for a given decision\. Moreover, favorable terminal outcomes can obscure redundant actions or detours within successful trajectories\. To address these two challenges, we proposeContextual Credit Policy Optimization \(CCPO\)\. For the first, CCPO weights observation\-matched peers from other trajectories by contextual similarity, computed from frozen LLM representations and accumulated trajectory statistics, allowing peers from similar contexts to contribute more to a step’s baseline\. For the second, CCPO pairs historical credit, which compares the realized return against the weighted peer baseline, with future credit, which measures the change in contextual outcome value over a short realized continuation\. A step is thus credited for the progress it makes, not only for its trajectory’s eventual outcome\. Both credits are computed in hindsight from rollouts already collected for training, enabling CCPO to address these challenges without an auxiliary value critic\. Across two Qwen2\.5\-Instruct backbones \(1\.5B and 7B\), CCPO achieves up to 97\.4% success on ALFWorld and 79\.4% on WebShop and outperforms GRPO by up to 21\.6 and 18\.5 points\. Ablations show that contextual weighting outperforms uniform weighting, and that historical and future credit serve as complementary learning signals\.

Figure 1:Overview of CCPO\.CCPO combines contextually weighted peer comparisons and short\-horizon future progress for critic\-free step\-level credit assignment\.## 1Introduction

Large language model \(LLM\) agents increasingly undertake long\-horizon tasks through sustained environment interaction\([Wang et al\., 2024](https://arxiv.org/html/2610.19103#bib.bib48);[Zhang et al\., 2026](https://arxiv.org/html/2610.19103#bib.bib49)\)\. Benchmarks such as WebArena\([Zhou et al\., 2024a](https://arxiv.org/html/2610.19103#bib.bib45)\)and SWE\-bench\([Jimenez et al\., 2024](https://arxiv.org/html/2610.19103#bib.bib46)\)require information gathering, coordinated actions, and adaptation to feedback\. Reinforcement learning \(RL\)\([Sutton and Barto, 2018](https://arxiv.org/html/2610.19103#bib.bib37)\)provides a post\-training framework that optimizes policies using rewards from interaction trajectories\. Recent applications span open\-web research, web navigation, and multi\-step software engineering\([Zheng et al\., 2025b](https://arxiv.org/html/2610.19103#bib.bib10);[Qi et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib8);[Luo et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib12)\)\. However, feedback often arrives only after many decisions\.

Learning from such delayed feedback depends on how it is translated into policy updates\. PPO\-based actor–critic methods use learned value estimates to guide optimization, at the cost of training an additional value model\([Schulman et al\., 2017](https://arxiv.org/html/2610.19103#bib.bib39)\)\. Alternatively, group\-based methods such as GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.19103#bib.bib43)\)estimate relative advantages from multiple sampled outcomes without a separately learned value critic\. When applied at the trajectory level, however, a single advantage is shared across all actions in a rollout\. Such comparisons differentiate outcomes across trajectories but provide limited information about the contributions of individual decisions, especially when rewards are sparse and delayed\([Zhang et al\., 2026](https://arxiv.org/html/2610.19103#bib.bib49)\)\. To enable fine\-grained policy updates, recent methods extend this approach to construct step\-level advantage estimates: GiGPO\([Feng et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib24)\)compares actions encountered at shared observations, and G2PO\([Wang et al\., 2026d](https://arxiv.org/html/2610.19103#bib.bib26)\)incorporates the structure of transitions between observation groups\. HGPO\([He et al\., 2026](https://arxiv.org/html/2610.19103#bib.bib25)\)accounts for historical context by organizing comparisons into hierarchical groups\. In each case, step\-level credit comes from comparing experiences in existing rollouts, so it depends on how they are compared\.

Two challenges remain when steps are compared at matched observations\. The first is whom a step should be compared with\. Identical observations can occur after different preceding actions, reasoning, exploration depth, and accumulated contexts\. Peers that share an observation are therefore not equally informative for the step being evaluated \(the query\)\. Requiring peers to share the same history instead leaves few peers per step, and a baseline averaged over so few returns gives noisy advantages\([He et al\., 2026](https://arxiv.org/html/2610.19103#bib.bib25)\)\. Our premise is that, within observation\-matched groups, contextual similarity should determine each peer’s contribution to the query’s baseline and value estimates\. The second is what a step should be credited for\. A return comparison credits a step for belonging to a trajectory that ended well, but an eventual success can include redundant actions or detours\. Separating a step that inherits a favorable context from one that makes progress from it requires a complementary comparison of realized gains over a short continuation\.

To address both challenges, we proposeContextual Credit Policy Optimization \(CCPO\)\. For the first, CCPO represents each step’s context with semantic features of an interaction window from a frozen LLM and accumulated trajectory statistics, and gives greater weight to observation\-matched peers with higher contextual similarity \(Figure[1](https://arxiv.org/html/2610.19103#S0.F1)\)\. For the second, CCPO builds two credit signals on these weighted peers\. Historical credit compares the realized return with its contextual peer baseline\. Future credit measures short\-horizon progress as the change in contextual value, a weighted average of peers’ discounted terminal rewards, between the current step and a later step of the same rollout\. Both are computed in hindsight from collected rollouts without requiring an auxiliary value critic or additional environment interactions\. On agentic multi\-step benchmarks, ALFWorld\([Shridhar et al\., 2021](https://arxiv.org/html/2610.19103#bib.bib31)\)and WebShop\([Yao et al\., 2022](https://arxiv.org/html/2610.19103#bib.bib32)\), CCPO outperforms prompting\- and RL\-based baselines using Qwen2\.5\-1\.5B\-Instruct and Qwen2\.5\-7B\-Instruct backbones\([Qwen et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib33)\)\.

Our main contributions are as follows:

- •Contextual credit estimation from history and future progress\.We assign step\-level credit by combining a history\-conditioned return residual with estimated value change over a future horizon, capturing outcome\-relative performance and local progress\. Analysis shows complementary effects of history and future credit as learning evolves\.
- •Contextual similarity for step\-level peer weighting\.We aggregate peer returns with continuous weights set by the similarity between query and peer context representations, producing a context\-dependent baseline\. Replacing these weights with uniform ones lowers success by 3\.7 and 2\.6 points\.
- •Semantic and structural context representations\.We combine semantic hidden states of recent interactions with accumulated trajectory statistics, enabling context\-sensitive comparisons across trajectories without training an auxiliary value critic\.

## 2Related Work

#### Agentic reinforcement learning for long\-horizon tasks\.

Reinforcement learning \(RL\)\([Sutton and Barto, 2018](https://arxiv.org/html/2610.19103#bib.bib37)\)enables large language model \(LLM\) agents to improve through extended interactions in which observations, reasoning, actions, and feedback accumulate into trajectories\([Zhou et al\., 2024b](https://arxiv.org/html/2610.19103#bib.bib4);[Zhang et al\., 2026](https://arxiv.org/html/2610.19103#bib.bib49)\)\. Applications span interactive gaming, web navigation, open\-web research, and multi\-step software engineering\([Wang et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib13);[Qi et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib8);[Zheng et al\., 2025b](https://arxiv.org/html/2610.19103#bib.bib10);[Li et al\., 2026b](https://arxiv.org/html/2610.19103#bib.bib53);[Luo et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib12)\), where agents act in an environment and adapt to its feedback over many steps\. Another line of work trains agents to reason with external tools, including search engines, web browsing tools, and code interpreters\([Jin et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib9);[Tongyi DeepResearch Team and others, 2026](https://arxiv.org/html/2610.19103#bib.bib52);[Feng et al\., 2026](https://arxiv.org/html/2610.19103#bib.bib11)\)\. Agentic RL can also learn from successes and failures through verbal reflection and trajectory preference learning\([Shinn et al\., 2023](https://arxiv.org/html/2610.19103#bib.bib2);[Yao et al\., 2024](https://arxiv.org/html/2610.19103#bib.bib3);[Song et al\., 2024](https://arxiv.org/html/2610.19103#bib.bib5);[Putta et al\., 2024](https://arxiv.org/html/2610.19103#bib.bib6)\)\. To collect high\-quality trajectories, methods use uncertainty\-guided branching and resampling\([Dong et al\., 2026](https://arxiv.org/html/2610.19103#bib.bib15);[Wang et al\., 2026a](https://arxiv.org/html/2610.19103#bib.bib16)\)\. Reward design\([Wang et al\., 2026e](https://arxiv.org/html/2610.19103#bib.bib14)\)is also crucial, spanning outcome\-supervised reward models\([Qi et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib8)\), process rewards from environment\-aware models such as DataPRM\([Qiu et al\., 2026](https://arxiv.org/html/2610.19103#bib.bib55)\), and turn\-level reward assignment\([Tao et al\., 2026](https://arxiv.org/html/2610.19103#bib.bib30)\)\. Despite these advances, sparse and delayed rewards, costly environment interaction, and training instability remain challenges\([Tao et al\., 2026](https://arxiv.org/html/2610.19103#bib.bib30);[Li et al\., 2026b](https://arxiv.org/html/2610.19103#bib.bib53)\), motivating efficient and stable policy optimization from trajectories already collected\.

#### Policy optimization for LLM agents\.

Existing approaches differ in their optimization objectives and advantage estimators\. Preference\-based methods, including DPO\([Rafailov et al\., 2023](https://arxiv.org/html/2610.19103#bib.bib41)\), ETO\([Song et al\., 2024](https://arxiv.org/html/2610.19103#bib.bib5)\), and Agent Q\([Putta et al\., 2024](https://arxiv.org/html/2610.19103#bib.bib6)\), optimize policies from response or trajectory preferences, including successful and failed agent rollouts, without explicit reward\-model fitting\. Actor–critic methods use learned value estimates, as in PPO\-based reinforcement learning from human feedback \(RLHF\) with InstructGPT\([Ouyang et al\., 2022](https://arxiv.org/html/2610.19103#bib.bib40)\)and multi\-step agent learning with ArCHer\([Zhou et al\., 2024b](https://arxiv.org/html/2610.19103#bib.bib4)\)\. To avoid a separate value critic, group\-based methods estimate advantages from multiple sampled responses or trajectories\. RLOO\([Ahmadian et al\., 2024](https://arxiv.org/html/2610.19103#bib.bib42)\)uses the mean reward of other samples as a baseline, while GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.19103#bib.bib43)\), used in DeepSeek\-R1\([Guo et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib44)\), standardizes group rewards\. Related variants include LOOP\([Chen et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib7)\)with leave\-one\-out PPO updates and Dr\. GRPO\([Liu et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib17)\)with revised length and reward normalization\. Further refinements target update stability and exploration efficiency through clipping, importance weighting, and sampling, including DAPO\([Yu et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib18)\), GSPO\([Zheng et al\., 2025a](https://arxiv.org/html/2610.19103#bib.bib20)\), CISPO\([MiniMax and others, 2025](https://arxiv.org/html/2610.19103#bib.bib21)\), SAPO\([Gao et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib22)\), VESPO\([Shen et al\., 2026](https://arxiv.org/html/2610.19103#bib.bib23)\), SAMPO\([Wang et al\., 2026b](https://arxiv.org/html/2610.19103#bib.bib51)\), Clip\-Cov\([Cui et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib19)\), and T2PO\([Wang et al\., 2026a](https://arxiv.org/html/2610.19103#bib.bib16)\)\. However, trajectory\-level applications of outcome\-supervised GRPO share a single advantage across all actions in a rollout, without explicitly differentiating individual contributions\([Feng et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib24)\)\. This motivates the step\-level methods discussed next\.

#### Step\-level credit assignment\.

Recent methods construct step\-level learning signals for multi\-step interactions\([Zhang, 2026](https://arxiv.org/html/2610.19103#bib.bib54)\)\. Critic\-based methods include SWEET\-RL\([Zhou et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib28)\)with additional training\-time information and HCAPO\([Tan et al\., 2026](https://arxiv.org/html/2610.19103#bib.bib29)\)with the agent as a hindsight critic\. TRACE\([Tao et al\., 2026](https://arxiv.org/html/2610.19103#bib.bib30)\)derives turn\-level rewards from a frozen model’s gold\-answer likelihoods\. OpenClaw\-RL\([Wang et al\., 2026c](https://arxiv.org/html/2610.19103#bib.bib56)\)combines evaluative rewards and corrective hints extracted from next\-state feedback\. Other methods estimate advantages by comparing collected trajectories\. GiGPO\([Feng et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib24)\)combines episode\-level advantages with comparisons at matching observations, while HGPO\([He et al\., 2026](https://arxiv.org/html/2610.19103#bib.bib25)\)aggregates advantages from hierarchical groups matched at different history lengths\. SALT\([Li et al\., 2026a](https://arxiv.org/html/2610.19103#bib.bib27)\)constructs graphs from trajectories of the same prompt, while G2PO\([Wang et al\., 2026d](https://arxiv.org/html/2610.19103#bib.bib26)\)aggregates observations into a state\-transition graph to estimate values and advantages\. GEPO\([Yuan et al\., 2026](https://arxiv.org/html/2610.19103#bib.bib57)\)uses task\-conditioned graph centrality to shape trajectory returns and scale step\-level advantages\. Peer matching faces a granularity trade\-off: observation\-only matching can mix different histories, whereas exact history matching reduces the available samples\([He et al\., 2026](https://arxiv.org/html/2610.19103#bib.bib25)\)\. CCPO keeps all observation\-matched peers and weights each by the similarity of its context to the query’s, characterized by local semantic and accumulated trajectory features, thus enabling fine\-grained comparisons utilizing available peers\.

## 3Contextual Credit Policy Optimization

Figure 2:Contextual credit policy optimization of multi\-step tasks\.Contextual similarity weights observation\-matched peers\. A history\-conditioned return residual and estimated outcome\-value change over a short future window yield step\-level credit from collected rollouts without an auxiliary value critic\.### 3\.1Problem Formulation

Notation is summarized in Appendix[A](https://arxiv.org/html/2610.19103#A1)\. We sampleNNtrajectories per task from behavior policyπold\\pi\_\{\\mathrm\{old\}\}\. Trajectoryiifor taskxix\_\{i\}containsTiT\_\{i\}actionsai,0:Ti−1a\_\{i,0:T\_\{i\}\-1\}, observationsoi,0:Tio\_\{i,0:T\_\{i\}\}, and terminal outcomeRiR\_\{i\}\. Promptpi,tp\_\{i,t\}containsxix\_\{i\},oi,to\_\{i,t\}, and up toKHK\_\{\\mathrm\{H\}\}preceding observation–action pairs\. Rewards come from environment feedback\. GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.19103#bib.bib43)\)uses the mean reward of group rollouts as a baseline to estimate normalized advantages without a learned value critic\. CCPO extends this critic\-free estimation to individual steps, weighting peers by contextual similarity to estimate return baselines\. The return target used by historical credit is

Yi,t=∑u=tTi−1γu−t​ri,uenv−η​1​\{ai,t​is invalid\}\.Y\_\{i,t\}=\\sum\_\{u=t\}^\{T\_\{i\}\-1\}\\gamma^\{u\-t\}r^\{\\mathrm\{env\}\}\_\{i,u\}\-\\eta\\,\\mathbf\{1\}\\\{a\_\{i,t\}\\text\{ is invalid\}\\\}\.\(1\)Hereri,uenvr^\{\\mathrm\{env\}\}\_\{i,u\}follows actionai,ua\_\{i,u\}andγ\\gammadiscounts rewards\. The penalty term subtractsη\\etafor an invalid action at stepttonly \(Appendix[C\.3](https://arxiv.org/html/2610.19103#A3.SS3)\)\.

### 3\.2Contextual Peer Estimation

CCPO assigns continuous, query\-specific weights to individual peers based on contextual similarity \(Figure[2](https://arxiv.org/html/2610.19103#S3.F2)\)\. Averaging uniformly within an observation\-matched group ignores that identical observations can follow different histories\. Soft weighting within observation\-matched groups accounts for these differences without exact history matching, which can leave few peers\([He et al\., 2026](https://arxiv.org/html/2610.19103#bib.bib25)\)\.

#### Peer selection\.

Exact peers share the task and observation and exclude the entire trajectory of the query step:

ℰi,t=\{\(j,s\):xj=xi,j≠i,0≤s<Tj,oj,s=oi,t\}\.\\mathcal\{E\}\_\{i,t\}=\\\{\(j,s\):x\_\{j\}=x\_\{i\},\\ j\\neq i,\\ 0\\leq s<T\_\{j\},\\ o\_\{j,s\}=o\_\{i,t\}\\\}\.\(2\)We use𝒫i,t=ℰi,t\\mathcal\{P\}\_\{i,t\}=\\mathcal\{E\}\_\{i,t\}when nonempty; otherwise, we use all same\-task steps withj≠ij\\neq i\. We quantify fallback rates and peer availability during training in Appendix[H](https://arxiv.org/html/2610.19103#A8)\.

#### Context representation\.

Each step combines a frozen LLM representationhi,th\_\{i,t\}with a trajectory summaryci,tc\_\{i,t\}\. The representation is the final\-layer hidden state of the last prompt token\. The summary records elapsed steps, distinct\-observation count, novelty fraction, and a revisit indicator through steptt\. Because the prompt holds only the most recent steps, the hidden state cannot reflect earlier history, and the summary supplies it\. We L2\-normalize each preprocessed component, concatenate them with equal weights, and L2\-normalize to obtainϕi,t\\phi\_\{i,t\}\.

#### Contextual readout\.

BucketBBcontains all batch steps matching the query’s task and observation, or task alone for fallback\. We aggregate scalar targetsrrover nonempty𝒫i,t\\mathcal\{P\}\_\{i,t\}:

w¯\(i,t\),\(j,s\)∝exp\(−∥ϕi,t−ϕj,s∥2/τB\),∑\(j,s\)∈𝒫i,tw¯\(i,t\),\(j,s\)=1\.\\bar\{w\}\_\{\(i,t\),\(j,s\)\}\\propto\\exp\\bigl\(\-\\\|\\phi\_\{i,t\}\-\\phi\_\{j,s\}\\\|\_\{2\}/\\tau\_\{B\}\\bigr\),\\qquad\\sum\_\{\(j,s\)\\in\\mathcal\{P\}\_\{i,t\}\}\\bar\{w\}\_\{\(i,t\),\(j,s\)\}=1\.\(3\)ℬi,t​\[r\]=∑\(j,s\)∈𝒫i,tw¯\(i,t\),\(j,s\)​rj,s\.\\mathcal\{B\}\_\{i,t\}\[r\]=\\sum\_\{\(j,s\)\\in\\mathcal\{P\}\_\{i,t\}\}\\bar\{w\}\_\{\(i,t\),\(j,s\)\}r\_\{j,s\}\.\(4\)The bandwidthτB\\tau\_\{B\}is proportional to the median pairwise context distance inBB, computed before trajectory exclusion\. This readout estimates both return baselines and outcome values\.

### 3\.3Temporal Credit Estimation

CCPO conditions return comparisons on aKHK\_\{\\mathrm\{H\}\}\-step history window and assesses progress over a short realized continuation ofKFK\_\{\\mathrm\{F\}\}steps\. The future window compares discounted contextual outcome values at its endpoints\. Historical credit depends on how the step’s trajectory ends, whereas future credit asks whether the step itself moved the trajectory to a more valuable context\.

#### Historical credit\.

We subtract the contextual peer baseline from the return:

Ci,tH=Yi,t−ℬi,t​\[Y\]\.C^\{\\mathrm\{H\}\}\_\{i,t\}=Y\_\{i,t\}\-\\mathcal\{B\}\_\{i,t\}\[Y\]\.\(5\)

#### Future credit\.

We estimate contextual values from unpenalized terminal outcomes:

Zi,s=γTi−s​Ri,Vi,s=\{ℬi,s​\[Z\],s<Ti,Ri,s=Ti\.Z\_\{i,s\}=\\gamma^\{T\_\{i\}\-s\}R\_\{i\},\\qquad V\_\{i,s\}=\\begin\{cases\}\\mathcal\{B\}\_\{i,s\}\[Z\],&s<T\_\{i\},\\\\ R\_\{i\},&s=T\_\{i\}\.\\end\{cases\}\(6\)For future horizonKFK\_\{\\mathrm\{F\}\}, lett\+=min⁡\(t\+KF,Ti\)t^\{\+\}=\\min\(t\+K\_\{\\mathrm\{F\}\},T\_\{i\}\)\. We compute

Δ​Vi,t=Vi,t\+−Vi,t,Ci,tF=zxi​\[Δ​V\]i,t,\\Delta V\_\{i,t\}=V\_\{i,t^\{\+\}\}\-V\_\{i,t\},\\qquad C^\{\\mathrm\{F\}\}\_\{i,t\}=z\_\{x\_\{i\}\}\[\\Delta V\]\_\{i,t\},\(7\)wherezxz\_\{x\}standardizes eligible differences within taskxx\. Each nonterminal endpoint uses its own context and peer set, and future endpoints are used only during training\. The endpoint difference measures growth in discounted contextual value, including changes due to shorter remaining time to success\. Compared with a discount\-aligned TD residual\([Schulman et al\., 2016](https://arxiv.org/html/2610.19103#bib.bib38)\), this difference includes an extra nonnegative term that grows with the endpoint value and can survive task\-wise centering, so future credit also rewards being closer to the end of a successful trajectory \(Appendix[C\.3](https://arxiv.org/html/2610.19103#A3.SS3)\)\.

### 3\.4Policy Optimization

We add historical credit and task\-standardized future credit with unit coefficients:

Ai,tCC=𝒩xi​\[CH\+CF\]i,t\.A^\{\\mathrm\{CC\}\}\_\{i,t\}=\\mathcal\{N\}\_\{x\_\{i\}\}\[C^\{\\mathrm\{H\}\}\+C^\{\\mathrm\{F\}\}\]\_\{i,t\}\.\(8\)The normalized contributions of the two credits, defined in Appendix[C\.3](https://arxiv.org/html/2610.19103#A3.SS3), satisfyACC=AH\+AFA^\{\\mathrm\{CC\}\}=A^\{\\mathrm\{H\}\}\+A^\{\\mathrm\{F\}\}\. The default usesAi,t=Ai,tCCA\_\{i,t\}=A^\{\\mathrm\{CC\}\}\_\{i,t\}\. To test whether trajectory\-level signals complement step\-level credit, the episode\-augmented variant adds a GRPO\-style episode advantage\([Shao et al\., 2024](https://arxiv.org/html/2610.19103#bib.bib43)\),Ai,t=Ai,tCC\+Ai,tEPA\_\{i,t\}=A^\{\\mathrm\{CC\}\}\_\{i,t\}\+A^\{\\mathrm\{EP\}\}\_\{i,t\}, with unit weight and no further normalization\. Each step’s advantage weights response tokens in a dual\-clipped objective\([Ye et al\., 2020](https://arxiv.org/html/2610.19103#bib.bib47)\), with a KL penalty toward the frozen reference policy\. See Appendix[C](https://arxiv.org/html/2610.19103#A3)for details\.

## 4Experiments

### 4\.1Experimental Setup

#### Benchmarks and metrics\.

We evaluate CCPO against baseline methods on ALFWorld\([Shridhar et al\., 2021](https://arxiv.org/html/2610.19103#bib.bib31)\)and WebShop\([Yao et al\., 2022](https://arxiv.org/html/2610.19103#bib.bib32)\), covering household task execution and online shopping through multi\-step interaction\. ALFWorld comprises six household task categories, with 3,553 training tasks and 140 held\-out tasks of the in\-distribution evaluation split\. For WebShop, we follow the goal partition implemented in verl\-agent\([Feng et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib24)\), reserving the first 500 goals as the evaluation pool and sampling training tasks from the remaining goals\. Following prior work\([Feng et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib24);[Wang et al\., 2026d](https://arxiv.org/html/2610.19103#bib.bib26)\), we report overall and per\-category success rates on ALFWorld\. For WebShop, we report success rate and mean task score\.

#### Models and baselines\.

We use Qwen2\.5\-1\.5B\-Instruct and Qwen2\.5\-7B\-Instruct\([Qwen et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib33)\)as policy backbones\. We compare CCPO with three categories of baselines: \(1\)Off\-the\-shelf LLMs:Gemini 3\.8 Flash\([Google DeepMind, 2026](https://arxiv.org/html/2610.19103#bib.bib34)\), DeepSeek\-V4\.1\-Flash\([DeepSeek\-AI and others, 2026](https://arxiv.org/html/2610.19103#bib.bib35)\), and Qwen3\.8\-27B\([Qwen Team, 2026](https://arxiv.org/html/2610.19103#bib.bib36)\)\. \(2\)Prompt\-based agents:the Qwen2\.5\-Instruct backbones without task\-specific RL fine\-tuning, ReAct\([Yao et al\., 2023](https://arxiv.org/html/2610.19103#bib.bib1)\), and Reflexion\([Shinn et al\., 2023](https://arxiv.org/html/2610.19103#bib.bib2)\)\. \(3\)RL training methods:PPO\([Schulman et al\., 2017](https://arxiv.org/html/2610.19103#bib.bib39)\), RLOO\([Ahmadian et al\., 2024](https://arxiv.org/html/2610.19103#bib.bib42)\), GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.19103#bib.bib43)\), GiGPO\([Feng et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib24)\), HGPO\([He et al\., 2026](https://arxiv.org/html/2610.19103#bib.bib25)\), and G2PO\([Wang et al\., 2026d](https://arxiv.org/html/2610.19103#bib.bib26)\)\. Baseline configurations are detailed in Appendix[B](https://arxiv.org/html/2610.19103#A2)\.

#### Implementation details\.

We train CCPO using verl\-agent\([Feng et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib24)\), built on veRL\([Sheng et al\., 2025](https://arxiv.org/html/2610.19103#bib.bib50)\), for 150 iterations, sampling 16 tasks with eight trajectories each per iteration\. We use a learning rate of10−610^\{\-6\}and discount factor 0\.95\. Interaction limits are 50 steps for ALFWorld and 15 for WebShop\. Settings are shared across both backbones and all group\-based baselines, except that HGPO retains its published limits of 4,096 prompt tokens on ALFWorld and 30 interaction steps on WebShop\. CCPO uses aKH=2K\_\{H\}=2history window and aKF=2K\_\{F\}=2future horizon\. We evaluate two CCPO variants, a default and an episode\-augmented one\. The default usesACCA\_\{\\mathrm\{CC\}\}from Equation \([8](https://arxiv.org/html/2610.19103#S3.E8)\)\. The variant usesACC\+AEPA\_\{\\mathrm\{CC\}\}\+A\_\{\\mathrm\{EP\}\}, adding a standard episode\-level advantage without further normalization\. All training and evaluation are completed on 8×\\timesH200 GPUs\. We report means and standard deviations across three random seeds\. Appendix[B](https://arxiv.org/html/2610.19103#A2)gives the shared training budgets and optimization settings\. Appendix[D](https://arxiv.org/html/2610.19103#A4)provides an ALFWorld prompt and response example\.

### 4\.2Main Results

#### Comparison with baselines\.

Across CCPO variants, mean success gains over GRPO are 16\.6–21\.6 percentage points on ALFWorld and 12\.3–18\.5 on WebShop at 1\.5B\. At 7B, they are 12\.3–15\.4 and 5\.5–6\.5 points \(Table[1](https://arxiv.org/html/2610.19103#S4.T1)\)\. WebShop score gains are 13\.1–14\.8 at 1\.5B and 1\.7–6\.5 at 7B\. CCPO leads the listed RL baselines in mean success, reaching 96\.9% and 97\.4% on ALFWorld and 75\.0% and 79\.4% on WebShop at 1\.5B and 7B\.

Table 1:Results on WebShop and ALFWorld\.We report the success rate \(%\) for each sub\-task and overall results for ALFWorld, and the success rate \(%\) and score for WebShop, averaged over 3 seeds with standard deviations\. Within each section, thebestmeans are highlighted and thesecond bestare underlined\. Higher is better\.\*marks baselines reproduced or off\-the\-shelf models evaluated by us\. Light\-gray rows show results from the original papers and are excluded from ranking highlights\.Δ\\DeltaGRPO gives the min–max gain over GRPO across both CCPO variants\.TypeMethodALFWorldWebShopAllPickLookCleanHeatCoolPick2Succ\.ScoreOff\-the\-Shelf ModelPromptingGemini 3\.8 Flash\*96\.4\\mathbf\{96\.4\}100\.0\\mathbf\{100\.0\}84\.6\\mathbf\{84\.6\}100\.0\\mathbf\{100\.0\}100\.0\\mathbf\{100\.0\}92\.0\\mathbf\{92\.0\}95\.8\\mathbf\{95\.8\}49\.2\\mathbf\{49\.2\}54\.0\\mathbf\{54\.0\}PromptingDeepSeek\-V4\.1\-Flash\*75\.0¯\\underline\{75\.0\}88\.6¯\\underline\{88\.6\}84\.6\\mathbf\{84\.6\}66\.7¯\\underline\{66\.7\}62\.5¯\\underline\{62\.5\}76\.0¯\\underline\{76\.0\}66\.7¯\\underline\{66\.7\}22\.022\.027\.927\.9PromptingQwen3\.8\-27B\*70\.770\.785\.785\.761\.5¯\\underline\{61\.5\}66\.7¯\\underline\{66\.7\}56\.356\.372\.072\.066\.7¯\\underline\{66\.7\}29\.0¯\\underline\{29\.0\}39\.5¯\\underline\{39\.5\}Qwen2\.5\-1\.5B\-InstructPromptingQwen2\.5\*6\.46\.411\.411\.47\.77\.73\.73\.76\.36\.34\.04\.04\.24\.25\.05\.021\.921\.9PromptingReAct12\.812\.817\.417\.420\.520\.515\.715\.76\.26\.27\.77\.72\.02\.011\.311\.340\.140\.1PromptingReflexion21\.821\.835\.335\.322\.222\.221\.721\.713\.613\.619\.419\.43\.73\.721\.921\.955\.855\.8RL TrainingPPO \(with critic\)54\.4±3\.154\.4\_\{\\pm 3\.1\}64\.8±3\.564\.8\_\{\\pm 3\.5\}40\.5±6\.940\.5\_\{\\pm 6\.9\}57\.1±4\.957\.1\_\{\\pm 4\.9\}60\.6±6\.660\.6\_\{\\pm 6\.6\}46\.4±4\.046\.4\_\{\\pm 4\.0\}47\.4±1\.947\.4\_\{\\pm 1\.9\}51\.5±2\.951\.5\_\{\\pm 2\.9\}73\.8±3\.073\.8\_\{\\pm 3\.0\}RL TrainingRLOO69\.7±2\.569\.7\_\{\\pm 2\.5\}88\.3±3\.088\.3\_\{\\pm 3\.0\}52\.8±8\.652\.8\_\{\\pm 8\.6\}71\.0±5\.971\.0\_\{\\pm 5\.9\}62\.8±8\.762\.8\_\{\\pm 8\.7\}66\.4±5\.566\.4\_\{\\pm 5\.5\}56\.9±4\.756\.9\_\{\\pm 4\.7\}52\.1±6\.752\.1\_\{\\pm 6\.7\}73\.9±5\.673\.9\_\{\\pm 5\.6\}RL TrainingGRPO\*75\.3±1\.075\.3\_\{\\pm 1\.0\}81\.8±0\.881\.8\_\{\\pm 0\.8\}66\.5±8\.766\.5\_\{\\pm 8\.7\}77\.2±0\.477\.2\_\{\\pm 0\.4\}73\.8±3\.473\.8\_\{\\pm 3\.4\}78\.0±3\.078\.0\_\{\\pm 3\.0\}65\.7±1\.965\.7\_\{\\pm 1\.9\}56\.5±2\.956\.5\_\{\\pm 2\.9\}70\.5±2\.270\.5\_\{\\pm 2\.2\}RL TrainingGiGPO\*83\.9±0\.783\.9\_\{\\pm 0\.7\}84\.8±0\.784\.8\_\{\\pm 0\.7\}77\.0±10\.077\.0\_\{\\pm 10\.0\}92\.4±0\.192\.4\_\{\\pm 0\.1\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}75\.0±0\.975\.0\_\{\\pm 0\.9\}74\.6±1\.974\.6\_\{\\pm 1\.9\}67\.4±2\.267\.4\_\{\\pm 2\.2\}80\.1±1\.880\.1\_\{\\pm 1\.8\}RL TrainingGiGPO86\.7±1\.786\.7\_\{\\pm 1\.7\}94\.4±5\.994\.4\_\{\\pm 5\.9\}67\.5±4\.667\.5\_\{\\pm 4\.6\}94\.8±3\.894\.8\_\{\\pm 3\.8\}94\.4±7\.894\.4\_\{\\pm 7\.8\}79\.8±4\.779\.8\_\{\\pm 4\.7\}76\.4±5\.476\.4\_\{\\pm 5\.4\}65\.0±3\.265\.0\_\{\\pm 3\.2\}83\.1±1\.683\.1\_\{\\pm 1\.6\}RL TrainingHGPO\*89\.1±0\.689\.1\_\{\\pm 0\.6\}93\.9¯±0\.3\\underline\{93\.9\}\_\{\\pm 0\.3\}80\.1±6\.080\.1\_\{\\pm 6\.0\}92\.4±0\.192\.4\_\{\\pm 0\.1\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}88\.3±1\.688\.3\_\{\\pm 1\.6\}76\.2±1\.676\.2\_\{\\pm 1\.6\}65\.4±1\.565\.4\_\{\\pm 1\.5\}82\.1±0\.882\.1\_\{\\pm 0\.8\}RL TrainingHGPO92\.8±1\.192\.8\_\{\\pm 1\.1\}Unreported71\.5±4\.071\.5\_\{\\pm 4\.0\}85\.6±2\.985\.6\_\{\\pm 2\.9\}RL TrainingG2PO\*94\.0¯±0\.7\\underline\{94\.0\}\_\{\\pm 0\.7\}93\.9¯±0\.3\\underline\{93\.9\}\_\{\\pm 0\.3\}90\.6¯±7\.4\\underline\{90\.6\}\_\{\\pm 7\.4\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}92\.7¯±1\.8\\underline\{92\.7\}\_\{\\pm 1\.8\}86\.6¯±0\.3\\underline\{86\.6\}\_\{\\pm 0\.3\}71\.6¯±0\.7\\underline\{71\.6\}\_\{\\pm 0\.7\}83\.5±0\.883\.5\_\{\\pm 0\.8\}RL TrainingG2PO95\.0±0\.895\.0\_\{\\pm 0\.8\}97\.2±3\.997\.2\_\{\\pm 3\.9\}96\.3±5\.296\.3\_\{\\pm 5\.2\}95\.1±1\.895\.1\_\{\\pm 1\.8\}94\.9±3\.694\.9\_\{\\pm 3\.6\}97\.1±2\.097\.1\_\{\\pm 2\.0\}91\.4±4\.191\.4\_\{\\pm 4\.1\}71\.2±2\.671\.2\_\{\\pm 2\.6\}85\.1±1\.485\.1\_\{\\pm 1\.4\}RL TrainingCCPO \(ours\)ACCA\_\{\\mathrm\{CC\}\}only96\.9±0\.8\\mathbf\{96\.9\}\_\{\\pm 0\.8\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}75\.2±4\.575\.2\_\{\\pm 4\.5\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}93\.1¯±0\.7\\underline\{93\.1\}\_\{\\pm 0\.7\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}97\.0±2\.6\\mathbf\{97\.0\}\_\{\\pm 2\.6\}68\.8±3\.468\.8\_\{\\pm 3\.4\}83\.6¯±2\.6\\underline\{83\.6\}\_\{\\pm 2\.6\}ACC\+AEPA\_\{\\mathrm\{CC\}\}\+A\_\{\\mathrm\{EP\}\}91\.9±0\.591\.9\_\{\\pm 0\.5\}93\.9¯±0\.2\\underline\{93\.9\}\_\{\\pm 0\.2\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}97\.5¯±2\.2\\underline\{97\.5\}\_\{\\pm 2\.2\}93\.1¯±0\.7\\underline\{93\.1\}\_\{\\pm 0\.7\}86\.9±0\.686\.9\_\{\\pm 0\.6\}83\.9±2\.283\.9\_\{\\pm 2\.2\}75\.0±0\.8\\mathbf\{75\.0\}\_\{\\pm 0\.8\}85\.4±1\.6\\mathbf\{85\.4\}\_\{\\pm 1\.6\}Δ​GRPO\\Delta\\ \\mathrm\{GRPO\}16\.6​–​21\.616\.6\\text\{\-\-\}21\.612\.1​–​18\.212\.1\\text\{\-\-\}18\.28\.7​–​33\.58\.7\\text\{\-\-\}33\.520\.3​–​22\.820\.3\\text\{\-\-\}22\.819\.319\.38\.9​–​22\.08\.9\\text\{\-\-\}22\.018\.2​–​31\.318\.2\\text\{\-\-\}31\.312\.3​–​18\.512\.3\\text\{\-\-\}18\.513\.1​–​14\.813\.1\\text\{\-\-\}14\.8Qwen2\.5\-7B\-InstructPromptingQwen2\.5\*19\.319\.337\.137\.138\.538\.514\.814\.812\.512\.58\.08\.04\.24\.23\.03\.026\.626\.6PromptingReAct31\.231\.248\.548\.535\.435\.434\.334\.313\.213\.218\.218\.217\.617\.619\.519\.546\.246\.2PromptingReflexion42\.742\.762\.062\.041\.641\.644\.944\.930\.930\.936\.336\.323\.823\.828\.828\.858\.158\.1RL TrainingPPO \(with critic\)80\.4±2\.780\.4\_\{\\pm 2\.7\}92\.3¯±4\.0\\underline\{92\.3\}\_\{\\pm 4\.0\}64\.0±8\.464\.0\_\{\\pm 8\.4\}92\.5±2\.492\.5\_\{\\pm 2\.4\}89\.5±7\.089\.5\_\{\\pm 7\.0\}80\.3±2\.080\.3\_\{\\pm 2\.0\}68\.8±8\.368\.8\_\{\\pm 8\.3\}68\.7±5\.168\.7\_\{\\pm 5\.1\}81\.4±3\.181\.4\_\{\\pm 3\.1\}RL TrainingRLOO75\.5±4\.675\.5\_\{\\pm 4\.6\}87\.6±4\.387\.6\_\{\\pm 4\.3\}78\.2±8\.378\.2\_\{\\pm 8\.3\}87\.3±5\.887\.3\_\{\\pm 5\.8\}81\.3±7\.681\.3\_\{\\pm 7\.6\}71\.9±5\.271\.9\_\{\\pm 5\.2\}48\.9±8\.448\.9\_\{\\pm 8\.4\}65\.7±4\.065\.7\_\{\\pm 4\.0\}80\.3±3\.280\.3\_\{\\pm 3\.2\}RL TrainingGRPO\*82\.0±0\.882\.0\_\{\\pm 0\.8\}85\.8±2\.185\.8\_\{\\pm 2\.1\}71\.9±10\.371\.9\_\{\\pm 10\.3\}92\.1±0\.792\.1\_\{\\pm 0\.7\}84\.2±2\.784\.2\_\{\\pm 2\.7\}81\.2±1\.881\.2\_\{\\pm 1\.8\}69\.1±0\.869\.1\_\{\\pm 0\.8\}72\.9±3\.272\.9\_\{\\pm 3\.2\}82\.9±0\.682\.9\_\{\\pm 0\.6\}RL TrainingGiGPO\*92\.2±0\.892\.2\_\{\\pm 0\.8\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}93\.3±11\.593\.3\_\{\\pm 11\.5\}96\.0¯±0\.3\\underline\{96\.0\}\_\{\\pm 0\.3\}95\.7¯±3\.7\\underline\{95\.7\}\_\{\\pm 3\.7\}89\.9¯±2\.1\\underline\{89\.9\}\_\{\\pm 2\.1\}76\.5±2\.376\.5\_\{\\pm 2\.3\}74\.7±4\.074\.7\_\{\\pm 4\.0\}84\.3±3\.584\.3\_\{\\pm 3\.5\}RL TrainingGiGPO90\.8±1\.390\.8\_\{\\pm 1\.3\}97\.7±1\.697\.7\_\{\\pm 1\.6\}82\.7±7\.982\.7\_\{\\pm 7\.9\}98\.8±1\.698\.8\_\{\\pm 1\.6\}83\.7±7\.283\.7\_\{\\pm 7\.2\}89\.3±8\.289\.3\_\{\\pm 8\.2\}79\.2±6\.679\.2\_\{\\pm 6\.6\}72\.8±3\.272\.8\_\{\\pm 3\.2\}84\.4±2\.984\.4\_\{\\pm 2\.9\}RL TrainingHGPO\*95\.8¯±0\.5\\underline\{95\.8\}\_\{\\pm 0\.5\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}96\.7¯±5\.8\\underline\{96\.7\}\_\{\\pm 5\.8\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}85\.6±1\.985\.6\_\{\\pm 1\.9\}92\.6¯±2\.6\\underline\{92\.6\}\_\{\\pm 2\.6\}77\.6±4\.377\.6\_\{\\pm 4\.3\}86\.1¯±3\.0\\underline\{86\.1\}\_\{\\pm 3\.0\}RL TrainingHGPO95\.4±0\.695\.4\_\{\\pm 0\.6\}Unreported78\.5±1\.478\.5\_\{\\pm 1\.4\}89\.0±1\.089\.0\_\{\\pm 1\.0\}RL TrainingG2PO\*94\.0±0\.594\.0\_\{\\pm 0\.5\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}96\.0¯±0\.3\\underline\{96\.0\}\_\{\\pm 0\.3\}95\.7¯±3\.7\\underline\{95\.7\}\_\{\\pm 3\.7\}85\.6±1\.985\.6\_\{\\pm 1\.9\}88\.2±2\.788\.2\_\{\\pm 2\.7\}72\.4±3\.572\.4\_\{\\pm 3\.5\}83\.8±2\.683\.8\_\{\\pm 2\.6\}RL TrainingG2PO96\.9±1\.396\.9\_\{\\pm 1\.3\}98\.7±2\.398\.7\_\{\\pm 2\.3\}90\.6±11\.590\.6\_\{\\pm 11\.5\}97\.6±2\.197\.6\_\{\\pm 2\.1\}100\.0±0\.0100\.0\_\{\\pm 0\.0\}98\.8±2\.198\.8\_\{\\pm 2\.1\}93\.7±8\.493\.7\_\{\\pm 8\.4\}78\.3±0\.678\.3\_\{\\pm 0\.6\}89\.8±1\.789\.8\_\{\\pm 1\.7\}RL TrainingCCPO \(ours\)ACCA\_\{\\mathrm\{CC\}\}only97\.4±0\.5\\mathbf\{97\.4\}\_\{\\pm 0\.5\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}95\.7¯±3\.7\\underline\{95\.7\}\_\{\\pm 3\.7\}94\.3±2\.2\\mathbf\{94\.3\}\_\{\\pm 2\.2\}94\.1±2\.5\\mathbf\{94\.1\}\_\{\\pm 2\.5\}79\.4±1\.6\\mathbf\{79\.4\}\_\{\\pm 1\.6\}89\.4±1\.0\\mathbf\{89\.4\}\_\{\\pm 1\.0\}ACC\+AEPA\_\{\\mathrm\{CC\}\}\+A\_\{\\mathrm\{EP\}\}94\.3±0\.594\.3\_\{\\pm 0\.5\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}85\.9±5\.185\.9\_\{\\pm 5\.1\}96\.0¯±0\.3\\underline\{96\.0\}\_\{\\pm 0\.3\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}89\.9¯±2\.1\\underline\{89\.9\}\_\{\\pm 2\.1\}88\.2±2\.788\.2\_\{\\pm 2\.7\}78\.4¯±3\.9\\underline\{78\.4\}\_\{\\pm 3\.9\}84\.6±3\.984\.6\_\{\\pm 3\.9\}Δ​GRPO\\Delta\\ \\mathrm\{GRPO\}12\.3​–​15\.412\.3\\text\{\-\-\}15\.414\.214\.214\.0​–​28\.114\.0\\text\{\-\-\}28\.13\.9​–​7\.93\.9\\text\{\-\-\}7\.911\.5​–​15\.811\.5\\text\{\-\-\}15\.88\.7​–​13\.18\.7\\text\{\-\-\}13\.119\.1​–​25\.019\.1\\text\{\-\-\}25\.05\.5​–​6\.55\.5\\text\{\-\-\}6\.51\.7​–​6\.51\.7\\text\{\-\-\}6\.5

### 4\.3Ablation Studies

We ablate CCPO with Qwen2\.5\-1\.5B\-Instruct, usingACCA\_\{\\mathrm\{CC\}\}on ALFWorld andACC\+AEPA\_\{\\mathrm\{CC\}\}\+A\_\{\\mathrm\{EP\}\}on WebShop as reference configurations\. All ablations follow the main evaluation protocol\.

Table 2:Ablations with Qwen2\.5\-1\.5B\-Instruct on ALFWorld and WebShop\.bestandsecond bestare marked\.HHandFFdenote active historical and future credit spans, withW=H\+F\+1W=H\+F\+1\. Zero marks a disabled credit channel\. Channel\-only rows retainKH=KF=2K\_\{H\}=K\_\{F\}=2\. Temporal\-window rows also change the actual window lengths \(Table[S3](https://arxiv.org/html/2610.19103#A2.T3)\)\. Markers:current,history, andfuture\.MethodALFWorldWebShopAllPickLookCleanHeatCoolPick2Succ\.ScoreQwen2\.5\-1\.5B\-InstructACCA\_\{\\mathrm\{CC\}\}onlyACC\+AEPA\_\{\\mathrm\{CC\}\}\+A\_\{\\mathrm\{EP\}\}w/o self\-trajectory peer exclusion90\.6±0\.890\.6\_\{\\pm 0\.8\}97\.0±0\.197\.0\_\{\\pm 0\.1\}85\.9±5\.185\.9\_\{\\pm 5\.1\}88\.1±1\.088\.1\_\{\\pm 1\.0\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}86\.9±0\.686\.9\_\{\\pm 0\.6\}83\.8±2\.883\.8\_\{\\pm 2\.8\}56\.0±5\.556\.0\_\{\\pm 5\.5\}75\.1±2\.675\.1\_\{\\pm 2\.6\}w/o context summary94\.3¯±0\.5\\underline\{94\.3\}\_\{\\pm 0\.5\}97\.0±0\.197\.0\_\{\\pm 0\.1\}81\.9±7\.181\.9\_\{\\pm 7\.1\}92\.1±0\.792\.1\_\{\\pm 0\.7\}95\.7¯±3\.7\\underline\{95\.7\}\_\{\\pm 3\.7\}98\.6¯±2\.4\\underline\{98\.6\}\_\{\\pm 2\.4\}92\.6¯±2\.6\\underline\{92\.6\}\_\{\\pm 2\.6\}73\.7±3\.573\.7\_\{\\pm 3\.5\}84\.5±3\.084\.5\_\{\\pm 3\.0\}Uniform weighting93\.2±0\.593\.2\_\{\\pm 0\.5\}94\.9±1\.994\.9\_\{\\pm 1\.9\}92\.6±6\.4\\mathbf\{92\.6\}\_\{\\pm 6\.4\}96\.1¯±0\.3\\underline\{96\.1\}\_\{\\pm 0\.3\}95\.7¯±3\.7\\underline\{95\.7\}\_\{\\pm 3\.7\}87\.0±0\.687\.0\_\{\\pm 0\.6\}92\.6¯±2\.7\\underline\{92\.6\}\_\{\\pm 2\.7\}72\.4±3\.972\.4\_\{\\pm 3\.9\}83\.1±3\.483\.1\_\{\\pm 3\.4\}Cosine\-based context similarity92\.7±0\.592\.7\_\{\\pm 0\.5\}97\.9¯±1\.8\\underline\{97\.9\}\_\{\\pm 1\.8\}89\.3¯±0\.6\\underline\{89\.3\}\_\{\\pm 0\.6\}92\.1±0\.792\.1\_\{\\pm 0\.7\}95\.7¯±3\.7\\underline\{95\.7\}\_\{\\pm 3\.7\}89\.9±2\.189\.9\_\{\\pm 2\.1\}88\.2±2\.788\.2\_\{\\pm 2\.7\}70\.3±2\.770\.3\_\{\\pm 2\.7\}86\.6±1\.4\\mathbf\{86\.6\}\_\{\\pm 1\.4\}Historical credit onlyH: 2, F: 0, W: 391\.4±0\.891\.4\_\{\\pm 0\.8\}93\.9±0\.293\.9\_\{\\pm 0\.2\}85\.9±5\.185\.9\_\{\\pm 5\.1\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}93\.1±0\.793\.1\_\{\\pm 0\.7\}86\.9±0\.686\.9\_\{\\pm 0\.6\}83\.8±2\.883\.8\_\{\\pm 2\.8\}72\.7±2\.872\.7\_\{\\pm 2\.8\}85\.1±0\.485\.1\_\{\\pm 0\.4\}Future credit onlyH: 0, F: 2, W: 384\.6±1\.684\.6\_\{\\pm 1\.6\}91\.9±1\.991\.9\_\{\\pm 1\.9\}71\.9±10\.371\.9\_\{\\pm 10\.3\}88\.1±1\.088\.1\_\{\\pm 1\.0\}82\.0±2\.482\.0\_\{\\pm 2\.4\}91\.3±0\.491\.3\_\{\\pm 0\.4\}70\.6±3\.070\.6\_\{\\pm 3\.0\}74\.0¯±2\.4\\underline\{74\.0\}\_\{\\pm 2\.4\}85\.1±2\.485\.1\_\{\\pm 2\.4\}Symmetric context variationH: 1, F: 1, W: 386\.7±0\.086\.7\_\{\\pm 0\.0\}91\.9±1\.991\.9\_\{\\pm 1\.9\}78\.5±1\.378\.5\_\{\\pm 1\.3\}92\.1±0\.792\.1\_\{\\pm 0\.7\}91\.0±3\.191\.0\_\{\\pm 3\.1\}85\.6±1\.985\.6\_\{\\pm 1\.9\}75\.0±2\.975\.0\_\{\\pm 2\.9\}——H: 3, F: 3, W: 791\.4±0\.891\.4\_\{\\pm 0\.8\}97\.0±0\.197\.0\_\{\\pm 0\.1\}85\.9±5\.185\.9\_\{\\pm 5\.1\}96\.0±0\.396\.0\_\{\\pm 0\.3\}93\.1±0\.793\.1\_\{\\pm 0\.7\}91\.3±0\.491\.3\_\{\\pm 0\.4\}79\.4±2\.879\.4\_\{\\pm 2\.8\}——Asymmetric context variationH: 4, F: 0, W: 594\.3¯±0\.9\\underline\{94\.3\}\_\{\\pm 0\.9\}97\.0±0\.197\.0\_\{\\pm 0\.1\}89\.3¯±0\.6\\underline\{89\.3\}\_\{\\pm 0\.6\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}95\.7¯±3\.7\\underline\{95\.7\}\_\{\\pm 3\.7\}91\.3±0\.491\.3\_\{\\pm 0\.4\}88\.2±2\.788\.2\_\{\\pm 2\.7\}——H: 3, F: 1, W: 592\.7±0\.992\.7\_\{\\pm 0\.9\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}89\.3¯±0\.6\\underline\{89\.3\}\_\{\\pm 0\.6\}96\.0±0\.396\.0\_\{\\pm 0\.3\}91\.0±3\.191\.0\_\{\\pm 3\.1\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}73\.6±3\.973\.6\_\{\\pm 3\.9\}——H: 1, F: 3, W: 589\.3±1\.689\.3\_\{\\pm 1\.6\}93\.9±0\.293\.9\_\{\\pm 0\.2\}82\.6±10\.982\.6\_\{\\pm 10\.9\}89\.6±1\.589\.6\_\{\\pm 1\.5\}82\.0±2\.482\.0\_\{\\pm 2\.4\}91\.3±0\.491\.3\_\{\\pm 0\.4\}88\.2±2\.788\.2\_\{\\pm 2\.7\}——H: 0, F: 4, W: 562\.0±1\.862\.0\_\{\\pm 1\.8\}71\.7±2\.671\.7\_\{\\pm 2\.6\}82\.6±10\.982\.6\_\{\\pm 10\.9\}58\.2±6\.158\.2\_\{\\pm 6\.1\}66\.1±3\.466\.1\_\{\\pm 3\.4\}59\.4±1\.359\.4\_\{\\pm 1\.3\}44\.1±1\.144\.1\_\{\\pm 1\.1\}——CCPO \(ours\)H: 2, F: 2, W: 596\.9±0\.8\\mathbf\{96\.9\}\_\{\\pm 0\.8\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}75\.2±4\.575\.2\_\{\\pm 4\.5\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}93\.1±0\.793\.1\_\{\\pm 0\.7\}100\.0±0\.0\\mathbf\{100\.0\}\_\{\\pm 0\.0\}97\.0±2\.6\\mathbf\{97\.0\}\_\{\\pm 2\.6\}75\.0±0\.8\\mathbf\{75\.0\}\_\{\\pm 0\.8\}85\.4¯±1\.6\\underline\{85\.4\}\_\{\\pm 1\.6\}#### Peer selection and contextual weighting\.

Relative to CCPO, including same\-trajectory peers lowers mean success by 6\.3 on ALFWorld and 19\.0 percentage points on WebShop \(Table[2](https://arxiv.org/html/2610.19103#S4.T2)\)\. Uniform weighting lowers success by 3\.7 and 2\.6 points, and cosine weighting by 4\.2 and 4\.7\.

#### Structured context summaries alter peer influence\.

Context summaries add accumulated trajectory statistics to semantic representations: step count, distinct\-observation count, novelty fraction, and a revisit indicator\. Removing them lowers mean success by 2\.6 percentage points on ALFWorld and 1\.3 on WebShop \(Table[2](https://arxiv.org/html/2610.19103#S4.T2)\)\. Adding summaries changes normalized weights by more than one percentage point for 33\.8% of matched query–peer pairs \(Table[3](https://arxiv.org/html/2610.19103#S4.T3); Appendix[E\.3](https://arxiv.org/html/2610.19103#A5.SS3)\)\.

Table 3:Absolute changes in matched peer weights from context summaries\.
#### Historical and future credit\.

We compare the combined, history\-only, and future\-only credit configurations, retaining episode\-level credit on WebShop\. These channel\-only ablations switch off the other credit component while retaining the default prompt\-history length and future horizon,KH=KF=2K\_\{\\mathrm\{H\}\}=K\_\{\\mathrm\{F\}\}=2\(Table[S3](https://arxiv.org/html/2610.19103#A2.T3)\)\. On ALFWorld, success falls from 96\.9% to 91\.4% when using only historical credit and 84\.6% when using only future credit\. On WebShop, it decreases from 75\.0% to 72\.7% and 74\.0%, respectively \(Table[2](https://arxiv.org/html/2610.19103#S4.T2)\)\. The combined configuration achieves the highest mean success on both benchmarks\. History\-only exceeds future\-only in mean success on ALFWorld but trails it on WebShop, suggesting task\-dependent credit utility\.

#### Temporal window allocation\.

We vary symmetric windows on history length and future horizon\. Increasing the symmetric window from\(2,2\)\(2,2\)to\(3,3\)\(3,3\)reduces success from 96\.9% to 91\.4%\. We then compare asymmetric temporal allocations at fixed context span\. The default CCPO \(2 steps each way\) outperforms all tested asymmetric windows\. We restrict these variations to ALFWorld, where successful 1\.5B trajectories average 8\.7 steps versus 5\.1 on WebShop under the reference configurations \(Appendix[G](https://arxiv.org/html/2610.19103#A7), Figure[S4](https://arxiv.org/html/2610.19103#A7.F4)\)\. These results favor a compact yet rich context window that balances locality with context span, since two steps each way outperforms both one and three steps\.

### 4\.4Contextual Credit Analysis

#### Identical observations can conceal different task progress\.

In Figure[3](https://arxiv.org/html/2610.19103#S4.F3), GiGPO and G2PO repeatedly visit an empty toiletpaper hanger without acquiring the target\. CCPO encounters the same observation after acquisition and completes the task \(Appendix[F\.1](https://arxiv.org/html/2610.19103#A6.SS1)\)\. Appendix[F\.2](https://arxiv.org/html/2610.19103#A6.SS2)presents a complementary failure mode: G2PO repeatedly removes and replaces one spraybottle, whereas CCPO places two distinct objects \(Figure[S3](https://arxiv.org/html/2610.19103#A6.F3)\)\. Across three evaluation draws, CCPO’s revisit rate is 22\.5%, versus 26\.8–41\.0% for the baselines \(Table[4](https://arxiv.org/html/2610.19103#S4.T4)\)\.

Figure 3:ALFWorld evaluation trajectories \(Qwen2\.5\-1\.5B\-Instruct\)\.Each cell denotes one step\. Revisit counts track navigation to previously visited locations\. CCPO usesACCA\_\{\\mathrm\{CC\}\}only\.Table 4:Revisit rates on ALFWorld evaluation rollouts\.

#### Contextual credit provides unique learning signals\.

Figure[4](https://arxiv.org/html/2610.19103#S4.F4)a comparesACCA\_\{\\mathrm\{CC\}\}with counterfactualAEPA\_\{\\mathrm\{EP\}\}from the sameACCA\_\{\\mathrm\{CC\}\}\-only ALFWorld rollouts\.AEPA\_\{\\mathrm\{EP\}\}is logged for comparison and excluded from policy updates\. Their mean absolute magnitudes diverge as validation success improves:AEPA\_\{\\mathrm\{EP\}\}diminishes whileACCA\_\{\\mathrm\{CC\}\}remains elevated, so a learning signal remains after most episodes succeed\.

\(a\) Contextual vs\. episode advantage\(b\) Credit decomposition

Figure 4:Contextual credit dynamics on ALFWorld \(Qwen2\.5\-1\.5B and 7B\)\.\(a\)ACCA\_\{\\mathrm\{CC\}\}versusAEPA\_\{\\mathrm\{EP\}\}from the same rollouts\. OnlyACCA\_\{\\mathrm\{CC\}\}is backpropagated\. \(b\) Historical, future, and combined credit\. Future\-to\-history ratios compare iterations 1–30 and 121–150\. The heatmap correlates unsmoothed historical and future magnitudes by non\-overlapping 10\-iteration windows\. Curves: mean absolute credit, centered 5\-iteration means, and faint raw traces\. Bands: 95% trajectory\-bootstrap CIs\.
#### The composition of contextual advantage changes during training\.

Figure[4](https://arxiv.org/html/2610.19103#S4.F4)b shows historical credit dominating initially, with future credit becoming more prominent despite fixed combination weights\. Between iterations 1–30 and 121–150, the future\-to\-history mean absolute magnitude ratio rises from 0\.25 to 0\.60 at 1\.5B and from 0\.32 to 1\.01 at 7B\. Correlations between historical and future magnitudes within 10\-iteration windows become less consistently positive later, especially at 7B\. The two components thus evolve differently across training, supporting complementary roles\.

Figure 5:Credit preferences on ALFWorld \(Qwen2\.5\-1\.5B\-Instruct\)\.Mean category\-relative credit within matched task–observation groups and iterations, from successful rollouts at iterations 101–150\. Asterisks: Bonferroni\-adjusted CIs excluding zero\. Dashes: fewer than 60 groups\.Off: inactive component\. Context markers denotecurrent,history, andfuture\.
#### What does future credit add beyond historical credit?

Using successful 1\.5B ALFWorld rollouts from iterations 101–150, we compare category\-relative credit within task–observation\-matched groups per iteration, with bootstrap CIs and Bonferroni correction across ten categories \(Figure[5](https://arxiv.org/html/2610.19103#S4.F5)\)\. Filtering and statistical procedures are detailed in Appendix[E\.1](https://arxiv.org/html/2610.19103#A5.SS1)\. Relative to other actions,AFA^\{\\mathrm\{F\}\}favors new\-location transitions more thanAHA^\{\\mathrm\{H\}\}\(difference:\+0\.037\+0\.037; adjusted CI:\[0\.001,0\.073\]\[0\.001,0\.073\]\), while both favor cleaning, heating, and cooling\. The\(H,F\)=\(4,0\)\(H,F\)=\(4,0\),\(0,4\)\(0,4\), and w/o\-context\-summary variants show similar patterns\. Paired action timing also differs: against full CCPO, the future\-only policy first opens a receptacle earlier in 72\.1% of paired games, and the history\-only policy later in 54\.3% \(Appendix[E\.2](https://arxiv.org/html/2610.19103#A5.SS2), Figure[S1](https://arxiv.org/html/2610.19103#A5.F1)\)\. The failure case in Figure[3](https://arxiv.org/html/2610.19103#S4.F3)shows that identical observations can accompany different task progress; the credit preferences in Figure[5](https://arxiv.org/html/2610.19103#S4.F5)complement this behavioral evidence by showing how historical and future credit evaluate exploratory transitions differently\.

## 5Conclusion

We introduced Contextual Credit Policy Optimization \(CCPO\) for two challenges in comparison\-based step\-level credit: whom a step is compared with, and what it is credited for\. CCPO weights observation\-matched peers by contextual similarity and pairs historical credit with future credit, both computed from collected rollouts without an auxiliary value critic\. With Qwen2\.5\-7B\-Instruct, it reaches 97\.4% success on ALFWorld and 79\.4% on WebShop\. In ablations, uniform weights lower success by 3\.7 and 2\.6 points, and either credit alone lowers it by up to 12\.3\.

#### Limitations\.

CCPO has three main limitations\. First, our evaluation covers two multi\-step benchmarks and one model family\. Broader generalization and applicability to short\-horizon or single\-step tasks remain untested\. Second, exact observation matching can limit peer availability, while fixed temporal windows and hand\-designed trajectory statistics may require adaptation across environments\. Finally, future credit vanishes when all rollouts in a task group receive zero terminal reward, limiting the future channel’s contribution under sparse rewards, particularly early in training when successful trajectories are scarce\.

#### Future work\.

The changing balance between historical and future credit motivates adaptive horizons and stage\-dependent weighting\. Further directions include streaming credit estimation from incomplete rollouts, multimodal contexts combining visual observations and textual histories, semantic peer retrieval beyond exact observation matching, and uncertainty estimates for correlated visits\. Broader evaluation should examine generalization across model families and domains, computational scaling, and the effects of realized\-future conditioning on policy updates\.

## References

- A\. Ahmadian, C\. Cremer, M\. Gallé, M\. Fadaee, J\. Kreutzer, O\. Pietquin, A\. Üstün, and S\. HookerBack to basics: revisiting REINFORCE\-style optimization for learning from human feedback in LLMs\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 12248–12267\.External Links:[Link](https://aclanthology.org/2024.acl-long.662/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.662)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2610.19103#S4.SS1.SSS0.Px2.p1.1)\.
- Chenet al\.\(2025\)K\. Chen, M\. Cusumano\-Towner, B\. Huval, A\. Petrenko, J\. Hamburger, V\. Koltun, and P\. KrähenbühlReinforcement learning for long\-horizon interactive LLM agents\.Note:arXiv preprint arXiv:2502\.01600External Links:2502\.01600,[Link](https://arxiv.org/abs/2502.01600v3)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px2.p1.1)\.
- Cuiet al\.\(2025\)G\. Cui, Y\. Zhang, J\. Chen, L\. Yuan, Z\. Wang, Y\. Zuo, H\. Li, Y\. Fan, H\. Chen, W\. Chen, Z\. Liu, H\. Peng, L\. Bai, W\. Ouyang, Y\. Cheng, B\. Zhou, and N\. DingThe entropy mechanism of reinforcement learning for reasoning language models\.Note:arXiv preprint arXiv:2505\.22617External Links:2505\.22617,[Link](https://arxiv.org/abs/2505.22617v1)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px2.p1.1)\.
- DeepSeek\-AIet al\.\(2026\)DeepSeek\-AIet al\.DeepSeek\-V4\.1\-Flash: pushing the limits of KV cache compression\.Note:arXiv preprint arXiv:2609\.19969External Links:2609\.19969,[Link](https://arxiv.org/abs/2609.19969v1)Cited by:[§4\.1](https://arxiv.org/html/2610.19103#S4.SS1.SSS0.Px2.p1.1)\.
- Donget al\.\(2026\)G\. Dong, H\. Mao, K\. Ma, L\. Bao, Y\. Chen, Z\. Wang, Z\. Chen, J\. Du, H\. Wang, F\. Zhang, G\. Zhou, Y\. Zhu, J\. Wen, and Z\. DouAgentic reinforced policy optimization\.InInternational Conference on Learning Representations,C\. Vondrick, B\. Hariharan, C\. Raffel, L\. Pinto, D\. Yang, and A\. Faust \(Eds\.\),Vol\.2026,pp\. 16981–17017\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/file/1c58e53bdf1fb045440256fd567531ae-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px1.p1.1)\.
- Fenget al\.\(2026\)J\. Feng, S\. Huang, X\. Qu, G\. Zhang, Y\. Qin, B\. Zhong, C\. Jiang, J\. Chi, and W\. ZhongReTool: reinforcement learning for strategic tool use in LLMs\.InInternational Conference on Learning Representations,C\. Vondrick, B\. Hariharan, C\. Raffel, L\. Pinto, D\. Yang, and A\. Faust \(Eds\.\),Vol\.2026,pp\. 37909–37926\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/file/4038c9208dfc22644c60ad39c24e5c53-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px1.p1.1)\.
- Fenget al\.\(2025\)L\. Feng, Z\. Xue, T\. Liu, and B\. AnGroup\-in\-group policy optimization for LLM agent training\.InAdvances in Neural Information Processing Systems,D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, and N\. Chen \(Eds\.\),Vol\.38,pp\. 46375–46408\.External Links:[Document](https://dx.doi.org/10.52202/085713-1544),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/420c9f777c0b4f78d515e53cf74d58b2-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2610.19103#S1.p2.1),[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2610.19103#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.19103#S4.SS1.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2610.19103#S4.SS1.SSS0.Px3.p1.1)\.
- Gaoet al\.\(2025\)C\. Gao, C\. Zheng, X\. Chen, K\. Dang, S\. Liu, B\. Yu, A\. Yang, S\. Bai, J\. Zhou, and J\. LinSoft adaptive policy optimization\.Note:arXiv preprint arXiv:2511\.20347External Links:2511\.20347,[Link](https://arxiv.org/abs/2511.20347v2)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px2.p1.1)\.
- Google DeepMind \(2026\)Google DeepMindGemini 3\.8 Flash\.Note:Model cardPublished September 2, 2026\. Accessed September 23, 2026External Links:[Link](https://deepmind.google/models/model-cards/gemini-3-8-flash/)Cited by:[§4\.1](https://arxiv.org/html/2610.19103#S4.SS1.SSS0.Px2.p1.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang,et al\.DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09422-z),[Link](https://doi.org/10.1038/s41586-025-09422-z)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px2.p1.1)\.
- Heet al\.\(2026\)S\. He, L\. Feng, q\. wei, X\. Cheng, L\. Feng, and B\. AnHierarchy\-of\-groups policy optimization for long\-horizon agentic tasks\.InInternational Conference on Learning Representations,C\. Vondrick, B\. Hariharan, C\. Raffel, L\. Pinto, D\. Yang, and A\. Faust \(Eds\.\),Vol\.2026,pp\. 27572–27593\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/file/2e7eb932f93d5db3b61c7a001e72fd0f-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2610.19103#S1.p2.1),[§1](https://arxiv.org/html/2610.19103#S1.p3.1),[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2610.19103#S3.SS2.p1.1),[§4\.1](https://arxiv.org/html/2610.19103#S4.SS1.SSS0.Px2.p1.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. NarasimhanSWE\-bench: can language models resolve real\-world GitHub issues?\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VTF8yNQM66)Cited by:[§1](https://arxiv.org/html/2610.19103#S1.p1.1)\.
- Jinet al\.\(2025\)B\. Jin, H\. Zeng, Z\. Yue, J\. Yoon, S\. O\. Arik, D\. Wang, H\. Zamani, and J\. HanSearch\-R1: training LLMs to reason and leverage search engines with reinforcement learning\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=Rwhi91ideu)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2026a\)J\. Li, Y\. Wang, Q\. Yan, Y\. Tian, Z\. Xu, H\. Song, P\. Xu, and L\. L\. CheongSALT: step\-level advantage assignment for long\-horizon agents via trajectory graph\.InFindings of the Association for Computational Linguistics: EACL 2026,V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 4709–4725\.External Links:[Link](https://aclanthology.org/2026.findings-eacl.247/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-eacl.247),ISBN 979\-8\-89176\-386\-9Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px3.p1.1)\.
- Liet al\.\(2026b\)W\. Li, B\. Qu, B\. Pan, J\. Zhang, Z\. Liu, P\. Zhang, W\. Chen, and B\. ZhangLiteResearcher: a scalable agentic RL training framework for deep research agent\.InThird Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=6qDucSpMvz)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2025\)Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. LinUnderstanding R1\-Zero\-like training: a critical perspective\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=5PAF7PAY2Y)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px2.p1.1)\.
- Luoet al\.\(2025\)M\. Luo, N\. Jain, J\. Singh, S\. Tan, A\. Patel, Q\. Wu, A\. Ariyak, C\. Cai, T\. Venkat, S\. Zhu, B\. Athiwaratkun, M\. Roongta, C\. Zhang, L\. E\. Li, R\. A\. Popa, K\. Sen, and I\. StoicaDeepSWE: training a fully open\-sourced, state\-of\-the\-art coding agent by scaling RL\.Note:Together AI and Agentica research blogExternal Links:[Link](https://www.together.ai/blog/deepswe)Cited by:[§1](https://arxiv.org/html/2610.19103#S1.p1.1),[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px1.p1.1)\.
- MiniMaxet al\.\(2025\)MiniMaxet al\.MiniMax\-M1: scaling test\-time compute efficiently with lightning attention\.Note:arXiv preprint arXiv:2506\.13585External Links:2506\.13585,[Link](https://arxiv.org/abs/2506.13585v1)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px2.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. F\. Christiano, J\. Leike, and R\. LoweTraining language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),Vol\.35,pp\. 27730–27744\.External Links:[Document](https://dx.doi.org/10.52202/068431-2011),[Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px2.p1.1)\.
- Puttaet al\.\(2024\)P\. Putta, E\. Mills, N\. Garg, S\. Motwani, C\. Finn, D\. Garg, and R\. RafailovAgent Q: advanced reasoning and learning for autonomous AI agents\.Note:arXiv preprint arXiv:2408\.07199External Links:2408\.07199,[Link](https://arxiv.org/abs/2408.07199v1)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px2.p1.1)\.
- Qiet al\.\(2025\)Z\. Qi, X\. Liu, I\. L\. Iong, H\. Lai, X\. Sun, J\. Sun, X\. Yang, Y\. Yang, S\. Yao, W\. Xu, J\. Tang, and Y\. DongWebRL: training LLM web agents via self\-evolving online curriculum reinforcement learning\.InInternational Conference on Learning Representations,Y\. Yue, A\. Garg, N\. Peng, F\. Sha, and R\. Yu \(Eds\.\),Vol\.2025,pp\. 79791–79821\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/c66e1fcc9691aae706250638f36f681b-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2610.19103#S1.p1.1),[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px1.p1.1)\.
- Qiuet al\.\(2026\)Z\. Qiu, S\. Qiao, K\. Xu, Y\. Zhu, L\. Du, N\. Zhang, and H\. ChenRewarding the scientific process: process\-level reward modeling for agentic data analysis\.InProceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\.2,pp\. 11866–11877\.External Links:[Document](https://dx.doi.org/10.1145/3770855.3819049),[Link](https://doi.org/10.1145/3770855.3819049)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px1.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.8\-Max: a new bar for coding and cowork\.External Links:[Link](https://qwen.ai/blog?id=qwen3.8)Cited by:[§4\.1](https://arxiv.org/html/2610.19103#S4.SS1.SSS0.Px2.p1.1)\.
- Qwenet al\.\(2025\)Qwen, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. QiuQwen2\.5 technical report\.Note:arXiv preprint arXiv:2412\.15115External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115v2)Cited by:[§1](https://arxiv.org/html/2610.19103#S1.p4.1),[§4\.1](https://arxiv.org/html/2610.19103#S4.SS1.SSS0.Px2.p1.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 53728–53741\.External Links:[Document](https://dx.doi.org/10.52202/075280-2338),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/a85b405ed65c6477a4fe8302b5e06ce7-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px2.p1.1)\.
- Schulmanet al\.\(2016\)J\. Schulman, P\. Moritz, S\. Levine, M\. Jordan, and P\. AbbeelHigh\-dimensional continuous control using generalized advantage estimation\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/1506.02438)Cited by:[§3\.3](https://arxiv.org/html/2610.19103#S3.SS3.SSS0.Px2.p1.3)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.Note:arXiv preprint arXiv:1707\.06347External Links:1707\.06347,[Link](https://arxiv.org/abs/1707.06347v2)Cited by:[§1](https://arxiv.org/html/2610.19103#S1.p2.1),[§4\.1](https://arxiv.org/html/2610.19103#S4.SS1.SSS0.Px2.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models\.Note:arXiv preprint arXiv:2402\.03300External Links:2402\.03300,[Link](https://arxiv.org/abs/2402.03300v3)Cited by:[§1](https://arxiv.org/html/2610.19103#S1.p2.1),[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2610.19103#S3.SS1.p1.1),[§3\.4](https://arxiv.org/html/2610.19103#S3.SS4.p1.2),[§4\.1](https://arxiv.org/html/2610.19103#S4.SS1.SSS0.Px2.p1.1)\.
- Shenet al\.\(2026\)G\. Shen, C\. Zhao, X\. Cheng, L\. Huang, and X\. YuVESPO: variational sequence\-level soft policy optimization for stable off\-policy LLM training\.Note:arXiv preprint arXiv:2602\.10693External Links:2602\.10693,[Link](https://arxiv.org/abs/2602.10693v3)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px2.p1.1)\.
- Shenget al\.\(2025\)G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. WuHybridFlow: a flexible and efficient RLHF framework\.InProceedings of the Twentieth European Conference on Computer Systems,pp\. 1279–1297\.External Links:[Document](https://dx.doi.org/10.1145/3689031.3696075),[Link](https://doi.org/10.1145/3689031.3696075)Cited by:[§4\.1](https://arxiv.org/html/2610.19103#S4.SS1.SSS0.Px3.p1.1)\.
- Shinnet al\.\(2023\)N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. YaoReflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 8634–8652\.External Links:[Document](https://dx.doi.org/10.52202/075280-0377),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/1b44b878bb782e6954cd888628510e90-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.19103#S4.SS1.SSS0.Px2.p1.1)\.
- Shridharet al\.\(2021\)M\. Shridhar, X\. Yuan, M\. Côté, Y\. Bisk, A\. Trischler, and M\. HausknechtALFWorld: aligning text and embodied environments for interactive learning\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2010.03768v2)Cited by:[§1](https://arxiv.org/html/2610.19103#S1.p4.1),[§4\.1](https://arxiv.org/html/2610.19103#S4.SS1.SSS0.Px1.p1.1)\.
- Songet al\.\(2024\)Y\. Song, D\. Yin, X\. Yue, J\. Huang, S\. Li, and B\. Y\. LinTrial and error: exploration\-based trajectory optimization of LLM agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 7584–7600\.External Links:[Link](https://aclanthology.org/2024.acl-long.409/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.409)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px2.p1.1)\.
- Sutton and Barto \(2018\)R\. S\. Sutton and A\. G\. BartoReinforcement learning: an introduction\.2nd edition,MIT Press\.External Links:ISBN 9780262039246,[Link](https://mitpress.mit.edu/9780262039246/reinforcement-learning/)Cited by:[§1](https://arxiv.org/html/2610.19103#S1.p1.1),[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px1.p1.1)\.
- Tanet al\.\(2026\)H\. Tan, X\. Yang, H\. Chen, J\. Shao, Y\. Wen, Y\. Shen, W\. Luo, X\. Du, L\. Guo, and Y\. LiHindsight credit assignment for long\-horizon LLM agents\.Note:arXiv preprint arXiv:2603\.08754External Links:2603\.08754,[Link](https://arxiv.org/abs/2603.08754v1)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px3.p1.1)\.
- Taoet al\.\(2026\)L\. Tao, B\. Peng, W\. Yao, T\. Ge, H\. Cheng, M\. H\. Wang, J\. Gao, and S\. LiTRACE: turn\-level reward assignment via credit estimation for long\-horizon agents\.Note:arXiv preprint arXiv:2607\.13988External Links:2607\.13988,[Link](https://arxiv.org/abs/2607.13988v1)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px3.p1.1)\.
- Tongyi DeepResearch Teamet al\.\(2026\)Tongyi DeepResearch Teamet al\.Tongyi DeepResearch technical report\.Note:arXiv preprint arXiv:2510\.24701External Links:2510\.24701,[Link](https://arxiv.org/abs/2510.24701v3)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2026a\)H\. Wang, H\. Cui, C\. Zhang, X\. Liu, S\. Jin, S\. Geng, X\. Zhang, N\. Zalmout, Z\. Shi, and Y\. SunT2\{\}^\{2\}PO: uncertainty\-guided exploration control for stable multi\-turn agentic reinforcement learning\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=aD1zjvdJN4)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2024\)L\. Wang, C\. Ma, X\. Feng, Z\. Zhang, H\. Yang, J\. Zhang, Z\. Chen, J\. Tang, X\. Chen, Y\. Lin, W\. X\. Zhao, Z\. Wei, and J\. WenA survey on large language model based autonomous agents\.Frontiers of Computer Science18\(6\),pp\. 186345\.External Links:[Document](https://dx.doi.org/10.1007/s11704-024-40231-1),[Link](https://doi.org/10.1007/s11704-024-40231-1)Cited by:[§1](https://arxiv.org/html/2610.19103#S1.p1.1)\.
- Wanget al\.\(2026b\)X\. Wang, H\. Zhang, H\. Wang, Y\. Shi, R\. Li, K\. Han, C\. Tong, H\. Deng, A\. K\. Taylor, R\. Sun, Y\. Zhu, J\. Cong, Y\. Sun, and W\. WangARLArena: a unified framework for stable agentic reinforcement learning\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=90kxFi9VGP)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2026c\)Y\. Wang, X\. Chen, X\. Jin, M\. Wang, and L\. YangOpenClaw\-RL: train any agent simply by talking\.Note:arXiv preprint arXiv:2603\.10165External Links:2603\.10165,[Link](https://arxiv.org/abs/2603.10165v2)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px3.p1.1)\.
- Wanget al\.\(2026d\)Y\. Wang, M\. Song, Z\. Zhang, S\. Huang, H\. Huang, F\. Wei, W\. Deng, F\. Sun, and Q\. ZhangGroup\-graph policy optimization for long\-horizon agentic reinforcement learning\.Note:arXiv preprint arXiv:2606\.22995External Links:2606\.22995,[Link](https://arxiv.org/abs/2606.22995v1)Cited by:[§1](https://arxiv.org/html/2610.19103#S1.p2.1),[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2610.19103#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.19103#S4.SS1.SSS0.Px2.p1.1)\.
- Wanget al\.\(2026e\)Z\. Wang, C\. Gui, X\. Jin, Q\. Wang, L\. Liu, K\. Wang, S\. Chen, L\. Li, Z\. Yang, P\. Zhang, Y\. Lu, J\. Wu, L\. Fei\-Fei, L\. Wang, Y\. Choi, and M\. LiRAGEN\-2: reasoning collapse in agentic RL\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=01caH9oj7C)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025\)Z\. Wang, K\. Wang, Q\. Wang, P\. Zhang, L\. Li, Z\. Yang, X\. Jin, K\. Yu, M\. N\. Nguyen, L\. Liu, E\. Gottlieb, Y\. Lu, K\. Cho, J\. Wu, L\. Fei\-Fei, L\. Wang, Y\. Choi, and M\. LiRAGEN: understanding self\-evolution in LLM agents via multi\-turn reinforcement learning\.Note:arXiv preprint arXiv:2504\.20073External Links:2504\.20073,[Link](https://arxiv.org/abs/2504.20073v2)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px1.p1.1)\.
- Yaoet al\.\(2022\)S\. Yao, H\. Chen, J\. Yang, and K\. NarasimhanWebShop: towards scalable real\-world web interaction with grounded language agents\.InAdvances in Neural Information Processing Systems,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),Vol\.35,pp\. 20744–20757\.External Links:[Document](https://dx.doi.org/10.52202/068431-1508),[Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/82ad13ec01f9fe44c01cb91814fd7b8c-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2610.19103#S1.p4.1),[§4\.1](https://arxiv.org/html/2610.19103#S4.SS1.SSS0.Px1.p1.1)\.
- Yaoet al\.\(2023\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. CaoReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=WE_vluYUL-X)Cited by:[§4\.1](https://arxiv.org/html/2610.19103#S4.SS1.SSS0.Px2.p1.1)\.
- Yaoet al\.\(2024\)W\. Yao, S\. Heinecke, J\. C\. Niebles, Z\. Liu, Y\. Feng, L\. Xue, R\. Ramapura Narasimha Murthy, Z\. Chen, J\. Zhang, D\. Arpit, R\. Xu, P\. Mui, H\. Wang, C\. Xiong, and S\. SavareseRetroformer: retrospective large language agents with policy gradient optimization\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 10091–10111\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/29f421fbdcc82aeb349d784d3aaccdb3-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px1.p1.1)\.
- Yeet al\.\(2020\)D\. Ye, Z\. Liu, M\. Sun, B\. Shi, P\. Zhao, H\. Wu, H\. Yu, S\. Yang, X\. Wu, Q\. Guo, Q\. Chen, Y\. Yin, H\. Zhang, T\. Shi, L\. Wang, Q\. Fu, W\. Yang, and L\. HuangMastering complex control in MOBA games with deep reinforcement learning\.Proceedings of the AAAI Conference on Artificial Intelligence34\(04\),pp\. 6672–6679\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v34i04.6144),[Link](https://doi.org/10.1609/aaai.v34i04.6144)Cited by:[§C\.4](https://arxiv.org/html/2610.19103#A3.SS4.p1.1),[§3\.4](https://arxiv.org/html/2610.19103#S3.SS4.p1.2)\.
- Yuet al\.\(2025\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, J\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, R\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, Y\. Wu, and M\. WangDAPO: an open\-source LLM reinforcement learning system at scale\.InAdvances in Neural Information Processing Systems,D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, and N\. Chen \(Eds\.\),Vol\.38,pp\. 113222–113244\.External Links:[Document](https://dx.doi.org/10.52202/085713-3775),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/a4277440d50f1f15d2cb4c14f7e0c0d2-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px2.p1.1)\.
- Yuanet al\.\(2026\)J\. Yuan, Z\. Gong, J\. Hang, Z\. Bai, and W\. ZhaoGraph\-enhanced policy optimization in LLM agent training\.Note:arXiv preprint arXiv:2510\.26270External Links:2510\.26270,[Link](https://arxiv.org/abs/2510.26270v2)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px3.p1.1)\.
- Zhang \(2026\)C\. ZhangFrom reasoning to agentic: credit assignment in reinforcement learning for large language models\.Note:arXiv preprint arXiv:2604\.09459External Links:2604\.09459,[Link](https://arxiv.org/abs/2604.09459v3)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px3.p1.1)\.
- Zhanget al\.\(2026\)G\. Zhang, H\. Geng, X\. Yu, Z\. Yin, Z\. Zhang, Z\. Tan, H\. Zhou, Z\. Li, X\. Xue, Y\. Li, Y\. Zhou, Y\. Chen, C\. Zhang, Y\. Fan, Z\. Wang, S\. Huang, F\. P\. Velez, Y\. Liao, H\. Wang, M\. Yang, H\. Ji, J\. Wang, S\. Yan, P\. Torr, and L\. BaiThe landscape of agentic reinforcement learning for LLMs: a survey\.Transactions on Machine Learning Research\.Note:Survey CertificationExternal Links:[Link](https://openreview.net/forum?id=RY19y2RI1O),ISSN 2835\-8856Cited by:[§1](https://arxiv.org/html/2610.19103#S1.p1.1),[§1](https://arxiv.org/html/2610.19103#S1.p2.1),[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px1.p1.1)\.
- Zhenget al\.\(2025a\)C\. Zheng, S\. Liu, M\. Li, X\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang, J\. Zhou, and J\. LinGroup sequence policy optimization\.Note:arXiv preprint arXiv:2507\.18071External Links:2507\.18071,[Link](https://arxiv.org/abs/2507.18071v2)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px2.p1.1)\.
- Zhenget al\.\(2025b\)Y\. Zheng, D\. Fu, X\. Hu, X\. Cai, L\. Ye, P\. Lu, and P\. LiuDeepResearcher: scaling deep research via reinforcement learning in real\-world environments\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 414–431\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.22/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.22),ISBN 979\-8\-89176\-332\-6Cited by:[§1](https://arxiv.org/html/2610.19103#S1.p1.1),[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px1.p1.1)\.
- Zhouet al\.\(2024a\)S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. NeubigWebArena: a realistic web environment for building autonomous agents\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 15585–15606\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/4410c0711e9154a7a2d26f9b3816d1ef-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2610.19103#S1.p1.1)\.
- Zhouet al\.\(2025\)Y\. Zhou, S\. Jiang, Y\. Tian, J\. Weston, S\. Levine, S\. Sukhbaatar, and X\. LiSWEET\-RL: training multi\-turn LLM agents on collaborative reasoning tasks\.Note:arXiv preprint arXiv:2503\.15478External Links:2503\.15478,[Link](https://arxiv.org/abs/2503.15478v1)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px3.p1.1)\.
- Zhouet al\.\(2024b\)Y\. Zhou, A\. Zanette, J\. Pan, S\. Levine, and A\. KumarArCHer: training language model agents via hierarchical multi\-turn RL\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 62178–62209\.External Links:[Link](https://proceedings.mlr.press/v235/zhou24t.html)Cited by:[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.19103#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix ANotation

Table S1:Notation for CCPO\.Subscripts index trajectories and steps;H\\mathrm\{H\}andF\\mathrm\{F\}denote historical and future channels\. Implementation details appear in Appendix[C](https://arxiv.org/html/2610.19103#A3)\.SymbolDefinitionTasks, trajectories, and temporal indexingx,xix,\\ x\_\{i\}Task instance;xix\_\{i\}is the task associated with trajectoryii\.NNNumber of trajectories sampled per task in a rollout batch\.τi\\tau\_\{i\}Trajectoryii, containing observationsoi,0:Tio\_\{i,0:T\_\{i\}\}, actionsai,0:Ti−1a\_\{i,0:T\_\{i\}\-1\}, and their outcomes\.i,ji,\\ jQuery\-trajectory and peer\-trajectory indices\.t,st,\\ sZero\-based query\-step and peer\-step indices:0≤t<Ti0\\leq t<T\_\{i\},0≤s<Tj0\\leq s<T\_\{j\}\.TiT\_\{i\}Number of actions in trajectoryii;TiT\_\{i\}also indexes its terminal endpoint\.oi,to\_\{i,t\}Observation before actionai,ta\_\{i,t\}; matching uses exact string equality\.ai,ta\_\{i,t\}Generated response, including reasoning and the executable action\.pi,tp\_\{i,t\}Task, current observation, and up toKHK\_\{\\mathrm\{H\}\}preceding observation–action pairs; excludes the current response\.ri,tenv,Rir^\{\\mathrm\{env\}\}\_\{i,t\},\\ R\_\{i\}Environment reward after actionai,ta\_\{i,t\}, and terminal trajectory outcome\.γ,η\\gamma,\\ \\etaReturn discount factor and current invalid\-action penalty coefficient\.KH,KFK\_\{\\mathrm\{H\}\},\\ K\_\{\\mathrm\{F\}\}Prompt\-history length and realized future horizon, both22by default\.t\+t^\{\+\}Future endpoint on the collected trajectory:t\+=min⁡\(t\+KF,Ti\)t^\{\+\}=\\min\(t\+K\_\{\\mathrm\{F\}\},T\_\{i\}\)\.H,F,WH,\\ F,\\ WActive credit spans in the ablation table, withW=H\+F\+1W=H\+F\+1; zero denotes a disabled channel\. Channel\-only rows retainKH=KF=2K\_\{\\mathrm\{H\}\}=K\_\{\\mathrm\{F\}\}=2, while temporal\-window variations useKH=HK\_\{\\mathrm\{H\}\}=HandKF=FK\_\{\\mathrm\{F\}\}=F\.Context representationhi,th\_\{i,t\}Frozen reference model's final\-layer hidden state at the last prompt token\.h¯,P⟂,3\\bar\{h\},\\ P\_\{\\perp,3\}Batch mean hidden state and projection removing its three leading principal directions\.ni,tn\_\{i,t\}Distinct\-observation count:\|\{oi,0,…,oi,t\}\|\|\\\{o\_\{i,0\},\\ldots,o\_\{i,t\}\\\}\|\.ni,t/\(t\+1\)n\_\{i,t\}/\(t\+1\)Observation\-novelty ratio over the observed prefix\.𝟏​\{revisiti,t\}\\mathbf\{1\}\\\{\\mathrm\{revisit\}\_\{i,t\}\\\}Whether the current observation appeared earlier in the same trajectory\.ci,tc\_\{i,t\}37\-dimensional encoding of elapsed steps, distinct observations, novelty ratio, and revisit indicator\.ν⁡\(v\)\\nu\(v\)L2 normalization:v/max⁡\(‖v‖2,10−9\)v/\\max\(\\\|v\\\|\_\{2\},10^\{\-9\}\)\.ϕi,t\\phi\_\{i,t\}Fused context feature; see Equation[9](https://arxiv.org/html/2610.19103#A3.E9)\.\[u;v\]\[u;v\]Vector concatenation\.Peer matching and contextual weightingℰi,t\\mathcal\{E\}\_\{i,t\}Exact task–observation peers outside the query trajectory; Equation[2](https://arxiv.org/html/2610.19103#S3.E2)\.𝒫i,t\\mathcal\{P\}\_\{i,t\}Peers used for readout:ℰi,t\\mathcal\{E\}\_\{i,t\}when nonempty, otherwise all same\-task nonterminal steps withj≠ij\\neq i\.u=\(i,t\),v=\(j,s\)u=\(i,t\),\\ v=\(j,s\)Shorthand for query and peer step identifiers\.wu,vw\_\{u,v\}Kernel weight:exp\(−∥ϕu−ϕv∥2/τB\)\\exp\(\-\\\|\\phi\_\{u\}\-\\phi\_\{v\}\\\|\_\{2\}/\\tau\_\{B\}\)\.w¯u,v\\bar\{w\}\_\{u,v\}Query\-normalized weight:wu,v/∑v′∈𝒫i,twu,v′w\_\{u,v\}/\\sum\_\{v^\{\\prime\}\\in\\mathcal\{P\}\_\{i,t\}\}w\_\{u,v^\{\\prime\}\}\.BBObservation or task bucket used to determine the bandwidth\.mB,m~Bm\_\{B\},\\ \\widetilde\{m\}\_\{B\}Median pairwise distance inBBand its safeguarded value; Appendix[C\.2](https://arxiv.org/html/2610.19103#A3.SS2)\.τB\\tau\_\{B\}Kernel bandwidth0\.15​m~B0\.15\\widetilde\{m\}\_\{B\}, distinct from trajectoryτi\\tau\_\{i\}\.rj,sr\_\{j,s\}Readout target:Yj,sY\_\{j,s\}for return baselines orZj,sZ\_\{j,s\}for outcome values\.ℬi,t​\[r\]\\mathcal\{B\}\_\{i,t\}\[r\]Contextual weighted readout; Equation[4](https://arxiv.org/html/2610.19103#S3.E4)\.Temporal credit signalsYi,tY\_\{i,t\}Historical return target including the current invalid\-action penalty; Equation[1](https://arxiv.org/html/2610.19103#S3.E1)\.ℬi,t​\[Y\]\\mathcal\{B\}\_\{i,t\}\[Y\]Contextual peer baseline for the realized return\.Ci,tHC^\{\\mathrm\{H\}\}\_\{i,t\}Historical credit:Yi,t−ℬi,t​\[Y\]Y\_\{i,t\}\-\\mathcal\{B\}\_\{i,t\}\[Y\]\.Zi,sZ\_\{i,s\}Unpenalized outcome target:γTi−s​Ri\\gamma^\{T\_\{i\}\-s\}R\_\{i\}\.Vi,sV\_\{i,s\}Contextual outcome value:ℬi,s​\[Z\]\\mathcal\{B\}\_\{i,s\}\[Z\]fors<Tis<T\_\{i\}, andRiR\_\{i\}ats=Tis=T\_\{i\}\.Δ​Vi,t\\Delta V\_\{i,t\}Outcome\-value change:Vi,t\+−Vi,tV\_\{i,t^\{\+\}\}\-V\_\{i,t\}\.Ci,tFC^\{\\mathrm\{F\}\}\_\{i,t\}Task\-standardized future credit:zxi​\[Δ​V\]i,tz\_\{x\_\{i\}\}\[\\Delta V\]\_\{i,t\}\.Support, normalization, and advantages𝒮xH\\mathcal\{S\}\_\{x\}^\{\\mathrm\{H\}\}Unique steps of taskxxwith a defined historical baseline\.𝒮xF\\mathcal\{S\}\_\{x\}^\{\\mathrm\{F\}\}Unique task\-xxsteps with both outcome\-value endpoints defined\.𝒮x,𝒮^x\\mathcal\{S\}\_\{x\},\\ \\widehat\{\\mathcal\{S\}\}\_\{x\}𝒮x=𝒮xH∪𝒮xF\\mathcal\{S\}\_\{x\}=\\mathcal\{S\}\_\{x\}^\{\\mathrm\{H\}\}\\cup\\mathcal\{S\}\_\{x\}^\{\\mathrm\{F\}\};𝒮^x\\widehat\{\\mathcal\{S\}\}\_\{x\}is the corresponding multiset of restored training rows\.ℛx\\mathcal\{R\}\_\{x\}All restored input step rows of taskxx, used for episode statistics\.u¯S,sS​\(u\)\\bar\{u\}\_\{S\},\\ s\_\{S\}\(u\)Mean and sample standard deviation over set or row multisetSS\.zS​\[u\]z\_\{S\}\[u\]Standardization:\(u−u¯S\)/\(sS​\(u\)\+10−6\)\(u\-\\bar\{u\}\_\{S\}\)/\(s\_\{S\}\(u\)\+10^\{\-6\}\)for\|S\|\>1\|S\|\>1\.zx​\[Δ​V\]z\_\{x\}\[\\Delta V\]Future\-credit standardization over𝒮xF\\mathcal\{S\}\_\{x\}^\{\\mathrm\{F\}\}\.𝒩x​\[⋅\]\\mathcal\{N\}\_\{x\}\[\\cdot\]Final contextual transformation\.ACC,i,tA\_\{\\mathrm\{CC\},i,t\}Contextual advantage𝒩xi​\[CH\+CF\]i,t\\mathcal\{N\}\_\{x\_\{i\}\}\[C^\{\\mathrm\{H\}\}\+C^\{\\mathrm\{F\}\}\]\_\{i,t\}, also denotedAi,tCCA^\{\\mathrm\{CC\}\}\_\{i,t\}\.Ai,tHA^\{\\mathrm\{H\}\}\_\{i,t\}Historical contribution after the final transformation; Equation[11](https://arxiv.org/html/2610.19103#A3.E11)\.Ai,tFA^\{\\mathrm\{F\}\}\_\{i,t\}Future contribution after the same transformation;ACC=AH\+AFA\_\{\\mathrm\{CC\}\}=A^\{\\mathrm\{H\}\}\+A^\{\\mathrm\{F\}\}\.μxd,σx\\mu\_\{x\}^\{d\},\\ \\sigma\_\{x\}Component mean and common scale for the additive decomposition; Appendix[C\.3](https://arxiv.org/html/2610.19103#A3.SS3)\.ei,te\_\{i,t\}Episode\-channel score:Ri−η​𝟏​\{ai,t​invalid\}R\_\{i\}\-\\eta\\mathbf\{1\}\\\{a\_\{i,t\}\\text\{ invalid\}\\\}\.AEP,i,tA\_\{\\mathrm\{EP\},i,t\}Episode\-score advantage overℛx\\mathcal\{R\}\_\{x\}\.αEP\\alpha\_\{\\mathrm\{EP\}\}Episode\-channel weight:00by default,11for the augmented variant\.Ai,tA\_\{i,t\}Optimizer advantage:ACC,i,t\+αEP​AEP,i,tA\_\{\\mathrm\{CC\},i,t\}\+\\alpha\_\{\\mathrm\{EP\}\}A\_\{\\mathrm\{EP\},i,t\}\.Policy optimizationθ,πθ\\theta,\\ \\pi\_\{\\theta\}Trainable policy parameters and the current LLM policy\.πold\\pi\_\{\\mathrm\{old\}\}Behavior policy used to collect rollouts\.πref\\pi\_\{\\mathrm\{ref\}\}Frozen reference policy for feature extraction and KL regularization\.
## Appendix BTraining and Ablation Configurations

Table S2:Training configurations shared by 1\.5B and 7B models on ALFWorld and WebShop\.SettingGRPOGiGPOHGPOG2POCCPOBoth benchmarksTraining iterations150150150150150Tasks×\\timesrollouts per iteration16×816\\times 816×816\\times 816×816\\times 816×816\\times 816×816\\times 8Total sampled trajectories19,20019,20019,20019,20019,200NVIDIA H200 GPUs per run88888Learning rate10−610^\{\-6\}10−610^\{\-6\}10−610^\{\-6\}10−610^\{\-6\}10−610^\{\-6\}Discount factor0\.950\.950\.950\.950\.95Actor KL coefficient0\.010\.010\.010\.010\.01KL estimatorlow\_var\_kllow\_var\_kllow\_var\_kllow\_var\_kllow\_var\_klKL included in rewardNoNoNoNoNoInvalid\-action penalty coeff\.0\.10\.10\.10\.10\.1Environment training seed00000Prompt history \(steps\)22222Response token limit / step512512512512512Prompt truncationerrorerrorlefterrorerrorValidation decodingSamplingSamplingSamplingSamplingSamplingValidation temperature0\.40\.40\.40\.40\.4Validation interval \(iterations\)55555ALFWorldPPO mini\-batch size256256256256256Maximum interaction steps5050505050Prompt token limit2,0482,0484,0962,0482,048WebShopPPO mini\-batch size6464646464Maximum interaction steps1515301515Prompt token limit4,0964,0964,0964,0964,096Table S3:CCPO variant and ablation configurations\.Each ablation changes only the listed settings relative to its corresponding reference configuration\. The default uses prompt historyKH=2K\_\{\\mathrm\{H\}\}=2, future horizonKF=2K\_\{\\mathrm\{F\}\}=2, and unit weights for both credit channels\. In Table[2](https://arxiv.org/html/2610.19103#S4.T2), a disabled channel has active span zero; its underlying window is retained in the channel\-only ablations\.VariantConfiguration changeACC\+AEPA\_\{\\mathrm\{CC\}\}\+A\_\{\\mathrm\{EP\}\}Add episode advantage with unit weight after contextual normalization, without renormalizing the sum\.w/o self\-trajectory peer exclusionAllow matching steps from the query trajectory, including the query itself, as peers\.w/o context summarySet the accumulated\-statistics weight from 1 to 0; retain frozen hidden\-state features and prompt history\.Cosine\-based context similarityReplace the exponential kernel with cosine\-based weighting\.Historical credit onlyTurn off future credit by setting its weight from 1 to 0\. Retain historical credit with unit weight and the default windowsKH=KF=2K\_\{\\mathrm\{H\}\}=K\_\{\\mathrm\{F\}\}=2; the active spans are\(H,F\)=\(2,0\)\(H,F\)=\(2,0\)\.Future credit onlyTurn off historical credit by setting its weight from 1 to 0\. Retain future credit with unit weight and horizonKF=2K\_\{\\mathrm\{F\}\}=2, together with prompt historyKH=2K\_\{\\mathrm\{H\}\}=2; the active spans are\(H,F\)=\(0,2\)\(H,F\)=\(0,2\)\.Temporal window variationChange the actual prompt\-history length and future horizon to the listed values:KH=HK\_\{\\mathrm\{H\}\}=HandKF=FK\_\{\\mathrm\{F\}\}=F\. Set a channel’s weight to 0 when its listed span is zero, and to 1 otherwise\. For example,\(H,F\)=\(4,0\)\(H,F\)=\(4,0\)uses four history steps and turns off future credit;\(0,4\)\(0,4\)uses a four\-step future horizon, no preceding prompt\-history pairs, and turns off historical credit\.
## Appendix CCCPO Implementation Details

Using complete, globally gathered rollouts, we remove identical step copies before feature construction, peer readouts, and future\-credit standardization\. Original rows are restored for final advantage computation before PPO minibatching\. Training and ablation configurations are in Appendix[B](https://arxiv.org/html/2610.19103#A2)\.

### C\.1Context Representation

The frozen reference model supplies the final\-layer hidden state at the last prompt token\. Across unique steps in the batch, we center these states, remove their three leading principal directions, and L2\-normalize\. The summary encodes elapsed stepstt, distinct\-observation countni,tn\_\{i,t\}, and novelty rationi,t/\(t\+1\)n\_\{i,t\}/\(t\+1\)in three 12\-dimensional thermometer blocks with upper bounds3030,2525, and11\. For valuevvand boundbb, the first⌊12​clip⁡\(v/b,0,1\)⌋\\lfloor 12\\operatorname\{clip\}\(v/b,0,1\)\\rfloorcoordinates are one; the rest are zero\. Appending the binary revisit indicator givesci,t∈ℝ37c\_\{i,t\}\\in\\mathbb\{R\}^\{37\}\. Withν⁡\(v\)=v/max⁡\(‖v‖2,10−9\)\\nu\(v\)=v/\\max\(\\\|v\\\|\_\{2\},10^\{\-9\}\), the fused feature is

ϕi,t=ν⁡\(\[ν⁡\(P⟂,3​\(hi,t−h¯\)\);ν⁡\(ci,t\)\]\)\.\\phi\_\{i,t\}=\\nu\\\!\\left\(\[\\nu\(P\_\{\\perp,3\}\(h\_\{i,t\}\-\\bar\{h\}\)\);\\nu\(c\_\{i,t\}\)\]\\right\)\.\(9\)The projection is recomputed per batch without reward labels\. With two or three unique steps, we skip principal\-direction removal; with one, we use the raw hidden block\. Nonterminal endpoints use their own prompt features and accumulated statistics\.

### C\.2Peer Readouts

Readouts use𝒫i,t\\mathcal\{P\}\_\{i,t\}from Section[3\.2](https://arxiv.org/html/2610.19103#S3.SS2); a step whose peer set is empty receives zero contextual credit\. For the full observation bucketBB, letmBm\_\{B\}be the median distance over distinct unordered pairs before trajectory exclusion\. Equation[3](https://arxiv.org/html/2610.19103#S3.E3)usesτB=0\.15​m~B\\tau\_\{B\}=0\.15\\widetilde\{m\}\_\{B\}, withm~B=mB\\widetilde\{m\}\_\{B\}=m\_\{B\}whenmB\>10−9m\_\{B\}\>10^\{\-9\}and11otherwise\. Task\-level fallback recomputes this bandwidth in the full task bucket\.

Repeated visits retain individual weights\. Each peer trajectory contributes its weighted mean, weighted again by its total kernel mass\. Both denominators are floored at10−1210^\{\-12\}; all\-zero weights yield zero while retaining support\. Away from these floors, this implements Equation[4](https://arxiv.org/html/2610.19103#S3.E4)\.

### C\.3Credit Construction and Normalization

With terminal outcomesRi∈\{0,10\}R\_\{i\}\\in\\\{0,10\\\}, the supplied historical target isYi,t=γTi−1−t​Ri−η​𝟏​\{ai,t​invalid\}Y\_\{i,t\}=\\gamma^\{T\_\{i\}\-1\-t\}R\_\{i\}\-\\eta\\mathbf\{1\}\\\{a\_\{i,t\}\\text\{ invalid\}\\\}, whereas future values useZi,t=γTi−t​RiZ\_\{i,t\}=\\gamma^\{T\_\{i\}\-t\}R\_\{i\}without penalties\. The current\-step penalty entersYYonce\. Att\+=min⁡\(t\+KF,Ti\)t^\{\+\}=\\min\(t\+K\_\{\\mathrm\{F\}\},T\_\{i\}\), terminal values equalRiR\_\{i\}, including zero for unsuccessful rollouts\. Both value endpoints must be defined; their difference has no additional discount factor\.

Leth=t\+−th=t^\{\+\}\-t\. Equation[7](https://arxiv.org/html/2610.19103#S3.E7)can be written as

Δ​Vi,t=\(γh​Vi,t\+−Vi,t\)\+\(1−γh\)​Vi,t\+\.\\Delta V\_\{i,t\}=\\bigl\(\\gamma^\{h\}V\_\{i,t^\{\+\}\}\-V\_\{i,t\}\\bigr\)\+\(1\-\\gamma^\{h\}\)V\_\{i,t^\{\+\}\}\.\(10\)Our value targets are nonnegative, so the second term is nonnegative and can vary across steps\. This time\-dependent contribution can persist after task\-wise centering\.

Standardization uses sample standard deviation plus10−610^\{\-6\}, with equal weight per included row\. Future differences are standardized over supported unique steps within each task in both benchmarks; fewer than two eligible steps give zero future credit\. Unavailable components are zero\-filled before combination\. Unsupported contextual advantages remain zero\. All\-zero outcomes give zero future credit; historical targets still include local penalties\.

For the plotted ALFWorld contributions, letμxd\\mu\_\{x\}^\{d\}be the mean ofCdC^\{d\}over the restored union support andσx=sx​\(CH\+CF\)\+10−6\\sigma\_\{x\}=s\_\{x\}\(C^\{\\mathrm\{H\}\}\+C^\{\\mathrm\{F\}\}\)\+10^\{\-6\}\. For multiple supported rows,

Ai,td=Ci,td−μxidσxi,d∈\{H,F\},ACC=AH\+AF\.A^\{d\}\_\{i,t\}=\\frac\{C^\{d\}\_\{i,t\}\-\\mu\_\{x\_\{i\}\}^\{d\}\}\{\\sigma\_\{x\_\{i\}\}\},\\qquad d\\in\\\{\\mathrm\{H\},\\mathrm\{F\}\\\},\\qquad A\_\{\\mathrm\{CC\}\}=A^\{\\mathrm\{H\}\}\+A^\{\\mathrm\{F\}\}\.\(11\)Both contributions are zero outside the union support\. These additive contributions share a normalization scale and sum toACCA\_\{\\mathrm\{CC\}\}\.

The optional episode channel usesei,t=Ri−η​𝟏​\{ai,t​invalid\}e\_\{i,t\}=R\_\{i\}\-\\eta\\mathbf\{1\}\\\{a\_\{i,t\}\\text\{ invalid\}\\\}over all restored task rows, including contextually unsupported steps\. Hence longer trajectories contribute more rows, and local penalties can produce different episode advantages within a trajectory\. The channel is added with unit weight after contextual normalization, without renormalizing\.

### C\.4Policy\-Update Integration

The adapter broadcasts each detached step advantage to all generated response tokens, including reasoning and executable actions; prompt and padding tokens are excluded\. The loss mask retains valid response tokens from steps with zero contextual credit\. The dual\-clipped policy objective\([Ye et al\., 2020](https://arxiv.org/html/2610.19103#bib.bib47)\)follows Section[3\.4](https://arxiv.org/html/2610.19103#S3.SS4)\. Features and credit estimates receive no gradients\. Estimation uses float64 NumPy arithmetic, while the adapter extracts and returns detached float32 features and token advantages\.

For a response tokenyywith preceding contextcc,low\_var\_klusesℓ=clip⁡\(log⁡πref​\(y∣c\)−log⁡πθ​\(y∣c\),−20,20\)\\ell=\\operatorname\{clip\}\(\\log\\pi\_\{\\mathrm\{ref\}\}\(y\\mid c\)\-\\log\\pi\_\{\\theta\}\(y\\mid c\),\-20,20\)and the penaltyclip⁡\(eℓ−ℓ−1,−10,10\)\\operatorname\{clip\}\(e^\{\\ell\}\-\\ell\-1,\-10,10\)\. The penalty is aggregated with the response\-token mask and added to the policy loss with coefficient0\.010\.01\.

Algorithm 1 CCPO training iteration

## Appendix DValidation Rollout Examples

### D\.1ALFWorld: Picking up a Soapbar

Table S4:ALFWorld prompt and response excerpts at step 6 \(Qwen2\.5\-1\.5B\-Instruct\)\.ComponentExcerptPromptRoleYou are an expert agent operating in the ALFRED Embodied Environment\.TaskYour task is to: put two soapbar in garbagecan\.HistoryPrior to this step, you have already taken 5 step\(s\)\. Below are the most recent 2 observations and the corresponding actions you took:History excerpt\[Observation 4: ’You arrive at cabinet 1\. The cabinet 1 is closed\.’, Action 4: ’open cabinet 1’\]History excerpt\[Observation 5: ’You open the cabinet 1\. The cabinet 1 is open\. In it, you see nothing\.’, Action 5: ’go to sinkbasin 1’\]Current stepYou are now at step 6 and your current observation is: You arrive at sinkbasin 1\. On the sinkbasin 1, you see a soapbar 3, and a soapbar 2\.ActionsYour admissible actions of the current situation are: \[’examine sinkbasin 1’, \[…\] ’take soapbar 2 from sinkbasin 1’, ’take soapbar 3 from sinkbasin 1’\]\.InstructionsNow it’s your turn to take an action\. You should first reason step\-by\-step about the current situation\. This reasoning process MUST be enclosed within<think\></think\>tags\. Once you’ve finished your reasoning, you should choose an admissible action for current step and present it within<action\></action\>tags\.Model responseResponse<think\>The agent has arrived at the sinkbasin 1 and has two soapbars available: soapbar 2 and soapbar 3\. The agent needs to put one of the soapbars in the garbagecan 1\.</think\><action\>take soapbar 2 from sinkbasin 1</action\>

## Appendix EStatistical Analysis

### E\.1Relative Credit Preferences

#### Data and filtering\.

For Figure[5](https://arxiv.org/html/2610.19103#S4.F5), we analyze ALFWorld training rollouts from iterations 101–150 of four Qwen2\.5\-1\.5B\-Instruct runs: full CCPO and CCPO without the accumulated context summary, both with\(KH,KF\)=\(2,2\)\(K\_\{\\mathrm\{H\}\},K\_\{\\mathrm\{F\}\}\)=\(2,2\), together with the asymmetric\(4,0\)\(4,0\)and\(0,4\)\(0,4\)window variants of Table[2](https://arxiv.org/html/2610.19103#S4.T2)\. All use theACCA\_\{\\mathrm\{CC\}\}\-only configuration\. Table[S5](https://arxiv.org/html/2610.19103#A5.T5)specifies the temporal windows and sample counts\. We retain successful trajectories, identified by a recorded episode return greater than0\.50\.5, and interior steps satisfyingt≥2t\\geq 2andtlast−t≥2t\_\{\\mathrm\{last\}\}\-t\\geq 2, wherettis the zero\-based logged step index andtlastt\_\{\\mathrm\{last\}\}is the last logged step in the trajectory\. We apply this interior\-step filter only to the analysis, consistently across all four runs\. Padding duplicates are removed upstream; trajectories with missing episode returns are excluded\. Each run is analyzed using its own rollouts, so comparisons across runs also reflect differences in the sampled behavior\.

Table S5:Filtered samples for the credit\-preference analysis\.Groups match training iteration, task instance, and observation\.MMcounts categories with at least 60 usable groups; each contrast requires both the category and other actions in a group\.
#### Action categories\.

The action at stepttis classified from the environment response recorded at stept\+1t\+1, without using credit values\. Fixed response\-prefix rules identify*task operations*\(clean, heat, or cool\),*take*,*examine*,*new location*,*revisit*,*open*,*put*,*toggle*,*invalid*,*close*, and*other*\. Arrivals are classified as new locations or revisits using the locations previously visited anywhere in the trajectory, including steps subsequently removed by filtering\. Invalid actions are identified by the environment’s “Nothing happens\.” response\. Task completion may require revisiting a location\.

#### Matched\-group contrast\.

A comparison groupBBcontains steps from the same training iteration and task instance with identical observation text\. Only exact task\-observation groups enter this analysis\. For action categorycc, letIB​\(c\)I\_\{B\}\(c\)contain steps of that category andOB​\(c\)O\_\{B\}\(c\)contain the other steps inBB\. We retain groups with both sets nonempty and, for a step\-level quantitymum\_\{u\}, define

dB​\(c,m\)=1\|IB​\(c\)\|​∑u∈IB​\(c\)mu−1\|OB​\(c\)\|​∑u∈OB​\(c\)mu,Δ⁡\(c,m\)=1\|𝒰⁡\(c\)\|​∑B∈𝒰⁡\(c\)dB​\(c,m\),\\begin\{split\}d\_\{B\}\(c;m\)&=\\frac\{1\}\{\|I\_\{B\}\(c\)\|\}\\sum\_\{u\\in I\_\{B\}\(c\)\}m\_\{u\}\-\\frac\{1\}\{\|O\_\{B\}\(c\)\|\}\\sum\_\{u\\in O\_\{B\}\(c\)\}m\_\{u\},\\\\ \\Delta\(c;m\)&=\\frac\{1\}\{\|\\mathcal\{U\}\(c\)\|\}\\sum\_\{B\\in\\mathcal\{U\}\(c\)\}d\_\{B\}\(c;m\),\\end\{split\}\(12\)where𝒰⁡\(c\)\\mathcal\{U\}\(c\)is the set of usable groups\. We apply this definition toAHA^\{\\mathrm\{H\}\},AFA^\{\\mathrm\{F\}\}, andACCA\_\{\\mathrm\{CC\}\}, using the additive decomposition in Equation[11](https://arxiv.org/html/2610.19103#A3.E11)\. Groups receive equal weight regardless of their size\. A positive contrast indicates higher average group\-relative credit for the category\. Individual advantages may have either sign, and the contrast describes an association between action category and credit\.

#### Bootstrap intervals and channel comparison\.

For each run, categories with fewer than 60 usable groups are excluded\. We draw 6,000 bootstrap samples of usable groups with replacement, using random seed 0\. Two\-sided percentile intervals use quantile levels0\.05/\(2​M\)0\.05/\(2M\)and1−0\.05/\(2​M\)1\-0\.05/\(2M\), whereMMis the number of eligible categories in that run\. Asterisks indicate intervals excluding zero\. Each combination of run, analysis window, and measured quantity defines a separate Bonferroni correction family\.

The channel comparison is paired within groups: we bootstrapdB​\(c,AF\)−dB​\(c,AH\)d\_\{B\}\(c;A^\{\\mathrm\{F\}\}\)\-d\_\{B\}\(c;A^\{\\mathrm\{H\}\}\)using the same sampled groups for both terms\. For new\-location transitions in full CCPO, the mean paired difference is0\.03720\.0372with adjusted interval\[0\.0011,0\.0727\]\[0\.0011,0\.0727\], based on 2,804 groups\. The paired\-contrast interval excludes zero; both channel\-specific intervals include zero\.

The analysis is conditional on successful trajectories and observations at which multiple action categories were sampled\.

### E\.2Paired Action Timing During Evaluation

Figure S1:Action timing on ALFWorld \(Qwen2\.5\-1\.5B\-Instruct\)\.First occurrence at checkpoint 150, paired across the same 140 evaluation games against full CCPO\. Percentages condition on both policies performing the action\. Dashes: fewer than 12 pairs or unavailable results; gray rows are excluded\. Context markers denotecurrent,history, andfuture\.#### Evaluation population\.

For Figure[S1](https://arxiv.org/html/2610.19103#A5.F1), we use evaluation rollouts from checkpoints at training iteration 150, collected following the main evaluation protocol\. Each ablation is paired with full CCPO on the same 140 ALFWorld tasks using their evaluation\-game identifiers\. This analysis includes successful and failed episodes\.

#### First occurrence\.

For gameggand categorycc, lettg,ct\_\{g,c\}denote the one\-based step of its first occurrence, identified by the response\-based categories above\. The paired timing difference is

Δ​tg,c=tg,cablation−tg,cfull\.\\Delta t\_\{g,c\}=t^\{\\mathrm\{ablation\}\}\_\{g,c\}\-t^\{\\mathrm\{full\}\}\_\{g,c\}\.\(13\)Only games where both policies perform the action are eligible\. The plotted*earlier*,*same*, and*later*percentages count pairs withΔ​tg,c<0\\Delta t\_\{g,c\}<0,=0=0, and\>0\>0, respectively\. Percentages are computed over eligible game pairs within each category\. Games where only one policy performs the action enter separate coverage counts and are excluded from these percentages\.

#### Displayed categories\.

Bars report descriptive proportions; dashes mark fewer than 12 eligible pairs or unavailable results\. New location is excluded: its first occurrence is fixed at step 1 under this labeling rule\. Invalid and other are also excluded\.

### E\.3Changes in Peer Contributions

#### Paired representation comparison\.

For Table[3](https://arxiv.org/html/2610.19103#S4.T3), we use full CCPO’s Qwen2\.5\-1\.5B\-Instruct ALFWorld training rollouts from iterations 101–150 to recompute normalized peer weights with and without the accumulated context summary\. Rollouts, queries, recorded peer sets, trajectory exclusions, and return targets are held fixed\. The full representation has 1,573 dimensions; the comparison retains its 1,536\-dimensional hidden\-state block and renormalizes that block to unit length\. The bandwidth rule is unchanged, but its numerical value is recomputed from distances in each representation\. This comparison isolates the effect on weights for fixed rollouts; it is separate from comparing independently trained ablation policies\.

#### Population and statistic\.

We retain the recorded peer\-pool selection, including task\-only fallback pools where used\. This diagnostic covers successful and failed rollouts and retains boundary steps\. It covers 99,424 queries and 1,678,986 query–peer pairs\. For each pair, define the absolute change in normalized weight, measured in percentage points, as

Du​v=100​\|w¯u,vfull−w¯u,vno​\-​summary\|\.D\_\{uv\}=100\\left\|\\bar\{w\}^\{\\mathrm\{full\}\}\_\{u,v\}\-\\bar\{w\}^\{\\mathrm\{no\\text\{\-\}summary\}\}\_\{u,v\}\\right\|\.\(14\)For a thresholddd, the reported percentage is

P\(d\)=100\|𝒬\|∑\(u,v\)∈𝒬𝟏\{Du​v\>d\},P\(d\)=\\frac\{100\}\{\|\\mathcal\{Q\}\|\}\\sum\_\{\(u,v\)\\in\\mathcal\{Q\}\}\\mathbf\{1\}\\\{D\_\{uv\}\>d\\\},\(15\)where𝒬\\mathcal\{Q\}is the set of query–peer step pairs\(u,v\)\(u,v\)\. Pairs receive equal weight, so queries with more peers contribute more terms\. Thresholds of 0\.25, 0\.5, 1, 5, and 10 percentage points yield 62\.1%, 48\.7%, 33\.8%, 8\.6%, and 3\.6% of pairs, respectively\. These are descriptive proportions of the recorded pairs\. These proportions quantify weight redistribution over fixed query\-peer pairs\.

## Appendix FFailure Mode Comparisons

We compare ALFWorld evaluation trajectories from Qwen2\.5\-1\.5B\-Instruct checkpoints at training step 150, with one sampled trajectory per method and task, temperature 0\.4, and a 50\-step limit\. Tables show selected actions and observation excerpts; italicized rows summarize repeated actions\.

### F\.1Repeated Visits Without Acquiring the Target

Figure S2:Evaluation trajectories for the navigation task in Table[S6](https://arxiv.org/html/2610.19103#A6.T6)\.Reproduced from Figure[3](https://arxiv.org/html/2610.19103#S4.F3)\. Each cell denotes one step; revisit counts track navigation to previously visited locations\. CCPO usesACCA\_\{\\mathrm\{CC\}\}only\.Table S6:Repeated navigation versus target acquisition\.Task: “put a toiletpaper in toiletpaperhanger\.” Revisit counts refer to navigation to previously visited locations\.StepActionObservation excerpt / summaryGiGPO: failed at 50 steps; 47 revisits3go to toiletpaperhanger 1On the toiletpaperhanger 1, you see nothing\.4–4946 navigation actions among countertop 1, toilet 1, and toiletpaperhanger 1; every returned observation repeats an earlier one\.G2PO: failed at 50 steps; 42 revisits5open cabinet 2In it, you see a soapbottle 1\.8–15Two navigation cycles: countertop 1→\\tocabinet 2→\\totoilet 1→\\totoiletpaperhanger 1\.HGPO: failed at 50 steps; 44 revisits4go to toiletpaperhanger 1On the toiletpaperhanger 1, you see nothing\.5–1814 navigation actions among toilet 1, countertop 1, and toiletpaperhanger 1; every returned observation repeats an earlier one\.CCPO \(ACCA\_\{\\mathrm\{CC\}\}only\): success at step 7; 0 revisits3go to cabinet 1The cabinet 1 is closed\.4open cabinet 1In it, you see \[…\] a toiletpaper 1\.5take toiletpaper 1 from cabinet 1You pick up the toiletpaper 1 from the cabinet 1\.6go to toiletpaperhanger 1On the toiletpaperhanger 1, you see nothing\.7move toiletpaper 1 to toiletpaperhanger 1You move the toiletpaper 1 to the toiletpaperhanger 1\.
### F\.2Repeating a Completed Subgoal

Figure S3:Evaluation trajectories for the two\-object task in Table[S7](https://arxiv.org/html/2610.19103#A6.T7)\.Each cell denotes one step; revisit counts track navigation to previously visited locations\. CCPO usesACCA\_\{\\mathrm\{CC\}\}only\.Table S7:Repeated manipulation of one object versus placing two distinct objects\.Task: “find two spraybottle and put them in toilet\.”StepActionObservation excerpt / summaryG2PO: failed at 50 steps10move spraybottle 2 to toilet 1You move the spraybottle 2 to the toilet 1\.14go to toilet 1On the toilet 1, you see a cloth 3, and a spraybottle 2\.15take spraybottle 2 from toilet 1You pick up the spraybottle 2 from the toilet 1\.16move spraybottle 2 to toilet 1You move the spraybottle 2 to the toilet 1\.37–39, 48–49Two further take–replace cycles with spraybottle 2; step 38 is an invalid action\.CCPO \(ACCA\_\{\\mathrm\{CC\}\}only\): success at step 154“1, and you have already taken ”Nothing happens\.5take spraybottle 1 from garbagecan 1You pick up the spraybottle 1 from the garbagecan 1\.8move spraybottle 1 to toilet 1You move the spraybottle 1 to the toilet 1\.11open cabinet 3In it, you see a spraybottle 2\.12take spraybottle 2 from cabinet 3You pick up the spraybottle 2 from the cabinet 3\.13“\. now, you are at step 13, and”Nothing happens\.15move spraybottle 2 to toilet 1You move the spraybottle 2 to the toilet 1\.GiGPO: failed at 50 steps11take soapbottle 2 from countertop 1You pick up the soapbottle 2 from the countertop 1\.13move soapbottle 2 to toilet 1You move the soapbottle 2 to the toilet 1\.HGPO: failed at 50 steps1–50Navigation and cabinet opening; no object is picked up\.

## Appendix GTrajectory Lengths

Figure[S4](https://arxiv.org/html/2610.19103#A7.F4)reports average steps among successful trajectories\. At 1\.5B, the ALFWorld reference configuration \(ACCA\_\{\\mathrm\{CC\}\}\) averages8\.65±0\.088\.65\\pm 0\.08steps, compared with5\.13±0\.115\.13\\pm 0\.11for the WebShop reference \(ACC\+AEPA\_\{\\mathrm\{CC\}\}\+A\_\{\\mathrm\{EP\}\}\)\. WebShop’s shorter successful trajectories motivate restricting the context\-window variations to ALFWorld\.

Figure S4:Average steps among successful trajectories on ALFWorld and WebShop\.Panels show Qwen2\.5\-1\.5B/7B\-Instruct agents and a shared off\-the\-shelf Gemini reference\. Error bars show sample standard deviations across three evaluation subsets; no deviation is reported for Gemini\.
## Appendix HExact\-Match Fallback and Peer Availability

Figure S5:Exact\-match fallback and peer availability during training\.Results use Qwen2\.5\-1\.5B\-Instruct on ALFWorld and WebShop\. \(a,b\) Percentage of rows falling back from the exact anchor\-observation key to a task\-only key, comparing CCPO \(full\) with five ablations\. \(c,d\) No\-peer row percentages, computed as the singleton\-bucket fraction divided by mean bucket size; CCPO \(full\) is highlighted and ablations are gray\. Faint traces show raw per\-step values, while smoothed curves use centered five\-step moving averages\. Shading marks steps 101–150; annotations report unweighted means of raw per\-step values over this window\. All ALFWorld arms useACCA\_\{\\mathrm\{CC\}\}only, and all WebShop arms useACC\+AEPA\_\{\\mathrm\{CC\}\}\+A\_\{\\mathrm\{EP\}\}\. Each arm represents one recorded training run\. Vertical\-axis ranges differ between benchmarks\.