Contrastive Branch Policy Optimization

# Contrastive Branch Policy Optimization
Source: [https://arxiv.org/html/2608.24300](https://arxiv.org/html/2608.24300)
CCS:Computing methodologies Reinforcement learningCCS:Computing methodologies Intelligent agentsYing WangAffiliation:Alibaba Group,Hangzhou,ChinaAffiliation:Harbin Institute of Technology,Harbin,Chinaemail:[24s005080@stu\.hit\.edu\.cn](mailto:[email protected])Changlin QiuNote:Corresponding author: Changlin Qiu \([qiuchanglin\.qcl@taobao\.com](mailto:[email protected])\)\.Affiliation:Alibaba Group,Hangzhou,Chinaemail:[qiuchanglin\.qcl@taobao\.com](mailto:[email protected]),Bang LinAffiliation:Alibaba Group,Hangzhou,Chinaemail:[linbang\.lb@taobao\.com](mailto:[email protected]),Linbo JinAffiliation:Alibaba Group,Hangzhou,Chinaemail:[yuyi\.jlb@alibaba\-inc\.com](mailto:[email protected]),Wen JiangAffiliation:Alibaba Group,Hangzhou,Chinaemail:[wen\.jiangw@alibaba\-inc\.com](mailto:[email protected]),Zhe SunAffiliation:Alibaba Group,Hangzhou,Chinaemail:[sz481403@alibaba\-inc\.com](mailto:[email protected])andJingli YangAffiliation:Harbin Institute of Technology,Harbin,Chinaemail:[jinglidg@hit\.edu\.cn](mailto:[email protected])

###### Abstract\.

Reinforcement learning with verifiable rewards \(RLVR\) enables language models to learn multi\-turn interaction with external tools, yet its sparse outcome rewards provide no signal for identifying which intermediate decisions are responsible for success\. Branch sampling induces local comparisons among alternative continuations, but existing methods tend to conflate two distinct problems: allocating a fixed rollout budget and translating branch outcomes into token\-level credit\. We introduce Contrastive Branch Policy Optimization \(CBPO\), which disentangles these two problems and assigns a dedicated mechanism to each\. Generation entropy screens candidate branch positions across the entire response, while path\-level and node\-level decay distribute a fixed budget across trajectories and positions to prevent exploration from collapsing onto a few paths or adjacent tokens\. A parent trajectory together with the branches that share an identical token prefix forms an exact\-prefix group, and the reward variation within this controlled group defines the Contrastive Branch Value \(CBV\), an outcome\-based estimate of local decision sensitivity that rescales continuation advantages without altering their sign\. When multiple nodes are selected along the same trajectory, CBPO partitions it into non\-overlapping credit segments, thereby avoiding duplicated gradients on shared tokens\. Requiring only outcome rewards and no process\-level annotation, CBPO provides a practical solution for fine\-grained credit assignment in tool\-integrated agent training\. Extensive experiments on ten benchmarks—five for mathematical reasoning and five for knowledge\-intensive search—show that CBPO consistently outperforms state\-of\-the\-art policy\-optimization and branch\-based methods, attaining the highest macro\-average accuracy in both domains and across two model scales\.

###### Keywords:

reinforcement learning, large language models, tool\-integrated reasoning, credit assignment

## 1\.Introduction

Chain\-of\-thought prompting\([Wei and others, 2022](https://arxiv.org/html/2608.24300#bib.bib1)\)and reinforcement learning with verifiable rewards \(RLVR\)\([Schulman and others, 2017](https://arxiv.org/html/2608.24300#bib.bib2);[Shao and others, 2024](https://arxiv.org/html/2608.24300#bib.bib3)\)have substantially improved the reasoning capabilities of large language models \(LLMs\)\. Many tasks, however, require current information, exact computation, or feedback from an external environment\. Addressing such tasks entails multi\-turn interaction with search engines, code interpreters, or both\([Yao and others, 2023](https://arxiv.org/html/2608.24300#bib.bib8);[Gou and others, 2024](https://arxiv.org/html/2608.24300#bib.bib9)\)\. Recent systems apply reinforcement learning to these interactions\([Feng and others, 2026](https://arxiv.org/html/2608.24300#bib.bib10);[Li et al\., 2025](https://arxiv.org/html/2608.24300#bib.bib11)\), including settings that require coordinated use of multiple tools\([Dong and others, 2025b](https://arxiv.org/html/2608.24300#bib.bib12)\)\.

The resulting training signal nevertheless remains coarse\. Most RLVR and agentic reinforcement\-learning methods assign a single terminal reward to every generated token in a trajectory\. Such uniform credit cannot distinguish a decisive tool request or correction from text that merely precedes a successful answer\. Process supervision\([Lightman and others, 2024](https://arxiv.org/html/2608.24300#bib.bib13)\)and step\-wise preference optimization\([Lai and others, 2024](https://arxiv.org/html/2608.24300#bib.bib14)\)provide finer signals but require intermediate labels or preferences\. Tree\-based sampling instead compares continuations from a shared history using only outcome rewards\([Hou et al\., 2025](https://arxiv.org/html/2608.24300#bib.bib15)\)\. This approach, however, still requires a principled policy for locating branches under a limited budget and mapping their outcomes to local policy updates\.

Existing methods address these requirements only partially\. GIGPO and Tree\-GRPO construct local advantages from shared or tree\-structured histories\([Feng et al\., 2025](https://arxiv.org/html/2608.24300#bib.bib16);[Ji et al\., 2026](https://arxiv.org/html/2608.24300#bib.bib17)\), whereas ARPO uses generation entropy to select branch points after tool feedback\([Dong and others, 2025a](https://arxiv.org/html/2608.24300#bib.bib18)\)\. Entropy provides an inexpensive proxy for token\-level uncertainty\([Lin et al\., 2024](https://arxiv.org/html/2608.24300#bib.bib19);[Cheng et al\., 2025](https://arxiv.org/html/2608.24300#bib.bib20);[Wang and others, 2025](https://arxiv.org/html/2608.24300#bib.bib21)\), but does not measure outcome sensitivity\. Allocation based solely on entropy can concentrate branches at adjacent positions, overlook consequential decisions outside tool boundaries, or emphasize variation that leaves the final answer unchanged\.

Contrastive Branch Policy Optimization \(CBPO\) separates candidate discovery from credit assignment\. CBPO scans local entropy throughout the response and applies path\-level and node\-level decay to distribute a fixed branch budget\. A parent trajectory and branches sharing an identical token prefix form an exact\-prefix group\. Reward variation within this controlled group defines Contrastive Branch Value \(CBV\), an outcome\-based estimate of local decision sensitivity\. A bounded form of CBV changes the magnitude, but not the sign, of continuation advantages\. Copied prefixes receive no branch gradient, and multiple branch nodes partition a parent trajectory into non\-overlapping credit segments\. The two signals therefore serve distinct functions: entropy identifies alternative continuations, whereas observed outcome variation estimates their task relevance\.

The evaluation spans two model scales and includes five tool\-augmented mathematical benchmarks and five knowledge\-intensive search benchmarks\. CBPO consistently outperforms state\-of\-the\-art policy\-optimization and branch\-based methods, attaining the highest macro average in both domains\.

The main contributions are as follows:

- •Branch selection is formulated as budgeted exploration over the entire response\. Fixed\-interval entropy windows expose candidates beyond tool boundaries, while path\-level and node\-level decay prevent allocation from collapsing onto a few trajectories or adjacent positions\.
- •CBV estimates local decision sensitivity from reward variation within exact\-prefix groups\. Standardized and bounded CBV modulates continuation advantages without changing their signs\. Prefix masking and non\-overlapping segmentation prevent repeated credit on shared tokens\.
- •Experiments across ten benchmarks and two model scales show that CBPO consistently outperforms strong policy\-optimization and branch\-based baselines\. Cross\-scale ablations further attribute these gains to balanced branch allocation and outcome\-contrastive credit assignment\.

## 2\.Related Work

### 2\.1\.Fine\-Grained Credit Assignment

PPO optimizes a policy from trajectory\-level returns\([Schulman and others, 2017](https://arxiv.org/html/2608.24300#bib.bib2)\)\. GRPO\-based reasoning systems likewise derive one advantage from each completed response\([Shao and others, 2024](https://arxiv.org/html/2608.24300#bib.bib3);[Guo and others, 2025](https://arxiv.org/html/2608.24300#bib.bib4)\)\. Subsequent variants improve optimization stability or scaling\([Yu and others, 2025](https://arxiv.org/html/2608.24300#bib.bib5);[Hu and others, 2025](https://arxiv.org/html/2608.24300#bib.bib6)\), while GSPO moves the optimization unit from tokens to sequences\([Zheng et al\., 2025](https://arxiv.org/html/2608.24300#bib.bib29)\)\. These methods train directly from outcome rewards but do not distinguish decisive intermediate choices from weakly related transitional text\. Problem decomposition, computational correction, tool selection, and answer verification can contribute differently to a multi\-step solution\. Assigning a single advantage may therefore reinforce effective decisions and incidental behavior together\.

Process rewards\([Lightman and others, 2024](https://arxiv.org/html/2608.24300#bib.bib13)\)and step\-wise preference optimization\([Lai and others, 2024](https://arxiv.org/html/2608.24300#bib.bib14)\)supervise intermediate reasoning more directly\. Preference objectives such as DPO\([Rafailov et al\., 2023](https://arxiv.org/html/2608.24300#bib.bib30)\)and token\-entropy methods\([He et al\., 2026](https://arxiv.org/html/2608.24300#bib.bib31)\)offer alternative local signals\. Process labels and learned value estimates can provide dense feedback but entail annotation costs or estimation error\. Entropy requires neither annotation nor learned value estimation, yet measures predictive uncertainty rather than a position’s empirical association with the final reward\.

Tree\-based methods reuse prefixes and sample several continuations from intermediate states, enabling local comparisons at controlled cost\. Online variants derive advantages from current\-policy descendants, within\-tree contrasts, or recurring histories\([Hou et al\., 2025](https://arxiv.org/html/2608.24300#bib.bib15);[Ji et al\., 2026](https://arxiv.org/html/2608.24300#bib.bib17)\)\. Entropy can identify uncertain candidates at low computational cost\([Lin et al\., 2024](https://arxiv.org/html/2608.24300#bib.bib19);[Cheng et al\., 2025](https://arxiv.org/html/2608.24300#bib.bib20)\)\. Moreover, a small fraction of high\-entropy tokens can dominate effective updates\([Wang and others, 2025](https://arxiv.org/html/2608.24300#bib.bib21)\)\. EAPO combines reward polarity with token entropy\. AEPO balances entropy during rollout and optimization while limiting repeated branches at consecutive high\-entropy positions\([He et al\., 2026](https://arxiv.org/html/2608.24300#bib.bib31);[Dong et al\., 2025](https://arxiv.org/html/2608.24300#bib.bib32)\)\. These approaches use uncertainty to guide optimization, but divergent token distributions need not yield divergent outcomes\. CBPO instead restricts entropy to candidate screening and estimates local credit from outcome contrasts among continuations sharing an identical prefix\.

### 2\.2\.Tool\-Integrated Reasoning

External tools allow language models to access current information, perform exact computation, and act on an environment\([Yao and others, 2023](https://arxiv.org/html/2608.24300#bib.bib8);[Lin and Xu, 2025](https://arxiv.org/html/2608.24300#bib.bib33)\)\. Tool\-integrated reasoning extends beyond the binary decision to invoke a tool\. A model must select a tool, construct the request, determine when to invoke it, verify the response, integrate the result, and terminate appropriately\. Search provides open\-world evidence, whereas code interpreters support calculation and programmatic verification\. Coordinating both requires repeated transitions between internal reasoning and external actions as new observations arrive\.

Work in this area has progressed from prompted reasoning and acting\([Yao and others, 2023](https://arxiv.org/html/2608.24300#bib.bib8);[Gou and others, 2024](https://arxiv.org/html/2608.24300#bib.bib9)\)to reinforcement learning for search\([Jin et al\., 2025](https://arxiv.org/html/2608.24300#bib.bib34);[Sun et al\., 2025](https://arxiv.org/html/2608.24300#bib.bib35)\), code interpreters\([Feng and others, 2026](https://arxiv.org/html/2608.24300#bib.bib10);[Li et al\., 2025](https://arxiv.org/html/2608.24300#bib.bib11)\), and multi\-tool coordination\([Dong and others, 2025b](https://arxiv.org/html/2608.24300#bib.bib12)\)\. Recent WSDM studies train autonomous programmatic agents and optimize tool selection with reinforcement learning\([Jiang et al\., 2026](https://arxiv.org/html/2608.24300#bib.bib38);[Zhang et al\., 2026a](https://arxiv.org/html/2608.24300#bib.bib39)\)\. Unlike prompting or supervised imitation, reinforcement learning can optimize the complete interaction from environmental feedback and terminal outcomes\. In long trajectories, however, success may depend on pre\-call reasoning, request construction, feedback interpretation, or post\-call synthesis\. Tool\-return boundaries therefore capture only a subset of potentially decisive positions\.

Knowledge\-intensive tasks also depend on retrieval quality and timing\. Adaptive retrieval can extend beyond an initial ranking when the first\-stage candidate pool has insufficient recall\([Rathee et al\., 2025](https://arxiv.org/html/2608.24300#bib.bib41)\)\. In LLM\-based fact\-checking, adaptive evidence retrieval allows a model to determine when internal knowledge is inadequate or conflicts with external evidence\([Zhang et al\., 2026b](https://arxiv.org/html/2608.24300#bib.bib40)\)\. These WSDM studies concern retrieval and verification rather than token\-level credit assignment, but reveal the same structural challenge\. An agent must determine where additional computation or evidence can alter the final outcome\.

Long\-horizon tool learning often improves planning or attribution through reward shaping, process supervision, or step\-level optimization\. Intermediate supervision is costly, while predefined states and step boundaries constrain the granularity of credit assignment\. GIGPO derives step\-level advantages from shared histories\. ARPO branches after tool feedback and assigns the resulting advantage to the suffix\([Feng et al\., 2025](https://arxiv.org/html/2608.24300#bib.bib16);[Dong and others, 2025a](https://arxiv.org/html/2608.24300#bib.bib18)\)\. The former depends on repeated history states, and the latter treats tool returns as fixed branch boundaries\. CBPO searches the full response for branch candidates, then attributes local credit by comparing outcomes under an identical prefix\.

## 3\.Preliminaries

Tool\-integrated rollout\.Given a problemxxdrawn from dataset𝒟\\mathcal\{D\}and a tool set𝒯\\mathcal\{T\}, policyπθ\\pi\_\{\\theta\}generates an interaction trajectoryτ\\tauthat interleaves model tokens, tool requests, and environmental observations\. Letz=\(z1,…,z\|z\|\)z=\(z\_\{1\},\\ldots,z\_\{\|z\|\}\)collect the model\-generated tokens ofτ\\tau, and leto<to\_\{<t\}denote the tool observations available before positiontt\. Trajectory generation then factorizes autoregressively over model tokens only:

\(1\)Pθ​\(τ∣x;𝒯\)=∏t=1\|z\|πθ​\(zt∣x,z<t,o<t;𝒯\)\.P\_\{\\theta\}\(\\tau\\mid x;\\mathcal\{T\}\)=\\prod\_\{t=1\}^\{\|z\|\}\\pi\_\{\\theta\}\\\!\\left\(z\_\{t\}\\mid x,z\_\{<t\},o\_\{<t\};\\mathcal\{T\}\\right\)\.
Observations are inserted by the environment upon each tool call; they condition subsequent generation but contribute neither likelihood terms nor policy gradients\. Upon termination, a verifier assigns a bounded outcome rewardR⁡\(τ\)R\(\\tau\)\. Following agentic reinforcement learning formulations\([Dong and others, 2025a](https://arxiv.org/html/2608.24300#bib.bib18);[Dong et al\., 2025](https://arxiv.org/html/2608.24300#bib.bib32)\), training maximizes the KL\-regularized expected reward:

\(2\)maxπθ𝔼x∼𝒟,τ∼πθ\(⋅∣x;𝒯\)\[R\(τ\)\]−βDKL\(πθ∥πref\),\\max\_\{\\pi\_\{\\theta\}\}\\;\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\},\\,\\tau\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x;\\mathcal\{T\}\)\}\\bigl\[R\(\\tau\)\\bigr\]\-\\beta\\,D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\_\{\\theta\}\\,\\\|\\,\\pi\_\{\\mathrm\{ref\}\}\\right\),
whereπref\\pi\_\{\\mathrm\{ref\}\}is a frozen reference policy andβ\\betacontrols the regularization strength\. Shared\-history, suffix\-based, and tree\-search agent RL methods adopt this trajectory\-level reward setting\([Feng et al\., 2025](https://arxiv.org/html/2608.24300#bib.bib16);[Ji et al\., 2026](https://arxiv.org/html/2608.24300#bib.bib17);[Dong and others, 2025a](https://arxiv.org/html/2608.24300#bib.bib18)\)\.

Group\-relative optimization\.For a given problem, Group Relative Policy Optimization \(GRPO\) samplesGGtrajectories and estimates the advantage of trajectorykkas\([Shao and others, 2024](https://arxiv.org/html/2608.24300#bib.bib3)\):

\(3\)Ak=R⁡\(τk\)−μRσR\+ϵ,μR=1G​∑j=1GR⁡\(τj\),A\_\{k\}=\\frac\{R\(\\tau\_\{k\}\)\-\\mu\_\{R\}\}\{\\sigma\_\{R\}\+\\epsilon\},\\qquad\\mu\_\{R\}=\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}R\(\\tau\_\{j\}\),
Here,μR\\mu\_\{R\}andσR\\sigma\_\{R\}are the within\-group reward mean and standard deviation, andϵ\\epsilonprovides numerical stability\. Standard GRPO assignsAkA\_\{k\}to every model\-generated token in trajectorykk, so the update magnitude is uniform along the trajectory regardless of which intermediate decisions determine the outcome\.

Generation uncertainty\.At generation positiontt, let the context beht=\(x,z<t\)h\_\{t\}=\(x,z\_\{<t\}\)and the next\-token distribution bept=πθ\(⋅∣ht\)p\_\{t\}=\\pi\_\{\\theta\}\(\\cdot\\mid h\_\{t\}\)\. Its Shannon entropy is

\(4\)Ht=−∑v∈𝒱pt,vlogpt,v,H\_\{t\}=\-\\sum\_\{v\\in\\mathcal\{V\}\}p\_\{t,v\}\\log p\_\{t,v\},
where𝒱\\mathcal\{V\}denotes the vocabulary\. LargerHtH\_\{t\}indicates greater uncertainty over the next token\. This quantity reflects the dispersion of the generation distribution at positionttrather than the confidence of any particular sampled token\. Prior work uses token entropy to identify critical decisions\([Lin et al\., 2024](https://arxiv.org/html/2608.24300#bib.bib19)\)and characterize exploration in language\-model reasoning\([Cheng et al\., 2025](https://arxiv.org/html/2608.24300#bib.bib20);[Wang and others, 2025](https://arxiv.org/html/2608.24300#bib.bib21)\)\.

## 4\.Contrastive Branch Policy Optimization

### 4\.1\.Overview

CBPO addresses two coupled limitations of branch\-based RL: concentrated exploration and imprecise local credit\. The method uses generation uncertainty for candidate selection and observed outcome variation to modulate update magnitude\. Figure[1](https://arxiv.org/html/2608.24300#acmlabel1)summarizes the resulting training framework\. Starting from complete interaction trajectories, CBPO scans each response for uncertain positions and allocates a fixed branch budget through path\-level and node\-level decay\. Rewards are then compared among continuations sharing an exact prefix to construct CBV\-aware local credit\. Shared\-prefix deduplication and non\-overlapping segmentation convert this credit into token\-level advantages\. Section[4\.2](https://arxiv.org/html/2608.24300#S4.SS2)constructs balanced exact\-prefix groups, and Section[4\.3](https://arxiv.org/html/2608.24300#S4.SS3)converts their outcomes into policy updates\.

The design maintains three invariants\. First, every problem receives the same total rollout budget because unused branch slots are reallocated to independent complete trajectories\. Second, each local comparison conditions on an identical token history, so reward variation arises only from resampled continuations\. Third, copied prefixes are excluded from branch losses, and parent trajectories are partitioned into non\-overlapping intervals when several nodes are selected\. In tool\-integrated trajectories, environmental observations condition later actions but are neither scanned nor assigned policy gradients, since they are not model\-generated\. These constraints separate allocation from attribution and prevent additional branches from duplicating shared\-token gradients\.

![End-to-end CBPO pipeline from tool-integrated trajectory sampling through balanced branching, CBV estimation, and policy optimization.](https://arxiv.org/html/2608.24300v1/figures/overview.png)Figure 1\.CBPO training framework\. The model first performs interactive reasoning with Python and Search tools and samples complete initial trajectories\. Candidate nodes are identified across the full response, and path\-level and node\-level decay allocate the branch budget\. Outcome rewards estimate CBV within exact\-prefix groups\. Shared\-prefix deduplication and non\-overlapping hierarchical segmentation then construct token\-level advantages for the policy update\.End\-to\-end CBPO pipeline from tool\-integrated trajectory sampling through balanced branching, CBV estimation, and policy optimization\.
### 4\.2\.Entropy\-Guided Balanced Branch Exploration

Exhaustive branching at every token is computationally infeasible, whereas branching only at tool\-return boundaries assumes that consequential decisions coincide with predefined interaction events\. Entropy provides an inexpensive candidate signal, but an unregularized ranking can allocate most of a fixed budget to a few trajectories or neighboring positions\. This stage therefore combines full\-response candidate discovery with explicit coverage control\. The procedure first samplesN0N\_\{0\}complete trajectories from the current policy\. Figure[2](https://arxiv.org/html/2608.24300#acmlabel2)illustrates how fixed\-interval scanning identifies high\-entropy decisions beyond tool\-call boundaries\.

![Line plot of token entropy across a full response, with fixed-interval windows, tool-call positions, and selected candidate branch points.](https://arxiv.org/html/2608.24300v1/figures/cbpo_candidate_discovery.png)Figure 2\.Full\-response candidate discovery\. Fixed\-interval entropy scans identify high\-entropy decisions beyond predefined tool\-call boundaries\.Line plot of token entropy across a full response, with fixed\-interval windows, tool\-call positions, and selected candidate branch points\.Full\-trajectory candidate identification\.For trajectoryii, CBPO places candidate boundaries throughout the response at intervals ofdmind\_\{\\min\}:

\(5\)ℬi=\{bi,q=qdmin\}q=1Qi,\\mathcal\{B\}\_\{i\}=\\\{b\_\{i,q\}=qd\_\{\\min\}\\\}\_\{q=1\}^\{Q\_\{i\}\},
whereQiQ\_\{i\}is the number of valid candidates\. Let𝒲i,b\\mathcal\{W\}\_\{i,b\}denote the window ofwwmodel\-generated tokens beginning at boundaryb∈ℬib\\in\\mathcal\{B\}\_\{i\}\. Its normalized entropy is

\(6\)Hi,b=−1\|𝒲i,b\|​log⁡\|𝒱\|∑t∈𝒲i,b∑v∈𝒱Ki,tpi,t,vlogpi,t,v,H\_\{i,b\}=\-\\frac\{1\}\{\|\\mathcal\{W\}\_\{i,b\}\|\\log\|\\mathcal\{V\}\|\}\\sum\_\{t\\in\\mathcal\{W\}\_\{i,b\}\}\\sum\_\{v\\in\\mathcal\{V\}\_\{K\}^\{i,t\}\}p\_\{i,t,v\}\\log p\_\{i,t,v\},
where𝒱Ki,t\\mathcal\{V\}\_\{K\}^\{i,t\}is the top\-KKcandidate set at tokentt, andpi,t,vp\_\{i,t,v\}is the probability of candidatevv\. The retained probability mass is not renormalized\. Thus,Hi,bH\_\{i,b\}is a truncated entropy proxy used only to rank candidate locations, not the entropy of a top\-KKdistribution\.

The first window of each trajectory provides a path\-specific reference:

\(7\)Hiroot=Hi,0\.H\_\{i\}^\{\\mathrm\{root\}\}=H\_\{i,0\}\.
Local uncertainty at boundarybbis measured by the entropy increase relative to this reference:

\(8\)Δ​Hi,b=Hi,b−Hiroot\.\\Delta H\_\{i,b\}=H\_\{i,b\}\-H\_\{i\}^\{\\mathrm\{root\}\}\.
The corresponding raw priority is

\(9\)Pi,braw=clip⁡\(α\+γ​Δ​Hi,b,0,1\),P\_\{i,b\}^\{\\mathrm\{raw\}\}=\\operatorname\{clip\}\\left\(\\alpha\+\\gamma\\Delta H\_\{i,b\},0,1\\right\),
whereα\\alphasets the base priority,γ\\gammascales the entropy increase, andclip\\operatorname\{clip\}restricts the score to\[0,1\]\[0,1\]\.

Path\-node balanced budgeting\.An entropy ranking alone can allocate most branches to a few trajectories or neighboring positions\. LetLiL\_\{i\}count branches assigned to pathii, and letli,bl\_\{i,b\}count branches at node\(i,b\)\(i,b\)\. CBPO updates the priority as

\(10\)Pi,bbal=Pi,braw\(1\+Li\)ρpath​\(1\+li,b\)ρnode,P\_\{i,b\}^\{\\mathrm\{bal\}\}=\\frac\{P\_\{i,b\}^\{\\mathrm\{raw\}\}\}\{\(1\+L\_\{i\}\)^\{\\rho\_\{\\mathrm\{path\}\}\}\(1\+l\_\{i,b\}\)^\{\\rho\_\{\\mathrm\{node\}\}\}\},
whereρpath,ρnode≥0\\rho\_\{\\mathrm\{path\}\},\\rho\_\{\\mathrm\{node\}\}\\geq 0control path\-level and node\-level decay\. Before allocation, CBPO retains at mostJmaxJ\_\{\\max\}candidates from each parent\. A candidate remains eligible whileli,b<Bnodel\_\{i,b\}<B\_\{\\mathrm\{node\}\}andLi<BpathL\_\{i\}<B\_\{\\mathrm\{path\}\}\. The hard caps are necessary because power decay never reduces a priority exactly to zero\. Updating both counts after each sampled branch progressively shifts allocation toward less\-explored paths and nodes\.

At each allocation step, CBPO selects the eligible candidate with the largestPbalP^\{\\mathrm\{bal\}\}\. A branch is sampled only if this score exceedsκ\\kappa\. Otherwise, independent complete trajectories fill the unused rollout slots, preserving the fixed budget\.

At a selected node, the trajectory is decomposed at token boundarybbas

\(11\)τ=\(pb,cb\),pb=τ<b,cb=τ≥b,\\tau=\(p\_\{b\},c\_\{b\}\),\\qquad p\_\{b\}=\\tau\_\{<b\},\\qquad c\_\{b\}=\\tau\_\{\\geq b\},
Branch generation holds prefixpbp\_\{b\}fixed and resamples only continuationcbc\_\{b\}\. The parent and all branches sharingpbp\_\{b\}form an exact\-prefix group𝒢⁡\(pb\)\\mathcal\{G\}\(p\_\{b\}\)\. Comparing their outcomes controls for the preceding token history and isolates variation among the sampled continuations\. The resulting balanced exact\-prefix groups serve as input to the outcome\-contrastive credit\-assignment stage\.

### 4\.3\.Hierarchical Policy Optimization with CBV

The first stage identifies positions at which the policy admits alternatives, but entropy alone cannot establish whether those alternatives affect the reward\. The second stage therefore measures reward variation within each exact\-prefix group and translates it into local credit\. This formulation emphasizes continuations sampled at outcome\-sensitive nodes while suppressing redundant gradients on their shared histories\. Figure[3](https://arxiv.org/html/2608.24300#acmlabel3)illustrates the distinction between shared\-prefix credit and CBV\-aware continuation credit\.

Figure 3\.Outcome\-contrastive credit assignment\. Exact\-prefix continuations are compared by outcome to construct CBV\-aware credit while deduplicating shared\-prefix credit\.Tree schematic in which three continuations share a prefix, receive different outcome rewards, and obtain CBV\-aware continuation credit alongside deduplicated shared\-prefix credit\.CBPO requires only bounded outcome rewards\. Following ReTool and ToRL\([Feng and others, 2026](https://arxiv.org/html/2608.24300#bib.bib10);[Li et al\., 2025](https://arxiv.org/html/2608.24300#bib.bib11)\), mathematical tasks use a binary correctness reward that compares trajectory answera^​\(τ\)\\hat\{a\}\(\\tau\)with reference answera⋆a^\{\\star\}:

\(12\)R⁡\(τ\)=\{1,a^​\(τ\)​is symbolically equivalent to​a⋆,0,otherwise\.R\(\\tau\)=\\begin\{cases\}1,&\\hat\{a\}\(\\tau\)\\text\{ is symbolically equivalent to \}a^\{\\star\},\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}
Search tasks use normalized token\-level F1 or an LLM\-as\-a\-Judge score\. Every reward satisfiesR⁡\(τ\)∈\[0,1\]R\(\\tau\)\\in\[0,1\]; Section[5\.1](https://arxiv.org/html/2608.24300#S5.SS1)gives the benchmark\-specific metrics\.

Contrastive Branch Value\.Consider exact\-prefix group𝒢i=\{τi,0,…,τi,Ki\}\\mathcal\{G\}\_\{i\}=\\\{\\tau\_\{i,0\},\\ldots,\\tau\_\{i,K\_\{i\}\}\\\}, whereτi,0\\tau\_\{i,0\}is the parent and the remaining trajectories are branches\. Contrastive Branch Value \(CBV\) is defined as the population standard deviation of group rewardsRi,0,…,Ri,KiR\_\{i,0\},\\ldots,R\_\{i,K\_\{i\}\}:

\(13\)CBVi=1Ki\+1​∑k=0Ki\(Ri,k−R¯i\)2,R¯i=1Ki\+1​∑k=0KiRi,k\.\\operatorname\{CBV\}\_\{i\}=\\sqrt\{\\frac\{1\}\{K\_\{i\}\+1\}\\sum\_\{k=0\}^\{K\_\{i\}\}\\left\(R\_\{i,k\}\-\\bar\{R\}\_\{i\}\\right\)^\{2\}\},\\qquad\\bar\{R\}\_\{i\}=\\frac\{1\}\{K\_\{i\}\+1\}\\sum\_\{k=0\}^\{K\_\{i\}\}R\_\{i,k\}\.
Larger CBV indicates greater reward variation among continuations sampled after the same prefix\. Because raw CBV is nonnegative, direct addition would only increase advantage values\. Standardization across valid nodes in the batch produces a comparable signed signal:

\(14\)ZiCBV=CBVi−μCBVσCBV\+ϵ\.Z\_\{i\}^\{\\mathrm\{CBV\}\}=\\frac\{\\operatorname\{CBV\}\_\{i\}\-\\mu\_\{\\mathrm\{CBV\}\}\}\{\\sigma\_\{\\mathrm\{CBV\}\}\+\\epsilon\}\.
Here,ZiCBV\>0Z\_\{i\}^\{\\mathrm\{CBV\}\}\>0denotes reward variation above the batch mean\. IfσCBV\\sigma\_\{\\mathrm\{CBV\}\}falls below a numerical threshold, everyZiCBVZ\_\{i\}^\{\\mathrm\{CBV\}\}is set to zero and the base advantages are retained\.

Credit assignment\.CBV identifies relative outcome sensitivity at the group level, but the optimizer still requires token\-level advantages\. Problem\-level GRPO normalization first yields base advantageAi,kA\_\{i,k\}for each trajectory\. The common prefix at nodeiireceives the group\-mean advantage:

\(15\)Aishared=1Ki\+1​∑k=0KiAi,k\.A\_\{i\}^\{\\mathrm\{shared\}\}=\\frac\{1\}\{K\_\{i\}\+1\}\\sum\_\{k=0\}^\{K\_\{i\}\}A\_\{i,k\}\.
Following bounded advantage modulation\([He et al\., 2026](https://arxiv.org/html/2608.24300#bib.bib31)\), CBV changes only the magnitude of continuation advantagekk\. The standardized credit is bounded as

\(16\)Ci,kCBV=clip⁡\(ZiCBV,−\|Ai,k\|ϕ,\|Ai,k\|ϕ\)\.C\_\{i,k\}^\{\\mathrm\{CBV\}\}=\\operatorname\{clip\}\\left\(Z\_\{i\}^\{\\mathrm\{CBV\}\},\-\\frac\{\|A\_\{i,k\}\|\}\{\\phi\},\\frac\{\|A\_\{i,k\}\|\}\{\\phi\}\\right\)\.
The CBV\-aware advantage is

\(17\)Ai,kfinal=Ai,k\+η​sign⁡\(Ai,k\)​stopgrad⁡\(Ci,kCBV\),A\_\{i,k\}^\{\\mathrm\{final\}\}=A\_\{i,k\}\+\\eta\\,\\operatorname\{sign\}\(A\_\{i,k\}\)\\operatorname\{stopgrad\}\\left\(C\_\{i,k\}^\{\\mathrm\{CBV\}\}\\right\),
whereη\\etacontrols modulation strength andϕ\\philimits its relative magnitude\. For0≤η<ϕ0\\leq\\eta<\\phi, the modulated advantage retains the original sign and therefore preserves the local optimization direction\.

Hierarchical segmentation for multiple nodes\.Let parent trajectoryiicontainJiJ\_\{i\}selected boundaries

\(18\)bi,1<bi,2<⋯<bi,Ji\.b\_\{i,1\}<b\_\{i,2\}<\\cdots<b\_\{i,J\_\{i\}\}\.
Multiple selected ancestors can otherwise assign credit repeatedly to the same suffix\. CBPO therefore partitions each parent trajectory into non\-overlapping intervals, with token advantage

\(19\)A~t=\{Ai,1shared,0≤t<bi,1,12​\(Ai,j,0final\+Ai,j\+1shared\),bi,j≤t<bi,j\+1,Ai,Ji,0final,t≥bi,Ji\.\\widetilde\{A\}\_\{t\}=\\begin\{cases\}A\_\{i,1\}^\{\\mathrm\{shared\}\},&0\\leq t<b\_\{i,1\},\\\\\[3\.0pt\] \\frac\{1\}\{2\}\\left\(A\_\{i,j,0\}^\{\\mathrm\{final\}\}\+A\_\{i,j\+1\}^\{\\mathrm\{shared\}\}\\right\),&b\_\{i,j\}\\leq t<b\_\{i,j\+1\},\\\\\[3\.0pt\] A\_\{i,J\_\{i\},0\}^\{\\mathrm\{final\}\},&t\\geq b\_\{i,J\_\{i\}\}\.\\end\{cases\}
whereAi,j,0finalA\_\{i,j,0\}^\{\\mathrm\{final\}\}is the parent’s continuation advantage at itsjj\-th selected node\. Each intermediate interval is simultaneously a suffix of the preceding node and a shared prefix of the following node\. The two associated terms therefore receive equal weight\.

CBPO retains the GRPO objective, replacing only trajectory\-level advantageAkA\_\{k\}with segmented token advantageA~k,t\\widetilde\{A\}\_\{k,t\}\. For model\-generated tokenyk,ty\_\{k,t\}, the importance ratio is

\(20\)rk,t​\(θ\)=πθ​\(yk,t∣x,yk,<t\)πold​\(yk,t∣x,yk,<t\),r\_\{k,t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(y\_\{k,t\}\\mid x,y\_\{k,<t\}\)\}\{\\pi\_\{\\mathrm\{old\}\}\(y\_\{k,t\}\\mid x,y\_\{k,<t\}\)\},and its clipped form

\(21\)r¯k,t​\(θ\)=clip⁡\(rk,t​\(θ\),1−ε,1\+ε\),\\bar\{r\}\_\{k,t\}\(\\theta\)=\\operatorname\{clip\}\\bigl\(r\_\{k,t\}\(\\theta\),1\-\\varepsilon,1\+\\varepsilon\\bigr\),\(22\)ℓk,t​\(θ\)=min⁡\(rk,t​\(θ\)​A~k,t,r¯k,t​\(θ\)​A~k,t\)\.\\ell\_\{k,t\}\(\\theta\)=\\min\\\!\\left\(r\_\{k,t\}\(\\theta\)\\widetilde\{A\}\_\{k,t\},\\bar\{r\}\_\{k,t\}\(\\theta\)\\widetilde\{A\}\_\{k,t\}\\right\)\.The CBPO objective is

\(23\)𝒥CBPO\(θ\)=𝔼\[1M∑k=1M1\|yk\|∑t=1\|yk\|\(ℓk,t\(θ\)−βDKL\(πθ∥πref\)\)\]\.\\mathcal\{J\}\_\{\\mathrm\{CBPO\}\}\(\\theta\)=\\mathbb\{E\}\\\!\\left\[\\frac\{1\}\{M\}\\sum\_\{k=1\}^\{M\}\\frac\{1\}\{\|y\_\{k\}\|\}\\sum\_\{t=1\}^\{\|y\_\{k\}\|\}\\left\(\\ell\_\{k,t\}\(\\theta\)\-\\beta D\_\{\\mathrm\{KL\}\}\\left\(\\pi\_\{\\theta\}\\,\\\|\\,\\pi\_\{\\mathrm\{ref\}\}\\right\)\\right\)\\right\]\.
The objective connects the two stages: balanced exploration defines the controlled comparisons, and CBV determines their contribution to non\-overlapping continuation updates\. Algorithm[1](https://arxiv.org/html/2608.24300#alg1)summarizes the end\-to\-end training procedure\.

Algorithm 1Contrastive Branch Policy Optimization1:initial policy

πθinit\\pi\_\{\\theta\_\{\\mathrm\{init\}\}\}, reference policy

πref\\pi\_\{\\mathrm\{ref\}\}, data

𝒟\\mathcal\{D\}, tools

𝒯\\mathcal\{T\}
2:training steps

SS, rollout budget

MM, initial count

N0N\_\{0\}, caps

Jmax,Bnode,BpathJ\_\{\\max\},B\_\{\\mathrm\{node\}\},B\_\{\\mathrm\{path\}\}
3:

α,γ,κ,ρpath,ρnode,dmin,w,K,η,ϕ\\alpha,\\gamma,\\kappa,\\rho\_\{\\mathrm\{path\}\},\\rho\_\{\\mathrm\{node\}\},d\_\{\\min\},w,K,\\eta,\\phi
4:trained policy

πθ\\pi\_\{\\theta\}
5:

πθ←πθinit\\pi\_\{\\theta\}\\leftarrow\\pi\_\{\\theta\_\{\\mathrm\{init\}\}\}
6:fortraining step

s=1,…,Ss=1,\\ldots,Sdo

7:

πold←πθ\\pi\_\{\\mathrm\{old\}\}\\leftarrow\\pi\_\{\\theta\}; sample minibatch

𝒟b⊂𝒟\\mathcal\{D\}\_\{b\}\\subset\\mathcal\{D\}
8:for allproblems

x∈𝒟bx\\in\\mathcal\{D\}\_\{b\}do

9:sample

N0N\_\{0\}parent trajectories; initialize rollout set

ℛx\\mathcal\{R\}\_\{x\}
10:for allparent trajectories

τi∈ℛx\\tau\_\{i\}\\in\\mathcal\{R\}\_\{x\}do

11:scan

dmind\_\{\\min\}\-spaced boundaries and compute

Hi,b,Δ​Hi,b,Pi,brawH\_\{i,b\},\\Delta H\_\{i,b\},P\_\{i,b\}^\{\\mathrm\{raw\}\}
12:retain the top

JmaxJ\_\{\\max\}candidates; set

Li←0L\_\{i\}\\leftarrow 0and

li,b←0l\_\{i,b\}\\leftarrow 0
13:while

\|ℛx\|<M\|\\mathcal\{R\}\_\{x\}\|<Mdo

14:form eligible set

𝒞x=\{\(i,b\):Li<Bpath,li,b<Bnode\}\\mathcal\{C\}\_\{x\}=\\\{\(i,b\):L\_\{i\}<B\_\{\\mathrm\{path\}\},\\ l\_\{i,b\}<B\_\{\\mathrm\{node\}\}\\\}
15:if

𝒞x=∅\\mathcal\{C\}\_\{x\}=\\varnothingthen

16:sample

M−\|ℛx\|M\-\|\\mathcal\{R\}\_\{x\}\|independent complete trajectories;break

17:compute

Pi,bbalP\_\{i,b\}^\{\\mathrm\{bal\}\}and choose

\(i∗,b∗\)=arg⁡max\(i,b\)∈𝒞x⁡Pi,bbal\(i^\{\*\},b^\{\*\}\)=\\arg\\max\_\{\(i,b\)\\in\\mathcal\{C\}\_\{x\}\}P\_\{i,b\}^\{\\mathrm\{bal\}\}
18:if

Pi∗,b∗bal\>κP\_\{i^\{\*\},b^\{\*\}\}^\{\\mathrm\{bal\}\}\>\\kappathen

19:fix prefix

τi∗,<b∗\\tau\_\{i^\{\*\},<b^\{\*\}\}, resample one continuation, and add it to

ℛx\\mathcal\{R\}\_\{x\}
20:

Li∗←Li∗\+1L\_\{i^\{\*\}\}\\leftarrow L\_\{i^\{\*\}\}\+1;

li∗,b∗←li∗,b∗\+1l\_\{i^\{\*\},b^\{\*\}\}\\leftarrow l\_\{i^\{\*\},b^\{\*\}\}\+1
21:else

22:sample

M−\|ℛx\|M\-\|\\mathcal\{R\}\_\{x\}\|independent complete trajectories;break

23:evaluate

R⁡\(τ\)R\(\\tau\)and compute prompt\-level base advantages

Ai,kA\_\{i,k\}over

ℛx\\mathcal\{R\}\_\{x\}
24:build exact\-prefix groups; compute

CBVi\\operatorname\{CBV\}\_\{i\},

ZiCBVZ\_\{i\}^\{\\mathrm\{CBV\}\},

AisharedA\_\{i\}^\{\\mathrm\{shared\}\}, and

Ai,kfinalA\_\{i,k\}^\{\\mathrm\{final\}\}
25:mask each copied branch prefix; assign

Ai,kfinalA\_\{i,k\}^\{\\mathrm\{final\}\}only to its sampled suffix

26:segment each parent by Eq\. \([19](https://arxiv.org/html/2608.24300#S4.E19)\); keep base advantages for fallback rollouts

27:maximize Eq\. \([23](https://arxiv.org/html/2608.24300#S4.E23)\) and update

θ\\theta

### 4\.4\.Theoretical Analysis

Two properties clarify the roles of the two signals\. Conditional entropy bounds the outcome information available in sampled continuations, which supports its use for candidate screening\. CBV estimates conditional outcome variance under a fixed prefix, which supports its use for local credit\. The bounded modulation additionally preserves the advantage sign and local PPO gradient direction\.

Property 1: generation entropy bounds attainable outcome information\.Given exact prefixpp, let continuationC=\(Y1,…,YL\)∼πθ\(⋅∣p\)C=\(Y\_\{1\},\\ldots,Y\_\{L\}\)\\sim\\pi\_\{\\theta\}\(\\cdot\\mid p\)produce final outcome rewardRR\. The chain rule for conditional entropy and the upper bound on conditional mutual information give

\(24\)H⁡\(C∣p\)\\displaystyle H\(C\\mid p\)=∑t=1L𝔼Y<t\|p​\[H⁡\(Yt∣p,Y<t\)\],\\displaystyle=\\sum\_\{t=1\}^\{L\}\\mathbb\{E\}\_\{Y\_\{<t\}\\mid p\}\\bigl\[H\(Y\_\{t\}\\mid p,Y\_\{<t\}\)\\bigr\],I⁡\(C;R∣p\)\\displaystyle I\(C;R\\mid p\)≤H⁡\(C∣p\)\.\\displaystyle\\leq H\(C\\mid p\)\.
For deterministic outcome verification, a nearly deterministic continuation can carry little information about alternative outcomes\. High conditional entropy is not sufficient, however, because variation in wording, formatting, or inconsequential reasoning may leave the answer unchanged\. Thus, entropy provides an upper bound rather than a direct estimate ofI⁡\(C;R∣p\)I\(C;R\\mid p\)\. CBPO uses it to screen for positions where meaningful alternatives may exist, then relies on observed branch rewards for credit assignment\.

Property 2: CBV estimates conditional outcome variance under a shared prefix\.Supposenncontinuations are sampled independently from prefixpp, yielding rewardsR1,…,RnR\_\{1\},\\ldots,R\_\{n\}, and letR¯=n−1​∑kRk\\bar\{R\}=n^\{\-1\}\\sum\_\{k\}R\_\{k\}\. Then

\(25\)CBV2⁡\(p\)=1n​∑k=1n\(Rk−R¯\)2=12​n2​∑i=1n∑j=1n\(Ri−Rj\)2,\\operatorname\{CBV\}^\{2\}\(p\)=\\frac\{1\}\{n\}\\sum\_\{k=1\}^\{n\}\(R\_\{k\}\-\\bar\{R\}\)^\{2\}=\\frac\{1\}\{2n^\{2\}\}\\sum\_\{i=1\}^\{n\}\\sum\_\{j=1\}^\{n\}\(R\_\{i\}\-R\_\{j\}\)^\{2\},
and

\(26\)𝔼⁡\[CBV2⁡\(p\)∣p\]=n−1n​Var⁡\(R∣p\)\.\\mathbb\{E\}\\\!\\left\[\\operatorname\{CBV\}^\{2\}\(p\)\\mid p\\right\]=\\frac\{n\-1\}\{n\}\\operatorname\{Var\}\(R\\mid p\)\.
The pairwise identity follows by expanding the squared differences:

\(27\)∑i=1n∑j=1n\(Ri−Rj\)2=2​n​∑i=1nRi2−2​\(∑i=1nRi\)2\.\\sum\_\{i=1\}^\{n\}\\sum\_\{j=1\}^\{n\}\(R\_\{i\}\-R\_\{j\}\)^\{2\}=2n\\sum\_\{i=1\}^\{n\}R\_\{i\}^\{2\}\-2\\left\(\\sum\_\{i=1\}^\{n\}R\_\{i\}\\right\)^\{2\}\.
Fori≠ji\\neq j, conditional independence gives𝔼⁡\[\(Ri−Rj\)2∣p\]=2​Var⁡\(R∣p\)\\mathbb\{E\}\[\(R\_\{i\}\-R\_\{j\}\)^\{2\}\\mid p\]=2\\operatorname\{Var\}\(R\\mid p\)\. Thenndiagonal terms are zero, leavingn⁡\(n−1\)n\(n\-1\)nonzero ordered pairs\. Substitution into Eq\. \([25](https://arxiv.org/html/2608.24300#S4.E25)\) yields Eq\. \([26](https://arxiv.org/html/2608.24300#S4.E26)\)\.

Equations \([25](https://arxiv.org/html/2608.24300#S4.E25)\) and \([26](https://arxiv.org/html/2608.24300#S4.E26)\) show thatnn−1​CBV2​\(p\)\\tfrac\{n\}\{n\-1\}\\operatorname\{CBV\}^\{2\}\(p\)is an unbiased estimator of conditional outcome variance\. Its pairwise form measures reward divergence among continuations sharing the same token history\. For binary correctness withqp=Pr⁡\(R=1∣p\)q\_\{p\}=\\Pr\(R=1\\mid p\), this variance isqp​\(1−qp\)q\_\{p\}\(1\-q\_\{p\}\)and is maximal atqp=1/2q\_\{p\}=1/2\. Consequently, an uncertain node with consistent outcomes receives little CBV, whereas a node separating success from failure receives more\. Bounded modulation changes update magnitude within a controlled interval while preserving the advantage sign and local PPO gradient direction\.

Property 3: bounded CBV modulation preserves the update direction\.LetC=CCBVC=C^\{\\mathrm\{CBV\}\}satisfy the clipping constraint\|C\|≤\|A\|/ϕ\|C\|\\leq\|A\|/\\phi\. For any nonzero base advantageAA, Eq\. \([17](https://arxiv.org/html/2608.24300#S4.E17)\) can be written as

\(28\)Afinal=w​A,w=1\+η​C\|A\|∈\[1−ηϕ,1\+ηϕ\]\.A^\{\\mathrm\{final\}\}=wA,\\qquad w=1\+\\eta\\frac\{C\}\{\|A\|\}\\in\\left\[1\-\\frac\{\\eta\}\{\\phi\},1\+\\frac\{\\eta\}\{\\phi\}\\right\]\.
When0≤η<ϕ0\\leq\\eta<\\phi, the lower endpoint is positive\. Therefore,AfinalA^\{\\mathrm\{final\}\}has the same sign asAA\. The PPO surrogate is positively homogeneous in its advantage argument, soℓ⁡\(r,w​A\)=w​ℓ​\(r,A\)\\ell\(r,wA\)=w\\ell\(r,A\)forw\>0w\>0\. Because the modulation is stop\-gradient, its local policy gradient is also scaled by the same positiveww\. CBV consequently reallocates update magnitude without reversing the preference induced by the outcome reward\. WhenA=0A=0, the clipping constraint forcesC=0C=0, and the statement holds trivially\.

## 5\.Experiments

### 5\.1\.Experimental Setup

Datasets and evaluation\.The evaluation covers mathematical reasoning and knowledge\-intensive search, two forms of tool\-integrated reasoning\. Each task provides access to Web Search and Python, allowing the policy to select tools without dataset\-specific routing\. Mathematical evaluation uses AIME 2024, AIME 2025, MATH\-500, MATH\([Hendrycks and others, 2021](https://arxiv.org/html/2608.24300#bib.bib23)\), and the complete GSM8K test set\([Cobbe and others, 2021](https://arxiv.org/html/2608.24300#bib.bib22)\)\. Reported metrics include Pass@1, the five\-task macro average, and Pass@3/5 computed from five samples per problem\. Knowledge\-intensive search evaluation uses WebWalker\([Wu et al\., 2025](https://arxiv.org/html/2608.24300#bib.bib24)\), HotpotQA\([Yang et al\., 2018](https://arxiv.org/html/2608.24300#bib.bib25)\), 2WikiMultiHopQA\([Ho et al\., 2020](https://arxiv.org/html/2608.24300#bib.bib26)\), MuSiQue\([Trivedi et al\., 2022](https://arxiv.org/html/2608.24300#bib.bib27)\), and Bamboogle\([Press et al\., 2023](https://arxiv.org/html/2608.24300#bib.bib28)\)\. WebWalker is evaluated with LLM\-as\-a\-Judge Pass@1, whereas the other four benchmarks use normalized token\-level F1\. Domain\-specific averages are reported because these metrics are not directly comparable\.

Table 1\.Measured results \(%\) for Qwen3\-1\.7B and Qwen3\-4B under a unified protocol\. Mathematical tasks report Pass@1; WebWalker reports LLM\-as\-a\-Judge Pass@1; the other search tasks report normalized token\-level F1\. Math Avg\. and Search Avg\. are unweighted five\-task means\. Green subscripts show improvement over size\-matched GIGPO\.MethodMathematical ReasoningKnowledge\-Intensive SearchAIME24AIME25MATH500GSM8KMATHMath Avg\.WebWalkerHotpotQA2WikiMuSiQueBamboogleSearch Avg\.Backbone: Qwen3\-1\.7BTraining\-Free MethodBase13\.316\.782\.290\.886\.757\.95\.022\.025\.09\.534\.019\.1TIR Prompting16\.723\.381\.488\.485\.159\.014\.536\.041\.016\.048\.031\.1\[3pt/3pt\]Supervised and Distillation MethodsSFT16\.723\.382\.891\.087\.660\.316\.548\.058\.022\.558\.040\.6OPD20\.026\.784\.492\.088\.762\.418\.552\.066\.525\.060\.544\.5\[3pt/3pt\]Classic RL MethodGRPO16\.730\.083\.691\.389\.062\.118\.053\.568\.526\.562\.045\.7REINFORCE\+\+20\.026\.783\.291\.988\.362\.018\.551\.065\.524\.061\.044\.0DAPO13\.326\.784\.891\.588\.861\.017\.552\.065\.025\.060\.544\.0GSPO20\.026\.784\.691\.088\.762\.218\.554\.068\.026\.063\.045\.9EAPO23\.330\.085\.691\.890\.064\.120\.555\.069\.532\.065\.048\.4OC\-GRPO20\.033\.385\.292\.489\.564\.119\.554\.569\.027\.064\.046\.8\[3pt/3pt\]Agentic RL MethodGIGPO23\.326\.784\.292\.489\.163\.1Δbase63\.1\_\{\\Delta\_\{\\mathrm\{base\}\}\}23\.558\.073\.029\.567\.050\.2Δbase50\.2\_\{\\Delta\_\{\\mathrm\{base\}\}\}Tree\-GRPO23\.330\.085\.092\.088\.863\.8\+1\.1%63\.8\_\{\{\\color\[rgb\]\{0,0\.88,0\}\+1\.1\\%\}\}24\.058\.573\.530\.068\.050\.8\+1\.2%50\.8\_\{\{\\color\[rgb\]\{0,0\.88,0\}\+1\.2\\%\}\}ARPO26\.730\.085\.091\.788\.364\.3¯\+1\.9%\\underline\{64\.3\}\_\{\{\\color\[rgb\]\{0,0\.88,0\}\+1\.9\\%\}\}24\.559\.074\.030\.569\.051\.4¯\+2\.4%\\underline\{51\.4\}\_\{\{\\color\[rgb\]\{0,0\.88,0\}\+2\.4\\%\}\}CBPO26\.733\.386\.493\.290\.466\.0\+4\.6%\\mathbf\{66\.0\}\_\{\{\\color\[rgb\]\{0,0\.88,0\}\\mathbf\{\+4\.6\\%\}\}\}26\.061\.075\.531\.572\.053\.2\+6\.0%\\mathbf\{53\.2\}\_\{\{\\color\[rgb\]\{0,0\.88,0\}\\mathbf\{\+6\.0\\%\}\}\}Backbone: Qwen3\-4BTraining\-Free MethodBase13\.320\.084\.690\.089\.859\.57\.027\.031\.012\.040\.023\.4TIR Prompting16\.726\.785\.192\.789\.362\.117\.541\.547\.019\.553\.535\.8\[3pt/3pt\]Supervised and Distillation MethodsSFT16\.726\.785\.793\.290\.562\.620\.052\.063\.025\.563\.044\.7OPD23\.333\.386\.594\.189\.665\.421\.556\.071\.528\.065\.548\.5\[3pt/3pt\]Classic RL MethodGRPO16\.740\.086\.093\.891\.365\.621\.557\.573\.029\.566\.549\.6REINFORCE\+\+26\.733\.385\.894\.291\.466\.322\.055\.070\.027\.065\.547\.9DAPO13\.330\.087\.893\.591\.563\.221\.056\.069\.028\.065\.047\.8GSPO20\.033\.387\.193\.491\.265\.021\.557\.072\.029\.066\.049\.1EAPO23\.336\.787\.294\.491\.866\.724\.059\.574\.535\.069\.052\.4OC\-GRPO23\.333\.387\.094\.290\.665\.722\.558\.573\.529\.567\.550\.3\[3pt/3pt\]Agentic RL MethodGIGPO26\.733\.386\.794\.691\.066\.5Δbase66\.5\_\{\\Delta\_\{\\mathrm\{base\}\}\}27\.062\.077\.032\.573\.054\.3Δbase54\.3\_\{\\Delta\_\{\\mathrm\{base\}\}\}Tree\-GRPO26\.733\.387\.594\.091\.966\.7\+0\.3%66\.7\_\{\{\\color\[rgb\]\{0,0\.88,0\}\+0\.3\\%\}\}28\.062\.577\.533\.074\.055\.0\+1\.3%55\.0\_\{\{\\color\[rgb\]\{0,0\.88,0\}\+1\.3\\%\}\}ARPO30\.033\.387\.494\.190\.767\.1¯\+0\.9%\\underline\{67\.1\}\_\{\{\\color\[rgb\]\{0,0\.88,0\}\+0\.9\\%\}\}29\.063\.078\.033\.575\.055\.7¯\+2\.6%\\underline\{55\.7\}\_\{\{\\color\[rgb\]\{0,0\.88,0\}\+2\.6\\%\}\}CBPO30\.036\.789\.596\.394\.069\.3\+4\.2%\\mathbf\{69\.3\}\_\{\{\\color\[rgb\]\{0,0\.88,0\}\\mathbf\{\+4\.2\\%\}\}\}31\.065\.580\.034\.576\.557\.5\+5\.9%\\mathbf\{57\.5\}\_\{\{\\color\[rgb\]\{0,0\.88,0\}\\mathbf\{\+5\.9\\%\}\}\}
Evaluation controls\.All methods use the same system prompt, tool interfaces, decoding settings, and answer\-extraction procedure\. Search calls return the top ten snippets and share an evaluation cache, preventing method\-specific retrieval variation from confounding policy comparisons\. Mathematical answers are checked with the same symbolic\-equivalence verifier\. Pass@3/5 uses five independent samples for every method and problem\. AIME 2024 and AIME 2025 each contain 30 problems, so one additional correct answer changes accuracy by 3\.3 percentage points\. Accordingly, the analysis emphasizes cross\-task averages, both model scales, and component ablations rather than small AIME differences in isolation\.

Training setup\.The experiments use Qwen3\-1\.7B and Qwen3\-4B backbones\([Yang and others, 2025](https://arxiv.org/html/2608.24300#bib.bib36)\)\. Training data are drawn from Tool\-Star\([Dong and others, 2025b](https://arxiv.org/html/2608.24300#bib.bib12)\)\. Mathematical experiments use 5,000 mathematical samples, whereas search experiments use search and dual\-tool trajectories from 10,000 open\-domain RL samples\. All reward\-based methods share the SFT initialization, training data, update count, outcome rewards, tool environment, and 16 rollout slots\. OPD checkpoints follow the same benchmark and decoding protocol during evaluation\. CBPO allocates six slots to initial trajectories and ten to a second branching stage\. Independent complete trajectories fill any unused branch slots\. Candidate scanning usesw=20w=20,K=10K=10,dmin=64d\_\{\\min\}=64, andJmax=3J\_\{\\max\}=3\. Allocation usesα=0\.2\\alpha=0\.2,γ=2\.0\\gamma=2\.0,κ=0\.25\\kappa=0\.25,ρpath=ρnode=0\.2\\rho\_\{\\mathrm\{path\}\}=\\rho\_\{\\mathrm\{node\}\}=0\.2,Bnode=3B\_\{\\mathrm\{node\}\}=3, andBpath=4B\_\{\\mathrm\{path\}\}=4\. Credit modulation usesη=0\.2\\eta=0\.2,ϕ=2\.0\\phi=2\.0, and a CBV standard\-deviation threshold of10−610^\{\-6\}\.

Baselines\.At both model scales, the comparison includes Base, TIR prompting\([Lin and Xu, 2025](https://arxiv.org/html/2608.24300#bib.bib33)\), SFT, and on\-policy distillation \(OPD\)\([Li and others, 2026](https://arxiv.org/html/2608.24300#bib.bib7)\)\. Policy\-optimization baselines are GRPO\([Shao and others, 2024](https://arxiv.org/html/2608.24300#bib.bib3)\), REINFORCE\+\+\([Hu and others, 2025](https://arxiv.org/html/2608.24300#bib.bib6)\), DAPO\([Yu and others, 2025](https://arxiv.org/html/2608.24300#bib.bib5)\), GSPO\([Zheng et al\., 2025](https://arxiv.org/html/2608.24300#bib.bib29)\), EAPO\([He et al\., 2026](https://arxiv.org/html/2608.24300#bib.bib31)\), and OC\-GRPO\([Agrawal et al\., 2026](https://arxiv.org/html/2608.24300#bib.bib37)\)\. Agentic and branch\-based baselines are GIGPO\([Feng et al\., 2025](https://arxiv.org/html/2608.24300#bib.bib16)\), Tree\-GRPO\([Ji et al\., 2026](https://arxiv.org/html/2608.24300#bib.bib17)\), and ARPO\([Dong and others, 2025a](https://arxiv.org/html/2608.24300#bib.bib18)\)\.

Figure 4\.Pass@K comparison between ARPO and CBPO\. The top and bottom rows show Qwen3\-1\.7B and Qwen3\-4B\. Pass@3/5 is computed from five samples per problem\. Each benchmark uses an independent vertical axis; consequently, bar heights should not be compared across benchmarks\.Grouped bar charts comparing ARPO and CBPO at Pass@1, Pass@3, and Pass@5 on five mathematical reasoning benchmarks and two model scales\.
### 5\.2\.Main Results

The primary comparison evaluates whether the complete CBPO pipeline improves closed\-form reasoning and open\-world retrieval under matched rollout budgets\. Table[5\.1](https://arxiv.org/html/2608.24300#S5.SS1)reports mathematical macro\-average Pass@1 scores of 66\.0% and 69\.3% with the 1\.7B and 4B backbones\. These scores exceed those of ARPO, the strongest size\-matched baseline by macro average, by 1\.7 and 2\.2 percentage points\. The corresponding search averages are 53\.2 and 57\.5, exceeding ARPO by 1\.8 points at both scales\. Relative to OPD, CBPO gains 3\.6 and 3\.9 points on the mathematical average and 8\.7 and 9\.0 points on the search average\. CBPO ranks first or ties for first on all five mathematical benchmarks at 1\.7B and on four of five at 4B\. At 4B, GRPO leads AIME 2025 by one problem\. CBPO also leads four of five search benchmarks at each scale, trailing EAPO on MuSiQue by 0\.5 points\. This consistency across domains and model scales supports the overall pipeline\. The following analyses examine the component\-specific hypotheses\.

Pass@K extension\.Figure[4](https://arxiv.org/html/2608.24300#acmlabel4)evaluates whether the gain persists when several candidate answers are available\. From five independent samples per problem, Qwen3\-1\.7B with CBPO reaches macro\-average Pass@1, Pass@3, and Pass@5 scores of 66\.0%, 74\.6%, and 77\.9%\. The corresponding Qwen3\-4B scores are 69\.3%, 76\.9%, and 80\.2%\. CBPO exceeds ARPO at each reported value, indicating that its advantage is not confined to single\-sample decoding\.

Training dynamics and branch selection\.Figure[5](https://arxiv.org/html/2608.24300#acmlabel5)compares reward, tool use, and selected branch positions\. CBPO attains a higher final mean reward than GIGPO and ARPO while averaging fewer tool calls\. The performance gain therefore cannot be attributed to more frequent tool invocation\. The qualitative examples show that decay can reorder high\-entropy candidates before allocation, consistent with separating candidate discovery from budget control\.

### 5\.3\.Full\-Response Candidate Analysis

Three Qwen3\-1\.7B trajectories from the MATH test set were inspected to locate decisions associated with their outcomes\. In one successful trajectory, the model enumerated all 16 one\- and two\-digit candidates before calling Python\. The interpreter then correctly identified the ten primes\. In a failed trajectory, the submitted code also executed correctly, but the preceding reasoning listed only six parenthesizations of an arithmetic expression\. Python evaluated this incomplete list and returned two values instead of the correct four\. The decisive error therefore occurred in the reasoning before tool invocation, not at the tool boundary\.

The third trajectory exhibited the converse pattern\. Python raised the same execution exception twice, yet the model recovered the correct answer from an algebraic derivation completed before either call\. A tool\-return position was therefore salient but not decisive for the final reward\. Together, these cases illustrate why candidate discovery should cover the full response rather than treat tool observations as universal branch boundaries\. These examples provide qualitative mechanism checks, not population\-level or causal evidence\. A controlled tool\-boundary\-only ablation remains necessary to quantify this design choice independently\.

![A single-column two-by-two panel grid with mean reward at upper left and mean tool calls at lower left. The upper-right qualitative word cloud sizes candidate tokens by entropy. The lower-right cloud shows retained candidates sized by post-decay branching priority, preserving an overall entropy relationship while introducing rank reversals from path- and node-level decay.](https://arxiv.org/html/2608.24300v1/cbpo_training_branch_grid.png)Figure 5\.Training dynamics and branch selection\. \(a\) Mean reward and \(b\) mean tool calls per trajectory; thick lines denote smoothed means, with light traces and shading showing variation\. In the qualitative word clouds, \(c\) token size increases monotonically with candidate entropy, whereas \(d\) token size increases monotonically with post\-decay branching priority among retained candidates\. Entropy therefore favors selection, but path\- and node\-level decay can change the ranking and retain some positions below the entropy\-only cutoff; sizes do not encode token frequency\.A single\-column two\-by\-two panel grid with mean reward at upper left and mean tool calls at lower left\. The upper\-right qualitative word cloud sizes candidate tokens by entropy\. The lower\-right cloud shows retained candidates sized by post\-decay branching priority, preserving an overall entropy relationship while introducing rank reversals from path\- and node\-level decay\.### 5\.4\.Ablation Study

The method motivates two quantitative predictions\. Path\-level and node\-level decay should reduce complementary forms of allocation concentration, whereas CBV should improve learning beyond branch exploration alone\. Table[2](https://arxiv.org/html/2608.24300#S5.T2)tests these predictions under matched training, rollout, and evaluation protocols\.

Removing either decay level reduces the mathematical macro average by 1\.5 points at both model scales\. Removing both reduces it by 2\.2 points, more than either individual ablation\. The two decay terms therefore regulate distinct sources of allocation concentration\. Removing CBV produces the largest losses, 3\.0 points for Qwen3\-1\.7B and 2\.9 points for Qwen3\-4B\. Branch exploration alone is therefore insufficient; branch outcomes must also inform local credit\. Most of this reduction occurs on AIME 2024 and AIME 2025, the two benchmarks with the lowest absolute accuracy\.

Table 2\.Component ablations for CBPO\. Values are Pass@1 \(%\) and the five\-task macro average\. Shading marks the full method, and bold indicates the best value in each column\.Table 3\.CBPO Pass@1 \(%\) under different rollout configurations\.M=N\+BM=N\+B\. Shading marks the default configuration, and bold indicates the best value in each column\.### 5\.5\.Branching Configuration

Balanced branching requires sufficient independent paths for global coverage and sufficient shared\-prefix continuations for local comparison\. This trade\-off is evaluated by varying the allocation between initial trajectoriesNNand dynamic branchesBB\. Total budgets areM=N\+B∈\{4,8,16\}M=N\+B\\in\\\{4,8,16\\\}, with all other settings held fixed\.

IncreasingMMfrom 4 to 16 improves the macro average at both model scales\. Within a fixed budget, too few initial trajectories restrict path coverage, whereas too few branches weaken exact\-prefix comparisons\. The highest macro average occurs atN=6,B=10N=6,B=10for both backbones\. This shared optimum favors branch sampling while retaining sufficient independent trajectories for global coverage\.

## 6\.Conclusion

CBPO addresses fine\-grained credit assignment by separating two operations that branch\-based RL often conflates\. Full\-response entropy scanning with path\-level and node\-level decay allocates a fixed rollout budget\. Reward variation within exact\-prefix groups then modulates the contribution of each sampled continuation\. Prefix masking and hierarchical segmentation prevent shared tokens from receiving duplicated credit\. With Qwen3\-1\.7B and Qwen3\-4B, CBPO attains mathematical macro averages of 66\.0% and 69\.3% and search averages of 53\.2 and 57\.5\. Cross\-scale ablations attribute distinct gains to balanced allocation and CBV\. However, the independent contribution of full\-response scanning still requires a controlled boundary\-only ablation\. Within the evaluated models, tasks, and budgets, the evidence supports a bounded design principle: uncertainty identifies alternatives, whereas observed outcome variation provides a stronger signal for local credit assignment\.

## Ethical Considerations

The study uses public benchmarks and collects no new personal or human\-subject data\. Open\-web retrieval may nevertheless expose a model to inaccurate, biased, offensive, or privacy\-sensitive material\. Benchmark accuracy does not establish the reliability or social neutrality of retrieved sources\. Generated Python code runs in a sandbox, and tool calls are capped to limit security and resource risks\. Like other branch\-based methods, CBPO samples several continuations and therefore incurs additional generation cost\. A fixed rollout budget bounds this cost, and all methods are compared using matched rollout slots\. The experiments evaluate benchmark accuracy rather than deployment safety\. High\-stakes applications would require domain\-specific testing, content filtering, access controls, and human oversight\.

## References

- Agrawalet al\.\(2026\)P\. Agrawal, A\. Samanta, S\. Ghasemlou, B\. Vidolov, J\. Bhandari, K\. Asadi, D\. Jiang, and A\. ModiOff\-Context GRPO: Learning to Reason on Hard Problems using Privileged Information\.Note:arXiv preprint arXiv:2607\.19313External Links:[Document](https://dx.doi.org/10.48550/arXiv.2607.19313),[Link](https://arxiv.org/abs/2607.19313)Cited by:[§5\.1](https://arxiv.org/html/2608.24300#S5.SS1.tab1.4.1.1)\.
- Chenget al\.\(2025\)D\. Cheng, S\. Huang, X\. Zhu, B\. Dai, W\. X\. Zhao, Z\. Zhang, and F\. WeiReasoning with Exploration: An Entropy Perspective\.Note:arXiv preprint arXiv:2506\.14758External Links:[Document](https://dx.doi.org/10.48550/arXiv.2506.14758),[Link](https://arxiv.org/abs/2506.14758)Cited by:[§1](https://arxiv.org/html/2608.24300#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.24300#S2.SS1.p3.1),[§3](https://arxiv.org/html/2608.24300#S3.p11.1)\.
- Cobbeet al\.\(2021\)K\. Cobbeet al\.Training Verifiers to Solve Math Word Problems\.Note:arXiv preprint arXiv:2110\.14168External Links:[Document](https://dx.doi.org/10.48550/arXiv.2110.14168),[Link](https://arxiv.org/abs/2110.14168)Cited by:[§5\.1](https://arxiv.org/html/2608.24300#S5.SS1.p1.1)\.
- Donget al\.\(2025\)G\. Dong, L\. Bao, Z\. Wang, K\. Zhao, X\. Li, J\. Jin, J\. Yang, H\. Mao, F\. Zhang, K\. Gai, G\. Zhou, Y\. Zhu, J\.\-R\. Wen, and Z\. DouAgentic Entropy\-Balanced Policy Optimization\.Note:arXiv preprint arXiv:2510\.14545External Links:[Document](https://dx.doi.org/10.48550/arXiv.2510.14545),[Link](https://arxiv.org/abs/2510.14545)Cited by:[§2\.1](https://arxiv.org/html/2608.24300#S2.SS1.p3.1),[§3](https://arxiv.org/html/2608.24300#S3.p3.1)\.
- Donget al\.\(2025a\)G\. Donget al\.Agentic Reinforced Policy Optimization\.Note:arXiv preprint arXiv:2507\.19849External Links:[Document](https://dx.doi.org/10.48550/arXiv.2507.19849),[Link](https://arxiv.org/abs/2507.19849)Cited by:[§1](https://arxiv.org/html/2608.24300#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.24300#S2.SS2.p4.1),[§3](https://arxiv.org/html/2608.24300#S3.p3.1),[§3](https://arxiv.org/html/2608.24300#S3.p5.1),[§5\.1](https://arxiv.org/html/2608.24300#S5.SS1.tab1.4.1.1)\.
- Donget al\.\(2025b\)G\. Donget al\.Tool\-Star: Empowering LLM\-Brained Multi\-Tool Reasoner via Reinforcement Learning\.Note:arXiv preprint arXiv:2505\.16410External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.16410),[Link](https://arxiv.org/abs/2505.16410)Cited by:[§1](https://arxiv.org/html/2608.24300#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.24300#S2.SS2.p2.1),[§5\.1](https://arxiv.org/html/2608.24300#S5.SS1.tab1.3.1.1)\.
- Fenget al\.\(2026\)J\. Fenget al\.ReTool: Reinforcement Learning for Strategic Tool Use in LLMs\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.24300#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.24300#S2.SS2.p2.1),[§4\.3](https://arxiv.org/html/2608.24300#S4.SS3.p2.1)\.
- Fenget al\.\(2025\)L\. Feng, Z\. Xue, T\. Liu, and B\. AnGroup\-in\-Group Policy Optimization for LLM Agent Training\.Note:arXiv preprint arXiv:2505\.10978External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.10978),[Link](https://arxiv.org/abs/2505.10978)Cited by:[§1](https://arxiv.org/html/2608.24300#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.24300#S2.SS2.p4.1),[§3](https://arxiv.org/html/2608.24300#S3.p5.1),[§5\.1](https://arxiv.org/html/2608.24300#S5.SS1.tab1.4.1.1)\.
- Gouet al\.\(2024\)Z\. Gouet al\.ToRA: A Tool\-Integrated Reasoning Agent for Mathematical Problem Solving\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.24300#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.24300#S2.SS2.p2.1)\.
- Guoet al\.\(2025\)D\. Guoet al\.DeepSeek\-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning\.Note:arXiv preprint arXiv:2501\.12948External Links:[Document](https://dx.doi.org/10.48550/arXiv.2501.12948),[Link](https://arxiv.org/abs/2501.12948)Cited by:[§2\.1](https://arxiv.org/html/2608.24300#S2.SS1.p1.1)\.
- Heet al\.\(2026\)Y\. He, H\. Wu, S\. Liu, H\. Ge, H\. Zhou, K\. Wu, Z\. Zheng, Q\. Lin, Z\. Zhong, and Y\. ZhangRethinking Token\-Level Credit Assignment in RLVR: A Polarity\-Entropy Analysis\.Note:arXiv preprint arXiv:2604\.11056External Links:[Document](https://dx.doi.org/10.48550/arXiv.2604.11056),[Link](https://arxiv.org/abs/2604.11056)Cited by:[§2\.1](https://arxiv.org/html/2608.24300#S2.SS1.p2.1),[§2\.1](https://arxiv.org/html/2608.24300#S2.SS1.p3.1),[§4\.3](https://arxiv.org/html/2608.24300#S4.SS3.p12.1),[§5\.1](https://arxiv.org/html/2608.24300#S5.SS1.tab1.4.1.1)\.
- Hendryckset al\.\(2021\)D\. Hendryckset al\.Measuring Mathematical Problem Solving with the MATH Dataset\.Note:arXiv preprint arXiv:2103\.03874External Links:[Document](https://dx.doi.org/10.48550/arXiv.2103.03874),[Link](https://arxiv.org/abs/2103.03874)Cited by:[§5\.1](https://arxiv.org/html/2608.24300#S5.SS1.p1.1)\.
- Hoet al\.\(2020\)X\. Ho, A\. Duong Nguyen, S\. Sugawara, and A\. AizawaConstructing A Multi\-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps\.InProceedings of the 28th International Conference on Computational Linguistics,Barcelona, Spain \(Online\),pp\. 6609–6625\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.coling-main.580),[Link](https://aclanthology.org/2020.coling-main.580/)Cited by:[§5\.1](https://arxiv.org/html/2608.24300#S5.SS1.p1.1)\.
- Houet al\.\(2025\)Z\. Hou, Z\. Hu, Y\. Li, R\. Lu, J\. Tang, and Y\. DongTreeRL: LLM Reinforcement Learning with On\-Policy Tree Search\.InProceedings of the Annual Meeting of the Association for Computational Linguistics,Cited by:[§1](https://arxiv.org/html/2608.24300#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.24300#S2.SS1.p3.1)\.
- Huet al\.\(2025\)J\. Huet al\.REINFORCE\+\+: Stabilizing Critic\-Free Policy Optimization with Global Advantage Normalization\.Note:arXiv preprint arXiv:2501\.03262External Links:[Document](https://dx.doi.org/10.48550/arXiv.2501.03262),[Link](https://arxiv.org/abs/2501.03262)Cited by:[§2\.1](https://arxiv.org/html/2608.24300#S2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2608.24300#S5.SS1.tab1.4.1.1)\.
- Jiet al\.\(2026\)Y\. Ji, Z\. Ma, Y\. Wang, G\. Chen, X\. Chu, and L\. WuTree Search for LLM Agent Reinforcement Learning\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.24300#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.24300#S2.SS1.p3.1),[§3](https://arxiv.org/html/2608.24300#S3.p5.1),[§5\.1](https://arxiv.org/html/2608.24300#S5.SS1.tab1.4.1.1)\.
- Jianget al\.\(2026\)C\. Jiang, M\. Cheng, X\. Tao, Q\. Mao, J\. Ouyang, and Q\. LiuTableMind: An Autonomous Programmatic Agent for Tool\-Augmented Table Reasoning\.InProceedings of the Nineteenth ACM International Conference on Web Search and Data Mining,Boise, ID, USA,pp\. 260–270\.External Links:[Document](https://dx.doi.org/10.1145/3773966.3777932),[Link](https://doi.org/10.1145/3773966.3777932)Cited by:[§2\.2](https://arxiv.org/html/2608.24300#S2.SS2.p2.1)\.
- Jinet al\.\(2025\)B\. Jin, H\. Zeng, Z\. Yue, J\. Yoon, S\. O\. Arik, D\. Wang, H\. Zamani, and J\. HanSearch\-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning\.InConference on Language Modeling,Cited by:[§2\.2](https://arxiv.org/html/2608.24300#S2.SS2.p2.1)\.
- Laiet al\.\(2024\)X\. Laiet al\.Step\-DPO: Step\-Wise Preference Optimization for Long\-Chain Reasoning of LLMs\.Note:arXiv preprint arXiv:2406\.18629External Links:[Document](https://dx.doi.org/10.48550/arXiv.2406.18629),[Link](https://arxiv.org/abs/2406.18629)Cited by:[§1](https://arxiv.org/html/2608.24300#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.24300#S2.SS1.p2.1)\.
- Liet al\.\(2025\)X\. Li, H\. Zou, and P\. LiuToRL: Scaling Tool\-Integrated Reinforcement Learning\.Note:arXiv preprint arXiv:2503\.23383External Links:[Document](https://dx.doi.org/10.48550/arXiv.2503.23383),[Link](https://arxiv.org/abs/2503.23383)Cited by:[§1](https://arxiv.org/html/2608.24300#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.24300#S2.SS2.p2.1),[§4\.3](https://arxiv.org/html/2608.24300#S4.SS3.p2.1)\.
- Liet al\.\(2026\)Y\. Liet al\.Rethinking On\-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe\.Note:arXiv preprint arXiv:2604\.13016External Links:[Document](https://dx.doi.org/10.48550/arXiv.2604.13016),[Link](https://arxiv.org/abs/2604.13016)Cited by:[§5\.1](https://arxiv.org/html/2608.24300#S5.SS1.tab1.4.1.1)\.
- Lightmanet al\.\(2024\)H\. Lightmanet al\.Let’s Verify Step by Step\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.24300#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.24300#S2.SS1.p2.1)\.
- Lin and Xu \(2025\)H\. Lin and Z\. XuUnderstanding Tool\-Integrated Reasoning\.Note:arXiv preprint arXiv:2508\.19201External Links:[Document](https://dx.doi.org/10.48550/arXiv.2508.19201),[Link](https://arxiv.org/abs/2508.19201)Cited by:[§2\.2](https://arxiv.org/html/2608.24300#S2.SS2.p1.1),[§5\.1](https://arxiv.org/html/2608.24300#S5.SS1.tab1.4.1.1)\.
- Linet al\.\(2024\)Z\. Lin, T\. Liang, J\. Xu, X\. Wang, R\. Luo, C\. Shi, S\. Li, Y\. Yang, and Z\. TuCritical Tokens Matter: Token\-Level Contrastive Estimation Enhances LLM’s Reasoning Capability\.Note:arXiv preprint arXiv:2411\.19943External Links:[Document](https://dx.doi.org/10.48550/arXiv.2411.19943),[Link](https://arxiv.org/abs/2411.19943)Cited by:[§1](https://arxiv.org/html/2608.24300#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.24300#S2.SS1.p3.1),[§3](https://arxiv.org/html/2608.24300#S3.p11.1)\.
- Presset al\.\(2023\)O\. Press, M\. Zhang, S\. Min, L\. Schmidt, N\. A\. Smith, and M\. LewisMeasuring and Narrowing the Compositionality Gap in Language Models\.InFindings of the Association for Computational Linguistics: EMNLP 2023,Singapore,pp\. 5687–5711\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.378),[Link](https://aclanthology.org/2023.findings-emnlp.378/)Cited by:[§5\.1](https://arxiv.org/html/2608.24300#S5.SS1.p1.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. FinnDirect Preference Optimization: Your Language Model Is Secretly a Reward Model\.InAdvances in Neural Information Processing Systems,Cited by:[§2\.1](https://arxiv.org/html/2608.24300#S2.SS1.p2.1)\.
- Ratheeet al\.\(2025\)M\. Rathee, S\. MacAvaney, and A\. AnandQuam: Adaptive Retrieval through Query Affinity Modelling\.InProceedings of the Eighteenth ACM International Conference on Web Search and Data Mining,Hannover, Germany,pp\. 954–962\.External Links:[Document](https://dx.doi.org/10.1145/3701551.3703584),[Link](https://doi.org/10.1145/3701551.3703584)Cited by:[§2\.2](https://arxiv.org/html/2608.24300#S2.SS2.p3.1)\.
- Schulmanet al\.\(2017\)J\. Schulmanet al\.Proximal Policy Optimization Algorithms\.Note:arXiv preprint arXiv:1707\.06347External Links:[Document](https://dx.doi.org/10.48550/arXiv.1707.06347),[Link](https://arxiv.org/abs/1707.06347)Cited by:[§1](https://arxiv.org/html/2608.24300#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.24300#S2.SS1.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shaoet al\.DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\.Note:arXiv preprint arXiv:2402\.03300External Links:[Document](https://dx.doi.org/10.48550/arXiv.2402.03300),[Link](https://arxiv.org/abs/2402.03300)Cited by:[§1](https://arxiv.org/html/2608.24300#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.24300#S2.SS1.p1.1),[§3](https://arxiv.org/html/2608.24300#S3.p6.1),[§5\.1](https://arxiv.org/html/2608.24300#S5.SS1.tab1.4.1.1)\.
- Sunet al\.\(2025\)H\. Sun, Z\. Qiao, J\. Guo, X\. Fan, Y\. Hou, Y\. Jiang, P\. Xie, Y\. Zhang, F\. Huang, and J\. ZhouZeroSearch: Incentivize the Search Capability of LLMs without Searching\.Note:arXiv preprint arXiv:2505\.04588External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.04588),[Link](https://arxiv.org/abs/2505.04588)Cited by:[§2\.2](https://arxiv.org/html/2608.24300#S2.SS2.p2.1)\.
- Trivediet al\.\(2022\)H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. SabharwalMuSiQue: Multihop Questions via Single\-hop Question Composition\.Transactions of the Association for Computational Linguistics10,pp\. 539–554\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00475),[Link](https://aclanthology.org/2022.tacl-1.31/)Cited by:[§5\.1](https://arxiv.org/html/2608.24300#S5.SS1.p1.1)\.
- Wanget al\.\(2025\)S\. Wanget al\.Beyond the 80/20 Rule: High\-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning\.Note:arXiv preprint arXiv:2506\.01939External Links:[Document](https://dx.doi.org/10.48550/arXiv.2506.01939),[Link](https://arxiv.org/abs/2506.01939)Cited by:[§1](https://arxiv.org/html/2608.24300#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.24300#S2.SS1.p3.1),[§3](https://arxiv.org/html/2608.24300#S3.p11.1)\.
- Weiet al\.\(2022\)J\. Weiet al\.Chain\-of\-Thought Prompting Elicits Reasoning in Large Language Models\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2608.24300#S1.p1.1)\.
- Wuet al\.\(2025\)J\. Wu, W\. Yin, Y\. Jiang, Z\. Wang, Z\. Xi, R\. Fang, L\. Zhang, Y\. He, D\. Zhou, P\. Xie, and F\. HuangWebWalker: Benchmarking LLMs in Web Traversal\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 10290–10305\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.508),[Link](https://aclanthology.org/2025.acl-long.508/)Cited by:[§5\.1](https://arxiv.org/html/2608.24300#S5.SS1.p1.1)\.
- Yanget al\.\(2025\)A\. Yanget al\.Qwen3 Technical Report\.Note:arXiv preprint arXiv:2505\.09388External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.09388),[Link](https://arxiv.org/abs/2505.09388)Cited by:[§5\.1](https://arxiv.org/html/2608.24300#S5.SS1.tab1.3.1.1)\.
- Yanget al\.\(2018\)Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. W\. Cohen, R\. Salakhutdinov, and C\. D\. ManningHotpotQA: A Dataset for Diverse, Explainable Multi\-hop Question Answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,Brussels, Belgium,pp\. 2369–2380\.External Links:[Document](https://dx.doi.org/10.18653/v1/D18-1259),[Link](https://aclanthology.org/D18-1259/)Cited by:[§5\.1](https://arxiv.org/html/2608.24300#S5.SS1.p1.1)\.
- Yaoet al\.\(2023\)S\. Yaoet al\.ReAct: Synergizing Reasoning and Acting in Language Models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.24300#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.24300#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2608.24300#S2.SS2.p2.1)\.
- Yuet al\.\(2025\)Q\. Yuet al\.DAPO: An Open\-Source LLM Reinforcement Learning System at Scale\.Note:arXiv preprint arXiv:2503\.14476External Links:[Document](https://dx.doi.org/10.48550/arXiv.2503.14476),[Link](https://arxiv.org/abs/2503.14476)Cited by:[§2\.1](https://arxiv.org/html/2608.24300#S2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2608.24300#S5.SS1.tab1.4.1.1)\.
- Zhanget al\.\(2026a\)J\. Zhang, D\. Bi, T\. Sun, M\. Yang, J\. Wang, and Y\. WangTOOL\-CURE: Tool Selection via Curriculum\-Enhanced Reinforcement Learning with Sample Screening for LLMs\.InProceedings of the Nineteenth ACM International Conference on Web Search and Data Mining,Boise, ID, USA,pp\. 946–954\.External Links:[Document](https://dx.doi.org/10.1145/3773966.3777952),[Link](https://doi.org/10.1145/3773966.3777952)Cited by:[§2\.2](https://arxiv.org/html/2608.24300#S2.SS2.p2.1)\.
- Zhanget al\.\(2026b\)Y\. Zhang, S\. Zhou, X\. Li, Z\. Tian, Y\. Gao, S\. Zhang, W\. Hou, Y\. Liu, and B\. ZhouKnowFC: Navigating Knowledge Conflicts in Large Language Model\-based Fact\-Checking\.InProceedings of the Nineteenth ACM International Conference on Web Search and Data Mining,Boise, ID, USA,pp\. 996–1006\.External Links:[Document](https://dx.doi.org/10.1145/3773966.3777935),[Link](https://doi.org/10.1145/3773966.3777935)Cited by:[§2\.2](https://arxiv.org/html/2608.24300#S2.SS2.p3.1)\.
- Zhenget al\.\(2025\)C\. Zheng, S\. Liu, M\. Li, X\.\-H\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang, J\. Zhou, and J\. LinGroup Sequence Policy Optimization\.Note:arXiv preprint arXiv:2507\.18071External Links:[Document](https://dx.doi.org/10.48550/arXiv.2507.18071),[Link](https://arxiv.org/abs/2507.18071)Cited by:[§2\.1](https://arxiv.org/html/2608.24300#S2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2608.24300#S5.SS1.tab1.4.1.1)\.