LAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoning

# Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoning
Source: [https://arxiv.org/html/2607.13501](https://arxiv.org/html/2607.13501)
###### Abstract

Reinforcement learning for multi\-turn search reasoning typically relies on terminal outcome rewards, which cannot distinguish useful, redundant, and harmful intermediate interactions\. We proposeLAPO, a self\-generated process\-supervision method based on backward leave\-one\-turn attribution\. For each search turn, LAPO replaces the turn and its retrieval observation with a fixed\[DELETE\]placeholder and measures the resulting change in the current policy’s mean log\-likelihood of the gold answer\. This*Answer\-Likelihood Gain*estimates the turn’s contribution while preserving all downstream interactions, allowing early evidence to be evaluated in the complete reasoning context\. LAPO further applies sign\-consistency gating, retaining only normalized process advantages whose directions agree with their raw attribution scores\. The method requires no additional reward model, teacher, verifier, or LLM\-as\-a\-Judge\. Across seven knowledge\-intensive question\-answering datasets with local retrieval, LAPO achieves an average exact\-match score of0\.3260\.326, outperforming the strongest step\-reward baseline, IGPO, by0\.0530\.053\. Ablations show complementary benefits from backward attribution and sign\-consistency gating, demonstrating that policy\-derived retrospective attribution can provide effective process supervision for multi\-turn search agents\. Code is available at[https://github\.com/zhuq\-111/LAPO\-Leave\-One\-Turn\-Attribution](https://github.com/zhuq-111/LAPO-Leave-One-Turn-Attribution)\.

## 1Introduction

Large language models increasingly need to interact with external information sources to solve knowledge\-intensive problems rather than relying solely on parametric knowledge\. In multi\-turn search reasoning, a model must iteratively formulate queries, retrieve information, interpret observations, and integrate evidence before producing an answer\. Reinforcement learning \(RL\) provides a natural training framework for this setting: it can optimize complete interaction trajectories according to task\-level objectives without requiring human demonstrations for every intermediate actionJinet al\.\([2025](https://arxiv.org/html/2607.13501#bib.bib1)\)\. However, the long\-horizon nature of search reasoning creates a key credit\-assignment problem: the final answer usually depends on multiple interrelated search and reasoning decisions, while both successful and failed trajectories may contain useful, redundant, and harmful intermediate interactions\.

When training relies only on final\-answer correctness, all actions within a trajectory are typically updated according to the same outcome\. Consequently, a successful trajectory may incorrectly reinforce ineffective searches, while a failed trajectory may penalize a turn that retrieved crucial evidence but whose evidence was not used correctly in subsequent reasoning\. A terminal reward indicates whether the entire trajectory succeeded, but it does not reveal the specific attribution of each intermediate interaction to the final outcome\. Multi\-turn search reasoning therefore calls for finer\-grained process\-level credit signals\.

Existing methods commonly obtain process supervision from reward models, teacher models, verifiers, or other step\-level evaluatorsLightmanet al\.\([2023](https://arxiv.org/html/2607.13501#bib.bib7)\); Wanget al\.\([2024](https://arxiv.org/html/2607.13501#bib.bib8)\); Fanget al\.\([2026](https://arxiv.org/html/2607.13501#bib.bib11)\)Uesatoet al\.\([2022](https://arxiv.org/html/2607.13501#bib.bib9)\); Zhanget al\.\([2024](https://arxiv.org/html/2607.13501#bib.bib10)\)\. Although these methods provide denser feedback, they introduce additional training and inference costs and make policy optimization depend on the quality of an external evaluator\. Moreover, an evaluator trained on another policy or data distribution may not accurately assess the intermediate behaviors produced by the current policy\. This raises a natural question: can process\-level supervision be constructed directly from signals produced by the current policy?

To this end, we proposeLAPO, a method for self\-generated process rewards based on backward leave\-one\-turn attribution\. Given a complete trajectory, LAPO replaces each eligible search turn and its retrieval observation in turn with\[DELETE\], and measures the resulting change in the current policy’s mean log\-likelihood of the gold answer\. We call this change the*Answer\-Likelihood Gain*and use it to estimate the attribution of each search turn within the complete reasoning process\. Unlike local confidence changes between adjacent states, LAPO preserves all remaining downstream interactions when evaluating a target turn, thereby identifying early evidence whose value becomes apparent only through subsequent reasoning\.

Raw answer\-likelihood gains, however, can be noisy and can vary in scale across sampled trajectories\. LAPO applies robust scaling and group normalization, then introduces sign\-consistency gating to retain only process signals whose normalized advantages agree in direction with their raw answer\-likelihood gains\. The gated process advantage is assigned to the policy\-generated tokens in the corresponding search turn and combined with the terminal outcome advantage used by GRPOShaoet al\.\([2024](https://arxiv.org/html/2607.13501#bib.bib17)\)\. The final\-answer turn remains supervised only by the terminal reward\. The entire procedure derives supervision directly from the current policy and requires no separately trained process reward model, teacher model, verifier, or LLM\-based judge\.

Our main contributions are as follows:

- •We identify a limitation of outcome rewards for within\-trajectory credit assignment in multi\-turn search reasoning: final\-answer correctness does not reflect the specific attribution of individual intermediate search interactions\.
- •We propose LAPO, a self\-generated process\-reward method based on backward leave\-one\-turn attribution\. LAPO estimates turn\-level attribution from changes in the current policy’s likelihood of the gold answer without relying on an additional process evaluator\.
- •We introduce group\-relative sign\-consistency gating to filter directionally unreliable attribution signals, and evaluate LAPO on seven knowledge\-intensive question\-answering datasets\.

## 2Related Work

### 2\.1Reinforcement Learning for Search\-Augmented Reasoning

Recent work uses reinforcement learning to train language models to actively acquire external information during reasoning\. Unlike conventional methods that treat retrieval as a fixed preprocessing step, these approaches allow a model to decide when to search, how to formulate a query, and how to integrate the retrieved evidence\. Search\-R1 shows that outcome\-reward\-based reinforcement learning can induce multi\-turn search behavior without supervised reasoning trajectoriesJinet al\.\([2025](https://arxiv.org/html/2607.13501#bib.bib1)\)\. R1\-Searcher and ReSearch likewise use task\-level outcome rewards to train search and reasoning capabilitiesSonget al\.\([2025](https://arxiv.org/html/2607.13501#bib.bib2)\); Chenet al\.\([2025](https://arxiv.org/html/2607.13501#bib.bib3)\)\. This paradigm has also been extended to web navigation, multimodal search, and retrieval\-augmented reasoning, as in WebAgent\-R1, MMSearch\-R1, and RAG\-R1Weiet al\.\([2025b](https://arxiv.org/html/2607.13501#bib.bib4)\); Wuet al\.\([2026](https://arxiv.org/html/2607.13501#bib.bib5)\); Tanet al\.\([2026](https://arxiv.org/html/2607.13501#bib.bib6)\)\. In addition to answer correctness, these methods may incorporate auxiliary signals such as format rewards, tool\-use constraints, search costs, or efficiency objectives\.

These studies indicate that trajectory\-level reinforcement learning can induce complex search behavior\. However, trajectory\-level rewards primarily reflect whether the complete interaction succeeds and provide limited information about which intermediate queries, retrieval observations, or reasoning turns contribute to the final outcome\. LAPO retains terminal task supervision while extracting turn\-level attribution signals from complete trajectories, providing finer\-grained process supervision for this setting\.

### 2\.2External Process Supervision and On\-Policy Distillation

One line of work introduces denser supervision through external process evaluators\. Let’s Verify Step by Step trains a process reward model using human step\-level annotationsLightmanet al\.\([2023](https://arxiv.org/html/2607.13501#bib.bib7)\)\. Math\-Shepherd reduces dependence on human annotation through continuation\-based automatic verificationWanget al\.\([2024](https://arxiv.org/html/2607.13501#bib.bib8)\)\. ReST\-MCTS\* combines process\-reward estimation with tree search and trains on the probability that an intermediate state eventually leads to a correct answer; CriticSearch instead uses a frozen retrospective critic to generate turn\-level rewards from the complete trajectory and gold answerZhanget al\.\([2024](https://arxiv.org/html/2607.13501#bib.bib10),[2026](https://arxiv.org/html/2607.13501#bib.bib32)\)\. These methods evaluate intermediate reasoning using human annotation, automatic verification, search\-derived value estimates, or external critics\.

Recent work further uses strong language models or structured semantic information to supervise trajectories generated by the current policy\. Rubric\-based On\-policy Distillation \(ROPD\) induces prompt\-specific rubrics from teacher–student discrepancies and uses them to evaluate on\-policy student responsesFanget al\.\([2026](https://arxiv.org/html/2607.13501#bib.bib11)\)\. OPID extracts turn\-level and step\-level skills from complete trajectories and constructs skill\-conditioned distillation signals for agent reinforcement learningYanget al\.\([2026](https://arxiv.org/html/2607.13501#bib.bib12)\)\.

Externally guided process supervision can provide semantically rich feedback and distinguish among different reasoning or planning errors\. However, such methods generally require additional teacher calls, reward models, rubric induction, skill extraction, or verification procedures, and their reliability depends on the capability and calibration of the external evaluator\. In contrast, LAPO neither generates semantic critiques or rubrics nor extracts reusable skills\. Instead, it estimates the conditional attribution of each search turn using the current policy’s likelihood of the gold answer\. LAPO therefore requires no separately learned process evaluator; its process signal depends only on terminal task supervision and the current policy’s own probabilistic estimates\.

### 2\.3Policy\-Derived Process Rewards and Turn\-Level Credit Assignment

The work most closely related to LAPO constructs intermediate rewards from changes in the policy’s own predictions or interaction states\. IGPO and TIPS both generate turn\-level rewards from increments in gold\-answer likelihood between adjacent states; TIPS uses a frozen, periodically updated copy of the policy to compute an information potential functionWanget al\.\([2026](https://arxiv.org/html/2607.13501#bib.bib13)\); Xieet al\.\([2026](https://arxiv.org/html/2607.13501#bib.bib14)\)\. StepSearch combines information gain, redundancy control, and token\-level process supervision, while IG\-Search estimates the contribution of retrieval evidence by comparing answer confidence under the retrieved documents and randomly substituted documentsZhenget al\.\([2025](https://arxiv.org/html/2607.13501#bib.bib15)\); Lianget al\.\([2026](https://arxiv.org/html/2607.13501#bib.bib16)\)\. Other work measures reasoning progress through semantic information gain, reward propagation, or turn\-level value estimationHuet al\.\([2026](https://arxiv.org/html/2607.13501#bib.bib31)\); Fenget al\.\([2026](https://arxiv.org/html/2607.13501#bib.bib33)\); Weiet al\.\([2025a](https://arxiv.org/html/2607.13501#bib.bib34)\)\.

Overall, among policy\-derived approaches that do not rely on external critics, prior work mainly estimates reasoning progress through changes between adjacent states or local counterfactual comparisons\. LAPO instead emphasizes retrospective leave\-one\-turn attribution over the complete trajectory\. Forward methods typically compare adjacent states before and after adding the current interaction:

s​\(y⋆∣c≤t\)−s​\(y⋆∣c<t\),s\(y^\{\\star\}\\mid c\_\{\\leq t\}\)\-s\(y^\{\\star\}\\mid c\_\{<t\}\),\(1\)which measures the immediate effect of the current interaction on the model’s likelihood of generating the gold answer\. In multi\-hop search, however, evidence obtained early may become useful only when combined with later observations, so a local state change may not accurately capture its attribution\.

LAPO therefore performs backward leave\-one\-turn attribution over the complete trajectory:

s​\(y⋆∣cfull\)−s​\(y⋆∣c∖t\),s\(y^\{\\star\}\\mid c\_\{\\mathrm\{full\}\}\)\-s\(y^\{\\star\}\\mid c\_\{\\setminus t\}\),\(2\)where the target search turn and its retrieval observation are jointly replaced with\[DELETE\], while all other downstream interactions remain unchanged\. This design evaluates the attribution of an early search turn in the full reasoning context\. Unlike IG\-Search, which performs document\-level attribution by substituting retrieved documents, LAPO uses an entire search interaction as its attribution unit and thereby provides turn\-level process supervision for multi\-turn search reasoning\.

## 3Method

![Refer to caption](https://arxiv.org/html/2607.13501v1/x1.png)Figure 1:Overview of LAPO\.For each complete search trajectory, LAPO replaces each eligible search turn and its retrieval observation in turn with\[DELETE\], and estimates turn\-level attribution using the difference in the current policy’s mean log\-likelihood of the gold answer before and after replacement\. The resulting attribution scores undergo robust scaling, a tanh bounded transformation, group normalization, and sign\-consistency gating\. They are then assigned to the policy\-generated tokens of the corresponding search turns and used jointly with the trajectory\-level outcome advantage for policy optimization\.### 3\.1Problem Formulation and Method Overview

We consider a multi\-turn search agent trained with reinforcement learning\. Given a questionxix\_\{i\}, the policyπθ\\pi\_\{\\theta\}alternates between generating search actions and receiving observations from the retrieval environment, and eventually generates an answer\. A complete trajectory is represented as

τi=\(xi,ui,1,oi,1,…,ui,Ti,oi,Ti,vi\),\\tau\_\{i\}=\\left\(x\_\{i\},u\_\{i,1\},o\_\{i,1\},\\ldots,u\_\{i,T\_\{i\}\},o\_\{i,T\_\{i\}\},v\_\{i\}\\right\),\(3\)whereui,tu\_\{i,t\}denotes the policy\-generated<think\>\.\.\.<search\>\.\.\.</search\>content at search turntt,oi,to\_\{i,t\}denotes the corresponding<information\>\.\.\.</information\>retrieval observation returned by the environment,TiT\_\{i\}is the number of search turns, andviv\_\{i\}denotes the policy\-generated final\-answer turn<think\>\.\.\.<answer\>\.\.\.</answer\>\. We usea^i\\hat\{a\}\_\{i\}to denote the final answer extracted fromviv\_\{i\}\. We treat a search action and its corresponding retrieval observation as one complete search interaction:

ei,t=\(ui,t,oi,t\)\.e\_\{i,t\}=\(u\_\{i,t\},o\_\{i,t\}\)\.\(4\)A*valid search turn*is a search interaction that contains both a complete policy\-generated search action and its corresponding environment observation; the final\-answer turn is not a valid search turn\.

Letai⋆a\_\{i\}^\{\\star\}denote the gold answer to questionxix\_\{i\}\. A conventional outcome reward provides a uniform training signal for the entire trajectory based on final\-answer quality\. It cannot distinguish the specific attribution of different search turns and thus offers limited fine\-grained credit assignment for crucial, redundant, or misleading search behavior\.

LAPO addresses this limitation through backward leave\-one\-turn attribution\. It measures the change in the current policy’s conditional likelihood of the gold answer when a single search turn is replaced in its entirety with\[DELETE\], and uses the change to construct a turn\-level process advantage\. After robust scaling, a tanh bounded transformation, group normalization, and sign\-consistency gating, this process advantage is assigned to the policy\-generated tokens of the corresponding search turn and combined with the original trajectory\-level outcome advantage in GRPO optimization\.

### 3\.2Backward Leave\-One\-Turn Attribution

#### Gold\-answer likelihood\.

Letccdenote the context before the policy generates its final answer\. To measure the current policy’s support for the gold answer, we append a fixed response template containing the gold answer toccand score only the gold\-answer tokens in the template\.

Letppdenote the fixed token sequence in the scoring template that precedes the gold\-answer tokens\. Writeai⋆=\(ai,1⋆,…,ai,Li⋆\)a\_\{i\}^\{\\star\}=\(a\_\{i,1\}^\{\\star\},\\ldots,a\_\{i,L\_\{i\}\}^\{\\star\}\), whereLiL\_\{i\}is the number of tokens in the gold answer\. The current policy’s score for the gold answer is

Sθ​\(ai⋆∣c\)=1Li​∑k=1Lilog⁡πθ​\(ai,k⋆∣c,p,ai,<k⋆\)\.S\_\{\\theta\}\(a\_\{i\}^\{\\star\}\\mid c\)=\\frac\{1\}\{L\_\{i\}\}\\sum\_\{k=1\}^\{L\_\{i\}\}\\log\\pi\_\{\\theta\}\\left\(a\_\{i,k\}^\{\\star\}\\mid c,p,a\_\{i,<k\}^\{\\star\}\\right\)\.\(5\)The mean token log\-likelihood reduces the effect of answer length on the score scale, making attribution scores more comparable across questions\.

Letcifullc\_\{i\}^\{\\mathrm\{full\}\}denote the pre\-answer context that contains all search interactions but excludes the policy\-generated final\-answer turnviv\_\{i\}\. Excludingviv\_\{i\}prevents the policy’s original final reasoning and answer from leaking into gold\-answer scoring, so the score reflects only the support supplied by the preceding search interactions\.

#### Leave\-one\-turn counterfactual attribution\.

For each valid search turntt, LAPO applies a leave\-one\-turn operation to the corresponding search action and retrieval observationei,t=\(ui,t,oi,t\)e\_\{i,t\}=\(u\_\{i,t\},o\_\{i,t\}\)in the full contextcifullc\_\{i\}^\{\\mathrm\{full\}\}\. Specifically, we replace the entire target turn with a fixed\[DELETE\]placeholder to explicitly mark the absence of that turn’s content in the counterfactual context\. This produces the counterfactual contextci,−tc\_\{i,\-t\}, in which every other search interaction and its order remain unchanged\.

The answer\-likelihood gain of search turnttis defined as

Δi,t=Sθ​\(ai⋆∣cifull\)−Sθ​\(ai⋆∣ci,−t\)\.\\Delta\_\{i,t\}=S\_\{\\theta\}\(a\_\{i\}^\{\\star\}\\mid c\_\{i\}^\{\\mathrm\{full\}\}\)\-S\_\{\\theta\}\(a\_\{i\}^\{\\star\}\\mid c\_\{i,\-t\}\)\.\(6\)
WhenΔi,t\>0\\Delta\_\{i,t\}\>0, replacing the target turn with the placeholder reduces the gold\-answer likelihood, indicating positive attribution to that turn under the current context and policy\. WhenΔi,t<0\\Delta\_\{i,t\}<0, the replacement instead increases the gold\-answer likelihood, suggesting that the turn may have a negative influence under the current context and policy\. A value ofΔi,t\\Delta\_\{i,t\}near zero indicates that the turn has little effect on the gold\-answer likelihood\.

Unlike forward attribution methods that measure an immediate change by comparing adjacent prefix states, LAPO preserves all downstream search interactions when evaluating the target turn\. This backward leave\-one\-turn design enables retrospective attribution in the full reasoning context even when evidence from an early turn becomes useful only through later search and reasoning\.

In our implementation, we flatten the full contexts and all leave\-one\-turn counterfactual contexts in a rollout batch into a single scoring batch, and append the same gold\-answer scoring template to every context\. A batched forward pass produces the gold\-answer log\-likelihoods under the full and counterfactual contexts simultaneously, reducing separate model calls and scheduling overhead\.

### 3\.3Gated Process\-Advantage Construction and Assignment

#### Trajectory\-level outcome advantage\.

For each question, we sampleGGtrajectories from the current policy\. The terminal task reward for trajectoryiiis

Ri=F1⁡\(a^i,ai⋆\)\.R\_\{i\}=\\operatorname\{F1\}\(\\hat\{a\}\_\{i\},a\_\{i\}^\{\\star\}\)\.\(7\)For a trajectory groupggcorresponding to the same question, the outcome advantage is obtained by within\-group standardization:

Aiout=Ri−μR\(g\)σR\(g\)\+ϵR,A\_\{i\}^\{\\mathrm\{out\}\}=\\frac\{R\_\{i\}\-\\mu\_\{R\}^\{\(g\)\}\}\{\\sigma\_\{R\}^\{\(g\)\}\+\\epsilon\_\{R\}\},\(8\)whereμR\(g\)\\mu\_\{R\}^\{\(g\)\}andσR\(g\)\\sigma\_\{R\}^\{\(g\)\}are the mean and standard deviation of rewards within the group, respectively, andϵR\\epsilon\_\{R\}is included for numerical stability\. This outcome advantage provides terminal task supervision to every policy\-generated token in the trajectory and remains the base advantage after LAPO introduces turn\-level credit signals\.

#### Robust scaling and group normalization\.

Answer\-likelihood gains may differ in scale across questions and trajectories, and a small number of extreme values may destabilize policy updates\. Letℰg\\mathcal\{E\}\_\{g\}be the set of all valid trajectory–turn pairs in groupgg\. LAPO estimates a scale factor using the median nonzero absolute answer\-likelihood gain within the group:

𝒟g=\{\|Δj,s\|:\(j,s\)∈ℰg,\|Δj,s\|\>ϵig\},τg=\{median⁡\(𝒟g\)\+ϵig,𝒟g≠∅,ϵig,𝒟g=∅\.\\mathcal\{D\}\_\{g\}=\\left\\\{\|\\Delta\_\{j,s\}\|:\(j,s\)\\in\\mathcal\{E\}\_\{g\},\\,\|\\Delta\_\{j,s\}\|\>\\epsilon\_\{\\mathrm\{ig\}\}\\right\\\},\\qquad\\tau\_\{g\}=\\begin\{cases\}\\operatorname\{median\}\(\\mathcal\{D\}\_\{g\}\)\+\\epsilon\_\{\\mathrm\{ig\}\},&\\mathcal\{D\}\_\{g\}\\neq\\varnothing,\\\\ \\epsilon\_\{\\mathrm\{ig\}\},&\\mathcal\{D\}\_\{g\}=\\varnothing\.\\end\{cases\}\(9\)Here,ϵig\\epsilon\_\{\\mathrm\{ig\}\}is a numerical\-stability constant\. When all answer\-likelihood gains in a group are zero,𝒟g\\mathcal\{D\}\_\{g\}is empty andτg\\tau\_\{g\}falls back toϵig\\epsilon\_\{\\mathrm\{ig\}\}\. All bounded gainszi,tz\_\{i,t\}are then zero, so LAPO adds no process advantage\.

We next apply a bounded transformation to the answer\-likelihood gain using the hyperbolic tangent:

zi,t=tanh⁡\(Δi,tτg\)\.z\_\{i,t\}=\\tanh\\left\(\\frac\{\\Delta\_\{i,t\}\}\{\\tau\_\{g\}\}\\right\)\.\(10\)This transformation preserves the sign and relative magnitude of the raw answer\-likelihood gain while limiting the influence of extreme values on subsequent advantage computation\.

Finally, LAPO standardizes the bounded gains within the group:

z~i,t=zi,t−μz\(g\)σz\(g\)\+ϵz,\\widetilde\{z\}\_\{i,t\}=\\frac\{z\_\{i,t\}\-\\mu\_\{z\}^\{\(g\)\}\}\{\\sigma\_\{z\}^\{\(g\)\}\+\\epsilon\_\{z\}\},\(11\)whereμz\(g\)\\mu\_\{z\}^\{\(g\)\}andσz\(g\)\\sigma\_\{z\}^\{\(g\)\}are the mean and standard deviation, respectively, of\{zj,s:\(j,s\)∈ℰg\}\\\{z\_\{j,s\}:\(j,s\)\\in\\mathcal\{E\}\_\{g\}\\\}\. This normalization spans all sampled trajectories for the same question and all valid search turns in those trajectories\. Thus,z~i,t\\widetilde\{z\}\_\{i,t\}measures the target turn’s contribution relative to the other valid turns in the same group and provides a scale\-aligned process signal across questions and trajectories\.

#### Sign\-consistency gating and advantage assignment\.

Although group normalization measures the relative contribution of a search turn, subtracting the group mean can reverse the sign of the raw answer\-likelihood gain\. For example, a turn with a negative raw attribution may receive a positive normalized advantage if its contribution is above the group mean\. Using such a directionally reversed signal directly could make the policy update contradict the raw attribution identified by the leave\-one\-turn comparison\.

LAPO therefore introduces sign\-consistency gating, which retains only process signals for which the raw answer\-likelihood gain and normalized turn advantage point in the same direction:

gi,t=𝕀​\[Δi,t​z~i,t\>0\]\.g\_\{i,t\}=\\mathbb\{I\}\\left\[\\Delta\_\{i,t\}\\widetilde\{z\}\_\{i,t\}\>0\\right\]\.\(12\)The gated process advantage is

Ai,tproc=gi,t​z~i,t\.A\_\{i,t\}^\{\\mathrm\{proc\}\}=g\_\{i,t\}\\widetilde\{z\}\_\{i,t\}\.\(13\)
WhenΔi,t\\Delta\_\{i,t\}andz~i,t\\widetilde\{z\}\_\{i,t\}have the same sign, the normalized turn advantage is retained; when their directions disagree, the process advantage is set to zero\. This gate filters only directionally unreliable auxiliary process signals and does not affect the trajectory’s original outcome advantage\.

LAPO then assigns the process advantage to the policy\-generated tokens in the corresponding search turn\. For thekk\-th policy\-generated token in trajectoryii, the training advantage is

Ai,ktoken=\{Aiout\+λ​Ai,tproc,k​belongs to valid search turn​t,Aiout,k​belongs to the final\-answer turn,A\_\{i,k\}^\{\\mathrm\{token\}\}=\\begin\{cases\}A\_\{i\}^\{\\mathrm\{out\}\}\+\\lambda A\_\{i,t\}^\{\\mathrm\{proc\}\},&k\\text\{ belongs to valid search turn \}t,\\\\ A\_\{i\}^\{\\mathrm\{out\}\},&k\\text\{ belongs to the final\-answer turn\},\\end\{cases\}\(14\)whereλ\\lambdacontrols the weight of the process advantage\. Thus, all policy\-generated tokens retain terminal task supervision, while search turns that pass sign\-consistency gating receive an additional turn\-level process signal\. The final\-answer turn participates in neither forward nor backward process\-reward computation and receives only the trajectory\-level outcome advantage\. Retrieval observations are generated by the environment and do not participate in policy optimization\.

#### Policy optimization\.

LAPO directly uses the clipped surrogate objective of GRPO for policy updates\. Letπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}be the old policy that generated the current trajectories, and lethi,kh\_\{i,k\}be the complete context before thekk\-th policy token, including the question, all preceding policy tokens, and environment observations\. The importance\-sampling ratio for thekk\-th policy token is

ρi,k​\(θ\)=πθ​\(yi,k∣hi,k\)πθold​\(yi,k∣hi,k\)\.\\rho\_\{i,k\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(y\_\{i,k\}\\mid h\_\{i,k\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(y\_\{i,k\}\\mid h\_\{i,k\}\)\}\.\(15\)
Using the token\-level advantage above, the LAPO objective is

𝒥LAPO​\(θ\)\\displaystyle\\mathcal\{J\}\_\{\\mathrm\{LAPO\}\}\(\\theta\)=𝔼x∼𝒟,τi∼πθold\[1\|𝒫i\|∑k∈𝒫i\(\\displaystyle=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\},\\,\\tau\_\{i\}\\sim\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\}\\Bigg\[\\frac\{1\}\{\|\\mathcal\{P\}\_\{i\}\|\}\\sum\_\{k\\in\\mathcal\{P\}\_\{i\}\}\\Bigg\(min\[ρi,k\(θ\)Ai,ktoken,\\displaystyle\\quad\\min\\Big\[\\rho\_\{i,k\}\(\\theta\)A\_\{i,k\}^\{\\mathrm\{token\}\},clip\(ρi,k\(θ\),1−ϵclip,1\+ϵclip\)Ai,ktoken\]−βDKL\(i,k\)\(πθ∥πref\)\)\],\\displaystyle\\qquad\\operatorname\{clip\}\\big\(\\rho\_\{i,k\}\(\\theta\),1\-\\epsilon\_\{\\mathrm\{clip\}\},1\+\\epsilon\_\{\\mathrm\{clip\}\}\\big\)A\_\{i,k\}^\{\\mathrm\{token\}\}\\Big\]\-\\beta D\_\{\\mathrm\{KL\}\}^\{\(i,k\)\}\\big\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\mathrm\{ref\}\}\\big\)\\Bigg\)\\Bigg\],\(16\)where𝒫i\\mathcal\{P\}\_\{i\}is the set of policy\-generated tokens in trajectoryiithat participate in policy optimization,ϵclip\\epsilon\_\{\\mathrm\{clip\}\}is the clipping range,πref\\pi\_\{\\mathrm\{ref\}\}is the reference policy,DKL\(i,k\)D\_\{\\mathrm\{KL\}\}^\{\(i,k\)\}is the KL\-regularization estimate at thekk\-th policy\-token position, andβ\\betacontrols the strength of KL regularization\. Training maximizes𝒥LAPO​\(θ\)\\mathcal\{J\}\_\{\\mathrm\{LAPO\}\}\(\\theta\)\.

LAPO does not change the form of GRPO policy optimization\. Before optimization, it uses backward leave\-one\-turn attribution to refine the trajectory\-level outcome advantage into search\-turn\-specific token\-level advantages\. Turn\-level credit signals can therefore be integrated directly into the existing GRPO training pipeline without an additional process reward model or changes to the underlying optimizer\.

## 4Experiments

### 4\.1Experimental Setup

#### Datasets and evaluation\.

We construct the reinforcement\-learning training set from Natural Questions \(NQ\), TriviaQA \(TQ\), HotpotQA, and 2WikiMultiHopQA \(2Wiki\)Kwiatkowskiet al\.\([2019](https://arxiv.org/html/2607.13501#bib.bib21)\); Joshiet al\.\([2017](https://arxiv.org/html/2607.13501#bib.bib22)\); Yanget al\.\([2018](https://arxiv.org/html/2607.13501#bib.bib23)\); Hoet al\.\([2020](https://arxiv.org/html/2607.13501#bib.bib24)\)\. Specifically, we randomly sample 20,000 examples from the training split of each dataset, yielding 80,000 training examples in total\. For each of the seven evaluation datasets below, we randomly sample 500 examples from the corresponding original split for validation and another 500 for testing, ensuring that the validation and test samples are disjoint\. We evaluate all methods on seven question\-answering benchmarks: NQ, TriviaQA, HotpotQA, 2Wiki, MuSiQue, Bamboogle, and PopQATrivediet al\.\([2022](https://arxiv.org/html/2607.13501#bib.bib25)\); Presset al\.\([2023](https://arxiv.org/html/2607.13501#bib.bib26)\); Mallenet al\.\([2023](https://arxiv.org/html/2607.13501#bib.bib27)\)\. Because NQ, TriviaQA, HotpotQA, and 2Wiki are used during training, we treat them as in\-domain datasets\. MuSiQue, Bamboogle, and PopQA are not used for training and evaluate out\-of\-domain generalization\. Together, these datasets cover both single\-hop knowledge\-intensive question answering and multi\-hop question answering that requires iterative retrieval and evidence aggregation\. At test time, we use exact match \(EM\) as the primary evaluation metric\.Avg\.in the tables denotes the macro\-average EM across the seven datasets, with equal weight assigned to each dataset\. During training, we use answer\-level F1 as the outcome\-reward signal, measuring token overlap between the model prediction and the reference answer\.

#### Baselines\.

We compare LAPO with three categories of representative methods\. The first category consists of*prompt\-based methods*without task\-level reinforcement\-learning optimization: Direct Inference, Chain\-of\-Thought \(CoT\)Weiet al\.\([2022](https://arxiv.org/html/2607.13501#bib.bib19)\), Search\-o1Liet al\.\([2025](https://arxiv.org/html/2607.13501#bib.bib18)\), and retrieval\-augmented generation \(RAG\)Lewiset al\.\([2020](https://arxiv.org/html/2607.13501#bib.bib20)\)\. These methods represent direct answering, explicit reasoning, and retrieval\-augmented paradigms\. The second category comprises*outcome\-reward\-based methods*: Search\-R1\-base, Search\-R1\-instructJinet al\.\([2025](https://arxiv.org/html/2607.13501#bib.bib1)\), and R1\-SearcherSonget al\.\([2025](https://arxiv.org/html/2607.13501#bib.bib2)\)\. These methods optimize entire search trajectories primarily according to final\-answer quality, without explicitly modeling the separate attribution of intermediate search turns to the final outcome\. The third category comprises*step\-reward\-based methods*: StepSearch\-base, StepSearch\-instructZhenget al\.\([2025](https://arxiv.org/html/2607.13501#bib.bib15)\), and IGPOWanget al\.\([2026](https://arxiv.org/html/2607.13501#bib.bib13)\)\. IGPO is the method most closely related to LAPO: both optimize multi\-turn search behavior using process rewards, but they use different attribution mechanisms\. In the main results, IGPO denotes the original forward\-attribution method without our sign\-consistency gating; the ablation study separately reports a variant that adds this gate to IGPO\.

#### Experimental configuration\.

All methods use a local retrieval corpus constructed from the 2018 English Wikipedia\. The corpus is processed following Karpukhin et al\.Karpukhinet al\.\([2020](https://arxiv.org/html/2607.13501#bib.bib28)\), and E5Wanget al\.\([2022](https://arxiv.org/html/2607.13501#bib.bib29)\)is used as the dense retriever\. Each search returns the three most relevant passages\. The model may perform at most three search turns before producing its final answer\. All methods share the same retrieval corpus, retriever, number of returned documents, and maximum number of search turns to prevent differences in retrieval environments from confounding the comparison\. All reinforcement\-learning methods are trained for 200 optimization steps on eight NVIDIA A100 GPUs\. The global training batch size is 512, andG=5G=5rollouts are sampled for each input prompt\. Unless otherwise specified, all methods use the same model initialization, training data, prompt template, decoding parameters, and retrieval budget\. The optimizer, learning rate, data preprocessing, and other generation settings are provided in Appendix[A](https://arxiv.org/html/2607.13501#A1)\.

### 4\.2Main Results

Table 1:Exact match \(EM\) on seven knowledge\-intensive question\-answering datasets\.NQ, TriviaQA, HotpotQA, and 2Wiki are in\-domain datasets, while MuSiQue, Bamboogle, and PopQA are out\-of\-domain datasets\. The average is the macro\-average across all seven datasets\. The best result in each column is shown in bold\.#### Overall performance\.

As shown in Table[1](https://arxiv.org/html/2607.13501#S4.T1), LAPO achieves the best result on all seven evaluation datasets, with an average EM of0\.3260\.326\. LAPO yields an absolute improvement of0\.0630\.063over the strongest outcome\-reward baseline, Search\-R1\-instruct, and improves average EM by0\.0530\.053over the strongest step\-reward baseline, IGPO, corresponding to a relative gain of19\.4%19\.4\\%\.

This result supports the view that final\-answer feedback alone may be insufficient for credit assignment in multi\-turn search trajectories\. Outcome rewards can encourage the policy to generate trajectories that succeed overall, but they do not further identify which intermediate interactions actually support the final answer\. LAPO converts the prediction change caused by replacing a single search turn with\[DELETE\]into a turn\-level learning signal, providing finer\-grained supervision without an external process reward model\.

#### Comparison with step\-level attribution methods\.

IGPO generally outperforms most prompt\-based and outcome\-reward\-based methods, indicating the value of process\-level supervision for training multi\-turn search agents\. Building on this observation, LAPO achieves higher EM than IGPO on all seven datasets\.

The main difference between the two methods lies in the context used to evaluate the attribution of an intermediate turn\. Forward information gain is computed immediately after the current turn is generated and therefore cannot account for evidence retrieved later\. LAPO instead reevaluates each intermediate turn after the complete search trajectory has been generated, using the final downstream context\. In multi\-turn retrieval, the value of an early query or an intermediate piece of evidence may become apparent only after it is combined with later evidence\. Retrospective attribution can therefore capture cross\-turn dependencies that are difficult to identify from a local prefix alone\. LAPO’s clear gains on multi\-hop datasets such as HotpotQA and 2Wiki are consistent with this motivation\.

#### In\-domain and out\-of\-domain generalization\.

LAPO’s advantage also transfers to data distributions not observed during training\. Across MuSiQue, Bamboogle, and PopQA, LAPO attains an average EM of0\.2330\.233, compared with0\.1770\.177for IGPO\. The improvement appears on all three out\-of\-domain datasets and is not driven by an anomalous gain on a single dataset\. These results indicate that LAPO’s performance advantage transfers to the three out\-of\-domain datasets considered here, suggesting some cross\-dataset generalization in the learned search and evidence\-use behavior\.

### 4\.3Training Dynamics and Process\-Reward Analysis

![Refer to caption](https://arxiv.org/html/2607.13501v1/x2.png)

Figure 2:Performance dynamics on held\-out validation sets\.Exact match \(EM\) for LAPO, IGPO, and Search\-R1 over training steps; Total denotes the aggregate result over the four in\-domain validation sets\.![Refer to caption](https://arxiv.org/html/2607.13501v1/x3.png)

Figure 3:Training\-performance dynamics\.Trajectory\-level answer F1 for the three methods over training steps\.
![Refer to caption](https://arxiv.org/html/2607.13501v1/x4.png)

Figure 4:Evolution of process signals\.Proportions of turns with positive and negative answer\-likelihood gains under LAPO and IGPO\.

#### Optimization dynamics and validation performance\.

Figures[4](https://arxiv.org/html/2607.13501#S4.F4)and[4](https://arxiv.org/html/2607.13501#S4.F4)compare the reinforcement\-learning dynamics and held\-out performance of LAPO, IGPO, and Search\-R1\. LAPO improves rapidly early in training and subsequently maintains a higher trajectory\-level answer F1\. Late in training, LAPO’s training F1 approaches0\.570\.57, while IGPO and Search\-R1 remain around0\.470\.47–0\.480\.48\. This pattern is consistent with LAPO’s motivation of providing finer\-grained turn\-level supervision\.

LAPO’s advantage on training trajectories also transfers to held\-out validation examples\. On the aggregate validation set, LAPO reaches the highest validation performance late in training and shows particularly clear advantages on HotpotQA and 2Wiki\. The training and validation curves exhibit consistent performance trends, indicating that higher outcome rewards on training trajectories accompany better validation performance on held\-out queries; these curves alone, however, cannot rule out other factors such as overfitting\.

In contrast, the differences among methods on TriviaQA are smaller and the curves overlap more strongly\. A possible explanation is that many TriviaQA questions can be answered through relatively direct retrieval, limiting the additional benefit of fine\-grained credit assignment\.

#### Evolution of process signals\.

Figure[4](https://arxiv.org/html/2607.13501#S4.F4)shows the proportions of search turns with positive and negative raw answer\-likelihood gains during training\. As training proceeds, the proportion of positively attributed search turns under LAPO gradually increases, while the proportion of negatively attributed turns continues to decrease\. IGPO exhibits a similar trend, but its positive proportion grows more slowly than LAPO’s and it retains more negative process updates late in training\. This change is consistent with the policy gradually increasing positively attributed turns and reducing negatively attributed turns, although turn proportions alone do not establish a causal improvement in specific search behavior\.

### 4\.4Ablation Study

#### Effect of backward leave\-one\-turn attribution\.

Table[2](https://arxiv.org/html/2607.13501#S4.T2)examines the effects of attribution direction and sign\-consistency gating\. Without gating, replacing IGPO’s forward attribution with LAPO’s backward leave\-one\-turn attribution improves average EM from0\.2730\.273to0\.2850\.285\. The gains appear primarily on multi\-hop datasets such as HotpotQA, 2Wiki, and MuSiQue\. This pattern is consistent with LAPO’s central motivation: it evaluates the attribution of a target turn while preserving the downstream context by replacing the entire turn with\[DELETE\]in the complete search trajectory and comparing the resulting answer likelihoods\.

Changing only the attribution direction, however, yields a relatively modest average improvement\. One reason is that standardization can reverse the direction of an attribution signal\. For example, a search turn with a negative raw answer\-likelihood gain may still receive a positive normalized advantage if its score is above the group mean\. Directly optimizing with this advantage would make the policy\-update direction contradict the raw attribution obtained from the leave\-one\-turn comparison\.

#### Effect of sign\-consistency gating\.

Adding sign\-consistency gating to IGPO improves its average EM from0\.2730\.273to0\.2820\.282, indicating that enforcing directional agreement between the normalized advantage and raw attribution can also improve forward attribution\. The effect is clearer for LAPO, where gating improves average EM from0\.2850\.285to0\.3260\.326\.

This result suggests that the performance gain of full LAPO arises neither solely from backward attribution nor simply from filtering process scores\. Backward attribution provides an estimate of turn\-level attribution over the complete trajectory, while sign\-consistency gating keeps the normalized relative advantage aligned with the direction of the raw attribution\. Together, they allow LAPO to use attribution signals that preserve downstream context while maintaining directionally consistent policy\-update signals\.

The benefit of gating is not fully uniform across datasets\. For example, LAPO without gating performs slightly better on HotpotQA than the full method\. Overall, however, full LAPO achieves the highest average EM and the best performance on six of the seven datasets\.

Table 2:Ablation results for attribution direction and sign\-consistency gating\.We compare IGPO’s forward attribution with LAPO’s backward leave\-one\-turn attribution, with and without gating, using exact match \(EM\)\. All variants use Qwen2\.5\-3B\-InstructQwenet al\.\([2024](https://arxiv.org/html/2607.13501#bib.bib30)\)with the same training data and optimization configuration\. The best result in each column is shown in bold\.

## 5Conclusion

We presented LAPO, a self\-generated process\-supervision method for multi\-turn search reasoning\. Through backward leave\-one\-turn attribution, LAPO compares the gold\-answer likelihood under the full context with that under a context in which a single search turn is replaced in its entirety with\[DELETE\], thereby estimating the attribution of each search turn under the current context and policy\. It further uses sign\-consistency gating to filter directionally unreliable process signals\. LAPO requires no additional process reward model, teacher model, verifier, or LLM\-as\-a\-Judge and can be integrated directly into an existing GRPO training framework\. Across seven knowledge\-intensive question\-answering datasets, LAPO outperforms the compared outcome\-reward\- and step\-reward\-based baselines, supporting the utility of retrospective turn\-level attribution for credit assignment in multi\-turn search\.

LAPO also has several limitations\. Its leave\-one\-turn score is a context\- and policy\-dependent counterfactual attribution rather than a formally identified causal effect, and it does not explicitly model interactions among multiple replaced turns\. The current signal also depends on gold answers, and our experiments are limited to the evaluated model scale, local retrieval environment, and relatively short search trajectories\. Future work may investigate more efficient counterfactual approximations, model higher\-order interactions among search turns, extend LAPO to longer trajectories, web search, and multimodal agents, and reduce its dependence on explicit gold\-answer supervision\.

## References

- M\. Chen, T\. Li, H\. Sun, Y\. Zhou, C\. Zhu, F\. Yang, Z\. Zhou, W\. Chen, H\. Wang, J\. Z\. Pan, W\. Zhang, and H\. Chen \(2025\)ReSearch: learning to reason with search for LLMs via reinforcement learning\.arXiv preprint arXiv:2503\.19470\.External Links:[Link](https://arxiv.org/abs/2503.19470)Cited by:[§2\.1](https://arxiv.org/html/2607.13501#S2.SS1.p1.1)\.
- J\. Fang, Z\. Hong, M\. Zheng, M\. Song, G\. Li, H\. Jiang, D\. Zhang, H\. Guo, X\. Wang, and T\. Chua \(2026\)Rubric\-based on\-policy distillation\.arXiv preprint arXiv:2605\.07396\.External Links:[Link](https://arxiv.org/abs/2605.07396)Cited by:[§1](https://arxiv.org/html/2607.13501#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.13501#S2.SS2.p2.1)\.
- X\. Feng, B\. Han, Z\. Zhou, J\. Fan, J\. Yao, K\. H\. Li, D\. Yu, and M\. K\. Ng \(2026\)RewardFlow: topology\-aware reward propagation on state graphs for agentic RL with large language models\.arXiv preprint arXiv:2603\.18859\.External Links:[Link](https://arxiv.org/abs/2603.18859)Cited by:[§2\.3](https://arxiv.org/html/2607.13501#S2.SS3.p1.1)\.
- X\. Ho, A\. Duong Nguyen, S\. Sugawara, and A\. Aizawa \(2020\)Constructing a multi\-hop QA dataset for comprehensive evaluation of reasoning steps\.InProceedings of the 28th International Conference on Computational Linguistics,pp\. 6609–6625\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.coling-main.580),[Link](https://aclanthology.org/2020.coling-main.580/)Cited by:[§4\.1](https://arxiv.org/html/2607.13501#S4.SS1.SSS0.Px1.p1.1)\.
- S\. Hu, Y\. Dai, Y\. Zhao, Y\. Tao, Y\. Guo, Z\. Fang, S\. T\. W\. Kwong, and Y\. Fang \(2026\)Optimizing agentic reasoning with retrieval via synthetic semantic information gain reward\.arXiv preprint arXiv:2602\.00845\.Note:Accepted at the 43rd International Conference on Machine LearningExternal Links:[Link](https://arxiv.org/abs/2602.00845)Cited by:[§2\.3](https://arxiv.org/html/2607.13501#S2.SS3.p1.1)\.
- B\. Jin, H\. Zeng, Z\. Yue, J\. Yoon, S\. O\. Arik, D\. Wang, H\. Zamani, and J\. Han \(2025\)Search\-R1: training LLMs to reason and leverage search engines with reinforcement learning\.InProceedings of the Second Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=Rwhi91ideu)Cited by:[§1](https://arxiv.org/html/2607.13501#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.13501#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.13501#S4.SS1.SSS0.Px2.p1.1)\.
- M\. Joshi, E\. Choi, D\. S\. Weld, and L\. Zettlemoyer \(2017\)TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1601–1611\.External Links:[Document](https://dx.doi.org/10.18653/v1/P17-1147),[Link](https://aclanthology.org/P17-1147/)Cited by:[§4\.1](https://arxiv.org/html/2607.13501#S4.SS1.SSS0.Px1.p1.1)\.
- V\. Karpukhin, B\. Oguz, S\. Min, P\. Lewis, L\. Wu, S\. Edunov, D\. Chen, and W\. Yih \(2020\)Dense passage retrieval for open\-domain question answering\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing,pp\. 6769–6781\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.550),[Link](https://aclanthology.org/2020.emnlp-main.550/)Cited by:[§4\.1](https://arxiv.org/html/2607.13501#S4.SS1.SSS0.Px3.p1.1)\.
- T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee, K\. Toutanova, L\. Jones, M\. Kelcey, M\. Chang, A\. M\. Dai, J\. Uszkoreit, Q\. Le, and S\. Petrov \(2019\)Natural questions: a benchmark for question answering research\.Transactions of the Association for Computational Linguistics7,pp\. 452–466\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00276),[Link](https://aclanthology.org/Q19-1026/)Cited by:[§4\.1](https://arxiv.org/html/2607.13501#S4.SS1.SSS0.Px1.p1.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela \(2020\)Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 9459–9474\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html)Cited by:[§4\.1](https://arxiv.org/html/2607.13501#S4.SS1.SSS0.Px2.p1.1)\.
- X\. Li, G\. Dong, J\. Jin, Y\. Zhang, Y\. Zhou, Y\. Zhu, P\. Zhang, and Z\. Dou \(2025\)Search\-o1: agentic search\-enhanced large reasoning models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 5420–5438\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.276),[Link](https://aclanthology.org/2025.emnlp-main.276/)Cited by:[§4\.1](https://arxiv.org/html/2607.13501#S4.SS1.SSS0.Px2.p1.1)\.
- Z\. Liang, Y\. Ma, B\. Chen, Z\. Qian, H\. Dai, L\. Mao, X\. Zhang, C\. Lei, and W\. Ou \(2026\)IG\-Search: step\-level information gain rewards for search\-augmented reasoning\.arXiv preprint arXiv:2604\.15148\.External Links:[Link](https://arxiv.org/abs/2604.15148)Cited by:[§2\.3](https://arxiv.org/html/2607.13501#S2.SS3.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2023\)Let’s verify step by step\.arXiv preprint arXiv:2305\.20050\.External Links:[Link](https://arxiv.org/abs/2305.20050)Cited by:[§1](https://arxiv.org/html/2607.13501#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.13501#S2.SS2.p1.1)\.
- A\. Mallen, A\. Asai, V\. Zhong, R\. Das, D\. Khashabi, and H\. Hajishirzi \(2023\)When not to trust language models: investigating effectiveness of parametric and non\-parametric memories\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9802–9822\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.546),[Link](https://aclanthology.org/2023.acl-long.546/)Cited by:[§4\.1](https://arxiv.org/html/2607.13501#S4.SS1.SSS0.Px1.p1.1)\.
- O\. Press, M\. Zhang, S\. Min, L\. Schmidt, N\. A\. Smith, and M\. Lewis \(2023\)Measuring and narrowing the compositionality gap in language models\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 5687–5711\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.378),[Link](https://aclanthology.org/2023.findings-emnlp.378/)Cited by:[§4\.1](https://arxiv.org/html/2607.13501#S4.SS1.SSS0.Px1.p1.1)\.
- Qwen, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2024\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.External Links:[Link](https://arxiv.org/abs/2412.15115)Cited by:[Table 2](https://arxiv.org/html/2607.13501#S4.T2)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.External Links:[Link](https://arxiv.org/abs/2402.03300)Cited by:[§1](https://arxiv.org/html/2607.13501#S1.p5.1)\.
- H\. Song, J\. Jiang, Y\. Min, J\. Chen, Z\. Chen, W\. X\. Zhao, L\. Fang, and J\. Wen \(2025\)R1\-Searcher: incentivizing the search capability in LLMs via reinforcement learning\.arXiv preprint arXiv:2503\.05592\.External Links:[Link](https://arxiv.org/abs/2503.05592)Cited by:[§2\.1](https://arxiv.org/html/2607.13501#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.13501#S4.SS1.SSS0.Px2.p1.1)\.
- Z\. Tan, J\. Huang, Q\. Wu, H\. Zhang, C\. Zhuang, and J\. Gu \(2026\)RAG\-R1: incentivizing the search and reasoning capabilities of LLMs through multi\-query parallelism\.Proceedings of the AAAI Conference on Artificial Intelligence40\(39\),pp\. 33187–33195\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v40i39.40603),[Link](https://ojs.aaai.org/index.php/AAAI/article/view/40603)Cited by:[§2\.1](https://arxiv.org/html/2607.13501#S2.SS1.p1.1)\.
- H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal \(2022\)MuSiQue: multihop questions via single\-hop question composition\.Transactions of the Association for Computational Linguistics10,pp\. 539–554\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00475),[Link](https://aclanthology.org/2022.tacl-1.31/)Cited by:[§4\.1](https://arxiv.org/html/2607.13501#S4.SS1.SSS0.Px1.p1.1)\.
- J\. Uesato, N\. Kushman, R\. Kumar, F\. Song, N\. Siegel, L\. Wang, A\. Creswell, G\. Irving, and I\. Higgins \(2022\)Solving math word problems with process\- and outcome\-based feedback\.arXiv preprint arXiv:2211\.14275\.External Links:[Link](https://arxiv.org/abs/2211.14275)Cited by:[§1](https://arxiv.org/html/2607.13501#S1.p3.1)\.
- G\. Wang, S\. Dai, G\. Ye, Z\. Gan, W\. Yao, Y\. Deng, X\. Wu, and Z\. Ying \(2026\)Information gain\-based policy optimization: a simple and effective approach for multi\-turn search agents\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=qkWP6phrvZ)Cited by:[§2\.3](https://arxiv.org/html/2607.13501#S2.SS3.p1.1),[§4\.1](https://arxiv.org/html/2607.13501#S4.SS1.SSS0.Px2.p1.1)\.
- L\. Wang, N\. Yang, X\. Huang, B\. Jiao, L\. Yang, D\. Jiang, R\. Majumder, and F\. Wei \(2022\)Text embeddings by weakly\-supervised contrastive pre\-training\.arXiv preprint arXiv:2212\.03533\.External Links:[Link](https://arxiv.org/abs/2212.03533)Cited by:[§4\.1](https://arxiv.org/html/2607.13501#S4.SS1.SSS0.Px3.p1.1)\.
- P\. Wang, L\. Li, Z\. Shao, R\. Xu, D\. Dai, Y\. Li, D\. Chen, Y\. Wu, and Z\. Sui \(2024\)Math\-Shepherd: verify and reinforce LLMs step\-by\-step without human annotations\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9426–9439\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.510),[Link](https://aclanthology.org/2024.acl-long.510/)Cited by:[§1](https://arxiv.org/html/2607.13501#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.13501#S2.SS2.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. V\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 24824–24837\.External Links:[Link](https://proceedings.neurips.cc/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html)Cited by:[§4\.1](https://arxiv.org/html/2607.13501#S4.SS1.SSS0.Px2.p1.1)\.
- Q\. Wei, S\. Zeng, C\. Li, W\. Brown, O\. Frunza, W\. Deng, A\. Schneider, Y\. Nevmyvaka, Y\. K\. Zhao, A\. Garcia, and M\. Hong \(2025a\)Reinforcing multi\-turn reasoning in LLM agents via turn\-level reward design\.arXiv preprint arXiv:2505\.11821\.External Links:[Link](https://arxiv.org/abs/2505.11821)Cited by:[§2\.3](https://arxiv.org/html/2607.13501#S2.SS3.p1.1)\.
- Z\. Wei, W\. Yao, Y\. Liu, W\. Zhang, Q\. Lu, L\. Qiu, C\. Yu, P\. Xu, C\. Zhang, B\. Yin, H\. Yun, and L\. Li \(2025b\)WebAgent\-R1: training web agents via end\-to\-end multi\-turn reinforcement learning\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 7909–7928\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.401),[Link](https://aclanthology.org/2025.emnlp-main.401/)Cited by:[§2\.1](https://arxiv.org/html/2607.13501#S2.SS1.p1.1)\.
- J\. Wu, Z\. Deng, W\. Li, Y\. Liu, B\. You, B\. Li, Z\. Ma, and Z\. Liu \(2026\)MMSearch\-R1: incentivizing LMMs to search\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2456–2487\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.114),[Link](https://aclanthology.org/2026.acl-long.114/)Cited by:[§2\.1](https://arxiv.org/html/2607.13501#S2.SS1.p1.1)\.
- Y\. Xie, N\. Thomas, N\. Hansen, Y\. Fu, L\. E\. Li, and X\. Wang \(2026\)TIPS: turn\-level information\-potential reward shaping for search\-augmented LLMs\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=eBMOr6a84z)Cited by:[§2\.3](https://arxiv.org/html/2607.13501#S2.SS3.p1.1)\.
- S\. Yang, J\. Wu, Z\. Lu, Y\. Shen, F\. Zhang, L\. Feng, S\. Zhang, H\. Luo, Z\. Lian, Z\. Wen, and J\. Tao \(2026\)OPID: on\-policy skill distillation for agentic reinforcement learning\.arXiv preprint arXiv:2606\.26790\.External Links:[Link](https://arxiv.org/abs/2606.26790)Cited by:[§2\.2](https://arxiv.org/html/2607.13501#S2.SS2.p2.1)\.
- Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning \(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp\. 2369–2380\.External Links:[Document](https://dx.doi.org/10.18653/v1/D18-1259),[Link](https://aclanthology.org/D18-1259/)Cited by:[§4\.1](https://arxiv.org/html/2607.13501#S4.SS1.SSS0.Px1.p1.1)\.
- D\. Zhang, S\. Zhoubian, Z\. Hu, Y\. Yue, Y\. Dong, and J\. Tang \(2024\)ReST\-MCTS\*: LLM self\-training via process reward guided tree search\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 64735–64772\.External Links:[Document](https://dx.doi.org/10.52202/079017-2066),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/76ec4dc30e9faaf0e4b6093eaa377218-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2607.13501#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.13501#S2.SS2.p1.1)\.
- Y\. Zhang, H\. Huang, Z\. Song, Y\. Zhu, Q\. Zhang, Z\. Zhao, and D\. Zhao \(2026\)CriticSearch: fine\-grained credit assignment for search agents via a retrospective critic\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 12272–12290\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.596),[Link](https://aclanthology.org/2026.findings-acl.596/)Cited by:[§2\.2](https://arxiv.org/html/2607.13501#S2.SS2.p1.1)\.
- X\. Zheng, K\. An, Z\. Wang, Y\. Wang, and Y\. Wu \(2025\)StepSearch: igniting LLMs search ability via step\-wise proximal policy optimization\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 21805–21830\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1106),[Link](https://aclanthology.org/2025.emnlp-main.1106/)Cited by:[§2\.3](https://arxiv.org/html/2607.13501#S2.SS3.p1.1),[§4\.1](https://arxiv.org/html/2607.13501#S4.SS1.SSS0.Px2.p1.1)\.

## Appendix AReproducibility Details

This section provides additional details about LAPO’s training configuration, retrieval environment, and counterfactual scoring implementation\. Unless otherwise specified, all reinforcement\-learning methods use the same model initialization, training data, prompt template, decoding parameters, and retrieval budget\.

### A\.1Training and Optimization Configuration

Table[3](https://arxiv.org/html/2607.13501#A1.T3)summarizes the key hyperparameters for training and LAPO process scoring\. The base policy is initialized from Qwen2\.5\-3B\-Instruct and optimized with GRPO’s group\-relative advantage estimation and clipped surrogate objective\. We use the AdamW optimizer, and all experiments are conducted on a single node with eight NVIDIA A100 GPUs\.

Table 3:Key configurations for training and LAPO process scoring\.ConfigurationValueGlobal training batch size512512PPO mini\-batch size256256PPO micro\-batch size6464Sampled trajectories per questionGG55OptimizerAdamWLearning rate1×10−61\\times 10^\{\-6\}PPO clipping rangeϵclip\\epsilon\_\{\\mathrm\{clip\}\}0\.20\.2KL coefficient0\.0010\.001Optimization steps200200Training hardware8×8\\timesNVIDIA A100Process\-advantage weightλ\\lambda0\.50\.5Answer\-likelihood\-gain stability termϵig\\epsilon\_\{\\mathrm\{ig\}\}\(ig\_eps\)10−610^\{\-6\}
### A\.2Retrieval Environment and Generation Settings

All methods share the same local retrieval environment\. The retrieval corpus is constructed from the 2018 English Wikipedia, and similarity search is performed using an E5 dense retriever with a FAISS index\. Each search returns the three most relevant passages\. The retrieval results are wrapped in a unified title\-and\-body format inside<information\>tags and appended to the current context as environment\-generated observations\.

The model may perform at most three search interactions\. After exhausting the search budget or generating a final response, the trajectory enters or completes the answer\-generation stage\. The final\-answer turn does not trigger retrieval and does not participate in LAPO process attribution\. We use stochastic sampling during training and greedy decoding for validation and testing\. Table[4](https://arxiv.org/html/2607.13501#A1.T4)provides the detailed configuration\.

Table 4:Key retrieval and generation configurations\.
### A\.3Counterfactual\-Context Construction and Batched Scoring

For each valid search turnttin trajectoryii, we replace the entire search action and its retrieval observation with a fixed\[DELETE\]marker to obtain the counterfactual contextci,−tc\_\{i,\-t\}\. This operation preserves the order of all interactions before and after the target turn and explicitly marks the absence of the target interaction with a placeholder\. The design avoids directly concatenating the text before and after the target turn, although we do not separately evaluate the placeholder’s effect on contextual coherence\. The final\-answer turn is always excluded from the scoring context, so the gold\-answer likelihood reflects only the support provided by preceding search interactions\.

To compute answer\-likelihood gains, we flatten the full contexts and all leave\-one\-turn counterfactual contexts in the same rollout batch into a single scoring batch\. We append the same gold\-answer scoring template to every context and compute the mean log\-likelihood only over the token span corresponding to the gold answer\. The full\-context score is reused across all turns in the same trajectory, reducing redundant forward computation and model\-scheduling overhead\.

### A\.4Training Cost

To assess the additional training cost of counterfactual process attribution, we measure the average training time per step for different methods under the same hardware environment and training configuration\. The results are shown in Figure[5](https://arxiv.org/html/2607.13501#A1.F5)\.

![Refer to caption](https://arxiv.org/html/2607.13501v1/x5.png)Figure 5:Training time per step for different methods\.Under the same hardware environment and training configuration, the average training times per step for Search\-R1, IGPO, and LAPO are 191, 196, and 198 seconds, respectively\. The red dashed lines and annotations indicate the additional wall\-clock time and relative increase of IGPO and LAPO over Search\-R1\.Search\-R1 requires an average of 191 seconds per training step\. Introducing turn\-level process signals increases this time to 196 seconds for IGPO, an overhead of 5 seconds \(2\.6%2\.6\\%\)\. LAPO requires 198 seconds per step, only 7 seconds \(3\.7%3\.7\\%\) more than Search\-R1 and 2 seconds \(approximately1\.0%1\.0\\%\) more than IGPO\. Under the current implementation and experimental configuration, backward leave\-one\-turn attribution therefore does not produce a large increase in training time\. Its additional overhead is controlled mainly by batching the full and counterfactual contexts for scoring and reusing the full\-context score across turns in the same trajectory\. These measurements are wall\-clock times for specific hardware and implementation choices, not platform\-independent theoretical computational costs\.

## Appendix BPrompt Template

### B\.1Agent Interaction Format

For each input question, we construct the model input with a fixed task prompt and encode it in conversation format using the base model’s native chat template, with an assistant generation prompt appended at the end\. The complete task prompt is:

You are answering a question that may require external search\.At each step, choose exactly one action:1\. If external information is needed:<think\>Write your brief reasoning about what information ismissing and why search is needed\.</think\><search\>one concise search query</search\>2\. If enough information is available:<think\>Write your brief reasoning leading to the answer\.</think\><answer\>final answer only</answer\>Rules:\- Do not output the words "reasoning", "query", or"final answer" as placeholders\.\- Use either <search\> or <answer\>, never both\.\- The <search\> tag must contain only one concise query\.\- Search results may be provided as<information\>\.\.\.</information\>; read them but nevergenerate <information\>\.\- The <answer\> tag should contain only the final answer,with no explanation\.\- Stop immediately after </search\> or </answer\>\.Question: \{question\}

At each interaction turn, the policy first produces brief reasoning for the current step inside<think\>and then selects either a search action or an answer action\. When the available information is insufficient, the policy outputs:

<think\>brief reasoning for search</think\><search\>one concise search query</search\>

The<search\>tag contains only one query to be sent to the retrieval service\. When the policy determines that the context contains sufficient information, it outputs:

<think\>brief reasoning leading to the answer</think\><answer\>final answer only</answer\>

The<answer\>tag contains only the final answer, without additional explanation\.

Each model call may generate exactly one action, either a search action or an answer action, but not both\. Decoding terminates immediately after the first complete and valid</search\>or</answer\>tag, preventing the model from executing multiple actions in a single generation\.

When the policy generates a search action, the environment extracts the query from the<search\>tag and invokes the external retrieval service\. The environment wraps the retrieved passages as:

<information\>retrieved passages</information\>

This retrieval observation is then appended to the current conversation context and used as input to the next model generation\. The<information\>tag is generated only by the environment; the policy never generates it\. When the policy generates an answer action, the current trajectory terminates immediately\.

A complete multi\-turn search trajectory can therefore be represented as:

<think\>\.\.\.</think\><search\>\.\.\.</search\><information\>\.\.\.</information\>\.\.\.<think\>\.\.\.</think\><search\>\.\.\.</search\><information\>\.\.\.</information\><think\>\.\.\.</think\><answer\>\.\.\.</answer\>

A trajectory contains zero or more<think\>\-<search\>\-<information\>interactions and ends with one<think\>\-<answer\>interaction\.

### B\.2Gold\-Answer Likelihood Scoring Template

LAPO appends the same fixed response template to both the full and counterfactual contexts:

<think\>Now there’s enough information to answer</think\><answer\>\{gold\_answer\}</answer\>

During scoring, we sum only the log probabilities of the tokens corresponding to\{gold\_answer\}and average them over the number of answer tokens\. The reasoning prefix, answer tags, and closing tag in the template are excluded from the score\. This design compares the full and counterfactual contexts under the same target sequence and reduces the effect of answer\-length differences on the attribution scale\.

## Appendix CData Processing and Evaluation Details

### C\.1Dataset Construction

The training set consists of the training splits of NQ, TriviaQA, HotpotQA, and 2Wiki, with 20,000 questions randomly sampled from each dataset and 80,000 training examples in total\. Validation and testing cover these four in\-domain datasets and the three out\-of\-domain datasets MuSiQue, Bamboogle, and PopQA\. Table[5](https://arxiv.org/html/2607.13501#A3.T5)summarizes the sampled data sizes\.

Table 5:Sample sizes for training, validation, and testing\. A dash indicates that the dataset is not used for reinforcement\-learning training\.Each example contains a question in conversation format and one or more reference answers\. When multiple reference answers are available, we compute both the training reward and test metric against every reference and take the best match\.

### C\.2Answer Extraction, Normalization, and Evaluation Metrics

Before scoring, we extract the last complete<answer\>block from the model output as the predicted answer\. We normalize answers by lowercasing, removing ASCII punctuation, removing the English articlesa/an/the, and collapsing consecutive whitespace, in that order\.

During training, answer\-level F1 is used as the terminal task reward\. Specifically, we tokenize the normalized prediction and reference answer on whitespace, compute precision and recall from the multiset intersection of tokens, and take their harmonic mean\. At test time, exact match \(EM\) is the primary metric: a prediction receives 1 if its normalized form exactly matches any normalized reference answer and 0 otherwise\. The average results reported in the main paper are macro\-average EM scores across the seven datasets, with equal weight assigned to every dataset\.

## Appendix DCase Studies

This section uses two concrete cases to illustrate two key components of LAPO\. First, backward leave\-one\-turn attribution evaluates a target search turn while preserving all downstream interactions, allowing it to identify delayed attribution that becomes apparent only through subsequent reasoning and evidence integration\. Second, sign\-consistency gating filters normalized process signals that lack support from the raw counterfactual attribution, preventing the policy from rewarding or penalizing turns with no observed attribution\. Unless otherwise specified, all values are rounded to three decimal places\.

### D\.1Backward Attribution Identifies Delayed Cross\-Turn Contributions

Figure[6](https://arxiv.org/html/2607.13501#A4.F6)presents a representative example\. The target search turn retrieves evidence directly related to the gold answer, but introducing this evidence does not immediately increase the likelihood of generating that answer\. Forward attribution, which compares adjacent prefixes before and after the target turn, therefore assigns it negative attribution\. In contrast, LAPO evaluates the turn retrospectively within the complete trajectory and identifies the evidence’s positive role in subsequent reasoning and evidence integration\.

Example Trajectory

Attribution Comparison

Interpretation\.Forward attribution measures only the immediate change after the target turn is added and therefore assigns it negative attribution\. LAPO evaluates the target turn with the complete downstream interaction preserved and identifies its delayed positive attribution\.

Figure 6:A case in which backward leave\-one\-turn attribution identifies a delayed cross\-turn contribution\. The target search turn retrieves key evidence supporting the gold answerSam\-I\-am, but the likelihood of generating the gold answer temporarily decreases when the evidence is introduced, leading forward attribution to assign a negative value\. LAPO compares the full context with a counterfactual context in which the target turn is replaced in its entirety with\[DELETE\]\. By preserving the complete search and reasoning context after the target turn, LAPO identifies the evidence’s positive attribution within the current trajectory\.This case illustrates that an immediate change in answer confidence caused by a search turn is not equivalent to the turn’s overall attribution to the final reasoning outcome\. New evidence may temporarily perturb the model’s answer distribution but later play an important role in further search, entity disambiguation, and evidence integration\. Forward attribution based only on adjacent prefixes cannot observe this delayed effect and may therefore assign negative attribution to a search turn with longer\-term value\.

LAPO preserves all interactions after the target turn and constructs a counterfactual context by replacing that turn in its entirety with\[DELETE\]\. If the replacement lowers the gold\-answer likelihood, the target turn receives positive attribution under the current full context and policy\. Backward attribution can therefore evaluate cross\-turn dependencies retrospectively and reduce errors caused by local\-prefix comparisons when attribution is delayed\.

### D\.2Sign\-Consistency Gating Filters Directional Distortions from Normalization

Group normalization constructs a process signal from the contribution of each search turn relative to the group mean\. Consequently, even when a turn has zero raw counterfactual attribution, its normalized score may be positive or negative\. Using this normalized score directly for policy optimization could reward a search turn with no observed attribution or penalize another such turn\. Table[6](https://arxiv.org/html/2607.13501#A4.T6)illustrates both cases\.

Table 6:Sign\-consistency gating filters normalized process signals that lack support from the raw counterfactual attribution\.AtprocA\_\{t\}^\{\\mathrm\{proc\}\}denotes the gated process advantage,Atno​gate=z~tA\_\{t\}^\{\\mathrm\{no\\ gate\}\}=\\widetilde\{z\}\_\{t\}denotes the process advantage without gating, andλ​Atno​gate\\lambda A\_\{t\}^\{\\mathrm\{no\\ gate\}\}andλ​Atproc\\lambda A\_\{t\}^\{\\mathrm\{proc\}\}denote the auxiliary advantages actually added to the policy\-generated tokens of the target search turn without and with gating, respectively\.In the first case, the target turn has a raw answer\-likelihood gain of zero, but group normalization produces a positive score\. Without gating, the process advantage for this turn would be\+0\.550\+0\.550\. The second case exhibits the opposite pattern: the raw attribution is again zero, but normalization produces a negative score, yielding a process advantage of−0\.558\-0\.558without gating\. Neither update direction is supported by the raw counterfactual attribution\. In the table, the gated process advantages are zero in both cases; after multiplication by the process\-advantage weightλ=0\.5\\lambda=0\.5, the auxiliary advantages added to the policy\-generated tokens also remain zero\. Without gating, the corresponding auxiliary advantages would instead be0\.550×0\.5=0\.2750\.550\\times 0\.5=0\.275and−0\.558×0\.5=−0\.279\-0\.558\\times 0\.5=\-0\.279\.

To filter such directionally distorted process signals, LAPO defines sign\-consistency gating as

gi,t=𝕀​\[Δi,t​z~i,t\>0\],g\_\{i,t\}=\\mathbb\{I\}\\left\[\\Delta\_\{i,t\}\\widetilde\{z\}\_\{i,t\}\>0\\right\],\(17\)
whereΔi,t\\Delta\_\{i,t\}is the raw answer\-likelihood gain for search turnttin trajectoryii, andz~i,t\\widetilde\{z\}\_\{i,t\}is the process signal after robust scaling, the tanh bounded transformation, and group normalization, in that order\. The gated process advantage is

Ai,tproc=gi,t​z~i,t\.A\_\{i,t\}^\{\\mathrm\{proc\}\}=g\_\{i,t\}\\widetilde\{z\}\_\{i,t\}\.\(18\)
The auxiliary advantage actually added to the policy\-generated tokens of the search turn isλ​Ai,tproc\\lambda A\_\{i,t\}^\{\\mathrm\{proc\}\}, consistent with Eq\. \([14](https://arxiv.org/html/2607.13501#S3.E14)\)\.

Thus, the auxiliary process advantage is retained only when the raw counterfactual attribution and normalized process signal have the same nonzero direction\. When the raw attribution is zero or normalization reverses the sign of the signal, the gate is zero and the corresponding turn receives no auxiliary process update\.

Importantly, sign\-consistency gating filters only fine\-grained process advantages that lack counterfactual support\. It does not remove the trajectory\-level outcome advantage or alter the outcome supervision received by the final\-answer turn\.