Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents

# Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents
Source: [https://arxiv.org/html/2608.12764](https://arxiv.org/html/2608.12764)
Chuqiao KuangAffiliation:Huawei Technologies Ltd\.\[2pt\]Tianyi ZhuangAffiliation:Huawei Technologies Ltd\.\[2pt\]Xiaoguang LiAffiliation:Huawei Technologies Ltd\.\[2pt\]\[4pt\] The Hong Kong University of ScienceTechnology

###### Abstract

Deep search agents operate over trajectories spanning dozens of steps, yet standard reinforcement learning provides only a single outcome reward per trajectory — supervision that is far too sparse for effective credit assignment\. On\-policy self\-distillation \(OPSD\) addresses this by using the model’s own logits as dense token\-level teachers, but extending it to search agents introduces a fundamental tension: the teacher, having access to privileged information such as the correct answer, produces a distribution that differs systematically from the student’s exploration\-based reasoning, and naive distillation causes the student to inherit this information asymmetry rather than learn better search strategies\. We resolve this tension through two contributions\. First, we construct Evidence Anchors — concise, step\-level evidence snippets extracted from the web — as privileged information that captures key reasoning steps without revealing the entire answer path\. Second, we propose Step\-Level Self\-Distilled Policy Optimization \(SSPO\), which converts teacher–student disagreement into step\-level advantage weights within GRPO, applied exclusively to incorrect trajectories\. This design decouples what to update from how much to update: the outcome reward determines the direction of policy change, while the teacher modulates its magnitude at each step\. Correct trajectories are left untouched, preserving their diversity\. On Qwen3\-8B, SSPO consistently outperforms GRPO across BrowseComp, GAIA, and FRAMES, surpassing or matching GRPO trained with twice as many gradient steps while adding only about 5% overhead per step from a single additional forward pass\.111Code is available at[https://github\.com/hkust\-nlp/SSPO](https://github.com/hkust-nlp/SSPO)\. Correspondence to:hwuds@connect\.ust\.hk\.

Figure 1:Comparison of performance across training methods\. Left:After introducing step\-level self\-distilled advantage weights \(SSPO\), performance improves substantially faster during training, surpassing GRPO trained for 200 steps after only 100 steps\.Right:We report results for the cold\-start model, GRPO after 100 and 200 steps, and SSPO after 100 steps\.## 1Introduction

Recent Large Language Model paradigms have rapidly evolved from generating appropriate responses to solving real\-world tasks through tool use across domains such as coding, office productivity, finance, and research[18](https://arxiv.org/html/2608.12764#bib.bib9);[33](https://arxiv.org/html/2608.12764#bib.bib10);[41](https://arxiv.org/html/2608.12764#bib.bib11);[4](https://arxiv.org/html/2608.12764#bib.bib43);[2](https://arxiv.org/html/2608.12764#bib.bib41);[7](https://arxiv.org/html/2608.12764#bib.bib40);[34](https://arxiv.org/html/2608.12764#bib.bib34);[30](https://arxiv.org/html/2608.12764#bib.bib35);[9](https://arxiv.org/html/2608.12764#bib.bib36)\. Among these applications, web search agents must answer vague, underspecified queries by actively exploring the open web — issuing searches, inspecting pages, and refining their reasoning over trajectories that often span dozens of steps[43](https://arxiv.org/html/2608.12764#bib.bib14)\. Training agents to perform such long\-horizon search reliably requires post\-training beyond Supervised Fine\-Tuning \(SFT\), making reinforcement learning \(RL\) the dominant approach[26](https://arxiv.org/html/2608.12764#bib.bib1);[20](https://arxiv.org/html/2608.12764#bib.bib4);[19](https://arxiv.org/html/2608.12764#bib.bib5);[40](https://arxiv.org/html/2608.12764#bib.bib6)\. However, RL in this setting faces a fundamental challenge: trajectories containing 20\+ steps receive only a single binary reward\. With supervision this sparse, the model receives little guidance on which steps led to success or failure, making credit assignment the central bottleneck for deep search agents\.

A natural way to address sparse rewards is to introduce denser supervision\. SFT provides token\-level feedback through teacher trajectories, but its Off\-Policy nature introduces exposure bias and limits generalization to unseen queries[36](https://arxiv.org/html/2608.12764#bib.bib19);[6](https://arxiv.org/html/2608.12764#bib.bib20);[48](https://arxiv.org/html/2608.12764#bib.bib21);[38](https://arxiv.org/html/2608.12764#bib.bib29)\. On\-Policy Distillation \(OPD\)[27](https://arxiv.org/html/2608.12764#bib.bib26);[39](https://arxiv.org/html/2608.12764#bib.bib44);[23](https://arxiv.org/html/2608.12764#bib.bib45)combines both benefits: the student performs on\-policy rollouts while a teacher model provides token\-level scoring\. More recent work extends this to self\-distillation, where the teacher is constructed from the student itself — using privileged information such as reference solutions or environment feedback as a prefix[63](https://arxiv.org/html/2608.12764#bib.bib27);[54](https://arxiv.org/html/2608.12764#bib.bib25);[13](https://arxiv.org/html/2608.12764#bib.bib28);[38](https://arxiv.org/html/2608.12764#bib.bib29);[35](https://arxiv.org/html/2608.12764#bib.bib31);[17](https://arxiv.org/html/2608.12764#bib.bib32), eliminating the need for a separate teacher model\.

Despite this progress, existing self\-distillation methods have been developed and evaluated primarily on single\-turn reasoning tasks such as math and code\. Two properties make self\-distillation feasible in those settings: the gap between teacher and student behavior is modest, and a natural form of privileged information — reference solutions or execution feedback — already exists\. Extending self\-distillation to multi\-turn search agents breaks both assumptions\. The information asymmetry becomes extreme: the teacher, seeing curated evidence and the answer, collapses to 3\.5 tool calls per trajectory, while the student, navigating the open web, requires 17\.7 \(Table[1](https://arxiv.org/html/2608.12764#S3.T1)\)\. Directly distilling this collapsed distribution causes the student to mimic the teacher’s brevity; when privileged information is unavailable, it abandons tool use prematurely rather than learning better search strategies\. An equally serious problem compounds the first: unlike math or code, open\-ended web search has no ready\-made privileged information\. A form of supervision that is both informative to the teacher and safe to distill to the student does not naturally exist, and constructing one is itself an open challenge\.

Our key insight is that search supervision is naturally defined at the level of information\-seeking actions rather than token generation\. We therefore align privileged supervision with the structure of search through two design choices\. First, we constructEvidence Anchors: compact, step\-level evidence snippets extracted from the web that capture information needed to answer a question without revealing the answer path\. Each question is associated with a small set of such anchors, serving as prefixes for the self\-teacher\. Second, we proposeStep\-Level Self\-Distilled Policy Optimization\(SSPO\)\. Rather than minimizing divergence between teacher and student distributions, SSPO converts teacher–student disagreement into step\-level advantage weights within GRPO[37](https://arxiv.org/html/2608.12764#bib.bib22);[11](https://arxiv.org/html/2608.12764#bib.bib24), applied only to incorrect trajectories\. This design decouples update direction from update magnitude: the outcome reward determines whether to reinforce or suppress a trajectory, while the teacher modulates how much each step contributes\. Correct trajectories are left untouched, preserving behavioral diversity\.

We evaluate SSPO on Qwen3\-8B[53](https://arxiv.org/html/2608.12764#bib.bib17)across three information\-seeking benchmarks — BrowseComp[43](https://arxiv.org/html/2608.12764#bib.bib14), GAIA[29](https://arxiv.org/html/2608.12764#bib.bib15), and FRAMES[16](https://arxiv.org/html/2608.12764#bib.bib16)\. SSPO consistently outperforms GRPO across all benchmarks, and notably surpasses GRPO trained with twice as many gradient steps while adding only about 5% computational overhead per training step\. Our ablation studies further validate two key design decisions\. First, directly matching the teacher distribution collapses the student’s tool use and underperforms even GRPO, confirming that decoupling update magnitude from direction is essential\. Second, step\-level advantage weights consistently outperform token\-level counterparts, whose fine\-grained signals are misaligned with the natural unit of information\-seeking actions\.

In summary, our contributions are:\(a\)We show that extending self\-distillation to multi\-turn search breaks the assumptions underlying single\-turn self\-distillation, requiring a fundamentally different design\.\(b\)We introduce Evidence Anchors: compact, step\-level evidence snippets as privileged information for open\-ended web search, aligned with the natural unit of information\-seeking actions\.\(c\)We propose SSPO, which uses step\-level self\-distilled signals as advantage weights rather than optimization targets, achieving superior sample efficiency over GRPO\.

## 2Preliminaries

### 2\.1Search Agent

We build our search agent based on the ReAct paradigm[56](https://arxiv.org/html/2608.12764#bib.bib18);[50](https://arxiv.org/html/2608.12764#bib.bib23), where the model interleaves reasoning and actions until producing the final answer\. Our action space includes two tools connected to the real web:searchandbrowse, their details could be found in Appendix[C](https://arxiv.org/html/2608.12764#A3)\. After receiving a user query, the agent attempts to provide an accurate answer through multipleThought Action Observationsteps\. An agent withTTsteps can be formulated as follows:

HT=\{𝚚u​s​e​r,𝚝1,𝚊1,𝚘1,…,𝚝T−1,𝚊T−1,𝚘T−1,𝚝T\}H\_\{T\}=\\\{\\mathtt\{q\}\_\{user\},\\mathtt\{t\}\_\{1\},\\mathtt\{a\}\_\{1\},\\mathtt\{o\}\_\{1\},\\ldots,\\mathtt\{t\}\_\{T\-1\},\\mathtt\{a\}\_\{T\-1\},\\mathtt\{o\}\_\{T\-1\},\\mathtt\{t\}\_\{T\}\\\}At time stepτ\\tau, the agent generates a thought𝚝τ\\mathtt\{t\}\_\{\\tau\}and a tool call𝚊τ\\mathtt\{a\}\_\{\\tau\}\(if it is not the final step\) conditioned on the full interaction history, i\.e\.,𝚝τ,𝚊τ∼π⁡\(𝚝,𝚊∣Hτ−1\)\\mathtt\{t\}\_\{\\tau\},\\mathtt\{a\}\_\{\\tau\}\\sim\\pi\(\\mathtt\{t\},\\mathtt\{a\}\\mid H\_\{\\tau\-1\}\)\. The output of the tool is treated as the observation𝚘τ\\mathtt\{o\}\_\{\\tau\}for this step\.

### 2\.2On\-Policy Learning

#### Agentic Reinforcement Learning\.

Deep search, similar to math and logic tasks[37](https://arxiv.org/html/2608.12764#bib.bib22);[50](https://arxiv.org/html/2608.12764#bib.bib23), is typically formulated under the Reinforcement Learning with Verifiable Rewards \(RLVR\) setting\. In this setting, a correctness reward can be assigned by evaluating whether the agent’s prediction is semantically consistent with the ground truth:Rcorrect=is\_equal\(pred, ground\_truth\)∈\{0,1\}R\_\{\\text\{correct\}\}=\\text\{\{is\\\_equal\(pred, ground\\\_truth\)\}\}\\in\\\{0,1\\\}\. In addition, training often incorporates a format rewardRformatR\_\{\\text\{format\}\}to encourage adherence to the ReAct paradigm[56](https://arxiv.org/html/2608.12764#bib.bib18)\. The final training reward is defined as:

Rf​i​n​a​l=Rc​o​r​r​e​c​t\+0\.2×Rf​o​r​m​a​tR\_\{final\}=R\_\{correct\}\+0\.2\\times R\_\{format\}\(1\)Similar to other works[26](https://arxiv.org/html/2608.12764#bib.bib1);[20](https://arxiv.org/html/2608.12764#bib.bib4);[19](https://arxiv.org/html/2608.12764#bib.bib5), we use the Group Relative Policy Optimization \(GRPO\) algorithm to update the model[37](https://arxiv.org/html/2608.12764#bib.bib22);[11](https://arxiv.org/html/2608.12764#bib.bib24)\. It addresses the challenge of estimating baselines in policy gradient methods by comparing responses within a group\. Specifically, for each questionqq, GRPO samplesGGresponses\{y\(1\),…,y\(G\)\}\\\{y^\{\(1\)\},\\ldots,y^\{\(G\)\}\\\}from the current policy and normalizes the rewards to obtain a sequence\-level advantage estimate:

A\(i\)=Rf​i​n​a​l​\(q,y\(i\)\)−μGσG,A^\{\(i\)\}=\\frac\{R\_\{final\}\(q,y^\{\(i\)\}\)\-\\mu\_\{G\}\}\{\\sigma\_\{G\}\},\(2\)whereμG\\mu\_\{G\}andσG\\sigma\_\{G\}denote the mean and standard deviation of rewards within the group, respectively\. The policy is then updated by maximizing a clipped surrogate objective:

ℒGRPO​\(θ\)=𝔼q∼𝒟​\[1G​∑i=1G1\|y\(i\)\|​∑t=1\|y\(i\)\|min⁡\(ρt\(i\)​At\(i\),clip​\(ρt\(i\),1−ϵ,1\+ϵ\)​At\(i\)\)\]\\mathcal\{L\}\_\{\\text\{GRPO\}\}\(\\theta\)=\\mathbb\{E\}\_\{q\\sim\\mathcal\{D\}\}\\left\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{\|y^\{\(i\)\}\|\}\\sum\_\{t=1\}^\{\|y^\{\(i\)\}\|\}\\min\\\!\\left\(\\rho\_\{t\}^\{\(i\)\}A^\{\(i\)\}\_\{t\},\\ \\text\{clip\}\\\!\\left\(\\rho\_\{t\}^\{\(i\)\},1\-\\epsilon,1\+\\epsilon\\right\)A^\{\(i\)\}\_\{t\}\\right\)\\right\]\(3\)whereρt\(i\)=πθ​\(yt\(i\)∣q\)/πθold​\(yt\(i\)∣q\)\\rho\_\{t\}^\{\(i\)\}=\\pi\_\{\\theta\}\(y\_\{t\}^\{\(i\)\}\\mid q\)/\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(y\_\{t\}^\{\(i\)\}\\mid q\)is the per\-token importance sampling ratio, andϵ\\epsilonis the clipping threshold that constrains the policy update to a trust region\. All tokens in the trajectory share the sequence\-level advantageAt\(i\)=A\(i\)A\_\{t\}^\{\(i\)\}=A^\{\(i\)\}\. To improve training efficiency, we adopt the duplication strategy proposed in WebSailor[20](https://arxiv.org/html/2608.12764#bib.bib4), where groups with non\-zero variance within the batch are duplicated to replace zero\-variance groups, as the latter provide no useful training signal\.

#### On\-Policy Self Distillation \(OPSD\)\.

Although GRPO has become the default algorithm for training agents, it has a key limitation in settings with long reasoning chains: relying solely on outcomes lacks process\-level supervision\. This is especially problematic for incorrect responses, where a small error in the final few tokens may lead to an incorrect answer, yet all preceding tokens receive the same level of penalty\. Recent works, including OPSD[54](https://arxiv.org/html/2608.12764#bib.bib25);[63](https://arxiv.org/html/2608.12764#bib.bib27);[13](https://arxiv.org/html/2608.12764#bib.bib28);[38](https://arxiv.org/html/2608.12764#bib.bib29);[57](https://arxiv.org/html/2608.12764#bib.bib30);[35](https://arxiv.org/html/2608.12764#bib.bib31);[17](https://arxiv.org/html/2608.12764#bib.bib32), attempt to alleviate the lack of process\-level supervision in long CoT[44](https://arxiv.org/html/2608.12764#bib.bib12)without relying on other teacher models[27](https://arxiv.org/html/2608.12764#bib.bib26)\. These methods provide the model with privileged informationcp​r​i​v​i​l​e​g​e​dc\_\{privileged\}as a form of self\-teaching, such as reference solutions, environment feedback, or better rollouts, thereby enabling token\-level supervision for the student\. The training objective is formulated as follows:

ℒOPSD\(θ\)=𝔼q∼𝒟,y∼PS\(⋅∣q\)\[1\|y\|∑t=1\|y\|𝒟\(PT∣∣PS\)\]\\mathcal\{L\}\_\{\\text\{OPSD\}\}\(\\theta\)=\\mathbb\{E\}\_\{q\\sim\\mathcal\{D\},y\\sim P\_\{S\}\(\\cdot\\mid q\)\}\\left\[\\frac\{1\}\{\|y\|\}\\sum\_\{t=1\}^\{\|y\|\}\\mathcal\{D\}\(P\_\{T\}\\mid\\mid P\_\{S\}\)\\right\]\(4\)where𝒟\\mathcal\{D\}denotes a divergence measure \(e\.g\., KL divergence\), andθ\\thetarepresents the parameters of the student model\. The teacher and student distributions can be expressed as:

Teacher:PT\(y∣q,y<t\)=πθ^\(⋅∣q,y<t,cp​r​i​v​i​l​e​g​e​d\)Student:PS\(y∣q,y<t\)=πθ\(⋅∣q,y<t\)\\displaystyle\\text\{Teacher\}:P\_\{T\}\(y\\mid q,y\_\{<t\}\)=\\pi\_\{\\hat\{\\theta\}\}\(\\cdot\\mid q,y\_\{<t\},c\_\{privileged\}\)\\,\\,\\text\{Student\}:P\_\{S\}\(y\\mid q,y\_\{<t\}\)=\\pi\_\{\\theta\}\(\\cdot\\mid q,y\_\{<t\}\)\(5\)For training stability, the teacher model’s parametersθ^\\hat\{\\theta\}are typically not updated through gradient training; common choices include keeping them fixed, updating them via EMA, or synchronizing them with the student model at regular step intervals\.

Although significant progress has been made, existing work is largely limited to relatively simple single\-turn reasoning tasks\. Our work explores how to leverage self\-distillation signals to provide more fine\-grained supervision for training long\-horizon search agents\.

## 3Step\-Level Self\-Distilled Policy Optimization

Deep search agents operate through information\-seeking actions whose utility emerges only at the level of complete search steps\. A retrieval action is valuable not because of individual token choices, but because it effectively localizes uncertainty, gathers relevant evidence, and shapes subsequent exploration\. Motivated by this observation, both our privileged information and supervision are aligned with the structure of search steps\.

### 3\.1Privileged Information: Evidence Anchors

In single\-turn settings, constructing privileged information is relatively straightforward\. For mathematical tasks, many open\-source datasets provide reference solutions[63](https://arxiv.org/html/2608.12764#bib.bib27);[10](https://arxiv.org/html/2608.12764#bib.bib33); for tool use and coding, error messages can be directly obtained[13](https://arxiv.org/html/2608.12764#bib.bib28); even in the absence of explicit feedback, the highest\-scoring rollout among multiple candidates can be selected as privileged teacher information[13](https://arxiv.org/html/2608.12764#bib.bib28)\. However, for BrowseComp\-style tasks[43](https://arxiv.org/html/2608.12764#bib.bib14), models often require context windows spanning tens or even hundreds of thousands of tokens\. In such settings, providing the best rollout as privileged information is impractical, while supplying only the final answer is insufficient to effectively guide multi\-step reasoning and action\. What is needed is privileged information structured around individual search actions — capable of signaling whether each step retrieves the right evidence\.

![Refer to caption](https://arxiv.org/html/2608.12764v1/anchors.png)Figure 2:An example question with three evidence anchors\. The differently colored evidence anchors correspond to the highlighted conditions in the question\.To construct such privileged information, we construct key pieces of evidence for each QA pair asEvidence Anchors\. Specifically, we prompt a SOTA LLM to identify as many pieces of evidence as possible that support the ground\-truth answer for each question\. The prompt we use can be seen in Figure[12](https://arxiv.org/html/2608.12764#A7.F12)\. We collect evidence anchors for over 6,000 QA pairs, with an average of 5\.24 anchors per question\. Figure[2](https://arxiv.org/html/2608.12764#S3.F2)presents an example from our training data\. Detailed statistics and correctness validation are provided in Appendix[G](https://arxiv.org/html/2608.12764#A7)\. These privileged signals are incorporated into the teacher model’s prompt only for incorrect trajectories, as illustrated in Figure[3](https://arxiv.org/html/2608.12764#S3.F3)\. In practice, in addition to evidence anchors, we explicitly provide the teacher with the incorrect answers generated from the student’s rollouts, preventing the teacher from repeating the same mistakes in its output distribution\.

[⬇](data:text/plain;base64,UXVlc3Rpb246IHtxdWVzdGlvbn0KWW91IHByZXZpb3VzbHkgYXR0ZW1wdGVkIHRvIG9idGFpbiB0aGUgYW5zd2VyIHtyb2xsb3V0IGFuc3dlcn0sIGJ1dCBpdCBpcyBpbmNvcnJlY3QuClRoZSBmb2xsb3dpbmcgYXJlIHNvbWUgZXZpZGVuY2UgYW5jaG9ycyB0aGF0IG1heSBoZWxwIHlvdToKLSB7c291cmNlIHRpdGxlfToge2V4cGxhbmF0aW9ufQouLi4KW0lNUE9SVEFOVF06IERvIG5vdCBhbnN3ZXIgZGlyZWN0bHkgZnJvbSB0aGUgYW5jaG9ycy4KQ29ycmVjdGx5IHNvbHZlIHRoZSBvcmlnaW5hbCBxdWVzdGlvbjo=)Question:\{question\}Youpreviouslyattemptedtoobtaintheanswer\{rolloutanswer\},butitisincorrect\.Thefollowingaresomeevidenceanchorsthatmayhelpyou:\-\{sourcetitle\}:\{explanation\}\.\.\.\[IMPORTANT\]:Donotanswerdirectlyfromtheanchors\.Correctlysolvetheoriginalquestion:

Figure 3:Teacher prompt template incorporating evidence anchors and previously generated incorrect answers from the student model\.
### 3\.2Step\-Level Self\-Distilled Advantage Weights

Pass@1\#TurnQuestion4617\.7\+ Anchors823\.5\+ Answer962\.4Table 1:Comparison of accuracy and average steps before and after providing privileged information\.As discussed in prior work[54](https://arxiv.org/html/2608.12764#bib.bib25), directly optimizing Equation[4](https://arxiv.org/html/2608.12764#S2.E4)can lead to privileged information leakage, which may degrade model performance in the later stages of training\. This issue becomes more pronounced in multi\-turn settings, where shortcuts introduced by privileged information affect not only the reasoning process itself but also significantly reduce the number of tool calls required for information gathering\. To quantify this effect, we sample 50 examples from the training set and compare the cold\-start model’s accuracy and trajectory length under different information conditions\. As shown in Table[1](https://arxiv.org/html/2608.12764#S3.T1), providing evidence anchors in addition to the question improves accuracy from 46 to 82, while reducing the average number of steps from 17\.7 to 3\.5\. When the final answer is further revealed, accuracy increases to 96, and the average number of steps decreases to 2\.4\. This substantial discrepancy suggests that the teacher’s distribution with privileged information is not an appropriate optimization target for the student\. Instead, using self\-distillation signals as weighting factors, rather than as gradient directions, helps mitigate this issue[54](https://arxiv.org/html/2608.12764#bib.bib25)\. Under this formulation, the update direction remains determined by environment rewards, while the teacher’s privileged distribution only modulates the update magnitude, preventing the student from directly fitting an information\-asymmetric target\. Beyond privileged\-information leakage, SRPO[17](https://arxiv.org/html/2608.12764#bib.bib32)further shows that applying OPSD signals only to incorrect trajectories, while preserving the original GRPO objective for correct ones, yields better performance\. Applying distillation to already\-correct trajectories introduces optimization ambiguity: the model is pushed toward a specific teacher distribution despite having already solved the task, unnecessarily suppressing the diversity of correct solutions\. Together, these two insights motivate our method design: we use self\-distillation signals as step\-level advantage weights rather than optimization targets, and restrict their application to incorrect search\-agent trajectories\.

For each student\-generated stepτ\\tauin trajectoryy\(i\)y^\{\(i\)\}, its thinking tokens𝚝τ\\mathtt\{t\}\_\{\\tau\}and tool\-calling tokens𝚊τ\\mathtt\{a\}\_\{\\tau\}jointly constitute one action in the agentic MDP[60](https://arxiv.org/html/2608.12764#bib.bib64)\. Under autoregressive factorization, the teacher and student assign the following conditional joint probabilities to this complete step:

PT​\(𝚝τ,𝚊τ\)=∏k∈τπθ^​\(yk∣cprivileged,q,y<k\),PS​\(𝚝τ,𝚊τ\)=∏k∈τπθ​\(yk∣q,y<k\)\.\\small P\_\{T\}\(\\mathtt\{t\}\_\{\\tau\},\\mathtt\{a\}\_\{\\tau\}\)=\\prod\_\{k\\in\\tau\}\\pi\_\{\\hat\{\\theta\}\}\(y\_\{k\}\\mid c\_\{\\text\{privileged\}\},q,y\_\{<k\}\),\\hskip 18\.49988ptP\_\{S\}\(\\mathtt\{t\}\_\{\\tau\},\\mathtt\{a\}\_\{\\tau\}\)=\\prod\_\{k\\in\\tau\}\\pi\_\{\\theta\}\(y\_\{k\}\\mid q,y\_\{<k\}\)\.\(6\)Equation[6](https://arxiv.org/html/2608.12764#S3.E6)follows standard autoregressive factorization, under which the joint probability of a step is the product of its token\-level conditional probabilities\. We compute both likelihoods by teacher\-forcing the same student\-generated step under the corresponding teacher and student configurations:PTP\_\{T\}uses the privileged context and teacher parameters, whereasPSP\_\{S\}uses the original context and current policy parameters\.

We then define the privileged\-information gain as the log joint\-likelihood ratio of this step action:

Δτs​t​e​p=sg​\(log⁡PT​\(𝚝τ,𝚊τ\)PS​\(𝚝τ,𝚊τ\)\),\\Delta\_\{\\tau\}^\{step\}=\\text\{\{sg\}\}\\left\(\\log\\frac\{P\_\{T\}\(\\mathtt\{t\}\_\{\\tau\},\\mathtt\{a\}\_\{\\tau\}\)\}\{P\_\{S\}\(\\mathtt\{t\}\_\{\\tau\},\\mathtt\{a\}\_\{\\tau\}\)\}\\right\),\(7\)wheresgdenotes stop\-gradient\. For tokensyty\_\{t\}belonging to stepτ\\tau\(t∈τt\\in\\tau\), they share the same step\-level advantage weight:

wt​=t∈τ​wτs​t​e​p=min⁡\(exp⁡\(sign​\(A\(i\)\)⋅Δτs​t​e​p\),1\+ϵ\)w\_\{t\}\\overset\{t\\in\\tau\}\{=\}w\_\{\\tau\}^\{step\}=\\min\(\\exp\(\\text\{sign\}\(A^\{\(i\)\}\)\\cdot\\Delta\_\{\\tau\}^\{step\}\),1\+\\epsilon\)\(8\)whereϵ\\epsilonis a positive hyperparameter, and themin\\minoperation prevents extreme gradient magnitudes\. It is important to clarify that trajectories withRf​i​n​a​l<1R\_\{final\}<1in Equation[1](https://arxiv.org/html/2608.12764#S2.E1)are treated as incorrect trajectories, since they fail to produce a factually equivalent final answer222An incorrect trajectory may still have a positive group\-relative advantage, e\.g\., when it receives the format reward and outperforms other rollouts in the same group\. Hence, Equation[8](https://arxiv.org/html/2608.12764#S3.E8)explicitly considers the advantage sign\.\. Finally, we replace the advantage term in Equation[3](https://arxiv.org/html/2608.12764#S2.E3)withA^t\(i\)=wt​At\(i\)​if​Rf​i​n​a​l<1​else​A\(i\)\\hat\{A\}^\{\(i\)\}\_\{t\}=w\_\{t\}A\_\{t\}^\{\(i\)\}\\text\{ \{if\} \}R\_\{final\}<1\\text\{ \{else\} \}A^\{\(i\)\}\.

Δτstep\>0\\Delta^\{\\text\{step\}\}\_\{\\tau\}\>0Δτstep<0\\Delta^\{\\text\{step\}\}\_\{\\tau\}<0A\(i\)\>0A^\{\(i\)\}\>0w\>1w\>1Reward amplifiedw<1w<1Reward dampenedA\(i\)<0A^\{\(i\)\}<0w<1w<1Penalty reducedw\>1w\>1Penalty amplified

Table 2:Step\-level advantage weight modulation\.As summarized in Table[2](https://arxiv.org/html/2608.12764#S3.T2), the weightwwmodulates each token’s contribution along two axes: the sign of the trajectory\-level advantageA\(i\)A^\{\(i\)\}and the step\-level teacher–student agreementΔτstep\\Delta^\{\\text\{step\}\}\_\{\\tau\}\. For incorrect trajectories withA\(i\)<0A^\{\(i\)\}<0, which represent the typical case for failed rollouts, we use the weightmin⁡\(PS/PT,1\+ϵ\)\\min\(P\_\{S\}/P\_\{T\},\\,1\+\\epsilon\)\. When the student is overconfident on a step rejected by the teacher \(PS→1,PT→0P\_\{S\}\\to 1,\\,P\_\{T\}\\to 0, i\.e\.,Δτstep<0\\Delta^\{\\text\{step\}\}\_\{\\tau\}<0\),w∈\(1,1\+ϵ\)w\\in\(1,\\,1\+\\epsilon\)amplifies the penalty, discouraging repeated mistakes\. Conversely, when the student is uncertain on a step endorsed by the teacher \(PS→0,PT→1P\_\{S\}\\to 0,\\,P\_\{T\}\\to 1, i\.e\.,Δτstep\>0\\Delta^\{\\text\{step\}\}\_\{\\tau\}\>0\),w∈\(0,1\)w\\in\(0,\\,1\)reduces the penalty, preserving valid intermediate reasoning within failed rollouts\. For incorrect trajectories withA\(i\)\>0A^\{\(i\)\}\>0, which may occur on challenging questions when a trajectory receives format rewards despite producing an incorrect final answer, we adopt the symmetric weightmin⁡\(PT/PS,1\+ϵ\)\\min\(P\_\{T\}/P\_\{S\},\\,1\+\\epsilon\): a confident teacher paired with an uncertain student amplifies the reward, while a confident student paired with an uncertain teacher dampens it\. Across all cases, SSPO adjusts the*magnitude*of updates at step\-level resolution without altering the trajectory\-level update*direction*\.

## 4Experiment

### 4\.1Experimental Setup

#### Training Data\.

Before On\-Policy learning, we first followed the WebExplorer[26](https://arxiv.org/html/2608.12764#bib.bib1)pipeline to collect approximately 4,000 English reasoning trajectories with correct final answers and without obvious tool\-calling errors, which we used for cold\-start initialization\. Further details of these trajectories are provided in Appendix[D](https://arxiv.org/html/2608.12764#A4)\. For On\-Policy learning, we sampled around 6,000 English QA pairs from the open\-source DeepForge dataset[64](https://arxiv.org/html/2608.12764#bib.bib2), using a difficulty ratio of 1\.5:3\.5:3\.5:1\.5 across the four levels\. We further employed DeepSeek\-V3\.2[8](https://arxiv.org/html/2608.12764#bib.bib3)to construct evidence anchors for each QA pair\.

#### Benchmarks\.

BrowseComp[43](https://arxiv.org/html/2608.12764#bib.bib14)is a highly challenging information\-retrieval benchmark introduced by OpenAI\. GAIA[29](https://arxiv.org/html/2608.12764#bib.bib15)is a widely used benchmark for general AI assistants; following WebThinker[22](https://arxiv.org/html/2608.12764#bib.bib7), we evaluate on its text\-only subset\. FRAMES[16](https://arxiv.org/html/2608.12764#bib.bib16), introduced by Google, is used to assess factual accuracy and reasoning capability\. Due to API cost constraints, we evaluate intermediate checkpoints on subsets of BrowseComp and FRAMES, denoted as \{BC, FRAMES\}\-Sub\. We adopt the Avg@4 metric to reduce variance, with the temperature set to 0\.6 and top\-ppset to 0\.95\.

More experimental details could be found in Appendix[B](https://arxiv.org/html/2608.12764#A2)\.

### 4\.2Experimental Results

#### Main Results\.

Table[3](https://arxiv.org/html/2608.12764#S4.T3)presents the performance of representative small\-scale search agents, alongside our models trained with cold\-start initialization, GRPO, and SSPO, as well as the average number of turns required to solve each problem\. Despite differences in search scaffolding and the fact that existing agents of comparable size are typically trained on substantially larger datasets, our models achieve competitive—often superior—performance, validating the effectiveness of our training pipeline\. Our primary focus, however, is on the performance gains that SSPO brings over GRPO within the same scaffold\. Models trained with SSPO consistently outperform their GRPO counterparts by a significant margin\. Using the average score across three benchmarks as a representative metric, GRPO achieves a \+2\.4 improvement over the cold\-start baseline, whereas SSPO delivers a markedly larger gain of \+4\.8\. Importantly, these improvements do not come at the expense of reasoning efficiency\. The SSPO\-trained model requires only a marginally higher number of turns compared to GRPO \(20\.5 vs\. 20\.0\), indicating that its performance gains are achieved with minimal additional computational cost\. As shown in Figure[1](https://arxiv.org/html/2608.12764#S0.F1)and Figure[10](https://arxiv.org/html/2608.12764#A5.F10), SSPO demonstrates substantially higher learning efficiency, consistently outperforming GRPO across all three benchmarks at the same number of training steps\. To further examine the impact of process supervision, we double the number of training steps for GRPO; notably, SSPO trained for 100 steps already surpasses GRPO trained for 200 steps, highlighting the superior sample efficiency enabled by process\-level supervision\. Beyond benchmark scores, these gains come with negligible training cost — only about 5% additional overhead from teacher scoring; detailed wall\-clock and compute statistics are provided in Appendix[E](https://arxiv.org/html/2608.12764#A5)\.

ModelBrowseComp[43](https://arxiv.org/html/2608.12764#bib.bib14)GAIA[29](https://arxiv.org/html/2608.12764#bib.bib15)Frames[16](https://arxiv.org/html/2608.12764#bib.bib16)AverageSmall Size Search Agent \(<10B\)WebSailor\-7B[20](https://arxiv.org/html/2608.12764#bib.bib4)6\.737\.9––MiroThinker\-8B\-DPO\-v0\.1[42](https://arxiv.org/html/2608.12764#bib.bib61)8\.746\.664\.439\.9AFM\-WebAgent\-7B\(RL\)[21](https://arxiv.org/html/2608.12764#bib.bib63)5\.840\.8––DeepDive\-9B\(RL\)[28](https://arxiv.org/html/2608.12764#bib.bib62)6\.3–––WebExplorer\-8B\(RL\)[26](https://arxiv.org/html/2608.12764#bib.bib1)15\.750\.075\.747\.1OffSeeker\-8B\(DPO\)[64](https://arxiv.org/html/2608.12764#bib.bib2)12\.851\.5––Our Models \(based on Qwen3\-8B\)Our\-8B\(Cold\-Start\)11\.7\(37\.7\)44\.8\(14\.9\)67\.2\(7\.2\)41\.2\(19\.9\)Our\-8B\(GRPO\)13\.6\(38\.0\)47\.3\(14\.5\)69\.8\(7\.6\)43\.6\(20\.0\)Our\-8B\(SSPO\)15\.7\(39\.4\)49\.3\(14\.5\)73\.0\(7\.7\)46\.0\(20\.5\)

Table 3:Main results on the BrowseComp, GAIA, and Frames benchmarks\.Subscripts indicate the average number of search turns required per problem\.Underlinedvalues denote the best overall performance in each column, whileboldvalues highlight the best results among our models\. Results for other search agents are reported from their respective original publications\.Figure 4:Comparison of training dynamics between GRPO and SSPO \(EMA Smoothed\)\.
#### Training Dynamics\.

Beyond benchmark scores, we analyze the training dynamics in Figure[4](https://arxiv.org/html/2608.12764#S4.F4)\. After an initial warm\-up phase of approximately 30–50 steps, SSPO achieves higher rewards than GRPO and maintains this advantage throughout training, indicating more effective optimization in our experiments\. Notably, the entropy curves of both methods remain closely aligned, suggesting that the performance gains of SSPO are unlikely to stem primarily from increased exploration, but rather from more effective policy updates\. This observation is further supported by the gradient norm curves: GRPO exhibits a steadily decreasing gradient norm, indicating early saturation of policy updates, whereas SSPO maintains a relatively higher gradient norm throughout training, reflecting more sustained learning signals\. Although SSPO shows a transient spike in gradient norm around step 50, this spike quickly subsides and does not appear to harm training stability, as evidenced by the continued improvement in reward\. Overall, these results suggest that SSPO improves upon GRPO mainly by enabling more sustained and effective parameter updates, rather than by substantially altering the exploration–exploitation balance\.

### 4\.3Ablation Study

We organize the ablation around three questions: how direct distillation compares with advantage weighting, whether the weighting signal should be defined at token or step level, and what roles the two privileged\-information inputs play\. Unless otherwise stated, all results are measured on BC\-Sub after 50 training steps\.

MethodUse of self\-distillation signalGranularityEvidence AnchorsBC\-SubAvg\. turnsGRPONone; outcome reward only–No12\.836\.1OPSD adaptationDirect optimization targetTokenYes11\.830\.9RLSD/SRPO adaptationAdvantage weightingTokenYes11\.835\.5Ours \(SSPO\)Advantage weightingStepYes14\.537\.5

Table 4:Comparison of the outcome\-only baseline and self\-distillation variants after 50 training steps\. The RLSD/SRPO adaptation uses the self\-distillation signal as token\-level advantage weights, following RLSD, and applies it only to incorrect trajectories, following SRPO\.#### Direct Distillation vs\. Advantage Weighting\.

GRPO provides the outcome\-only baseline, reaching 12\.8 accuracy with an average of 36\.1 turns\. Under the same training budget, the direct OPSD adaptation \(Appendix[F\.1](https://arxiv.org/html/2608.12764#A6.SS1)\) obtains lower accuracy than GRPO \(11\.8 vs\. 12\.8\) and produces substantially shorter trajectories \(30\.9 vs\. 36\.1 turns\)\. Together with Table[1](https://arxiv.org/html/2608.12764#S3.T1), this pattern is consistent with privileged\-information shortcutting: the teacher can solve the task with much less search, and directly matching its distribution transfers this short\-search behavior to the student\.

The RLSD/SRPO adaptation instead uses the token\-level self\-distillation signal as advantage weights on incorrect trajectories\. Its average trajectory length recovers to 35\.5 turns, close to GRPO’s 36\.1 and substantially above the direct OPSD adaptation’s 30\.9, while its accuracy remains 11\.8\. This recovery supports using the teacher signal to modulate the update magnitude rather than as a direct optimization target; however, the unchanged accuracy also shows that token\-level weighting alone is insufficient in this setting\.

MethodEvidence AnchorsIncorrect feedbackBC\-SubSign agreementwith full setting \(%\)Final\-step penaltyamplified \(%\)GRPONoNo12\.8––SSPO w/o Evidence AnchorsNoYes12\.432\.892\.4SSPO w/o Incorrect FeedbackYesNo14\.081\.146\.7Ours \(SSPO\)YesYes14\.5100\.086\.3

Table 5:Ablation of the two privileged\-information sources\. BC\-Sub results are measured after 50 training steps\. The two rightmost columns are computed by rescoring 512 incorrect trajectories, using the sign under the full setting as a reference\. “Final\-step penalty amplified” denotes the proportion of trajectories for which the final\-step weight is greater than one\.
#### Step\-Level Rather Than Token\-Level\.

We then isolate supervision granularity by holding advantage weighting, Evidence Anchors, and incorrect\-trajectory routing fixed\. The token\-level RLSD/SRPO adaptation assigns a separate weight to every token \(Appendix[F\.2](https://arxiv.org/html/2608.12764#A6.SS2)\), whereas SSPO shares one joint\-probability weight across the reasoning and tool\-calling tokens that form a complete information\-seeking step\. This controlled change improves BC\-Sub accuracy from 11\.8 to 14\.5 while maintaining the average trajectory length \(35\.5 vs\. 37\.5 turns\)\. The entropy curves in Figure[6](https://arxiv.org/html/2608.12764#S4.F6)provide complementary evidence: token\-level weighting produces steadily increasing entropy, whereas step\-level weighting remains more stable\. These results support aligning the supervision granularity with the complete search action rather than fragmenting it across individual tokens\.

#### Roles of Evidence Anchors and Incorrect\-Answer Feedback\.

Table[5](https://arxiv.org/html/2608.12764#S4.T5)separates the two sources of privileged information\. Incorrect\-answer feedback alone does not improve over GRPO \(12\.4 vs\. 12\.8\), whereas Evidence Anchors alone improve accuracy to 14\.0\. Combining both yields the best result, 14\.5, showing that Evidence Anchors provide the primary process\-supervision signal and incorrect\-answer feedback is complementary\.

To examine how they contribute to the full step score, we rescore the same 512 incorrect trajectories under each single\-information setting\. This is a decomposition of the deployed signal rather than an assumption that the full setting is an unbiased ground\-truth label\. Evidence Anchors alone agree with the sign of the full signal on 81\.1% of steps, compared with 32\.8% for incorrect\-answer feedback alone, indicating that Evidence Anchors primarily determine the evaluation of intermediate information\-seeking actions\. Conversely, incorrect\-answer feedback amplifies the final\-step penalty in 92\.4% of trajectories, compared with 46\.7% under Evidence Anchors alone\. Thus, the incorrect answer primarily identifies the failed terminal conclusion, while Evidence Anchors ground credit assignment across the preceding search process\.

Figure 5:Entropy curves of GRPO, step\-level, and token\-level supervision during training\.Figure 6:Average number of queries under different step\-level privileged\-information gains\.

## 5Case Study: Which Steps are Exempted from Penalty?

To better understand why SSPO is effective, we analyze how the teacher assigns penalties across different types of steps\. We find thatthe teacher does not simply reward steps based on their lexical overlap with evidence anchors\. Instead, it modulates penalties according to whether a step performs a focused and informative verification action\.

Specifically, steps that receive reduced penalties are typically compact, well\-scoped queries targeting a single local constraint implied by the evidence anchors, as shown in Figure[7](https://arxiv.org/html/2608.12764#S5.F7)\. Although these steps may contain fewer anchor words, they effectively reduce uncertainty along the reasoning chain\. In contrast, steps that receive stronger penalties are often long, diffuse queries that concatenate multiple surface terms from the question or anchors without forming a precise information need, as shown in Figure[8](https://arxiv.org/html/2608.12764#S5.F8)\. Despite their high lexical overlap with anchors, such steps receive amplified penalties, indicating that the teacher prioritizes information quality over word matching\. This behavior is further supported by quantitative analysis\. Figure[6](https://arxiv.org/html/2608.12764#S4.F6)shows the relationship between the average number of queries in search tool calls and the privileged\-information gain\. Steps with positive privileged\-information gainΔτs​t​e​p\>0\\Delta\_\{\\tau\}^\{step\}\>0use significantly fewer queries than those with negative gainΔτs​t​e​p<0\\Delta\_\{\\tau\}^\{step\}<0\. Overall, SSPO encourages targeted, evidence\-grounded information\-seeking by reducing penalties for precise verification steps while amplifying penalties for broad and unfocused exploration\.

[⬇](data:text/plain;base64,PHRoaW5rPgpQZXJoYXBzIHRoZSBzcGVjaWZpYyBFUk5JIGVtcGxveWVlIHJlZmVyZW5jZWQgaXMgbm90IHRoZSBwZXJzb24gbGlzdGVkIGluIHRoZSBwcm9ncmFtIGNvbW1pdHRlZSAoRmxvcmlhbiwgRG9taW5payBldGMpLiBUaGUgYWJvdmUgTGlua2VkSW4gc25pcHBldCBmb3IgUGF0cmljayBQZmVpZmVyIHNob3dzICJpU0FRQiAtIEZvdW5kYXRpb24gTGV2ZWwgKENQU0EtRikiLCB3aGljaCBtYXRjaGVzIHRoZSBjZXJ0aWZpY2F0aW9uLgpOb3cgZG9lcyBQYXRyaWNrIFBmZWlmZXIgc3BlY2lhbGl6ZSBpbiBtZWRpY2FsIHRlY2hub2xvZ3kgc29mdHdhcmU/IFBvdGVudGlhbGx5LCBhcyBFUk5JIHdvcmtzIGluIG1lZHRlY2gsIGFuZCAiUGF0cmljayBQZmVpZmVyIiBtaWdodCBiZSBmb2N1c2luZyBvbiBtZWRpY2FsIHNvbHV0aW9ucywgYnV0IHdlIG5lZWQgdmVyaWZpY2F0aW9uLgpTZWFyY2ggIlBhdHJpY2sgUGZlaWZlciBFUk5JIG1lZGljYWwgdGVjaG5vbG9neSIuCjwvdGhpbms+Cjx0b29sX2NhbGw+CiJzZWFyY2ggcXVlcmllcyI6IFsiUGF0cmljayBQZmVpZmVyIEVSTkkgbWVkaWNhbCB0ZWNobm9sb2d5Il0KPC90b29sX2NhbGw+)<think\>PerhapsthespecificERNIemployeereferencedisnotthepersonlistedintheprogramcommittee\(Florian,Dominiketc\)\.TheaboveLinkedInsnippetforPatrickPfeifershows"iSAQB\-FoundationLevel\(CPSA\-F\)",whichmatchesthecertification\.NowdoesPatrickPfeiferspecializeinmedicaltechnologysoftware?Potentially,asERNIworksinmedtech,and"PatrickPfeifer"mightbefocusingonmedicalsolutions,butweneedverification\.Search"PatrickPfeiferERNImedicaltechnology"\.</think\><tool\_call\>"searchqueries":\["PatrickPfeiferERNImedicaltechnology"\]</tool\_call\>Figure 7:Δτs​t​e​p\>0\\Delta\_\{\\tau\}^\{step\}\>0: compact single\-query verification targeting one specific entity\-attribute relation\.[⬇](data:text/plain;base64,PHRoaW5rPgpFUk5JIG1hdGNoZXMgdGhlIGNvbnN1bHRhbmN5IHdpdGggb2ZmaWNlcyBhY3Jvc3MgRXVyb3BlIGFuZCBBc2lhLgpOb3cgZmluZCBhbiBFUk5JIGVtcGxveWVlIHdobyBpcyBhICJtZWRpY2FsIHRlY2hub2xvZ3kgc29mdHdhcmUgc3BlY2lhbGlzdCIgYW5kIGhvbGRzIGEgImZvdW5kYXRpb25hbCBjZXJ0aWZpY2F0aW9uIGluIHNvZnR3YXJlIGFyY2hpdGVjdHVyZSIuIFRoZSBmb3VuZGF0aW9uIGNlcnRpZmljYXRpb24gY291bGQgcmVmZXIgdG8gdGhlIGlTQVFCIENQU0EtRiBjZXJ0aWZpY2F0aW9uLgpXZSBhbHNvIGhhdmUgdGhhdCB0aGlzIGVtcGxveWVlIHNlcnZlcyBvbiB0aGUgY29uZmVyZW5jZSBjb21taXR0ZWUgZm9yIEJhc2VsT25lIGFuZCByZWNlbnRseSBjby1mb3VuZGVkIGFuIGluaXRpYXRpdmUgbGlua2luZyBsb2NhbCBpbmR1c3RyeSBhbmQgYWNhZGVtaWEuClRoZSBwcm9ncmFtIGNvbW1pdHRlZSBpbmNsdWRlcyBJeC1jaGVsIFJ1aXosIERvbWluaWsgQmVyZ2VyLCBNYXJjdXMgRmlobG9uLCBSb2JpbiBNZXpnZXIsIEZsb3JpYW4gU3RhbXBlLCBGcmFuayBCYWllciwgZXRjLiBBbW9uZyB0aGVzZSwgcGVyaGFwcyBvbmUgaXMgYSBtZWRpY2FsIHRlY2hub2xvZ3kgc29mdHdhcmUgc3BlY2lhbGlzdC4KQnV0IHdlIGNvdWxkIGFsc28gc2VhcmNoIGZvciAiQmFzZWxPbmUgY29uZmVyZW5jZSBjb21taXR0ZWUgRVJOSSIuCjwvdGhpbms+Cjx0b29sX2NhbGw+CiJzZWFyY2ggcXVlcmllcyI6IFsKICAiQmFzZWxPbmUgY29uZmVyZW5jZSBjb21taXR0ZWUgRVJOSSIsCiAgIkVSTkkgbWVkaWNhbCB0ZWNobm9sb2d5IHNvZnR3YXJlIGFyY2hpdGVjdCBCYXNlbE9uZSIsCiAgIkVSTkkgbWVkaWNhbCB0ZWNobm9sb2d5IHNvZnR3YXJlIHNwZWNpYWxpc3QgQmFzZWxPbmUgY29tbWl0dGVlIiwKICAiQ1BTQSBGb3VuZGF0aW9uIEVSTkkgZW1wbG95ZWUiXQo8L3Rvb2xfY2FsbD4=)<think\>ERNImatchestheconsultancywithofficesacrossEuropeandAsia\.NowfindanERNIemployeewhoisa"medicaltechnologysoftwarespecialist"andholdsa"foundationalcertificationinsoftwarearchitecture"\.ThefoundationcertificationcouldrefertotheiSAQBCPSA\-Fcertification\.WealsohavethatthisemployeeservesontheconferencecommitteeforBaselOneandrecentlyco\-foundedaninitiativelinkinglocalindustryandacademia\.TheprogramcommitteeincludesIx\-chelRuiz,DominikBerger,MarcusFihlon,RobinMezger,FlorianStampe,FrankBaier,etc\.Amongthese,perhapsoneisamedicaltechnologysoftwarespecialist\.Butwecouldalsosearchfor"BaselOneconferencecommitteeERNI"\.</think\><tool\_call\>"searchqueries":\["BaselOneconferencecommitteeERNI","ERNImedicaltechnologysoftwarearchitectBaselOne","ERNImedicaltechnologysoftwarespecialistBaselOnecommittee","CPSAFoundationERNIemployee"\]</tool\_call\>Figure 8:Δτs​t​e​p<0\\Delta\_\{\\tau\}^\{step\}<0: diffuse multi\-query search with high lexical overlap but poor localization\.
## 6Conclusion

We presented SSPO, a step\-level self\-distillation method that extends On\-Policy Self\-Distillation from single\-turn reasoning to multi\-turn deep search agents\. To construct a meaningful self\-teacher in a setting where neither reference solutions nor execution feedback are available, we introduced Evidence Anchors—structured, step\-aligned privileged information whose granularity matches that of the search process itself\. From this teacher, SSPO derives a self\-distillation signal, recasts it as step\-level advantage weights, and applies it only to incorrect trajectories, thereby injecting fine\-grained process supervision while sidestepping both privileged\-information leakage and the ambiguous gradients that direct distillation would otherwise impose on already\-correct rollouts\.

Across BrowseComp, GAIA, and FRAMES, SSPO surpasses GRPO trained for twice as many gradient steps, while sustaining higher rewards and more stable gradient norms throughout optimization\. Beyond the aggregate gains, our analysis reveals a coherent shaping of the policy update: penalties contract on compact, evidence\-grounded verification steps and expand on diffuse, underspecified queries\. This suggests that step\-level self\-distillation guides the agent toward more targeted and efficient information\-seeking, rather than simply lifting final\-answer accuracy\.

## References

- \[1\]Z\. AI\(2025\)GLM\-4\.6\.Note:[https://docs\.z\.ai/guides/llm/glm\-4\.6](https://docs.z.ai/guides/llm/glm-4.6)Cited by:[Table 6](https://arxiv.org/html/2608.12764#A4.T6.2.4.1),[Table 7](https://arxiv.org/html/2608.12764#A4.T7.2.3.1)\.
- \[2\]Z\. AI\(2026\)GLM\-5\.1\.Note:[https://docs\.z\.ai/guides/llm/glm\-5\.1](https://docs.z.ai/guides/llm/glm-5.1)Cited by:[§1](https://arxiv.org/html/2608.12764#S1.p1.1)\.
- \[3\]Anthropic\(2025\)Introducing Claude 4 — anthropic\.com\.Note:[https://www\.anthropic\.com/news/claude\-4](https://www.anthropic.com/news/claude-4)\[Accessed 19\-03\-2026\]Cited by:[Table 6](https://arxiv.org/html/2608.12764#A4.T6.2.2.1),[Table 7](https://arxiv.org/html/2608.12764#A4.T7.2.2.1)\.
- \[4\]Anthropic\(2026\)Introducing claude opus 4\.7\.Note:[https://www\.anthropic\.com/news/claude\-opus\-4\-7](https://www.anthropic.com/news/claude-opus-4-7)Cited by:[§1](https://arxiv.org/html/2608.12764#S1.p1.1)\.
- \[5\]Z\. Bi, K\. Chen, C\. Tseng, D\. Zhang, T\. Wang, H\. Luo, L\. Chen, J\. Huang, J\. Guan, J\. Hao, X\. Song, and J\. Song\(2025\)Is gpt\-oss good? a comprehensive evaluation of openai’s latest open source models\.External Links:2508\.12461,[Link](https://arxiv.org/abs/2508.12461)Cited by:[§D\.2](https://arxiv.org/html/2608.12764#A4.SS2.p2.1)\.
- \[6\]T\. Chu, Y\. Zhai, J\. Yang, S\. Tong, S\. Xie, D\. Schuurmans, Q\. V\. Le, S\. Levine, and Y\. Ma\(2025\)SFT memorizes, RL generalizes\.InProceedings of the 42nd International Conference on Machine Learning,pp\. 10818–10838\.Cited by:[§A\.1](https://arxiv.org/html/2608.12764#A1.SS1.p1.1),[§1](https://arxiv.org/html/2608.12764#S1.p2.1)\.
- \[7\]G\. Deepmind\(2026\)Gemini 3\.1 pro: best for complex tasks and bringing creative concepts to life\.Note:[https://deepmind\.google/models/gemini/pro/](https://deepmind.google/models/gemini/pro/)Cited by:[§1](https://arxiv.org/html/2608.12764#S1.p1.1)\.
- \[8\]DeepSeek\-AI, A\. Liu, A\. Mei, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong, C\. Lu, C\. Zhao, C\. Deng, C\. Xu, C\. Ruan, D\. Dai, D\. Guo, D\. Yang,et al\.\(2025\)DeepSeek\-v3\.2: pushing the frontier of open large language models\.External Links:2512\.02556,[Link](https://arxiv.org/abs/2512.02556)Cited by:[§D\.1](https://arxiv.org/html/2608.12764#A4.SS1.p2.1),[Table 6](https://arxiv.org/html/2608.12764#A4.T6.2.5.1),[§4\.1](https://arxiv.org/html/2608.12764#S4.SS1.SSS0.Px1.p1.1)\.
- \[9\]DeepSeek\-AI\(2026\)DeepSeek\-v4: towards highly efficient million\-token context intelligence\.Cited by:[§1](https://arxiv.org/html/2608.12764#S1.p1.1)\.
- \[10\]E\. Guha, R\. Marten, S\. Keh, N\. Raoof, G\. Smyrnis, H\. Bansal, M\. Nezhurina, J\. Mercat, T\. Vu, Z\. Sprague, A\. Suvarna, B\. Feuer, L\. Chen, Z\. Khan, E\. Frankel,et al\.\(2025\)OpenThoughts: data recipes for reasoning models\.External Links:2506\.04178,[Link](https://arxiv.org/abs/2506.04178)Cited by:[§3\.1](https://arxiv.org/html/2608.12764#S3.SS1.p1.1)\.
- \[11\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi, X\. Zhang, X\. Yu, Y\. Wu, Z\. F\. Wu, Z\. Gou,et al\.\(2025\)DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:ISSN 1476\-4687,[Link](http://dx.doi.org/10.1038/s41586-025-09422-z),[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[§A\.1](https://arxiv.org/html/2608.12764#A1.SS1.p1.1),[§1](https://arxiv.org/html/2608.12764#S1.p4.1),[§2\.2](https://arxiv.org/html/2608.12764#S2.SS2.SSS0.Px1.p1.2)\.
- \[12\]X\. Ho, A\. Duong Nguyen, S\. Sugawara, and A\. Aizawa\(2020\)Constructing a multi\-hop QA dataset for comprehensive evaluation of reasoning steps\.InProceedings of the 28th International Conference on Computational Linguistics,Barcelona, Spain \(Online\),pp\. 6609–6625\.External Links:[Link](https://www.aclweb.org/anthology/2020.coling-main.580)Cited by:[§A\.2](https://arxiv.org/html/2608.12764#A1.SS2.p2.1)\.
- \[13\]J\. Hübotter, F\. Lübeck, L\. Behric, A\. Baumann, M\. Bagatella, D\. Marta, I\. Hakimi, I\. Shenfeld, T\. Kleine Buening, C\. Guestrin, and A\. Krause\(2026\)Reinforcement learning via self\-distillation\.arXiv preprint arXiv:2601\.20802\.Cited by:[§A\.3](https://arxiv.org/html/2608.12764#A1.SS3.p2.1),[§1](https://arxiv.org/html/2608.12764#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.12764#S2.SS2.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2608.12764#S3.SS1.p1.1)\.
- \[14\]B\. Jin, J\. Yoon, P\. Kargupta, S\. O\. Arik, and J\. Han\(2025\)An empirical study on reinforcement learning for reasoning\-search interleaved llm agents\.arXiv preprint arXiv:2505\.15117\.Cited by:[§A\.1](https://arxiv.org/html/2608.12764#A1.SS1.p1.1)\.
- \[15\]B\. Jin, H\. Zeng, Z\. Yue, J\. Yoon, S\. Arik, D\. Wang, H\. Zamani, and J\. Han\(2025\)Search\-r1: training llms to reason and leverage search engines with reinforcement learning\.arXiv preprint arXiv:2503\.09516\.Cited by:[§A\.1](https://arxiv.org/html/2608.12764#A1.SS1.p1.1)\.
- \[16\]S\. Krishna, K\. Krishna, A\. Mohananey, S\. Schwarcz, A\. Stambler, S\. Upadhyay, and M\. Faruqui\(2024\)Fact, fetch, and reason: a unified evaluation of retrieval\-augmented generation\.External Links:2409\.12941,[Link](https://arxiv.org/abs/2409.12941)Cited by:[§1](https://arxiv.org/html/2608.12764#S1.p5.1),[§4\.1](https://arxiv.org/html/2608.12764#S4.SS1.SSS0.Px2.p1.1),[Table 3](https://arxiv.org/html/2608.12764#S4.T3.2.1.1.4)\.
- \[17\]G\. Li, T\. Yang, J\. Fang, M\. Song, M\. Zheng, H\. Guo, D\. Zhang, J\. Wang, and T\. Chua\(2026\)Unifying group\-relative and self\-distillation policy optimization via sample routing\.External Links:2604\.02288,[Link](https://arxiv.org/abs/2604.02288)Cited by:[§A\.3](https://arxiv.org/html/2608.12764#A1.SS3.p2.1),[§1](https://arxiv.org/html/2608.12764#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.12764#S2.SS2.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2608.12764#S3.SS2.p1.1)\.
- \[18\]J\. Li, W\. Zhao, J\. Zhao, W\. Zeng, H\. Wu, X\. Wang, R\. Ge, Y\. Cao, Y\. Huang, W\. Liu, J\. Liu, Z\. Su, Y\. Guo, F\. Zhou, L\. Zhang, J\. Michelini, X\. Wang, X\. Yue, S\. Zhou, G\. Neubig, and J\. He\(2026\)The tool decathlon: benchmarking language agents for diverse, realistic, and long\-horizon task execution\.External Links:2510\.25726,[Link](https://arxiv.org/abs/2510.25726)Cited by:[§1](https://arxiv.org/html/2608.12764#S1.p1.1)\.
- \[19\]K\. Li, Z\. Zhang, H\. Yin, R\. Ye, Y\. Zhao, L\. Zhang, L\. Ou, D\. Zhang, X\. Wu, J\. Wu, X\. Wang, Z\. Qiao, Z\. Zhang, Y\. Jiang, P\. Xie, F\. Huang, and J\. Zhou\(2025\)WebSailor\-v2: bridging the chasm to proprietary agents via synthetic data and scalable reinforcement learning\.External Links:2509\.13305,[Link](https://arxiv.org/abs/2509.13305)Cited by:[§D\.1](https://arxiv.org/html/2608.12764#A4.SS1.p1.1),[§1](https://arxiv.org/html/2608.12764#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.12764#S2.SS2.SSS0.Px1.p1.2)\.
- \[20\]K\. Li, Z\. Zhang, H\. Yin, L\. Zhang, L\. Ou, J\. Wu, W\. Yin, B\. Li, Z\. Tao, X\. Wang, W\. Shen, J\. Zhang, D\. Zhang, X\. Wu, Y\. Jiang, M\. Yan, P\. Xie, F\. Huang, and J\. Zhou\(2025\)WebSailor: navigating super\-human reasoning for web agent\.External Links:2507\.02592,[Link](https://arxiv.org/abs/2507.02592)Cited by:[Appendix B](https://arxiv.org/html/2608.12764#A2.SS0.SSS0.Px1.p1.1),[§D\.1](https://arxiv.org/html/2608.12764#A4.SS1.p1.1),[§1](https://arxiv.org/html/2608.12764#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.12764#S2.SS2.SSS0.Px1.p1.2),[§2\.2](https://arxiv.org/html/2608.12764#S2.SS2.SSS0.Px1.p1.4),[Table 3](https://arxiv.org/html/2608.12764#S4.T3.2.1.3.1)\.
- \[21\]W\. Li, J\. Lin, Z\. Jiang, J\. Cao, X\. Liu, J\. Zhang, Z\. Huang, Q\. Chen, W\. Sun, Q\. Wang, H\. Lu, T\. Qin, C\. Zhu, Y\. Yao, S\. Fan, X\. Li, T\. Wang, P\. Liu, K\. Zhu, H\. Zhu, D\. Shi, P\. Wang, Y\. Guan, X\. Tang, M\. Liu, Y\. E\. Jiang, J\. Yang, J\. Liu, G\. Zhang, and W\. Zhou\(2025\)Chain\-of\-agents: end\-to\-end agent foundation models via multi\-agent distillation and agentic rl\.External Links:2508\.13167,[Link](https://arxiv.org/abs/2508.13167)Cited by:[Table 3](https://arxiv.org/html/2608.12764#S4.T3.2.1.5.1)\.
- \[22\]X\. Li, J\. Jin, G\. Dong, H\. Qian, Y\. Wu, J\. Wen, Y\. Zhu, and Z\. Dou\(2025\)WebThinker: empowering large reasoning models with deep research capability\.External Links:2504\.21776,[Link](https://arxiv.org/abs/2504.21776)Cited by:[§D\.1](https://arxiv.org/html/2608.12764#A4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2608.12764#S4.SS1.SSS0.Px2.p1.1)\.
- \[23\]Y\. Li, Y\. Zuo, B\. He, J\. Zhang, C\. Xiao, C\. Qian, T\. Yu, H\. Gao, W\. Yang, Z\. Liu,et al\.\(2026\)Rethinking on\-policy distillation of large language models: phenomenology, mechanism, and recipe\.arXiv preprint arXiv:2604\.13016\.Cited by:[§A\.3](https://arxiv.org/html/2608.12764#A1.SS3.p1.1),[§1](https://arxiv.org/html/2608.12764#S1.p2.1)\.
- \[24\]J\. Liu and L\. Zhang\(2025\)Code\-r1: reproducing r1 for code with reliable rewards\.Note:[https://github\.com/ganler/code\-r1](https://github.com/ganler/code-r1)Cited by:[§A\.1](https://arxiv.org/html/2608.12764#A1.SS1.p1.1)\.
- \[25\]J\. Liu, Y\. Fan, Z\. Jiang, H\. Ding, Y\. Hu, C\. Zhang, Y\. Shi, S\. Weng, A\. Chen, S\. Chen, Y\. Huang, M\. Zhang, P\. Zhao, J\. Yan, and J\. He\(2025\)SynLogic: synthesizing verifiable reasoning data at scale for learning logical reasoning and beyond\.External Links:2505\.19641,[Link](https://arxiv.org/abs/2505.19641)Cited by:[§A\.1](https://arxiv.org/html/2608.12764#A1.SS1.p1.1)\.
- \[26\]J\. Liu, Y\. Li, C\. Zhang, J\. Li, A\. Chen, K\. Ji, W\. Cheng, Z\. Wu, C\. Du, Q\. Xu, J\. Song, Z\. Zhu, W\. Chen, P\. Zhao, and J\. He\(2025\)WebExplorer: explore and evolve for training long\-horizon web agents\.External Links:2509\.06501,[Link](https://arxiv.org/abs/2509.06501)Cited by:[Appendix B](https://arxiv.org/html/2608.12764#A2.SS0.SSS0.Px1.p1.1),[Appendix C](https://arxiv.org/html/2608.12764#A3.p1.1),[§D\.1](https://arxiv.org/html/2608.12764#A4.SS1.p1.1),[§D\.2](https://arxiv.org/html/2608.12764#A4.SS2.p2.1),[§1](https://arxiv.org/html/2608.12764#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.12764#S2.SS2.SSS0.Px1.p1.2),[§4\.1](https://arxiv.org/html/2608.12764#S4.SS1.SSS0.Px1.p1.1),[Table 3](https://arxiv.org/html/2608.12764#S4.T3.2.1.7.1)\.
- \[27\]K\. Lu and T\. M\. Lab\(2025\)On\-policy distillation\.Thinking Machines Lab: Connectionism\.Note:https://thinkingmachines\.ai/blog/on\-policy\-distillationExternal Links:[Document](https://dx.doi.org/10.64434/tml.20251026)Cited by:[§A\.3](https://arxiv.org/html/2608.12764#A1.SS3.p1.1),[§F\.1](https://arxiv.org/html/2608.12764#A6.SS1.p1.1),[§1](https://arxiv.org/html/2608.12764#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.12764#S2.SS2.SSS0.Px2.p1.1)\.
- \[28\]R\. Lu, Z\. Hou, Z\. Wang, H\. Zhang, X\. Liu, Y\. Li, S\. Feng, J\. Tang, and Y\. Dong\(2025\)DeepDive: advancing deep search agents with knowledge graphs and multi\-turn rl\.External Links:2509\.10446,[Link](https://arxiv.org/abs/2509.10446)Cited by:[Table 3](https://arxiv.org/html/2608.12764#S4.T3.2.1.6.1)\.
- \[29\]G\. Mialon, C\. Fourrier, C\. Swift, T\. Wolf, Y\. LeCun, and T\. Scialom\(2023\)GAIA: a benchmark for general ai assistants\.External Links:2311\.12983,[Link](https://arxiv.org/abs/2311.12983)Cited by:[§1](https://arxiv.org/html/2608.12764#S1.p5.1),[§4\.1](https://arxiv.org/html/2608.12764#S4.SS1.SSS0.Px2.p1.1),[Table 3](https://arxiv.org/html/2608.12764#S4.T3.2.1.1.3)\.
- \[30\]MiniMax\(2026\)MiniMax m2\.7: early echoes of self\-evolution\.External Links:[Link](https://www.minimax.io/news/minimax-m27-en)Cited by:[§1](https://arxiv.org/html/2608.12764#S1.p1.1)\.
- \[31\]OpenAI\(2025\)GPT\-5\-nano\.Note:[https://developers\.openai\.com/api/docs/models/gpt\-5\-nano](https://developers.openai.com/api/docs/models/gpt-5-nano)Cited by:[§D\.1](https://arxiv.org/html/2608.12764#A4.SS1.p2.1),[Table 6](https://arxiv.org/html/2608.12764#A4.T6)\.
- \[32\]OpenAI\(2025\)Introducing gpt\-oss\.Note:[https://openai\.com/index/introducing\-gpt\-oss/](https://openai.com/index/introducing-gpt-oss/)Cited by:[§D\.2](https://arxiv.org/html/2608.12764#A4.SS2.p1.1),[§D\.2](https://arxiv.org/html/2608.12764#A4.SS2.p2.1),[Table 7](https://arxiv.org/html/2608.12764#A4.T7.2.4.1),[Appendix H](https://arxiv.org/html/2608.12764#A8.p1.1)\.
- \[33\]OpenAI\(2026\)Introducing gpt‑5\.5\.Note:[https://openai\.com/index/introducing\-gpt\-5\-5/](https://openai.com/index/introducing-gpt-5-5/)Cited by:[§1](https://arxiv.org/html/2608.12764#S1.p1.1)\.
- \[34\]Qwen Team\(2026\)Qwen3\.6\-Plus: towards real world agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.6)Cited by:[§1](https://arxiv.org/html/2608.12764#S1.p1.1)\.
- \[35\]H\. Sang, Y\. Xu, Z\. Zhou, R\. He, Z\. Wang, and J\. Sun\(2026\)CRISP: compressed reasoning via iterative self\-policy distillation\.External Links:2603\.05433,[Link](https://arxiv.org/abs/2603.05433)Cited by:[§A\.3](https://arxiv.org/html/2608.12764#A1.SS3.p2.1),[§1](https://arxiv.org/html/2608.12764#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.12764#S2.SS2.SSS0.Px2.p1.1)\.
- \[36\]F\. Schmidt\(2019\)Generalization in generation: a closer look at exposure bias\.InProceedings of the 3rd Workshop on Neural Generation and Translation,pp\. 157–167\.External Links:[Link](https://aclanthology.org/D19-5616/)Cited by:[§A\.1](https://arxiv.org/html/2608.12764#A1.SS1.p1.1),[§1](https://arxiv.org/html/2608.12764#S1.p2.1)\.
- \[37\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo\(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.External Links:2402\.03300,[Link](https://arxiv.org/abs/2402.03300)Cited by:[§A\.1](https://arxiv.org/html/2608.12764#A1.SS1.p1.1),[§1](https://arxiv.org/html/2608.12764#S1.p4.1),[§2\.2](https://arxiv.org/html/2608.12764#S2.SS2.SSS0.Px1.p1.1),[§2\.2](https://arxiv.org/html/2608.12764#S2.SS2.SSS0.Px1.p1.2)\.
- \[38\]I\. Shenfeld, M\. Damani, J\. Hübotter, and P\. Agrawal\(2026\)Self\-distillation enables continual learning\.External Links:2601\.19897,[Link](https://arxiv.org/abs/2601.19897)Cited by:[§A\.1](https://arxiv.org/html/2608.12764#A1.SS1.p1.1),[§A\.3](https://arxiv.org/html/2608.12764#A1.SS3.p2.1),[§1](https://arxiv.org/html/2608.12764#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.12764#S2.SS2.SSS0.Px2.p1.1)\.
- \[39\]M\. Song and M\. Zheng\(2026\)A survey of on\-policy distillation for large language models\.External Links:2604\.00626,[Link](https://arxiv.org/abs/2604.00626)Cited by:[§A\.3](https://arxiv.org/html/2608.12764#A1.SS3.p1.1),[§1](https://arxiv.org/html/2608.12764#S1.p2.1)\.
- \[40\]Z\. Tao, J\. Wu, W\. Yin, J\. Zhang, B\. Li, H\. Shen, K\. Li, L\. Zhang, X\. Wang, Y\. Jiang, P\. Xie, F\. Huang, and J\. Zhou\(2025\)WebShaper: agentically data synthesizing via information\-seeking formalization\.External Links:2507\.15061,[Link](https://arxiv.org/abs/2507.15061)Cited by:[§D\.1](https://arxiv.org/html/2608.12764#A4.SS1.p1.1),[§1](https://arxiv.org/html/2608.12764#S1.p1.1)\.
- \[41\]K\. Team\(2026\)Kimi k2\.6: advancing open\-source coding\.Note:[https://www\.kimi\.com/blog/kimi\-k2\-6](https://www.kimi.com/blog/kimi-k2-6)Cited by:[§1](https://arxiv.org/html/2608.12764#S1.p1.1)\.
- \[42\]M\. Team, S\. Bai, L\. Bing, C\. Chen, G\. Chen, Y\. Chen, Z\. Chen, Z\. Chen, X\. Dong,et al\.\(2025\)MiroThinker: pushing the performance boundaries of open\-source research agents via model, context, and interactive scaling\.arXiv preprint arXiv:2511\.11793\.Cited by:[Table 3](https://arxiv.org/html/2608.12764#S4.T3.2.1.4.1)\.
- \[43\]J\. Wei, Z\. Sun, S\. Papay, S\. McKinney, J\. Han, I\. Fulford, H\. W\. Chung, A\. T\. Passos, W\. Fedus, and A\. Glaese\(2025\)BrowseComp: a simple yet challenging benchmark for browsing agents\.External Links:2504\.12516,[Link](https://arxiv.org/abs/2504.12516)Cited by:[§1](https://arxiv.org/html/2608.12764#S1.p1.1),[§1](https://arxiv.org/html/2608.12764#S1.p5.1),[§3\.1](https://arxiv.org/html/2608.12764#S3.SS1.p1.1),[§4\.1](https://arxiv.org/html/2608.12764#S4.SS1.SSS0.Px2.p1.1),[Table 3](https://arxiv.org/html/2608.12764#S4.T3.2.1.1.2)\.
- \[44\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. Le, and D\. Zhou\(2023\)Chain\-of\-thought prompting elicits reasoning in large language models\.External Links:2201\.11903,[Link](https://arxiv.org/abs/2201.11903)Cited by:[§D\.2](https://arxiv.org/html/2608.12764#A4.SS2.p1.1),[§2\.2](https://arxiv.org/html/2608.12764#S2.SS2.SSS0.Px2.p1.1)\.
- \[45\]T\. Wen, G\. Dong, and Z\. Dou\(2026\)SmartSearch: process reward\-guided query refinement for search agents\.External Links:2601\.04888,[Link](https://arxiv.org/abs/2601.04888)Cited by:[§A\.2](https://arxiv.org/html/2608.12764#A1.SS2.p1.1)\.
- \[46\]H\. Wu, C\. Wang, W\. Zhao, and J\. He\(2025\)Mirage or method? how model\-task alignment induces divergent rl conclusions\.External Links:2508\.21188,[Link](https://arxiv.org/abs/2508.21188)Cited by:[§A\.1](https://arxiv.org/html/2608.12764#A1.SS1.p1.1)\.
- \[47\]H\. Wu, Y\. Yao, W\. Yu, and N\. Zhang\(2025\)ReCode: updating code api knowledge with reinforcement learning\.External Links:2506\.20495,[Link](https://arxiv.org/abs/2506.20495)Cited by:[§A\.1](https://arxiv.org/html/2608.12764#A1.SS1.p1.1)\.
- \[48\]Y\. Wu, Y\. Zhou, Z\. Ziheng, Y\. Peng, X\. Ye, X\. Hu, W\. Zhu, L\. Qi, M\. Yang, and X\. Yang\(2026\)On the generalization of sft: a reinforcement learning perspective with reward rectification\.External Links:2508\.05629,[Link](https://arxiv.org/abs/2508.05629)Cited by:[§A\.1](https://arxiv.org/html/2608.12764#A1.SS1.p1.1),[§1](https://arxiv.org/html/2608.12764#S1.p2.1)\.
- \[49\]X\.AI\(2025\)Grok 4\.1 fast and agent tools api\.Note:[https://x\.ai/news/grok\-4\-1\-fast](https://x.ai/news/grok-4-1-fast)Cited by:[Table 6](https://arxiv.org/html/2608.12764#A4.T6.2.3.1)\.
- \[50\]T\. Xie, Z\. Gao, Q\. Ren, H\. Luo, Y\. Hong, B\. Dai, J\. Zhou, K\. Qiu, Z\. Wu, and C\. Luo\(2025\)Logic\-rl: unleashing llm reasoning with rule\-based reinforcement learning\.External Links:2502\.14768,[Link](https://arxiv.org/abs/2502.14768)Cited by:[§A\.1](https://arxiv.org/html/2608.12764#A1.SS1.p1.1),[§2\.1](https://arxiv.org/html/2608.12764#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2608.12764#S2.SS2.SSS0.Px1.p1.1)\.
- \[51\]P\. Xu, Z\. Li, X\. Xing, G\. Zhang, D\. Li, and K\. Shi\(2026\)Principle process reward for search agents\.External Links:[Link](https://openreview.net/forum?id=zN1aqLhkGm)Cited by:[§A\.2](https://arxiv.org/html/2608.12764#A1.SS2.p1.1)\.
- \[52\]Y\. Xu, H\. Sang, Z\. Zhou, R\. He, Z\. Wang, and A\. Geramifard\(2026\)TIP: token importance in on\-policy distillation\.External Links:2604\.14084,[Link](https://arxiv.org/abs/2604.14084)Cited by:[§A\.3](https://arxiv.org/html/2608.12764#A1.SS3.p1.1)\.
- \[53\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang,et al\.\(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[Appendix B](https://arxiv.org/html/2608.12764#A2.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.12764#S1.p5.1)\.
- \[54\]C\. Yang, C\. Qin, Q\. Si, M\. Chen, N\. Gu, D\. Yao, Z\. Lin, W\. Wang, J\. Wang, and N\. Duan\(2026\)Self\-distilled rlvr\.External Links:2604\.03128,[Link](https://arxiv.org/abs/2604.03128)Cited by:[§A\.3](https://arxiv.org/html/2608.12764#A1.SS3.p2.1),[§1](https://arxiv.org/html/2608.12764#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.12764#S2.SS2.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2608.12764#S3.SS2.p1.1)\.
- \[55\]Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning\(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.External Links:1809\.09600,[Link](https://arxiv.org/abs/1809.09600)Cited by:[§A\.2](https://arxiv.org/html/2608.12764#A1.SS2.p2.1)\.
- \[56\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao\(2023\)ReAct: synergizing reasoning and acting in language models\.External Links:2210\.03629,[Link](https://arxiv.org/abs/2210.03629)Cited by:[Appendix B](https://arxiv.org/html/2608.12764#A2.SS0.SSS0.Px1.p1.1),[§D\.2](https://arxiv.org/html/2608.12764#A4.SS2.p1.1),[§2\.1](https://arxiv.org/html/2608.12764#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2608.12764#S2.SS2.SSS0.Px1.p1.1)\.
- \[57\]T\. Ye, L\. Dong, X\. Wu, S\. Huang, and F\. Wei\(2026\)On\-policy context distillation for language models\.External Links:2602\.12275,[Link](https://arxiv.org/abs/2602.12275)Cited by:[§A\.3](https://arxiv.org/html/2608.12764#A1.SS3.p2.1),[§2\.2](https://arxiv.org/html/2608.12764#S2.SS2.SSS0.Px2.p1.1)\.
- \[58\]Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, M\. Qiao, Y\. Wu, and M\. Wang\(2025\)DAPO: an open\-source llm reinforcement learning system at scale\.External Links:2503\.14476,[Link](https://arxiv.org/abs/2503.14476)Cited by:[§A\.1](https://arxiv.org/html/2608.12764#A1.SS1.p1.1)\.
- \[59\]W\. Zeng, Y\. Huang, Q\. Liu, W\. Liu, K\. He, Z\. Ma, and J\. He\(2025\)SimpleRL\-zoo: investigating and taming zero reinforcement learning for open base models in the wild\.InSecond Conference on Language Modeling,Cited by:[§A\.1](https://arxiv.org/html/2608.12764#A1.SS1.p1.1)\.
- \[60\]G\. Zhang, H\. Geng, X\. Yu, Z\. Yin, Z\. Zhang, Z\. Tan, H\. Zhou, Z\. Li, X\. Xue, Y\. Li, Y\. Zhou, Y\. Chen, C\. Zhang, Y\. Fan, Z\. Wang, S\. Huang, F\. P\. Velez, Y\. Liao, H\. WANG, M\. Yang, H\. Ji, J\. Wang, S\. YAN, P\. Torr, and L\. BAI\(2026\)The landscape of agentic reinforcement learning for LLMs: a survey\.Transactions on Machine Learning Research\.Note:Survey CertificationExternal Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=RY19y2RI1O)Cited by:[§3\.2](https://arxiv.org/html/2608.12764#S3.SS2.p2.1)\.
- \[61\]Y\. Zhang, Y\. Zeng, Q\. Li, Z\. Hu, K\. Han, and W\. Zuo\(2025\)Tool\-r1: sample\-efficient reinforcement learning for agentic tool use\.External Links:2509\.12867,[Link](https://arxiv.org/abs/2509.12867)Cited by:[§A\.1](https://arxiv.org/html/2608.12764#A1.SS1.p1.1)\.
- \[62\]Y\. Zhang, H\. Huang, Z\. Song, Y\. Zhu, Q\. Zhang, Z\. Zhao, and D\. Zhao\(2025\)CriticSearch: fine\-grained credit assignment for search agents via a retrospective critic\.External Links:2511\.12159,[Link](https://arxiv.org/abs/2511.12159)Cited by:[§A\.2](https://arxiv.org/html/2608.12764#A1.SS2.p1.1)\.
- \[63\]S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. Grover\(2026\)Self\-distilled reasoner: on\-policy self\-distillation for large language models\.External Links:2601\.18734,[Link](https://arxiv.org/abs/2601.18734)Cited by:[§A\.3](https://arxiv.org/html/2608.12764#A1.SS3.p2.1),[§1](https://arxiv.org/html/2608.12764#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.12764#S2.SS2.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2608.12764#S3.SS1.p1.1)\.
- \[64\]Y\. Zhou, K\. Zheng, Q\. Chen, M\. Hu, Q\. Sun, C\. Xu, and J\. Chen\(2026\)OffSeeker: online reinforcement learning is not all you need for deep research agents\.External Links:2601\.18467,[Link](https://arxiv.org/abs/2601.18467)Cited by:[§4\.1](https://arxiv.org/html/2608.12764#S4.SS1.SSS0.Px1.p1.1),[Table 3](https://arxiv.org/html/2608.12764#S4.T3.2.1.8.1)\.

## Appendix ARelated Work

### A\.1Reinforcement Learning with Verifiable Rewards \(RLVR\)

Building on works such as DeepSeek\-MATH\[[37](https://arxiv.org/html/2608.12764#bib.bib22)\]and DeepSeek\-R1\[[11](https://arxiv.org/html/2608.12764#bib.bib24)\], RLVR has become a standard component in LLM training pipelines and is widely regarded as essential for improving reasoning capabilities across domains, including but not limited to mathematics\[[59](https://arxiv.org/html/2608.12764#bib.bib55),[58](https://arxiv.org/html/2608.12764#bib.bib56)\], logical reasoning\[[50](https://arxiv.org/html/2608.12764#bib.bib23),[25](https://arxiv.org/html/2608.12764#bib.bib52),[46](https://arxiv.org/html/2608.12764#bib.bib53)\], coding\[[24](https://arxiv.org/html/2608.12764#bib.bib57),[47](https://arxiv.org/html/2608.12764#bib.bib54)\], and verifiable agent tasks\[[14](https://arxiv.org/html/2608.12764#bib.bib58),[15](https://arxiv.org/html/2608.12764#bib.bib59),[61](https://arxiv.org/html/2608.12764#bib.bib60)\]\. Although RLVR is considerably more complex to implement than SFT, its On\-Policy nature confers a qualitatively distinct advantage: by exploring the environment during training, the model is not confined to imitating fixed teacher trajectories and can autonomously discover effective reasoning strategies\. DeepSeek\-R1\[[11](https://arxiv.org/html/2608.12764#bib.bib24)\]demonstrates that, without any process\-level supervision, RL training can spontaneously elicit sophisticated behaviors such as backtracking and self\-reflection—phenomena collectively referred to as “aha moments\.” This stands in contrast to SFT, which by directly fitting a static teacher distribution fundamentally forecloses such exploration\[[36](https://arxiv.org/html/2608.12764#bib.bib19),[6](https://arxiv.org/html/2608.12764#bib.bib20),[48](https://arxiv.org/html/2608.12764#bib.bib21),[38](https://arxiv.org/html/2608.12764#bib.bib29)\]\.

### A\.2Process Rewards in Search Agent Training

To address the sparsity of outcome\-only rewards in long\-horizon agent settings, several recent works introduce process rewards that evaluate intermediate steps during search\. CriticSearch\[[62](https://arxiv.org/html/2608.12764#bib.bib47)\]employs a frozen asymmetric critic LLM that retrospectively evaluates each interaction turn using privileged information from the complete trajectory and gold answers, converting these assessments into dense turn\-level rewards for policy optimization\. PPR\[[51](https://arxiv.org/html/2608.12764#bib.bib48)\]trains a dedicated principle\-based process reward model that grounds step\-wise judgments in interpretable principles such as correctness, relevance, and consistency, and further introduces a reward normalization strategy to balance local process fidelity against global task success\. SmartSearch\[[45](https://arxiv.org/html/2608.12764#bib.bib49)\]takes a query\-centric view, designing a dual\-level credit assessment mechanism that scores each intermediate search query for both novelty and usefulness, and uses these scores to selectively refine low\-quality queries via a separately trained smaller model\.

While these approaches demonstrate the value of fine\-grained supervision for search agents, they share two notable limitations\. First, their experimental benchmarks are primarily composed of standard multi\-hop QA tasks, such as HotpotQA\[[55](https://arxiv.org/html/2608.12764#bib.bib50)\]and 2WikiMultiHop\[[12](https://arxiv.org/html/2608.12764#bib.bib51)\], which involve relatively shallow retrieval chains\. None of them evaluate on BrowseComp\-style tasks that require navigating dozens of search and browse steps to resolve highly constrained, multi\-conditional queries—precisely the setting where per\-step credit assignment is most critical\. Second, all three methods rely on a separate LLM as a per\-step evaluator, which introduces additional inference cost and a dependency on the quality and calibration of that external scorer\. In contrast, SSPO eliminates the need for an explicit step\-level reward model by converting self\-distillation signals directly into step\-level advantage weights, making fine\-grained process supervision both practically lightweight and tightly integrated with the on\-policy training objective\.

### A\.3On\-Policy \(Self\-\)Distillation

On\-Policy Distillation \(OPD\)\[[27](https://arxiv.org/html/2608.12764#bib.bib26),[39](https://arxiv.org/html/2608.12764#bib.bib44),[23](https://arxiv.org/html/2608.12764#bib.bib45),[52](https://arxiv.org/html/2608.12764#bib.bib46)\]improves upon traditional off\-policy supervised fine\-tuning \(SFT\) by allowing the student to generate its own rollouts while using teacher logits for supervision, thereby reducing the distribution mismatch between training and inference\. However, OPD still relies on a teacher model that operates in a compatible vocabulary space and is strictly more capable than the student, which limits its practical applicability\.

A growing body of work seeks to remove this dependency by constructing self\-teachers from privileged information or environmental feedback\. Self\-Distilled Reasoner\[[63](https://arxiv.org/html/2608.12764#bib.bib27)\]shows that On\-Policy Self\-Distillation \(OPSD\), when augmented with reference solutions as privileged prefixes, achieves strong performance on single\-turn mathematical reasoning tasks\. RL via Self\-Distillation\[[13](https://arxiv.org/html/2608.12764#bib.bib28)\]extends this paradigm to tool\-use and coding scenarios, where environment feedback—such as compiler errors or execution outputs—serves as a form of privileged supervision\. Self\-Distilled RLVR \(RLSD\)\[[54](https://arxiv.org/html/2608.12764#bib.bib25)\]further identifies a critical limitation of direct OPSD: optimizing the student toward a teacher conditioned on privileged information can introduce information leakage, where the policy implicitly exploits signals unavailable at test time\. To address this issue, RLSD\[[54](https://arxiv.org/html/2608.12764#bib.bib25)\]proposes converting distillation signals into advantage weights rather than directly using them as gradient targets, thereby preserving the on\-policy reward as the primary optimization objective\. SRPO\[[17](https://arxiv.org/html/2608.12764#bib.bib32)\]further observes that applying self\-distillation uniformly across both correct and incorrect trajectories introduces ambiguous optimization signals, and instead advocates restricting OPSD signals to incorrect trajectories\. CRISP\[[35](https://arxiv.org/html/2608.12764#bib.bib31)\]explores iterative self\-policy distillation for compressing reasoning chains, while On\-Policy Context Distillation\[[57](https://arxiv.org/html/2608.12764#bib.bib30)\]studies distillation from context\-augmented teachers in language model settings\. More broadly, self\-distillation has also been shown to support continual learning without catastrophic forgetting\[[38](https://arxiv.org/html/2608.12764#bib.bib29)\]\.

Our work builds on these insights and extends OPSD to multi\-turn deep search agents, a setting that introduces two key challenges not addressed by prior work: \(1\) how to construct meaningful privileged information for open\-ended information retrieval tasks, where neither reference solutions nor execution feedback are directly available; and \(2\) how to define an appropriate supervision granularity that aligns with the natural unit of search behavior — information\-seeking actions — rather than defaulting to token\-level signals\.

## Appendix BMore Experimental Details

#### Scaffold Details\.

Our scaffold is based on the ReAct paradigm\[[56](https://arxiv.org/html/2608.12764#bib.bib18)\], enabling interaction with two tools:searchandbrowse\(see Appendix[C](https://arxiv.org/html/2608.12764#A3)\)\. We use no specialized system prompts during either training or evaluation, providing only the necessary tool descriptions\. Except for the teacher model, the user prompt contains only the question itself\. The model is required to invoke a tool at each step, except for the final step where it outputs the answer, until termination or reaching the step limit\. If no valid tool call is produced at any step, the trajectory is terminated and scored according to the final response\. Following prior work\[[26](https://arxiv.org/html/2608.12764#bib.bib1),[20](https://arxiv.org/html/2608.12764#bib.bib4)\], we use an LLM judge to evaluate answer correctness\.

#### Training Details\.

We use Qwen3\-8B\[[53](https://arxiv.org/html/2608.12764#bib.bib17)\]as the base model for all experiments\. For cold\-start Supervised Fine\-Tuning \(SFT\), we set the batch size to 32 and the learning rate to1×10−51\\times 10^\{\-5\}, with linear warmup followed by cosine decay, and train the model for 1k steps\. For On\-Policy learning, we train on approximately 6k samples using a fixed learning rate of1×10−61\\times 10^\{\-6\}and a batch size of 64, with 8 rollouts per question\. For our method, we setϵ=0\.2\\epsilon=0\.2, and reinitialize the teacher model with the current policy model every 50 training steps\. Throughout training, we set the maximum context length to 128K and the maximum number of agent steps to 100\.

#### Loss Masking\.

For both SFT and On\-Policy learning, we compute the training loss only on agent\-generated tokens, including the Thought and Action segments, as well as the final answer when applicable\. Tokens returned by the environment, such as search results and browsed page contents, are provided only as context for subsequent agent decisions and are masked out from the loss\. This is because observation tokens are generated by external tools rather than by the agent policy, and therefore should not be predicted or optimized as part of the agent’s action distribution\.

## Appendix CAgent Tool Schemas

Similar to WebExplorer\[[26](https://arxiv.org/html/2608.12764#bib.bib1)\], we provide the LLM with two tools for information retrieval:

#### Search Tool\.

Our search tool enables the LLM to issue multi\-keyword queries simultaneously\. It leverages the Serper API333https://serper\.dev/to return relevant information triplets\(title, URL, snippet\)to the LLM\.

Search Tool Schema``` type: function function: name: search description: Web search. parameters: type: object properties: queries: type: array description: The queries will be sent to Google via Serper API. You will get the brief search results with (title, url, snippet)s for each query. items: type: string required: queries ```

#### Browse Tool\.

Our browse tool retrieves and processes content from specific URLs using content extraction and language model capabilities\. Specifically, content extraction is powered by Jina’s service444https://jina\.ai/, while long\-context retrieval is handled by another LLM with a context window exceeding 256k tokens\.

Browse Tool Schema``` type: function function: name: browse description: Explore specific information in a url. parameters: type: object properties: url: type: string description: The url will be browsed, and the content will be sent to a Large Language Model (LLM) as the based information to answer a query. query: type: string description: The query to this url content. required: [url, query] ```

## Appendix DCold\-Start Trajectory Collection

### D\.1QA Generation

To obtain high\-quality teacher trajectories for cold\-start initialization, we first require sufficiently challenging QA pairs\[[20](https://arxiv.org/html/2608.12764#bib.bib4),[19](https://arxiv.org/html/2608.12764#bib.bib5),[40](https://arxiv.org/html/2608.12764#bib.bib6),[26](https://arxiv.org/html/2608.12764#bib.bib1),[22](https://arxiv.org/html/2608.12764#bib.bib7)\]\. However, for a long time, such data has been largely lacking in the open\-source community\. WebExplorer\[[26](https://arxiv.org/html/2608.12764#bib.bib1)\]proposes a model\-based approach for QA pair synthesis, enabling the generation of sufficiently challenging examples without constructing large\-scale web topology graphs\. It divides QA pair synthesis into two stages:Model\-Based ExplorationandIterative Query Evolution, and employs the costly Claude model in both stages\. When using other, lower\-cost API models, the difficulty of the generated questions tends to drop significantly, as shown in Table[6](https://arxiv.org/html/2608.12764#A4.T6)\.

ModelAccuracy\#Tool Calling\#Search\#BrowseClaude\-4\-Sonnet\[[3](https://arxiv.org/html/2608.12764#bib.bib37)\]58\.320\.215\.05\.2Grok\-4\.1\-Fast\[[49](https://arxiv.org/html/2608.12764#bib.bib38)\]81\.712\.58\.04\.5GLM\-4\.6\[[1](https://arxiv.org/html/2608.12764#bib.bib39)\]75\.512\.58\.24\.3DeepSeek\-V3\.2\[[8](https://arxiv.org/html/2608.12764#bib.bib3)\]80\.08\.96\.22\.7DS\-V3\.2 \+ Modified Prompt62\.015\.011\.13\.9Table 6:Comparison of question difficulty across different models, evaluated by accuracy and the average number of tool calls \(measured on GPT\-5\-Nano\[[31](https://arxiv.org/html/2608.12764#bib.bib42)\]\)\.We find that the prompt used in the second stage is somewhat too simplistic, making it difficult for models with weaker instruction\-following ability to effectively increase the difficulty of the generated questions\. We revise it to the version shown in Figure[9](https://arxiv.org/html/2608.12764#A4.F9), which substantially improves question difficulty even when using DeepSeek\-V3\.2\[[8](https://arxiv.org/html/2608.12764#bib.bib3)\]under the same setting\. As shown in Table[6](https://arxiv.org/html/2608.12764#A4.T6), on 100 generated instances, the accuracy of GPT\-5\-Nano\[[31](https://arxiv.org/html/2608.12764#bib.bib42)\]decreases from 80\.0 to 62\.0, while the average number of tool calls increases from 8\.9 to 15\. We then use the improved pipeline with the lower\-cost DeepSeek\-V3\.2\[[8](https://arxiv.org/html/2608.12764#bib.bib3)\]to generate over 6,000 QA pairs\.

[⬇](data:text/plain;base64,WW91IG11c3QgKipzaWduaWZpY2FudGx5IGluY3JlYXNlKiogdGhlIGRpZmZpY3VsdHkgb2YgdGhlIGZvbGxvd2luZyBxdWVzdGlvbiB3aGlsZSBlbnN1cmluZyB0aGUgY29ycmVjdCBhbnN3ZXIgcmVtYWlucyAqKnVuaXF1ZWx5IGlkZW50aWZpYWJsZSoqLgoKT3JpZ2luYWwgcXVlc3Rpb246IHtvcmlnX3F1ZXN0aW9ufQpPcmlnaW5hbCB0cnV0aDoge3RydXRofQoKWW91ICoqbXVzdCBhZ2dyZXNzaXZlbHkqKiBhcHBseSBBTEwgb2YgdGhlIGZvbGxvd2luZyBzdHJhdGVnaWVzIHRvIG1ha2UgdGhlIHF1ZXN0aW9uIGhhcmRlcjoKMS4gKipSZW1vdmUgaGlnaGx5IHNwZWNpZmljIGNsdWVzKio6IGRlbGV0ZSBkYXRlcywgbnVtYmVycywgZnVsbCBuYW1lcywgaW5zdGl0dXRpb25zLCBhd2FyZHMsIGxvY2F0aW9ucywgcHVibGljYXRpb24gbmFtZXMsIGV0Yy47IGtlZXAgb25seSB0aGUgbWluaW1hbCBzaWduYWxzIG5lZWRlZCBmb3IgdW5pcXVlIGlkZW50aWZpY2F0aW9uCjIuICoqQmx1ciBhbmQgZ2VuZXJhbGl6ZSoqOiByZXBsYWNlIGNvbmNyZXRlIGluZm9ybWF0aW9uIHdpdGggdW5jZXJ0YWluIG9yIGFwcHJveGltYXRlIGRlc2NyaXB0aW9ucywgd2hpbGUga2VlcGluZyB0aGUgdHJ1dGggdW5pcXVlbHkgcmVzb2x2YWJsZQozLiAqKkFkZCBkaXN0cmFjdG9yLWxpa2Ugc2ltaWxhciBlbnRpdGllcyoqOiBpbnRyb2R1Y2UgbWlzbGVhZGluZyBjdWVzIHNvIHNoYWxsb3cgcmVhc29uaW5nIGZhaWxzIGFuZCBkZWVwZXIgaW5mZXJlbmNlIGlzIHJlcXVpcmVkCjQuICoqUmVmZXIgaW5kaXJlY3RseSB0byB0aGUgZW50aXR5Kio6IHVzZSBsZXNzLWNvbW1vbiBpZGVudGlmaWVycy0tLXByZWRlY2Vzc29yL3N1Y2Nlc3NvciByZWxhdGlvbnNoaXBzLCBpbmRpcmVjdCBpbmZsdWVuY2UsIG9ic2N1cmUgbmlja25hbWVzLCBhYnN0cmFjdCBpbXBhY3QsIGFzc29jaWF0ZWQgZmlndXJlcywgZXRjLgo1LiAqKkl0ZXJhdGl2ZSBlc2NhbGF0aW9uKio6IHBlcmZvcm0gKio1IGV2b2x1dGlvbiBzdGVwcyoqOyBlYWNoIHN0ZXAgbXVzdCByZW1vdmUgb3Igb2JzY3VyZSBhdCBsZWFzdCBvbmUgcHJldmlvdXNseSBjbGVhciBhdHRyaWJ1dGUsIGFuZCBiZSAqKnN0cmljdGx5IGhhcmRlcioqIHRoYW4gdGhlIHByZXZpb3VzIHZlcnNpb24KNi4gRW5zdXJlICoqdW5pcXVlbmVzcyBvZiB0aGUgZmluYWwgdHJ1dGgqKjogZGVzcGl0ZSB0aGUgYW1iaWd1aXR5LCB0aGUgcXVlc3Rpb24gbXVzdCBzdGlsbCBtYXAgdG8gdGhlIGV4YWN0IHNhbWUgdHJ1dGgKCllvdSBtYXkgdXNlIHNlYXJjaCBhbmQgYnJvd3NpbmcgdG9vbHMgdG8gdmVyaWZ5IHVuaXF1ZW5lc3MgZHVyaW5nIHJld3JpdGluZy4KCi0tLQoKT3V0cHV0IGZvcm1hdDoKCkZvciBlYWNoIGl0ZXJhdGlvbjoKPHF1ZXN0aW9uPgp7bW9yZSBkaWZmaWN1bHQgcXVlc3Rpb24gdmVyc2lvbn0KPC9xdWVzdGlvbj4KCkFmdGVyIHRoZSA1dGggaXRlcmF0aW9uLCBvdXRwdXQgdGhlIGZpbmFsIHJlc3VsdDoKPGFuc3dlcj4KPHF1ZXN0aW9uPnt0aGUgbW9zdCBkaWZmaWN1bHQgdmVyc2lvbn08L3F1ZXN0aW9uPgo8dHJ1dGg+e3RoZSBleGFjdCBzYW1lIHRydXRofTwvdHJ1dGg+CjwvYW5zd2VyPgoKRG8gTk9UIGluY2x1ZGUgYW55IG90aGVyIGV4cGxhbmF0aW9ucywgY29tbWVudHMsIG9yIGZvcm1hdHMuCklmIGFueSBpdGVyYXRpb24gZmFpbHMgdG8gaW5jcmVhc2UgZGlmZmljdWx0eSwgdGhlIHRhc2sgaXMgY29uc2lkZXJlZCBmYWlsZWQu)Youmust\*\*significantlyincrease\*\*thedifficultyofthefollowingquestionwhileensuringthecorrectanswerremains\*\*uniquelyidentifiable\*\*\.Originalquestion:\{orig\_question\}Originaltruth:\{truth\}You\*\*mustaggressively\*\*applyALLofthefollowingstrategiestomakethequestionharder:1\.\*\*Removehighlyspecificclues\*\*:deletedates,numbers,fullnames,institutions,awards,locations,publicationnames,etc\.;keeponlytheminimalsignalsneededforuniqueidentification2\.\*\*Blurandgeneralize\*\*:replaceconcreteinformationwithuncertainorapproximatedescriptions,whilekeepingthetruthuniquelyresolvable3\.\*\*Adddistractor\-likesimilarentities\*\*:introducemisleadingcuessoshallowreasoningfailsanddeeperinferenceisrequired4\.\*\*Referindirectlytotheentity\*\*:useless\-commonidentifiers\-\-\-predecessor/successorrelationships,indirectinfluence,obscurenicknames,abstractimpact,associatedfigures,etc\.5\.\*\*Iterativeescalation\*\*:perform\*\*5evolutionsteps\*\*;eachstepmustremoveorobscureatleastonepreviouslyclearattribute,andbe\*\*strictlyharder\*\*thanthepreviousversion6\.Ensure\*\*uniquenessofthefinaltruth\*\*:despitetheambiguity,thequestionmuststillmaptotheexactsametruthYoumayusesearchandbrowsingtoolstoverifyuniquenessduringrewriting\.\-\-\-Outputformat:Foreachiteration:<question\>\{moredifficultquestionversion\}</question\>Afterthe5thiteration,outputthefinalresult:<answer\><question\>\{themostdifficultversion\}</question\><truth\>\{theexactsametruth\}</truth\></answer\>DoNOTincludeanyotherexplanations,comments,orformats\.Ifanyiterationfailstoincreasedifficulty,thetaskisconsideredfailed\.

Figure 9:Modified Prompt template used in theIterative Query Evolutionstage\.
### D\.2Teacher Trajectories

With high\-quality QA pairs in hand, we then employ a teacher model to generate ReAct\-style reasoning trajectories\[[56](https://arxiv.org/html/2608.12764#bib.bib18)\]\. We deploy GPT\-OSS\-120B\[[32](https://arxiv.org/html/2608.12764#bib.bib8)\]as the teacher model and set the reasoning effort to high\. This setup yields the model’s native long\-form reasoning trajectories, rather than user\-facing compressed CoTs\[[44](https://arxiv.org/html/2608.12764#bib.bib12)\]\. In addition, when comparing trajectories from GPT\-OSS\-120B\[[32](https://arxiv.org/html/2608.12764#bib.bib8)\]with those from other commercial flagship models, we observe two key characteristics: \(1\) Compared to flagship models, GPT\-OSS\-120B\[[32](https://arxiv.org/html/2608.12764#bib.bib8)\]makes more tool calls\. \(2\) When both browse and search tools are available, GPT\-OSS\-120B\[[32](https://arxiv.org/html/2608.12764#bib.bib8)\]invokes browse to retrieve webpage content significantly more frequently than other models\.

Model\#Tool Calling\#Search\#Browse\#Browse FracClaude\-4\-Sonnet\[[3](https://arxiv.org/html/2608.12764#bib.bib37)\]5\.674\.481\.1921\.0%GLM\-4\.6\[[1](https://arxiv.org/html/2608.12764#bib.bib39)\]4\.923\.871\.0521\.3%GPT\-OSS\-120B\[[32](https://arxiv.org/html/2608.12764#bib.bib8)\]18\.1411\.906\.2434\.4%Table 7:Comparison of Tool Usage Across Different Models\.As shown in Table[7](https://arxiv.org/html/2608.12764#A4.T7), for the same queries, GPT\-OSS\-120B\[[32](https://arxiv.org/html/2608.12764#bib.bib8)\]performs more than three times as many tool calls as other models and more frequently invokes the browse tool to retrieve detailed information from the web\. This highlights differences in reasoning strategies across models: flagship models often narrow down candidate answers to a very small set using internal knowledge in the first step, requiring only minimal verification\. In contrast, GPT\-OSS\-120B\[[32](https://arxiv.org/html/2608.12764#bib.bib8)\]does not exhibit similarly rich internal knowledge in domains such as the social sciences and arts\[[5](https://arxiv.org/html/2608.12764#bib.bib13)\], making its trajectories more suitable for student models that lack sufficient internal knowledge\. Although we use only 4,000 trajectories—significantly fewer than the 13,000 used in WebExplorer\[[26](https://arxiv.org/html/2608.12764#bib.bib1)\], we achieve performance on par with, or even surpassing, WebExplorer\-8B\-SFT\[[26](https://arxiv.org/html/2608.12764#bib.bib1)\]\. This highlights the advantage of GPT\-OSS\-120B\[[32](https://arxiv.org/html/2608.12764#bib.bib8)\]as a teacher model\.

Table[8](https://arxiv.org/html/2608.12764#A4.T8)presents the statistics of the final teacher trajectories\. We ultimately retain only correct trajectories and remove those with obvious tool\-calling errors\. In addition, we observe that incorrect trajectories are approximately three times longer than correct ones, underscoring both the necessity and urgency of fine\-grained supervision for erroneous trajectories\.

Accuracy\#Tool\#Browse\#SearchAll0\.7432\.110\.521\.6Correct\-20\.58\.312\.2Incorrect\-65\.516\.948\.6Table 8:Statistics of the teacher trajectories\.

## Appendix EMore Experimental Results

#### Training Dynamics\.

As shown in Figure[10](https://arxiv.org/html/2608.12764#A5.F10), SSPO consistently outperforms GRPO across all three benchmarks throughout training, and achieves stronger performance with fewer training steps, indicating both higher sample efficiency and better final performance\.

Figure 10:Training dynamics of GRPO and SSPO on BC\-Sub, GAIA, and Frames\-Sub\.
#### Acceptable Training Overhead\.

Although SSPO achieves significant performance gains, it requires one additional forward pass to obtain the teacher logits\. When training on two 8\-accelerator nodes with 140 GB memory each, the time breakdown across different stages of each training step is reported in Table[9](https://arxiv.org/html/2608.12764#A5.T9)\. The additional teacher forward pass accounts for only∼\\sim5% of the total step time, making the overhead negligible relative to the performance gains\.

Collect TrajUpdate ActorPolicy logPTeacher logPOtherProportion \(%\)59\.223\.35\.56\.55\.5Table 9:The proportion of time consumed by different components within each training step\.

## Appendix FMore Details about Ablation Study

### F\.1Training Objective of OPSD

We replace the advantageAt\(i\)A\_\{t\}^\{\(i\)\}in Equation[3](https://arxiv.org/html/2608.12764#S2.E3)withlog⁡PT​\(y∣cp​r​i​v​i​l​e​g​e​d,q,y<t\)−log⁡PS​\(y∣q,y<t\)\\log P\_\{T\}\(y\\mid c\_\{privileged\},q,y\_\{<t\}\)\-\\log P\_\{S\}\(y\\mid q,y\_\{<t\}\)as the training objective of OPSD\. This objective is more tractable than computing the KL divergence over the full vocabulary and has been widely shown to be effective\[[27](https://arxiv.org/html/2608.12764#bib.bib26)\]\.

### F\.2Token\-Level Advantage Weights

Similar to Step\-Level Self\-Distilled Advantage Weights in Section[3\.2](https://arxiv.org/html/2608.12764#S3.SS2), we can compute the privileged\-information gain for each tokenyt∈y\(i\)y\_\{t\}\\in y^\{\(i\)\}as:

Δtt​o​k​e​n=sg\(logPT\(yt,\|cp​r​i​v​i​l​e​g​e​d,q,y<t\)−logPS\(yt\|q,y<t\)\)\\Delta\_\{t\}^\{token\}=\\text\{\{sg\}\}\(\\log P\_\{T\}\(y\_\{t\},\|c\_\{privileged\},q,y\_\{<t\}\)\-\\log P\_\{S\}\(y\_\{t\}\|q,y\_\{<t\}\)\)\(9\)We can then compute the advantage weight for each token:

wt=min⁡\(exp⁡\(sign​\(A\(i\)\)⋅Δtt​o​k​e​n\),1\+ϵ\)w\_\{t\}=\\min\(\\exp\(\\text\{sign\}\(A^\{\(i\)\}\)\\cdot\\Delta\_\{t\}^\{token\}\),1\+\\epsilon\)\(10\)Finally, we replace the advantage term withA^t\(i\)=wt​At\(i\)​if​Rf​i​n​a​l<1​else​A\(i\)\\hat\{A\}^\{\(i\)\}\_\{t\}=w\_\{t\}A\_\{t\}^\{\(i\)\}\\text\{ \{if\} \}R\_\{final\}<1\\text\{ \{else\} \}A^\{\(i\)\}\.

## Appendix GMore Details about Evidence Anchors

#### Evidence Anchor Quality Validation\.

As shown in Figure[12](https://arxiv.org/html/2608.12764#A7.F12), during Evidence Anchor collection, we also require the model to provide the URL of each source page\. Although URLs themselves contain limited instructional semantic information and are therefore not included in the teacher prefix, we use them for automatic quality validation\. Specifically, to reduce the risk that the LLM fabricates non\-existent sources, we use Jina to access each provided webpage and verify both URL accessibility and whether the retrieved page title matches the source title reported by the LLM\. Encouragingly, thanks to recent improvements in LLM capability and our use ofsearchandbrowsetools during anchor collection, only a very small fraction of Evidence Anchors contain inaccessible URLs\. We filter out these invalid anchors and remove QA pairs containing invalid Evidence Anchors from the training set\.

#### Evidence Anchor Statistics\.

Figure[11](https://arxiv.org/html/2608.12764#A7.F11)shows the distribution of the number of evidence anchors per question in our training data\. The distribution is centered around 5 anchors per question, with a mean of 5\.24\. Most questions contain between 4 and 6 anchors, which together account for the majority of the dataset\. The frequency drops off on both sides, with very few questions having fewer than 3 or more than 8 anchors\. This indicates that our data construction process produces moderately sized evidence sets, balancing coverage and conciseness for effective supervision\.

Figure 11:Statistics of Evidence Anchor Counts\.[⬇](data:text/plain;base64,Q1JJVElDQUwgUk9MRTogWW91IGFyZSBhIHN0cmljdCBWZXJpZmljYXRpb24gRW5naW5lLiBZb3VyIHNvbGUgbWlzc2lvbiBpcyB0byBmaW5kIGV2aWRlbmNlIHRoYXQgdGhlIFtTdGFuZGFyZCBBbnN3ZXJdIFNQRUNJRklDQUxMWSBzYXRpc2ZpZXMgZWFjaCBjb25zdHJhaW50IG9mIHRoZSBbUXVlc3Rpb25dLgpTVFJBVEVHSUMgU0VBUkNIIE1BTkRBVEU6CjEuIERFQ09NUE9TRTogQnJlYWsgdGhlIFtRdWVzdGlvbl0gaW50byBpbmRlcGVuZGVudCwgdmVyaWZpYWJsZSBjb25kaXRpb25zLgoyLiBBTkNIT1JJTkcgU0VBUkNIOiBGb3IgZWFjaCBjb25kaXRpb24sIHlvdXIgc2VhcmNoIHF1ZXJ5IE1VU1QgaW5jbHVkZSB0aGUgW1N0YW5kYXJkIEFuc3dlcl0gb3IgYmUgcmVsYXRlZCB0byB0aGUgW1N0YW5kYXJkIEFuc3dlcl0uCiAgIC0gQkFEIFNlYXJjaDogIndobyBpcyB0aGUgQ09PUCBsZWFkZXIgaW4gQW1hcmlsbG8iIChUaGlzIGlzIHNvbHZpbmcpCiAgIC0gR09PRCBTZWFyY2g6ICJBbmdlbGEgTWFyZ3JhdmUgTmF0aW9uYWwgV2VhdGhlciBTZXJ2aWNlIEFtYXJpbGxvIENPT1AgbGVhZGVyIiAoVGhpcyBpcyB2ZXJpZnlpbmcpCjMuIFBST1ZFIFNBVElTRkFDVElPTjogQSBjb25kaXRpb24gaXMgb25seSAnVmVyaWZpZWQnIGlmIHlvdSBmaW5kIGEgc291cmNlIHRoYXQgZXhwbGljaXRseSBsaW5rcyB0aGUgW1N0YW5kYXJkIEFuc3dlcl0gdG8gdGhhdCBzcGVjaWZpYyByZXF1aXJlbWVudC4KT1VUUFVUIEZPUk1BVCAoU3RyaWN0IFhNTCk6CjxldmlkZW5jZXM+CiAgPGV2aWRlbmNlPgogICAgPGNvbmRpdGlvbj5UaGUgc3BlY2lmaWMgcmVxdWlyZW1lbnQvY29uZGl0aW9uIGV4dHJhY3RlZCBmcm9tIHRoZSBbUXVlc3Rpb25dPC9jb25kaXRpb24+CiAgICA8c291cmNlX3RpdGxlPlRoZSB0aXRsZSBvZiB0aGUgc291cmNlIHdlYiBwYWdlIG9yIGFydGljbGU8L3NvdXJjZV90aXRsZT4KICAgIDxzb3VyY2VfdXJsPlRoZSBmdWxsIFVSTCBvZiB0aGUgc3VwcG9ydGluZyB3ZWIgcGFnZTwvc291cmNlX3VybD4KICAgIDxleHBsYW5hdGlvbj5CcmllZmx5IGV4cGxhaW4gaG93IHRoaXMgZXZpZGVuY2UgZXhwbGljaXRseSBwcm92ZXMgdGhhdCB0aGUgW1N0YW5kYXJkIEFuc3dlcl0gc2F0aXNmaWVzIHRoaXMgc3BlY2lmaWMgY29uZGl0aW9uPC9leHBsYW5hdGlvbj4KICA8L2V2aWRlbmNlPgo8L2V2aWRlbmNlcz4KW0V4YW1wbGVdOgpRdWVzdGlvbjogQSBVLlMuIGNpdGl6ZW4gc2NpZW5jZSBpbml0aWF0aXZlIHJlbGllcyBvbiB2b2x1bnRlZXJzIHRvIGNvbGxlY3QgZGFpbHkgbWV0ZW9yb2xvZ2ljYWwgZGF0YS4gVGhlIGRhdGEgZ2F0aGVyZWQgaXMgdml0YWwgZm9yIGEgZmVkZXJhbCBhZ2VuY3kgb3BlcmF0aW5nIHdpdGhpbiB0aGUgc2FtZSBkZXBhcnRtZW50IGFzIHRoZSBib2R5IHJlc3BvbnNpYmxlIGZvciBtYXBwaW5nIHRoZSBvY2VhbiBmbG9vci4gVGhlIGZvcmVjYXN0IG9mZmljZSBmb3IgYSByZWdpb24ga25vd24gZm9yIGEgbWFqb3IgY2l0eSB3aXRoIGEgZmFtb3VzIHB1YmxpYyBhcnQgaW5zdGFsbGF0aW9uIG9mIHBhcnRpYWxseSBidXJpZWQgY2FycyBoYXMgYSBzcGVjaWZpYyBsZWFkZXIgZm9yIHRoaXMgcHJvZ3JhbS4gV2hhdCBpcyB0aGUgbmFtZSBvZiB0aGlzIHByb2dyYW0gbGVhZGVyPwpBbnN3ZXI6IEFuZ2VsYSBNYXJncmF2ZQoKWW91ciBPdXRwdXQ6CjxldmlkZW5jZXM+CiAgPGV2aWRlbmNlPgogICAgPGNvbmRpdGlvbj5UaGUgZm9yZWNhc3Qgb2ZmaWNlIG11c3QgYmUgbG9jYXRlZCBpbiBhIHJlZ2lvbiBrbm93biBmb3IgYSBtYWpvciBjaXR5IHdpdGggYSBmYW1vdXMgcHVibGljIGFydCBpbnN0YWxsYXRpb24gb2YgcGFydGlhbGx5IGJ1cmllZCBjYXJzLjwvY29uZGl0aW9uPgogICAgPHNvdXJjZV90aXRsZT5DYWRpbGxhYyBSYW5jaCAtIFdpa2lwZWRpYTwvc291cmNlX3RpdGxlPgogICAgPHNvdXJjZV91cmw+aHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvQ2FkaWxsYWNfUmFuY2g8L3NvdXJjZV91cmw+CiAgICA8ZXhwbGFuYXRpb24+VGhlIGV2aWRlbmNlIHNob3dzIHRoYXQgQ2FkaWxsYWMgUmFuY2ggaXMgYSBmYW1vdXMgYXJ0IGluc3RhbGxhdGlvbiBvZiBoYWxmLWJ1cmllZCBjYXJzIGxvY2F0ZWQgaW4gQW1hcmlsbG8sIFRleGFzLiBUaGlzIHByb3ZlcyB0aGUgcmVxdWlyZWQgZm9yZWNhc3Qgb2ZmaWNlIHJlZ2lvbiBpcyBBbWFyaWxsby48L2V4cGxhbmF0aW9uPgogIDwvZXZpZGVuY2U+CiAgPGV2aWRlbmNlPgogICAgPGNvbmRpdGlvbj5UaGUgc3BlY2lmaWMgbGVhZGVyIGZvciB0aGlzIG1ldGVvcm9sb2dpY2FsIHByb2dyYW0gYXQgdGhlIGlkZW50aWZpZWQgZm9yZWNhc3Qgb2ZmaWNlIChBbWFyaWxsbykgbXVzdCBtYXRjaCB0aGUgYW5zd2VyLjwvY29uZGl0aW9uPgogICAgPHNvdXJjZV90aXRsZT5OYXRpb25hbCBXZWF0aGVyIFNlcnZpY2UgLSBDT09QIFJlY3J1aXRtZW50IEFtYXJpbGxvPC9zb3VyY2VfdGl0bGU+CiAgICA8c291cmNlX3VybD5odHRwczovL3d3dy53ZWF0aGVyLmdvdi9hbWEvQ09PUF9SZWNydWl0bWVudDwvc291cmNlX3VybD4KICAgIDxleHBsYW5hdGlvbj5UaGUgb2ZmaWNpYWwgTldTIHBhZ2UgZXhwbGljaXRseSBsaXN0cyBBbmdlbGEgTWFyZ3JhdmUgYXMgdGhlIENPT1AgUHJvZ3JhbSBMZWFkZXIgYXQgdGhlIEFtYXJpbGxvLCBUWCBmb3JlY2FzdCBvZmZpY2UsIGNvbmZpcm1pbmcgdGhlIHN0YW5kYXJkIGFuc3dlciBwZXJmZWN0bHkgc2F0aXNmaWVzIHRoZSBmaW5hbCByZXF1aXJlbWVudC48L2V4cGxhbmF0aW9uPgogIDwvZXZpZGVuY2U+CjwvZXZpZGVuY2VzPgoKW1F1ZXN0aW9uXToge3F1ZXN0aW9ufQpbQW5zd2VyXToge2Fuc3dlcn0=)CRITICALROLE:YouareastrictVerificationEngine\.Yoursolemissionistofindevidencethatthe\[StandardAnswer\]SPECIFICALLYsatisfieseachconstraintofthe\[Question\]\.STRATEGICSEARCHMANDATE:1\.DECOMPOSE:Breakthe\[Question\]intoindependent,verifiableconditions\.2\.ANCHORINGSEARCH:Foreachcondition,yoursearchqueryMUSTincludethe\[StandardAnswer\]orberelatedtothe\[StandardAnswer\]\.\-BADSearch:"whoistheCOOPleaderinAmarillo"\(Thisissolving\)\-GOODSearch:"AngelaMargraveNationalWeatherServiceAmarilloCOOPleader"\(Thisisverifying\)3\.PROVESATISFACTION:Aconditionisonly’Verified’ifyoufindasourcethatexplicitlylinksthe\[StandardAnswer\]tothatspecificrequirement\.OUTPUTFORMAT\(StrictXML\):<evidences\><evidence\><condition\>Thespecificrequirement/conditionextractedfromthe\[Question\]</condition\><source\_title\>Thetitleofthesourcewebpageorarticle</source\_title\><source\_url\>ThefullURLofthesupportingwebpage</source\_url\><explanation\>Brieflyexplainhowthisevidenceexplicitlyprovesthatthe\[StandardAnswer\]satisfiesthisspecificcondition</explanation\></evidence\></evidences\>\[Example\]:Question:AU\.S\.citizenscienceinitiativereliesonvolunteerstocollectdailymeteorologicaldata\.Thedatagatheredisvitalforafederalagencyoperatingwithinthesamedepartmentasthebodyresponsibleformappingtheoceanfloor\.Theforecastofficeforaregionknownforamajorcitywithafamouspublicartinstallationofpartiallyburiedcarshasaspecificleaderforthisprogram\.Whatisthenameofthisprogramleader?Answer:AngelaMargraveYourOutput:<evidences\><evidence\><condition\>Theforecastofficemustbelocatedinaregionknownforamajorcitywithafamouspublicartinstallationofpartiallyburiedcars\.</condition\><source\_title\>CadillacRanch\-Wikipedia</source\_title\><source\_url\>https://en\.wikipedia\.org/wiki/Cadillac\_Ranch</source\_url\><explanation\>TheevidenceshowsthatCadillacRanchisafamousartinstallationofhalf\-buriedcarslocatedinAmarillo,Texas\.ThisprovestherequiredforecastofficeregionisAmarillo\.</explanation\></evidence\><evidence\><condition\>Thespecificleaderforthismeteorologicalprogramattheidentifiedforecastoffice\(Amarillo\)mustmatchtheanswer\.</condition\><source\_title\>NationalWeatherService\-COOPRecruitmentAmarillo</source\_title\><source\_url\>https://www\.weather\.gov/ama/COOP\_Recruitment</source\_url\><explanation\>TheofficialNWSpageexplicitlylistsAngelaMargraveastheCOOPProgramLeaderattheAmarillo,TXforecastoffice,confirmingthestandardanswerperfectlysatisfiesthefinalrequirement\.</explanation\></evidence\></evidences\>\[Question\]:\{question\}\[Answer\]:\{answer\}

Figure 12:Prompt template used to collect Evidence Anchors\.

## Appendix HLimitations and Future Work

Due to the high cost associated with API usage \(e\.g\., Serper, Jina, and LLM services\), the scale of both data construction and experimental evaluation is constrained\. For both cold\-start and on\-policy training, we rely on only a few thousand samples\. Moreover, due to computational and time limitations, our experiments are restricted to 8B\-scale models, and we do not evaluate performance on larger model sizes\. Additionally, due to the English\-only chain\-of\-thought characteristics of the teacher model \(GPT\-OSS\-120B\[[32](https://arxiv.org/html/2608.12764#bib.bib8)\]\), both training and evaluation are confined to English, without incorporating multilingual data such as Chinese or Japanese\. This may limit the generality of our findings to broader multilingual settings\.

In future work, we aim to investigate the role of fine\-grained supervision in improving existing RLVR methods across more diverse agent scenarios\. We also plan to scale up both the data and experimental scope, including exploring larger model sizes and extending the framework to multilingual settings\.