Planner-Centric Reinforcement Learning for Deep Research with Structure-Aware Reward

# Planner-Centric Reinforcement Learning for Deep Research with Structure-Aware Reward
Source: [https://arxiv.org/html/2605.30824](https://arxiv.org/html/2605.30824)
Mustafa Anis Hussain, Xinle Wu, Yao Lu National University of Singapore \{mustafah,wuxl,luyao\}@comp\.nus\.edu\.sg

###### Abstract

Deep research tasks require LLMs to plan what to investigate, retrieve evidence, and synthesize long\-form answers across multiple branches of inquiry\. Existing training paradigms either rely on short\-form verifiable QA as a proxy or optimize monolithic long trajectories, which makes planning and execution difficult to disentangle and yields weak credit assignment for the planning process\. We propose DecomposeR, a planner\-centric deep research framework that represents research plans as typed directed acyclic graphs \(DAGs\), allowing planning to be made explicit, structured, and rewardable\. We train a Qwen3\-8B model in two stages: planner reinforcement learning \(RL\) first learns graph structure and query decomposition to improve research planning, and answerer reinforcement learning \(RL\) then learns branch\-level execution and final synthesis conditioned on the learned plan\. By assigning rewards to explicit planner tokens and structured components rather than to a flat trajectory, DecomposeR enables finer\-grained optimization of planning while reducing the ambiguity of end\-to\-end training\. Experiments show that DecomposeR\-8B improves over strong comparable open baselines by 5\.1–8\.0 points on popular long\-form benchmarks due to improved planning and answering capabilities\.

DecomposeR: Planner\-Centric Reinforcement Learning for Deep Research with Structure\-Aware Reward

Mustafa Anis Hussain, Xinle Wu, Yao LuNational University of Singapore\{mustafah,wuxl,luyao\}@comp\.nus\.edu\.sg

## 1Introduction

Deep research \(DR\) requires planning what to investigate, issuing multiple searches, selecting evidence, and synthesizing findings into a citation\-grounded long\-form report\. Such workflows are increasingly important for scientific literature synthesis, clinical evidence aggregation, and general investigative tasks\(Asai et al\.,[2024a](https://arxiv.org/html/2605.30824#bib.bib2); Singh et al\.,[2025](https://arxiv.org/html/2605.30824#bib.bib23); Arora et al\.,[2025](https://arxiv.org/html/2605.30824#bib.bib1); Du et al\.,[2025](https://arxiv.org/html/2605.30824#bib.bib6)\)\. Frontier proprietary systems demonstrate the practical value of this workflow\(OpenAI,[2025](https://arxiv.org/html/2605.30824#bib.bib18); Comanici et al\.,[2025](https://arxiv.org/html/2605.30824#bib.bib5); Perplexity Team,[2025](https://arxiv.org/html/2605.30824#bib.bib19)\), while open alternatives either wrap capable LLMs in training\-free retrieval pipelines\(Li et al\.,[2026a](https://arxiv.org/html/2605.30824#bib.bib14),[b](https://arxiv.org/html/2605.30824#bib.bib15)\)or post\-train smaller models with reinforcement learning\(Jin et al\.,[2025](https://arxiv.org/html/2605.30824#bib.bib10); Song et al\.,[2025](https://arxiv.org/html/2605.30824#bib.bib24); Chen et al\.,[2026](https://arxiv.org/html/2605.30824#bib.bib4); Mei et al\.,[2025](https://arxiv.org/html/2605.30824#bib.bib17)\)\.

Despite this progress, current RL\-trained DR systems share two reward\-side limitations that bound their training signal\. First, credit assignment is ambiguous\. Reasoning, search, evidence selection, branch synthesis, and final writing are interleaved in a flat ReAct\-style trajectory\(Yao et al\.,[2023](https://arxiv.org/html/2605.30824#bib.bib28)\), and a single scalar reward derived from the final answer is propagated across every model action\. A weak final answer therefore cannot be attributed to a specific failure point so the gradient cannot direct improvement towards the responsible component\. Second, reward is sparse\. Intermediate planning, search, and synthesis decisions receive little to no direct supervision\(Jin et al\.,[2025](https://arxiv.org/html/2605.30824#bib.bib10); Song et al\.,[2025](https://arxiv.org/html/2605.30824#bib.bib24); Chen et al\.,[2026](https://arxiv.org/html/2605.30824#bib.bib4)\); every intermediate action is shaped only through its eventual effect on the terminal answer, which both slows learning and destabilizes credit propagation across long DR trajectories\.

![Refer to caption](https://arxiv.org/html/2605.30824v1/figures/framework_overview.png)Figure 1:DecomposeR rollout structure\. The planner emits an initial typed DAG, receives search results, revises the graph and fetch set, and then the answerer executes the revised graph in topological waves before writing the final answer\.We introduce DecomposeR, which addresses both limitations through a single design principle: structure\-aware reward modeling for deep research\. The research plan is materialized as an explicit typed DAG \(Figure[1](https://arxiv.org/html/2605.30824#S1.F1)\) in which search nodes issue web queries, aggregate nodes synthesize branch\-level conclusions, and a terminal answer node integrates branches into the final report\. Because every plan component is individually addressable, reward can be assigned to the corresponding part of the research process rather than only to the final report\. Planner rewards score rubric coverage over plan\-node fields, search quality over search nodes, and structural behavior such as branch breadth and evidence reuse; answerer rewards score branch\-level execution and final synthesis\. This design improves credit assignment by tying reward to the component responsible for each decision, and reduces sparsity by replacing a single trajectory\-end scalar with signals distributed across the decomposition\.

We evaluate DecomposeR on three long\-form benchmarks: DeepResearchBench, ResearchQA\-Mini, and HealthBench\. DecomposeR improves over strong comparable open baselines by 5\.1–8\.0 points despite a substantially smaller end\-to\-end training budget\.

Our contributions are: \(1\)Structure\-aware reward modeling for deep research: we materialize the DR plan as a typed DAG whose structural components are directly addressable by reward functions, allowing separate signals to optimize the corresponding plan and execution components rather than being read off a trajectory terminus; \(2\)DecomposeR framework: we instantiate these rewards with typed decomposition graphs and staged reinforcement learning, making search breadth, evidence reuse, branch integration, execution fidelity, and final answer quality addressable optimization targets; and \(3\)Empirical validation: we evaluate on three long\-form benchmarks, with analysis showing that the reward components, structured decomposition interface, and search\-conditioned revision each contribute to the final gains\.

## 2Related Work

Deep research agents\.Recent DR systems use web\-scale retrieval and long\-form synthesis to answer open\-ended queries\. Frontier systems such as OpenAI Deep Research rely on strong base models and inference\-time orchestration\(OpenAI,[2025](https://arxiv.org/html/2605.30824#bib.bib18)\)\. WebWeaver moves toward more explicit structure through dynamic outlines, evidence memory, and citation\-grounded writing, but remains an inference\-time dual\-agent framework rather than an RL objective over a typed plan\(Li et al\.,[2026b](https://arxiv.org/html/2605.30824#bib.bib15)\)\. WebThinker interleaves thinking, searching, navigation, and drafting inside a web\-enabled reasoning loop and improves tool use with online preference optimization\(Li et al\.,[2026a](https://arxiv.org/html/2605.30824#bib.bib14)\)\. WebSailor and WebExplorer focus on long\-horizon web agents, constructing high\-uncertainty or evolved web\-navigation tasks and training agents to handle many tool calls\(Li et al\.,[2025](https://arxiv.org/html/2605.30824#bib.bib13); Liu et al\.,[2025](https://arxiv.org/html/2605.30824#bib.bib16)\)\. These systems demonstrate the importance of search and synthesis, while DecomposeR focuses on making the research plan itself a structured, separately rewarded policy object\.

RL for search and tool use\.GRPO and RLVR\-style post\-training have been applied to reasoning, search, and tool use\(Shao et al\.,[2024](https://arxiv.org/html/2605.30824#bib.bib21); Guo et al\.,[2025](https://arxiv.org/html/2605.30824#bib.bib7)\)\. Search\-R1 trains models to interleave reasoning and search with retrieved\-token masking and outcome rewards\(Jin et al\.,[2025](https://arxiv.org/html/2605.30824#bib.bib10)\), while R1\-Searcher studies outcome\-based RL for invoking search without process supervision\(Song et al\.,[2025](https://arxiv.org/html/2605.30824#bib.bib24)\)\. ReSearch similarly treats search calls as part of the reasoning chain using explicit<search\>and<result\>tags, and masks retrieved results from the policy\-gradient loss\(Chen et al\.,[2026](https://arxiv.org/html/2605.30824#bib.bib4)\)\. O2\-Searcher targets both open\-ended and closed\-ended QA through a local search environment and task\-aware rewards\(Mei et al\.,[2025](https://arxiv.org/html/2605.30824#bib.bib17)\)\. ParallelSearch addresses an orthogonal efficiency problem by rewarding models for decomposing independent sub\-queries that can be searched in parallel\(Zhao et al\.,[2025](https://arxiv.org/html/2605.30824#bib.bib29)\)\. DecomposeR differs by exposing the full research plan as a typed graph and assigning reward to graph components before answer synthesis is trained, rather than treating the search trajectory as a flat sequence of tool actions\.

Query decomposition and retrieval\-augmented generation\.Prior decomposition methods decompose complex questions into subproblems through prompting or modular inference\(Khot et al\.,[2023](https://arxiv.org/html/2605.30824#bib.bib11); Wang et al\.,[2023](https://arxiv.org/html/2605.30824#bib.bib26); Trivedi et al\.,[2023](https://arxiv.org/html/2605.30824#bib.bib25)\)\. Retrieval\-augmented generation systems learn when and what to retrieve\(Asai et al\.,[2024b](https://arxiv.org/html/2605.30824#bib.bib3); Jiang et al\.,[2023](https://arxiv.org/html/2605.30824#bib.bib9)\)\. DecomposeR brings these ideas into RL for open\-ended DR by making decomposition a trainable graph policy rather than an unobserved latent reasoning trace\.

## 3Methodology

### 3\.1Problem Formulation

Existing RL\-trained DR systems model the agent rollout as a flat sequence

τflat=\(x,m1,o1,…,mT,oT,y\),\\tau\_\{\\mathrm\{flat\}\}=\(x,m\_\{1\},o\_\{1\},\\ldots,m\_\{T\},o\_\{T\},y\),\(1\)in which model actionsmtm\_\{t\}and tool observationsoto\_\{t\}interleave until a final answeryyis emitted, and a single scalar rewardR​\(x,y\)R\(x,y\)is propagated through everymtm\_\{t\}for policy update\. The limitations identified in Sec\.[1](https://arxiv.org/html/2605.30824#S1)are directly readable from this formulation\. First, everymtm\_\{t\}– whether it carries a planning decision, a search query, or a synthesis sentence – is updated under the sameRR, so the gradient cannot localize the failure to a specific role\. Second,R​\(x,y\)R\(x,y\)supervises the entire\{mt\}t=1T\\\{m\_\{t\}\\\}\_\{t=1\}^\{T\}, leaving each intermediatemtm\_\{t\}shaped only through its eventual effect on the terminalyy\. Third,mtm\_\{t\}is an unstructured token sequence, so dependencies between subtasks and the reuse of evidence across branches have no native representation\.

DecomposeR replaces this flat trajectory with a factored one in which an explicit plan𝒢\\mathcal\{G\}mediates between a planner policyπP\\pi\_\{P\}and an answerer policyπA\\pi\_\{A\}:

τ=\(x,𝒢0,Z,𝒢1,F,O1:K,y\),\\tau=\(x,\\mathcal\{G\}\_\{0\},Z,\\mathcal\{G\}\_\{1\},F,O\_\{1:K\},y\),\(2\)where𝒢0\\mathcal\{G\}\_\{0\}is an initial plan emitted byπP\\pi\_\{P\},Z=ℰ​\(𝒢0\)Z=\\mathcal\{E\}\(\\mathcal\{G\}\_\{0\}\)are search observations returned by the environmentℰ\\mathcal\{E\},𝒢1\\mathcal\{G\}\_\{1\}is the planner’s revised plan,FFare fetched documents,O1:KO\_\{1:K\}are intermediate outputs produced byπA\\pi\_\{A\}inKKtopological waves, andyyis the final answer\. The joint trajectory distribution factorises as

π​\(τ\|x\)\\displaystyle\\pi\(\\tau\|x\)=πP​\(𝒢0\|x\)​ℰ​\(Z\|𝒢0\)\\displaystyle=\\pi\_\{P\}\(\\mathcal\{G\}\_\{0\}\|x\)\\,\\mathcal\{E\}\(Z\|\\mathcal\{G\}\_\{0\}\)\(3\)⋅πP​\(𝒢1,F\|x,𝒢0,Z\)​πA​\(O1:K,y\|x,𝒢1,F\)\\displaystyle\\cdot\\pi\_\{P\}\(\\mathcal\{G\}\_\{1\},F\|x,\\mathcal\{G\}\_\{0\},Z\)\\,\\pi\_\{A\}\(O\_\{1:K\},y\|x,\\mathcal\{G\}\_\{1\},F\)This structure directly resolves the limitations of Eq\.[1](https://arxiv.org/html/2605.30824#S3.E1):πP\\pi\_\{P\}andπA\\pi\_\{A\}become separately optimisable policies, each amenable to its own reward \(resolving role entanglement\); rewards can be assigned to𝒢1\\mathcal\{G\}\_\{1\}, to individualOkO\_\{k\}, and toyyrespectively, distributing supervision across the rollout \(resolving sparsity\); and𝒢\\mathcal\{G\}is an explicit structured object that the planner emits and the answerer consumes \(resolving expressiveness, with the structure detailed in Sec\.[3\.3](https://arxiv.org/html/2605.30824#S3.SS3)\)\. Consequently the training objective decomposes into two role\-specific objectives,

𝒥P​\(θP\)\\displaystyle\\mathcal\{J\}\_\{P\}\(\\theta\_\{P\}\)=𝔼x​\[RP​\(𝒢1;x,Z\)\],\\displaystyle=\\mathbb\{E\}\_\{x\}\\\!\\left\[R\_\{P\}\(\\mathcal\{G\}\_\{1\};x,Z\)\\right\],\(4\)𝒥A​\(θA\)\\displaystyle\\mathcal\{J\}\_\{A\}\(\\theta\_\{A\}\)=𝔼x​\[RA​\(O1:K,y;x,𝒢1\)\]\\displaystyle=\\mathbb\{E\}\_\{x\}\\\!\\left\[R\_\{A\}\(O\_\{1:K\},y;x,\\mathcal\{G\}\_\{1\}\)\\right\]optimised in two sequential RL stages withRPR\_\{P\}andRAR\_\{A\}representing the planner and answerer reward respectively\.

### 3\.2Overview

Figure[1](https://arxiv.org/html/2605.30824#S1.F1)illustrates the DecomposeR rollout, which is identical at training and inference time up to the application of gradients\. Given a queryxx, the planner first emits an initial plan𝒢0\\mathcal\{G\}\_\{0\}in the form of a typed DAG \(the construction of which is detailed in Sec\.[3\.3](https://arxiv.org/html/2605.30824#S3.SS3)\)\. The environmentℰ\\mathcal\{E\}executes every search node in𝒢0\\mathcal\{G\}\_\{0\}and returns the observation setZZ\. The planner consumesZZand emits a revised plan𝒢1\\mathcal\{G\}\_\{1\}together with a small set of URLsFFto be fetched;ℰ\\mathcal\{E\}then runs only the newly added or modified search nodes in𝒢1\\mathcal\{G\}\_\{1\}and fetches the URLs inFF, reusing observations from unchanged search nodes\. Finally, the answerer fills the aggregate nodes of𝒢1\\mathcal\{G\}\_\{1\}in topological waves and produces the final citation\-grounded report at the answer node\. The three design choices underlying this pipeline are justified below\.

Why a typed DAG\.A typed DAG with search\-, aggregate\-, and answer\-type nodes makes cross\-source evidence reuse and hierarchical synthesis explicit at the structural level, both of which linear lists and tree\-structured plans cannot natively express\. The structure further exposes well\-defined properties– branch breadth, cross\-branch integration, query distinctness– as addressable units that the planner reward can target directly \(Sec\.[3\.4\.2](https://arxiv.org/html/2605.30824#S3.SS4.SSS2)\)\.

Why two\-turn planning\.𝒢0\\mathcal\{G\}\_\{0\}is generated fromxxalone and therefore reflects only the planner’s parametric guess at what is worth investigating\. Parametric knowledge frequently misaligns with what is actually retrievable on the open web: queries can return sparse, off\-topic, or outdated snippets\. The second turn closes the planning loop with retrieval reality, letting the planner drop ineffective subqueries, add new subqueries inspired by retrieved snippets, and concentrate full\-page fetching on the URLs most likely to support the synthesis\.

Why decouple planner and answerer\.Collapsing planning and answering into one policy under a single trajectory\-level reward conflates their failure modes: a low answer score cannot distinguish a weak plan executed faithfully from a strong plan executed poorly\. By training the two roles sequentially, Planner RL optimises rewards that depend only on𝒢1\\mathcal\{G\}\_\{1\}and never on answerer output, so plan quality is isolated from execution noise\. Answerer RL then trains against a fixed planner, so answerer credit is not confounded by drifting plan quality\. Staging also reduces the non\-stationarity inherent to jointly training two policies that act as each other’s environment, leaving two stable single\-policy optimisation problems\.

### 3\.3Plan Representation

The plan𝒢\\mathcal\{G\}must satisfy three requirements: expose addressable structure for fine\-grained reward design, capture cross\-source evidence reuse and hierarchical synthesis, and admit efficient execution\. Linear subquery lists provide only a flat sequence with no notion of branch reuse, and tree\-structured subgoal plans forbid a node from being shared across multiple descendants\. We therefore represent𝒢\\mathcal\{G\}as a typed directed acyclic graph, whose construction and execution we now detail\.

Node types\.𝒢\\mathcal\{G\}contains three node types\. A search nodes∈𝒩ss\\in\\mathcal\{N\}\_\{s\}carries a query string and a list of key points specifying what evidence the query should return\. An aggregate nodea∈𝒩aa\\in\\mathcal\{N\}\_\{a\}carries a synthesis brief consisting of a need statement and key points to be covered in its output\. The terminal answer nodevansv\_\{\\mathrm\{ans\}\}carries the final synthesis brief; it has no outgoing edges, and exactly one such node exists per plan\.

Edges and validity\.A directed edgeu→vu\\to vdeclares that the output ofuuis consumed byvv: edges from search nodes carry retrieved evidence, edges between aggregate nodes carry branch\-level conclusions, and edges intovansv\_\{\\mathrm\{ans\}\}carry the inputs to the final report\. A validator enforces that𝒢\\mathcal\{G\}is a connected DAG with valid JSON syntax, valid node IDs, allowed node types, no cycles, and exactly onevansv\_\{\\mathrm\{ans\}\}\. Invalid graphs are rejected at the validity gate before any reward is computed \(Sec\.[3\.4\.2](https://arxiv.org/html/2605.30824#S3.SS4.SSS2)\)\.

Topological\-wave execution\.Given a valid𝒢1\\mathcal\{G\}\_\{1\}, the answerer executes it wave by wave\. WaveWℓW\_\{\\ell\}contains all aggregate nodes whose parents have already been completed; all nodes in a wave are generated in a single model turn, exploiting the fact that same\-wave aggregates are conditionally independent given their parents\. After all aggregate nodes are filled, a final turn generatesvansv\_\{\\mathrm\{ans\}\}conditioned on its parent branches and the cumulative cited evidence\. Topological execution preserves the dependency order implied by𝒢\\mathcal\{G\}while minimising the number of model turns required\.

### 3\.4Staged Training

We train the planner and the answerer sequentially atop a shared Qwen3\-8B backbone with role\-conditioned prompts and role\-specific LoRA adapters\(Hu et al\.,[2021](https://arxiv.org/html/2605.30824#bib.bib8)\): a planner adapter is active when the model emits a graph, and an answerer adapter is active when the model emits aggregate or answer outputs\. Training proceeds in three phases: a cold\-start SFT phase that initialises both adapters from teacher trajectories, a planner RL phase that refines the planner adapter, and an answerer RL phase that refines the answerer adapter starting from a checkpoint with the planner\-RL adapter merged in\. Both RL phases use Group Relative Policy Optimization \(GRPO\)\(Shao et al\.,[2024](https://arxiv.org/html/2605.30824#bib.bib21)\)with token\-level masking that excludes environment observations from the policy gradient; the full GRPO objective and masking definitions are deferred to Appendix[A\.2](https://arxiv.org/html/2605.30824#A1.SS2)\. We describe each phase below, focusing on the DecomposeR\-specific reward design\.

#### 3\.4\.1Cold\-Start SFT

We first perform supervised fine\-tuning to teach the model the basic graph\-emission protocol and wave\-execution format, so that subsequent RL operates on a policy that already produces parseable outputs rather than wasting samples on protocol violations\. A frontier teacher model \(GPT\-5\-mini\) generates complete planner–answerer trajectories for a held\-out subset of training queries; each trajectory comprises planner turns \(initial and revised graphs\) and answerer turns \(aggregate waves and final answer\)\. We retain only trajectories that pass the graph validator and the answerer parser end\-to\-end, discarding those with malformed JSON, invalid node references, or schema\-violating output\. Surviving trajectories yield two streams of turn\-level examples– planner turns under the planner system prompt, and answerer turns under the answerer system prompt– on which the planner and answerer adapters are jointly initialised\.

#### 3\.4\.2Planner RL

The planner adapter is refined with a reward over three plan\-quality dimensions: semantic coverage of the user’s information need, retrieval quality of the search queries, and structural expressiveness of the DAG, rewarding each independently before combining them:

Rplan=𝟏\[Valid\(𝒢\)\]\(\\displaystyle R\_\{\\mathrm\{plan\}\}=\\mathbf\{1\}\[\\mathrm\{Valid\}\(\\mathcal\{G\}\)\]\\,\\bigl\(λcov​Crub\+λsearch​Qsearch\\displaystyle\\lambda\_\{\\mathrm\{cov\}\}\\,C\_\{\\mathrm\{rub\}\}\+\\lambda\_\{\\mathrm\{search\}\}\\,Q\_\{\\mathrm\{search\}\}\(5\)\+λgraphEgraph\)\.\\displaystyle\+\\lambda\_\{\\mathrm\{graph\}\}\\,E\_\{\\mathrm\{graph\}\}\\bigr\)\.The hard validity gate𝟏​\[Valid​\(𝒢\)\]\\mathbf\{1\}\[\\mathrm\{Valid\}\(\\mathcal\{G\}\)\]ensures that protocol correctness is established before quality signals take effect so invalid graphs receive zero reward regardless of their semantic content\. Allλ\\lambdaterms in this subsection are scalar reward weights\.

Rubric coverageCrubC\_\{\\mathrm\{rub\}\}\.The rubric for each training query is generated in advance by a frontier LLM conditioned on the query together with retrieved web context, producing query\-specific evaluation criteriaℛ=\{\(ru,wu\)\}\\mathcal\{R\}=\\\{\(r\_\{u\},w\_\{u\}\)\\\}of the form “the response should coverrur\_\{u\}”\. Rubric coverage measures how well the plan has semantically anticipated these criteria, by taking, for each rubric item, its best normalised cosine similarity with any non\-answer node field \(query,key\_points,need\):

Crub=∑uwu​maxn∈𝒩∖\{vans\}⁡cos\+⁡\(ϕ​\(u\),ϕ​\(n\)\)∑uwu,C\_\{\\mathrm\{rub\}\}=\\frac\{\\sum\_\{u\}w\_\{u\}\\max\_\{n\\in\\mathcal\{N\}\\setminus\\\{v\_\{\\mathrm\{ans\}\}\\\}\}\\cos^\{\+\}\\\!\\bigl\(\\phi\(u\),\\phi\(n\)\\bigr\)\}\{\\sum\_\{u\}w\_\{u\}\},\(6\)whereϕ​\(⋅\)\\phi\(\\cdot\)denotes the sentence embedding function andcos\+⁡\(⋅,⋅\)≡max⁡\(0,cos⁡\(⋅,⋅\)\)\\cos^\{\+\}\(\\cdot,\\cdot\)\\equiv\\max\(0,\\cos\(\\cdot,\\cdot\)\)clamps negative similarities to zero\. Anchoring the rubric to plan\-node fields rather than to the final answer gives the planner direct semantic feedback on the emitted plan\.

Search qualityQsearchQ\_\{\\mathrm\{search\}\}\.Search quality measures whether the search nodes target the right aspects and avoid redundancy:

Qsearch=λsearch,cov​Csearch\+λsearch,dist​Dsearch,Q\_\{\\mathrm\{search\}\}=\\lambda\_\{\\mathrm\{search,cov\}\}\\,C\_\{\\mathrm\{search\}\}\+\\lambda\_\{\\mathrm\{search,dist\}\}\\,D\_\{\\mathrm\{search\}\},\(7\)whereCsearchC\_\{\\mathrm\{search\}\}is the rubric\-to\-search\-query similarity \(CrubC\_\{\\mathrm\{rub\}\}restricted to search nodes\), andDsearch=1−s¯pairD\_\{\\mathrm\{search\}\}=1\-\\bar\{s\}\_\{\\mathrm\{pair\}\}is the complement of the mean pairwise cosine similarity among search queries\. ThusCsearchC\_\{\\mathrm\{search\}\}rewards on\-topic retrieval, whileDsearchD\_\{\\mathrm\{search\}\}rewards nonredundant queries\.

Graph expressivenessEgraphE\_\{\\mathrm\{graph\}\}\.Graph expressiveness combines synthesis breadthBsynthB\_\{\\mathrm\{synth\}\}, cross\-branch integrationIcrossI\_\{\\mathrm\{cross\}\}, and search breadthBsearchB\_\{\\mathrm\{search\}\}:

Egraph=\\displaystyle E\_\{\\mathrm\{graph\}\}=λgraph,synth​Bsynth\+λgraph,cross​Icross\\displaystyle\\lambda\_\{\\mathrm\{graph,synth\}\}\\,B\_\{\\mathrm\{synth\}\}\+\\lambda\_\{\\mathrm\{graph,cross\}\}\\,I\_\{\\mathrm\{cross\}\}\(8\)\+λgraph,search​Bsearch\.\\displaystyle\+\\lambda\_\{\\mathrm\{graph,search\}\}\\,B\_\{\\mathrm\{search\}\}\.The subterms are

Bsynth\\displaystyle B\_\{\\mathrm\{synth\}\}=f​\(\|pa𝒩a​\(vans\)\|\)⋅dpar,\\displaystyle=f\\\!\\bigl\(\|\\mathrm\{pa\}\_\{\\mathcal\{N\}\_\{a\}\}\(v\_\{\\mathrm\{ans\}\}\)\|\\bigr\)\\cdot d\_\{\\mathrm\{par\}\},\(9\)Icross\\displaystyle I\_\{\\mathrm\{cross\}\}=12​\(rfan\+rint\),\\displaystyle=\\tfrac\{1\}\{2\}\\bigl\(r\_\{\\mathrm\{fan\}\}\+r\_\{\\mathrm\{int\}\}\\bigr\),\(10\)Bsearch\\displaystyle B\_\{\\mathrm\{search\}\}=f​\(\|𝒩s\|\),\\displaystyle=f\\\!\\bigl\(\|\\mathcal\{N\}\_\{s\}\|\\bigr\),\(11\)wheref​\(k\)=1−1/kf\(k\)=1\-1/kis a diminishing\-returns count score,dpard\_\{\\mathrm\{par\}\}is the mean pairwise cosine diversity among the answer node’s aggregate parents,rfanr\_\{\\mathrm\{fan\}\}is the fraction of search nodes feeding at least two aggregate nodes, andrintr\_\{\\mathrm\{int\}\}is the fraction of aggregate nodes integrating at least two parent inputs\. ThusBsynthB\_\{\\mathrm\{synth\}\}rewards diverse direct inputs to the final answer,IcrossI\_\{\\mathrm\{cross\}\}rewards evidence reuse and multi\-source branch integration, andBsearchB\_\{\\mathrm\{search\}\}rewards search breadth\.

020204040606080801001001201201401401601601801802002002202202402402602602802803003003203200\.40\.40\.50\.50\.60\.60\.70\.70\.80\.8RL steprewardPlanner RewardFigure 2:Planner RL reward dynamics\. The black curve shows the smoothed planner reward and the grey trace shows raw rollout means\. Structural diagnostics are reported in Appendix[A\.6](https://arxiv.org/html/2605.30824#A1.SS6)\.
#### 3\.4\.3Answerer RL

With the planner adapter fixed, the answerer adapter is refined to execute the plan faithfully and synthesise a high\-quality final report\. Graphs are generated by the planner\-RL model and rolled out through the environment in advance, so the answerer trains on a stable distribution of plans rather than on a moving target\. We decompose answerer quality into two dimensions: branch\-level execution fidelity \(measuring whether each aggregate node covers its declared key points\) and global synthesis quality \(measuring whether the final answer integrates the branches and satisfies the rubric\) and combine them as

Rans=λexec​Rexec\+λsynth​RsynthR\_\{\\mathrm\{ans\}\}=\\lambda\_\{\\mathrm\{exec\}\}\\,R\_\{\\mathrm\{exec\}\}\+\\lambda\_\{\\mathrm\{synth\}\}\\,R\_\{\\mathrm\{synth\}\}\(12\)Allλ\\lambdaterms in this subsection are scalar reward weights\. Parse failures receive partial credit computed over the successfully completed waves; this eliminates the binary reward cliff that destabilised early experiments, in which a single malformed node would zero out a trajectory whose mean baseline was well above zero\.

Execution rewardRexecR\_\{\\mathrm\{exec\}\}\.Execution reward scores each aggregate node independently against its declared key points\. Only aggregate nodes are scored, since search nodes are executed byℰ\\mathcal\{E\}rather than by the policy\. For eachv∈𝒩av\\in\\mathcal\{N\}\_\{a\}, we combine an embedding\-level and a lexical key\-point coverage signal:

rv=λexec,emb​Kemb​\(v\)\+λexec,lex​Klex​\(v\)r\_\{v\}=\\lambda\_\{\\mathrm\{exec,emb\}\}\\,K\_\{\\mathrm\{emb\}\}\(v\)\+\\lambda\_\{\\mathrm\{exec,lex\}\}\\,K\_\{\\mathrm\{lex\}\}\(v\)\(13\)Kemb​\(v\)K\_\{\\mathrm\{emb\}\}\(v\)is a greedy bipartite matching score between the node’skey\_pointsembeddings and its output\-sentence embeddings, weighted by recall;Klex​\(v\)K\_\{\\mathrm\{lex\}\}\(v\)is the mean fraction of each key point’s content tokens present in the output\. The embedding signal captures semantic coverage, while the lexical signal grounds it deterministically\. Per\-node scores are aggregated with a shifted harmonic mean:

Rexec=\|𝒩a\|∑v∈𝒩a11\+rv−1∈\[0,1\]R\_\{\\mathrm\{exec\}\}=\\frac\{\|\\mathcal\{N\}\_\{a\}\|\}\{\\sum\_\{v\\in\\mathcal\{N\}\_\{a\}\}\\frac\{1\}\{1\+r\_\{v\}\}\}\-1\\in\[0,1\]\(14\)The shifted harmonic mean maps all\-zero to0and all\-one to11\.

Synthesis rewardRsynthR\_\{\\mathrm\{synth\}\}\.Synthesis reward targets the final answer rather than individual branches:

Rsynth=λsynth,judge​Jrub\+λsynth,util​Ubranch\.R\_\{\\mathrm\{synth\}\}=\\lambda\_\{\\mathrm\{synth,judge\}\}\\,J\_\{\\mathrm\{rub\}\}\+\\lambda\_\{\\mathrm\{synth,util\}\}\\,U\_\{\\mathrm\{branch\}\}\.\(15\)JrubJ\_\{\\mathrm\{rub\}\}is a rubric\-weighted judge score from an external LLM on a five\-point scale\.UbranchU\_\{\\mathrm\{branch\}\}is the mean fraction of each branch’s content tokens that appear in the final answer; it provides a deterministic signal that the model is actually consuming its intermediate branch outputs\.

050501001001501502002002502503003003503500\.30\.30\.40\.40\.50\.50\.60\.60\.70\.70\.80\.8RL stepweighted rewardAnswerer Reward ComponentsoverallexecutionsynthesisFigure 3:Answerer RL reward dynamics\. The plot shows overall answerer reward together with its execution and synthesis contributions\. Length diagnostics are reported in Appendix[A\.6](https://arxiv.org/html/2605.30824#A1.SS6)\.

## 4Experiments

### 4\.1Experimental Setup

Benchmarks and metrics\.We evaluate DecomposeR on three long\-form deep\-research benchmarks spanning different domains\.DeepResearchBench \(DRBench\)provides expert\-written multi\-domain research tasks that require web exploration, citation\-backed synthesis, and report\-quality judgment\.ResearchQA\-Mini \(ResQA\-Mini\)evaluates scholarly question answering with survey\-derived research questions and rubric items\.HealthBenchevaluates responses to health\-related conversations against physician\-informed rubric criteria, testing evidence synthesis in a high\-stakes domain\. For all three benchmarks we report the official rubric\- or judge\-based score; each metric captures coverage, factual correctness, and response quality on a scale where higher is better\. Appendix[A\.4](https://arxiv.org/html/2605.30824#A1.SS4)provides benchmark\-specific evaluation details\.

Base model\.We build DecomposeR on Qwen3\-8B\(Yang et al\.,[2025](https://arxiv.org/html/2605.30824#bib.bib27)\)with two LoRA adapters\(Hu et al\.,[2021](https://arxiv.org/html/2605.30824#bib.bib8)\): one for the planner role and one for the answerer role\. All ablation variants use this same backbone with the same SFT initialization, so that comparisons across variants isolate the effect of the studied design choice\.

Baselines\.We compare DecomposeR\-8B against two broad types of systems:*\(i\) Commercial deep\-research systems*: OpenAI \(o4\-mini\) Deep Research, Gemini Deep Research and Perplexity Deep Research, accessed through their official APIs\.*\(ii\) Open deep\-research and search agents*: WebThinker \(QwQ\-32B and 32B\-DPO\), Tongyi DeepResearch\-30B, WebExplorer\-8B, Search\-R1, ASearcher\-Web\-7B and Qwen3\-8B equipped with search\. We additionally report a DecomposeR\-8B \(SFT only\) variant to isolate the effect of the two RL stages\.

Training data\.We obtain a filtered pool of roughly 4,000 long\-form English prompts/queries across OpenScholar, SearchArena, and ScholarQA sources\. We use 2,325 queries for cold\-start SFT, converted from teacher planner–answerer trajectories into 10,532 turn\-level supervision examples\. For RL, both planner RL and answerer RL use the same 1,432\-query prepared set, with planner graphs, search results, fetched pages, rubrics, and citation catalogs attached to each query artifact\.

Training protocol\.Both RL stages use GRPO\(Shao et al\.,[2024](https://arxiv.org/html/2605.30824#bib.bib21)\)with verl\(Sheng et al\.,[2025](https://arxiv.org/html/2605.30824#bib.bib22)\)and vLLM\(Kwon et al\.,[2023](https://arxiv.org/html/2605.30824#bib.bib12)\)for training and rollout generation respectively\. The environment uses Jina Search for search\-node execution \(up to five results per node\) and Serper for URL fetching \(up to four URLs per revision turn\)\. Full GRPO details, hyperparameters, and prompt templates are in Appendix[A](https://arxiv.org/html/2605.30824#A1)\. Reported numbers are averaged over multiple random seeds\. A central design decision is thatreward weights are specified once and kept fixedacross the main experiment and every ablation variant reported in this section\. The reward specification uses no benchmark\-specific weight calibration\. Consequently, performance differences across variants reflect the design choices being studied rather than reward retuning\. Reward weights used are reported in full in Appendix[A\.5](https://arxiv.org/html/2605.30824#A1.SS5)\.

### 4\.2Main Results

Overall performance\.Table[1](https://arxiv.org/html/2605.30824#S4.T1)shows that DecomposeR\-8B \(SFT\+RL\) is the strongest open model across the three long\-form benchmarks compared against models of similar size, achieving 41\.8 on DRBench, 42\.0 on HealthBench, and 71\.4 on ResearchQA\-Mini\. Compared with Qwen3\-8B \+ Search, DecomposeR improves by 7\.4 points on DRBench, 22\.2 points on HealthBench, and 14\.4 points on ResearchQA\-Mini, despite using the same backbone scale\. It also substantially outperforms WebExplorer\-8B and other RL\-trained search agents such as Search\-R1\-7B and ASearcher\-Web\-7B indicating that the improvement is not simply due to giving an 8B model access to search, but training it to organize retrieval and synthesis through the DecomposeR framework\.

RL improves substantially over SFT\.The SFT\-only model already benefits from learning the typed graph and wave\-execution protocol, but the two RL stages provide large additional gains\. The improvement is largest on ResearchQA\-Mini, where rubric\-style coverage and synthesis are central, and remains positive on HealthBench, where domain sensitivity makes unsupported synthesis more costly\. These gains suggest that SFT is sufficient to initialize the graph\-emission and wave\-execution protocol, but that staged RL is needed for the planner and answerer to use this protocol effectively \.

Comparison with larger open systems\.Among open deep\-research systems, DecomposeR is the best on average and is competitive with or stronger than substantially larger models on individual benchmarks\. It outperforms Tongyi DeepResearch\-30B\-A3B on DRBench and ResearchQA\-Mini and exceeds WebThinker\-32B\-DPO on DRBench and HealthBench, while remaining close on ResearchQA\-Mini\. This is notable because those systems rely on much larger backbones and/or more carefully\-engineered inference behavior\.

Table 1:Main results on three long\-form deep\-research benchmarks\. DecomposeR\-8B \(SFT\+RL\) refers to the full model trained with both planner\-RL and answerer\-RL stages\. Entries marked “–” indicate that no public result was available or that running the evaluation was prohibitively expensive\.
### 4\.3Staged Training is Necessary

The two experiments in this section answer two distinct questions: whether staged training outperforms joint training \(B1\), and whether the improvement actually traces back to better plans rather than to the answerer compensating for weak plans \(B2\)\. Both halves of Table[2](https://arxiv.org/html/2605.30824#S4.T2)share the same fixed reward weights as the main experiment\.

B1: Staged vs\. Joint RL\.We construct a Joint RL baseline that shares with DecomposeR the same Qwen3\-8B backbone, separate planner and answerer LoRA adapters, SFT initialization, and typed DAG plan representation\. The variants differ only in training schedule and reward assignment\. In Joint RL, the planner and answerer adapters are updated simultaneously from a single trajectory\-level reward computed from the final answer\. This setup isolates the effect of staged, role\-specific optimization from changes in model architecture or plan representation\. The top half of Table[2](https://arxiv.org/html/2605.30824#S4.T2)shows that staged training outperforms joint training by 3\.7–4\.7 points across benchmarks\. The gap is largest on DRBench, whose general\-domain queries produce the longest rollouts and therefore suffer most from sparse trajectory\-end credit assignment\.

B2: Planner quality with a fixed external answerer\.A natural concern is that the gains in B1 might come from the answerer learning to compensate for weak plans rather than from the planner producing better plans\. We address this by holding the answerer fixed and varying only the planner\. Specifically, we use GPT\-4\.1 with a standardized execution prompt as an external answerer and feed it plans from four planners: GPT\-5 with a planning prompt \(strong training\-free reference\), the SFT\-only planner, the Joint\-RL planner from B1, and the DecomposeR planner\. The same external answerer executes all four plan distributions, so differences in final answer quality reflect only differences in plan quality\. The bottom half of Table[2](https://arxiv.org/html/2605.30824#S4.T2)shows that the DecomposeR planner produces plans whose downstream answer quality exceeds the SFT\-only and Joint\-RL planners considerably and closely trails or exceeds the GPT\-5 planning baseline despite using a substantially smaller backbone\. Combined with B1, this confirms that the gains from staged training trace back to the planner, not to answerer compensation\.

Table 2:Validating staged training\. B1 compares RL\-pipeline variants trained on the same SFT model\. B2 isolates planner quality by holding the answerer fixed and varying only the planner\.Additional ablation studies reported in Appendix[A\.7](https://arxiv.org/html/2605.30824#A1.SS7)and[A\.8](https://arxiv.org/html/2605.30824#A1.SS8)further isolate the design choices behind these gains\. The reward\-component ablation tests whether performance comes from a single dominant signal or varied sources\. The plan\-representation and planner\-revision ablations then test whether typed DAG structure and search\-conditioned revision are necessary by replacing the DAG with linear/tree structures and by disabling the post\-search revision turn\.

### 4\.4Case Study

Appendix[C](https://arxiv.org/html/2605.30824#A3)gives a representative DecomposeR rollout, including the pre\- and post\-revision graphs in Figures[15](https://arxiv.org/html/2605.30824#A3.F15)and[17](https://arxiv.org/html/2605.30824#A3.F17), the turn flow in TableLABEL:tab:appendix\_rollout\_turn\_flow, and the revised graph summary in TableLABEL:tab:appendix\_rollout\_graph\_summary\. The example highlights framework\-specific behavior to look for: the initial decomposition is revised after search feedback; added or modified search nodes and fetch requests change the evidence available to the answerer; search evidence can feed multiple downstream aggregates; and aggregate outputs are executed in topological waves before being combined by the terminal answer node\.

## 5Conclusion

We introduced DecomposeR, a planner\-centric framework for deep research that represents plans as typed DAGs and trains planner and answerer roles in separate RL stages\. By assigning reward to explicit graph components, DecomposeR improves credit assignment for the planning process\. The planner RL run shows strong gains in reward, validity, search breadth, aggregate branching, and cross\-branch structure, supporting the core hypothesis that structured plans are a useful optimization target for deep research systems\. The answerer RL run confirms that execution fidelity and synthesis quality are separable learning objectives\.

## 6Limitations

The current system fixes the plan at graph revision time and does not support iterative replanning after the answerer begins execution\. In principle, partial evidence from early aggregate nodes could inform later search decisions, but this would require variable\-length rollouts that make GRPO credit assignment substantially harder\. The two\-turn bounded planner is a deliberate tradeoff that preserves RL tractability; extending to adaptive replanning is a natural future direction\.

DecomposeR also depends on external search and fetched pages\. Noisy, incomplete, or poorly ranked evidence can still propagate into the graph and final answer, and the current system does not perform a separate source\-faithfulness verification step\.

Finally, the reward design uses fixed weights across several planner and answerer components\. We keep these weights fixed for all experiments to avoid benchmark\-specific tuning, but this may not be optimal across domains\. In addition, several reward terms rely on embedding\-based rubric or key\-point matching, which can over\-credit semantically similar but incomplete content\.

## References

- Arora et al\. \(2025\)Rahul K\. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero\-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal\. 2025\.[Healthbench: Evaluating large language models towards improved human health](https://arxiv.org/abs/2505.08775)\.*Preprint*, arXiv:2505\.08775\.
- Asai et al\. \(2024a\)Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D’arcy, David Wadden, Matt Latzke, Minyang Tian, Pan Ji, Shengyan Liu, Hao Tong, Bohao Wu, Yanyu Xiong, Luke Zettlemoyer, and 6 others\. 2024a\.[Openscholar: Synthesizing scientific literature with retrieval\-augmented lms](https://arxiv.org/abs/2411.14199)\.*Preprint*, arXiv:2411\.14199\.
- Asai et al\. \(2024b\)Akari Asai, Zeqiu Wu, Yizhong Wang, Avi Sil, and Hannaneh Hajishirzi\. 2024b\.Self\-rag: Learning to retrieve, generate, and critique through self\-reflection\.In*International conference on learning representations*, volume 2024, pages 9112–9141\.
- Chen et al\. \(2026\)Mingyang Chen, Linzhuang Sun, Tianpeng Li, Haoze Sun, Chenzheng Zhu, Haofen Wang, Jeff Pan, Wen Zhang, Huajun Chen, Fan Yang, and 1 others\. 2026\.Learning to reason with search for llms via reinforcement learning\.*Advances in Neural Information Processing Systems*, 38:85287–85307\.
- Comanici et al\. \(2025\)Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others\. 2025\.Gemini 2\.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.*arXiv preprint arXiv:2507\.06261*\.
- Du et al\. \(2025\)Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao\. 2025\.Deepresearch bench: A comprehensive benchmark for deep research agents\.*arXiv preprint arXiv:2506\.11763*\.
- Guo et al\. \(2025\)Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, and 1 others\. 2025\.Deepseek\-r1: Incentivizing reasoning capability in llms via reinforcement learning\.*arXiv preprint arXiv:2501\.12948*\.
- Hu et al\. \(2021\)Edward J\. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\. 2021\.[Lora: Low\-rank adaptation of large language models](https://arxiv.org/abs/2106.09685)\.*Preprint*, arXiv:2106\.09685\.
- Jiang et al\. \(2023\)Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi\-Yu, Yiming Yang, Jamie Callan, and Graham Neubig\. 2023\.Active retrieval augmented generation\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 7969–7992\.
- Jin et al\. \(2025\)Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan O\. Arik, Dong Wang, Hamed Zamani, and Jiawei Han\. 2025\.[Search\-r1: Training llms to reason and leverage search engines with reinforcement learning](https://arxiv.org/abs/2503.09516)\.In*Proceedings of the Conference on Language Modeling*\.
- Khot et al\. \(2023\)Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal\. 2023\.Decomposed prompting: A modular approach for solving complex tasks\.In*International Conference on Learning Representations*\.
- Kwon et al\. \(2023\)Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E\. Gonzalez, Hao Zhang, and Ion Stoica\. 2023\.[Efficient memory management for large language model serving with pagedattention](https://arxiv.org/abs/2309.06180)\.*Preprint*, arXiv:2309\.06180\.
- Li et al\. \(2025\)Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baixuan Li, Zhengwei Tao, Xinyu Wang, Weizhou Shen, Junkai Zhang, Dingchu Zhang, Xixi Wu, Yong Jiang, Ming Yan, Pengjun Xie, Fei Huang, and Jingren Zhou\. 2025\.[Websailor: Navigating super\-human reasoning for web agent](https://arxiv.org/abs/2507.02592)\.*Preprint*, arXiv:2507\.02592\.
- Li et al\. \(2026a\)Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yongkang Wu, Ji\-Rong Wen, Yutao Zhu, and Zhicheng Dou\. 2026a\.Webthinker: Empowering large reasoning models with deep research capability\.*Advances in Neural Information Processing Systems*, 38:120091–120131\.
- Li et al\. \(2026b\)Zijian Li, Xin Guan, Bo Zhang, Shen Huang, Houquan Zhou, Shaopeng Lai, Ming Yan, Yong Jiang, Pengjun Xie, Fei Huang, Jun Zhang, and Jingren Zhou\. 2026b\.[Webweaver: Structuring web\-scale evidence with dynamic outlines for open\-ended deep research](https://openreview.net/forum?id=MtNCJjlrKt)\.In*The Fourteenth International Conference on Learning Representations*\.
- Liu et al\. \(2025\)Junteng Liu, Yunji Li, Chi Zhang, Jingyang Li, Aili Chen, Ke Ji, Weiyu Cheng, Zijia Wu, Chengyu Du, Qidi Xu, Jiayuan Song, Zhengmao Zhu, Wenhu Chen, Pengyu Zhao, and Junxian He\. 2025\.[Webexplorer: Explore and evolve for training long\-horizon web agents](https://arxiv.org/abs/2509.06501)\.*Preprint*, arXiv:2509\.06501\.
- Mei et al\. \(2025\)Jianbiao Mei, Tao Hu, Daocheng Fu, Licheng Wen, Xuemeng Yang, Rong Wu, Pinlong Cai, Xinyu Cai, Xing Gao, Yu Yang, Chengjun Xie, Botian Shi, Yong Liu, and Yu Qiao\. 2025\.[O2\-searcher: A searching\-based agent model for open\-domain open\-ended question answering](https://arxiv.org/abs/2505.16582)\.*Preprint*, arXiv:2505\.16582\.
- OpenAI \(2025\)OpenAI\. 2025\.Introducing deep research\.[https://openai\.com/index/introducing\-deep\-research/](https://openai.com/index/introducing-deep-research/)\.
- Perplexity Team \(2025\)Perplexity Team\. 2025\.Introducing perplexity deep research\.[https://www\.perplexity\.ai/hub/blog/introducing\-perplexity\-deep\-research](https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research)\.
- Reimers and Gurevych \(2019\)Nils Reimers and Iryna Gurevych\. 2019\.[Sentence\-bert: Sentence embeddings using siamese bert\-networks](https://arxiv.org/abs/1908.10084)\.In*Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing*\. Association for Computational Linguistics\.
- Shao et al\. \(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y\. K\. Li, Y\. Wu, and Daya Guo\. 2024\.[Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://arxiv.org/abs/2402.03300)\.*Preprint*, arXiv:2402\.03300\.
- Sheng et al\. \(2025\)Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu\. 2025\.[Hybridflow: A flexible and efficient rlhf framework](https://doi.org/10.1145/3689031.3696075)\.In*Proceedings of the Twentieth European Conference on Computer Systems*, EuroSys ’25, page 1279–1297\. ACM\.
- Singh et al\. \(2025\)Amanpreet Singh, Joseph Chee Chang, Dany Haddad, Aakanksha Naik, Jena D Hwang, Rodney Kinney, Daniel S Weld, Doug Downey, and Sergey Feldman\. 2025\.AI2 scholar QA: Organized literature synthesis with attribution\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 3: System Demonstrations\)*, pages 513–523\.
- Song et al\. \(2025\)Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji\-Rong Wen\. 2025\.[R1\-searcher: Incentivizing the search capability in llms via reinforcement learning](https://arxiv.org/abs/2503.05592)\.*Preprint*, arXiv:2503\.05592\.
- Trivedi et al\. \(2023\)Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal\. 2023\.Interleaving retrieval with chain\-of\-thought reasoning for knowledge\-intensive multi\-step questions\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics*, pages 10014–10037\.
- Wang et al\. \(2023\)Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka\-Wei Lee, and Ee\-Peng Lim\. 2023\.[Plan\-and\-solve prompting: Improving zero\-shot chain\-of\-thought reasoning by large language models](https://doi.org/10.18653/v1/2023.acl-long.147)\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others\. 2025\.[Qwen3 technical report](https://arxiv.org/abs/2505.09388)\.*Preprint*, arXiv:2505\.09388\.
- Yao et al\. \(2023\)Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao\. 2023\.[React: Synergizing reasoning and acting in language models](https://openreview.net/forum?id=WE_vluYUL-X)\.In*The Eleventh International Conference on Learning Representations*\.
- Zhao et al\. \(2025\)Shu Zhao, Tan Yu, Anbang Xu, Japinder Singh, Aaditya Shukla, and Rama Akkiraju\. 2025\.[Parallelsearch: Train your llms to decompose query and search sub\-queries in parallel with reinforcement learning](https://arxiv.org/abs/2508.09303)\.*Preprint*, arXiv:2508\.09303\.

## Appendix

## Appendix ATraining Details

### A\.1Training Data Sources

Table[3](https://arxiv.org/html/2605.30824#A1.T3)gives the exact source breakdown for the filtered long\-form training pool and the artifacts derived from it\. The active training sets contain 2,325 SFT queries, converted into 10,532 supervised turns, and a shared 1,432\-query RL set used by both planner RL and answerer RL\. Candidate split and rubric columns are included to document data construction\.

Table 3:Training data source breakdown\. “SFT used” counts planner–answerer teacher trajectories used for cold\-start SFT\. “Rubrics generated” counts generated query\-specific rubrics\. “RL used” counts the shared prepared query set used by both planner RL and answerer RL\.
### A\.2Masked GRPO Objective

This subsection expands the Group Relative Policy Optimization \(GRPO\) training protocol summarized in Section[4\.1](https://arxiv.org/html/2605.30824#S4.SS1)\(Shao et al\.,[2024](https://arxiv.org/html/2605.30824#bib.bib21)\)\. Search\-RL systems such as Search\-R1 and ReSearch adapt GRPO to tool environments by treating retrieved text as environment output and masking it out of the policy\-gradient loss\(Jin et al\.,[2025](https://arxiv.org/html/2605.30824#bib.bib10); Chen et al\.,[2026](https://arxiv.org/html/2605.30824#bib.bib4)\); O2\-Searcher further uses GRPO with composite rewards for open\-ended search tasks\(Mei et al\.,[2025](https://arxiv.org/html/2605.30824#bib.bib17)\)\. DecomposeR follows the same separation principle, but the model\-generated objects are typed planner graphs and answerer node reports rather than inline search traces\.

##### Masked GRPO objective\.

For each input query or prepared answerer artifactxx, the old policy samples a group ofGGrollouts\{τi\}i=1G∼πθold\(⋅\|x\)\\\{\\tau\_\{i\}\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(\\cdot\|x\)\. Each rollout receives a scalar stage rewardRiR\_\{i\}:Ri=RplanR\_\{i\}=R\_\{\\mathrm\{plan\}\}for planner RL andRi=RansR\_\{i\}=R\_\{\\mathrm\{ans\}\}for answerer RL\. We use the full clipped GRPO objective ofShao et al\. \([2024](https://arxiv.org/html/2605.30824#bib.bib21)\), with the token average restricted to the model\-generated assistant tokens:

𝒥GRPO​\(θ\)\\displaystyle\\mathcal\{J\}\_\{\\mathrm\{GRPO\}\}\(\\theta\)=𝔼​\[1G​∑i=1G1\|ℳi\|​∑t∈ℳi\{min⁡\(ρi,t​A^i,clip⁡\(ρi,t,1−ϵ,1\+ϵ\)​A^i\)−β​D^KLi,t\}\],\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{\|\\mathcal\{M\}\_\{i\}\|\}\\sum\_\{t\\in\\mathcal\{M\}\_\{i\}\}\\left\\\{\\min\\\!\\left\(\\rho\_\{i,t\}\\widehat\{A\}\_\{i\},\\operatorname\{clip\}\(\\rho\_\{i,t\},1\-\\epsilon,1\+\\epsilon\)\\widehat\{A\}\_\{i\}\\right\)\-\\beta\\,\\widehat\{D\}\_\{\\mathrm\{KL\}\}^\{i,t\}\\right\\\}\\right\],\(16\)ρi,t\\displaystyle\\rho\_\{i,t\}=πθ​\(yi,t\|hi,t\)πθold​\(yi,t\|hi,t\),A^i=Ri−1G​∑j=1GRjstd⁡\(\{Rj\}j=1G\)\+δ,\\displaystyle=\\frac\{\\pi\_\{\\theta\}\(y\_\{i,t\}\|h\_\{i,t\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(y\_\{i,t\}\|h\_\{i,t\}\)\},\\qquad\\widehat\{A\}\_\{i\}=\\frac\{R\_\{i\}\-\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}R\_\{j\}\}\{\\operatorname\{std\}\(\\\{R\_\{j\}\\\}\_\{j=1\}^\{G\}\)\+\\delta\},D^KLi,t\\displaystyle\\widehat\{D\}\_\{\\mathrm\{KL\}\}^\{i,t\}=πref​\(yi,t\|hi,t\)πθ​\(yi,t\|hi,t\)−log⁡\(πref​\(yi,t\|hi,t\)πθ​\(yi,t\|hi,t\)\)−1\.\\displaystyle=\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{i,t\}\|h\_\{i,t\}\)\}\{\\pi\_\{\\theta\}\(y\_\{i,t\}\|h\_\{i,t\}\)\}\-\\log\\\!\\left\(\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{i,t\}\|h\_\{i,t\}\)\}\{\\pi\_\{\\theta\}\(y\_\{i,t\}\|h\_\{i,t\}\)\}\\right\)\-1\.Herehi,th\_\{i,t\}is the full dialogue context before tokenyi,ty\_\{i,t\},ϵ\\epsilonis the PPO\-style clip range,β\\betais the KL coefficient, andδ\\deltais a small numerical constant\. The termD^KLi,t\\widehat\{D\}\_\{\\mathrm\{KL\}\}^\{i,t\}is the positive KL estimator used by GRPO; unlike RLHF formulations that add a per\-token KL penalty into the reward, this form regularizes the policy directly in the objective\. No learned critic or value head is used; the group mean and standard deviation provide the baseline and scale\. Since our rewards are trajectory\-level stage rewards, the same normalized advantage is assigned to every unmasked policy token in the rollout\.

The maskℳi\\mathcal\{M\}\_\{i\}contains only policy\-generated assistant tokens\. For planner RL, this includes the planner<think\>tokens, graph JSON, revision JSON, and fetch\-request JSON\. For answerer RL, this includes the answerer<think\>tokens and XML<node\>blocks for aggregate waves and the final answer\. System prompts, user payloads, Jina search results, Serper fetch contents, compact dependency payloads, and citation catalogs are conditioned on but excluded from the policy\-gradient and KL terms\. This is the DecomposeR analogue of retrieved\-token masking in inline search\-agent rollouts\.

### A\.3Training Hyper\-parameters

Tables[4](https://arxiv.org/html/2605.30824#A1.T4)and[5](https://arxiv.org/html/2605.30824#A1.T5)report the cold\-start SFT and RL hyper\-parameters used in the reported run\. All training stages were run on a single node with 8 x NVIDIA RTX PRO 6000 GPUs, each with 96GB of memory\.

Table 4:Cold\-start SFT hyper\-parameters\.Table 5:GRPO hyper\-parameters used for the two RL stages\.
### A\.4Benchmark Evaluation Details

##### DeepResearchBench\.

We evaluated all 100 DeepResearchBench prompts, consisting of 50 Chinese and 50 English tasks from the official query file\. Following the benchmark protocol, we generated one cited long\-form report per prompt, ran the official RACE and FACT evaluation pipelines separately for the Chinese, English, and combined outputs\. RACE was judged with the official DeepResearchBench LLM judge usingopenai/gpt\-5\.5, while the auxiliary FACT pipeline usedopenai/gpt\-5\.4\-mini; the main results table reports the combined RACE score\.

##### ResearchQA\-Mini\.

We evaluated the full 776\-questiontest\_minisplit of ResearchQA\-Mini\. The generated reports were exported in the expected ResearchQA response\-map format and scored with the official recommended rubric\-coverage judge used by the evaluation wrapper,gpt\-4\.1\-mini\.

##### HealthBench\.

We evaluated 500 samples sampled from the official HealthBench dataset, including 100 prompts from the hard subset under a hard\-sampling fraction of 20 %, consistent with the original ratio\. Generated completions were attached back to the original HealthBench rows and graded with the officialsimple\-evalsHealthBench rubric implementation using the recommendedgpt\-4\.1\-2025\-04\-14as the grader\.

### A\.5Planner and Answerer Reward Details

Table 6:All reward weights used in the planner and answerer RL stages\.Table[6](https://arxiv.org/html/2605.30824#A1.T6)separates weights by the object they score\. Planner terms are applied to the revised graph before any final answer is written, while answerer terms are applied to generated node reports and the terminal answer under a fixed graph\. Throughout this subsection,𝒢=\(𝒩,ℰ\)\\mathcal\{G\}=\(\\mathcal\{N\},\\mathcal\{E\}\)denotes the revised planner graph,𝒩s⊂𝒩\\mathcal\{N\}\_\{s\}\\subset\\mathcal\{N\}denotes its search nodes,𝒩a⊂𝒩\\mathcal\{N\}\_\{a\}\\subset\\mathcal\{N\}denotes its aggregate nodes, andvansv\_\{\\mathrm\{ans\}\}denotes the terminal answer node\. The functionϕ​\(⋅\)\\phi\(\\cdot\)is the sentence\-embedding encoder used for reward\-side semantic matching,cos⁡\(⋅,⋅\)\\cos\(\\cdot,\\cdot\)is cosine similarity in that embedding space, andmax⁡\(0,cos⁡\(⋅,⋅\)\)\\max\(0,\\cos\(\\cdot,\\cdot\)\)clamps negative similarities to zero\.

##### Planner\-stage reward\.

The planner rollout consists of an initial graph, environment search results, and a planner revision that emits the final graph and fetch requests\. Only the revised graph𝒢\\mathcal\{G\}is rewarded\. Invalid graphs receive zero reward:

Rplan=𝟏​\[Valid​\(𝒢\)\]​\(0\.40​Crub\+0\.30​Qsearch\+0\.30​Egraph\)\.R\_\{\\mathrm\{plan\}\}=\\mathbf\{1\}\[\\mathrm\{Valid\}\(\\mathcal\{G\}\)\]\\left\(0\.40\\,C\_\{\\mathrm\{rub\}\}\+0\.30\\,Q\_\{\\mathrm\{search\}\}\+0\.30\\,E\_\{\\mathrm\{graph\}\}\\right\)\.\(17\)Here𝟏​\[Valid​\(𝒢\)\]\\mathbf\{1\}\[\\mathrm\{Valid\}\(\\mathcal\{G\}\)\]is one only when the revised graph parses and satisfies the typed\-DAG constraints; otherwise the planner reward is zero\. The three terms correspond to distinct planner responsibilities:CrubC\_\{\\mathrm\{rub\}\}checks whether the plan creates places for the rubric content to be answered later,QsearchQ\_\{\\mathrm\{search\}\}checks whether the search nodes expose useful and non\-duplicative retrieval routes, andEgraphE\_\{\\mathrm\{graph\}\}checks whether the graph contains enough reusable synthesis structure for the answerer to execute\. For a rubric itemuuwith weightwuw\_\{u\}, letfields​\(n\)\\mathrm\{fields\}\(n\)be the non\-emptyquery,key\_points, andneedfields of a non\-answer nodenn\. Rubric coverageCrubC\_\{\\mathrm\{rub\}\}is the weighted average best match from each rubric item to any searchable or aggregate plan field:

Crub=∑uwu​maxn∈𝒩∖\{vans\}⁡maxc∈fields​\(n\)⁡max⁡\(0,cos⁡\(ϕ​\(u\),ϕ​\(c\)\)\)∑uwu\.C\_\{\\mathrm\{rub\}\}=\\frac\{\\sum\_\{u\}w\_\{u\}\\max\_\{n\\in\\mathcal\{N\}\\setminus\\\{v\_\{\\mathrm\{ans\}\}\\\}\}\\max\_\{c\\in\\mathrm\{fields\}\(n\)\}\\max\(0,\\cos\(\\phi\(u\),\\phi\(c\)\)\)\}\{\\sum\_\{u\}w\_\{u\}\}\.\(18\)Search quality is

Qsearch=0\.50​Csearch\+0\.50​Dsearch,Q\_\{\\mathrm\{search\}\}=0\.50\\,C\_\{\\mathrm\{search\}\}\+0\.50\\,D\_\{\\mathrm\{search\}\},\(19\)whereCsearchC\_\{\\mathrm\{search\}\}is computed likeCrubC\_\{\\mathrm\{rub\}\}but restricted to search\-node queries\. Letqiq\_\{i\}be the query string emitted by theiith search node in𝒩s\\mathcal\{N\}\_\{s\}\. The distinctness termDsearchD\_\{\\mathrm\{search\}\}is one minus the mean clamped cosine similarity over unordered pairs of search queries:

Dsearch=1−2\|𝒩s\|​\(\|𝒩s\|−1\)​∑i<jmax⁡\(0,cos⁡\(ϕ​\(qi\),ϕ​\(qj\)\)\)\.D\_\{\\mathrm\{search\}\}=1\-\\frac\{2\}\{\|\\mathcal\{N\}\_\{s\}\|\(\|\\mathcal\{N\}\_\{s\}\|\-1\)\}\\sum\_\{i<j\}\\max\(0,\\cos\(\\phi\(q\_\{i\}\),\\phi\(q\_\{j\}\)\)\)\.\(20\)When there are fewer than two search queries,DsearchD\_\{\\mathrm\{search\}\}is set to zero\. ThusCsearchC\_\{\\mathrm\{search\}\}rewards query\-level recall of important facets, whileDsearchD\_\{\\mathrm\{search\}\}discourages multiple search nodes from becoming paraphrases of the same retrieval request\.

Graph expressiveness is

Egraph=0\.40​Bsynth\+0\.40​Icross\+0\.20​Bsearch\.E\_\{\\mathrm\{graph\}\}=0\.40\\,B\_\{\\mathrm\{synth\}\}\+0\.40\\,I\_\{\\mathrm\{cross\}\}\+0\.20\\,B\_\{\\mathrm\{search\}\}\.\(21\)All three subterms use the diminishing\-returns functionf​\(k\)=max⁡\(0,1−1/max⁡\(k,1\)\)f\(k\)=\\max\(0,1\-1/\\max\(k,1\)\)\. LetPansP\_\{\\mathrm\{ans\}\}be the set of aggregate parents of the answer node anddpard\_\{\\mathrm\{par\}\}be one minus the mean pairwise normalized cosine similarity among their node texts\. Then

Bsynth\\displaystyle B\_\{\\mathrm\{synth\}\}=f​\(\|Pans\|\)​dpar,\\displaystyle=f\(\|P\_\{\\mathrm\{ans\}\}\|\)\\,d\_\{\\mathrm\{par\}\},\(22\)Icross\\displaystyle I\_\{\\mathrm\{cross\}\}=12​\(rfan\+rint\),\\displaystyle=\\tfrac\{1\}\{2\}\(r\_\{\\mathrm\{fan\}\}\+r\_\{\\mathrm\{int\}\}\),\(23\)Bsearch\\displaystyle B\_\{\\mathrm\{search\}\}=f​\(\|𝒩s\|\),\\displaystyle=f\(\|\\mathcal\{N\}\_\{s\}\|\),\(24\)whereBsynthB\_\{\\mathrm\{synth\}\}measures how many distinct aggregate branches feed the answer,IcrossI\_\{\\mathrm\{cross\}\}measures cross\-branch connectivity, andBsearchB\_\{\\mathrm\{search\}\}measures search breadth\. The termrfanr\_\{\\mathrm\{fan\}\}is the fraction of search nodes with at least two direct downstream children, so it rewards evidence reuse across branches\. The termrintr\_\{\\mathrm\{int\}\}is the fraction of aggregate nodes with at least two parent inputs, so it rewards synthesis nodes that combine evidence or upstream conclusions\. The diminishing\-returns function gives most of its gain when moving from one to two useful branches or search nodes, then saturates as counts grow; the diversity multiplierdpard\_\{\\mathrm\{par\}\}prevents duplicate answer\-parent branches from increasingBsynthB\_\{\\mathrm\{synth\}\}cheaply\.

##### Answerer\-stage reward\.

The answerer rollout executes the fixed planner graph in topological aggregate waves and then writes the terminal answer node\. For a fully parsed trajectory,

Rans=0\.50​Rexec\+0\.50​Rsynth\.R\_\{\\mathrm\{ans\}\}=0\.50\\,R\_\{\\mathrm\{exec\}\}\+0\.50\\,R\_\{\\mathrm\{synth\}\}\.\(25\)The execution term is local to aggregate nodes: it asks whether each branch report fulfilled the key points declared by the planner\. The synthesis term is reserved for the terminal answer, so missing branch coverage and poor final integration affect different parts of the reward\. Only aggregate nodes are scored for execution\. For each aggregate nodevvwith key pointsPvP\_\{v\}and generated node textovo\_\{v\},

rv=0\.50​Kemb​\(v\)\+0\.50​Klex​\(v\)\.r\_\{v\}=0\.50\\,K\_\{\\mathrm\{emb\}\}\(v\)\+0\.50\\,K\_\{\\mathrm\{lex\}\}\(v\)\.\(26\)A highrvr\_\{v\}therefore means that a node report covers its assigned content targets, not merely that it is fluent or long\. HerePvP\_\{v\}is the set of planner\-declared key points for aggregate nodevv, andovo\_\{v\}is the generated XML node text for that aggregate\. The embedding termKemb​\(v\)K\_\{\\mathrm\{emb\}\}\(v\)is the greedy one\-to\-one matching score between key pointsp∈Pvp\\in P\_\{v\}and output sentencesssfromovo\_\{v\}, normalized by the number of key points:

Kemb​\(v\)=1\|Pv\|​∑\(p,s\)∈match​\(v\)max⁡\(0,cos⁡\(ϕ​\(p\),ϕ​\(s\)\)\)\.K\_\{\\mathrm\{emb\}\}\(v\)=\\frac\{1\}\{\|P\_\{v\}\|\}\\sum\_\{\(p,s\)\\in\\mathrm\{match\}\(v\)\}\\max\(0,\\cos\(\\phi\(p\),\\phi\(s\)\)\)\.\(27\)The lexical term strips inline citations and averages key\-point token recall\. Heretok​\(⋅\)\\mathrm\{tok\}\(\\cdot\)denotes the set of normalized content tokens after citation removal:

Klex​\(v\)=1\|Pv\|​∑p∈Pv\|tok​\(p\)∩tok​\(ov\)\|\|tok​\(p\)\|\.K\_\{\\mathrm\{lex\}\}\(v\)=\\frac\{1\}\{\|P\_\{v\}\|\}\\sum\_\{p\\in P\_\{v\}\}\\frac\{\|\\mathrm\{tok\}\(p\)\\cap\\mathrm\{tok\}\(o\_\{v\}\)\|\}\{\|\\mathrm\{tok\}\(p\)\|\}\.\(28\)The embedding term gives credit for semantic coverage of key points, while the lexical term provides a deterministic guard against high semantic similarity without concrete key\-point content\. Node scores are combined with the shifted harmonic mean

Rexec=\|𝒩a\|∑v∈𝒩a11\+rv−1\.R\_\{\\mathrm\{exec\}\}=\\frac\{\|\\mathcal\{N\}\_\{a\}\|\}\{\\sum\_\{v\\in\\mathcal\{N\}\_\{a\}\}\\frac\{1\}\{1\+r\_\{v\}\}\}\-1\.\(29\)This preserves sensitivity to weak branches without turning one failed branch into a total zero reward\. We use the shifted harmonic mean rather than a min operator because early experiments with a hard minimum made GRPO advantage estimates unstable under small group sizes\.

Synthesis reward combines an LLM rubric judge and deterministic branch utilization:

Rsynth=0\.50​Jrub\+0\.50​Ubranch\.R\_\{\\mathrm\{synth\}\}=0\.50\\,J\_\{\\mathrm\{rub\}\}\+0\.50\\,U\_\{\\mathrm\{branch\}\}\.\(30\)These two subterms separate external answer quality from structural fidelity:JrubJ\_\{\\mathrm\{rub\}\}measures whether the final report satisfies the query\-level rubric, whileUbranchU\_\{\\mathrm\{branch\}\}checks whether the final report actually uses the branch reports produced during graph execution\. The judge termJrubJ\_\{\\mathrm\{rub\}\}is a rubric\-weighted average of five\-level scores\{0,0\.25,0\.5,0\.75,1\}\\\{0,0\.25,0\.5,0\.75,1\\\}returned by the configured judge \(gpt\-5\-miniin the reported run\)\. The five\-level scale is used because coarser three\-level scales merge adjacent rubric\-satisfaction levels, while broader numeric scales introduce more calibration noise across queries\. Branch utilization is

Ubranch=1\|B\|​∑b∈B\|tok​\(b\)∩tok​\(y\)\|\|tok​\(b\)\|,U\_\{\\mathrm\{branch\}\}=\\frac\{1\}\{\|B\|\}\\sum\_\{b\\in B\}\\frac\{\|\\mathrm\{tok\}\(b\)\\cap\\mathrm\{tok\}\(y\)\|\}\{\|\\mathrm\{tok\}\(b\)\|\},\(31\)whereBBis the set of aggregate branch outputs andyyis the final answer, both after stripping inline citations\. Each branch outputb∈Bb\\in Bis one completed aggregate\-node report\. This term is deterministic and targets the failure mode in which the answerer ignores its completed branch reports and writes the final answer directly from parametric knowledge or retrieved evidence\.

All reward\-side semantic matching uses thesentence\-transformers/all\-MiniLM\-L6\-v2embedding model\(Reimers and Gurevych,[2019](https://arxiv.org/html/2605.30824#bib.bib20)\)\.

##### Partial credit for parse failures\.

Planner parse or graph\-validity failures receive zero because the planner reward is defined on a valid typed DAG\. Answerer parse failures are handled differently: if some aggregate waves have already completed, missing aggregate nodes are assigned zero node scores and the rollout receives0\.50​Rexec0\.50\\,R\_\{\\mathrm\{exec\}\}withRsynth=0R\_\{\\mathrm\{synth\}\}=0; if the final answer fails to parse after all aggregate waves succeed, it receives the same execution credit with zero synthesis\. If no aggregate node is successfully parsed, the reward remains zero\. This change was used because fully zeroing a long answerer rollout after a late XML parse error produced unstable GRPO advantages despite useful completed branch reports\.

### A\.6Training Diagnostics

Figures[4](https://arxiv.org/html/2605.30824#A1.F4)and[5](https://arxiv.org/html/2605.30824#A1.F5)report the two training diagnostics omitted from the main text for space\. The planner graph curve shows that planner RL does not merely improve scalar reward: it also increases typed graph structure, with edge count growing faster than node count as the model learns to reuse evidence across branches\. The answerer length curve shows that responses become longer during answerer RL\. We interpret this as a consequence of key\-point coverage pressure rather than length reward hacking, since the active reward has no direct length term and the reward curve in Figure[3](https://arxiv.org/html/2605.30824#S3.F3)tracks execution and synthesis improvements\.

020204040606080801001001201201401401601601801802002002202202402402602602802803003003203200224466881010121214141616RL stepmean countGraph Size by Stepnodessearchaggregateedges

Figure 4:Planner graph size and typed component counts over RL\.050501001001501502002002502503003003503507,0007\{,\}0008,0008\{,\}0009,0009\{,\}00010,00010\{,\}000RL stepmean tokensAnswerer Response Length

Figure 5:Answerer response length over RL\.
### A\.7Reward Component Ablations

We ablate the planner and answerer reward components separately, removing one component at a time from the full reward and retraining from the SFT checkpoint with the same schedule, data, and fixed weights on the remaining components\. Results are reported in Table[7](https://arxiv.org/html/2605.30824#A1.T7)\.

##### C1: Planner reward components\.

The top half of Table[7](https://arxiv.org/html/2605.30824#A1.T7)reports leave\-one\-out ablations of the three top\-level planner reward terms\. Removing rubric coverageCrubC\_\{\\mathrm\{rub\}\}causes the largest drop on ResQA\-Mini, whose rubric\-based evaluations directly reward semantic coverage of specific aspects\. Removing search qualityQsearchQ\_\{\\mathrm\{search\}\}degrades all three benchmarks, falling from 51\.7 to 49\.1 on average, confirming that this term shapes retrieval coverage rather than only acting as a noise term\. Removing graph expressivenessEgraphE\_\{\\mathrm\{graph\}\}has a sizeable effect on final answer scores and causes graphs to collapse toward less integrated, simpler structures\. This contrasts with the full planner\-RL diagnostic in Figure[4](https://arxiv.org/html/2605.30824#A1.F4), where edge count grows faster than node count, indicating increasing cross\-branch reuse\.

##### C2: Answerer reward components\.

The bottom half of Table[7](https://arxiv.org/html/2605.30824#A1.T7)reports the same ablation for the answerer reward\. Removing per\-aggregate execution rewardRexecR\_\{\\mathrm\{exec\}\}causes branch\-level key\-point coverage to drop substantially, resulting in the largest drop\-off across all answerer sub\-components\. Removing the rubric judgeJrubJ\_\{\\mathrm\{rub\}\}reduces final answer quality on all three benchmarks\. The branch utilization termUbranchU\_\{\\mathrm\{branch\}\}has the smallest standalone effect on benchmark scores, but when removed the content overlap between the final answer and intermediate branch outputs drops from 0\.82 to 0\.61, indicating that withoutUbranchU\_\{\\mathrm\{branch\}\}the answerer increasingly ignores its own branch outputs and re\-derives the final report from scratch\.

Table 7:Leave\-one\-out ablation of reward components\. All variants keep the fixed reward weights on remaining components unchanged\.

### A\.8Plan Representation and Revision Ablations

This ablation complements the reward\-component analysis in Appendix[A\.7](https://arxiv.org/html/2605.30824#A1.SS7)by isolating two structural choices in the planner: whether the plan is a typed DAG rather than a weaker linear or tree structure, and whether the planner can revise the plan after seeing search results\. Results are reported in Table[8](https://arxiv.org/html/2605.30824#A1.T8)\.

##### D1: Plan structure ablation\.

We replace the typed DAG with two alternative plan representations while retaining the rest of the DecomposeR training pipeline\.Linear: the plan consists of search nodes feeding directly into the answer node with no aggregate layer, approximating the parallel\-subquery structures used by ParallelSearch and SubSearch; the graph expressiveness rewardEgraphE\_\{\\mathrm\{graph\}\}is removed because there is no DAG structure to score, leavingCrubC\_\{\\mathrm\{rub\}\}andQsearchQ\_\{\\mathrm\{search\}\}active\.Tree: aggregate nodes are restricted to at most one parent, approximating hierarchical outline structures of WebWeaver\-style systems; the structural rewards remain active butrintr\_\{\\mathrm\{int\}\}inIcrossI\_\{\\mathrm\{cross\}\}is necessarily zero\. The top half of Table[8](https://arxiv.org/html/2605.30824#A1.T8)shows that the typed DAG outperforms both Linear and Tree on every benchmark, with the largest gap on tasks that require synthesizing evidence across multiple thematic branches\.

##### D2: Single\-turn vs\. two\-turn planning\.

We further ablate the two\-turn planner protocol by training a single\-turn variant in which the planner emits𝒢0\\mathcal\{G\}\_\{0\}and immediately commits to it, with no opportunity to revise after seeing search results\. The bottom half of Table[8](https://arxiv.org/html/2605.30824#A1.T8)shows that two\-turn planning improves over single\-turn planning on all three benchmarks, consistent with the intuition that an initial plan generated from parametric knowledge alone benefits from being adjusted to what is actually retrievable\.

Table 8:Plan representation ablations\. D1 replaces the typed DAG with a linear \(parallel\-subquery\) or tree \(hierarchical outline\) structure\. D2 disables the revision turn that follows initial search\.

## Appendix BPrompt and Rollout Appendix

This appendix records the exact system prompts used by the DecomposeR pipeline and a model\-output rollout from the final 8B model\. Planner JSON, evidence payloads, and long answerer text are excerpted to avoid printing full fetched pages or repetitive answer prose; the graph, turn flow, execution waves, final answer structure, and citation catalog are preserved\.

### B\.1Execution and SFT Prompts

This subsection groups the static system prompts and turn\-specific user payloads used for production rollouts and cold\-start SFT\. The initial planner turn receives only the raw query; after search, the planner receives a structured revision payload\. The answerer then receives aggregate\-wave payloads and a final\-answer payload\. The SFT corpus stores these same prompt/completion turns from teacher trajectories\.

##### Planner prompts\.

`Figure 6: Planner system prompt Figure 7: Planner revision\-turn user payload`

`Answerer prompts\. Figure 8: Answerer system prompt Figure 9: Answerer aggregate\-wave user payload Figure 10: Answerer final\-answer user payload SFT trajectory generation\. The same planner and answerer prompts above are used for cold\-start SFT trajectory generation\. Teacher rollouts follow the production sequence: the planner first receives the raw query and emits an initial graph; the environment executes the initial search nodes; the planner then receives the after\-search payload, revises the graph, and selects URLs to fetch; the environment executes any newly added searches and fetches the selected pages; and the answerer executes aggregate waves and the terminal answer node using the answerer prompts above\. The turn\-level SFT corpus stores the resulting planner and answerer chat turns as prompt/completion examples\. B\.2 Judge and Reward Prompts The answerer synthesis reward uses two terms: the LLM final\-answer rubric judge below and deterministic branch utilization\. Branch utilization is computed as the mean fraction of each aggregate branch’s content tokens that appear in the final answer, after stripping inline citations; it does not call an LLM judge and has no prompt\. Figure 11: Rubric generation system prompt Figure 12: Answer judge system prompt Appendix C Example Planner–Answerer Rollout Figure 13: User query Abbreviation policy\. This example keeps the control flow and graph structure intact while excerpting long evidence payloads and repetitive answer prose\. The important structural property is that the revised graph has two aggregate execution waves before the final answer: N7–N9 are executed first, N10 is executed second, and the final answer consumes N7, N8, and N10\. N9 therefore shapes the final report indirectly through N10 rather than as a direct parent of the answer node\. Turn Flow\. Table 9: Turn flow for the agroforestry nitrogen case\-study rollout\. Turn Model / environment action Nodes or artifacts Planner turn 1 Emit initial typed DAG Search nodes N1–N5, aggregate nodes N6–N8, answer node N9\. Environment Execute initial searches with Jina Search Search results attached to N1–N5\. Planner turn 2 Revise graph and request fetches Adds search node N6, revises aggregate nodes N7–N10, answer node N11, and selects four fetches\. Environment Execute new search and fetch selected pages with Serper New search: N6\. Fetched evidence: N3\-F1, N3\-F2, N3\-F3, N5\-F4\. Answerer wave 1 Execute ready aggregate nodes N7: tree\-root uptake and retention; N8: agroforestry nitrogen cycling; N9: preferential flow and soil transport\. Answerer wave 2 Execute dependent aggregate node N10: evidence that subsoil nitrogen leaching is reduced by tree\-root systems\. Answerer final Write terminal answer node N11 integrated answer using N7, N8, and N10\. Revised Graph Summary\. Table 10: Revised graph summary for the agroforestry nitrogen rollout\. Node Type Need Inputs N1 search Mechanisms by which tree roots take up nitrogen and reduce nitrogen loss – N2 search Agroforestry intercropping systems and how they influence nitrogen cycling – N3 search Evidence on subsoil nitrogen leaching reduction in agroforestry or tree\-root systems – N4 search Role of root architecture, depth, and temporal dynamics in nitrogen capture – N5 search Soil water and nutrient transport processes affecting leaching – N6 search Direct evidence on preferential flow and how tree roots alter leaching pathways – N7 aggregate How tree roots reduce nitrogen loss through uptake and retention N1, N4 N8 aggregate How agroforestry intercropping changes nitrogen cycling and belowground environment N2, N5 N9 aggregate How preferential flow and soil transport processes influence leaching N5, N6 N10 aggregate Evidence that subsoil nitrogen leaching is reduced by tree\-root systems N3, N5, N9 N11 answer Integrated explanation of how nitrogen uptake by tree roots in agroforestry intercropping reduces subsoil nitrogen leaching N7, N8, N10 Figure 14: Planner initial output \(abbreviated\) Figure 15: Initial agroforestry nitrogen planner graph before search\-result revision\. Red nodes are search nodes, blue nodes are aggregate nodes, and the orange node is the terminal answer node\. Figure 16: Planner revision output \(abbreviated\) Figure 17: Revised graph after search\-result revision\. Figure 18: Answerer execution payload excerpt \(abbreviated\) Figure 19: Answerer turn 1 excerpt: answer\_execution \(N7, N8, N9\) Figure 20: Answerer turn 2 excerpt: answer\_execution \(N10\) Figure 21: Answerer final output: answer\_final \(N11\) Citation Catalog Used by the Final Answer\. Table 11: Citation catalog used by the final answer in the agroforestry nitrogen rollout\. ID Kind Node Title URL N1\-R3 search N1 Molecular fundamentals of nitrogen uptake and transport in trees https://academic\.oup\.com/jxb/article/68/10/2489/3091614 N1\-R4 search N1 Nitrogen Journey in Plants: From Uptake to Metabolism, Stress … https://pmc\.ncbi\.nlm\.nih\.gov/articles/PMC10605003/ N2\-R1 search N2 Belowground Dynamics Influence Nitrogen Cycling and Crop … https://www\.frontiersin\.org/journals/sustainable\-food\-systems/articles/10\.3389/fsufs\.2021\.705577/full N2\-R2 search N2 Nitrogen Cycling and Root Dynamics in an Agroforestry System https://ui\.adsabs\.harvard\.edu/abs/2024EGUGA\.\.26\.4555E/abstract N2\-R3 search N2 Below\-ground interactions in dryland agroforestry https://www\.sciencedirect\.com/science/article/abs/pii/S0378112798003223 N2\-R4 search N2 Nitrogen dynamics in agroforestry systems\. A review https://hal\.science/hal\-04140609/document N3\-F1 fetch N3 Nitrogen dynamics in agroforestry systems\. A review https://link\.springer\.com/article/10\.1007/s13593\-022\-00791\-7 N3\-F2 fetch N3 Expanding agroforestry can increase nitrate retention https://www\.nature\.com/articles/s43016\-022\-00657\-x N3\-R5 search N3 Reduced soil nutrient leaching following the establishment of tree … https://www\.researchgate\.net/publication/225134004\_Reduced\_soil\_nutrient\_leaching\_following\_the\_establishment\_of\_tree\-based\_intercropping\_systems\_in\_eastern\_Canada N4\-R1 search N4 Root phenotypes for improved nitrogen capture https://pmc\.ncbi\.nlm\.nih\.gov/articles/PMC11420291/ N4\-R4 search N4 What Is Root System Architecture and Why Does It Matter? https://cid\-inc\.com/blog/what\-is\-root\-system\-architecture\-and\-why\-does\-it\-matter\-in\-plants/ N5\-F4 fetch N5 Characterization of preferential flow and transport pathways under farmland with different land uses https://www\.sciencedirect\.com/science/article/abs/pii/S0167198725001278 N5\-R1 search N5 Characterization of preferential flow and transport pathways under … https://www\.sciencedirect\.com/science/article/abs/pii/S0167198725001278 N5\-R2 search N5 Characteristics of Preferential Flow Paths and Their Impact on … https://www\.pjoes\.com/pdf\-89388\-23248?filename=Characteristics\-of\-Prefer\.pdf Figure 22: Citation block appended to final answer`