TalentRL：用于层次化技能路由的策略引导强化学习

# Strategy-Guided Reinforcement Learningfor Hierarchical Skill Routing
Source: [https://arxiv.org/html/2610.41143](https://arxiv.org/html/2610.41143)
## TalentRL: Strategy\-Guided Reinforcement Learning for Hierarchical Skill Routing

Xiaodi SunZhaoqiong HuangMichelle ChangJingwen LuAffiliation:Microsoft AIEmail:[huancchen@microsoft\.com](mailto:)

###### Abstract

Natural\-language skills enable large language model \(LLM\) agents to reuse procedures across tasks, yet most systems retrieve individual skills from a flat SkillBank without explicitly modeling the high\-level strategy that governs their selection and use\. We introduce TalentRL, a strategy\-guided reinforcement learning framework for hierarchical routing over procedural memory\. Coarse\-grained*talents*represent recurring strategies for skill use, while fine\-grained*skills*provide executable procedures\. For each task, the policy model selects a talent that guides execution and softly prioritizes its associated skills during similarity\-based retrieval\. However, since task outcomes depend jointly on talent selection, retrieved skill context, and downstream execution, terminal rewards alone cannot isolate the contribution of the routing decision\. TalentRL therefore trains talent routing with reward contrasts between rollouts of the same task under matched interventions on the talent\-conditioned context, and optimizes downstream execution with group\-relative advantages\. Across seven open\-domain and multi\-hop question\-answering benchmarks, talent\-aware supervised fine\-tuning improves average success rate by 7\.53%, 6\.71%, and 9\.21% over the corresponding zero\-shot baselines for Qwen2\.5\-7B, Qwen3\-8B, and Llama\-3\.1\-8B, respectively\. Reinforcement learning further improves these SFT checkpoints by 7\.58% to 9\.32%, and TalentRL achieves the highest average success rate among all compared methods on all three backbones\. On the agentic benchmarks ALFWorld and WebShop, TalentRL achieves the highest ALFWorld success rate \(99\.3%\) and the highest WebShop score \(90\.7\) among all compared methods\.

## 1Introduction

Large language model \(LLM\) agents solve interactive tasks by interleaving reasoning, retrieval, and environment actions\[[42](https://arxiv.org/html/2610.41143#bib.bib42)\]\. To improve behavior across episodes, verbal\-memory methods retain linguistic reflections from prior interactions\[[25](https://arxiv.org/html/2610.41143#bib.bib25)\], while outcome\-based reinforcement learning \(RL\) optimizes the policy using task rewards\[[12](https://arxiv.org/html/2610.41143#bib.bib12)\]\. Procedural\-memory systems further store reusable skills distilled from past experience, yet most retrieve individual skills from a flat SkillBank without having the policy model explicitly select a high\-level strategy to guide their selection and use\. This motivates a central question: how can an agent learn, from task\-level outcomes, to select a high\-level strategy that organizes its procedural memory?

Prior work has progressed from extracting reusable insights across trajectories to building evolving libraries of procedural skills from successful and failed interactions\[[49](https://arxiv.org/html/2610.41143#bib.bib49),[19](https://arxiv.org/html/2610.41143#bib.bib19),[46](https://arxiv.org/html/2610.41143#bib.bib46),[34](https://arxiv.org/html/2610.41143#bib.bib34)\]\. Recent methods couple skill generation or curation with hierarchical, joint, or cross\-task RL\[[15](https://arxiv.org/html/2610.41143#bib.bib15),[24](https://arxiv.org/html/2610.41143#bib.bib24),[43](https://arxiv.org/html/2610.41143#bib.bib43)\]\. However, two challenges remain\. First, existing retrieval and selection methods typically operate directly on individual skills, leaving the high\-level strategy that governs their selection and use neither explicit nor learned\. Second, terminal rewards reflect the joint effects of strategy selection, skill retrieval, and execution, and therefore cannot isolate the contribution of strategy selection\.

To this end, we propose TalentRL, which separates strategy selection, skill retrieval, and downstream execution through a predefined hierarchical structure for procedural memory\. Coarse\-grained*talents*represent recurring strategies for skill use, while fine\-grained*skills*encode executable procedures\. For each task, the policy model selects a talent, which guides execution and softly prioritizes its associated skills during retrieval\. Specifically, the retriever combines task–skill similarity with this talent prior and supplies the top\-ranked skills to the policy model during execution\. Before RL, we construct talent\-aware supervised fine\-tuning \(SFT\) demonstrations from paired rollouts to initialize both talent routing and downstream execution\.

TalentRL addresses the routing credit\-assignment challenge by training talent routing with reward contrasts across matched intervention rollouts and optimizing downstream execution with group\-relative advantages\. Specifically, the interventions hold the task fixed while withholding retrieved skills, reducing the retrieval budget, or removing both talent guidance and its retrieval prior\. These reward contrasts therefore measure the utility of the talent\-conditioned context, rather than the effect of the talent label alone\. TalentRL applies the intervention\-derived routing signal to tokens in the talent\-selection response, while tokens in subsequent execution responses receive group\-relative advantages\. Our contributions are threefold:

- •We propose an outcome\-guided talent\-aware SFT data generation pipeline that compares candidate talents through paired rollouts and converts positive\-utility talent\-conditioned trajectories into routing and execution supervision\.
- •We introduce TalentRL, a strategy\-guided RL framework that combines hierarchical skill routing with intervention\-based credit for talent selection and group\-relative optimization of downstream execution\.
- •We demonstrate that talent\-aware SFT improves performance across seven open\-domain and multi\-hop question\-answering benchmarks, and subsequent RL achieves the highest average success rate among all compared methods on all three backbones\. On ALFWorld and WebShop, TalentRL achieves the highest ALFWorld success rate and WebShop score among all compared methods\.

## 2Related Work

![Refer to caption](https://arxiv.org/html/2610.41143v1/talentrl.png)Figure 1:Overview of TalentRL\.\(A\)The policy model selects a talent that guides downstream execution and provides a soft prior for similarity\-based skill retrieval\.\(B\)Reward contrasts across matched intervention rollouts train talent routing, while group\-relative advantages optimize downstream execution, both updating the shared policy model\.\(C\)Talent\-aware SFT initializes talent routing and downstream execution from outcome\-guided demonstrations\.### 2\.1Skill Memory and Representation

Procedural memory, which the CoALA taxonomy\[[27](https://arxiv.org/html/2610.41143#bib.bib27)\]distinguishes from episodic and semantic memory, enables LLM agents to transfer experience across tasks by storing reusable knowledge of how to act\. Existing methods represent such knowledge in different forms and at different levels of abstraction\. Natural\-language representations include verbal feedback in Reflexion\[[25](https://arxiv.org/html/2610.41143#bib.bib25)\], cross\-trajectory insights in ExpeL\[[49](https://arxiv.org/html/2610.41143#bib.bib49)\], context\-conditioned guidelines in AutoGuide\[[6](https://arxiv.org/html/2610.41143#bib.bib6)\], and step\-level instructions or higher\-level scripts in MemP\[[4](https://arxiv.org/html/2610.41143#bib.bib4)\]\. Buffer of Thoughts\[[38](https://arxiv.org/html/2610.41143#bib.bib38)\]retains reusable reasoning templates, while AWM\[[32](https://arxiv.org/html/2610.41143#bib.bib32)\]abstracts trajectories into workflows\.

Executable procedural memory represents reusable behavior as programs or callable tools, as in Voyager\[[30](https://arxiv.org/html/2610.41143#bib.bib30)\], LATM\[[1](https://arxiv.org/html/2610.41143#bib.bib1)\], and CRAFT\[[44](https://arxiv.org/html/2610.41143#bib.bib44)\]\. Together with natural\-language memory, these approaches maintain procedural knowledge outside model parameters, where it can be inspected, retrieved, invoked, or composed at inference time\. In contrast, SKILL0\[[17](https://arxiv.org/html/2610.41143#bib.bib17)\]internalizes procedural knowledge into model parameters, while SoftSkill\[[28](https://arxiv.org/html/2610.41143#bib.bib28)\]encodes it as continuous representations\. Our work focuses on external natural\-language skills, which remain inspectable and can be retrieved and composed at inference time\.

### 2\.2Self\-Evolving Skill Learning

Self\-evolving skill memory methods use accumulated agent experience to update skill libraries\. One line of work updates external skills without changing the policy model’s parameters, through procedure revision in Skill\-Pro\[[19](https://arxiv.org/html/2610.41143#bib.bib19)\], validation\-guided editing in SkillOpt\[[39](https://arxiv.org/html/2610.41143#bib.bib39)\], consensus\-gated evolution followed by structure specialization in SkillSpec\[[2](https://arxiv.org/html/2610.41143#bib.bib2)\], or trajectory\-based consolidation in Trace2Skill\[[21](https://arxiv.org/html/2610.41143#bib.bib21)\]\.

Another line of work co\-adapts procedural memory and the policy model\. SkillRL\[[34](https://arxiv.org/html/2610.41143#bib.bib34)\], CoSkill\[[5](https://arxiv.org/html/2610.41143#bib.bib5)\], and SAGE\[[31](https://arxiv.org/html/2610.41143#bib.bib31)\]couple skill\-library evolution with RL\. Skill1\[[24](https://arxiv.org/html/2610.41143#bib.bib24)\]unifies skill selection, utilization, and distillation; SkillRise\[[43](https://arxiv.org/html/2610.41143#bib.bib43)\]credits skill edits according to their cross\-task utility; ReSkill\[[9](https://arxiv.org/html/2610.41143#bib.bib9)\]embeds skill creation into GRPO\-based policy optimization; SkillForge\[[7](https://arxiv.org/html/2610.41143#bib.bib7)\]manages the skill library through fitness\-driven skill lifecycles; and SearchSkill\[[11](https://arxiv.org/html/2610.41143#bib.bib11)\]evolves skills for search\. Following SkillRL\[[34](https://arxiv.org/html/2610.41143#bib.bib34)\], TalentRL also evolves the SkillBank during RL\.

### 2\.3Skill Routing and Credit Assignment

Agents must determine which skills to retrieve and how to use or combine them for a given task\. One line of work develops structured mechanisms for skill retrieval and composition\. SkillRouter\[[51](https://arxiv.org/html/2610.41143#bib.bib51)\]studies full\-text retrieval and reranking at registry scale, while SkillLens\[[20](https://arxiv.org/html/2610.41143#bib.bib20)\]organizes strategies, procedures, and primitives in a multi\-granularity graph\. SkillComposer\[[50](https://arxiv.org/html/2610.41143#bib.bib50)\]jointly predicts a skill subset, its cardinality, and the execution order of the selected skills; HyperSkill\[[35](https://arxiv.org/html/2610.41143#bib.bib35)\]connects subtasks and reusable skills through a hypergraph\.

Another line of work studies learning signals for skill selection and the evaluation of skill use\. ARISE\[[15](https://arxiv.org/html/2610.41143#bib.bib15)\]introduces a hierarchical manager–worker policy with skill\-level rewards, while RSIPO\[[45](https://arxiv.org/html/2610.41143#bib.bib45)\]uses a skill\-conditioned teacher to provide token\-level supervision during agentic RL\. SkillCoach\[[52](https://arxiv.org/html/2610.41143#bib.bib52)\]assesses skill selection, following, composition, and reflection, providing a more detailed view of skill use than final task success alone\. When the policy model selects a high\-level strategy to guide skill retrieval, terminal rewards reflect both that selection and downstream execution, making the routing decision difficult to evaluate on its own\. TalentRL addresses this credit\-assignment challenge by training talent routing with reward contrasts across matched intervention rollouts and optimizing downstream execution with group\-relative advantages\.

## 3TalentRL: Strategy\-Guided Skill Routing

Existing skill\-augmented LLM frameworks such as SkillRL\[[34](https://arxiv.org/html/2610.41143#bib.bib34)\]retrieve task\-relevant skills by similarity before execution, without requiring the policy model to explicitly select a high\-level strategy\. In contrast, TalentRL introduces a policy\-selected talent that guides both skill retrieval and downstream execution\. We first formalize skill\-augmented task execution, then describe talent organization, routing, retrieval, and execution, followed by talent\-aware SFT data construction\.

### 3\.1Preliminaries

#### LLM agents with skill memory\.

Let𝒟\\mathcal\{D\}denote the task distribution andx∼𝒟x\\sim\\mathcal\{D\}a task description\. The agent has access to a SkillBank𝒮\\mathcal\{S\}of natural\-language skills, each describing a reusable procedure\. Given a taskxx, the retriever ranks skillss∈𝒮s\\in\\mathcal\{S\}by their embedding similarityq⁡\(x,s\)q\(x,s\)to the task and retrieves the top\-KKskills:

𝒮K​\(x\)=TopKs∈𝒮⁡q⁡\(x,s\),\\mathcal\{S\}\_\{K\}\(x\)=\\operatorname\{TopK\}\_\{s\\in\\mathcal\{S\}\}q\(x,s\),\(1\)whereKKis the retrieval budget andTopK\\operatorname\{TopK\}returns theKKhighest\-scoring skills\. Leth0=xh\_\{0\}=x\. At execution turnj≥1j\\geq 1, the agent observes the historyhj−1=\(x,a1,o1,…,aj−1,oj−1\)h\_\{j\-1\}=\(x,a\_\{1\},o\_\{1\},\\ldots,a\_\{j\-1\},o\_\{j\-1\}\), whereaja\_\{j\}denotes the action at turnjjandojo\_\{j\}the corresponding environment observation\. Conditioned on the interaction history and retrieved skills, the policy modelπθ\\pi\_\{\\theta\}, parameterized byθ\\theta, samples the next action:

aj∼πθ\(⋅∣hj−1,𝒮K\(x\)\)\.a\_\{j\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid h\_\{j\-1\},\\mathcal\{S\}\_\{K\}\(x\)\)\.\(2\)The environment returnsojo\_\{j\}, and the interaction history is updated by appending\(aj,oj\)\(a\_\{j\},o\_\{j\}\)tohj−1h\_\{j\-1\}\. Letτ=\(x,a1,o1,…,aT,oT\)\\tau=\(x,a\_\{1\},o\_\{1\},\\ldots,a\_\{T\},o\_\{T\}\)denote the interaction trajectory, whereTTis the number of execution turns\. We maximize the expected episodic return:

J\(θ\)=𝔼x∼𝒟,τ∼πθ\(⋅∣x\)\[R\(τ\)\],J\(\\theta\)=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\},\\,\\tau\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\left\[R\(\\tau\)\\right\],\(3\)whereR⁡\(τ\)R\(\\tau\)is the episodic return of trajectoryτ\\tau\. In this formulation, retrieval depends only on task similarity and is not learned, while the policy model learns to execute tasks using the retrieved skills\. We next introduce talent selection as a policy action that guides both retrieval and execution\.

### 3\.2Talent organization, routing, and execution

Let𝒯\\mathcal\{T\}denote a catalog of talents, each representing a coarse\-grained strategy\. Each talentc∈𝒯c\\in\\mathcal\{T\}has a descriptiondcd\_\{c\}for talent routing, guidancegcg\_\{c\}for task execution, and a set of associated skillsℳ⁡\(c\)⊆𝒮\\mathcal\{M\}\(c\)\\subseteq\\mathcal\{S\}\. Each skill is associated with one or more talents\. Following the trajectory\-distillation strategy of SkillRL\[[34](https://arxiv.org/html/2610.41143#bib.bib34)\], we use an LLM to analyze successful and failed trajectories, distill reusable procedures into skills, and identify recurring strategies to represent as talents\. During RL, the SkillBank, talents, and talent–skill associations are updated following SkillRL\[[34](https://arxiv.org/html/2610.41143#bib.bib34)\]\. Table[5](https://arxiv.org/html/2610.41143#A2.T5)in Appendix[B](https://arxiv.org/html/2610.41143#A2)lists the eight initial talents for search tasks and their preset skill associations\.

#### Talent routing\.

TalentRL uses a shared policyπθ\\pi\_\{\\theta\}for both routing and execution\. Before interacting with the environment, the policy model observes the task and talent catalog and samples

c∼πθ\(⋅∣x,𝒯\),c∈𝒯\.c\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x,\\mathcal\{T\}\),\\qquad c\\in\\mathcal\{T\}\.\(4\)Its first response contains a routing rationale in<think\>and the selected talent identifier in<talent\>; together, these fields constitute the routing action\. The following simplified example shows the policy model’s talent\-selection response and the guidance supplied for execution\.

Talent\-selection responseTask:Where was the author of the novel that inspired Blade Runner born? Model Response: <think\>Identify the source novel and its author before retrieving the author’s birthplace\.</think\> <talent\>talent\_bridge\_entity</talent\> Talent Guidance \(gcg\_\{c\}\): Resolve the bridge entity first, preserve relation direction, then issue a targeted follow\-up query for the requested attribute\.

#### Talent\-conditioned skill retrieval\.

For taskxx, the retriever first constructs a candidate poolℛ⁡\(x\)⊆𝒮\\mathcal\{R\}\(x\)\\subseteq\\mathcal\{S\}of the skills with the highest similarityq⁡\(x,s\)q\(x,s\)\. Given the selected talentcc, it reranks these candidates using

q~\(x,s∣c\)=q\(x,s\)\+λ𝕀\[s∈ℳ\(c\)\],s∈ℛ\(x\),\\widetilde\{q\}\(x,s\\mid c\)=q\(x,s\)\+\\lambda\\mathbb\{I\}\\\!\\left\[s\\in\\mathcal\{M\}\(c\)\\right\],\\qquad s\\in\\mathcal\{R\}\(x\),\(5\)whereλ\>0\\lambda\>0is the score bonus assigned to skills associated withcc, and𝕀\\mathbb\{I\}is the indicator function\. The retriever then selects the top\-KKskills for execution:

𝒮K​\(x,c\)=TopKs∈ℛ⁡\(x\)⁡q~​\(x,s∣c\)\.\\mathcal\{S\}\_\{K\}\(x,c\)=\\operatorname\{TopK\}\_\{s\\in\\mathcal\{R\}\(x\)\}\\widetilde\{q\}\(x,s\\mid c\)\.\(6\)Talent–skill associations thus provide a soft prior rather than a hard filter, allowing highly relevant skills outsideℳ⁡\(c\)\\mathcal\{M\}\(c\)to be selected from the candidate pool\. The policy model guides skill selection through its talent choice, while the retriever selects individual skills by reranked similarity within the budgetKK\.

#### Talent\-conditioned execution\.

The same policy model generates execution actions conditioned on the history, talent guidance, and retrieved skills:

aj∼πθ\(⋅∣hj−1,gc,𝒮K\(x,c\)\)\.a\_\{j\}\\sim\\pi\_\{\\theta\}\\\!\\left\(\\cdot\\mid h\_\{j\-1\},g\_\{c\},\\mathcal\{S\}\_\{K\}\(x,c\)\\right\)\.\(7\)τ=\(x,c,𝒮K​\(x,c\),a1,o1,…,aT,oT\)\\tau=\(x,c,\\mathcal\{S\}\_\{K\}\(x,c\),a\_\{1\},o\_\{1\},\\ldots,a\_\{T\},o\_\{T\}\)denotes the resulting trajectory\. The objective in Equation[3](https://arxiv.org/html/2610.41143#S3.E3)averages over both talent selection and task execution\.

Algorithm 1TalentRL1:

Initial policy modelπ0\\pi\_\{0\}; environmentℰ\\mathcal\{E\}; task sets𝒟sft,𝒟rl\\mathcal\{D\}\_\{\\mathrm\{sft\}\},\\mathcal\{D\}\_\{\\mathrm\{rl\}\};talent catalog𝒯\\mathcal\{T\}; SkillBank𝒮\\mathcal\{S\}; skill budgetKK
2:Trained policy

πθ\\pi\_\{\\theta\}
3:// Phase I: Talent\-Aware SFT

4:

𝒟talent←∅\\mathcal\{D\}\_\{\\mathrm\{talent\}\}\\leftarrow\\varnothing
5:foreach task

x∈𝒟sftx\\in\\mathcal\{D\}\_\{\\mathrm\{sft\}\}do

6:Collect

nnreference rollouts

τ∅,r\\tau\_\{\\varnothing,r\}with similarity\-only skills\.

7:For each

c∈𝒯c\\in\\mathcal\{T\}, collect

nnrollouts

τc,r\\tau\_\{c,r\}using

\(gc,𝒮K​\(x,c\)\)\(g\_\{c\},\\mathcal\{S\}\_\{K\}\(x,c\)\)\.

8:

u¯c​\(x\)←1n​∑r=1nuc,r​\(x\),c∈𝒯\\overline\{u\}\_\{c\}\(x\)\\leftarrow\\frac\{1\}\{n\}\\sum\_\{r=1\}^\{n\}u\_\{c,r\}\(x\),\\hskip 8\.50012ptc\\in\\mathcal\{T\}\(Section[3\.3](https://arxiv.org/html/2610.41143#S3.SS3)\)

9:

c∗←arg⁡maxc∈𝒯​u¯c​\(x\)c^\{\*\}\\leftarrow\\arg\\max\_\{c\\in\\mathcal\{T\}\}\\overline\{u\}\_\{c\}\(x\)
10:if

u¯c∗​\(x\)\>0\\overline\{u\}\_\{c^\{\*\}\}\(x\)\>0then

11:Retain a completed, valid

τc∗,r\\tau\_\{c^\{\*\},r\}with

uc∗,r​\(x\)\>0u\_\{c^\{\*\},r\}\(x\)\>0\.

12:Add its routing and execution targets to

𝒟talent\\mathcal\{D\}\_\{\\mathrm\{talent\}\}\.

13:Fine\-tune

π0\\pi\_\{0\}on

𝒟talent\\mathcal\{D\}\_\{\\mathrm\{talent\}\}using

ℒsft\\mathcal\{L\}\_\{\\mathrm\{sft\}\}\(Eq\.[9](https://arxiv.org/html/2610.41143#S3.E9)\) to obtain

πθ\\pi\_\{\\theta\}\.

14:Initialize and freeze

πref←πθ\\pi\_\{\\mathrm\{ref\}\}\\leftarrow\\pi\_\{\\theta\}\.

15:// Phase II: Matched\-Intervention RL

16:foreach optimization iterationdo

17:foreach task

xxin a batch sampled from

𝒟rl\\mathcal\{D\}\_\{\\mathrm\{rl\}\}do

18:Sample

ca∼πθ\(⋅∣x,𝒯\)c\_\{\\mathrm\{a\}\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x,\\mathcal\{T\}\); collect

τa\\tau\_\{\\mathrm\{a\}\}using

\(gca,𝒮K​\(x,ca\)\)\(g\_\{c\_\{\\mathrm\{a\}\}\},\\mathcal\{S\}\_\{K\}\(x,c\_\{\\mathrm\{a\}\}\)\)\.

19:Collect controls

τ−s,τk,τnp\\tau\_\{\-s\},\\tau\_\{k\},\\tau\_\{\\mathrm\{np\}\}as in Section[4\.1](https://arxiv.org/html/2610.41143#S4.SS1)\.

20:Collect ordinary rollouts; let

𝒫\\mathcal\{P\}index

τa\\tau\_\{\\mathrm\{a\}\}and the ordinary rollouts\.

21:Compute all

RiR\_\{i\}; obtain

A^iexec\\hat\{A\}\_\{i\}^\{\\mathrm\{exec\}\}for

i∈𝒫i\\in\\mathcal\{P\}\(Eq\.[10](https://arxiv.org/html/2610.41143#S4.E10)\)\.

22:Compute

A^talent\\hat\{A\}^\{\\mathrm\{talent\}\}\(Eq\.[13](https://arxiv.org/html/2610.41143#S4.E13)\) and assign

Ai,tA\_\{i,t\}\(Eq\.[14](https://arxiv.org/html/2610.41143#S4.E14)\)\.

23:Update

πθ\\pi\_\{\\theta\}using the masked, KL\-regularized

ℒrl\\mathcal\{L\}\_\{\\mathrm\{rl\}\}\(Eq\.[15](https://arxiv.org/html/2610.41143#S4.E15)\)\.

24:return

πθ\\pi\_\{\\theta\}

Table 1:Comparison of SkillRL\-SFT and TalentRL\-SFT on Search7, with zero\-shot results as a reference\. Success rates \(%\) are reported, with the best result for each metric within each backbone in bold\.

### 3\.3Talent\-aware SFT data construction

Before RL, we use talent\-aware SFT to initialize the policy model for both talent routing and downstream execution\. For each taskxxin the training set𝒟sft\\mathcal\{D\}\_\{\\mathrm\{sft\}\}and each talentc∈𝒯c\\in\\mathcal\{T\}, we collectnntrajectoriesτc,r\\tau\_\{c,r\}conditioned on guidancegcg\_\{c\}and retrieved skills𝒮K​\(x,c\)\\mathcal\{S\}\_\{K\}\(x,c\), wherer=1,…,nr=1,\\ldots,nindexes the rollouts\. For comparison, we collectnnreference trajectoriesτ∅,r\\tau\_\{\\varnothing,r\}for the same task using similarity\-based skill retrieval, without talent guidance or a talent\-specific retrieval prior\. We define the utility of each talent\-conditioned rollout as

uc,r​\(x\)=R⁡\(τc,r\)−R⁡\(τ∅,r\)\.u\_\{c,r\}\(x\)=R\(\\tau\_\{c,r\}\)\-R\(\\tau\_\{\\varnothing,r\}\)\.\(8\)To account for variability across rollouts, we evaluate each task–talent pair using its mean utility,u¯c​\(x\)=1n​∑r=1nuc,r​\(x\)\\overline\{u\}\_\{c\}\(x\)=\\frac\{1\}\{n\}\\sum\_\{r=1\}^\{n\}u\_\{c,r\}\(x\)\. This mean utility reflects the joint effects of talent guidance and the retrieval prior\. We select the talent with the highest mean utility,c∗=arg⁡maxc∈𝒯​u¯c​\(x\)c^\{\*\}=\\arg\\max\_\{c\\in\\mathcal\{T\}\}\\overline\{u\}\_\{c\}\(x\), and retain the task for talent\-aware data construction only ifu¯c∗​\(x\)\>0\\overline\{u\}\_\{c^\{\*\}\}\(x\)\>0\. For the selected talentc∗c^\{\*\}, we retain one rollout that completes normally, contains only valid actions, and has positive utility, preferring higher utility and fewer turns\.

Each retained rollout provides supervision for both talent routing and downstream execution\. The routing target contains a short rationale in<think\>and the identifier ofc∗c^\{\*\}in<talent\>, and the execution targets are the actions of the retained rollout\. Individual skill selection requires no separate supervision since the same talent\-conditioned retriever handles it during data collection, training, and inference\. Starting from the initial policy modelπ0\\pi\_\{0\}, we minimize the cross\-entropy loss over response tokens:

ℒsft=−∑t∈ℐasstlogπθ\(yt∣ξ,y<t\),\\mathcal\{L\}\_\{\\mathrm\{sft\}\}=\-\\sum\_\{t\\in\\mathcal\{I\}\_\{\\mathrm\{asst\}\}\}\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid\\xi,y\_\{<t\}\),\(9\)whereξ\\xidenotes the input context for the current response,yty\_\{t\}is the target token at positiontt,y<ty\_\{<t\}denotes the preceding tokens, andℐasst\\mathcal\{I\}\_\{\\mathrm\{asst\}\}is the set of token indices in the model’s response\. The resulting policy model initializes the matched\-intervention RL stage described in Section[4](https://arxiv.org/html/2610.41143#S4)\.

## 4Talent\-Aware RL

Since an episodic return reflects both talent routing and subsequent task execution, using it as a shared training signal does not distinguish their respective contributions\. To address this issue, TalentRL compares matched rollouts of the same task under different routing contexts and assigns separate credit to routing and execution tokens\.

### 4\.1Matched rollouts and execution credit

For each task, we generate an anchor rolloutτa\\tau\_\{\\mathrm\{a\}\}using the policy\-selected talentcac\_\{\\mathrm\{a\}\}, its guidancegcag\_\{c\_\{\\mathrm\{a\}\}\}, and the full skill budgetKK\. We construct three matched controls\. \(a\) Theno\-skill controlτ−s\\tau\_\{\-s\}retains the same talent and guidance but withholds all retrieved skills\. \(b\) Thereduced\-context controlτk\\tau\_\{k\}, where0<k<K0<k<K, retains the same talent, guidance, and retrieval ordering but supplies only the top\-kkskills\. \(c\) Theno\-prior controlτnp\\tau\_\{\\mathrm\{np\}\}removes both talent guidance and the talent\-specific retrieval bonus while retaining the full budget of similarity\-retrieved skills\. We collect these matched rollouts alongside ordinary, independently sampled policy rollouts for the same task\. LetRi=R⁡\(τi\)R\_\{i\}=R\(\\tau\_\{i\}\)denote the return of rolloutii\. Let𝒫\\mathcal\{P\}denote the index set of the anchor and ordinary policy rollouts for the same task, excluding the matched control rollouts\. Following GRPO\[[23](https://arxiv.org/html/2610.41143#bib.bib23)\], the execution advantage for eachi∈𝒫i\\in\\mathcal\{P\}is

A^iexec=Ri−μ𝒫σ𝒫\+ϵ\.\\hat\{A\}\_\{i\}^\{\\mathrm\{exec\}\}=\\frac\{R\_\{i\}\-\\mu\_\{\\mathcal\{P\}\}\}\{\\sigma\_\{\\mathcal\{P\}\}\+\\epsilon\}\.\(10\)Here,μ𝒫\\mu\_\{\\mathcal\{P\}\}andσ𝒫\\sigma\_\{\\mathcal\{P\}\}are the mean and standard deviation of returns in𝒫\\mathcal\{P\}, andϵ\>0\\epsilon\>0is a numerical stability constant\. Routing credit is computed by comparing the anchor return with the control returns, as described next\.

### 4\.2Matched\-intervention routing credit

LetRaR\_\{\\mathrm\{a\}\},R−sR\_\{\-s\},RkR\_\{k\}, andRnpR\_\{\\mathrm\{np\}\}denote the task returns of the anchor, no\-skill, reduced\-context, and no\-prior rollouts, respectively\. To ignore return differences with magnitude below a thresholdη≥0\\eta\\geq 0, we define a dead\-zone operator:

Dη​\(z\)=\{0,\|z\|<η,z,otherwise\.D\_\{\\eta\}\(z\)=\\begin\{cases\}0,&\|z\|<\\eta,\\\\ z,&\\text\{otherwise\}\\end\{cases\}\.\(11\)Let𝒥=\{skill,budget,talent\}\\mathcal\{J\}=\\\{\\mathrm\{skill\},\\mathrm\{budget\},\\mathrm\{talent\}\\\}index the three reward contrasts\. We compute

δℓ=Dη​\(Ra−Rℓ\),ℓ∈𝒥\.\\delta\_\{\\ell\}=D\_\{\\eta\}\\\!\\bigl\(R\_\{\\mathrm\{a\}\}\-R\_\{\\ell\}\\bigr\),\\quad\\ell\\in\\mathcal\{J\}\.\(12\)Here,RℓR\_\{\\ell\}is the return of the no\-skill, reduced\-context, or no\-prior control forℓ=skill,budget,talent\\ell=\\mathrm\{skill\},\\mathrm\{budget\},\\mathrm\{talent\}, respectively\. We combine these contrasts into a routing advantage:

A^talent=clip⁡\(α​∑ℓ∈𝒥wℓ​δℓ,−κ,κ\),\\hat\{A\}^\{\\mathrm\{talent\}\}=\\operatorname\{clip\}\\\!\\left\(\\alpha\\sum\_\{\\ell\\in\\mathcal\{J\}\}w\_\{\\ell\}\\delta\_\{\\ell\},\\,\-\\kappa,\\,\\kappa\\right\),\(13\)wherewℓ≥0w\_\{\\ell\}\\geq 0and∑ℓ∈𝒥wℓ=1\\sum\_\{\\ell\\in\\mathcal\{J\}\}w\_\{\\ell\}=1\. The weights control the relative contributions of the three contrasts, whileα\>0\\alpha\>0scales the combined signal andκ\>0\\kappa\>0sets its clipping bound\. This routing advantage provides task\-specific feedback on the utility of the talent\-conditioned context\.

Table 2:Search7 success rate \(%\) after RL\. Within each backbone, the highest and second\-highest distinct values for each metric are bold and underlined, respectively; ties receive the same formatting\. Evaluation details are provided in Appendix[A](https://arxiv.org/html/2610.41143#A1)\.
### 4\.3Token\-level policy objective

We optimize the shared policy model by assigning routing and execution advantages to their corresponding response tokens\. Letmi,t=1m\_\{i,t\}=1if tokenttin rolloutiiis generated by the policy model, either during routing or execution, andmi,t=0m\_\{i,t\}=0for input tokens, including the task prompt, talent guidance, retrieved skills, and environment observations\. This mask excludes input tokens from the policy surrogate loss;Ai,tA\_\{i,t\}determines the advantage assigned to each generated token\. Letℐiroute\\mathcal\{I\}\_\{i\}^\{\\mathrm\{route\}\}andℐiexec\\mathcal\{I\}\_\{i\}^\{\\mathrm\{exec\}\}denote the routing and execution token positions in rolloutii, respectively\. Letiai\_\{\\mathrm\{a\}\}denote the index of the anchor rollout\. We define the token\-specific advantage as

Ai,t=\{A^talent,i=ia,t∈ℐiroute,A^iexec,i∈𝒫,t∈ℐiexec,0,otherwise\.A\_\{i,t\}=\\begin\{cases\}\\hat\{A\}^\{\\mathrm\{talent\}\},&i=i\_\{\\mathrm\{a\}\},\\ t\\in\\mathcal\{I\}\_\{i\}^\{\\mathrm\{route\}\},\\\\\[3\.0pt\] \\hat\{A\}\_\{i\}^\{\\mathrm\{exec\}\},&i\\in\\mathcal\{P\},\\ t\\in\\mathcal\{I\}\_\{i\}^\{\\mathrm\{exec\}\},\\\\\[3\.0pt\] 0,&\\text\{otherwise\}\.\\end\{cases\}\(14\)Letri,t​\(θ\)r\_\{i,t\}\(\\theta\)denote the probability ratio for the sampled token at positionttin rolloutiiunder the current policy and the behavior policy that generated the rollout, conditioned on the same context\. We minimize the following KL\-regularized clipped surrogate objective:

ℒrl\(θ\)=−1N∑i,tmi,tℓi,t\(θ\)\+βDKL\(πθ∥πref\),\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{rl\}\}\(\\theta\)=\-\\frac\{1\}\{N\}\\sum\_\{i,t\}m\_\{i,t\}\\,\\ell\_\{i,t\}\(\\theta\)\+\\beta D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\mathrm\{ref\}\}\\right\),\(15\)ℓi,t​\(θ\)=min⁡\(ri,t​\(θ\)​Ai,t,clip⁡\(ri,t​\(θ\),1−ϵc,1\+ϵc\)​Ai,t\)\.\\displaystyle\\ell\_\{i,t\}\(\\theta\)=\\min\\\!\\bigl\(r\_\{i,t\}\(\\theta\)A\_\{i,t\},\\operatorname\{clip\}\\\!\\left\(r\_\{i,t\}\(\\theta\),1\-\\epsilon\_\{c\},1\+\\epsilon\_\{c\}\\right\)A\_\{i,t\}\\bigr\)\.Here,N=∑i,tmi,tN=\\sum\_\{i,t\}m\_\{i,t\}counts the response tokens,ϵc\\epsilon\_\{c\}controls probability\-ratio clipping, andβ\\betacontrols the strength of KL regularization toward the frozen reference policyπref\\pi\_\{\\mathrm\{ref\}\}, initialized from the SFT checkpoint\. The routing rationale and talent identifier receive the same routing advantage since together they form the routing action\. Algorithm[1](https://arxiv.org/html/2610.41143#alg1)summarizes the complete training procedure, from talent\-aware SFT to matched\-intervention RL\.

## 5Experiments

Table 3:ALFWorld success rate and WebShop average score/success rate \(%\)\. Baseline values are paper\-reported, not protocol\-matched reproductions\. Plain SFT and TalentRL\-SFT are Qwen2\.5\-7B\-Instruct students; their ALFWorld columns use the 140 valid\-seen games\. TalentRL is initialized from the TalentRL\-SFT checkpoints\. Best and second\-best results are bold and underlined, respectively\.### 5\.1General setup

We use Qwen2\.5\-7B\-Instruct\[[36](https://arxiv.org/html/2610.41143#bib.bib36)\], Qwen3\-8B\[[37](https://arxiv.org/html/2610.41143#bib.bib37)\], and Llama\-3\.1\-8B\-Instruct\[[8](https://arxiv.org/html/2610.41143#bib.bib8)\]as backbone models\. We evaluate on seven open\-domain and multi\-hop question\-answering benchmarks: Natural Questions\[[14](https://arxiv.org/html/2610.41143#bib.bib14)\], TriviaQA\[[13](https://arxiv.org/html/2610.41143#bib.bib13)\], PopQA\[[18](https://arxiv.org/html/2610.41143#bib.bib18)\], HotpotQA\[[40](https://arxiv.org/html/2610.41143#bib.bib40)\], 2WikiMultiHopQA\[[10](https://arxiv.org/html/2610.41143#bib.bib10)\], MuSiQue\[[29](https://arxiv.org/html/2610.41143#bib.bib29)\], and Bamboogle\[[22](https://arxiv.org/html/2610.41143#bib.bib22)\]\. We refer to this suite as Search7 and report the average success rate \(%\) across the seven benchmarks, with equal weight assigned to each benchmark\. We also evaluate on two interactive benchmarks, ALFWorld\[[26](https://arxiv.org/html/2610.41143#bib.bib26)\]for household tasks and WebShop\[[41](https://arxiv.org/html/2610.41143#bib.bib41)\]for online shopping\. We use GRPO for RL training with a batch size of 64 and 8 rollouts per task\. RL runs for 200 steps on the NQ and HotpotQA training sets for Search7 and 150 steps on ALFWorld and WebShop, with five skills retrieved by Qwen3\-Embedding\-0\.6B\[[48](https://arxiv.org/html/2610.41143#bib.bib48)\]andλ=0\.3\\lambda=0\.3\. Further experimental details are provided in Appendix[A](https://arxiv.org/html/2610.41143#A1)\.

### 5\.2Baselines

We consider the following baselines that use RL to improve the policy model with an external skill library\. \(a\)CoSkill\[[5](https://arxiv.org/html/2610.41143#bib.bib5)\]jointly trains reasoning and meta\-skill agents to evolve a hierarchical SkillBank\. \(b\)SkillRL\[[34](https://arxiv.org/html/2610.41143#bib.bib34)\]retrieves procedural skills and recursively evolves its skill library during RL\. \(c\)SAGE\[[31](https://arxiv.org/html/2610.41143#bib.bib31)\]couples RL with a self\-improving external skill library\. \(d\)Skill1\[[24](https://arxiv.org/html/2610.41143#bib.bib24)\]unifies skill selection, utilization, and distillation in one policy\. \(e\)ReSkill\[[9](https://arxiv.org/html/2610.41143#bib.bib9)\]embeds skill creation into GRPO\-based policy optimization\. On ALFWorld and WebShop, we additionally compare with the memory\-based agents MemRL\[[47](https://arxiv.org/html/2610.41143#bib.bib47)\], EvolveR\[[33](https://arxiv.org/html/2610.41143#bib.bib33)\], Mem0\[[3](https://arxiv.org/html/2610.41143#bib.bib3)\], and SimpleMem\[[16](https://arxiv.org/html/2610.41143#bib.bib16)\], using the results reported by SkillRL\[[34](https://arxiv.org/html/2610.41143#bib.bib34)\]\.

### 5\.3Main results

#### Talent\-aware SFT strengthens policy initialization\.

To examine whether talent\-aware supervision improves policy initialization, we start from the search SFT data released by SkillRL\[[34](https://arxiv.org/html/2610.41143#bib.bib34)\], label its tasks with talents following Section[3\.3](https://arxiv.org/html/2610.41143#S3.SS3), and add the resulting talent\-conditioned trajectories to the original SFT trajectories\. Starting from each zero\-shot backbone, we separately perform full\-parameter fine\-tuning on this augmented data \(TalentRL\-SFT\) and on the original trajectories alone \(SkillRL\-SFT\), and compare the two in Table[1](https://arxiv.org/html/2610.41143#S3.T1)\. TalentRL\-SFT reaches average Search7 success rates of 34\.72%, 36\.31%, and 34\.45% on Qwen2\.5\-7B, Qwen3\-8B, and Llama\-3\.1\-8B, respectively, improving over SkillRL\-SFT by \+3\.13%, \+1\.33%, and \+3\.65%\. TalentRL\-SFT improves over SkillRL\-SFT on both the single\-hop and multi\-hop averages for every backbone, by \+1\.72%, \+1\.71%, and \+2\.79% on single\-hop benchmarks and by \+4\.19%, \+1\.04%, and \+4\.30% on multi\-hop benchmarks for Qwen2\.5\-7B, Qwen3\-8B, and Llama\-3\.1\-8B, respectively\. The multi\-hop gains are driven mainly by Bamboogle on Qwen2\.5\-7B \(\+8\.80%\) and Llama\-3\.1\-8B \(\+12\.80%\) and by 2Wiki on Qwen3\-8B \(\+5\.36%\), whereas Qwen3\-8B trails slightly on MuSiQue \(−0\.91\-0\.91%\) and Bamboogle \(−0\.80\-0\.80%\), which explains its smaller multi\-hop gain\. Across all three model architectures, the combined routing and execution supervision provides a stronger policy initialization for subsequent RL\.

Figure 2:Talent selection of the final TalentRL policies\. \(a\) Search7 test sets, as % of questions per dataset\. \(b\) ALFWorld by task type over the seen \(140\) and unseen \(134\) splits\. \(c\) WebShop on 500 goals\. Percentages below 10% are omitted from the bars\.
#### TalentRL on Search7 Benchmarks\.

We then examine whether talent\-aware RL further improves the policy model\. Starting from each TalentRL\-SFT checkpoint, we train the policy model with the objective in Section[4](https://arxiv.org/html/2610.41143#S4), whereas all other baselines in Table[2](https://arxiv.org/html/2610.41143#S4.T2)start from the corresponding SkillRL\-SFT checkpoint\. We compare TalentRL with these baselines in Table[2](https://arxiv.org/html/2610.41143#S4.T2)and with the published interactive\-task results in Table[3](https://arxiv.org/html/2610.41143#S5.T3)\. As shown in Table[2](https://arxiv.org/html/2610.41143#S4.T2), TalentRL reaches average Search7 success rates of 43\.13%, 43\.89%, and 43\.77% on Qwen2\.5\-7B, Qwen3\-8B, and Llama\-3\.1\-8B, respectively, achieving the highest average among all compared methods on each backbone\. Relative to the strongest baseline on each backbone, TalentRL improves the average success rate by \+1\.23% over SkillRL on Qwen2\.5\-7B, \+1\.09% over SkillRL on Qwen3\-8B, and \+0\.67% over SAGE on Llama\-3\.1\-8B\. On multi\-hop benchmarks, TalentRL achieves the highest average on all three backbones, exceeding the strongest baseline by \+2\.37% over SAGE on Qwen2\.5\-7B, \+1\.22% over Skill1 on Qwen3\-8B, and \+0\.50% over Skill1 on Llama\-3\.1\-8B\. The advantage is most pronounced on Bamboogle, where TalentRL ranks first on all three backbones and exceeds the second\-best method by 4\.80%, 1\.60%, and 4\.80%, respectively\. On single\-hop benchmarks, it achieves the highest average on Qwen3\-8B and Llama\-3\.1\-8B and the second highest on Qwen2\.5\-7B, where it trails SkillRL by 0\.68%\.

#### Interactive\-task results\.

Table[3](https://arxiv.org/html/2610.41143#S5.T3)presents ALFWorld and WebShop results\. We use GPT\-5\.5 to generate SFT trajectories with retrieved skills\. Plain SFT fine\-tunes Qwen2\.5\-7B\-Instruct on the successful trajectories, and TalentRL\-SFT further fine\-tunes this checkpoint on talent\-labeled trajectories constructed as in Section[3\.3](https://arxiv.org/html/2610.41143#S3.SS3)\(Appendix[A](https://arxiv.org/html/2610.41143#A1)\)\. Without interactive RL, TalentRL\-SFT raises ALFWorld overall success from 90\.7% to 93\.6% \(131/140\), with the largest gains on Clean \(\+7\.4%\), Pick2 \(\+4\.2%\), and Cool \(\+4\.0%\)\. On WebShop, it raises the average score from 73\.0 to 75\.8 and the success rate from 58\.4% to 63\.0% \(315/500\)\. Interactive RL from the TalentRL\-SFT checkpoints further improves both benchmarks\. On ALFWorld, TalentRL reaches 99\.3% overall success \(139/140\), \+5\.7% over TalentRL\-SFT and the best among all methods, with 100% on every task type except Cool \(96\.0%\)\. On WebShop, TalentRL improves over TalentRL\-SFT by \+14\.9 in average score and \+16\.6% in success rate, reaching the best average score among all methods \(90\.7\) and a success rate of 79\.6% \(398/500\), second only to Skill1 \(82\.9%\)\.

#### Learned talent routing\.

Figure[2](https://arxiv.org/html/2610.41143#S5.F2)shows that the trained policies select different talents for different types of tasks, and the selected talent matches the structure of each task\. On Search7, most TriviaQA and PopQA questions ask for a single attribute of a single entity and are routed mainly toconstraint application, which narrows candidate answers by the conditions stated in the question\. NQ is also mostly single\-hop, but many of its questions ask*when*or*how many*, and 33% of them are routed totemporal and numeric reasoning, compared with at most 8% on the other benchmarks\. MuSiQue and Bamboogle require resolving an intermediate entity first and are routed mainly tobridge\-entity resolution\(89% and 61%\)\. HotpotQA and 2Wiki also contain comparison questions such as*Who was born first, A or B?*, and 24% and 46% of their questions are routed toevidence comparison\. On ALFWorld, the selected talent reflects the main bottleneck of each task type\. Pick and Pick2 tasks, which are limited by locating objects, are routed mainly tosystematic receptacle search, Clean tasks mainly toexact object selection, and Heat tasks, which require operating the microwave, selectfixture interactionmore often than any other task type \(23%\)\. On WebShop, the talents correspond to product domains, and the selected talent follows the category of the requested product\. All 322 goals routed toapparelrequest clothing\. The 31 goals routed togeneric matchingfall outside the four specific domains and are mainly electronics and food\. Appendix[B\.3](https://arxiv.org/html/2610.41143#A2.SS3)gives further details\.

## 6Conclusion

We propose TalentRL, a reinforcement learning framework in which a single policy model learns both to select a talent for each task and to execute skills under that talent\. The selected talent guides how the policy model solves the task and adds a soft prior to skill retrieval\. Talent\-aware SFT teaches the policy model this select\-then\-execute format, and RL then trains talent selection with matched\-intervention contrasts and skill execution with group\-relative advantages\. On Search7, TalentRL improves over prior skill\-based methods after both SFT and RL on all three backbones, with the largest gains on multi\-hop questions, and it remains competitive on ALFWorld and WebShop\. Future work will extend TalentRL to more domains and larger models, and explore discovering new talents directly from RL experience\.

## References

- \[1\]Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou\.Large language models as tool makers\.In*International Conference on Learning Representations*, 2024\.
- \[2\]Huancheng Chen, Xiaodi Sun, Zhaoqiong Huang, Shenyang Huang, Shreya Singhal, and Jingwen Lu\.Skillspec: Consensus\-gated agent skill evolution via representation specialization\.*arXiv preprint arXiv:2610\.00704*, 2026\.
- \[3\]Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav\.Mem0: Building production\-ready AI agents with scalable long\-term memory\.*arXiv preprint arXiv:2504\.19413*, 2025\.
- \[4\]Runnan Fang, Yuan Liang, Xiaobin Wang, Jialong Wu, Shuofei Qiao, Pengjun Xie, Fei Huang, Huajun Chen, and Ningyu Zhang\.MemP: Exploring agent procedural memory\.*arXiv preprint arXiv:2508\.06433*, 2025\.
- \[5\]Jinyuan Feng, Dongmin Li, Yiqun Chen, Yang Gao, Xing Chen, Huimu Wang, and Zhiqiang Pu\.Coskill: Joint reinforcement learning of reasoning and meta\-skill agents for hierarchical skill evolution\.*arXiv preprint arXiv:2609\.04865*, 2026\.
- \[6\]Yao Fu, Dong\-Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lee\.Autoguide: Automated generation and selection of context\-aware guidelines for large language model agents\.*arXiv preprint arXiv:2403\.08978*, 2024\.
- \[7\]Yuyao Ge, Yiwei Wang, Yuchen He, Baolong Bi, Lingrui Mei, Jiayu Yao, Lizhe Chen, and Shenghua Liu\.SkillForge: Co\-evolving skills and agents via dynamic skill lifecycles\.In*Advances in Neural Information Processing Systems*, 2026\.
- \[8\]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al\.The llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*, 2024\.
- \[9\]Zelin He, Haotian Lin, Boran Han, Wei Zhu, Haoyang Fang, Bernie Wang, Xuan Zhu, Runze Li, and Matthew Reimherr\.ReSkill: Reconciling skill creation with policy optimization in agentic RL\.*arXiv preprint arXiv:2606\.01619*, 2026\.
- \[10\]Xanh Ho, Anh\-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa\.Constructing a multi\-hop qa dataset for comprehensive evaluation of reasoning steps\.In*International Conference on Computational Linguistics*, 2020\.
- \[11\]Jinchao Hu, Meizhi Zhong, Kehai Chen, and Min Zhang\.Searchskill: Teaching llms to use search tools with evolving skill banks\.*arXiv preprint arXiv:2605\.09038*, 2026\.
- \[12\]Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan O\. Arik, Dong Wang, Hamed Zamani, and Jiawei Han\.Search\-r1: Training llms to reason and leverage search engines with reinforcement learning\.*arXiv preprint arXiv:2503\.09516*, 2025\.
- \[13\]Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer\.Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension\.In*Annual Meeting of the Association for Computational Linguistics*, 2017\.
- \[14\]Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al\.Natural questions: A benchmark for question answering research\.*Transactions of the Association for Computational Linguistics*, 7:453–466, 2019\.
- \[15\]Yu Li, Rui Miao, Zhengling Qi, and Tian Lan\.ARISE: Agent reasoning with intrinsic skill evolution in hierarchical reinforcement learning\.*arXiv preprint arXiv:2603\.16060*, 2026\.
- \[16\]Jiaqi Liu, Yaofeng Su, Peng Xia, Siwei Han, Zeyu Zheng, Cihang Xie, Mingyu Ding, and Huaxiu Yao\.SimpleMem: Efficient lifelong memory for LLM agents\.*arXiv preprint arXiv:2601\.02553*, 2026\.
- \[17\]Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Chengcheng Han, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen\.SKILL0: In\-context agentic reinforcement learning for skill internalization\.*arXiv preprint arXiv:2604\.02268*, 2026\.
- \[18\]Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi\.When not to trust language models: Investigating effectiveness of parametric and non\-parametric memories\.In*Annual Meeting of the Association for Computational Linguistics*, 2023\.
- \[19\]Qirui Mi, Zhijian Ma, Mengyue Yang, Haoxuan Li, Yisen Wang, Haifeng Zhang, and Jun Wang\.Skill\-pro: Learning reusable skills from experience via non\-parametric ppo for llm agents\.*arXiv preprint arXiv:2602\.01869*, 2026\.
- \[20\]Yongliang Miao, Ziyang Yu, Liang Zhao, Bowen Zhu, and Hasibul Haque\.Skilllens: Adaptive multi\-granularity skill reuse for cost\-efficient llm agents\.*arXiv preprint arXiv:2605\.08386*, 2026\.
- \[21\]Jingwei Ni, Yihao Liu, Xinpeng Liu, Yutao Sun, Mengyu Zhou, Pengyu Cheng, Dexin Wang, Erchao Zhao, Xiaoxi Jiang, and Guanjun Jiang\.Trace2skill: Distill trajectory\-local lessons into transferable agent skills\.*arXiv preprint arXiv:2603\.25158*, 2026\.
- \[22\]Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A\. Smith, and Mike Lewis\.Measuring and narrowing the compositionality gap in language models\.In*Findings of the Association for Computational Linguistics: EMNLP*, 2023\.
- \[23\]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y\. K\. Li, Y\. Wu, and Daya Guo\.Deepseekmath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.
- \[24\]Yaorui Shi, Yuxin Chen, Zhengxi Lu, Yuchun Miao, Shugui Liu, Qi Gu, Xunliang Cai, Xiang Wang, and An Zhang\.Skill1: Unified evolution of skill\-augmented agents via reinforcement learning\.*arXiv preprint arXiv:2605\.06130*, 2026\.
- \[25\]Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao\.Reflexion: Language agents with verbal reinforcement learning\.*Advances in Neural Information Processing Systems*, 36, 2023\.
- \[26\]Mohit Shridhar, Xingdi Yuan, Marc\-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht\.ALFWorld: Aligning text and embodied environments for interactive learning\.In*International Conference on Learning Representations*, 2021\.
- \[27\]Theodore R\. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L\. Griffiths\.Cognitive architectures for language agents\.*Transactions on Machine Learning Research*, 2024\.
- \[28\]Xijia Tao, Yihua Teng, Xinyu Fu, Ziru Liu, Kecheng Chen, Yuzhi Zhao, Suiyun Zhang, Rui Liu, and Lingpeng Kong\.Softskill: Behavioral compression for contextual adaptation\.*arXiv preprint arXiv:2606\.20333*, 2026\.
- \[29\]Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal\.Musique: Multihop questions via single\-hop question composition\.*Transactions of the Association for Computational Linguistics*, 2022\.
- \[30\]Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar\.Voyager: An open\-ended embodied agent with large language models\.*arXiv preprint arXiv:2305\.16291*, 2023\.
- \[31\]Jiongxiao Wang, Qiaojing Yan, Yawei Wang, Yijun Tian, Soumya Smruti Mishra, Zhichao Xu, Megha Gandhi, Panpan Xu, and Lin Lee Cheong\.Reinforcement learning for self\-improving agent with skill library\.*arXiv preprint arXiv:2512\.17102*, 2025a\.
- \[32\]Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig\.Agent workflow memory\.In*International Conference on Machine Learning*, 2025b\.
- \[33\]Rong Wu, Xiaoman Wang, Jianbiao Mei, Pinlong Cai, Daocheng Fu, Cheng Yang, Licheng Wen, Xuemeng Yang, Yufan Shen, Yuxin Wang, and Botian Shi\.EvolveR: Self\-evolving LLM agents through an experience\-driven lifecycle\.*arXiv preprint arXiv:2510\.16079*, 2025\.
- \[34\]Peng Xia, Jianwen Chen, Hanyang Wang, Jiaqi Liu, Kaide Zeng, Yu Wang, Siwei Han, Yiyang Zhou, Xujiang Zhao, Haifeng Chen, Zeyu Zheng, Cihang Xie, and Huaxiu Yao\.Skillrl: Evolving agents via recursive skill\-augmented reinforcement learning\.*arXiv preprint arXiv:2602\.08234*, 2026\.
- \[35\]Ruiyao Xu, Tiankai Yang, and Wei\-Chieh Huang\.Hyperskill: Self\-evolving llm agents via hypergraph\-structured skill memory\.*arXiv preprint arXiv:2608\.16114*, 2026\.
- \[36\]An Yang, Baosong Yang, Beichen Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al\.Qwen2\.5 technical report\.*arXiv preprint arXiv:2412\.15115*, 2024a\.
- \[37\]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.
- \[38\]Ling Yang, Zhaochen Yu, Tianjun Zhang, Shiyi Cao, Minkai Xu, Wentao Zhang, Joseph E\. Gonzalez, and Bin Cui\.Buffer of thoughts: Thought\-augmented reasoning with large language models\.In*Advances in Neural Information Processing Systems*, 2024b\.
- \[39\]Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang, Ziwei Zhou, Zisu Huang, Yan Li, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, Yuqing Yang, Dongdong Chen, Xue Yang, and Chong Luo\.Skillopt: Executive strategy for self\-evolving agent skills\.*arXiv preprint arXiv:2605\.23904*, 2026\.
- \[40\]Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W\. Cohen, Ruslan Salakhutdinov, and Christopher D\. Manning\.Hotpotqa: A dataset for diverse, explainable multi\-hop question answering\.In*Conference on Empirical Methods in Natural Language Processing*, 2018\.
- \[41\]Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan\.WebShop: Towards scalable real\-world web interaction with grounded language agents\.*arXiv preprint arXiv:2207\.01206*, 2022\.
- \[42\]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao\.React: Synergizing reasoning and acting in language models\.In*International Conference on Learning Representations*, 2023\.
- \[43\]Zhiyuan Yao, Yuxin Chen, Zhengxi Lu, Zishan Xu, Yueqing Sun, Yifu Guo, Yuquan Lu, Zhengzhou Cai, Kangning Zhang, Zhuowen Han, Zi\-Han Wang, Ziang Ye, Qi Gu, Xunliang Cai, Weiwen Liu, and Yongliang Shen\.Skillrise: Agentic reinforcement learning for cross\-task skill evolution\.*arXiv preprint arXiv:2607\.26784*, 2026\.
- \[44\]Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi R\. Fung, Hao Peng, and Heng Ji\.CRAFT: Customizing LLMs by creating and retrieving from specialized toolsets\.In*International Conference on Learning Representations*, 2024\.
- \[45\]Fan Zhang, Hao Wu, Jinyang Wu, Hongyu Qu, Zheng Lian, Xiao Luo, Yefeng Zheng, Bo Zhang, Haoxuan Li, and Pheng\-Ann Heng\.RSIPO: Recursive self\-improving policy optimization for agentic reinforcement learning via skill\-policy co\-evolution\.*arXiv preprint arXiv:2610\.15772*, 2026a\.
- \[46\]Haozhen Zhang, Quanyu Long, Jianzhu Bao, Tao Feng, Weizhi Zhang, Haodong Yue, and Wenya Wang\.Memskill: Learning and evolving memory skills for self\-evolving agents\.*arXiv preprint arXiv:2602\.02474*, 2026b\.
- \[47\]Shengtao Zhang, Jiaqian Wang, Ruiwen Zhou, Junwei Liao, Yuchen Feng, Zhuo Li, Yujie Zheng, Weinan Zhang, Ying Wen, Zhiyu Li, Feiyu Xiong, Yutao Qi, Bo Tang, and Muning Wen\.MemRL: Self\-evolving agents via runtime reinforcement learning on episodic memory\.*arXiv preprint arXiv:2601\.03192*, 2026c\.
- \[48\]Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou\.Qwen3 embedding: Advancing text embedding and reranking through foundation models\.*arXiv preprint arXiv:2506\.05176*, 2025\.
- \[49\]Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong\-Jin Liu, and Gao Huang\.Expel: Llm agents are experiential learners\.In*AAAI Conference on Artificial Intelligence*, 2024\.
- \[50\]Xinyu Zhao, Zhen Tan, Vaishnav Tadiparthi, Nakul Agarwal, Kwonjoon Lee, Ehsan Moradi Pari, Hossein Nourkhiz Mahjoub, and Tianlong Chen\.Generative skill composition for llm agents\.*arXiv preprint arXiv:2606\.32025*, 2026\.
- \[51\]Yanzhao Zheng, Zhentao Zhang, Chao Ma, Yuanqiang Yu, Jihuai Zhu, Yong Wu, Tianze Xu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, and Gang Yu\.Skillrouter: Skill routing for llm agents at scale\.*arXiv preprint arXiv:2603\.22455*, 2026\.
- \[52\]Jiayin Zhu, Kelong Mao, Yudong Guo, Dengbo He, Sulong Xu, Simiu Gu, and Yutao Yue\.Skillcoach: Self\-evolving rubrics for evaluating and enhancing agentic skill\-use\.*arXiv preprint arXiv:2607\.01874*, 2026\.

## Appendix AExperimental details

Table 4:Primary Qwen2\.5\-7B Search7 training, routing, and retrieval configuration\. Interactive\-task protocols are described separately below\.StageHyperparameterValueSFTEpochs2SFTLearning rate1×10−51\\times 10^\{\-5\}SFTEffective batch size16SFTMaximum sequence length8,192RLPolicy updates200RLLearning rate1×10−61\\times 10^\{\-6\}RLTask batch size64RLRollouts per task8RLKL coefficient0\.01RLPolicy\-ratio clipping radius0\.2RLMaximum interaction steps4RoutingReduced skill budgetkk2RoutingDead\-zone thresholdη\\eta0\.1RoutingAdvantage scaleα\\alpha2\.0RoutingAdvantage clipping boundκ\\kappa2\.0RetrievalEmbedding modelQwen3\-Embedding\-0\.6BRetrievalSimilarity recall size20RetrievalExecution skill budget5RetrievalSimilarity threshold0\.2RetrievalTalent\-prior weightλ\\lambda0\.3Training rollouts are sampled to produce within\-task diversity for group\-relative optimization; groups in which all rollouts receive the same return receive zero execution advantage\. Final Search7 evaluation uses deterministic decoding\. Skill and query embeddings are L2\-normalized, soq⁡\(x,s\)q\(x,s\)is cosine similarity\. The three routing contrasts are equally weighted\. The full generated talent response receives the routing advantage\.

#### Search7 baseline configurations\.

Table[2](https://arxiv.org/html/2610.41143#S4.T2)uses our Search7 adaptations of the baselines, aligned with the official implementations\. As described in Section[5\.3](https://arxiv.org/html/2610.41143#S5.SS3), TalentRL starts from TalentRL\-SFT, whereas the baselines start from the corresponding SkillRL\-SFT checkpoints\. TalentRL is evaluated after 200 RL steps\.

#### Interactive\-task baseline results\.

Baseline values in Table[3](https://arxiv.org/html/2610.41143#S5.T3)are reported by SkillRL\[[34](https://arxiv.org/html/2610.41143#bib.bib34)\], except the Skill1 row, which comes from[Shi et al\. \[24\]](https://arxiv.org/html/2610.41143#bib.bib24)\. Skill1 uses Qwen2\.5\-7B\-Instruct and averages results over three runs\. These published results are not protocol\-matched reproductions\. Highlighting includes all rows: the highest and second\-highest distinct values for each metric are bold and underlined, respectively, with ties receiving the same formatting\.

#### Interactive\-task evaluation protocols\.

Plain SFT and TalentRL\-SFT use Qwen2\.5\-7B\-Instruct without interactive RL\. TalentRL continues each TalentRL\-SFT checkpoint with interactive GRPO and the evolving SkillBank for 150 steps on each benchmark; we evaluate the step\-150 checkpoints with the SkillBank state saved at that step and the same evaluation protocol as TalentRL\-SFT\. Plain SFT fine\-tunes all parameters on successful GPT\-5\.5 trajectories with embedding\-retrieved skills \(batch 16, two epochs, learning rate10−510^\{\-5\}\)\. TalentRL\-SFT continues this checkpoint with the same recipe on talent\-labeled rows only: 4,961 ALFWorld and 1,212 WebShop rows, each task contributing one routing example and its talent\-conditioned actions\. The supervision comes from matched rollouts of the Plain SFT student under no talent and under each of the five ALFWorld or five WebShop talents, with three repeats per condition; each task keeps the talent with the highest positive mean utility, breaking ties by score and then by fewer turns\. For ALFWorld, the training tasks are resampled so that each of the five talents labels 85–86 of the 429 training tasks\. During TalentRL\-SFT and TalentRL evaluation, the model first selects a talent, then acts with its guidance and five skills reranked from the 20 most task\-similar skills with a bonusλ=0\.3\\lambda=0\.3for the talent’s skills; an invalid route falls back to no talent\. Talent selection uses greedy decoding; execution responses use temperature 0\.4 and up to 512 tokens per response\. Plain SFT uses embedding\-retrieved skills without talent routing or guidance\. On ALFWorld, the table reports 140 held\-outvalid\_seentasks with state\-feedback prompts and a 50\-step limit; on the 134valid\_unseentasks, Plain SFT succeeds on 131 and TalentRL\-SFT on 133\. On WebShop, evaluation covers the first 500 goals in the canonical seed\-42 ordering within the 1,000\-product environment with a 15\-step limit\. All tasks remain in the denominator\. TalentRL\-SFT succeeds on 131/140 ALFWorld and 315/500 WebShop tasks, with mean WebShop reward 0\.7582 \(75\.8 on the table’s 0–100 scale\); all TalentRL\-SFT routes are valid\. After interactive RL, TalentRL succeeds on 139/140 ALFWorld games \(99\.3%\) and 398/500 WebShop goals \(79\.6%\), with a WebShop average score of 90\.7, as reported in Table[3](https://arxiv.org/html/2610.41143#S5.T3)\. These are single\-checkpoint native\-protocol evaluations, not protocol\-matched reproductions of the published baselines\.

## Appendix BTalent taxonomy

The initial Search7 catalog uses eight talents:task decomposition,bridge\-entity resolution,entity disambiguation,constraint application,evidence comparison,long\-tail query expansion,temporal and numeric reasoning, andanswer verification\. Each talent contains a natural\-language description, generation guidance, and references to a subset of reusable skills\.

### B\.1Preset talent\-to\-skill assignments

Table[5](https://arxiv.org/html/2610.41143#A2.T5)reports the talent\-specific generation guidance and initial talent–skill associations\. The guidance conditions execution, while the associations supply a soft prior rather than a hard retrieval boundary\. The retriever recalls the 20 most similar skills above a similarity threshold of 0\.2, adds the bonusλ=0\.3\\lambda=0\.3to skills associated with the policy\-selected talent, and supplies the top five reranked skills during execution\. Skills, talents, and associations are updated during the reported RL runs, starting from these preset assignments\.

Table 5:Talent\-specific generation guidance and preset talent–skill associations for Search7\. The selected talent provides natural\-language guidance for execution and a soft prior for skill retrieval\. Associated skills are not necessarily all included in the execution context\. Table[6](https://arxiv.org/html/2610.41143#A2.T6)provides the referenced skill details\.
### B\.2Skill details

Table[6](https://arxiv.org/html/2610.41143#A2.T6)lists the principles and applicability conditions of the 26 distinct skills referenced in Table[5](https://arxiv.org/html/2610.41143#A2.T5)\. Each skill is listed once, even when it is associated with multiple talents\.

Table 6:Details of the preset SkillBank skills referenced by the Search7 talent catalog\. Principles and applicability conditions reproduce the corresponding skill fields; only typography is normalized\.Skill IDPrincipleWhen to applygen\_001Break the question into minimal sub\-questions \(entities, attributes, relations\) and handle each with its own query before synthesizing the final answer\.Any complex or multi\-part question, especially multi\-hop or comparative tasks\.gen\_003If top results don’t yield direct evidence, rephrase by adding qualifiers \(dates, roles, context\) or alternate names instead of repeating the same query\.After scanning first result set and finding no definitive evidence\.gen\_004Only state an answer once the supporting evidence \(reliable source snippet\) has been located and read; refrain from guessing or summarizing from memory\.Before committing to a final answer, especially for contentious or less\-known facts\.gen\_005Validate critical facts \(dates, numbers, names\) against at least two independent sources to avoid outdated or incorrect information\.When the fact may have changed over time or when initial sources disagree\.gen\_006Detect ambiguous entity mentions and disambiguate by adding clarifiers \(e\.g\., occupation, year, nationality\) in the query or by exploring disambiguation pages\.When an entity name maps to multiple possible referents or results appear mixed\.gen\_007For multi\-hop tasks, first retrieve the intermediate entity \(e\.g\., a film’s director\) then run a second targeted query for that entity’s attribute \(e\.g\., birthplace\) before synthesizing\.Whenever the answer depends on an attribute of a related entity rather than the entity in the question\.gen\_008When comparing entities \(earlier/later, larger/smaller\), collect each relevant value separately, list them side by side, then decide using explicit comparison logic\.Any comparative question involving dates, counts, or quantitative attributes\.gen\_009For time\-sensitive queries \(e\.g\., upcoming releases, current versions\), include temporal cues like “latest”, year, or “release date 2023” and prioritize most recent authoritative sources\.Questions asking for “current”, “next”, or “latest” information\.gen\_010Stop issuing further queries once you have clear, corroborated evidence; conversely, avoid premature termination if no source explicitly supports an answer\.After each read step—decide to answer only if confidence is justified; otherwise refine search\.dir\_002If first search yields weak/no hits, instantly reformulate using synonyms, alternate names, dates, or quoted phrases instead of repeating the same query\.After an initial search returns no clear answer or only tangential results\.dir\_003For song titles, quotes, episode names, etc\., wrap the unique phrase in quotation marks to pull exact\-match sources\.When the query contains distinctive phrases, lyrics, book/film titles, or direct quotations\.dir\_004Include recency cues \(e\.g\., “current”, year\) in the search and verify publication date to avoid outdated or speculative info\.For questions about ‘current’, ‘latest’, or future events/releases\.dir\_005Only state an answer that is explicitly supported by the retrieved text; if unclear, continue searching rather than guess or hallucinate\.Before finalizing any factoid answer\.mul\_001Split the main query into explicit sub\-questions to identify each entity and the specific attribute or relationship needed before searching\.Any multi\-hop question that links two or more entities/facts \(e\.g\., director’s birthplace, album vs film comparisons\)\.mul\_002Issue separate, focused searches for each sub\-question or entity instead of one broad query to gather precise intermediate facts\.After decomposition, when distinct pieces of information must be collected individually\.mul\_003Retrieve concrete values for all items involved \(dates, places, relations\) before performing any comparison or conclusion to avoid premature or unsupported answers\.Comparative tasks \(earlier/later, older/younger, bigger/smaller, etc\.\)\.mul\_004Add clarifying descriptors \(profession, year, medium, location\) to queries to distinguish between entities with identical or similar names and prevent mixing facts\.Whenever an entity name is generic or shared by multiple subjects \(e\.g\., common personal names, titles appearing in different media\)\.mul\_005If initial search fails or returns ambiguous results, promptly rephrase using synonyms, alternate titles, or additional context instead of repeating the same query\.After a search yields no relevant results or conflicting information\.ent\_002If the name is common or ambiguous, append clarifiers \(birth year, nationality, known work\) to the query or scan result snippets for matching context before selecting a source\.When multiple people/characters share the same name or early results mix different entities\.ent\_003Confirm the attribute in at least two independent, reputable sources \(e\.g\., Wikipedia infobox \+ biography site\) to avoid hallucinations\.After the first plausible answer appears or when the attribute seems uncommon or uncertain\.ent\_004If initial search returns irrelevant hits, adjust the query instead of repeating it—use alternative spellings, middle initials, or related works/titles\.When consecutive top results don’t mention the sought attribute for the intended entity\.com\_001First split the question into \(a\) each entity and \(b\) the single attribute to be compared; this ensures every subsequent search is targeted and comparable\.At the moment you read any comparison\-type question\.com\_002Independently retrieve the identical attribute for each entity—run separate, attribute\-focused searches \(e\.g\., “\[Entity\] headquarters”, “\[Entity\] release date”\) and store the raw values\.Immediately after identifying entities and the comparison attribute\.com\_003Convert retrieved values to a common, directly comparable form \(e\.g\., standardize dates, compute ages, map demonyms to countries\) before judging equality or ordering\.After gathering each entity’s attribute but before drawing any conclusion\.com\_004If a search yields no clear or multiple results, refine the query with context \(profession, year, language\) or acknowledge data insufficiency—never infer without evidence\.Whenever an attribute lookup returns ambiguous, missing, or conflicting information\.com\_005State each normalized value and perform the logical comparison \(equal, earlier, larger, etc\.\) explicitly to derive the answer, ensuring traceability and avoiding hallucination\.As the final synthesis step before outputting the comparison result\.
### B\.3Talent selection distribution

Figure[2](https://arxiv.org/html/2610.41143#S5.F2)reports the talents selected at evaluation by the Qwen3\-8B TalentRL policy on Search7 and by the final TalentRL policies on ALFWorld and WebShop\. All selections use greedy decoding\. The Search7 distribution is discussed in Section[5\.3](https://arxiv.org/html/2610.41143#S5.SS3); this subsection analyzes the interactive tasks\.

#### ALFWorld\.

Panel \(b\) groups ALFWorld tasks by type over the seen \(140\) and unseen \(134\) games\. The selected talent follows the step most likely to fail in each task type\. Pick and Pick2 tasks only require finding objects and placing them, so their main difficulty is searching drawers, cabinets, and countertops; they usesystematic receptacle searchin 83% and 100% of games, respectively, and Pick2 additionally requires locating a second instance of the same object type\. Clean tasks are routed mainly toexact object selection\(71%\)\. Cleaning itself takes a single action at the sink basin, and many Clean targets are kitchen utensils with similar names, such as knife and butter knife or spoon and ladle, so selecting the correct object matters more than the cleaning step\. Heat tasks require operating the microwave, the most involved appliance interaction among the six task types, and selectfixture interactionmore often than any other task type \(23%\)\. Cool tasks use the fridge, which needs simpler interaction, and are routed mainly tosystematic receptacle search\(61%\)\. Look tasks require finding an object and then turning on a desk lamp, and are split betweensystematic receptacle search\(45%\) andexact object selection\(48%\)\.Goal decompositionanddestination deliveryare not selected: ALFWorld tasks follow short fixed templates that need no further decomposition, and the final placement step rarely fails\. Because TalentRL already succeeds on 139 of the 140 seen games, these distributions show the learned routing preference rather than evidence that another talent would fail on the same task\.

#### WebShop\.

WebShop talents correspond to product domains, so the selection follows the category of the requested product, and the distribution mainly reflects the composition of the 500 test goals\. All 322 goals routed toapparelrequest clothing items, such as shirts, hoodies, dresses, and activewear\.Home decorreceives furniture, window coverings, and bedding \(76 goals\), andfootwearreceives shoes, boots, and sandals \(65 goals\)\. The 31 goals routed togeneric matchingfall outside the four specific domains: about 17 request electronics, such as cables, phone accessories, and storage devices, about 9 request food, and the rest are mostly beauty products\. Thepersonal caretalent is defined narrowly around personal\-care items and travel containers, and is selected for only 6 skincare, hair\-removal, and hair\-growth goals; similar beauty goals are sometimes routed togeneric matchinginstead\. Success rates range from 61\.5% for footwear to 83\.9% for apparel\. Footwear goals are the hardest because they require selecting exact size and color variants, and a single wrong option prevents a full match\.

## Appendix CSearch7 SFT data construction

This section details the Search7 construction in Section[3\.3](https://arxiv.org/html/2610.41143#S3.SS3); the interactive\-task SFT stages are described in Appendix[A](https://arxiv.org/html/2610.41143#A1)\. We start from the SkillRL Search SFT split, group its step\-level rows by question, and reconstruct task trajectories\. For each taskxx, we collectnnrollouts for every forced talentccandnnreference rollouts with similarity\-retrieved skills but no talent guidance or retrieval bonus\. Following Section[3\.3](https://arxiv.org/html/2610.41143#S3.SS3), we computeuc,r​\(x\)=R⁡\(τc,r\)−R⁡\(τ∅,r\)u\_\{c,r\}\(x\)=R\(\\tau\_\{c,r\}\)\-R\(\\tau\_\{\\varnothing,r\}\)and average over repeats\. We select the talent with the largest mean utility and retain the task only if this utility is positive, breaking talent ties by mean conditioned score, positive\-outcome rate, fewer tool calls, and a deterministic task–talent hash\. We then choose one normally completed, valid, positive\-utility trajectory for that talent, prioritizing higher utility and fewer tool calls\. Normal completion and action validity are eligibility requirements, not tie\-breaking preferences\.

Each accepted paired trajectory yields a talent\-selection row and one ReAct row for every search or answer action\. The routing target contains a task\-grounded rationale in<think\>and the selected talent identifier in<talent\>\. Skills are selected by the same talent\-conditioned similarity retriever used at inference and therefore require no policy target\. The ReAct targets preserve the accepted talent\-conditioned rollout rather than the original SkillRL actions\. Tasks without a positive talent do not produce routing\-aware rows\. Original SkillRL rows for the source tasks are retained as replay, and the two sources are concatenated without enforcing a fixed ratio\.

Training and validation are separated by a deterministic task hash, ensuring that no task appears in both partitions\. The final Search7 evaluation sets are not used to construct routing targets\. We initialize from the base instruction model and run two epochs of full\-parameter, response\-only SFT with a low learning rate\. Prompt tokens and injected skill context are excluded from the loss\. Replay rows retain the original search protocol without adding a talent\-selection target\.

## Appendix DMatched\-intervention RL details

The RL source row contains the task prompt and benchmark metadata\. For each task, the system creates an anchor rollout, matched no\-skill, reduced\-context, and no\-prior controls, and ordinary policy rollouts\. Retrieved observations, talent guidance, and injected skills are environment context and are masked out of the policy loss\.

### D\.1Anchor and control rollouts

The anchor uses the policy\-selected talent, its guidance, and five talent\-conditioned skills\. The no\-skill control retains the talent and guidance but withholds skills\. The reduced\-context control \(k=2k=2\) retains the same talent, guidance, and retrieval ordering while supplying only the top two skills\. The no\-prior control removes talent guidance and the retrieval bonus, supplying five skills retrieved by similarity alone\. The matched controls are retained for routing\-credit estimation, but all their response tokens have zero surrogate advantage, as specified by Equation[14](https://arxiv.org/html/2610.41143#S4.E14)\. Only the anchor’s routing response, including both its rationale and talent identifier, receives the routing advantage; ordinary rollouts’ routing tokens have zero surrogate advantage\. Execution tokens in the anchor and ordinary rollouts receive the group\-relative advantage computed from those rollouts only, excluding the matched controls\.

### D\.2Reward and credit details

The return is the task score alone, with no additional reward shaping\. Talent routing credit averages three signed, dead\-zoned return differences, namely the anchor return minus the no\-skill, reduced\-context, and no\-prior control returns\. Following Equation[13](https://arxiv.org/html/2610.41143#S4.E13), we multiply this average byα=2\.0\\alpha=2\.0, then clip it to\[−κ,κ\]\[\-\\kappa,\\kappa\]withκ=2\.0\\kappa=2\.0before applying it to the anchor routing response\. We useη=0\.1\\eta=0\.1and equal contrast weights\. The shared policy is optimized with the masked surrogate and KL regularization in Equation[15](https://arxiv.org/html/2610.41143#S4.E15); the reference policy is initialized from the SFT checkpoint and frozen throughout RL\. The intervention contrasts are auxiliary routing\-utility signals rather than unbiased estimators of the original episodic\-return advantage\.