Write, Execute, Refine: From Skill Followers to Skill Optimizers via Reinforcement Learning from Execution Feedback

# From Skill Followers to Skill Optimizersvia Reinforcement Learning from Execution Feedback
Source: [https://arxiv.org/html/2608.17587](https://arxiv.org/html/2608.17587)
## Write, Execute, Refine: From Skill Followers to Skill Optimizers via Reinforcement Learning from Execution Feedback

Kang PengZhiwei Zhang11footnotemark:1Affiliation:The Chinese University of Hong KongAffiliation:MoE Key Laboratory of High Confidence Software TechnologiesYichen ZhangAffiliation:Harbin Institute of Technology, Harbin, ChinaAffiliation:Huawei Technologies Co\., Ltd\.\[1mm\]26s165138@stu\.hit\.edu\.cn, zhangzhiwei1019@link\.cuhk\.edu\.hkZezhong WangAffiliation:The Chinese University of Hong KongYiming DuAffiliation:The Chinese University of Hong KongGeng TuAffiliation:\[2mm\] Harbin Institute of Technology, Shenzhen, ChinaBaojun WangBin LiangAffiliation:The Chinese University of Hong KongAffiliation:MoE Key Laboratory of High Confidence Software TechnologiesRuifeng XuThanks:Corresponding author\.Affiliation:\[2mm\] Harbin Institute of Technology, Shenzhen, ChinaKam\-Fai WongAffiliation:The Chinese University of Hong KongAffiliation:MoE Key Laboratory of High Confidence Software Technologies

###### Abstract

Expert\-written natural language skills can improve tool\-using agents, yet agent\-authored skills perform 8–11 points worse than using no skill\. This gap suggests that following procedural guidance and improving it from execution evidence are distinct capabilities\. Inference\-time loops can repair skills but do not improve the model that writes the next one\. We study how to organize execution experience from intermediate skills into training states for an optimizer\. We introduceWER\(Write,Execute, andRefine\), a multi\-phase framework that trains a Skill Optimizer outside a frozen executor\. The optimizer proposes skills, a frozen agent executes each repeatedly, and a programmatic verifier scores the outcomes\. The scores provide relative credit and select mixed\-outcome records\. Matched successful and failed trajectories from these records form the next phase’s refinement states, so the optimizer learns from the consequences of its earlier outputs\. On BFCL v4 multi\-turn andτ2\\tau^\{2\}\-bench, WER improves average Pass@1 over the no\-skill baseline by 7\.80 and 3\.85 points, respectively\. Under an identical refinement workflow, it outperforms the same backbone without optimizer training by 9\.35 and 10\.29 points\. The trained 4B optimizer reaches 76\.63% on BFCL v4, outperforming all evaluated off\-the\-shelf general\-purpose models used as skill optimizers on average\. Our code is available at[https://github\.com/littlepkk/WER4skill\-optimizer\-training](https://github.com/littlepkk/WER4skill-optimizer-training)\.

## 1Introduction

Reliable tool use requires procedural knowledge: which tool to invoke, how to validate its arguments, and when to retry[28](https://arxiv.org/html/2608.17587#bib.bib1);[14](https://arxiv.org/html/2608.17587#bib.bib2);[13](https://arxiv.org/html/2608.17587#bib.bib3)\. A common approach represents this knowledge as*skills*, compact natural\-language instructions added to an agent’s context at inference time[20](https://arxiv.org/html/2608.17587#bib.bib6);[5](https://arxiv.org/html/2608.17587#bib.bib20)\. On SkillsBench[7](https://arxiv.org/html/2608.17587#bib.bib11), expert\-curated skills raise the average pass rate from 33\.9% to 50\.5%, yet agent\-authored skills fall 8–11 points below using no skill at all\. This reveals a capability gap: agents can benefit substantially from procedural guidance without being able to write it reliably\.

![Refer to caption](https://arxiv.org/html/2608.17587v1/teaser_fig_v2.png)Figure 1:The Skill Optimizer remains outside the environment and observes execution\.It writes a skill for a*frozen*agent and revises the skill from the resulting trajectory\.A common response to this gap is an inference\-time loop in which an LLM drafts a skill, observes its execution, and revises it[18](https://arxiv.org/html/2608.17587#bib.bib4);[8](https://arxiv.org/html/2608.17587#bib.bib26);[1](https://arxiv.org/html/2608.17587#bib.bib9);[27](https://arxiv.org/html/2608.17587#bib.bib10);[9](https://arxiv.org/html/2608.17587#bib.bib19)\. More broadly, reflecting on trajectories and execution outcomes, then distilling the lessons into reusable experience, has become a common workflow recipe for improving agents\. Yet this recipe presupposes a capability that language models may not reliably possess: converting behavioral evidence into the procedural instruction needed to prevent the observed failure\. Although such loops can improve the current artifact, they typically leave the skill writer unchanged: each new task again relies on the base model to diagnose an execution log and translate the failure into a general procedural correction, and standard pretraining or instruction tuning does not directly optimize this execution\-grounded reflection capability\. Plausible but incorrect corrections can be costly: injecting only 10% plausible but wrong experience reducesτ2\\tau^\{2\}\-bench Pass@1 from 82\.5 to 77\.2, while self\-verification recovers almost none of the loss \(83\.3→\\rightarrow83\.2\)[31](https://arxiv.org/html/2608.17587#bib.bib12)\. Repairing one skill at inference time therefore does not by itself teach the skill writer to repair the next\.

Recent reinforcement\-learning methods have established that skill improvement can itself be learned\. SkillMaster[26](https://arxiv.org/html/2608.17587#bib.bib18)learns post\-episode skill mutations through counterfactual probe evaluation, SkillOS[12](https://arxiv.org/html/2608.17587#bib.bib7)trains skill\-repository management according to the utility of its updates on subsequent tasks, and Skill\-R1[19](https://arxiv.org/html/2608.17587#bib.bib14)optimizes recurrent skill revisions with intra\- and inter\-generation advantages\. Together, these methods move beyond prompting a fixed skill writer and instead learn a refinement policy\. Within this emerging direction, we ask a complementary training question: how should execution experience from intermediate skills be structured into refinement states for training a skill optimizer? To study this question, we construct a multi\-phase training framework for iterative skill refinement, jointly designing its rollout mechanism, credit assignment, and cross\-phase experience construction\. Verified outcomes are used both to compare candidate revisions and to select and reorganize diagnostically useful successful and failed trajectories into the next phase’s refinement states\. By reusing this experience across phases, the optimizer learns to correct specific deficiencies exposed by its own previous outputs\.

We instantiate this framework asWER\(Write,Execute, andRefine\)\. A Skill Optimizerπθ\\pi\_\{\\theta\}remains outside the sandbox \(Figure[1](https://arxiv.org/html/2608.17587#S1.F1)\) and proposes multiple revisions of a skill\. A frozen skill\-conditioned agent executes each candidate repeatedly, and a deterministic verifier supplies group\-relative rewards\. Candidates with mixed outcomes are especially informative: with the task, skill, and executor held fixed, their successful and failed trajectories provide a controlled local comparison between different behavioral branches\. WER reassembles this paired evidence with the corresponding skill as the next phase’s refinement state\. WER thus trains the optimizer on the execution consequences of its own previous outputs without modifying the downstream agent\.

Our main contributions are:

- •We identify refinement\-state construction as a central training problem in iterative skill optimization and formulate the Skill Optimizer as a dedicated execution\-conditioned policy over skill documents\.
- •We introduce phase\-wise self\-bootstrapping that couples candidate\-level relative optimization with cross\-phase construction of diagnostic experience from matched successful and failed executions\.
- •Under an identical refinement workflow, WER improves average Pass@1 over the same backbone without optimizer training by 9\.35 points on BFCL v4 multi\-turn[13](https://arxiv.org/html/2608.17587#bib.bib3)and 10\.29 points onτ2\\tau^\{2\}\-bench[2](https://arxiv.org/html/2608.17587#bib.bib27)\. Despite having only 4B parameters, the trained Skill Optimizer also outperforms every evaluated off\-the\-shelf general\-purpose model in the same role on BFCL v4, showing that specialized refinement training provides gains beyond generic in\-context reasoning\.

## 2Related Work

#### Skill construction and inference\-time refinement\.

Many systems represent skills as external natural language artifacts that can be created, stored, retrieved, and revised without changing the acting model\. Voyager[20](https://arxiv.org/html/2608.17587#bib.bib6)builds a library of executable skills, while Trace2Skill[11](https://arxiv.org/html/2608.17587#bib.bib15)extracts lessons from individual trajectories and organizes them into transferable skill directories\. SkillsBench[7](https://arxiv.org/html/2608.17587#bib.bib11)and a recent systematization[5](https://arxiv.org/html/2608.17587#bib.bib20)study the value and lifecycle of these artifacts\. Other systems use execution feedback to revise skills at inference time\. EvoSkill[1](https://arxiv.org/html/2608.17587#bib.bib9)derives revisions from failure analysis, SkillOpt[27](https://arxiv.org/html/2608.17587#bib.bib10)edits skill documents under validation constraints, and SkillRevise[9](https://arxiv.org/html/2608.17587#bib.bib19)performs successive revisions conditioned on execution traces\. Execute\-Distill\-Verify[31](https://arxiv.org/html/2608.17587#bib.bib12)further improves reliability through heterogeneous execution and consensus verification\. In these systems, the skill may evolve, but the off\-the\-shelf model that writes it typically does not learn from the refinement experience\. Automatic prompt optimization makes a similar distinction\. OPRO, PromptAgent, EvoPrompt, DSPy, and TextGrad improve textual artifacts through search, task feedback, demonstrations, or textual gradients[25](https://arxiv.org/html/2608.17587#bib.bib21);[22](https://arxiv.org/html/2608.17587#bib.bib22);[3](https://arxiv.org/html/2608.17587#bib.bib23);[6](https://arxiv.org/html/2608.17587#bib.bib24);[29](https://arxiv.org/html/2608.17587#bib.bib25)\. These methods optimize the current artifact\. WER keeps the external text interface but trains the Skill Optimizer itself from multi\-turn trajectories and programmatically verified outcomes in stateful tool environments\.

#### Skills in agentic reinforcement learning\.

Most skill\-augmented reinforcement learning methods update the task agent itself\. SAGE[21](https://arxiv.org/html/2608.17587#bib.bib17)integrates a skill library into GRPO; Skill1[17](https://arxiv.org/html/2608.17587#bib.bib8)jointly learns skill selection, utilization, and distillation; and SkillRL[23](https://arxiv.org/html/2608.17587#bib.bib30)recursively evolves an externally distilled skill bank alongside the agent policy\. ReSkill[4](https://arxiv.org/html/2608.17587#bib.bib29)likewise evaluates competing skill\-bank versions while the executor continues to learn\. Another direction internalizes skills into model parameters: Skill0[10](https://arxiv.org/html/2608.17587#bib.bib13)progressively removes external skills during training, whereas Skill0\.5[30](https://arxiv.org/html/2608.17587#bib.bib31)combines general\-skill internalization with task\-specific retrieval\. In these approaches, adaptation occurs in the acting model or develops alongside it\. WER studies a different setting: the executor remains frozen while a separate policy learns to revise the natural language instructions that condition it\.

#### Learned skill management and refinement\.

The closest work trains the skill\-level decisions themselves\. SkillMaster[26](https://arxiv.org/html/2608.17587#bib.bib18)reviews completed trajectories to propose, update, or retain a skill after each episode, and evaluates the proposed mutation counterfactually on related probe tasks\. SkillOS[12](https://arxiv.org/html/2608.17587#bib.bib7)trains an independent curator to insert, update, or delete entries in an external repository, using subsequent tasks in a related stream to measure the long\-horizon utility of those updates\. Skill\-R1[19](https://arxiv.org/html/2608.17587#bib.bib14)freezes the task model and trains a lightweight skill generator over multiple generations, combining rollout comparisons within a generation with improvement across successive generations\. These methods learn skill improvement at different decision and credit horizons: mutations after an episode, repository operations evaluated on later tasks, and progress across recurrent generations\. WER differs in how it assigns credit within a phase and constructs experience across phases\. Alternative revisions for the same refinement state are compared within a group, while mixed\-outcome executions of the same intermediate skill are paired to form the next phase’s refinement states\. The optimizer is thus trained on selected execution consequences of its own previous revisions\.

## 3Method

![Refer to caption](https://arxiv.org/html/2608.17587v1/framework_v3.png)Figure 2:Training loop\.A refinement state is assembled from the task context, interaction history, and current skill, from which the Skill Optimizer samplesKKcandidate skills \(§[3\.1](https://arxiv.org/html/2608.17587#S3.SS1)\)\. A frozen skill\-conditioned agent executes each candidate in the sandbox, producing trajectories and verifier outcomes \(§[3\.2](https://arxiv.org/html/2608.17587#S3.SS2)\)\. The outcomes provide group\-relative advantages for updatingπθ\\pi\_\{\\theta\}\(§[3\.3](https://arxiv.org/html/2608.17587#S3.SS3)\), while the skills and trajectories enter the experience buffer to assemble next\-phase states \(§[3\.4](https://arxiv.org/html/2608.17587#S3.SS4)\)\. As illustrated on the left, retained skills seed the candidates of the next round, forming a revision tree across phases\. For clarity, the figure shows binary verifier rewards; the actual reward also includes the format and length terms in Eq\.[4](https://arxiv.org/html/2608.17587#S3.E4)\.We consider an agent that solves a multi\-turn task by calling tools against a stateful environment, and that receives, in addition to the task, a short natural\-language skill describing how to proceed\. The task is scored by the environment: an episode succeeds when the terminal state matches the one a reference solution reaches\. Our object of study is not the agent but the skill it is given, and the framework has a single moving part accordingly\. It is a policy that takes a skill together with the record of what happened when that skill ran, and returns a better skill\. Everything else exists to produce that record honestly\. Figure[2](https://arxiv.org/html/2608.17587#S3.F2)shows the loop\. We instantiate it in two multi\-turn, stateful tool environments that expose programmatic verifiers, BFCL multi\-turn[13](https://arxiv.org/html/2608.17587#bib.bib3)andτ2\\tau^\{2\}\-bench[2](https://arxiv.org/html/2608.17587#bib.bib27), and defer the full configuration to the experimental setup\. We define the operator in §[3\.1](https://arxiv.org/html/2608.17587#S3.SS1)and the two channels through which execution reaches it in §[3\.2](https://arxiv.org/html/2608.17587#S3.SS2)\. §[3\.3](https://arxiv.org/html/2608.17587#S3.SS3)gives the optimization procedure, and §[3\.4](https://arxiv.org/html/2608.17587#S3.SS4)applies the operator to its own output across phases\.

### 3\.1Skill Refinement as a Learned Operator

A skill can be rewritten only if it is a well\-defined object with a well\-defined effect, so we fix both before defining the policy that edits it\.

#### Skills\.

A skill is a short markdown document with four sections: a name, a one\-line description of the task family it covers, a numbered workflow, and a list of notes recording assumptions, edge cases and known failure modes\. At execution time the body of the skill is prepended to the downstream agent’s system prompt and nothing else about the agent changes\. It must describe a procedure rather than a solution, since concrete argument values and entity identifiers belong to one instance and would not survive a change of task\.

#### The operator\.

Let

x=\(q,𝒞,h,s,e\)x=\(q,\\;\\mathcal\{C\},\\;h,\\;s,\\;e\)\(1\)denote a*refinement state*, whereqqis the user query,𝒞\\mathcal\{C\}the tool definitions and environment description visible to the agent,hhthe observed interaction history,ssthe skill currently in force, andeethe execution evidence produced the last timesswas run\. The Skill Optimizer is a policy over skill documents,

s′∼πθ\(⋅∣x\)\.s^\{\\prime\}\\sim\\pi\_\{\\theta\}\(\\,\\cdot\\mid x\\,\)\.\(2\)
The optimizer’s confinement to text is what makes the rest of the framework possible\.πθ\\pi\_\{\\theta\}never emits an action in the tool environment\. Its entire output is a document, and whatever the downstream agent subsequently does is mediated by that document\. It is present in the loop only as an observer\.

Eq\.[2](https://arxiv.org/html/2608.17587#S3.E2)is also the sole generation rule in the framework, with no separate mode for writing a skill from nothing\. A cold start is the special case in whichssis a draft induced from\(q,𝒞\)\(q,\\mathcal\{C\}\)alone andeeis the evidence from that draft’s first rollout\. Every later round differs only in wheress,hhandeecame from\. The operator learned in one round is therefore exactly the operator applied in the next\. We call one application of the operator to a task a*round*, and a training stage over the whole dataset a*phase*\. Each phase advances every task by one round, so the number of phases sets how deep a revision chain the optimizer practises on\. How many rounds to run at inference time is then free\.

### 3\.2What the Optimizer Observes

A skill can only be improved from what its execution reveals, so the design question for this subsection is what the optimizer is allowed to see, and who decides whether it worked\.

#### Execution\.

A candidate skills′s^\{\\prime\}is injected into a frozen agentπA\\pi\_\{A\}, which interacts with an instrumented environmentℰ\\mathcal\{E\}and produces a trajectory

τ=\(a1,o1,…,aT,oT,y\),\\tau\\;=\\;\\big\(a\_\{1\},o\_\{1\},\\ldots,a\_\{T\},o\_\{T\},\\;y\\big\),\(3\)whereata\_\{t\}is a tool call together with its arguments,oto\_\{t\}is the environment response including any error returned, andyyis the terminal environment state\. The parameters ofπA\\pi\_\{A\}are never updated, at any point in training\.

#### Channel one: the trajectory, as context\.

The trajectory is not summarized before it reaches the optimizer\. The next refinement state carries the call sequence verbatim, so the model can see which tool was chosen, what arguments it was given, what the environment returned, and what the agent did after a failure\. Only the trajectory tells the optimizer where to edit\. A scalar cannot\. A reward of zero reports that the skill was inadequate\. A trajectory showing the same call issued three times against an identifier the environment has already invalidated reports that the skill never told the agent to check state before acting\.

#### Channel two: the outcome, as reward\.

Whethers′s^\{\\prime\}helped is decided by the environment rather than by a model\. In BFCL multi\-turn we compare the final state of every environment object against the state reached by the reference solution; inτ2\\tau^\{2\}\-bench we compare the terminal database against the reference database and check that the required actions were taken\. Both checks are programmatic and deterministic\. The argument in §[1](https://arxiv.org/html/2608.17587#S1)turns on this\. The failure mode we set out to avoid is a loop whose quality signal comes from the same family of models that produced the behaviour being scored\. Freezing the executor keeps the optimizer out of the trajectory, and using a verifier keeps a model out of the score\. Neither alone is sufficient\.

#### Attribution, and its limits\.

BecauseπA\\pi\_\{A\}is fixed, a difference in outcome between two candidates written for the samexxis evidence about the two documents rather than about an agent that moved between them\. A workflow loop cannot claim that signal, which is why the executor stays frozen even though training it jointly would score higher\. The property holds in expectation rather than per rollout, since one trajectory also reflects the executor’s own sampling, so we score each candidate overnnrollouts and aggregate\. The executor is by design the larger of the two models: the optimizer is small enough to train, while the executor is chosen for capability rather than trainability\.

### 3\.3Group\-Relative Optimization

Absolute outcome varies far more across tasks than across the candidate skills written for any one of them, so an absolute reward would mostly measure task difficulty\. We therefore score candidates against each other under a matched refinement state\.

#### Reward\.

For a refinement statexxthe optimizer samplesKKcandidatess1′,…,sK′s^\{\\prime\}\_\{1\},\\ldots,s^\{\\prime\}\_\{K\}, and each is executednntimes\. The scalar assigned to a candidate combines three terms,

R⁡\(s′\)=13​\(Rfmt\+Rtask\+Rlen\),R\(s^\{\\prime\}\)\\;=\\;\\tfrac\{1\}\{3\}\\Big\(R\_\{\\mathrm\{fmt\}\}\+R\_\{\\mathrm\{task\}\}\+R\_\{\\mathrm\{len\}\}\\Big\),\(4\)with the following roles\.RfmtR\_\{\\mathrm\{fmt\}\}requires the generation to be parseable, with reasoning and skill body in separate delimited blocks so that only the body is injected downstream\. It is annealed downward during training, since format compliance is acquired early and should stop competing with the task term\.RtaskR\_\{\\mathrm\{task\}\}is the verifier outcome aggregated over thennrollouts\.RlenR\_\{\\mathrm\{len\}\}constrains the reasoning block, penalising both empty reasoning and reasoning that runs past a budget\.

#### Why the task reward stays coarse\.

It is tempting to decomposeRtaskR\_\{\\mathrm\{task\}\}into finer credit: was the right tool selected, was the argument correct, did the agent recover after the error, was an irreversible action taken without confirmation\. In these environments none of those questions can be settled programmatically without putting a model in the loop, and a model\-produced score is precisely what §[1](https://arxiv.org/html/2608.17587#S1)argued against\. We therefore let the verifier decide only what it can decide, and rely on comparison within a group to expose the finer differences\. Two candidates for the same state differ only in their text, so a consistent gap between them is informative even when the individual signal is coarse\.

#### Update\.

The advantage of a candidate is its deviation from the group mean,

A^k=Rk−μ⁡\(𝐑\),𝐑=\(R1,…,RK\),\\hat\{A\}\_\{k\}\\;=\\;R\_\{k\}\-\\mu\(\\mathbf\{R\}\),\\qquad\\mathbf\{R\}=\(R\_\{1\},\\ldots,R\_\{K\}\),\(5\)optionally rescaled byσ⁡\(𝐑\)\\sigma\(\\mathbf\{R\}\)\. The optimizer is updated with the clipped GRPO surrogate[15](https://arxiv.org/html/2608.17587#bib.bib5),

𝒥⁡\(θ\)=𝔼x​\[1K​∑k=1Kmin⁡\(ρk​A^k,clip⁡\(ρk,1−ϵ,1\+ϵ\)​A^k\)\]\\displaystyle\\mathcal\{J\}\(\\theta\)=\\mathbb\{E\}\_\{x\}\\\!\\left\[\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\min\\\!\\Big\(\\rho\_\{k\}\\hat\{A\}\_\{k\},\\ \\mathrm\{clip\}\(\\rho\_\{k\},1\{\-\}\\epsilon,1\{\+\}\\epsilon\)\\,\\hat\{A\}\_\{k\}\\Big\)\\right\]

\(6\)whereρk=πθ​\(sk′∣x\)/πold​\(sk′∣x\)\\rho\_\{k\}=\\pi\_\{\\theta\}\(s^\{\\prime\}\_\{k\}\\mid x\)/\\pi\_\{\\text\{old\}\}\(s^\{\\prime\}\_\{k\}\\mid x\)\. We use no KL term and no reference policy\. The output we want is a document format and a revision style that the base model does not yet have, so anchoring the policy to its initialization works against the objective\.

### 3\.4Phase\-Wise Self\-Bootstrapping

A single round of revision closes whichever gap the last execution happened to expose, and the next gap becomes visible only after the revised skill runs again\. The training signal for roundt\+1t\+1therefore has to be manufactured by roundtt\. The left side of Figure[2](https://arxiv.org/html/2608.17587#S3.F2)visualizes this temporal structure\. For a given task, moving down one level advances the refinement by one round: a retained skill from roundtt, together with its execution evidence, becomes the parent state from which the candidates of roundt\+1t\+1are sampled\. Repeating this buffer–reassembly step grows a revision tree rather than restarting each phase from the initial draft\.

#### The buffer\.

Every scored candidate is written to an experience buffer as a tuple\(x,s′,\{τ1,…,τn\},R\)\\big\(x,\\;s^\{\\prime\},\\;\\\{\\tau\_\{1\},\\ldots,\\tau\_\{n\}\\\},\\;R\\big\), retaining allnntrajectories rather than their aggregate\. At the end of a phase the buffer is converted into the refinement states of the next phase, so that the optimizer is next asked to improve its own previous output rather than a draft it has never seen\.

#### Retention\.

Not every record is useful as a next\-phase input\. A candidate whose rollouts all succeed leaves no failure to diagnose; one whose rollouts all fail leaves no working path to contrast against\. With a binary verifier andn=2n=2rollouts a candidate scores00,11or22, and we retain exactly the middle case\. The kept record therefore holds a*matched pair*: two runs of the same skill on the same task, one of which reached the reference state and one of which did not\. Because skill and task are fixed across the pair, the trajectories differ only where the skill left the agent unconstrained\. One run exhibits a path that works, the other the branch the skill failed to rule out, and the difference between them is the edit\. A single trajectory, of either sign, does not localize the gap this way\. Retention is decided by the verifier outcome alone\. A well\-formatted skill that fails and a badly formatted one that succeeds carry very different information, and only the second is worth revising from\.

#### Assembling the next refinement state\.

The next state is built by concatenation rather than summarization\. The skill that produced the pair becomes the current skillss, and the tool context𝒞\\mathcal\{C\}is restored from the original task instance\. The run that reached the reference state is placed in a success block, the run that did not in a failure block\. The assembled state is then presented to the optimizer as a single request to revise\. Nothing is compressed along the way, so the optimizer reads the same call sequences the verifier scored\.

#### Consequences\.

Across phases the input distribution shifts toward states in which the skill in force is nearly but not quite sufficient, which is where a single targeted edit is most likely to change the outcome\. A static dataset cannot supply them: whether a skill is nearly sufficient is a fact about the current optimizer paired with the current executor, and it moves as training proceeds\. Depth also becomes a property of training rather than of inference: having practised the operator,πθ\\pi\_\{\\theta\}can be applied once at test time or repeated until the outcome stops improving\.

The overall training procedure is summarized in Algorithm[1](https://arxiv.org/html/2608.17587#alg1)\.

Algorithm 1Phase\-Wise Training of the Skill Optimizer1:initial refinement states

𝒳\(0\)\\mathcal\{X\}^\{\(0\)\}; optimizer

πθ\\pi\_\{\\theta\}; frozen executor

πA\\pi\_\{A\}; environment

ℰ\\mathcal\{E\}; phases

PP; candidates

KK; rollouts

nn
2:trained Skill Optimizer

πθ\\pi\_\{\\theta\}
3:for

p=0,…,P−1p=0,\\ldots,P\-1do

4:

ℬ\(p\)←∅\\mathcal\{B\}^\{\(p\)\}\\leftarrow\\varnothing⊳\\trianglerightexperience from phasepp

5:for all

x=\(q,𝒞,h,s,e\)∈𝒳\(p\)x=\(q,\\mathcal\{C\},h,s,e\)\\in\\mathcal\{X\}^\{\(p\)\}do

6:Sample

s1′,…,sK′∼πθ\(⋅∣x\)s^\{\\prime\}\_\{1\},\\ldots,s^\{\\prime\}\_\{K\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)
7:for

k=1,…,Kk=1,\\ldots,Kdo

8:for

j=1,…,nj=1,\\ldots,ndo

9:

τk,j←Execute​\(πA,sk′,ℰ\)\\tau\_\{k,j\}\\leftarrow\\textsc\{Execute\}\(\\pi\_\{A\},s^\{\\prime\}\_\{k\},\\mathcal\{E\}\)
10:endfor

11:

Rk←Reward​\(sk′,\{τk,j\}j=1n\)R\_\{k\}\\leftarrow\\textsc\{Reward\}\(s^\{\\prime\}\_\{k\},\\\{\\tau\_\{k,j\}\\\}\_\{j=1\}^\{n\}\)⊳\\trianglerightEq\.[4](https://arxiv.org/html/2608.17587#S3.E4)

12:Add

\(x,sk′,\{τk,j\}j=1n,Rk\)\(x,s^\{\\prime\}\_\{k\},\\\{\\tau\_\{k,j\}\\\}\_\{j=1\}^\{n\},R\_\{k\}\)to

ℬ\(p\)\\mathcal\{B\}^\{\(p\)\}
13:endfor

14:Compute

A^1,…,A^K\\hat\{A\}\_\{1\},\\ldots,\\hat\{A\}\_\{K\}from

R1,…,RKR\_\{1\},\\ldots,R\_\{K\}⊳\\trianglerightEq\.[5](https://arxiv.org/html/2608.17587#S3.E5)

15:endfor

16:Update

θ\\thetaon the collected groups⊳\\trianglerightEq\.[6](https://arxiv.org/html/2608.17587#S3.E6)

17:

𝒳\(p\+1\)←Assemble⁡\(Retain⁡\(ℬ\(p\)\)\)\\mathcal\{X\}^\{\(p\+1\)\}\\leftarrow\\operatorname\{Assemble\}\\\!\\left\(\\operatorname\{Retain\}\(\\mathcal\{B\}^\{\(p\)\}\)\\right\)⊳\\trianglerightmatched outcomes

18:endfor

19:return

πθ\\pi\_\{\\theta\}

Here𝒳\(p\)\\mathcal\{X\}^\{\(p\)\}is the set of refinement states in phasepp,ℬ\(p\)\\mathcal\{B\}^\{\(p\)\}is the experience collected in that phase, andPPis the number of phases\.

## 4Experiments

MethodBFCL v4 Multi\-Turnτ2\\tau^\{2\}\-benchFile SystemTradingTravelVehicleAvg\.↑\\uparrowAirlineRetailTelecomAvg\.↑\\uparrowNo Skill68\.4286\.8462\.1657\.8968\.8345\.0064\.8430\.7746\.87GPT\-5\.1 Seed Skill63\.1678\.9462\.1664\.8667\.2847\.5063\.7331\.8647\.70Qwen3\-4B as Skill Optimizer57\.8984\.2162\.1664\.8667\.2840\.0060\.4320\.8740\.43Skill\-R1[19](https://arxiv.org/html/2608.17587#bib.bib14)71\.0584\.2162\.1667\.5671\.2547\.5056\.0420\.8741\.47Trace2Skill[11](https://arxiv.org/html/2608.17587#bib.bib15)63\.1687\.5562\.1675\.6872\.1452\.5058\.3419\.7843\.54WER \(Ours\)71\.0586\.8470\.2778\.3876\.6350\.0068\.1334\.0650\.72Table 1:Main results on BFCL v4 andτ2\\tau^\{2\}\-bench \(Pass@1, %\)\. Best results are inbold\.### 4\.1Experimental Setup

#### Benchmarks and Metrics\.

We evaluate WER on two benchmarks for long\-horizon tool use: the multi\-turn\-base tasks from BFCL v4[13](https://arxiv.org/html/2608.17587#bib.bib3)andτ2\\tau^\{2\}\-bench[2](https://arxiv.org/html/2608.17587#bib.bib27)\. BFCL v4 contains 200 instances, which we partition into 50 training instances and 150 test instances, balanced across its four task domains\. We report domain\-level results and an overall average that weights each domain equally\. Forτ2\\tau^\{2\}\-bench, which provides its own subdomain partition, we split the base tasks into training and test sets using the same 1:3 ratio\. Because a skill serves as an experience guide distilled by an agent, our primary goal is to assess whether an optimized skill can reliably support successful task completion in a single attempt\. We therefore report Pass@1 rather than Pass@kk, avoiding the extra inference cost of repeated attempts\. To reduce variance on these relatively small test sets, we run each evaluation three times and report the average Pass@1\. Appendix[A\.1](https://arxiv.org/html/2608.17587#A1.SS1)provides further details on both benchmarks\.

#### Data Construction\.

Because the optimizer only revises text skills, its action space does not depend on the interface of the task environment\. We pool the benchmark\-specific training splits defined in[Benchmarks and Metrics](https://arxiv.org/html/2608.17587#S4.SS1.SSS0.Px1), exposing the optimizer to diverse refinement contexts and encouraging cross\-benchmark generalization\. GPT\-5\.1 generates a zero\-shot initial skill for each task\. These skills serve as the first refinement targets and initiate phase\-wise self\-bootstrapping, which continually collects and organizes the training data needed for later rounds of refinement\. Further details are provided in §[3\.4](https://arxiv.org/html/2608.17587#S3.SS4)\.

#### Baselines\.

We compare WER with five baselines\. No Skill lets the base agent solve each task without an external skill\. GPT\-5\.1 Seed Skill gives the agent the initial skill generated by GPT\-5\.1 without further refinement\. Qwen3\-4B as Skill Optimizer replaces our trained optimizer with the untrained Qwen3\-4B base model[24](https://arxiv.org/html/2608.17587#bib.bib16), while keeping the refinement protocol fixed\. This comparison measures the effect of optimizer training\. Skill\-R1[19](https://arxiv.org/html/2608.17587#bib.bib14)trains a lightweight skill generator with reinforcement learning and uses intra\- and inter\-generation advantages from verified recurrent rollouts\. Trace2Skill[11](https://arxiv.org/html/2608.17587#bib.bib15)extracts lessons from individual trajectories in parallel and organizes them into a transferable skill directory without parameter updates\. Appendix[A\.2](https://arxiv.org/html/2608.17587#A1.SS2)provides additional details on Skill\-R1 and Trace2Skill\.

#### Evaluation Protocol\.

We use GPT\-4o as the frozen executor for all training rollouts and downstream evaluations\. At inference time, refinement starts from the GPT\-5\.1 seed skills\. The WER\-trained optimizer performs two rounds of refinement on the training split, producing a set of skills for each domain\. GPT\-5\.5 then merges them into one domain\-specific skill, which is appended to the executor’s system prompt and evaluated on the corresponding test domain\. We use the same protocol for BFCL v4 andτ2\\tau^\{2\}\-bench\. The optimizer is trained with GRPO using the verl framework[16](https://arxiv.org/html/2608.17587#bib.bib28)\. The full training configuration is provided in Appendix[A\.3](https://arxiv.org/html/2608.17587#A1.SS3)\.

### 4\.2Main Results

We first ask whether WER produces skills that outperform skill\-authoring baselines under a single\-attempt protocol\. Table[1](https://arxiv.org/html/2608.17587#S4.T1)reports Pass@1 on both benchmarks\. We examine the results from two angles: the gain a WER skill provides over the base agent and the isolated contribution of optimizer training\.

#### WER Improves Agentic Performance of Base LLMs\.

Table[1](https://arxiv.org/html/2608.17587#S4.T1)shows that WER raises the domain\-averaged Pass@1 over the No Skill baseline from 68\.83% to 76\.63% on BFCL v4 and from 46\.87% to 50\.72% onτ2\\tau^\{2\}\-bench, gains of 7\.80 and 3\.85 percentage points\. On BFCL v4, WER improves three of the four domains and matches the No Skill baseline in Trading\. It obtains the best or tied\-best result in File System, Travel, and Vehicle\. Onτ2\\tau^\{2\}\-bench, WER improves all three domains and ranks first in Retail and Telecom\. Relative to the fixed GPT\-5\.1 seed skills, WER gains 9\.35 points on BFCL v4 and 3\.02 points onτ2\\tau^\{2\}\-bench\. This margin indicates that the gains stem from effective skill optimization rather than solely from GPT\-5\.1’s ability to generate the initial skills\.

#### WER Learns to Refine Skills from Execution Feedback\.

To isolate the effect of optimizer training, we use the untrained Qwen3\-4B base model as the skill optimizer and keep the rest of the refinement protocol fixed \(see[Evaluation Protocol](https://arxiv.org/html/2608.17587#S4.SS1.SSS0.Px4)\)\. It does not improve on the GPT\-5\.1 seed skills on BFCL v4 \(67\.28% in both cases\) and reduces performance from 47\.70% to 40\.43% onτ2\\tau^\{2\}\-bench\. It also trails the No Skill baseline on both benchmarks\. A fixed refinement workflow is therefore not sufficient when the optimizer cannot turn execution experience into useful revisions\. After WER training, the same backbone improves by 9\.35 points on BFCL v4 and 10\.29 points onτ2\\tau^\{2\}\-bench\. Since the backbone and refinement interface are fixed, this comparison measures the learned ability to diagnose trajectories, extract reusable lessons, and revise skills from feedback\. WER has the highest average among the evaluated methods on both benchmarks, including Skill\-R1 and Trace2Skill\.

### 4\.3Phase\-Wise Training Improves Iterative Refinement

PhaseFile Sys\.TradingTravelVehicleAvg\.↑\\uparrow160\.5376\.3262\.1678\.3869\.35273\.6881\.5859\.6470\.2771\.29371\.0586\.8470\.2778\.3876\.63Table 2:Phase\-wise BFCL v4 performance \(Pass@1, %\)\. Best results are inbold\.Does iterative refinement improve because of phase\-wise self\-bootstrapping \(§[3\.4](https://arxiv.org/html/2608.17587#S3.SS4)\), or is one phase of training sufficient? This comparison tests whether later phases provide a distinct training benefit rather than merely repeating the same optimization procedure\. We evaluate the checkpoint from each training phase under the same protocol \(see[Evaluation Protocol](https://arxiv.org/html/2608.17587#S4.SS1.SSS0.Px4)\)\. Average Pass@1 increases monotonically from 69\.35% after Phase 1 to 71\.29% after Phase 2 and 76\.63% after Phase 3, an overall gain of 7\.28 points\. Individual domains fluctuate, but the final checkpoint has the best average, leads in Trading and Travel, and ties for the best Vehicle score\. This pattern supports phase\-wise self\-bootstrapping: training on refinement states induced by earlier optimizer outputs progressively teaches the model to diagnose newly exposed failures and revise skills over multiple rounds\.

### 4\.4Refinement Gains Saturate After a Few Rounds

Given that phase\-wise training helps, we next ask how many successive revisions are useful at inference time\. Each run starts from the same GPT\-5\.1 seed skills\. Depth 0 denotes the seeds, and each subsequent depth adds one WER revision\.

DepthFile Sys\.TradingTravelVehicleAvg\.↑\\uparrow063\.1678\.9462\.1664\.8667\.28160\.5386\.8467\.5767\.5770\.67271\.0586\.8470\.2778\.3876\.63368\.4289\.4767\.5775\.6875\.33Table 3:BFCL v4 performance across refinement depths \(Pass@1, %\)\. Best results are inbold\.Figure 3:BFCL v4 performance trends across refinement depths\.Table[3](https://arxiv.org/html/2608.17587#S4.T3)and Figure[3](https://arxiv.org/html/2608.17587#S4.F3)show diminishing returns\. The first two revisions account for almost all of the gain, raising average Pass@1 from 67\.28% to 76\.63%\. A third revision provides no additional benefit and instead lowers the average slightly to 75\.33%\. These results suggest that iterative skill refinement is most effective in its early rounds and gradually saturates as the number of revisions increases\. Appendix[B](https://arxiv.org/html/2608.17587#A2)shows how successive revisions correct different execution failures\.

### 4\.5Skill Refinement Is Not Subsumed by General Reasoning

The preceding experiments use a fixed optimizer backbone\. Many artifact\-optimization systems rely on the in\-context reasoning of a general\-purpose language model embedded in a fixed workflow\. We therefore ask whether dedicated optimizer training is necessary or whether such generic reasoning is sufficient for skill refinement\. We use DeepSeek\-V4\-Flash, GPT\-5\.5, Gemini 3\.5 Flash, and Claude Sonnet 4\.6 as skill optimizers in the workflow described in[Evaluation Protocol](https://arxiv.org/html/2608.17587#S4.SS1.SSS0.Px4)\. We compare them with the WER\-trained Qwen3\-4B optimizer[24](https://arxiv.org/html/2608.17587#bib.bib16)\. Keeping the refinement workflow fixed separates the effect of specialized optimizer training from the benefit of simply using a more capable model in the same loop\.

Figure 4:Skill\-optimizer backbones on BFCL v4 \(Pass@1, %\)\. The WER\-trained Qwen3\-4B optimizer has the highest average despite its smaller size\. Per\-domain results are reported in Table[4](https://arxiv.org/html/2608.17587#A1.T4)\(Appendix[A\.4](https://arxiv.org/html/2608.17587#A1.SS4)\)\.Figure[4](https://arxiv.org/html/2608.17587#S4.F4)shows that the general\-purpose models reach average Pass@1 scores from 69\.91% to 74\.75%\. The WER\-trained Qwen3\-4B optimizer reaches 76\.63%, 1\.88 points above GPT\-5\.5, the strongest general\-purpose model in this comparison\. WER leads in File System and Travel and ties GPT\-5\.5 in Vehicle\. These findings indicate that skill\-refinement ability is not fully subsumed by general reasoning alone\. Although strong general\-purpose models can serve as capable skill optimizers, WER shows that targeted training offers complementary improvements in diagnosing execution traces, extracting reusable experience, and turning that experience into effective skill revisions\. This refinement capability may also be useful for bootstrapping general\-purpose language models\.

### 4\.6Qualitative Analysis: Successive Revisions Target Distinct Failures

The aggregate results show that refinement helps but not what changes in each revision\. We examine three rounds of WER on a representative BFCL v4 multi\-turn task\. The seed skill fails in both file handling and numerical aggregation, and neither rollout succeeds\. The first revision fixes the file\-creation procedure but leaves one numerical failure\. The second fixes the remaining aggregation error, after which both rollouts succeed\. Different failures are therefore corrected as execution makes them visible\. Figure[5](https://arxiv.org/html/2608.17587#A2.F5)in Appendix[B](https://arxiv.org/html/2608.17587#A2)provides the complete skills and trajectories\.

## 5Conclusion

We introduced WER, a framework for learning to refine reusable agent skills from execution feedback\. WER leaves the downstream executor unchanged and trains a separate Skill Optimizer to revise natural\-language skills from execution trajectories and programmatic verification signals\. Through phase\-wise self\-bootstrapping, each training phase uses refinement states constructed from the consequences of earlier revisions, allowing the optimizer to learn from its own outputs\. Experiments on BFCL v4 andτ2\\tau^\{2\}\-bench show that WER produces more effective revisions than the same backbone without optimizer training\. On BFCL v4, the trained 4B optimizer also outperforms all evaluated general\-purpose models used in the same role\. These results support treating skill refinement as a distinct capability for turning execution feedback into reusable procedural guidance\.

## Limitations

Our study has two main limitations\. First, WER has been evaluated only on BFCL v4 multi\-turn andτ2\\tau^\{2\}\-bench, both of which provide programmatic verifiers\. The current results therefore do not establish whether the learned refinement capability transfers to settings with open\-ended evaluation, unseen tool interfaces, or different executor models\.

Second, WER preserves matched successful and failed trajectories verbatim when constructing subsequent refinement states\. This retains diagnostically useful execution details, but the state size grows as agent interactions become longer and more complex\. We have not yet evaluated this design on substantially longer\-horizon tasks, multimodal trajectories, or large skill repositories, where context scalability may become a bottleneck\. Future work will extend WER to such more complex settings and investigate how to scale experience construction without losing the diagnostic evidence needed for effective refinement\.

## References

- Alzubiet al\.\(2026\)S\. Alzubi, N\. Provenzano, J\. Bingham, W\. Chen, and T\. VuEvoSkill: automated skill discovery for multi\-agent systems\.arXiv preprint arXiv:2603\.02766\.Cited by:[§1](https://arxiv.org/html/2608.17587#S1.p2.1),[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px1.p1.1)\.
- Barreset al\.\(2025\)V\. Barres, H\. Dong, S\. Ray, X\. Si, and K\. Narasimhanτ2\\tau^\{2\}\-Bench: evaluating conversational agents in a dual\-control environment\.arXiv preprint arXiv:2506\.07982\.Cited by:[§A\.1](https://arxiv.org/html/2608.17587#A1.SS1.SSS0.Px2.p1.1),[3rd item](https://arxiv.org/html/2608.17587#S1.I1.i3.p1.1),[§3](https://arxiv.org/html/2608.17587#S3.p1.1),[§4\.1](https://arxiv.org/html/2608.17587#S4.SS1.SSS0.Px1.p1.1)\.
- Guoet al\.\(2024\)Q\. Guo, R\. Wang, J\. Guo, B\. Li, K\. Song, X\. Tan, G\. Liu, J\. Bian, and Y\. YangConnecting large language models with evolutionary algorithms yields powerful prompt optimizers\.InProceedings of ICLR,Cited by:[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px1.p1.1)\.
- Heet al\.\(2026\)Z\. He, H\. Lin, B\. Han, W\. Zhu, H\. Fang, B\. Wang, X\. Zhu, R\. Li, and M\. ReimherrReSkill: reconciling skill creation with policy optimization in agentic RL\.arXiv preprint arXiv:2606\.01619\.Cited by:[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px2.p1.1)\.
- Jianget al\.\(2026\)Y\. Jiang, D\. Li, H\. Deng, B\. Ma, X\. Wang, Q\. Wang, and G\. YuSoK: agentic skills – beyond tool use in LLM agents\.arXiv preprint arXiv:2602\.20867\.Cited by:[§1](https://arxiv.org/html/2608.17587#S1.p1.1),[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px1.p1.1)\.
- Khattabet al\.\(2024\)O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. Vardhamanan, S\. Haq, A\. Sharma, T\. T\. Joshi, H\. Moazam, H\. Miller, M\. Zaharia, and C\. PottsDSPy: compiling declarative language model calls into state\-of\-the\-art pipelines\.InProceedings of the 12th International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2026\)X\. Li, Y\. Liu, W\. Chen, B\. You, Z\. Di, Y\. He, S\. Zheng, K\. W\. Choe, J\. Sun, S\. Wang,et al\.SkillsBench: benchmarking how well agent skills work across diverse tasks\.arXiv preprint arXiv:2602\.12670\.Cited by:[§1](https://arxiv.org/html/2608.17587#S1.p1.1),[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2025\)Y\. Liu, C\. Si, K\. Narasimhan, and S\. YaoContextual experience replay for self\-improvement of language agents\.arXiv preprint arXiv:2506\.06698\.Cited by:[§1](https://arxiv.org/html/2608.17587#S1.p2.1)\.
- Liuet al\.\(2026\)Y\. Liu, Z\. Su, L\. Xie, Y\. Zhang, Q\. Zong, J\. Guo, Z\. Xie, Y\. Ji, Y\. Yim, H\. Luo, X\. Ren, R\. Chenyu, H\. Li, and Y\. SongSkillRevise: improving LLM\-authored agent skills via trace\-conditioned skill revision\.arXiv preprint arXiv:2606\.01139\.Cited by:[§1](https://arxiv.org/html/2608.17587#S1.p2.1),[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px1.p1.1)\.
- Luet al\.\(2026\)Z\. Lu, Z\. Yao, J\. Wu, C\. Han, Q\. Gu, X\. Cai, W\. Lu, J\. Xiao, Y\. Zhuang, and Y\. ShenSkill0: in\-context agentic reinforcement learning for skill internalization\.arXiv preprint arXiv:2604\.02268\.Cited by:[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px2.p1.1)\.
- Niet al\.\(2026\)J\. Ni, Y\. Liu, X\. Liu, Y\. Sun, M\. Zhou, P\. Cheng, D\. Wang, E\. Zhao, X\. Jiang, and G\. JiangTrace2Skill: distill trajectory\-local lessons into transferable agent skills\.arXiv preprint arXiv:2603\.25158\.Cited by:[§A\.2](https://arxiv.org/html/2608.17587#A1.SS2.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2608.17587#S4.SS1.SSS0.Px3.p1.1),[Table 1](https://arxiv.org/html/2608.17587#S4.T1.2.7.1)\.
- Ouyanget al\.\(2026\)S\. Ouyang, J\. Yan, Y\. Chen, R\. Han, Z\. Wang, B\. D\. Mishra, R\. Meng, C\. Li, Y\. Jiao, K\. Zha, M\. Shen, V\. Tirumalashetty, G\. Lee, J\. Han, T\. Pfister, and C\. LeeSkillOS: learning skill curation for self\-evolving agents\.arXiv preprint arXiv:2605\.06614\.Cited by:[§1](https://arxiv.org/html/2608.17587#S1.p3.1),[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px3.p1.1)\.
- Patilet al\.\(2025\)S\. G\. Patil, H\. Mao, C\. C\. Ji, F\. Yan, V\. Suresh, I\. Stoica, and J\. E\. GonzalezThe berkeley function calling leaderboard \(BFCL\): from tool use to agentic evaluation of large language models\.InProceedings of the 42nd International Conference on Machine Learning,Cited by:[§A\.1](https://arxiv.org/html/2608.17587#A1.SS1.SSS0.Px1.p1.1),[3rd item](https://arxiv.org/html/2608.17587#S1.I1.i3.p1.1),[§1](https://arxiv.org/html/2608.17587#S1.p1.1),[§3](https://arxiv.org/html/2608.17587#S3.p1.1),[§4\.1](https://arxiv.org/html/2608.17587#S4.SS1.SSS0.Px1.p1.1)\.
- Schicket al\.\(2023\)T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, L\. Zettlemoyer, N\. Cancedda, and T\. ScialomToolformer: language models can teach themselves to use tools\.arXiv preprint arXiv:2302\.04761\.Cited by:[§1](https://arxiv.org/html/2608.17587#S1.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§3\.3](https://arxiv.org/html/2608.17587#S3.SS3.SSS0.Px3.p1.2)\.
- Shenget al\.\(2025\)G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. WuHybridFlow: a flexible and efficient RLHF framework\.InProceedings of the Twentieth European Conference on Computer Systems,pp\. 1279–1297\.External Links:[Document](https://dx.doi.org/10.1145/3689031.3696075)Cited by:[§A\.3](https://arxiv.org/html/2608.17587#A1.SS3.p1.1),[§4\.1](https://arxiv.org/html/2608.17587#S4.SS1.SSS0.Px4.p1.1)\.
- Shiet al\.\(2026\)Y\. Shi, Y\. Chen, Z\. Lu, Y\. Miao, S\. Liu, Q\. Gu, X\. Cai, X\. Wang, and A\. ZhangSkill1: unified evolution of skill\-augmented agents via reinforcement learning\.arXiv preprint arXiv:2605\.06130\.Cited by:[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px2.p1.1)\.
- Shinnet al\.\(2023\)N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. YaoReflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§1](https://arxiv.org/html/2608.17587#S1.p2.1)\.
- Visheet al\.\(2026\)Y\. Vishe, R\. Surana, X\. Jiang, Z\. Huang, X\. Li, N\. L\. Kuang, T\. Yu, R\. A\. Rossi, J\. Shang, J\. McAuley, and J\. WuSkill\-r1: agent skill evolution via reinforcement learning\.arXiv preprint arXiv:2605\.09359\.Cited by:[§A\.2](https://arxiv.org/html/2608.17587#A1.SS2.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.17587#S1.p3.1),[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2608.17587#S4.SS1.SSS0.Px3.p1.1),[Table 1](https://arxiv.org/html/2608.17587#S4.T1.2.6.1)\.
- Wanget al\.\(2023\)G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. AnandkumarVoyager: an open\-ended embodied agent with large language models\.arXiv preprint arXiv:2305\.16291\.Cited by:[§1](https://arxiv.org/html/2608.17587#S1.p1.1),[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025\)J\. Wang, Q\. Yan, Y\. Wang, Y\. Tian, S\. S\. Mishra, Z\. Xu, M\. Gandhi, P\. Xu, and L\. L\. CheongSAGE: reinforcement learning for self\-improving agent with skill library\.arXiv preprint arXiv:2512\.17102\.Cited by:[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2024\)X\. Wang, C\. Li, Z\. Wang, F\. Bai, H\. Luo, J\. Zhang, N\. Jojic, E\. P\. Xing, and Z\. HuPromptAgent: strategic planning with language models enables expert\-level prompt optimization\.InProceedings of ICLR,Cited by:[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px1.p1.1)\.
- Xiaet al\.\(2026\)P\. Xia, J\. Chen, H\. Wang, J\. Liu, K\. Zeng, Y\. Wang, S\. Han, Y\. Zhou, X\. Zhao, H\. Chen, Z\. Zheng, C\. Xie, and H\. YaoSkillRL: evolving agents via recursive skill\-augmented reinforcement learning\.arXiv preprint arXiv:2602\.08234\.Cited by:[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px2.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.1](https://arxiv.org/html/2608.17587#S4.SS1.SSS0.Px3.p1.1),[§4\.5](https://arxiv.org/html/2608.17587#S4.SS5.p1.1)\.
- Yanget al\.\(2024\)C\. Yang, X\. Wang, Y\. Lu, H\. Liu, Q\. V\. Le, D\. Zhou, and X\. ChenLarge language models as optimizers\.InProceedings of ICLR,Cited by:[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2026a\)M\. Yang, J\. Piao, X\. Xia, X\. Lan, J\. Chen, Y\. Gong, and Y\. LiSkillMaster: toward autonomous skill mastery in LLM agents\.arXiv preprint arXiv:2605\.08693\.Cited by:[§1](https://arxiv.org/html/2608.17587#S1.p3.1),[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px3.p1.1)\.
- Yanget al\.\(2026b\)Y\. Yang, Z\. Gong, W\. Huang, Q\. Yang, Z\. Zhou, Z\. Huang, Y\. Li, X\. Gao, Q\. Dai, B\. Liu, K\. Qiu, Y\. Yang, D\. Chen, X\. Yang, and C\. LuoSkillOpt: executive strategy for self\-evolving agent skills\.arXiv preprint arXiv:2605\.23904\.Cited by:[§1](https://arxiv.org/html/2608.17587#S1.p2.1),[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px1.p1.1)\.
- Yaoet al\.\(2023\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. CaoReAct: synergizing reasoning and acting in language models\.InProceedings of ICLR,Cited by:[§1](https://arxiv.org/html/2608.17587#S1.p1.1)\.
- Yuksekgonulet al\.\(2024\)M\. Yuksekgonul, F\. Bianchi, J\. Boen, S\. Liu, Z\. Huang, C\. Guestrin, and J\. ZouTextGrad: automatic “differentiation” via text\.arXiv preprint arXiv:2406\.07496\.Cited by:[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px1.p1.1)\.
- Zhuet al\.\(2026a\)J\. Zhu, J\. Yu, Y\. Zhao, C\. Han, Q\. Gu, X\. Cai, X\. Li, and W\. QianSkill0\.5: joint skill internalization and utilization for out\-of\-distribution generalization in agentic reinforcement learning\.arXiv preprint arXiv:2605\.28424\.Cited by:[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px2.p1.1)\.
- Zhuet al\.\(2026b\)S\. Zhu, Y\. Qi, Y\. Wang, J\. Li, C\. Song, Y\. Shi, Y\. Miao, H\. Gao, and K\. ZhangEscaping the self\-confirmation trap: an execute\-distill\-verify paradigm for agentic experience learning\.arXiv preprint arXiv:2606\.24428\.Cited by:[§1](https://arxiv.org/html/2608.17587#S1.p2.1),[§2](https://arxiv.org/html/2608.17587#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix AAdditional Experimental Details

### A\.1Benchmark Details

#### BFCL v4\.

We use the 200 multi\-turn\-base tasks from BFCL v4[13](https://arxiv.org/html/2608.17587#bib.bib3), which are evenly distributed across File System, Vehicle Control, Trading Bot, and Travel Booking\. Each task may require multiple tool calls over several conversational turns, requiring the agent to maintain interaction context, use intermediate tool outputs, and execute a coherent sequence of state\-dependent actions\. The available tools include both domain\-specific APIs and cross\-functional utilities such as messaging and mathematical operations\. Evaluation is execution based: the predicted calls are run in the corresponding environment, and success is determined from the resulting environment state rather than exact trajectory matching\.

#### τ2\\tau^\{2\}\-bench\.

τ2\\tau^\{2\}\-bench[2](https://arxiv.org/html/2608.17587#bib.bib27)evaluates conversational tool agents in the Airline, Retail, and Telecom domains\. Each domain supplies a policy, task\-specific tools, a stateful environment, and an LLM\-based user simulator\. Airline and Retail center on policy\-constrained customer\-service workflows, whereas Telecom introduces a dual\-control setting in which the agent and user operate distinct tools over a shared environment\. The agent must therefore not only reason about the task and call its own tools, but also communicate effectively and guide actions that only the user can perform\. Tasks are scored by verifiable outcomes, including the final environment state and required information communicated to the user, rather than by matching a single reference trajectory\. For both benchmarks, we report Pass@1 as the single\-attempt task success rate\.

### A\.2Baseline Details

#### Skill\-R1\.

Skill\-R1[19](https://arxiv.org/html/2608.17587#bib.bib14)formulates skill optimization as a recurrent reinforcement\-learning problem while keeping the task model frozen\. A lightweight skill generator conditions on the task context, previous rollouts, and their verified outcomes to produce the next skill revision\. Its bi\-level group\-relative objective combines intra\-generation advantages, which compare rollouts under the same skill, with inter\-generation advantages that reward improvements across successive revisions\.

#### Trace2Skill\.

Trace2Skill[11](https://arxiv.org/html/2608.17587#bib.bib15)constructs transferable skills from a pool of agent execution trajectories without updating model parameters\. Multiple analyst agents process trajectories in parallel to extract trajectory\-local lessons and propose skill patches; these patches are then consolidated hierarchically into a unified, conflict\-free skill directory\. This design supports both refining an existing skill and constructing one from an initial weak draft while reducing sensitivity to any single trajectory\.

### A\.3Training Details

Optimizer training is conducted with the verl framework[16](https://arxiv.org/html/2608.17587#bib.bib28)on a single node equipped with eight Ascend 910B NPUs\. We use GRPO with a batch size of 6 and sample four rollouts per prompt\. The learning rate is set to1×10−61\\times 10^\{\-6\}following a cosine schedule with warmup\. The maximum prompt and response lengths are 19,000 and 4,096 tokens, respectively, and rollouts are sampled with a temperature of 0\.95 and top\-kksampling withk=50k=50\.

### A\.4Per\-Domain Results for Skill\-Optimizer Backbones

Table[4](https://arxiv.org/html/2608.17587#A1.T4)reports the full per\-domain breakdown for the skill\-optimizer backbone comparison summarized in Figure[4](https://arxiv.org/html/2608.17587#S4.F4)\.

OptimizerFile Sys\.TradingTravelVehicleAvg\.↑\\uparrowDeepSeek\-V4\-Flash68\.4284\.2167\.5670\.2772\.61GPT\-5\.569\.7489\.4863\.5278\.3874\.75Gemini 3\.5 Flash60\.5389\.4864\.8672\.9771\.96Claude Sonnet 4\.668\.4284\.2154\.0572\.9769\.91WER \(Qwen3\-4B\)71\.0586\.8470\.2778\.3876\.63Table 4:BFCL v4 performance with different skill optimizers \(Pass@1, %\)\. Best results are inbold\.

## Appendix BExtended Case Study Analysis

To illustrate how WER improves a skill over successive rounds, Figure[5](https://arxiv.org/html/2608.17587#A2.F5)traces three refinement stages on a representative BFCL v4 multi\-turn task\. The task requires the agent to read a financial report, aggregate several values, and write the rounded result to a newly created file\. At each stage, we evaluate the current skill with two independent agent rollouts\. In the visualization, highlighted spans identify the key skill instructions introduced or reinforced at each stage, while red and green bubbles in the execution traces denote erroneous and correct steps, respectively\.

The initial seed skill leaves both file creation and numerical aggregation insufficiently specified, causing both rollouts to fail\. After observing these failures, the first revision explicitly instructs the agent to create a missing file before writing to it\. This eliminates the file\-operation error, although one rollout still computes the aggregate incorrectly\. The second revision then clarifies that all relevant values should be collected before aggregation and that rounding should be applied only to the final result\. With these complementary corrections, both rollouts succeed\. The progression from 0/2 to 1/2 and finally 2/2 successful rollouts provides a concrete example of WER addressing newly exposed failure modes across successive revisions\.

Figure 5:Three\-stage skill refinement on a representative BFCL v4 multi\-turn task\.
## Appendix CPrompt Templates

### C\.1Skill Revision Prompt

Figure 6:Prompt used by the skill optimizer to revise a skill from successful and failed execution trajectories\.
### C\.2Skill Merging Prompt

Figure 7:Prompt used to consolidate a set of related skills into a single reusable skill\.