R²Orch：通过技能内化循环中的自博弈实现智能体编排的递归自我改进

# Recursive Self-Improvement for Agent Orchestrationvia Self-Play within a Skill-Internalization Loop
Source: [https://arxiv.org/html/2610.22740](https://arxiv.org/html/2610.22740)
Yun HuangAffiliation:The Chinese University of Hong Kong, Shenzhen, ChinaQiang HuangAffiliation:Fudan University, ChinaYanjin LiAffiliation:University of Illinois Urbana\-Champaign, USARui QianAffiliation:Fudan University, ChinaQichao ZhaoAffiliation:Tsinghua University, ChinaTiesunlong ShenAffiliation:National University of Singapore, SingaporeNa ZhaoAffiliation:The Chinese University of Hong Kong, Shenzhen, ChinaJian WangAffiliation:The Chinese University of Hong Kong, Shenzhen, ChinaXiaoying TangAffiliation:The Chinese University of Hong Kong, Shenzhen, China

###### Abstract

In recent years, agentic orchestration has been widely applied to solve complex human tasks\. However, existing orchestration learning still faces key challenges, including*hand\-designed curricula*,*advantage collapse*when a task’s rollouts all reach the same outcome, and*unreliable evaluation signals*that judge each skill by one noisy score\. To address these challenges, we propose R2Orch, a new paradigm of*Recursive Self\-Play Orchestration*: an inner loop of self\-play that improves the orchestrator on self\-chosen tasks, and an outer loop that improves how the inner loop improves\. To support this paradigm, we introduce*Recursive Learnability*, a training objective that rewards a Proposer for choosing tasks whose rollouts keep reaching different outcomes as skills are withdrawn\. Built on this objective, we further propose*Skill Withdrawal*, in which the outer loop withdraws each skill once a controlled evaluation within self\-play shows that the orchestrator performs as well without it\. Moreover, the two loops form one cycle: failures are distilled into skills, skills into parameters, and withdrawals lead the Proposer to new failures, so the orchestrator moves toward zero\-skill deployment\. Experimental results on twelve datasets show that R2Orch consistently outperforms baselines across question answering, math, code, and interactive tasks\. Our code is available at[anonymous\.4open\.science/r/R2Orch\-36F1](https://anonymous.4open.science/r/R2Orch-36F1/)\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2610.22740v1/R2Orch-Figure1.png)Figure 1:A recursion needs two channels: the outer loop sets the inner loop’s objective, and the inner loop, Proposer–Director self\-play on a randomized rollout group, returns the evidence for withdrawing skills\.![Refer to caption](https://arxiv.org/html/2610.22740v1/R2Orch-Figure2.png)Figure 2:Paradigm comparison\. \(a\) Learned orchestration trains on a hand\-designed*static curriculum*\. \(b\) Self\-improving agents measure the frontier with skills present, a*skill\-anchored frontier*where advantages collapse once skills leave\. \(c\) Skill scaffolds withdraw guidance through*success\-rate gates*, unreliable evaluation signals pinned by self\-play\. \(d\) R2Orch nests two loops joined by downward \(objective\) and upward \(evidence\) channels\.In recent years, a variety of powerful agentic systems have been applied to solve a wide range of human problems\([Zhang et al\., 2025](https://arxiv.org/html/2610.22740#bib.bib47);[Li et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib18);[Zhang et al\., 2026c](https://arxiv.org/html/2610.22740#bib.bib48)\), gradually moving beyond hand\-written agent pipelines toward orchestrators that learn to compose agent teams\.

In this process, the agent graph has become a key bridge from task goals to executable collaboration: by giving each node a role and each edge a hand\-off, orchestrators solve tasks with improved controllability, compositionality, and reusability\([Zhang et al\., 2025](https://arxiv.org/html/2610.22740#bib.bib47);[Zhang et al\., 2026c](https://arxiv.org/html/2610.22740#bib.bib48)\), as shown in Figure[1](https://arxiv.org/html/2610.22740#S1.F1)\. However, in practice, orchestrators still learn from tasks people collect and lean on in\-prompt skills\([Li et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib18);[Zhang et al\., 2026d](https://arxiv.org/html/2610.22740#bib.bib49)\), making them costly to scale and hard to improve on their own\.

To address these issues, three main lines of work have emerged, as shown in Figure[2](https://arxiv.org/html/2610.22740#S1.F2)\. First,*learned orchestration*optimizes workflows and planning from task feedback\([Zhang et al\., 2025](https://arxiv.org/html/2610.22740#bib.bib47);[Li et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib18);[Zhang et al\., 2026c](https://arxiv.org/html/2610.22740#bib.bib48)\), taking the task pool as given\. Second,*self\-improving agents*refine their own memory, harness code, answers, or tasks through repeated loops\([Zhu et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib54);[Srikanth et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib37);[Zhang et al\., 2026b](https://arxiv.org/html/2610.22740#bib.bib46);[Lu et al\., 2026a](https://arxiv.org/html/2610.22740#bib.bib22);[Fu et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib6)\)\. Third,*skill scaffolds*hand procedural knowledge to a policy\([Zhang et al\., 2026d](https://arxiv.org/html/2610.22740#bib.bib49);[Shi et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib35);[Lu et al\., 2026b](https://arxiv.org/html/2610.22740#bib.bib23);[Shen et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib34)\), in context at inference or as training\-time support\.

However, these methods still face three challenges \(Figure[2](https://arxiv.org/html/2610.22740#S1.F2)a–c\)\.\(i\) Hand\-designed curricula\.Learned orchestration and skill internalization practice on task pools that people collect in advance\([Zhang et al\., 2026c](https://arxiv.org/html/2610.22740#bib.bib48);[Shi et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib35)\), and self\-play chooses its own tasks while the solver keeps retrieving its skills\([Fu et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib6)\), so choosing tasks and withdrawing skills do not inform each other\.\(ii\) Advantage collapse\.Group\-based RL such as GRPO learns from differences among a task’s rollouts, which vanish when they all reach the same outcome\([He et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib8)\); self\-play keeps tasks at the solver’s frontier with skills in context\([Fu et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib6)\), so withdrawing a skill pushes the tasks it supported toward all\-wrong rollouts, and advantage collapse returns\.\(iii\) Unreliable evaluation signals\.Training\-time scaffolds fade guidance on a success rate mixing task choice with the skill’s effect\([Shi et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib35)\), and skill libraries keep or retire skills on one noisy score each, so they drift until injecting a skill scores worse than injecting nothing\([Zhang et al\., 2026e](https://arxiv.org/html/2610.22740#bib.bib50);[Zhang et al\.,](https://arxiv.org/html/2610.22740#bib.bib51)\)\.

To address these challenges, we proposeR2Orch\(Fig\.[2](https://arxiv.org/html/2610.22740#S1.F2)d\), a new paradigm of*Recursive Self\-Play Orchestration*, in which an inner loop of self\-play improves the orchestrator, a Director, on tasks a Proposer chooses, and an outer loop improves how the inner loop improves\. Two channels join the loops, one carrying the objective down and one the evidence up\. To support this paradigm, we introduce*Recursive Learnability*, the downward channel, which rewards the Proposer for choosing tasks whose rollouts keep reaching different outcomes as skills are withdrawn, scoring each task with the candidate skill withheld\. Built on this objective, we further propose*Skill Withdrawal*, the upward channel, in which a random part of each rollout group hides one skill, turning self\-play into a controlled evaluation whose Bayesian posterior over the skill’s remaining benefit decides when it can leave\. The two loops thus close into one cycle: failures are distilled into skills, skills into parameters, and withdrawals lead the Proposer to new failures, so the Director improves on its own and moves toward zero\-skill deployment\.

We evaluate R2Orch on twelve benchmarks in question answering, math, code, and interactive tasks\. Results show that R2Orch outperforms all baselines on every benchmark, improves six other Worker backbones without retraining, and keeps its accuracy when tested with zero skills\. Together, these results mark a step toward the recursive self\-improvement of agent orchestrators\.

## 2Related Work

#### Learned Orchestration and Self\-Play\.

Prior work searches agent workflows\([Zhang et al\., 2025](https://arxiv.org/html/2610.22740#bib.bib47)\), learns graph editing\([Zhang et al\., 2026c](https://arxiv.org/html/2610.22740#bib.bib48)\), or trains the planner of an agent system\([Li et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib18)\), over tasks that people collect in advance\. Self\-play lets a Proposer pose tasks to a solver, rewarding regret\([Dennis et al\., 2020](https://arxiv.org/html/2610.22740#bib.bib5)\), solver uncertainty\([Huang et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib10)\), or learning progress while distilling failures into skills\([Fu et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib6)\)\. Group\-based RL\([Shao et al\., 2024](https://arxiv.org/html/2610.22740#bib.bib32);[Liu et al\., 2025](https://arxiv.org/html/2610.22740#bib.bib21)\)gives no advantage to a task whose rollouts all reach the same outcome\([He et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib8)\), so curricula favor learnable tasks\([Rutherford et al\., 2024](https://arxiv.org/html/2610.22740#bib.bib29)\)or filter the rest out\([Yu et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib44)\)\. R2Orch scores tasks with the candidate skill withheld, so its Proposer picks tasks whose rollouts keep reaching different outcomes instead of filtering collapsed ones\.

#### Skill Evolution and Self\-Improvement\.

Skills enter the prompt at inference\([Zhang et al\., 2026d](https://arxiv.org/html/2610.22740#bib.bib49)\)or support training and are withdrawn on a success\-rate threshold\([Shi et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib35)\), a preset budget\([Lu et al\., 2026b](https://arxiv.org/html/2610.22740#bib.bib23)\), or leave\-one\-skill\-out validation\([Shen et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib34)\)\. Rollouts with and without a skill score it\([Tu et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib40);[Lin et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib19)\), paired tests accept self\-modifications\([Shawn, 2026](https://arxiv.org/html/2610.22740#bib.bib33)\), and one noisy score per skill lets a library drift\([Zhang et al\.,](https://arxiv.org/html/2610.22740#bib.bib51);[Zhang et al\., 2026e](https://arxiv.org/html/2610.22740#bib.bib50)\)\. Recursive self\-improvement systems revise their own memory, code, or answers\([Zhu et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib54);[Srikanth et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib37);[Lu et al\., 2026a](https://arxiv.org/html/2610.22740#bib.bib22);[Zhang et al\., 2026b](https://arxiv.org/html/2610.22740#bib.bib46);[Wang et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib41)\)or adopt a self\-modification once it is shown to help\([Schmidhuber, 2007](https://arxiv.org/html/2610.22740#bib.bib30);[Zhang et al\., 2026a](https://arxiv.org/html/2610.22740#bib.bib45)\)\. R2Orch nests self\-play in a skill\-withdrawal loop and decides each withdrawal from the controlled evaluation within every randomized rollout group, at no extra rollouts\.

![Refer to caption](https://arxiv.org/html/2610.22740v1/R2Orch-Figure3.png)Figure 3:Overview of R2Orch\. Top: Proposer–Director self\-play \(inner loop\) and skill withdrawal \(outer loop\) share one randomized rollout group\. Bottom left: Recursive Learnability scores a task with the candidate skill hidden\. Bottom right: Skill Withdrawal withdraws or admits each skill by the posterior of its paired contrast\.

## 3Preliminaries

Definition 1: Agent Graph\.An agent graph is a directed graph𝒢=\(V,E,κ,o\)\\mathcal\{G\}=\(V,E,\\kappa,o\), where each nodev∈Vv\\in Vis an agent with a role prompt and a model route, each edgee∈Ee\\in Eis a directed or bidirectional hand\-off between two agents,κ\\kappaassigns every agent a layer, ando∈Vo\\in Vis the output agent whose answer is returned \(our role\-prompted nodes extend the operator nodes of[Zhang et al\., 2026c](https://arxiv.org/html/2610.22740#bib.bib48)\)\.

Definition 2: Director Trajectory\.A complete sequence ofTTatomic edits from an empty agent graph toFinishdefines a trajectory, in which every edit is validated and applied before the next one and its feedback returns to the Director:

ξ=\(a1,…,aT\)⇒g,𝒢t=Apply\(𝒢t−1,at\),𝒢0=∅,g=𝒢T,\\begin\{gathered\}\\xi=\(a\_\{1\},\\dots,a\_\{T\}\)\\Rightarrow g,\\\\ \\mathcal\{G\}\_\{t\}=\\textsc\{Apply\}\(\\mathcal\{G\}\_\{t\-1\},a\_\{t\}\),\\ \\ \\mathcal\{G\}\_\{0\}=\\varnothing,\\ \\ g=\\mathcal\{G\}\_\{T\},\\end\{gathered\}\(1\)whereata\_\{t\}is one ofAddAgent,SetPrompt,SetModel,SetLayer,ConsiderRelation,DeleteAgent,SetOutput, andFinish, andggis the final agent graph that frozen Workers execute\.

Problem Statement\.Given an evaluation distributionρ\\rho, a frozen Worker pool, a verifierrr, and a skill library, we seek a Directorπθ\\pi\_\{\\theta\}that builds a successful agent graph with no skill in its context, trained on tasks chosen by a Proposerρϕ\\rho\_\{\\phi\}:

maxθ𝔼x∼ρ𝔼g∼πθ\(⋅∣x,∅\)\[r\(g,x\)\],\\max\_\{\\theta\}\\ \\mathbb\{E\}\_\{x\\sim\\rho\}\\,\\mathbb\{E\}\_\{g\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x,\\varnothing\)\}\\big\[r\(g,x\)\\big\],\(2\)wherer⁡\(g,x\)∈\{0,1\}r\(g,x\)\\in\\\{0,1\\\}is the verifier’s correctness after the Workers executeggonxx, and the empty set marks zero\-skill deployment\. Skills scaffold training, and the Proposer chooses tasks whose rollouts reach different outcomes\. Equation \([2](https://arxiv.org/html/2610.22740#S3.E2)\) is the deployed target; Section[4](https://arxiv.org/html/2610.22740#S4)shows how a shrinking scaffold and a learned curriculum approach it\.

## 4Method

As illustrated in Figure[3](https://arxiv.org/html/2610.22740#S2.F3), this section introduces the R2Orch framework, including Recursive Self\-Play Orchestration \(Section[4\.1](https://arxiv.org/html/2610.22740#S4.SS1)\), Recursive Learnability \(Section[4\.2](https://arxiv.org/html/2610.22740#S4.SS2)\), and Skill Withdrawal \(Section[4\.3](https://arxiv.org/html/2610.22740#S4.SS3)\), all built on one randomized rollout group\.

### 4\.1Recursive Self\-Play Orchestration

Recursive Self\-Play Orchestration nests two self\-improvement loops \(Figure[3](https://arxiv.org/html/2610.22740#S2.F3), top\)\. In the inner loop, self\-play improves the Director on tasks that the Proposer chooses; the outer loop improves how the inner loop improves, by withdrawing the Director’s skills as each stops helping\.

Two time scales\.Following two\-time\-scale stochastic approximation\([Borkar, 1997](https://arxiv.org/html/2610.22740#bib.bib4)\), both policies take proximal best responses on the fast time scale, each a clipped step\([Schulman et al\., 2017](https://arxiv.org/html/2610.22740#bib.bib31)\):

θt\+1≈argmaxθ𝔼vK\(x;θ\)−1βKL\(πθ∥πθt\),ϕt\+1≈argmaxϕ𝔼RP\(x\)−τKL\(ρϕ∥ρ0\)\.\\begin\{gathered\}\\theta\_\{t\+1\}\\approx\\arg\\max\_\{\\theta\}\\mathbb\{E\}\\,v\_\{K\}\(x;\\theta\)\-\\tfrac\{1\}\{\\beta\}\\,\\mathrm\{KL\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\theta\_\{t\}\}\),\\\\ \\phi\_\{t\+1\}\\approx\\arg\\max\_\{\\phi\}\\mathbb\{E\}\\,R\_\{P\}\(x\)\-\\tau\\,\\mathrm\{KL\}\(\\rho\_\{\\phi\}\\\|\\rho\_\{0\}\)\.\\end\{gathered\}\(3\)Here the Director, trained on the Proposer’s picks withK∼qxK\\sim q\_\{x\}, is scored by the valuevK\(x;θ\)=𝔼ξ∼πθ\(⋅∣x,K\)\[u\(ξ,x\)\]v\_\{K\}\(x;\\theta\)=\\mathbb\{E\}\_\{\\xi\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x,K\)\}\[u\(\\xi,x\)\]of the context it reads under the rewarduu\(correctness less a Worker cost\), and the Proposer by the rewardRPR\_\{P\}of Section[4\.2](https://arxiv.org/html/2610.22740#S4.SS2)\. On the slow scale, the outer loop updates the scaffold from the same rollout groups, and the scaffold in turn sets the randomized context of every group:

𝒮t\+1=Φ⁡\(𝒮t,\{Πk\}k∈𝒮t,ℱt\),qx=nhG​δKx∖kx\+nvG​δKx,\\begin\{gathered\}\\mathcal\{S\}\_\{t\+1\}=\\Phi\\big\(\\mathcal\{S\}\_\{t\};\\,\\\{\\Pi\_\{k\}\\\}\_\{k\\in\\mathcal\{S\}\_\{t\}\},\\,\\mathcal\{F\}\_\{t\}\\big\),\\\\ q\_\{x\}=\\tfrac\{n\_\{h\}\}\{G\}\\,\\delta\_\{K\_\{x\}\\setminus k\_\{x\}\}\+\\tfrac\{n\_\{v\}\}\{G\}\\,\\delta\_\{K\_\{x\}\},\\end\{gathered\}\(4\)whereKx⊆𝒮tK\_\{x\}\\subseteq\\mathcal\{S\}\_\{t\}are the skills retrieved forxx,kx∈Kxk\_\{x\}\\in K\_\{x\}is the skill under test,Πk\\Pi\_\{k\}the posterior overkk’s benefit,ℱt\\mathcal\{F\}\_\{t\}the groups in which both parts fail, andΦ\\Phithe decisions of Section[4\.3](https://arxiv.org/html/2610.22740#S4.SS3); of theG=nh\+nvG=n\_\{h\}\+n\_\{v\}rollouts, rolloutiireadsKiK\_\{i\}, which isKx∖kxK\_\{x\}\\setminus k\_\{x\}in the hidden partℋx\\mathcal\{H\}\_\{x\}\(nh\>nvn\_\{h\}\>n\_\{v\}\) andKxK\_\{x\}in the visible part𝒱x\\mathcal\{V\}\_\{x\}\. Because𝒮t\\mathcal\{S\}\_\{t\}constrains bothqxq\_\{x\}and the context on whichRPR\_\{P\}is computed, each outer step rewrites the objectives of the inner loop\.

Director update\.The randomized context is also how the inner loop prepares each withdrawal: becauseqxq\_\{x\}puts most of its mass onKx∖kxK\_\{x\}\\setminus k\_\{x\}, most of each update already trains the Director without the candidate skill\. The Director uses the group\-relative estimator of GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.22740#bib.bib32);[Liu et al\., 2025](https://arxiv.org/html/2610.22740#bib.bib21)\)with this adaptation; with centered advantages, it is a leave\-one\-out gradient\([Ahmadian et al\., 2024](https://arxiv.org/html/2610.22740#bib.bib1)\)of the first objective in Eq\. \([3](https://arxiv.org/html/2610.22740#S4.E3)\):

∇θ𝔼K∼qx\[vK\(x;θ\)\]=𝔼\[1G−1∑i=1G\(ui−u¯\)\\displaystyle\\nabla\_\{\\theta\}\\,\\mathbb\{E\}\_\{K\\sim q\_\{x\}\}\\big\[v\_\{K\}\(x;\\theta\)\\big\]=\\mathbb\{E\}\\Big\[\\tfrac\{1\}\{G\-1\}\\textstyle\\sum\_\{i=1\}^\{G\}\\big\(u\_\{i\}\-\\bar\{u\}\\big\)\(5\)⋅∇θlogπθ\(ξi∣x,Ki\)\],\\displaystyle\\cdot\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\\big\(\\xi\_\{i\}\\mid x,\\,K\_\{i\}\\big\)\\Big\],whereξi\\xi\_\{i\}is the edit trajectory of rolloutii\(Definition 2\) andui−u¯=\(1−1G\)​\(ui−u¯−i\)u\_\{i\}\-\\bar\{u\}=\(1\-\\tfrac\{1\}\{G\}\)\(u\_\{i\}\-\\bar\{u\}\_\{\-i\}\)is the leave\-one\-out baseline of rolloutii\.

What each loop contributes\.To see how the two loops share the work, letJ𝒮​\(θ\)=𝔼x∼ρ​\[pKx​\(x\)\]J\_\{\\mathcal\{S\}\}\(\\theta\)=\\mathbb\{E\}\_\{x\\sim\\rho\}\[p\_\{K\_\{x\}\}\(x\)\]be the success rate under scaffold𝒮\\mathcal\{S\}onρ\\rho, withpK​\(x\)p\_\{K\}\(x\)the success probability under contextKK\. Over the rounds it telescopes into inner gainsΓt=J𝒮t​\(θt\+1\)−J𝒮t​\(θt\)\\Gamma\_\{t\}=J\_\{\\mathcal\{S\}\_\{t\}\}\(\\theta\_\{t\+1\}\)\-J\_\{\\mathcal\{S\}\_\{t\}\}\(\\theta\_\{t\}\)and outer costsCt=J𝒮t​\(θt\+1\)−J𝒮t\+1​\(θt\+1\)C\_\{t\}=J\_\{\\mathcal\{S\}\_\{t\}\}\(\\theta\_\{t\+1\}\)\-J\_\{\\mathcal\{S\}\_\{t\+1\}\}\(\\theta\_\{t\+1\}\):

J𝒮T​\(θT\)=J𝒮0​\(θ0\)\+∑t<T\(Γt−Ct\)\.J\_\{\\mathcal\{S\}\_\{T\}\}\(\\theta\_\{T\}\)=J\_\{\\mathcal\{S\}\_\{0\}\}\(\\theta\_\{0\}\)\+\\textstyle\\sum\_\{t<T\}\\big\(\\Gamma\_\{t\}\-C\_\{t\}\\big\)\.\(6\)HereΓt\\Gamma\_\{t\}is what self\-play adds on a fixed scaffold, andCtC\_\{t\}is what the scaffold change takes away: for one withdrawn skillkk, its benefitpKx−pKx∖kp\_\{K\_\{x\}\}\-p\_\{K\_\{x\}\\setminus k\}weighted by how oftenkkis retrieved\. With𝒮T=∅\\mathcal\{S\}\_\{T\}=\\varnothingthe left side is the zero\-skill objective, and the other two components each serve one term: Recursive Learnability drivesΓt\\Gamma\_\{t\}\(Section[4\.2](https://arxiv.org/html/2610.22740#S4.SS2)\), and Skill Withdrawal limits the withdrawal cost in eachCtC\_\{t\}\(Section[4\.3](https://arxiv.org/html/2610.22740#S4.SS3)\)\.

Proposition 1\.*Zero\-skill success is the starting success plus inner gains less outer costs, and self\-play tracks each scaffold’s equilibrium\.**Proof\.*Appendix[A](https://arxiv.org/html/2610.22740#A1), via Lemmas[A\.5](https://arxiv.org/html/2610.22740#Thmlemma5)and[A\.6](https://arxiv.org/html/2610.22740#Thmlemma6), following the two\-time\-scale analysis of[Borkar \(1997\)](https://arxiv.org/html/2610.22740#bib.bib4)\.

### 4\.2Recursive Learnability

Recursive Learnability is the downward channel, through which the outer loop sets the inner loop’s objective: it rewards the Proposer for tasks whose rollouts keep reaching different outcomes with the candidate skill withheld \(Figure[3](https://arxiv.org/html/2610.22740#S2.F3), bottom left\)\.

Tilt and its log\-partition\.How much a task can raise its success follows from the proximal step itself\. Following relative\-entropy policy search\([Peters et al\., 2010](https://arxiv.org/html/2610.22740#bib.bib24)\)and KL\-regularized RL as inference\([Korbak et al\., 2022](https://arxiv.org/html/2610.22740#bib.bib15)\), a step of sizeβ\\betatilts the Director toward success, and its log\-partition is the outcome’s cumulant generating function:

πβ​\(g∣x,K\)=πθ​\(g∣x,K\)​eβ​r​\(g,x\)/ZK,log⁡ZK=β​pK\+β22​pK​\(1−pK\)\+O⁡\(β3\)\.\\begin\{gathered\}\\pi\_\{\\beta\}\(g\\mid x,K\)=\\pi\_\{\\theta\}\(g\\mid x,K\)\\,e^\{\\beta r\(g,x\)\}/Z\_\{K\},\\\\ \\log Z\_\{K\}=\\beta p\_\{K\}\+\\tfrac\{\\beta^\{2\}\}\{2\}p\_\{K\}\(1\-p\_\{K\}\)\+O\(\\beta^\{3\}\)\.\\end\{gathered\}\(7\)HereZK​\(x,β\)=1\+pK​\(eβ−1\)Z\_\{K\}\(x;\\beta\)=1\+p\_\{K\}\(e^\{\\beta\}\-1\)withpK=pK​\(x\)p\_\{K\}=p\_\{K\}\(x\), so the slope and curvature oflog⁡ZK\\log Z\_\{K\}atβ=0\\beta=0are the success probability and the outcome variance\. A step gainsβ​pK​\(1−pK\)\\beta\\,p\_\{K\}\(1\-p\_\{K\}\)in success at a KL cost ofβ2​pK​\(1−pK\)/2\\beta^\{2\}p\_\{K\}\(1\-p\_\{K\}\)/2to leading order, so within a KL budget the gain grows with the curvature\.

Recursive target\.On the current Director, this curvature is the learnability of the task\([Rutherford et al\., 2024](https://arxiv.org/html/2610.22740#bib.bib29)\)\. The progress that counts, however, is what survives withdrawal \(Eqs\. \([5](https://arxiv.org/html/2610.22740#S4.E5)\) and \([6](https://arxiv.org/html/2610.22740#S4.E6)\)\), so Recursive Learnability takes the curvature on the context with the candidate skill withheld:

ℓrec​\(x\)≔4​∂β2log⁡ZKx∖kx\|β=0=4​p−​\(1−p−\)=4G−1​𝔼​\[∑i\(yi−−y¯−\)2\],\\begin\{gathered\}\\ell\_\{\\mathrm\{rec\}\}\(x\)\\coloneqq 4\\,\\partial^\{2\}\_\{\\beta\}\\log Z\_\{K\_\{x\}\\setminus k\_\{x\}\}\\big\|\_\{\\beta=0\}\\\\ =4\\,p^\{\-\}\\big\(1\-p^\{\-\}\\big\)=\\tfrac\{4\}\{G\-1\}\\,\\mathbb\{E\}\\big\[\\textstyle\\sum\_\{i\}\(y^\{\-\}\_\{i\}\-\\bar\{y\}^\{\-\}\)^\{2\}\\big\],\\end\{gathered\}\(8\)a learnability proportional to the expected advantage energy of the group, wherep−=pKx∖kx​\(x\)p^\{\-\}=p\_\{K\_\{x\}\\setminus k\_\{x\}\}\(x\),yi−y^\{\-\}\_\{i\}are the outcomes ofGGrollouts that readKx∖kxK\_\{x\}\\setminus k\_\{x\}with meany¯−\\bar\{y\}^\{\-\}, andℓ⁡\(p\)=4​p​\(1−p\)\\ell\(p\)=4p\(1\-p\)\. The target vanishes exactly whenp−∈\{0,1\}p^\{\-\}\\in\\\{0,1\\\}, where every group collapses, and peaks at one half; since each withdrawal removes one more skill fromKx∖kxK\_\{x\}\\setminus k\_\{x\}, it follows the scaffold down to the zero\-skill Director\.

Proposer reward\.The hidden part of each group already readsKx∖kxK\_\{x\}\\setminus k\_\{x\}, so it measuresp−p^\{\-\}at no extra cost\. The reward is an empirical\-Bayes plug\-in estimate ofℓrec\\ell\_\{\\mathrm\{rec\}\}on this part\([Gelman et al\., 1995](https://arxiv.org/html/2610.22740#bib.bib7)\), discounted byγ<1\\gamma<1when only the visible part progresses and penalized byλP\\lambda\_\{P\}on collapse:

RP​\(x\)=\(1−cx\)​γhx​ℓ​\(p~x\)−λP​cx,p~x=𝔼⁡\[p∣y𝒜x\]=∑i∈𝒜xyi\+m​p¯\|𝒜x\|\+m\.\\begin\{gathered\}R\_\{P\}\(x\)=\(1\-c\_\{x\}\)\\,\\gamma^\{h\_\{x\}\}\\,\\ell\(\\tilde\{p\}\_\{x\}\)\-\\lambda\_\{P\}\\,c\_\{x\},\\\\ \\tilde\{p\}\_\{x\}=\\mathbb\{E\}\\big\[p\\mid y\_\{\\mathcal\{A\}\_\{x\}\}\\big\]=\\frac\{\\sum\_\{i\\in\\mathcal\{A\}\_\{x\}\}y\_\{i\}\+m\\,\\bar\{p\}\}\{\|\\mathcal\{A\}\_\{x\}\|\+m\}\.\\end\{gathered\}\(9\)Herecx=𝟏\[y¯∈\{0,1\}\]c\_\{x\}=\\mathbf\{1\}\[\\bar\{y\}\\in\\\{0,1\\\}\]flags collapse,hx=𝟏\[y¯ℋ=0<y¯𝒱\]h\_\{x\}=\\mathbf\{1\}\[\\bar\{y\}\_\{\\mathcal\{H\}\}=0<\\bar\{y\}\_\{\\mathcal\{V\}\}\]switches𝒜x\\mathcal\{A\}\_\{x\}fromℋx\\mathcal\{H\}\_\{x\}to𝒱x\\mathcal\{V\}\_\{x\}, and the Beta prior shrinks toward the dataset’s running hidden\-part success ratep¯\\bar\{p\}with strengthmm\.

Proposer update\.These rewards become a curriculum through the Proposer\. As a KL\-regularized environment designer\([Dennis et al\., 2020](https://arxiv.org/html/2610.22740#bib.bib5)\), it maximizesJP\(ρ\)=𝔼x∼ρ\[RP\(x\)\]−τKL\(ρ∥ρ0\)J\_\{P\}\(\\rho\)=\\mathbb\{E\}\_\{x\\sim\\rho\}\[R\_\{P\}\(x\)\]\-\\tau\\,\\mathrm\{KL\}\(\\rho\\\|\\rho\_\{0\}\)over the candidate list𝒞t\\mathcal\{C\}\_\{t\}, whose optimum is the closed\-form Gibbs curriculum \(Lemma[A\.9](https://arxiv.org/html/2610.22740#Thmlemma9)\):

ρ⋆​\(x\)=ρ0​\(x\)​eR¯P​\(x\)/τ∑x′∈𝒞tρ0​\(x′\)​eR¯P​\(x′\)/τ\.\\rho^\{\\star\}\(x\)=\\frac\{\\rho\_\{0\}\(x\)\\,e^\{\\bar\{R\}\_\{P\}\(x\)/\\tau\}\}\{\\sum\_\{x^\{\\prime\}\\in\\mathcal\{C\}\_\{t\}\}\\rho\_\{0\}\(x^\{\\prime\}\)\\,e^\{\\bar\{R\}\_\{P\}\(x^\{\\prime\}\)/\\tau\}\}\.\(10\)WithR¯P​\(x\)\\bar\{R\}\_\{P\}\(x\)the expected reward of taskxx, the curriculum tilts the base modelρ0\\rho\_\{0\}toward tasks of high Recursive Learnability\. A pairwise update, with each pick as the other’s baseline\([Ahmadian et al\., 2024](https://arxiv.org/html/2610.22740#bib.bib1)\), moves the Proposer toward these tasks\.

Proposition 2\.*Under a small trust region, a task’s success gain grows with its Recursive Learnability\.**Proof\.*Appendix[A](https://arxiv.org/html/2610.22740#A1), via Lemmas[A\.1](https://arxiv.org/html/2610.22740#Thmlemma1)–[A\.3](https://arxiv.org/html/2610.22740#Thmlemma3)on tilt and energy \(graph\-determined outcomes\)\.

Table 1:Main results \(five\-run mean±\\pmstd, %\), with benchmarks grouped by task domain\. All methods share the frozen Workers, the training tasks, and the compute budget, each in its best configuration, and R2Orch is tested with zero skills\. Each test set has 128 items \(AIME 2026: 30\); bold marks the best score in each benchmark column\.
### 4\.3Skill Withdrawal

Skill Withdrawal is the upward channel, through which the inner loop supplies the outer loop’s evidence: it keeps eachCtC\_\{t\}small by withdrawing a skill once the same rollout groups, read as a controlled evaluation, show that the Director performs as well without it \(Figure[3](https://arxiv.org/html/2610.22740#S2.F3), bottom right\)\.

Estimand and estimator\.Because the hidden part is drawn at random, each rollout group is a randomized block\([Rubin, 1974](https://arxiv.org/html/2610.22740#bib.bib27);[Imbens and Rubin, 2015](https://arxiv.org/html/2610.22740#bib.bib11)\), and the difference of its part means is an unbiased \(Neyman\) estimator of the skill’s effect:

zx=1nv​∑i∈𝒱xyi−1nh​∑i∈ℋxyi,𝔼\[zx∣x,kx=k\]=pKx−pKx∖k≕Δk\(x\),\\begin\{gathered\}z\_\{x\}=\\tfrac\{1\}\{n\_\{v\}\}\\textstyle\\sum\_\{i\\in\\mathcal\{V\}\_\{x\}\}y\_\{i\}\-\\tfrac\{1\}\{n\_\{h\}\}\\sum\_\{i\\in\\mathcal\{H\}\_\{x\}\}y\_\{i\},\\\\ \\mathbb\{E\}\[z\_\{x\}\\mid x,k\_\{x\}\{=\}k\]=p\_\{K\_\{x\}\}\-p\_\{K\_\{x\}\\setminus k\}\\eqqcolon\\Delta\_\{k\}\(x\),\\end\{gathered\}\(11\)wherek=kxk=k\_\{x\}\. Averaging over the tasks that testkkgives its benefitΔk=𝔼⁡\[Δk​\(x\)∣kx=k\]\\Delta\_\{k\}=\\mathbb\{E\}\[\\Delta\_\{k\}\(x\)\\mid k\_\{x\}=k\], which estimates its cost per retrieval inCtC\_\{t\}\(Lemma[A\.5](https://arxiv.org/html/2610.22740#Thmlemma5)\)\. The success rate mixes it with the curriculum:

𝔼\[y¯∣x,kx=k\]=pKx∖k​\(x\)⏟curriculum\+nvG​Δk​\(x\)⏟skill\.\\mathbb\{E\}\[\\bar\{y\}\\mid x,k\_\{x\}=k\]=\\underbrace\{p\_\{K\_\{x\}\\setminus k\}\(x\)\}\_\{\\smash\{\\text\{curriculum\}\}\}\+\\underbrace\{\\tfrac\{n\_\{v\}\}\{G\}\\,\\Delta\_\{k\}\(x\)\}\_\{\\smash\{\\text\{skill\}\}\}\.\(12\)Recursive Learnability pulls the curriculum term toward one half, so a threshold ony¯\\bar\{y\}reads the curriculum rather than the skill, whilezxz\_\{x\}removes it\.

Posterior and decisions\.The contrasts accumulate into a Bayesian\-bootstrap posterior per skill\([Rubin, 1981](https://arxiv.org/html/2610.22740#bib.bib28)\), and a Thompson draw tests first the skill closest to withdrawal\([William and others, 1933](https://arxiv.org/html/2610.22740#bib.bib38)\):

Πk=Law\(∑x∈𝒟kwxzx\),w∼Dir\(𝟏nk\),kx=argmink∈KxΔ~k,Δ~k∼Π0k,\\begin\{gathered\}\\Pi\_\{k\}=\\mathrm\{Law\}\\big\(\\textstyle\\sum\_\{x\\in\\mathcal\{D\}\_\{k\}\}w\_\{x\}z\_\{x\}\\big\),\\ \\ w\\sim\\mathrm\{Dir\}\(\\mathbf\{1\}\_\{n\_\{k\}\}\),\\\\ k\_\{x\}=\\arg\\min\_\{k\\in K\_\{x\}\}\\ \\tilde\{\\Delta\}\_\{k\},\\qquad\\tilde\{\\Delta\}\_\{k\}\\sim\\Pi^\{0\}\_\{k\},\\end\{gathered\}\(13\)where𝒟k\\mathcal\{D\}\_\{k\}holds thenkn\_\{k\}records of skillkk, andΠk0\\Pi^\{0\}\_\{k\}adds one prior record so that untested skills are also drawn\. On this posterior, withdrawal and admission are Bayes decisions\([Berger, 1985](https://arxiv.org/html/2610.22740#bib.bib3)\)under lossΔ\+\\Delta^\{\+\}for withdrawing,ϵ\\epsilonfor keeping, and an asymmetric zero\-one loss for admitting:

𝒲t=\{k:nk≥nmin,𝔼Δ∼Πk\[Δ\+\]≤ϵ\},𝒩t=\{k:nk≥nmin,qk≤α\},\\begin\{gathered\}\\mathcal\{W\}\_\{t\}=\\big\\\{k:\\ n\_\{k\}\\geq n\_\{\\min\},\\ \\mathbb\{E\}\_\{\\Delta\\sim\\Pi\_\{k\}\}\[\\Delta^\{\+\}\]\\leq\\epsilon\\big\\\},\\\\ \\mathcal\{N\}\_\{t\}=\\big\\\{k:\\ n\_\{k\}\\geq n\_\{\\min\},\\ q\_\{k\}\\leq\\alpha\\big\\\},\\end\{gathered\}\(14\)withΔ\+=max⁡\(Δ,0\)\\Delta^\{\+\}=\\max\(\\Delta,0\),qk=Πk​\(Δ≤0\)q\_\{k\}=\\Pi\_\{k\}\(\\Delta\\leq 0\),nminn\_\{\\min\}the minimum record, andα\\alphathe admission risk, so each withdrawal costs at mostϵ\\epsilonin posterior expectation on the tasks on which its skill was tested\.

Failures to skills\.The same posterior closes the outer loop: the least helpful skill is revised first, and new skills are distilled from groups where both parts fail \(Appendix[C](https://arxiv.org/html/2610.22740#A3)\), so failures exposed by withdrawals become the next skills to internalize\.

Proposition 3\.*Random hiding makes the paired contrast unbiased for a skill’s benefit\.**Proof\.*Appendix[A](https://arxiv.org/html/2610.22740#A1), via randomization\([Rubin, 1974](https://arxiv.org/html/2610.22740#bib.bib27)\)\.

## 5Experiments

We evaluate R2Orch through the following research questions \(RQs\)\.RQ1:How does R2Orch compare with the baselines in distribution and on held\-out benchmarks?RQ2:Does the trained Director transfer to other Worker backbones without retraining?RQ3:Do both loops of Recursive Self\-Play Orchestration contribute, and does the Director keep its accuracy at zero skills \(Section[4\.1](https://arxiv.org/html/2610.22740#S4.SS1)\)?RQ4:Does Recursive Learnability choose the tasks that keep yielding a learning signal as skills are withdrawn \(Section[4\.2](https://arxiv.org/html/2610.22740#S4.SS2)\)?RQ5:Does Skill Withdrawal remove each skill only once the Director no longer needs it \(Section[4\.3](https://arxiv.org/html/2610.22740#S4.SS3)\)?

Table 2:Ablation of R2Orch, tested with zero skills \(five\-run mean, %\)\. The first row of each group removes the whole component and the other rows change one part \(Appendix[D](https://arxiv.org/html/2610.22740#A4)\); free\-form QA benchmarks report Ans EM, and the Qwen3\.5\-9B and R2Orch rows repeat their scores in Table[1](https://arxiv.org/html/2610.22740#S4.T1)\.\(a\) Held\-out accuracy\(b\) Graph diversity\(c\) Policy mass\(d\) Solved candidates

\(e\) Backbone transfer: IID \(radars\) and OOD \(bars\)

Figure 4:Training dynamics and backbone transfer\. \(a\) Held\-out accuracy with skills and at zero skills\. \(b\) Graph types over the six IID datasets per Proposer objective \(final counts at the curve ends\)\. \(c\) The Proposer’s probability mass on unseen, unsolved \(p=0p=0\), learnable \(0<p<10<p<1\), and solved \(p=1p=1\) candidates by previous success ratepp\(power\-scaled axis\)\. \(d\) Probability of choosing a solved candidate as a moving average over 10 cycles \(dots: single cycles\) for the trained Proposer, the untrained model with the same prompt, and random choice\. \(e\) Backbone transfer: the Director trained with Qwen3\.5\-9B Workers is used unchanged at zero skills with six other backbones\. Radars: IID scores with R2Orch \(solid\) and without it \(dashed\), the radius linear over 20–80 \(inner half\) and 80–100 \(outer half\)\. Bars: OOD scores per task domain, the frozen backbone in blue and R2Orch’s gain in pink \(Code: SWE\-Bench; Interactive: WebShop; QA: mean of TriviaQA, MuSiQue, and GPQA; Math: MATH\-Hard\)\.\(a\) IID scores per Proposer objective

\(b\) Pairwise OOD differences

\(c\) Per effective group\(d\) Per cycle\(e\) Evidence and outcome\(f\) Skill withdrawals

Figure 5:Proposer objectives, selection cost, and skill withdrawals \(RQ4, RQ5\); only the Proposer’s objective changes in \(a\)–\(d\) \(AZR: Absolute Zero\)\. \(a\) IID scores\. \(b\) Row minus column of the mean OOD score\. \(c\) Rollouts and tokens per effective group; bubble area: A100 hours\. \(d\) Training \(pink\) and selection \(blue\) rollouts per cycle\. \(e\) Seen\-task picks over cycles 50–199 by previous success ratepp: realized success and share of mixed new groups\. \(f\) The seed cards’ held\-out gain over no cards, before training and around each withdrawal \(95% CI\)\.Setup\.Six in\-distribution \(IID\) benchmarks supply the training tasks and IID tests, and six out\-of\-distribution \(OOD\) benchmarks are held out\. The baselines span direct prompting, backbone training, workflow search, RL\-trained orchestration, skill internalization, and self\-play with skill evolution, under the shared protocol of Table[1](https://arxiv.org/html/2610.22740#S4.T1)\(Appendix[D](https://arxiv.org/html/2610.22740#A4)\)\.

Main results \(RQ1\)\.R2Orch is best on all twelve benchmarks against six families of baselines, and its lead holds on the held\-out half at zero skills \(Table[1](https://arxiv.org/html/2610.22740#S4.T1)\)\. Training the backbone alone changes little, while orchestration lifts the frozen Workers far higher, so the gains come from how the Workers are organized\. The other orchestrators learn from signals that stop tracking the Director: fixed task pools \(workflow search, RL\-trained orchestration\), success rates or budgets \(skill internalization\), and frontiers measured with skills in context \(self\-play\)\. R2Orch replaces these signals with Recursive Learnability and Skill Withdrawal, so its lead is largest on AIME 2026, WebShop, and ALFWorld, where the ablations cost most \(Table[2](https://arxiv.org/html/2610.22740#S5.T2)\)\.

Backbone transfer \(RQ2\)\.The Director trained with Qwen3\.5\-9B Workers improves all six other backbones on every benchmark without retraining \(Figure[4](https://arxiv.org/html/2610.22740#S5.F4)e\), since the roles and hand\-offs it builds fit any Worker\. Weaker backbones gain the most, narrowing the IID range across backbones from 25\.7 to 6\.4 points, because the graph supplies the planning and checking they lack\. On OOD benchmarks, the mean gain is largest on interactive tasks, whose many steps compound each hand\-off, and smallest on math, which is already near its ceiling\.

Recursive Self\-Play Orchestration \(RQ3\)\.Both loops are needed \(Table[2](https://arxiv.org/html/2610.22740#S5.T2)\): skills give the Director the early successes that make hard tasks learnable, and the Proposer keeps groups away from uniform outcomes\. Since most of each group hides the candidate skill \(Eq\. \([4](https://arxiv.org/html/2610.22740#S4.E4)\)\), updates teach the Director to succeed without it, and zero\-skill accuracy overtakes the accuracy with skills midway \(Figure[4](https://arxiv.org/html/2610.22740#S5.F4)a\)\. A skill has thus stopped helping before it leaves, keeping the outer costs of Eq\. \([6](https://arxiv.org/html/2610.22740#S4.E6)\) small as the inner gains of self\-play accumulate \(Proposition 1\)\.

Recursive Learnability \(RQ4\)\.Recursive Learnability picks the tasks that Proposition 2 expects to gain most, those whose rollouts still disagree with the candidate skill withheld\. New groups on tasks of intermediate success rate reach mixed outcomes, and thus a nonzero GRPO signal, about four times as often as atp=0p=0orp=1p=1\(Figure[5](https://arxiv.org/html/2610.22740#S5.F5)e\), and the trained Proposer learns to skip solved candidates \(Figure[4](https://arxiv.org/html/2610.22740#S5.F4)c,d\)\. Learnability measured with the skill in context picks tasks that collapse once the skill leaves, so it trails on every IID benchmark and the mean OOD score, as does every other objective \(Table[2](https://arxiv.org/html/2610.22740#S5.T2), Figure[5](https://arxiv.org/html/2610.22740#S5.F5)a,b\)\. Since the hidden part already measures this score, selection costs no extra rollouts and the fewest per effective group \(Figure[5](https://arxiv.org/html/2610.22740#S5.F5)c,d\), and these groups lead the Director to build the most distinct graph types \(Figure[4](https://arxiv.org/html/2610.22740#S5.F4)b\)\.

Skill Withdrawal \(RQ5\)\.Skill Withdrawal removes a skill once it no longer helps the current Director, as it reads the skill’s own effect, not the success rate\. The success rate mixes that effect with the curriculum that Recursive Learnability shapes \(Eq\. \([12](https://arxiv.org/html/2610.22740#S4.E12)\)\), so a success\-rate threshold misreads the skill and trails the paired contrast, as do a preset budget and a validation gap \(Proposition 3, Table[2](https://arxiv.org/html/2610.22740#S5.T2)\)\. All 12 withdrawals pass Eq\. \([14](https://arxiv.org/html/2610.22740#S4.E14)\) on the card’s own contrasts, with posterior𝔼⁡\[Δ\+\]\\mathbb\{E\}\[\\Delta^\{\+\}\]at most 0\.019, and around each the seed cards no longer raise held\-out accuracy \(Figure[5](https://arxiv.org/html/2610.22740#S5.F5)f\)\. A fixed scaffold costs most at zero skills, as the Director keeps leaning on skills that never leave its context throughout training\.

## 6Conclusion

R2Orch nests two loops: Recursive Learnability picks tasks learnable without the candidate skill, and Skill Withdrawal retires skills that stop helping\. It leads on all twelve benchmarks at zero skills\.

## Limitations

R2Orch starts from initial skills that we customize by hand for each dataset, and the withdrawals, revisions, and distillations that follow build on them, so deriving the initial skills automatically for a new domain remains an open problem\.

## Ethical considerations

R2Orch is trained and evaluated on public benchmarks that we use for research evaluation as their creators intend, and the study involves no human participants, annotators, or personal data\. ALFWorld and WebShop are simulated environments, and the code benchmarks run in their standard evaluation harnesses\. Agents that orchestrate tools can be misused like any agent system, and a self\-improving orchestrator shares this risk, so we release the code for inspection and recommend running it in a sandboxed environment\.

## Acknowledgements

We used AI assistants to polish the language and to help withLaTeXtypesetting; all other work, including the experiments and the first drafts, is the authors’ own, and the authors take full responsibility for the content of the paper\.

## References

- Ahmadianet al\.\(2024\)A\. Ahmadian, C\. Cremer, M\. Gallé, M\. Fadaee, J\. Kreutzer, O\. Pietquin, A\. Üstün, and S\. HookerBack to basics: revisiting reinforce\-style optimization for learning from human feedback in llms\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 12248–12267\.Cited by:[§4\.1](https://arxiv.org/html/2610.22740#S4.SS1.p3.2),[§4\.2](https://arxiv.org/html/2610.22740#S4.SS2.p5.2)\.
- Austinet al\.\(2021\)J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le,et al\.Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px1.p1.1)\.
- Berger \(1985\)J\. O\. BergerStatistical decision theory and bayesian analysis\.Springer New York\.External Links:ISBN 9781475742862,ISSN 0172\-7397,[Link](http://dx.doi.org/10.1007/978-1-4757-4286-2),[Document](https://dx.doi.org/10.1007/978-1-4757-4286-2)Cited by:[§4\.3](https://arxiv.org/html/2610.22740#S4.SS3.p3.2)\.
- Borkar \(1997\)V\. S\. BorkarStochastic approximation with two time scales\.Systems & Control Letters29\(5\),pp\. 291–294\.Cited by:[§4\.1](https://arxiv.org/html/2610.22740#S4.SS1.p2.1),[§4\.1](https://arxiv.org/html/2610.22740#S4.SS1.p5.1)\.
- Denniset al\.\(2020\)M\. Dennis, N\. Jaques, E\. Vinitsky, A\. Bayen, S\. Russell, A\. Critch, and S\. LevineEmergent complexity and zero\-shot transfer via unsupervised environment design\.Advances in neural information processing systems33,pp\. 13049–13061\.Cited by:[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2610.22740#S4.SS2.p5.1)\.
- Fuet al\.\(2026\)Z\. Fu, Z\. Li, Q\. Ai, H\. Wu, M\. Wu, C\. Zhao, A\. Wang, G\. He, and C\. WangSelf\-play meets skill evolution: self\-evolving search agents that pose, solve, and remember\.arXiv preprint arXiv:2607\.29468\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.22740#S1.p3.1),[§1](https://arxiv.org/html/2610.22740#S1.p4.1),[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px1.p1.1)\.
- Gelmanet al\.\(1995\)A\. Gelman, J\. B\. Carlin, H\. S\. Stern, and D\. B\. RubinBayesian data analysis\.chapman and hall/cRc\.Cited by:[§4\.2](https://arxiv.org/html/2610.22740#S4.SS2.p4.1)\.
- Heet al\.\(2026\)X\. He, Q\. Sun, A\. Cheng, X\. Li, X\. Ji, H\. Lu, R\. Huang, and Q\. HuAdvantage collapse in group relative policy optimization: diagnosis and mitigation\.arXiv preprint arXiv:2605\.21125\.Cited by:[§1](https://arxiv.org/html/2610.22740#S1.p4.1),[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px1.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the math dataset\.arXiv preprint arXiv:2103\.03874\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px1.p1.1)\.
- Huanget al\.\(2026\)C\. Huang, W\. Yu, X\. Wang, H\. Zhang, Z\. Li, R\. Li, J\. Huang, H\. Mi, and D\. YuR\-zero: self\-evolving reasoning llm from zero data\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 130770–130790\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px5.p1.1),[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px1.p1.1)\.
- Imbens and Rubin \(2015\)G\. W\. Imbens and D\. B\. RubinCausal inference in statistics, social, and biomedical sciences\.New York517\.Cited by:[§4\.3](https://arxiv.org/html/2610.22740#S4.SS3.p2.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. NarasimhanSwe\-bench: can language models resolve real\-world github issues?\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 54107–54157\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px1.p1.1)\.
- Jinet al\.\(2021\)D\. Jin, E\. Pan, N\. Oufattole, W\. Weng, H\. Fang, and P\. SzolovitsWhat disease does this patient have? a large\-scale open domain question answering dataset from medical exams\.Applied Sciences11\(14\),pp\. 6421\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px1.p1.1)\.
- Joshiet al\.\(2017\)M\. Joshi, E\. Choi, D\. S\. Weld, and L\. ZettlemoyerTriviaqa: a large scale distantly supervised challenge dataset for reading comprehension\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1601–1611\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px1.p1.1)\.
- Korbaket al\.\(2022\)T\. Korbak, E\. Perez, and C\. BuckleyRL with kl penalties is better viewed as bayesian inference\.InFindings of the Association for Computational Linguistics: EMNLP 2022,pp\. 1083–1091\.Cited by:[§4\.2](https://arxiv.org/html/2610.22740#S4.SS2.p2.1)\.
- Kwiatkowskiet al\.\(2019\)T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee,et al\.Natural questions: a benchmark for question answering research\.Transactions of the Association for Computational Linguistics7,pp\. 453–466\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px1.p1.1)\.
- Leeet al\.\(2019\)K\. Lee, M\. Chang, and K\. ToutanovaLatent retrieval for weakly supervised open domain question answering\.InProceedings of the 57th annual meeting of the association for computational linguistics,pp\. 6086–6096\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2026\)Z\. Li, H\. Zhang, S\. Han, S\. Liu, J\. Xie, Y\. Zhang, Y\. Choi, J\. Y\. Zou, and P\. LuIn\-the\-flow agentic system optimization for effective planning and tool use\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 50524–50570\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.22740#S1.p1.1),[§1](https://arxiv.org/html/2610.22740#S1.p2.1),[§1](https://arxiv.org/html/2610.22740#S1.p3.1),[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px1.p1.1)\.
- Linet al\.\(2026\)H\. Lin, Z\. Kuai, E\. Xue, and L\. WangSKILLC: learning autonomous skill internalization in llm agents via contrastive credit assignment\.arXiv preprint arXiv:2605\.27899\.Cited by:[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2023\)J\. Liu, C\. S\. Xia, Y\. Wang, and L\. ZhangIs your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation\.Advances in neural information processing systems36,pp\. 21558–21572\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2025\)Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. LinUnderstanding r1\-zero\-like training: a critical perspective\.arXiv preprint arXiv:2503\.20783\.Cited by:[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.22740#S4.SS1.p3.2)\.
- Luet al\.\(2026a\)S\. Lu, C\. Li, K\. Luo, Z\. Zhang, H\. Wang, H\. Xiao, L\. Xiong, J\. Wang, S\. Wang, X\. Jiang,et al\.Arex: towards a recursively self\-improving agent for deep research\.arXiv preprint arXiv:2607\.21461\.Cited by:[§1](https://arxiv.org/html/2610.22740#S1.p3.1),[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px2.p1.1)\.
- Luet al\.\(2026b\)Z\. Lu, Z\. Yao, J\. Wu, C\. Han, Q\. Gu, X\. Cai, W\. Lu, J\. Xiao, Y\. Zhuang, and Y\. ShenSkill0: in\-context agentic reinforcement learning for skill internalization\.arXiv preprint arXiv:2604\.02268\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.22740#S1.p3.1),[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px2.p1.1)\.
- Peterset al\.\(2010\)J\. Peters, K\. Mulling, and Y\. AltunRelative entropy policy search\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.24,pp\. 1607–1612\.Cited by:[§4\.2](https://arxiv.org/html/2610.22740#S4.SS2.p2.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px3.p1.1)\.
- Reinet al\.\(2023\)D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. BowmanGpqa: a graduate\-level google\-proof q&a benchmark\.arXiv preprint arXiv:2311\.12022\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px1.p1.1)\.
- Rubin \(1974\)D\. B\. RubinEstimating causal effects of treatments in randomized and nonrandomized studies\.\.Journal of educational Psychology66\(5\),pp\. 688\.Cited by:[§4\.3](https://arxiv.org/html/2610.22740#S4.SS3.p2.1),[§4\.3](https://arxiv.org/html/2610.22740#S4.SS3.p5.1)\.
- Rubin \(1981\)D\. B\. RubinThe bayesian bootstrap\.The annals of statistics,pp\. 130–134\.Cited by:[§4\.3](https://arxiv.org/html/2610.22740#S4.SS3.p3.1)\.
- Rutherfordet al\.\(2024\)A\. Rutherford, M\. Beukman, T\. Willi, B\. Lacerda, N\. Hawes, and J\. FoersterNo regrets: investigating and improving regret approximations for curriculum discovery\.Advances in Neural Information Processing Systems37,pp\. 16071–16101\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px5.p1.1),[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2610.22740#S4.SS2.p3.1)\.
- Schmidhuber \(2007\)J\. SchmidhuberGödel machines: fully self\-referential optimal universal self\-improvers\.InArtificial general intelligence,pp\. 199–226\.Cited by:[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px2.p1.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§4\.1](https://arxiv.org/html/2610.22740#S4.SS1.p2.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.22740#S4.SS1.p3.2)\.
- Shawn \(2026\)Z\. ShawnPACE: anytime\-valid acceptance tests for self\-evolving agents\.arXiv preprint arXiv:2606\.08106\.Cited by:[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px2.p1.1)\.
- Shenet al\.\(2026\)J\. Shen, T\. Zhang, X\. Zhao, and H\. ChengDynamic skill lifecycle management for agentic reinforcement learning\.arXiv preprint arXiv:2605\.10923\.Cited by:[§1](https://arxiv.org/html/2610.22740#S1.p3.1),[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px2.p1.1)\.
- Shiet al\.\(2026\)Y\. Shi, Z\. Ma, Y\. Wang, Q\. Tan, Y\. Li, P\. Chen, and Z\. ZhuPATS: policy\-aware training scaffolding for agentic reinforcement learning\.arXiv preprint arXiv:2607\.21419\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.22740#S1.p3.1),[§1](https://arxiv.org/html/2610.22740#S1.p4.1),[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px2.p1.1)\.
- Shridharet al\.\(2020\)M\. Shridhar, X\. Yuan, M\. Côté, Y\. Bisk, A\. Trischler, and M\. HausknechtAlfworld: aligning text and embodied environments for interactive learning\.arXiv preprint arXiv:2010\.03768\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px1.p1.1)\.
- Srikanthet al\.\(2026\)D\. Srikanth, B\. Zhao, D\. Xu, Y\. Wu, and Z\. JiangRecursive self\-improvement of ai research agents\.arXiv preprint arXiv:2609\.26457\.Cited by:[§1](https://arxiv.org/html/2610.22740#S1.p3.1),[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px2.p1.1)\.
- Trivediet al\.\(2022\)H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal♫ Musique: multihop questions via single\-hop question composition\.Transactions of the Association for Computational Linguistics10,pp\. 539–554\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px1.p1.1)\.
- Tuet al\.\(2026\)S\. Tu, C\. Xu, Q\. Zhang, Y\. Zhang, X\. Lan, L\. Li, D\. Li, and D\. ZhaoDynamic dual\-granularity skill bank for agentic rl\.arXiv preprint arXiv:2603\.28716\.Cited by:[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2026\)Z\. Wang, M\. Yan, J\. Bi, S\. Yan, V\. Tresp, and Y\. MaMetaSkill\-evolve: recursive self\-improvement of llm agents via two\-timescale meta\-skill evolution\.arXiv preprint arXiv:2607\.05297\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px2.p1.1)\.
- Williamet al\.\(1933\)R\. T\. Williamet al\.On the likelihood that one unknown probability exceeds another in view of the evidence of two samples\.Biometrika25\(3/4\),pp\. 285–294\.Cited by:[§4\.3](https://arxiv.org/html/2610.22740#S4.SS3.p3.1)\.
- Yanget al\.\(2018\)Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. ManningHotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 2369–2380\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px1.p1.1)\.
- Yaoet al\.\(2022\)S\. Yao, H\. Chen, J\. Yang, and K\. NarasimhanWebshop: towards scalable real\-world web interaction with grounded language agents\.Advances in Neural Information Processing Systems35,pp\. 20744–20757\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px1.p1.1)\.
- Yuet al\.\(2026\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu,et al\.Dapo: an open\-source llm reinforcement learning system at scale\.Advances in Neural Information Processing Systems38,pp\. 113222–113244\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px5.p1.1),[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026a\)J\. Zhang, S\. Hu, C\. Lu, R\. Lange, and J\. CluneDarwin gödel machine: open\-ended evolution of self\-improving agents\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 104223–104294\.Cited by:[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2026b\)J\. Zhang, B\. Zhao, W\. Yang, J\. Foerster, J\. Clune, M\. Jiang, S\. Devlin, and T\. ShavrinaHyperagents\.arXiv preprint arXiv:2603\.19461\.Cited by:[§1](https://arxiv.org/html/2610.22740#S1.p3.1),[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2025\)J\. Zhang, J\. Xiang, Z\. Yu, F\. Teng, X\. Chen, J\. Chen, M\. Zhuge, X\. Cheng, S\. Hong, J\. Wang,et al\.Aflow: automating agentic workflow generation\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 34040–34077\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.22740#S1.p1.1),[§1](https://arxiv.org/html/2610.22740#S1.p2.1),[§1](https://arxiv.org/html/2610.22740#S1.p3.1),[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026c\)M\. Zhang, W\. Liu, T\. Shen, Q\. Lin, R\. Mao, E\. Cambria, X\. Tang, and H\. LuoFlowSteer: towards agents designing agentic workflows via reinforced progressive canvas editing\.arXiv preprint arXiv:2602\.01664\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.22740#S1.p1.1),[§1](https://arxiv.org/html/2610.22740#S1.p2.1),[§1](https://arxiv.org/html/2610.22740#S1.p3.1),[§1](https://arxiv.org/html/2610.22740#S1.p4.1),[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2610.22740#S3.p1.1)\.
- Zhanget al\.\(2026d\)M\. Zhang, T\. Shen, H\. Luo, W\. Liu, Z\. Xiao, E\. Cambria, and X\. TangSkillFlow: flow\-driven recursive skill evolution for agentic orchestration\.arXiv preprint arXiv:2605\.14089\.Cited by:[§1](https://arxiv.org/html/2610.22740#S1.p2.1),[§1](https://arxiv.org/html/2610.22740#S1.p3.1),[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px2.p1.1)\.
- \[50\]X\. Zhang, Y\. Cui, G\. Wang, Z\. Li, W\. Qiu, B\. Zhu, and P\. HeRatchet: how reliable must an llm judge be to retire a skill?, 2026d\.URL https://arxiv\. org/abs/2605\.22148\.Cited by:[§1](https://arxiv.org/html/2610.22740#S1.p4.1),[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2026e\)X\. Zhang, Y\. Cui, G\. Wang, Z\. Li, W\. Qiu, B\. Zhu, and P\. HeLibrary drift: diagnosing and fixing a silent failure mode in self\-evolving llm skill libraries\.arXiv preprint arXiv:2605\.19576\.Cited by:[§1](https://arxiv.org/html/2610.22740#S1.p4.1),[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px2.p1.1)\.
- Zhaoet al\.\(2026\)A\. Zhao, Y\. Wu, T\. Wu, Q\. Xu, Y\. Yue, M\. Lin, S\. Wang, Q\. Wu, Z\. Zheng, and G\. HuangAbsolute zero: reinforced self\-play reasoning with zero data\.Advances in Neural Information Processing Systems38,pp\. 105816–105879\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px5.p1.1)\.
- Zhenget al\.\(2026\)H\. Zheng, Y\. Zhou, B\. Bartoldson, B\. Kailkhura, F\. Lai, J\. Zhao, and B\. ChenAct only when it pays: efficient reinforcement learning for llm reasoning via selective rollouts\.Advances in Neural Information Processing Systems38,pp\. 124321–124346\.Cited by:[Appendix D](https://arxiv.org/html/2610.22740#A4.SS0.SSS0.Px5.p1.1)\.
- Zhuet al\.\(2026\)S\. Zhu, S\. Fan, X\. Wang, W\. Wu, K\. Zhou, and B\. HuangRSIAgent: autonomous exploration for recursive self\-improvement in new environments\.arXiv preprint arXiv:2609\.15364\.Cited by:[§1](https://arxiv.org/html/2610.22740#S1.p3.1),[§2](https://arxiv.org/html/2610.22740#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix ATheory and Proofs

This appendix analyzes the two channels of R2Orch: outcome diversity for the curriculum and randomized evidence for skill withdrawal\. We derive the policy\-space progress calculation, the rollout estimators, and the posterior decisions, then connect them through round accounting\. Proposition 1 rests on Lemmas[A\.4](https://arxiv.org/html/2610.22740#Thmlemma4)–[A\.6](https://arxiv.org/html/2610.22740#Thmlemma6), Proposition 2 on Lemmas[A\.1](https://arxiv.org/html/2610.22740#Thmlemma1)–[A\.3](https://arxiv.org/html/2610.22740#Thmlemma3)and[A\.9](https://arxiv.org/html/2610.22740#Thmlemma9), and Proposition 3 on Lemmas[A\.7](https://arxiv.org/html/2610.22740#Thmlemma7)and[A\.8](https://arxiv.org/html/2610.22740#Thmlemma8); the final subsection collects the three proofs, and Table[3](https://arxiv.org/html/2610.22740#A1.T3)the notation\.

Table 3:Notation of the main text\.### A\.1Conditional setup and policy likelihoods

Fix a taskxx, a scaffold, a retrieved contextKK, and a candidate skillk∈Kk\\in K\. WriteK−=K∖\{k\}K^\{\-\}=K\\setminus\\\{k\\\},p\+=pK​\(x,θ\)p^\{\+\}=p\_\{K\}\(x;\\theta\), andp−=pK−​\(x,θ\)p^\{\-\}=p\_\{K^\{\-\}\}\(x;\\theta\)\. The success probability averages both Director sampling and Worker execution\. The two parts have fixed positive sizesnh,nvn\_\{h\},n\_\{v\}, withG=nh\+nvG=n\_\{h\}\+n\_\{v\}\. During one group, the policy, candidate, contexts, and pre\-group statistics are held fixed\. Conditional on the assigned contexts, different rollouts use independent randomness\. Task rewards average candidate\-skill selection whenever several skills can be tested for the same task\. Gradients hold the task and context sampling laws fixed and assume the usual measurability and integrability conditions\.

For an execution that returns intermediate Worker feedback, letω\\omegadenote its complete record andhth\_\{t\}the history before Director actionata\_\{t\}\. Frozen Worker kernels are independent ofθ\\theta, so the likelihood score is

Sθ​\(ω\)=∇θ​log​Pθ​\(ω\)=∑t∇θ​log​πθ​\(at∣ht\)\.S\_\{\\theta\}\(\\omega\)=\\nabla\_\{\\theta\}\\log P\_\{\\theta\}\(\\omega\)=\\sum\_\{t\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid h\_\{t\}\)\.\(15\)Thus𝔼⁡\[Sθ\]=0\\mathbb\{E\}\[S\_\{\\theta\}\]=0and∇θ𝔼​\[u\]=𝔼⁡\[u​Sθ\]\\nabla\_\{\\theta\}\\mathbb\{E\}\[u\]=\\mathbb\{E\}\[uS\_\{\\theta\}\]for a reward with no direct parameter dependence\. Multiple editing histories may produce the same terminal graph; their individual action likelihoods are not the marginal likelihood of that graph\. The group\-gradient results use the record score in Eq\. \([15](https://arxiv.org/html/2610.22740#A1.E15)\)\. A terminal\-graph score gives the same value gradient when the conditional reward mean given that graph is independent ofθ\\theta\. Otherwise the complete record is required, for example when editing histories with different costs reach the same graph\.

The policy\-space tilt below considers a choiceggfollowed by a fixed execution kernel, so its stochastic extension leaves Worker randomness outside the Director’s control\. We use a fixed evaluation distributionρ\\rhofor round accounting\. The training curriculumρϕ\\rho\_\{\\phi\}and the distribution of tasks recorded for a particular skill need not equalρ\\rho\.

In Eq\. \([3](https://arxiv.org/html/2610.22740#S4.E3)\), the Director KL is understood as averaged over the fixed task and context distribution\. Its moving referenceπθt\\pi\_\{\\theta\_\{t\}\}defines an ideal proximal response; the base\-model anchor in the training loss is a separate regularizer\. The Proposer instead uses the fixed referenceρ0\\rho\_\{0\}in both its KL term and the trust region on its pair tilt \(Appendix[B](https://arxiv.org/html/2610.22740#A2)\), so its response is proximal toρ0\\rho\_\{0\}\.

### A\.2KL tilting and controllable progress

###### Lemma A\.1\(KL\-tilt progress\)\.

Letπ\\pibe a reference distribution andF⁡\(g\)F\(g\)a bounded, parameter\-independent score\. Forβ\>0\\beta\>0, the unique maximizer, on the support ofπ\\pi, of𝔼π′\[F\]−β−1KL\(π′∥π\)\\mathbb\{E\}\_\{\\pi^\{\\prime\}\}\[F\]\-\\beta^\{\-1\}\\mathrm\{KL\}\(\\pi^\{\\prime\}\\\|\\pi\)is

πβ​\(g\)=π⁡\(g\)​eβ​F​\(g\)Z⁡\(β\),Z⁡\(β\)=𝔼π​\[eβ​F\]\.\\pi\_\{\\beta\}\(g\)=\\frac\{\\pi\(g\)e^\{\\beta F\(g\)\}\}\{Z\(\\beta\)\},\\qquad Z\(\\beta\)=\\mathbb\{E\}\_\{\\pi\}\[e^\{\\beta F\}\]\.\(16\)Writingψ=log⁡Z\\psi=\\log Z,μ=𝔼π​F\\mu=\\mathbb\{E\}\_\{\\pi\}F, andV=Varπ​\(F\)V=\\mathrm\{Var\}\_\{\\pi\}\(F\)gives

ψ⁡\(β\)\\displaystyle\\psi\(\\beta\)=β​μ\+12​β2​V\+O⁡\(β3\),\\displaystyle=\\beta\\mu\+\\tfrac\{1\}\{2\}\\beta^\{2\}V\+O\(\\beta^\{3\}\),\(17\)𝔼πβ​F−μ\\displaystyle\\mathbb\{E\}\_\{\\pi\_\{\\beta\}\}F\-\\mu=β​V\+O⁡\(β2\),\\displaystyle=\\beta V\+O\(\\beta^\{2\}\),KL\(πβ∥π\)\\displaystyle\\mathrm\{KL\}\(\\pi\_\{\\beta\}\\\|\\pi\)=12​β2​V\+O⁡\(β3\)\.\\displaystyle=\\tfrac\{1\}\{2\}\\beta^\{2\}V\+O\(\\beta^\{3\}\)\.For fixedV\>0V\>0, a small KL budgetδ\\deltayields optimal score gain2​δ​V\+O⁡\(δ\)\\sqrt\{2\\delta V\}\+O\(\\delta\)\.

###### Proof\.

Substitution of the tilted density gives, for every admissibleπ′\\pi^\{\\prime\},

𝔼π′F−β−1KL\(π′∥π\)=β−1\{logZ−KL\(π′∥πβ\)\}\.\\mathbb\{E\}\_\{\\pi^\{\\prime\}\}F\-\\beta^\{\-1\}\\mathrm\{KL\}\(\\pi^\{\\prime\}\\\|\\pi\)=\\beta^\{\-1\}\\\{\\log Z\-\\mathrm\{KL\}\(\\pi^\{\\prime\}\\\|\\pi\_\{\\beta\}\)\\\}\.\(18\)Nonnegativity of KL proves optimality and uniqueness\. Boundedness permits differentiation ofZZ:ψ′​\(β\)=𝔼πβ​F\\psi^\{\\prime\}\(\\beta\)=\\mathbb\{E\}\_\{\\pi\_\{\\beta\}\}Fandψ′′​\(β\)=Varπβ​\(F\)\\psi^\{\\prime\\prime\}\(\\beta\)=\\mathrm\{Var\}\_\{\\pi\_\{\\beta\}\}\(F\)\. Expanding at zero proves the first two identities\. Since the log density ratio isβ​F−ψ⁡\(β\)\\beta F\-\\psi\(\\beta\), its expectation isβ​ψ′​\(β\)−ψ⁡\(β\)\\beta\\psi^\{\\prime\}\(\\beta\)\-\\psi\(\\beta\), proving the KL expansion\. ForV\>0V\>0, inversion givesβ=2​δ/V\+O⁡\(δ\)\\beta=\\sqrt\{2\\delta/V\}\+O\(\\delta\)\. The variational inequality also shows that this tilt maximizes expected score among distributions with its KL cost\. Substitution proves the budget statement\. IfV=0V=0, the score is constant on the available support and no such tilt improves it\. ∎

For a binary outcome determined bygg, takeF=rF=randp=𝔼π​rp=\\mathbb\{E\}\_\{\\pi\}r\. Then

Z⁡\(β\)\\displaystyle Z\(\\beta\)=1−p\+peβ,pβ=p​eβZ⁡\(β\),\\displaystyle=1\-p\+pe^\{\\beta\},\\qquad p\_\{\\beta\}=\\frac\{pe^\{\\beta\}\}\{Z\(\\beta\)\},\(19\)pβ−p\\displaystyle p\_\{\\beta\}\-p=p⁡\(1−p\)​\(eβ−1\)1\+p⁡\(eβ−1\)\.\\displaystyle=\\frac\{p\(1\-p\)\(e^\{\\beta\}\-1\)\}\{1\+p\(e^\{\\beta\}\-1\)\}\.Consequentlyψ′′​\(0\)=p⁡\(1−p\)\\psi^\{\\prime\\prime\}\(0\)=p\(1\-p\)is also the Fisher information of this one\-dimensional exponential family\. At a common small tilt, gains rank byp⁡\(1−p\)p\(1\-p\)to first order; at a common small KL budget, they scale with its square root\. At a finite tilt the ranking can change: atβ=log⁡2\\beta=\\log 2,p=0\.3p=0\.3yields gain0\.21/1\.30\.21/1\.3, whereasp=0\.6p=0\.6yields the smaller gain0\.24/1\.60\.24/1\.6despite its larger variance\. Gain per unit KL cost is2/β\+O⁡\(1\)2/\\beta\+O\(1\)\.

#### Stochastic Workers\.

LetQ⁡\(g\)=𝔼⁡\[Y∣g\]Q\(g\)=\\mathbb\{E\}\[Y\\mid g\]for a fixed Worker execution kernel andY∈\{0,1\}Y\\in\\\{0,1\\\}\. Optimizing expected correctness over graph distributions applies Lemma[A\.1](https://arxiv.org/html/2610.22740#Thmlemma1)withF=QF=Q, not with a sampledYY\. The controllable graph\-level variance is

Vctrl\\displaystyle V\_\{\\mathrm\{ctrl\}\}=Varπ​\(Q\),\\displaystyle=\\mathrm\{Var\}\_\{\\pi\}\(Q\),\(20\)p⁡\(1−p\)\\displaystyle p\(1\-p\)=Vctrl\+𝔼π​\[Q⁡\(1−Q\)\]\.\\displaystyle=V\_\{\\mathrm\{ctrl\}\}\+\\mathbb\{E\}\_\{\\pi\}\[Q\(1\-Q\)\]\.The second identity follows by conditioningY2=YY^\{2\}=Yongg, or by total variance\. A Director\-only update reweights graphs and keeps the Worker kernel, whereas a joint tilt of graph and execution outcomes byeβ​Ye^\{\\beta Y\}would change that kernel\. In particular, ifQ⁡\(g\)=1/2Q\(g\)=1/2for every graph, correctness variance is1/41/4but no graph redistribution improves success\. Recursive Learnability measures outcome diversity under stochastic execution, which equals its controllable part for graph\-determined outcomes, the setting of Proposition 2\.

A local statement also covers feedback\-dependent records and restricted policy classes\. Letb=𝔼⁡\[\(Y−p\)​Sθ\]b=\\mathbb\{E\}\[\(Y\-p\)S\_\{\\theta\}\]andI=𝔼⁡\[Sθ​Sθ⊤\]I=\\mathbb\{E\}\[S\_\{\\theta\}S\_\{\\theta\}^\{\\top\}\]\. Under a differentiable local KL expansion, maximizingb⊤​db^\{\\top\}dsubject tod⊤​I​d/2≤δd^\{\\top\}Id/2\\leq\\deltagives

2​δ​b⊤​I†​b,b⊤​I†​b≤p⁡\(1−p\)\.\\sqrt\{2\\delta\\,b^\{\\top\}I^\{\\dagger\}b\},\\qquad b^\{\\top\}I^\{\\dagger\}b\\leq p\(1\-p\)\.\(21\)Indeed, a null direction ofIIhas zero score almost surely and is orthogonal tobb\. Cauchy–Schwarz in the range ofIIgives the optimum\. MoreoverSθ⊤​I†​bS\_\{\\theta\}^\{\\top\}I^\{\\dagger\}bis the least\-squares projection ofY−pY\-ponto the score span, so its variance cannot exceedVar⁡\(Y\)\\mathrm\{Var\}\(Y\)\. This identifies the accessible part of outcome diversity for any differentiable policy class\.

For the cost\-adjusted rewardu=Y−λ​cu=Y\-\\lambda c, defineU⁡\(g\)=Q⁡\(g\)−λ​𝔼​\[c∣g\]U\(g\)=Q\(g\)\-\\lambda\\mathbb\{E\}\[c\\mid g\]\. Tilting byUUchanges correctness at the initial rateCovπ​\(Q,U\)=Varπ​\(Q\)−λ​Covπ​\(Q,c¯\)\\mathrm\{Cov\}\_\{\\pi\}\(Q,U\)=\\mathrm\{Var\}\_\{\\pi\}\(Q\)\-\\lambda\\,\\mathrm\{Cov\}\_\{\\pi\}\(Q,\\bar\{c\}\)withc¯​\(g\)=𝔼​\[c∣g\]\\bar\{c\}\(g\)=\\mathbb\{E\}\[c\\mid g\], which differs fromVarπ​\(Q\)\\mathrm\{Var\}\_\{\\pi\}\(Q\)by at mostλ​\(Varπ​\(Q\)​Varπ​\(c¯\)\)1/2\\lambda\\,\(\\mathrm\{Var\}\_\{\\pi\}\(Q\)\\,\\mathrm\{Var\}\_\{\\pi\}\(\\bar\{c\}\)\)^\{1/2\}, small for the weightλ\\lambdaof Table[4](https://arxiv.org/html/2610.22740#A3.T4)\.

###### Lemma A\.2\(Equal tilt\)\.

For finitely many tasks with fixed weightsaj\>0a\_\{j\}\>0and variancesVj\>0V\_\{j\}\>0, the quadratic\-budget problem

max⁡∑jβj≥0⁡aj​βj​Vjsubject to12​∑jaj​βj2​Vj≤δ\\max\_\{\\beta\_\{j\}\\geq 0\}\\sum\_\{j\}a\_\{j\}\\beta\_\{j\}V\_\{j\}\\quad\\text\{subject to\}\\quad\\tfrac\{1\}\{2\}\\sum\_\{j\}a\_\{j\}\\beta\_\{j\}^\{2\}V\_\{j\}\\leq\\delta\(22\)has the common solutionβj=2​δ/∑jaj​Vj\\beta\_\{j\}=\\sqrt\{2\\delta/\\sum\_\{j\}a\_\{j\}V\_\{j\}\}\.

###### Proof\.

Cauchy–Schwarz bounds the squared objective by\(∑jaj​βj2​Vj\)​\(∑jaj​Vj\)\(\\sum\_\{j\}a\_\{j\}\\beta\_\{j\}^\{2\}V\_\{j\}\)\(\\sum\_\{j\}a\_\{j\}V\_\{j\}\), with equality exactly when all positive\-variance tasks have the same tilt\. Saturating the budget gives the stated value\. Zero\-variance tasks add no gain or cost\. ∎

This allocation result for independently adjustable policy distributions motivates one advantage scale across tasks: the batch whitening of Appendix[B](https://arxiv.org/html/2610.22740#A2)rescales every task by the same factor, whereas dividing by each task’s standard deviation would give larger multipliers to low\-variance tasks\.

### A\.3Advantage energy, collapse, and shrinkage

###### Lemma A\.3\(Advantage energy\)\.

Letn≥2n\\geq 2independent outcomes have distributionBernoulli⁡\(p\)\\mathrm\{Bernoulli\}\(p\)and letAi=Yi−Y¯A\_\{i\}=Y\_\{i\}\-\\bar\{Y\}\. Then

𝔼​∑iAi2\\displaystyle\\mathbb\{E\}\\sum\_\{i\}A\_\{i\}^\{2\}=\(n−1\)​p​\(1−p\),\\displaystyle=\(n\-1\)p\(1\-p\),\(23\)Pr⁡\(∑iAi2=0\)\\displaystyle\\Pr\\\!\\left\(\\sum\_\{i\}A\_\{i\}^\{2\}=0\\right\)=pn\+\(1−p\)n\.\\displaystyle=p^\{n\}\+\(1\-p\)^\{n\}\.Thusℓ⁡\(p\)=4​p​\(1−p\)\\ell\(p\)=4p\(1\-p\)is4/\(n−1\)4/\(n\-1\)times the expected advantage energy, and a realized group collapses exactly when its outcomes agree\.

###### Proof\.

ForB=∑iYiB=\\sum\_\{i\}Y\_\{i\}, binary outcomes give∑iAi2=B−B2/n\\sum\_\{i\}A\_\{i\}^\{2\}=B\-B^\{2\}/n\. Since𝔼​B=n​p\\mathbb\{E\}B=npand𝔼​B2=n​p​\(1−p\)\+n2​p2\\mathbb\{E\}B^\{2\}=np\(1\-p\)\+n^\{2\}p^\{2\}, taking expectations proves the first identity\. The sum vanishes exactly atB∈\{0,n\}B\\in\\\{0,n\\\}, giving the second identity\. ∎

For the actual hidden part,n=nhn=n\_\{h\}, and the hypothetical homogeneous group in Eq\. \([8](https://arxiv.org/html/2610.22740#S4.E8)\) hasn=Gn=G\. Population learnability vanishes only atp∈\{0,1\}p\\in\\\{0,1\\\}, while a finite group can collapse at anypp: even atp=1/2p=1/2, with probability21−n2^\{1\-n\}\. Forn\>1n\>1, differentiating the collapse probability givesn⁡\[pn−1−\(1−p\)n−1\]n\[p^\{n\-1\}\-\(1\-p\)^\{n\-1\}\], which is negative below one half and positive above it\. Thus the same population frontier that maximizes learnability minimizes homogeneous\-group collapse probability\. The hidden part and the full group consequently have different finite\-sample collapse probabilities even when their success probabilities coincide\.

The mixed group has a different energy\. LetV−=p−​\(1−p−\)V\_\{\-\}=p^\{\-\}\(1\-p^\{\-\}\)andV\+=p\+​\(1−p\+\)V\_\{\+\}=p^\{\+\}\(1\-p^\{\+\}\)\. Conditional independence gives

𝔼​∑i\(Yi−Y¯\)2\\displaystyle\\mathbb\{E\}\\sum\_\{i\}\(Y\_\{i\}\-\\bar\{Y\}\)^\{2\}=\(1−1G\)​\(nh​V−\+nv​V\+\)\\displaystyle=\\left\(1\-\\tfrac\{1\}\{G\}\\right\)\(n\_\{h\}V\_\{\-\}\+n\_\{v\}V\_\{\+\}\)\(24\)\+nh​nvG​\(p\+−p−\)2\.\\displaystyle\+\\tfrac\{n\_\{h\}n\_\{v\}\}\{G\}\(p^\{\+\}\-p^\{\-\}\)^\{2\}\.To verify this, expand∑iYi2−G​Y¯2\\sum\_\{i\}Y\_\{i\}^\{2\}\-G\\bar\{Y\}^\{2\}and separate within\-context variances from the squared differences between context means\. Its collapse probability is\(p−\)nh​\(p\+\)nv\+\(1−p−\)nh​\(1−p\+\)nv\(p^\{\-\}\)^\{n\_\{h\}\}\(p^\{\+\}\)^\{n\_\{v\}\}\+\(1\-p^\{\-\}\)^\{n\_\{h\}\}\(1\-p^\{\+\}\)^\{n\_\{v\}\}\. The extra energy term measures dependence on the skill rather than post\-withdrawal learning potential, which is why Eq\. \([9](https://arxiv.org/html/2610.22740#S4.E9)\) reads the hidden part\. If advantages useuu, the same expansion concerns reward variance, and correctness agreement need not make them zero when costs differ\.

#### Meaning of the Beta reward\.

Condition on a pre\-group running meanp¯∈\(0,1\)\\bar\{p\}\\in\(0,1\)and strengthm\>0m\>0\. A proper working prior isp∼Beta⁡\(a,b\)p\\sim\\mathrm\{Beta\}\(a,b\)witha=m​p¯a=m\\bar\{p\},b=m⁡\(1−p¯\)b=m\(1\-\\bar\{p\}\)\. For a fixed context withsssuccesses innnindependent trials, seta′=a\+sa^\{\\prime\}=a\+s,b′=b\+n−sb^\{\\prime\}=b\+n\-s, andN=n\+mN=n\+m\. Multiplication of prior and Bernoulli likelihood gives the posteriorBeta⁡\(a′,b′\)\\mathrm\{Beta\}\(a^\{\\prime\},b^\{\\prime\}\), with

p~\\displaystyle\\tilde\{p\}=a′/N,Var\(p∣y\)=p~​\(1−p~\)N\+1,\\displaystyle=a^\{\\prime\}/N,\\qquad\\mathrm\{Var\}\(p\\mid y\)=\\frac\{\\tilde\{p\}\(1\-\\tilde\{p\}\)\}\{N\+1\},\(25\)𝔼⁡\[ℓ⁡\(p\)∣y\]\\displaystyle\\mathbb\{E\}\[\\ell\(p\)\\mid y\]=NN\+1​ℓ​\(p~\)\.\\displaystyle=\\frac\{N\}\{N\+1\}\\ell\(\\tilde\{p\}\)\.The last identity follows from𝔼⁡\[p⁡\(1−p\)\]=a′​b′/\[N⁡\(N\+1\)\]\\mathbb\{E\}\[p\(1\-p\)\]=a^\{\\prime\}b^\{\\prime\}/\[N\(N\+1\)\]\. Hence Eq\. \([9](https://arxiv.org/html/2610.22740#S4.E9)\) uses the posterior\-mean*plug\-in*, which exceeds the posterior expectation of learnability by4​Var​\(p∣y\)4\\mathrm\{Var\}\(p\\mid y\)\. A running mean estimated from data gives an empirical\-Bayes interpretation conditional on that mean\.

The plug\-in also has a precise frequentist interpretation when the running mean is fixed before sampling\. Letμn=\(n​p\+m​p¯\)/\(n\+m\)\\mu\_\{n\}=\(np\+m\\bar\{p\}\)/\(n\+m\)\. Since𝔼​p~=μn\\mathbb\{E\}\\tilde\{p\}=\\mu\_\{n\}andVar⁡\(p~\)=n​p​\(1−p\)/\(n\+m\)2\\mathrm\{Var\}\(\\tilde\{p\}\)=np\(1\-p\)/\(n\+m\)^\{2\}, expanding the quadratic learnability function gives

𝔼⁡\[ℓ⁡\(p~\)\]=ℓ⁡\(μn\)−4​n​p​\(1−p\)\(n\+m\)2\.\\mathbb\{E\}\[\\ell\(\\tilde\{p\}\)\]=\\ell\(\\mu\_\{n\}\)\-\\frac\{4np\(1\-p\)\}\{\(n\+m\)^\{2\}\}\.\(26\)There are therefore two separate effects: shrinkage changes the center, while finite sampling subtracts a variance term\. The identity holds before the outcome\-dependent branch selection below\.

For paired groups, the stated branching can be written as follows:cxc\_\{x\}indicates agreement of allGGoutcomes; on a noncollapsed group,hx=1h\_\{x\}=1when the hidden part has no success and the visible part does\. Use𝒜x=𝒱x\\mathcal\{A\}\_\{x\}=\\mathcal\{V\}\_\{x\}in this transfer branch and𝒜x=ℋx\\mathcal\{A\}\_\{x\}=\\mathcal\{H\}\_\{x\}otherwise\. A no\-skill group uses all its rollouts\. The visible branch estimates visible\-context successp\+p^\{\+\}and supplies a discounted transfer incentive\. The collapse penalty and the compute factor further shape the operational reward, whose expectationR¯P\\bar\{R\}\_\{P\}enters Eq\. \([10](https://arxiv.org/html/2610.22740#S4.E10)\)\.

There is nevertheless a direct large\-group connection in the ordinary hidden\-success branch\. Fix0<p−<10<p^\{\-\}<1, let both part sizes increase, and take fixedmmorm/nh→0m/n\_\{h\}\\to 0\. Consistency of the hidden sample mean givesp~x→p−\\tilde\{p\}\_\{x\}\\to p^\{\-\}in probability\. The hidden\-zero event has probability\(1−p−\)nh→0\(1\-p^\{\-\}\)^\{n\_\{h\}\}\\to 0\. Full\-group collapse is contained in the event that the hidden outcomes all agree, whose probability also tends to zero\. Therefore, with the compute factor omitted, the bounded reward in Eq\. \([9](https://arxiv.org/html/2610.22740#S4.E9)\) converges toℓ⁡\(p−\)\\ell\(p^\{\-\}\)in probability and in expectation\.

The exceptional branch still matters\. Atp−=0p^\{\-\}=0and0<p\+<10<p^\{\+\}<1, the hidden part keeps failing while the visible part includes a success with high probability\. With a consistent visible estimate, the reward approachesγ​ℓ​\(p\+\)\\gamma\\ell\(p^\{\+\}\), so the transfer term keeps a discounted incentive for assisted practice on tasks whose hidden\-context learnability is zero\.

### A\.4Mixed\-context gradients and the last skill

###### Lemma A\.4\(Mixed\-group gradient\)\.

Condition on an assignment of exactlynhn\_\{h\}hidden andnvn\_\{v\}visible contexts\. LetSiS\_\{i\}be each rollout’s likelihood score anduiu\_\{i\}its integrable reward\. With independent rollouts conditional on this assignment and a common group mean,

𝔼\[∑i\(ui−u¯\)​SiG−1\]=nhG∇vK−\+nvG∇vK\.\\mathbb\{E\}\\left\[\\frac\{\\sum\_\{i\}\(u\_\{i\}\-\\bar\{u\}\)S\_\{i\}\}\{G\-1\}\\right\]=\\frac\{n\_\{h\}\}\{G\}\\nabla v\_\{K^\{\-\}\}\+\\frac\{n\_\{v\}\}\{G\}\\nabla v\_\{K\}\.\(27\)If each part is centered separately, the expectation of the unnormalized sum becomes

\(nh−1\)∇vK−\+\(nv−1\)∇vK\.\(n\_\{h\}\-1\)\\nabla v\_\{K^\{\-\}\}\+\(n\_\{v\}\-1\)\\nabla v\_\{K\}\.\(28\)

###### Proof\.

Write𝐊\\mathbf\{K\}for the complete context assignment\. Forj≠ij\\neq i, conditional independence and the zero\-mean score identity imply𝔼⁡\[uj​Si∣𝐊\]=𝔼⁡\[uj∣𝐊\]​𝔼​\[Si∣𝐊\]=0\\mathbb\{E\}\[u\_\{j\}S\_\{i\}\\mid\\mathbf\{K\}\]=\\mathbb\{E\}\[u\_\{j\}\\mid\\mathbf\{K\}\]\\mathbb\{E\}\[S\_\{i\}\\mid\\mathbf\{K\}\]=0\. Forj=ij=i,𝔼⁡\[ui​Si∣𝐊\]=∇vKi\\mathbb\{E\}\[u\_\{i\}S\_\{i\}\\mid\\mathbf\{K\}\]=\\nabla v\_\{K\_\{i\}\}\. Therefore

𝔼\[\(ui−u¯\)Si∣𝐊\]=\(1−1G\)∇vKi\.\\mathbb\{E\}\[\(u\_\{i\}\-\\bar\{u\}\)S\_\{i\}\\mid\\mathbf\{K\}\]=\\left\(1\-\\tfrac\{1\}\{G\}\\right\)\\nabla v\_\{K\_\{i\}\}\.\(29\)Summing and dividing byG−1G\-1proves Eq\. \([27](https://arxiv.org/html/2610.22740#A1.E27)\)\. It also follows from the exact leave\-one\-out identityui−u¯=\(1−1/G\)​\(ui−u¯−i\)u\_\{i\}\-\\bar\{u\}=\(1\-1/G\)\(u\_\{i\}\-\\bar\{u\}\_\{\-i\}\)\. Repeating the calculation within a part of sizencn\_\{c\}gives a total\(nc−1\)∇vc\(n\_\{c\}\-1\)\\nabla v\_\{c\}, proving Eq\. \([28](https://arxiv.org/html/2610.22740#A1.E28)\)\. Averaging over assignments preserves both conclusions\. ∎

If the last\-skill caseK=\{k\}K=\\\{k\\\}uses separate centering, normalizing Eq\. \([28](https://arxiv.org/html/2610.22740#A1.E28)\) byG−2G\-2gives the mixture

qx′=nh−1G−2​δ∅\+nv−1G−2​δ\{k\}\.q^\{\\prime\}\_\{x\}=\\frac\{n\_\{h\}\-1\}\{G\-2\}\\delta\_\{\\varnothing\}\+\\frac\{n\_\{v\}\-1\}\{G\-2\}\\delta\_\{\\\{k\\\}\}\.\(30\)ForG\>2G\>2andnh\>nv≥1n\_\{h\}\>n\_\{v\}\\geq 1, its hidden weight exceedsnh/Gn\_\{h\}/Gby\(nh−nv\)/\[G⁡\(G−2\)\]\(n\_\{h\}\-n\_\{v\}\)/\[G\(G\-2\)\]\. Keeping the divisorG−1G\-1instead gives\(G−2\)/\(G−1\)\(G\-2\)/\(G\-1\)times this mixture gradient: the3/23/2split yields weights2:12:1and relative scale3/43/4\. A singleton part contributes no centered gradient\. Once no skill is retrieved, allGGrollouts share the empty context and ordinary whole\-group centering applies\.

These identities describe the on\-policy reward gradient; PPO clipping, base\-model anchoring, batch whitening, token averaging, and advantage\-based filtering turn it into the training surrogate of Appendix[B](https://arxiv.org/html/2610.22740#A2)on the same context mixture\.

### A\.5Round accounting and interacting withdrawals

###### Lemma A\.5\(Round decomposition\)\.

Fixρ\\rhoand defineJ𝒮​\(θ\)=𝔼x∼ρ​\[pKx​\(𝒮\)​\(x,θ\)\]J\_\{\\mathcal\{S\}\}\(\\theta\)=\\mathbb\{E\}\_\{x\\sim\\rho\}\[p\_\{K\_\{x\}\(\\mathcal\{S\}\)\}\(x;\\theta\)\]\. For any finite sequence of policies and scaffolds, defineΓt\\Gamma\_\{t\}andCtC\_\{t\}as in Eq\. \([6](https://arxiv.org/html/2610.22740#S4.E6)\)\. Then

J𝒮T​\(θT\)=J𝒮0​\(θ0\)\+∑t<T\(Γt−Ct\)\.J\_\{\\mathcal\{S\}\_\{T\}\}\(\\theta\_\{T\}\)=J\_\{\\mathcal\{S\}\_\{0\}\}\(\\theta\_\{0\}\)\+\\sum\_\{t<T\}\(\\Gamma\_\{t\}\-C\_\{t\}\)\.\(31\)If an outer change removes distinct skillsk1,…,kmk\_\{1\},\\ldots,k\_\{m\}without changing or refilling the remaining retrieved context, putKxj=Kx∖\{k1,…,kj\}K\_\{x\}^\{j\}=K\_\{x\}\\setminus\\\{k\_\{1\},\\ldots,k\_\{j\}\\\}\. Atθt\+1\\theta\_\{t\+1\}its cost is

Ct=∑j=1m𝔼ρ​\[pKxj−1​\(x\)−pKxj​\(x\)\]\.C\_\{t\}=\\sum\_\{j=1\}^\{m\}\\mathbb\{E\}\_\{\\rho\}\\left\[p\_\{K\_\{x\}^\{j\-1\}\}\(x\)\-p\_\{K\_\{x\}^\{j\}\}\(x\)\\right\]\.\(32\)A term is zero on tasks not retrievingkjk\_\{j\}\.

###### Proof\.

Add and subtractJ𝒮t​\(θt\+1\)J\_\{\\mathcal\{S\}\_\{t\}\}\(\\theta\_\{t\+1\}\)in each round’s total change to obtainΓt−Ct\\Gamma\_\{t\}\-C\_\{t\}, and telescope over rounds\. For pure removals, telescope again overKx0,…,KxmK\_\{x\}^\{0\},\\ldots,K\_\{x\}^\{m\}before taking the task expectation\. Both calculations are algebraic, so they hold for any sequence of policies and scaffolds\. ∎

The sequential effects in Eq\. \([32](https://arxiv.org/html/2610.22740#A1.E32)\) can differ from single\-removal effects\. For two redundant skillsa,ba,bwith success probabilitiesp\{a,b\}=p\{a\}=p\{b\}=1p\_\{\\\{a,b\\\}\}=p\_\{\\\{a\\\}\}=p\_\{\\\{b\\\}\}=1andp∅=0p\_\{\\varnothing\}=0, each single removal costs zero, but removing both costs one\. Bounds therefore add over the successive contexts; in our runs no two withdrawals share a round \(Figure[5](https://arxiv.org/html/2610.22740#S5.F5)f\), and re\-testing withdrawn skills \(Appendix[C](https://arxiv.org/html/2610.22740#A3)\) can restore a skill whose removal hurts\. Admission, revision, and retrieval replacement enter through Eq\. \([31](https://arxiv.org/html/2610.22740#A1.E31)\)\.

For one skill, letak=Prρ⁡\(k∈Kx\)a\_\{k\}=\\Pr\_\{\\rho\}\(k\\in K\_\{x\}\)and letμρ,k\\mu\_\{\\rho,k\}be its mean current effect conditional on retrieval\. ThenCt=ak​μρ,kC\_\{t\}=a\_\{k\}\\mu\_\{\\rho,k\}\. A ledger instead targets the tasks on which that skill was selected for testing\. These distributions can differ even when the task curriculum is fixed, because skill selection depends on the task and ledger\. They can also differ from the fixed evaluation distribution in Eq\. \([31](https://arxiv.org/html/2610.22740#A1.E31)\)\.

The posterior bound transfers to the round cost up to a driftdkd\_\{k\}: if a posterior concernsμk\\mu\_\{k\}, the desired current mean satisfiesμρ,k≤μk\+dk\\mu\_\{\\rho,k\}\\leq\\mu\_\{k\}\+d\_\{k\}under the same model, and withdrawal gives𝔼⁡\[μk\+\]≤ϵ\\mathbb\{E\}\[\\mu\_\{k\}^\{\+\}\]\\leq\\epsilon, then

𝔼⁡\[Ct\]≤ak​\(ϵ\+dk\)\.\\mathbb\{E\}\[C\_\{t\}\]\\leq a\_\{k\}\(\\epsilon\+d\_\{k\}\)\.\(33\)Matched policy, context, and task scope suffice fordk=0d\_\{k\}=0\. For effects in\[−1,1\]\[\-1,1\], a uniform effect driftbkb\_\{k\}and a task\-distribution mismatch contribute at mostbk\+2TV\(ρ\(⋅∣k∈Kx\),νk\)b\_\{k\}\+2\\mathrm\{TV\}\(\\rho\(\\cdot\\mid k\\in K\_\{x\}\),\\nu\_\{k\}\), whereνk\\nu\_\{k\}is the distribution of the tasks that testkk\. This follows by adding and subtracting the old effect under the new distribution and using the total\-variation bound\.

When several withdrawals are justified on successive contexts, letμj\\mu\_\{j\}denote each retrieval\-conditional mean effect andaja\_\{j\}its evaluation retrieval mass\. Under a common posterior model with𝔼⁡\[μj\+\]≤ϵj\\mathbb\{E\}\[\\mu\_\{j\}^\{\+\}\]\\leq\\epsilon\_\{j\}for each actual successive effect, linearity gives𝔼⁡\[Ct\]≤∑jaj​ϵj\\mathbb\{E\}\[C\_\{t\}\]\\leq\\sum\_\{j\}a\_\{j\}\\epsilon\_\{j\}\. Independence between the effects is unnecessary; the marginals only need to refer to the successive contexts\.

The fixed evaluation distribution also matters\. For changing distributionsρt\\rho\_\{t\}, define inner gains and outer costs usingρt\\rho\_\{t\}within roundtt\. WritingJνJ^\{\\nu\}for evaluation underν\\nu, an additional term is

Dt=J𝒮t\+1ρt\+1​\(θt\+1\)−J𝒮t\+1ρt​\(θt\+1\)\.D\_\{t\}=J^\{\\rho\_\{t\+1\}\}\_\{\\mathcal\{S\}\_\{t\+1\}\}\(\\theta\_\{t\+1\}\)\-J^\{\\rho\_\{t\}\}\_\{\\mathcal\{S\}\_\{t\+1\}\}\(\\theta\_\{t\+1\}\)\.\(34\)Adding and subtracting the second expectation at every transition yields a telescoping sum ofΓt−Ct\+Dt\\Gamma\_\{t\}\-C\_\{t\}\+D\_\{t\}, so changingρt\\rho\_\{t\}alters measured success without changing the Director\. Keepingρ\\rhofixed in the deployed target removesDtD\_\{t\}, while the Proposer still chooses training tasks adaptively\.

### A\.6Tracking under separated scaffold changes

###### Lemma A\.6\(Two\-time\-scale tracking\)\.

Let the inner state obey

wt\+1=wt\+at​\{h⁡\(wt,𝒮t\)\+Mt\+1\}\.w\_\{t\+1\}=w\_\{t\}\+a\_\{t\}\\\{h\(w\_\{t\};\\mathcal\{S\}\_\{t\}\)\+M\_\{t\+1\}\\\}\.\(35\)Assume deterministic positive step sizes with∑tat=∞\\sum\_\{t\}a\_\{t\}=\\inftyand∑tat2<∞\\sum\_\{t\}a\_\{t\}^\{2\}<\\infty; adapted iterates; predictable modes𝒮t\\mathcal\{S\}\_\{t\}; and martingale\-difference noise with uniformly bounded conditional second moment\. Suppose all iterates lie in a compact setDD, every mode has a locally Lipschitz mean field, andDDis forward invariant for its ODE\. Assume finitely many modes, each with a Lyapunov\-stable equilibriumw⋆​\(𝒮\)w^\{\\star\}\(\\mathcal\{S\}\)attracting every point ofDD\. Finally, either mode changes stop, or their successive spacings on the clockst=∑u<taus\_\{t\}=\\sum\_\{u<t\}a\_\{u\}tend to infinity\. For everyζ\>0\\zeta\>0, almost surely,

1sT∑t<Tat𝟏\{∥wt−w⋆\(𝒮t\)∥\>ζ\}⟶0\.\\frac\{1\}\{s\_\{T\}\}\\sum\_\{t<T\}a\_\{t\}\\mathbf\{1\}\\\{\\\|w\_\{t\}\-w^\{\\star\}\(\\mathcal\{S\}\_\{t\}\)\\\|\>\\zeta\\\}\\longrightarrow 0\.\(36\)

###### Proof\.

The martingale∑t<nat​Mt\+1\\sum\_\{t<n\}a\_\{t\}M\_\{t\+1\}has bounded total second moment and converges almost surely\. Its increments over all sufficiently late finite clock windows therefore vanish uniformly\. Boundedness ofhhonDD,at→0a\_\{t\}\\to 0, and the local Lipschitz property give the corresponding Euler approximation error\. Applying the integral form of the ODE and Gronwall’s inequality shows that, on any fixed\-length interval without switching, the interpolation of the recursion uniformly approaches the ODE started at the same point\. The constants can be chosen uniformly over the finite set of modes\.

Fixζ\\zeta\. Stability provides a smaller neighborhood of each equilibrium whose ODE trajectories stay inside theζ/2\\zeta/2neighborhood\. Attraction and compactness give a common finite timeHHafter which every trajectory starting inDDis inside an even smaller neighborhood\. To see uniformity, choose an entry time for each initial point, use continuity of the finite\-time flow to obtain a neighborhood with the same entry property, and take a finite subcover ofDD\. Stability then preserves the entry property at later times\. Taking the maximum over modes gives the commonHH\.

On sufficiently late unswitched windows, the approximation error is smaller than both neighborhood margins\. After the first window of lengthHH, the stochastic interpolation is inside the inner neighborhood\. Applying the same argument on consecutive windows keeps their endpoints inside it and their intervening paths within distanceζ\\zeta\. Thus each switch contributes at most a fixed transient clock length, up to vanishing discretization errors\. IfN⁡\(s\)N\(s\)counts switches before clock timess, diverging spacings implyN⁡\(s\)/s→0N\(s\)/s\\to 0: eventually every complete spacing exceeds any prescribed constant\. The total transient fraction is consequently bounded by a vanishing initial fraction plusO⁡\(H​N​\(s\)/s\)O\(HN\(s\)/s\), proving Eq\. \([36](https://arxiv.org/html/2610.22740#A1.E36)\)\. If switching stops, the argument gives convergence to the final equilibrium\. ∎

We hold the running statistics, such as the ledger and the reward normalization, fixed within each mode, so that each mode has an autonomous mean field\. The finite\-round accounting identity of Lemma[A\.5](https://arxiv.org/html/2610.22740#Thmlemma5)holds without these conditions\.

### A\.7Randomized contrasts, bootstrap means, and decisions

###### Lemma A\.7\(Randomization\)\.

For a fixed task, policy, and candidate, assign exactlynvn\_\{v\}ofGGrollout slots to the visible context uniformly at random, independently of their potential outcomes\. Assume no interference between slots\. Then

𝔼\[zx∣x,k\]\\displaystyle\\mathbb\{E\}\[z\_\{x\}\\mid x,k\]=p\+−p−=Δk​\(x\),\\displaystyle=p^\{\+\}\-p^\{\-\}=\\Delta\_\{k\}\(x\),\(37\)𝔼\[Y¯∣x,k\]\\displaystyle\\mathbb\{E\}\[\\bar\{Y\}\\mid x,k\]=p−\+nvG​Δk​\(x\)\.\\displaystyle=p^\{\-\}\+\\tfrac\{n\_\{v\}\}\{G\}\\Delta\_\{k\}\(x\)\.For fixed slot\-level potential outcomes, the design variance is

Varassign​\(zx\)=S12nv\+S02nh−SΔ2G,\\mathrm\{Var\}\_\{\\mathrm\{assign\}\}\(z\_\{x\}\)=\\frac\{S\_\{1\}^\{2\}\}\{n\_\{v\}\}\+\\frac\{S\_\{0\}^\{2\}\}\{n\_\{h\}\}\-\\frac\{S\_\{\\Delta\}^\{2\}\}\{G\},\(38\)where the finite\-population variances use denominatorG−1G\-1\. Under independent superpopulation rollout sampling, the unconditional variance given the task isV\+/nv\+V−/nhV\_\{\+\}/n\_\{v\}\+V\_\{\-\}/n\_\{h\}\.

###### Proof\.

LetIiI\_\{i\}mark a visible assignment and letYi​\(1\),Yi​\(0\)Y\_\{i\}\(1\),Y\_\{i\}\(0\)be fixed potential outcomes\. Since𝔼​Ii=nv/G\\mathbb\{E\}I\_\{i\}=n\_\{v\}/G, the two observed part means have expectations equal to their corresponding population means\. Their difference is the population mean treatment effect\. Averaging over potential\-outcome sampling yields the first identity, and the weighted average of the part means yields the second\.

For the variance,Var⁡\(Ii\)=nv​nh/G2\\mathrm\{Var\}\(I\_\{i\}\)=n\_\{v\}n\_\{h\}/G^\{2\}andCov\(Ii,Ij\)=−nvnh/\[G2\(G−1\)\]\\mathrm\{Cov\}\(I\_\{i\},I\_\{j\}\)=\-n\_\{v\}n\_\{h\}/\[G^\{2\}\(G\-1\)\]for distinct slots\. Writezx=∑iIi​ci−∑iYi​\(0\)/nhz\_\{x\}=\\sum\_\{i\}I\_\{i\}c\_\{i\}\-\\sum\_\{i\}Y\_\{i\}\(0\)/n\_\{h\}withci=Yi​\(1\)/nv\+Yi​\(0\)/nhc\_\{i\}=Y\_\{i\}\(1\)/n\_\{v\}\+Y\_\{i\}\(0\)/n\_\{h\}\. Substitution of these covariances givesnv​nh​∑i\(ci−c¯\)2/\[G⁡\(G−1\)\]n\_\{v\}n\_\{h\}\\sum\_\{i\}\(c\_\{i\}\-\\bar\{c\}\)^\{2\}/\[G\(G\-1\)\]\. Expanding this expression and usingSΔ2=S12\+S02−2​S10S\_\{\\Delta\}^\{2\}=S\_\{1\}^\{2\}\+S\_\{0\}^\{2\}\-2S\_\{10\}proves Eq\. \([38](https://arxiv.org/html/2610.22740#A1.E38)\)\. Under independent rollout sampling, the two observed part means concern disjoint independent slots, so their variances add toV\+/nv\+V−/nhV\_\{\+\}/n\_\{v\}\+V\_\{\-\}/n\_\{h\}\. ∎

A re\-test group adds a withdrawn skillkkto the visible part, so by the same lemma its contrast estimatespKx∪\{k\}​\(x\)−pKx​\(x\)p\_\{K\_\{x\}\\cup\\\{k\\\}\}\(x\)\-p\_\{K\_\{x\}\}\(x\)\. The design is blocked by task: the contrast removes the baseline term at a fixed task, while task selection can still change the average effect\. A fixed mean group success does not identifyΔk​\(x\)\\Delta\_\{k\}\(x\), since differentp−p^\{\-\}values offset different effects in Eq\. \([37](https://arxiv.org/html/2610.22740#A1.E37)\)\. For example, writeqv=nv/Gq\_\{v\}=n\_\{v\}/Gand choose anyc∈\(0,1\)c\\in\(0,1\)\. The probabilitiesp−=c−qv​dp^\{\-\}=c\-q\_\{v\}dandp\+=c\+\(1−qv\)​dp^\{\+\}=c\+\(1\-q\_\{v\}\)dgive mean group successccand effectdd\. For all sufficiently small positivedd, both are valid probabilities\. Takingd=0d=0gives the same mean and zero effect\. Thus the nonidentifiability is exact even under a fixed group ratio, while the randomized contrast distinguishes these cases in expectation\.

#### What adaptive records estimate\.

Predictable task and skill selection does not destroy within\-group randomization\. To see what it preserves, index the successive records for a fixed skill version byjjand letℋj−1\\mathcal\{H\}\_\{j\-1\}contain the information available before the next tested group is generated\. Defineμj=𝔼⁡\[zj∣ℋj−1\]\\mu\_\{j\}=\\mathbb\{E\}\[z\_\{j\}\\mid\\mathcal\{H\}\_\{j\-1\}\], including the current task\-selection law in that expectation\. Thenej=zj−μje\_\{j\}=z\_\{j\}\-\\mu\_\{j\}is a martingale difference, and for deterministicnn,

𝔼⁡\[\(z¯n−1n​∑j=1nμj\)2\]≤1n\.\\mathbb\{E\}\\left\[\\left\(\\bar\{z\}\_\{n\}\-\\frac\{1\}\{n\}\\sum\_\{j=1\}^\{n\}\\mu\_\{j\}\\right\)^\{2\}\\right\]\\leq\\frac\{1\}\{n\}\.\(39\)Fori<ji<j, conditioning onℋj−1\\mathcal\{H\}\_\{j\-1\}gives𝔼⁡\[ei​ej\]=0\\mathbb\{E\}\[e\_\{i\}e\_\{j\}\]=0\. Alsozj∈\[−1,1\]z\_\{j\}\\in\[\-1,1\]implies𝔼⁡\[ej2∣ℋj−1\]=Var⁡\(zj∣ℋj−1\)≤1\\mathbb\{E\}\[e\_\{j\}^\{2\}\\mid\\mathcal\{H\}\_\{j\-1\}\]=\\mathrm\{Var\}\(z\_\{j\}\\mid\\mathcal\{H\}\_\{j\-1\}\)\\leq 1\. Expanding the square proves the bound, so the ledger mean tracks the average of the predictable effectsμj\\mu\_\{j\}\.

For a current targetμnow\\mu\_\{\\mathrm\{now\}\}, the error is this sampling term plus the driftn−1​∑jμj−μnown^\{\-1\}\\sum\_\{j\}\\mu\_\{j\}\-\\mu\_\{\\mathrm\{now\}\}; more records shrink the former, and the 30\-cycle evidence window of Appendix[C](https://arxiv.org/html/2610.22740#A3)keeps old records from dominating the latter\.

#### Bootstrap mean\.

Treat the ledger as an indexed collection ofnncontrast recordsz1,…,znz\_\{1\},\\ldots,z\_\{n\}, allowing repeated task occurrences\. For weightsw∼Dirichlet⁡\(1,…,1\)w\\sim\\mathrm\{Dirichlet\}\(1,\\ldots,1\), defineΔBB=∑jwj​zj\\Delta^\{\\mathrm\{BB\}\}=\\sum\_\{j\}w\_\{j\}z\_\{j\}\. Conditional on the records,

𝔼BB​ΔBB\\displaystyle\\mathbb\{E\}\_\{\\mathrm\{BB\}\}\\Delta^\{\\mathrm\{BB\}\}=z¯,\\displaystyle=\\bar\{z\},\(40\)VarBB​\(ΔBB\)\\displaystyle\\mathrm\{Var\}\_\{\\mathrm\{BB\}\}\(\\Delta^\{\\mathrm\{BB\}\}\)=∑j\(zj−z¯\)2n⁡\(n\+1\)\.\\displaystyle=\\frac\{\\sum\_\{j\}\(z\_\{j\}\-\\bar\{z\}\)^\{2\}\}\{n\(n\+1\)\}\.Indeed,𝔼​wj=1/n\\mathbb\{E\}w\_\{j\}=1/n,Var⁡\(wj\)=\(n−1\)/\[n2​\(n\+1\)\]\\mathrm\{Var\}\(w\_\{j\}\)=\(n\-1\)/\[n^\{2\}\(n\+1\)\], and distinct weights have covariance−1/\[n2\(n\+1\)\]\-1/\[n^\{2\}\(n\+1\)\]; expanding the weighted sum proves both identities\. Under a stationary exchangeable record model, this is the Bayesian\-bootstrap posterior of the mean observed\-contrast distribution\. That mean equals the mean latent skill effect when randomization is unbiased on the same scope\. No latent\-effect likelihood is needed to define this mean\.

For the adaptive ledger, we use the same Dirichlet construction as the posterior of the recorded comparison model, and the decisions below minimize posterior risk under it\.

###### Lemma A\.8\(Bayes decisions\)\.

Fix a posteriorΠ\\Piover a scope\-specific mean effectΔ\\Delta\. With lossesΔ\+\\Delta^\{\+\}for withdrawal andϵ\\epsilonfor retention, withdrawal minimizes posterior expected loss exactly when𝔼Π​\[Δ\+\]≤ϵ\\mathbb\{E\}\_\{\\Pi\}\[\\Delta^\{\+\}\]\\leq\\epsilon, allowing withdrawal at a tie\. With loss1−α1\-\\alphafor admitting a nonbeneficial skill andα\\alphafor withholding a beneficial one, admission is optimal exactly whenΠ⁡\(Δ≤0\)≤α\\Pi\(\\Delta\\leq 0\)\\leq\\alpha, ties included\.

###### Proof\.

The withdrawal and retention risks are precisely𝔼Π​\[Δ\+\]\\mathbb\{E\}\_\{\\Pi\}\[\\Delta^\{\+\}\]andϵ\\epsilon\. For admission, putq=Π⁡\(Δ≤0\)q=\\Pi\(\\Delta\\leq 0\)\. The two risks are\(1−α\)​q\(1\-\\alpha\)qandα⁡\(1−q\)\\alpha\(1\-q\); their comparison reduces toq≤αq\\leq\\alpha\. Finally𝔼Π​Δ≤𝔼Π​Δ\+\\mathbb\{E\}\_\{\\Pi\}\\Delta\\leq\\mathbb\{E\}\_\{\\Pi\}\\Delta^\{\+\}, proving the signed\-mean posterior bound used in Eq\. \([33](https://arxiv.org/html/2610.22740#A1.E33)\)\. ∎

The positive part is taken*after*forming the mean effect, and convexity places the resulting risk between two simpler quantities,z¯\+≤𝔼BB​\[\(ΔBB\)\+\]≤n−1​∑jzj\+\\bar\{z\}^\{\+\}\\leq\\mathbb\{E\}\_\{\\mathrm\{BB\}\}\[\(\\Delta^\{\\mathrm\{BB\}\}\)^\{\+\}\]\\leq n^\{\-1\}\\sum\_\{j\}z\_\{j\}^\{\+\}\. The rule thus bounds the modeled aggregate harm over the test scope\. Minimum record counts restrict eligibility, the withdrawal\-first precedence resolves overlap between admission and withdrawal, and trial expiration \(Appendix[C](https://arxiv.org/html/2610.22740#A3)\) is a separate management rule\.

Versioning a skill by identifier, version, and content hash keeps evidence from pooling across edits to that skill, and the 30\-cycle evidence window keeps it close to the current Director, context, and curriculum; the driftdkd\_\{k\}of Eq\. \([33](https://arxiv.org/html/2610.22740#A1.E33)\) carries the rest into the round cost\. The prior pseudo\-record enters only the exploration distributionΠk0\\Pi\_\{k\}^\{0\}of Thompson selection, so decisions use the data\-onlyΠk\\Pi\_\{k\}, and the minimum\-record gate keeps an untested skill from qualifying for a decision\.

### A\.8The anchored curriculum and distinct pairs

###### Lemma A\.9\(Anchored curriculum\)\.

Fix a finite candidate list, a full\-support referenceρ0\\rho\_\{0\}, finite conditional expected rewardsR¯x\\bar\{R\}\_\{x\}, andτ\>0\\tau\>0\. Over all distributions on the list, the unique maximizer of∑xρxR¯x−τKL\(ρ∥ρ0\)\\sum\_\{x\}\\rho\_\{x\}\\bar\{R\}\_\{x\}\-\\tau\\mathrm\{KL\}\(\\rho\\\|\\rho\_\{0\}\)is the Gibbs distribution in Eq\. \([10](https://arxiv.org/html/2610.22740#S4.E10)\)\. For two independent picksa,b∼ρϕa,b\\sim\\rho\_\{\\phi\}, with conditional reward means depending only on the respective task,

𝔼\[12\(Ra−Rb\)\(sa−sb\)\]=∇ϕ∑xρϕ\(x\)R¯x,\\mathbb\{E\}\\left\[\\tfrac\{1\}\{2\}\(R\_\{a\}\-R\_\{b\}\)\(s\_\{a\}\-s\_\{b\}\)\\right\]=\\nabla\_\{\\phi\}\\sum\_\{x\}\\rho\_\{\\phi\}\(x\)\\bar\{R\}\_\{x\},\(41\)wheresx=∇ϕ​log​ρϕ​\(x\)s\_\{x\}=\\nabla\_\{\\phi\}\\log\\rho\_\{\\phi\}\(x\)and rewards are held fixed when differentiating\.

###### Proof\.

WithZ=∑xρ0​\(x\)​eR¯x/τZ=\\sum\_\{x\}\\rho\_\{0\}\(x\)e^\{\\bar\{R\}\_\{x\}/\\tau\}, the objective isτlogZ−τKL\(ρ∥ρ⋆\)\\tau\\log Z\-\\tau\\mathrm\{KL\}\(\\rho\\\|\\rho^\{\\star\}\), proving uniqueness\. Expanding the pair estimator, the two self\-terms each have expectation∇ϕ∑xρϕ\(x\)R¯x\\nabla\_\{\\phi\}\\sum\_\{x\}\\rho\_\{\\phi\}\(x\)\\bar\{R\}\_\{x\}\. The cross\-terms vanish by independence and𝔼⁡\[sx\]=0\\mathbb\{E\}\[s\_\{x\}\]=0, and the factor one half gives Eq\. \([41](https://arxiv.org/html/2610.22740#A1.E41)\)\. ∎

For the stated distinct\-pair sampler, putpx=ρϕ​\(x\)p\_\{x\}=\\rho\_\{\\phi\}\(x\)and assume at least two candidates with positive probability\. Keeping the first pick and redrawing the second from the remaining candidates gives

Pr⁡\(a,b\)=pa​pb1−pa,a≠b\.\\Pr\(a,b\)=\\frac\{p\_\{a\}p\_\{b\}\}\{1\-p\_\{a\}\},\\qquad a\\neq b\.\(42\)The unordered\-pair probability sums both orders\. Under the same task\-local reward\-mean assumption, the expectation of the raw estimator is

∑a<bpa​pb​wa​b​\(R¯a−R¯b\)​\(sa−sb\),\\displaystyle\\sum\_\{a<b\}p\_\{a\}p\_\{b\}\\,w\_\{ab\}\(\\bar\{R\}\_\{a\}\-\\bar\{R\}\_\{b\}\)\(s\_\{a\}\-s\_\{b\}\),\(43\)wa​b=12​\(11−pa\+11−pb\)\.\\displaystyle w\_\{ab\}=\\tfrac\{1\}\{2\}\\left\(\\frac\{1\}\{1\-p\_\{a\}\}\+\\frac\{1\}\{1\-p\_\{b\}\}\\right\)\.This follows directly by multiplying the unordered\-pair probability by the symmetric half\-difference estimator\. The independent\-pick gradient is the same sum withwa​b=1w\_\{ab\}=1\. These pair\-dependent factors cannot generally be absorbed into a single learning rate\. For pair\-dependent rewards, replaceR¯a−R¯b\\bar\{R\}\_\{a\}\-\\bar\{R\}\_\{b\}by𝔼⁡\[Ra−Rb∣\{a,b\}\]\\mathbb\{E\}\[R\_\{a\}\-R\_\{b\}\\mid\\\{a,b\\\}\], with reward labels attached to tasks rather than pick order\.

A two\-candidate example already separates the stationary points\. LetR¯1−R¯2=1\\bar\{R\}\_\{1\}\-\\bar\{R\}\_\{2\}=1,ρ0=\(1/2,1/2\)\\rho\_\{0\}=\(1/2,1/2\),τ=1\\tau=1, and parameterizep1p\_\{1\}by its logit\. The raw distinct\-pair reward gradient is1/21/2, whereas the independent\-pick gradient isp1​\(1−p1\)p\_\{1\}\(1\-p\_\{1\}\)\. At the Gibbs logit of one, the KL gradient equalsp1​\(1−p1\)p\_\{1\}\(1\-p\_\{1\}\), so the raw distinct\-pair regularized gradient remains nonzero\. Gibbs optimality thus characterizes the ideal objective, while the stationary point of the practical update also depends on the pair weights, reward scaling, and trust region\.

If both selected tasks supply training groups, the pooled task marginal within a list is

ν⁡\(x\)=12​px​\(1\+∑a≠xpa1−pa\),\\nu\(x\)=\\tfrac\{1\}\{2\}p\_\{x\}\\left\(1\+\\sum\_\{a\\neq x\}\\frac\{p\_\{a\}\}\{1\-p\_\{a\}\}\\right\),\(44\)obtained by averaging first\- and second\-position marginals\. Conditioning further on selection of skillkkgives its test\-task scope\. Finally, a smaller expected reward implies a smaller Gibbs ratioρ⋆​\(x\)/ρ0​\(x\)\\rho^\{\\star\}\(x\)/\\rho\_\{0\}\(x\), and the absolute probability also scales with the reference probabilityρ0​\(x\)\\rho\_\{0\}\(x\)\.

### A\.9The three propositions

###### Proof of Proposition 1\.

Lemma[A\.5](https://arxiv.org/html/2610.22740#Thmlemma5)proves the accounting statement for every finite sequence\. Appending a test\-time step𝒮T\+1=∅\\mathcal\{S\}\_\{T\+1\}=\\varnothingwithθT\+1=θT\\theta\_\{T\+1\}=\\theta\_\{T\}makes its left side the zero\-skill success underρ\\rhofor any run, with the benefit of the remaining scaffold as the last outer cost\. Lemma[A\.6](https://arxiv.org/html/2610.22740#Thmlemma6)proves the tracking statement under its assumptions, and Lemma[A\.4](https://arxiv.org/html/2610.22740#Thmlemma4)identifies the context mixture asqxq\_\{x\}, orqx′q^\{\\prime\}\_\{x\}under separate last\-skill centering\. ∎

###### Proof of Proposition 2\.

Apply Lemma[A\.1](https://arxiv.org/html/2610.22740#Thmlemma1)to the policy onK−K^\{\-\}to obtain gainβ​p−​\(1−p−\)\+O⁡\(β2\)\\beta p^\{\-\}\(1\-p^\{\-\}\)\+O\(\\beta^\{2\}\), or2​δ​p−​\(1−p−\)\+O⁡\(δ\)\\sqrt\{2\\delta p^\{\-\}\(1\-p^\{\-\}\)\}\+O\(\\delta\)at a common small budget, which grows withℓrec\\ell\_\{\\mathrm\{rec\}\}to first order; for a parametric Director, Eq\. \([21](https://arxiv.org/html/2610.22740#A1.E21)\) bounds the accessible part\. Lemma[A\.3](https://arxiv.org/html/2610.22740#Thmlemma3)identifies4​p−​\(1−p−\)4p^\{\-\}\(1\-p^\{\-\}\)with normalized expected energy\. Lemma[A\.2](https://arxiv.org/html/2610.22740#Thmlemma2)gives the shared\-tilt optimum of the quadratic allocation problem\. With stochastic Workers, Eq\. \([20](https://arxiv.org/html/2610.22740#A1.E20)\) instead governs graph\-controllable progress\. The operational shrinkage, transfer, and collapse terms turn this target into the expected reward of Lemma[A\.9](https://arxiv.org/html/2610.22740#Thmlemma9)\. ∎

###### Proof of Proposition 3\.

The first identity of Lemma[A\.7](https://arxiv.org/html/2610.22740#Thmlemma7)proves conditional unbiasedness\. Averaging it over the actual test\-task distribution proves unbiasedness for that distribution’s mean effect\. The second identity mixes the post\-withdrawal baseline with the skill effect, as in Eq\. \([12](https://arxiv.org/html/2610.22740#S4.E12)\), while the contrast cancels the baseline within the task\. Bootstrap aggregation and Lemma[A\.8](https://arxiv.org/html/2610.22740#Thmlemma8)then give the decisions of Eq\. \([14](https://arxiv.org/html/2610.22740#S4.E14)\), and Eq\. \([33](https://arxiv.org/html/2610.22740#A1.E33)\) carries them to the round costs\. ∎

## Appendix BEstimators and Implementation Details

#### Shrinkage and compute factor\.

The success ratep~x\\tilde\{p\}\_\{x\}in Eq\. \([9](https://arxiv.org/html/2610.22740#S4.E9)\) shrinks toward the running hidden\-part success rate of the task’s dataset with strengthmm, an exponential moving average over previous cycles with decay 0\.9 that starts at one half\. The positive term of Eq\. \([9](https://arxiv.org/html/2610.22740#S4.E9)\) is multiplied by\(T¯d/Tq\)η\(\\bar\{T\}\_\{d\}/T\_\{q\}\)^\{\\eta\}, whereTqT\_\{q\}is the mean duration of the task’s rollouts andT¯d\\bar\{T\}\_\{d\}the running mean duration of the dataset, so tasks are compared within a dataset by learning progress per unit of compute\. A task without both parts, for instance one for which no skill is retrieved, is scored byℓ\\ellof the shrunk success rate of the whole group, with the same collapse penalty\.

#### Prior record and cost term\.

The Thompson draw of Eq\. \([13](https://arxiv.org/html/2610.22740#S4.E13)\) adds one pseudo\-record from a wide zero\-mean prior to the bootstrap, so a skill with no data is drawn from the prior alone\. The Director’s reward subtractsλ\\lambdatimes the Workers’ normalized cost from correctness; the contrastszxz\_\{x\}always use raw correctness\.

#### Director update\.

The Director’s loss is the clipped surrogate of PPO on the advantages of Eq\. \([5](https://arxiv.org/html/2610.22740#S4.E5)\), whitened over the batch, which rescales every task by the same factor, and averaged over tokens, plus a KL anchor to the base model\. Samples whose advantage is below a small threshold in magnitude, mostly from groups whose rollouts agree in correctness and differ only in cost, are dropped except for a small share kept for the anchor\.

#### Proposer update\.

Each candidate list yields two distinct picksaaandbb: drawaafrom the choice policy and drawbbfrom that policy conditioned onb≠ab\\neq a\. The advantage ofaais half the reward differenceRP​\(a\)−RP​\(b\)R\_\{P\}\(a\)\-R\_\{P\}\(b\), scaled to unit root mean square over the batch, andbbtakes its negative; the loss addsτ\\tautimes the KL of the list’s choice distribution to the reference\. A trust region drops the policy term once the pair’s tilt\(log⁡ρϕ​\(a\)−log⁡ρϕ​\(b\)\)−\(log⁡ρ0​\(a\)−log⁡ρ0​\(b\)\)\(\\log\\rho\_\{\\phi\}\(a\)\-\\log\\rho\_\{\\phi\}\(b\)\)\-\(\\log\\rho\_\{0\}\(a\)\-\\log\\rho\_\{0\}\(b\)\)passes a bound in the direction the advantage pushes, so the choice within a list stays stochastic\.

#### Graph canvas\.

Agents, layers, and relations are edited through the action set of Definition 2, and the canvas validates each edit before it is applied\.

## Appendix CTraining Loop and Skill Ledger

Algorithm[1](https://arxiv.org/html/2610.22740#alg1)lists one round of R2Orch; every step cites the equation it implements\. An LLM editor revises skills, the least helpful first, and distills new ones from groups in which both parts fail, while withdrawals follow Eq\. \([14](https://arxiv.org/html/2610.22740#S4.E14)\)\. A skill that the editor adds or revises enters the scaffold on trial under a new ledger key \(identifier, version, and content hash\), so its records start afresh; records for an unchanged skill identity remain pooled across Director updates\.𝒩t\\mathcal\{N\}\_\{t\}admits it, a trial skill still unadmitted after3​nmin3n\_\{\\min\}records is discarded, and𝒲t\\mathcal\{W\}\_\{t\}is checked first, so the two rules never act on one skill in the same round\. A task withKx=∅K\_\{x\}=\\varnothingruns itsGGrollouts without skills and adds no record\. Each posterior uses the records of the last 30 cycles, and 30% of the tasks re\-test withdrawn skills, which𝒩t\\mathcal\{N\}\_\{t\}can restore\.

Algorithm 1One round of R2Orch1:Director

πθ\\pi\_\{\\theta\}, Proposer

ρϕ\\rho\_\{\\phi\}, base models

πref,ρ0\\pi\_\{\\mathrm\{ref\}\},\\rho\_\{0\}, scaffold

𝒮t\\mathcal\{S\}\_\{t\}with summary

sts\_\{t\}, ledger

𝒟t\\mathcal\{D\}\_\{t\}
2:foreach candidate list

𝒞\\mathcal\{C\}of the rounddo

3:draw two distinct tasks from

ρϕ\(⋅∣𝒞,st\)\\rho\_\{\\phi\}\(\\cdot\\mid\\mathcal\{C\},s\_\{t\}\)⊳\\trianglerightAppendix[B](https://arxiv.org/html/2610.22740#A2)

4:foreach selected task

xxdo

5:retrieve

Kx⊆𝒮tK\_\{x\}\\subseteq\\mathcal\{S\}\_\{t\}; on a re\-test task, add one withdrawn skill to

KxK\_\{x\}as

kxk\_\{x\}⊳\\trianglerightAppendix[C](https://arxiv.org/html/2610.22740#A3)

6:otherwise, if

Kx≠∅K\_\{x\}\\neq\\varnothing, draw

kxk\_\{x\}by Thompson sampling; draw

ℋx\\mathcal\{H\}\_\{x\}uniformly, or set

ℋx=∅\\mathcal\{H\}\_\{x\}=\\varnothingif

Kx=∅K\_\{x\}=\\varnothing⊳\\trianglerightEq\. \([13](https://arxiv.org/html/2610.22740#S4.E13)\)

7:set

Ki←KxK\_\{i\}\\leftarrow K\_\{x\}for all

ii; for

i∈ℋxi\\in\\mathcal\{H\}\_\{x\}, set

Ki←Kx∖\{kx\}K\_\{i\}\\leftarrow K\_\{x\}\\setminus\\\{k\_\{x\}\\\}
8:run the Director under

\(x,Ki\)\(x,K\_\{i\}\)and execute its graph on the Workers; record correctness

yiy\_\{i\},

i≤Gi\\leq G
9:compute

RP​\(x\)R\_\{P\}\(x\)using Eq\. \([9](https://arxiv.org/html/2610.22740#S4.E9)\) or the skill\-free fallback

10:if

Kx≠∅K\_\{x\}\\neq\\varnothingthen

11:

zx←y¯𝒱−y¯ℋz\_\{x\}\\leftarrow\\bar\{y\}\_\{\\mathcal\{V\}\}\-\\bar\{y\}\_\{\\mathcal\{H\}\}; append

\(kx,zx\)\(k\_\{x\},z\_\{x\}\)to

𝒟t\\mathcal\{D\}\_\{t\}⊳\\trianglerightEq\. \([11](https://arxiv.org/html/2610.22740#S4.E11)\)

12:

θ←\\theta\\leftarrowclipped step on the advantages of Eq\. \([5](https://arxiv.org/html/2610.22740#S4.E5)\) with the KL anchor to

πref\\pi\_\{\\mathrm\{ref\}\}
13:

ϕ←\\phi\\leftarrowdistinct\-pair step of Appendix[B](https://arxiv.org/html/2610.22740#A2)with the KL anchor to

ρ0\\rho\_\{0\}and the trust region on the tilt

14:foreach skill

kkin

𝒮t\\mathcal\{S\}\_\{t\}or under re\-test with

nk≥nminn\_\{k\}\\geq n\_\{\\min\}do

15:

Πk←\\Pi\_\{k\}\\leftarrowEq\. \([13](https://arxiv.org/html/2610.22740#S4.E13)\); withdraw

kkif

k∈𝒲tk\\in\\mathcal\{W\}\_\{t\}, otherwise admit or restore it if

k∈𝒩tk\\in\\mathcal\{N\}\_\{t\}⊳\\trianglerightEq\. \([14](https://arxiv.org/html/2610.22740#S4.E14)\)

16:discard each trial skill still unadmitted after

3​nmin3n\_\{\\min\}records

17:if a revision candidate with a defined posterior exists, revise the one with the lowest posterior benefit

18:distill new candidates from groups in which both parts fail

19:return

θ,ϕ,𝒮t\+1,𝒟t\+1\\theta,\\phi,\\mathcal\{S\}\_\{t\+1\},\\mathcal\{D\}\_\{t\+1\}

SymbolMeaningValueGG\(nhn\_\{h\},nvn\_\{v\}\)rollouts per task \(hidden, visible\)5 \(3, 2\)ϵ\\epsilonwithdrawal tolerance on the success rate0\.02α\\alphaadmission risk0\.05nminn\_\{\\min\}minimum records per decision12BBBayesian\-bootstrap draws2000–Thompson prior standard deviation0\.25–evidence window of the posterior \(cycles\)30–share of tasks that re\-test withdrawn skills0\.3γ\\gammadiscount when only the visible part succeeds0\.5λP\\lambda\_\{P\}penalty when all rollouts agree0\.2mmshrinkage strength2η\\etacompute\-factor exponent0\.5λ\\lambdaWorker\-cost weight in the Director reward0\.005τ\\tauProposer KL anchor1\.0–trust region on the Proposer’s tilt \(nats\)4\.0–Director KL anchor; PPO clip \(low, high\)0\.005; \(0\.2, 0\.28\)–learning rate \(Director, Proposer\), cosine decay10−510^\{\-5\},10−510^\{\-5\}–base model \(Director, Proposer, Workers\)Qwen3\.5\-9B–LoRA rank64–tasks per cycle24 \(2 lists of 8 per benchmark, 2 picks each\)–Director updates200–sampling temperature \(Director, Proposer, Worker\)1\.0, 1\.0, 0\.3Table 4:Hyperparameters of R2Orch\.
## Appendix DExperimental Details

#### Benchmarks and splits\.

The six IID benchmarks, HotpotQA\([Yang et al\., 2018](https://arxiv.org/html/2610.22740#bib.bib42)\), NQ\-Open\([Kwiatkowski et al\., 2019](https://arxiv.org/html/2610.22740#bib.bib16);[Lee et al\., 2019](https://arxiv.org/html/2610.22740#bib.bib17)\), MedQA\([Jin et al\., 2021](https://arxiv.org/html/2610.22740#bib.bib13)\), AIME, MBPP\+\([Austin et al\., 2021](https://arxiv.org/html/2610.22740#bib.bib2);[Liu et al\., 2023](https://arxiv.org/html/2610.22740#bib.bib20)\), and ALFWorld\([Shridhar et al\., 2020](https://arxiv.org/html/2610.22740#bib.bib36)\), supply the training tasks and the IID tests; AIME trains on the pooled problems of earlier years and is tested on AIME 2026\. The six OOD benchmarks, TriviaQA\([Joshi et al\., 2017](https://arxiv.org/html/2610.22740#bib.bib14)\), MuSiQue\([Trivedi et al\., 2022](https://arxiv.org/html/2610.22740#bib.bib39)\), GPQA\([Rein et al\., 2023](https://arxiv.org/html/2610.22740#bib.bib26)\), MATH\-Hard\([Hendrycks et al\., 2021](https://arxiv.org/html/2610.22740#bib.bib9)\), SWE\-Bench\([Jimenez et al\., 2024](https://arxiv.org/html/2610.22740#bib.bib12)\), and WebShop\([Yao et al\., 2022](https://arxiv.org/html/2610.22740#bib.bib43)\), are used only for testing\. Each test set has 128 items \(AIME 2026: 30\) disjoint from the training tasks, so one run’s 0/1 metric lies on ak/128k/128grid \(k/30k/30for AIME 2026\) and a five\-run mean on ak/640k/640grid \(k/150k/150\)\. The training pools and sizes are released with the code\. The training curves in Figure[4](https://arxiv.org/html/2610.22740#S5.F4)a use a separate held\-out set of 32 tasks from each IID dataset \(AIME: 30 problems of earlier years\), disjoint from the training tasks and distinct from the IID and OOD test sets\.

#### Metrics\.

HotpotQA, NQ\-Open, TriviaQA, and MuSiQue report exact match \(EM\) and F1 of the final answer, with Ans EM in Table[2](https://arxiv.org/html/2610.22740#S5.T2); MedQA, AIME 2026, GPQA, and MATH\-Hard report accuracy; MBPP\+ reports Pass@1; ALFWorld and WebShop report the success rate \(SR\); and SWE\-Bench reports the share of resolved issues\.

#### Baselines and fairness\.

Qwen3\.5\-9B\([Qwen Team, 2026](https://arxiv.org/html/2610.22740#bib.bib25)\)and DeepSeek\-V4\-Flash are prompted directly\. SFT and GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.22740#bib.bib32)\), the latter with dynamic sampling, train Qwen3\.5\-9B on the same training tasks, and each keeps the configuration whose trained weights score best under its default recipe\. The other baselines are workflow search \(AFlow\([Zhang et al\., 2025](https://arxiv.org/html/2610.22740#bib.bib47)\)\), RL\-trained orchestration \(AgentFlow, FlowSteer\([Li et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib18);[Zhang et al\., 2026c](https://arxiv.org/html/2610.22740#bib.bib48)\)\), skill internalization \(SKILL0, PATS\([Lu et al\., 2026b](https://arxiv.org/html/2610.22740#bib.bib23);[Shi et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib35)\)\), and self\-play with skill evolution \(SESA, MetaSkill\([Fu et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib6);[Wang et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib41)\)\), and each runs the best configuration reported in its paper\. All methods share the frozen Workers, training tasks, and compute budget; R2Orch is tested at zero skills\.

#### Ablation variants\.

Each ablation changes one component and keeps the rest of R2Orch fixed, including the Workers, the training tasks, the training rollouts, and the zero\-skill test\. In each group of Table[2](https://arxiv.org/html/2610.22740#S5.T2), the first row removes the whole component: GRPO only for Recursive Self\-Play Orchestration, a failure reward for Recursive Learnability, and a fixed scaffold for Skill Withdrawal\. The other rows remove one part of the orchestration \(the skills or the Proposer’s picks\), replace only the Proposer’s reward, or replace only the withdrawal rule, whose thresholds are tuned on validation\.

#### Proposer objectives\.

Figure[5](https://arxiv.org/html/2610.22740#S5.F5)a–e replace only the Proposer’s objective, by uniform sampling, dynamic sampling \(DAPO\([Yu et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib44)\)\), selective rollout \(GRESO\([Zheng et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib53)\)\), learnability on the current scaffold \(SFL\([Rutherford et al\., 2024](https://arxiv.org/html/2610.22740#bib.bib29)\)\), a solve\-rate reward \(Absolute Zero, AZR\([Zhao et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib52)\)\), or an uncertainty reward \(R\-Zero\([Huang et al\., 2026](https://arxiv.org/html/2610.22740#bib.bib10)\)\), and keep the Director update, the scaffold, the withdrawal rule, and the training rollouts fixed; the Uniform, DAPO, SFL, and R2Orch rows of Figure[5](https://arxiv.org/html/2610.22740#S5.F5)a repeat Table[2](https://arxiv.org/html/2610.22740#S5.T2)\. Every objective trains 24 groups of five rollouts per cycle, DAPO’s discarded rollouts count toward its cost, and scores are five\-run means at zero skills\.

#### Backbone transfer\.

For Figure[4](https://arxiv.org/html/2610.22740#S5.F4)e, the Director keeps the weights trained with Qwen3\.5\-9B Workers, and only the Workers are replaced by each of six other backbones, with skills off\. Frozen prompts the backbone directly, and \+R2Orch runs the same backbone under the trained Director\.

#### Training dynamics\.

Figure[4](https://arxiv.org/html/2610.22740#S5.F4)a evaluates the checkpoint of every training cycle on the held\-out set, once with the current skills and once with none\. Figure[4](https://arxiv.org/html/2610.22740#S5.F4)b counts the distinct graph types the Director builds over the six IID datasets\. A candidate task is unseen if it has never been attempted; otherwise its previous success rateppover its earlier rollouts makes it unsolved \(p=0p=0\), learnable \(0<p<10<p<1\), or solved \(p=1p=1\)\. Figure[4](https://arxiv.org/html/2610.22740#S5.F4)c sums the Proposer’s probability over the candidates of each class, and Figure[4](https://arxiv.org/html/2610.22740#S5.F4)d gives its probability on solved candidates as a moving average over 10 cycles, beside the untrained model with the same prompt and random choice \(the share of solved candidates\)\.

#### Selection evidence and withdrawals\.

Each candidate list of eight tasks yields two distinct picks \(Appendix[B](https://arxiv.org/html/2610.22740#A2)\)\. Figure[5](https://arxiv.org/html/2610.22740#S5.F5)e counts the 2,220 picks of seen tasks over cycles 50–199 by their previous success rate, and a new group is mixed if its rollouts contain at least one success and one failure\. These counts include both picks of each list, the second drawn from the remaining candidates \(Eq\. \([42](https://arxiv.org/html/2610.22740#A1.E42)\)\), whereas Figure[4](https://arxiv.org/html/2610.22740#S5.F4)d gives the policy’s probability on solved candidates as a moving average over 10 cycles\. Figure[5](https://arxiv.org/html/2610.22740#S5.F5)f measures the held\-out gain of a dataset’s three original seed cards, used together, over no cards, once at checkpoint 0 and as the mean over the 10 checkpoints around each withdrawal with its 95% interval\. Each row names the withdrawal cycle and the card;*restored*marks the MedQA topology card, which re\-testing restored after its withdrawal at cycle 6\. This is dataset\-level evidence; each withdrawal follows the posterior of the withdrawn card’s contrast \(Eq\. \([14](https://arxiv.org/html/2610.22740#S4.E14)\)\)\.

#### Hyperparameters\.

Table[4](https://arxiv.org/html/2610.22740#A3.T4)lists every constant that appears as a symbol in the main text; the PPO clip and the learning rate set the step sizeβ\\beta\.

#### Compute\.

R2Orch trains on A100 GPUs for 200 cycles, about 106 A100 hours measured in training, twice its 100\-cycle cost\. Figure[5](https://arxiv.org/html/2610.22740#S5.F5)c,d report the selection cost of the Proposer objectives: extra and total rollouts per cycle, rollouts and tokens per effective group, and A100 hours per 100 cycles\. An effective group mixes successes and failures, so its GRPO advantages are not all zero\.

#### Repeated runs\.

Table[1](https://arxiv.org/html/2610.22740#S4.T1)reports the mean and standard deviation over five runs, and Table[2](https://arxiv.org/html/2610.22740#S5.T2)and Figures[4](https://arxiv.org/html/2610.22740#S5.F4)e and[5](https://arxiv.org/html/2610.22740#S5.F5)a report five\-run means\.