RODS: Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

# Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents
Source: [https://arxiv.org/html/2606.19047](https://arxiv.org/html/2606.19047)
††footnotetext:∗This work was supported by Ant Group Research Intern Program\.††footnotetext:†Corresponding Authors\.Ruishan Fang1,2,4,∗Siyuan Lu1,2,3,4Chenyi Zhuang1,†Tao Lin4,1,†1Inclusion AI, Ant Group2Zhejiang University3Shanghai Innovation Institute4Westlake University

###### Abstract

Multi\-turn tool\-use RL is bottlenecked by the rapid depletion of informative samples in static datasets\. We observe that the gradient signal in GRPO concentrates on tasks with the highest rollout reward variance, a consequence of the Popoviciu upper bound\. Consequently, samples near the agent’s*capability boundary*—where successes and failures are roughly balanced—contribute disproportionately large policy gradients\. As training progresses, this boundary continuously shifts, which gradually depletes the pool of informative samples in a static dataset\. We proposeRODS\(Reward\-drivenOnlineDataSynthesis\) to resolve this depletion\.RODScloses the loop between RL training and data generation by repurposing the progress reward variance as a practical, zero\-cost boundary detector that requires no extra inference beyond the rollouts already computed for training\. It continuously identifies such boundary samples, synthesizes new multi\-turn variants matching their structural complexity \(e\.g\., API topology and dependency depth\) via a skill\-aligned resampling pipeline, and manages a dynamic replay buffer that co\-evolves with the policy\. Starting from 400 human seeds and maintaining an active training pool of∼\\sim800 samples,RODSachieves comparable performance to a 17K\-sample offline pipeline while requiring roughly20×20\\timesfewer trajectories, and improves over fixed\-data RL and environment augmentation in our controlled setting\.

![Refer to caption](https://arxiv.org/html/2606.19047v1/x2.png)Figure 1:Current limitations in static data training vs\.RODSdynamic synthesis paradigm\.\(a\) Data scarcity \(ℂ​1\\mathbb\{C\}1\):High\-quality multi\-turn tool\-use datasets require massive human annotation effort\.\(b\) Capability vs\. static Data \(ℂ​2\\mathbb\{C\}2\):As the model learns, static data becomes mastered, leading to a loss of gradient signal\.\(c\) Semantic disjointedness \(ℂ​3\\mathbb\{C\}3\):Naively stitching single\-turn queries creates disjointed interactions\.
\(d\) TheRODSsolution:A closed\-loop data engine continuously ingests static data to detect the evolving capability boundary, synthesizing coherent, targeted active corpora where the model needs them\.## 1Introduction

Large Language Model \(LLM\)\-based agents\(wang2024survey,weng2023agent\)have demonstrated strong potential in solving complex tasks by using external tools and environment interactions\(anthropic2026claude,openai2026gpt54thinking\)\. Recent work frames tool\-use agent training as an RL problem \(Agentic RL\), optimizing policies through direct environment interaction\(jin2025search,feng2025retool,wang2025ragen,liu2025exploratory\)\. However, extending Agentic RL to multi\-turn tool\-use environments introduces unique challenges that have not been fully addressed by existing methods, as it simultaneously demands large training corpora and long\-horizon structural coherence\(patil2025bfcl,tbench,tau2bench\)\.

Current RL systems face three main challenges in this regime \(see Figure[1](https://arxiv.org/html/2606.19047#S0.F1)\)\. First, high\-quality multi\-turn datasets are scarce \(ℂ​1\\mathbb\{C\}1\) due to prohibitive annotation and validation costs, as seen in BFCL V3\(patil2025bfcl\)which contains only 800 samples\. Second, as agent capabilities evolve, static datasets suffer from shifting boundaries \(ℂ​2\\mathbb\{C\}2\), leading to signal depletion and wasted compute on mastered or unreachable tasks\(li2025adacurl,dai2025harder\)\. Third, on\-the\-fly synthesis often causes semantic disjointedness \(ℂ​3\\mathbb\{C\}3\) by lacking a unifying goal, which produces trajectories without coreference or coherence that fail to teach reliable reasoning\.

Despite these hurdles, existing efforts generally fall into two categories: large\-scale offline synthesis and online environment augmentation\. Offline pipelines\(prabhakar2025apigen,toucan,xu2025funreasonmt\)address data scarcity \(ℂ​1\\mathbb\{C\}1\) by generating massive corpora upfront, yet they remain decoupled from the training loop and thus fail to track the model’s evolving capability boundary \(ℂ​2\\mathbb\{C\}2\)\. Conversely, RL training methods like EnvTuning\(tuneenv\)enable learning from minimal data but are ultimately constrained by the signal depletion of their fixed seed corpora \(ℂ​2\\mathbb\{C\}2\)\. While online self\-play and self\-evolution approaches\(toolr0,li2025closeloop,zhai2025agentevolver\)theoretically close this loop, unconstrained zero\-data generation frequently fails to maintain semantic coherence in complex multi\-turn settings \(ℂ​3\\mathbb\{C\}3\), leading to disjointed trajectories that offer little pedagogical value\.

*How can we train multi\-turn tool\-use agents under extremedata scarcity, by dynamically synthesizing data that strictly tracks the evolvingcapability boundary while maintainingmulti\-turn semantic coherence?*

We proposeRODS\(Reward\-drivenOnlineDataSynthesis\) to bridge this gap, a framework that tightly couples data generation with the RL training loop\. Our approach rests on a simple insight: the Progress Reward in policy gradient methods like GRPO serves as a boundary detector that reuses existing rollout statistics—since rollouts are already computed for advantage estimation—because rollout variance is highest near the agent’s capability boundary \(μ≈0\.5\\mu\\approx 0\.5, as suggested by the Popoviciu upper bound\)\. By synthesizing data in this high\-variance region and managing its lifecycle through a co\-evolving replay buffer,RODSmaintains a continuous stream of gradient\-informative samples\. This paper makes three main contributions\. To address signal starvation inherent to static datasets \(ℂ​2\\mathbb\{C\}2\), we proposeRODS, a reward\-driven boundary expansion method that repurposes the RL progress reward to identify and expand boundary tasks in real time\. To preserve multi\-turn semantic coherence \(ℂ​3\\mathbb\{C\}3\), we introduce a skill\-aligned resampling synthesis pipeline that anchors novel trajectories to the complexity profiles of verified seeds\. Instead of simple entity substitution, it preserves the functional dependency structure of the seed while generating novel narratives and environment states\. Finally, we demonstrate significant data efficiency \(ℂ​1\\mathbb\{C\}1\): starting from 400 human seeds and maintaining an active training pool of∼\\sim800 samples,RODSachieves performance comparable to a 17K\-sample offline pipeline \(using roughly20×20\\timesless data\) and improves over fixed\-data RL in our controlled setting\.

## 2Related Work

#### Data synthesis for tool use\.

Although high\-quality human\-annotated benchmarks\(apibank,tbench,tau2bench,acebench,travelplanner,funcbenchgen\)provide rigorous evaluation protocols for multi\-turn tool use, their limited scale of typically hundreds of instances is insufficient for training reliable agentic policies\. This data scarcity has prompted a shift toward large\-scale offline synthesis\. Frameworks such as APIGen\-MT\(prabhakar2025apigen\), TOUCAN\(toucan\), and Magnet\(magnet\)prioritize corpus scale and structural complexity, often generating millions of trajectories upfront\. While effective for pre\-training, these static pipelines remain decoupled from the training process, producing uniform data distributions that cannot track the model’s evolving capability boundary \(ℂ​2\\mathbb\{C\}2\)\.

Directed synthesis methods attempt to narrow this focus by targeting specific failure modes\. FunReason\-MT\(xu2025funreasonmt\)utilizes environment\-API graphs and advanced tool\-query synthesis to tackle hard query generation, while LoopTool\(looptool\)employs a feedback loop to correct algorithmic errors\. Although these represent a clear advance in targeting known weaknesses, they remain offline snapshots that cannot adapt to the shifting capability boundary during active RL training\.RODSaddresses this limitation by introducing a reward\-driven synthesis loop that operates*during*training, ensuring that the generated data remains at the model’s immediate capability boundary\.

#### Online RL and curriculum learning\.

Data efficiency in agentic RL is traditionally addressed through environment simulation or corrective feedback\. Simulation\-based approaches like ScaleEnv\(scaleenv\), Agent World Model\(agentworldmodel\), and Simia\-Agent\(simulatingenv\)construct high\-fidelity interactive loops to maximize the signal extracted from existing tasks\. EnvTuning\(tuneenv\)further improves efficiency by orchestrating a four\-stage curriculum with actionable environment augmentation\. However, these methods are constrained by the fixed diversity of their seed corpora; as the agent improves, the proportion of gradient\-informative samples in the static pool shrinks\.

Self\-play paradigms\(toolr0,selfplayevolve\)attempt to resolve this via absolute zero\-data generation, where a generator proposes tasks from scratch\. While theoretically appealing, such unconstrained generation frequently struggles with the long\-horizon logical chains and interdependent API calls required for multi\-turn tool use \(ℂ​3\\mathbb\{C\}3\)\.RODStakes a highly complementary approach by using a critically small set of human data as*structural anchors*\. By reusing the inherent rollout variance of GRPO as a boundary detector and grounding synthesis in skill\-aligned resampling,RODSenables a targeted, complexity\-aligned curriculum that bypasses the high\-variance search of generating multi\-turn logic from scratch\. Our approach aligns with prioritized data selection methods, such as prioritized experience replay\(schaul2016prioritized\), hard\-example mining\(shrivastava2016training\), and competence\-based curricula\(platanios2019competence\), which focus learning on the most informative samples\. The key distinction is that these classical methods select or re\-weight*existing*experiences, whereasRODSgenerates*new*data at the reward\-defined boundary, combining the targeting principle of prioritized replay with the distributional expansion of generative synthesis\.

## 3RODS

To resolve multi\-turn data scarcity and signal depletion, we introduceRODS, a reward\-driven data synthesis framework\.RODSmaintains a saturated learning signal through three co\-evolving modules\. First, reward\-based seed detection identifies high\-variance boundary tasks \(§[3\.2](https://arxiv.org/html/2606.19047#S3.SS2)\)\. Second, skill\-aligned synthesis generates coherent, structurally\-isomorphic variants \(§[3\.3](https://arxiv.org/html/2606.19047#S3.SS3)\)\. Third, dynamic replay buffer management tracks the shifting capability boundary \(§[3\.4](https://arxiv.org/html/2606.19047#S3.SS4)\)\.

![Refer to caption](https://arxiv.org/html/2606.19047v1/x3.png)Figure 2:TheRODSclosed\-loop RL\-data synthesis architecture\.\(Top\)*Reward Calculation & Agent Training:*The agent trains on a mixed dataset via GRPO; the Progress Reward \(RPR\_\{P\}\) identifies boundary seeds and feeds them back to the data engine\.\(Bottom\-left\)*Plan & Execute:*The Planner Agent selects a function sequence from the API graph; the Execution Orchestrator instantiates it on a simulation environment with environment feedback \(Repeat×N\\times N\), producing an executable raw trajectory\.\(Bottom\-right\)*Refine & Judge:*A Query Agent converts the trajectory into per\-turn natural\-language queries; a Rewrite Agent grounds all queries in the Planner’s narrative for cross\-turn coherence; a Critique Agent validates semantic quality with a feedback loop \(Repeat×N\\times N\), yielding validated synthetic datasets that are injected into training\.### 3\.1Problem Formulation and Design Motivation

Multi\-turn tool use is formalized as a POMDP where an agent resolves interdependent queries through API calls and environment feedback \(details in Appendix[A](https://arxiv.org/html/2606.19047#A1)\)\. We use GRPO\(shao2024deepseekmath\)for optimization, though our variance\-based boundary tracking extends to other trajectory\-sampling methods like PPO \(see Appendix[K](https://arxiv.org/html/2606.19047#A11)\)\.

#### Reward sparsity and progress reward\.

Because sparse binary rewards fail to assign credit across complex long\-horizon trajectories\(feng2025group\), we adopt the progress reward \(RP∈\[0,1\]R\_\{P\}\\in\[0,1\]\) fromtuneenv:RP=1/N​∑t=1N\(rtstate⋅rtexec\)R\_\{P\}=\\nicefrac\{\{1\}\}\{\{N\}\}\\sum\_\{t=1\}^\{N\}\(r\_\{t\}^\{\\text\{state\}\}\\cdot r\_\{t\}^\{\\text\{exec\}\}\)\. This assigns continuous credit for partial completion, enriching the advantage signal\. Note that ground truth is used exclusively for simulation\-based reward computation; the policy never observes it\.

#### Design heuristic: variance peaks near the capability boundary\.

In policy gradient methods like GRPO\(shao2024deepseekmath\), the gradient signal density of a sample is governed by its rollout reward variance\. While Popoviciu’s inequality\(popoviciu1935sur\)shows that the variance of a bounded variable is upper\-bounded byμ​\(1−μ\)\\mu\(1\-\\mu\)\(maximizing atμ≈0\.5\\mu\\approx 0\.5\), this provides a theoretical upper bound rather than an exact description of the continuous reward variance\. We nonetheless validate this as an*empirically supported heuristic*: the continuous progress reward variance peaks near the capability boundary \(μ≈0\.5\\mu\\approx 0\.5\), as confirmed by the per\-task variance analysis in Figure[3](https://arxiv.org/html/2606.19047#S4.F3)\(right\)\. Thus, this heuristic underpins the design ofRODS: identifying these boundary samples serves as an effective, cheap proxy for finding the highest concentration of informative gradient signals \(see Appendix[B\.5](https://arxiv.org/html/2606.19047#A2.SS5)for theoretical details\)\.

### 3\.2Reward\-Based Seed Data Detection

To operationalize the boundary\-targeting heuristic identified in §[3\.1](https://arxiv.org/html/2606.19047#S3.SS1), we use the average Progress Rewardr¯i\\bar\{r\}\_\{i\}acrossKKrollouts as a density\-based probe to partition the task space𝒟\\mathcal\{D\}into three dynamically evolving regions:

𝒟mastered=\{xi:r¯i\>α\+\},𝒟boundary=\{xi:α−≤r¯i≤α\+\},𝒟hard=\{xi:r¯i<α−\},\\mathcal\{D\}\_\{\\text\{mastered\}\}=\\\{x\_\{i\}:\\bar\{r\}\_\{i\}\>\\alpha^\{\+\}\\\}\\,,\\mathcal\{D\}\_\{\\text\{boundary\}\}=\\\{x\_\{i\}:\\alpha^\{\-\}\\leq\\bar\{r\}\_\{i\}\\leq\\alpha^\{\+\}\\\}\\,,\\mathcal\{D\}\_\{\\text\{hard\}\}=\\\{x\_\{i\}:\\bar\{r\}\_\{i\}<\\alpha^\{\-\}\\\}\\,,\(1\)whereα−\\alpha^\{\-\}andα\+\\alpha^\{\+\}are boundary thresholds \(set to0\.200\.20and0\.850\.85respectively, see Appendix[B](https://arxiv.org/html/2606.19047#A2)\)\. We define the tasks within the intermediate regionxi∈𝒟boundaryx\_\{i\}\\in\\mathcal\{D\}\_\{\\text\{boundary\}\}asboundary seeds, as they represent the model’s immediate capability boundary and harbor the highest potential for informative gradient signals\.

#### Type\-quota seed emission\.

At each training step, we select up toMMseeds from𝒟boundary\\mathcal\{D\}\_\{\\text\{boundary\}\}\. To ensure diverse skill coverage, we enforce a per\-type quotaMτM\_\{\\tau\}\(e\.g\., base/long\-context, missing\-function, missing\-parameter\) such that∑τMτ=M\\sum\_\{\\tau\}M\_\{\\tau\}=M\. Within each type, candidates are ranked by the variance\-proxyϕ​\(r¯i\)=4​r¯i​\(1−r¯i\)\\phi\(\\bar\{r\}\_\{i\}\)=4\\bar\{r\}\_\{i\}\(1\-\\bar\{r\}\_\{i\}\)in descending order, prioritizing seeds closest to the capability midpoint \(r¯i=0\.5\\bar\{r\}\_\{i\}=0\.5\) where gradient signal potential is highest \(cf\.ℂ​2\\mathbb\{C\}2\)\. A temporal exclusion window ofccsteps prevents redundant sampling, and the selected seeds are asynchronously dispatched to the synthesis pipeline\.

### 3\.3Skill\-Aligned Data Synthesis

The synthesis pipeline transforms boundary seeds into novel, structurally valid variants that preserve the informative complexity of the original task \(see Appendix[P](https://arxiv.org/html/2606.19047#A16)for detailed seed\-to\-variant examples\)\. Rather than simple paraphrasing, we enforce*structural similarity*: we extract the complexity profileΦ​\(xseed\)\\Phi\(x\_\{\\text\{seed\}\}\)\(e\.g\., the directed acyclic graph of API dependencies and parameter flows\) and sample a new taskx′∼p\(⋅\|Φ\(xseed\)\)x^\{\\prime\}\\sim p\(\\cdot\|\\Phi\(x\_\{\\text\{seed\}\}\)\)such thatΦ​\(x′\)\\Phi\(x^\{\\prime\}\)approximatesΦ​\(xseed\)\\Phi\(x\_\{\\text\{seed\}\}\)in dependency depth and API topology\. This constraint ensures thatx′x^\{\\prime\}has structural difficulty similar to the seed, placing it near the capability boundary \(ℂ​2\\mathbb\{C\}2\) as validated by the injection reward analysis in Figure[3](https://arxiv.org/html/2606.19047#S4.F3)\(middle\), while forcing the model to generalize across novel abstract logic and environment states, preventing overfitting to static execution paths \(ℂ​3\\mathbb\{C\}3\)\. We implement this via a five\-stage multi\-agent pipeline \(see Appendix[C](https://arxiv.org/html/2606.19047#A3)for full prompts\): In Stage I \(schema\-guided planning, addressingℂ​2\\mathbb\{C\}2andℂ​3\\mathbb\{C\}3\), the planner agent ingestsxseedx\_\{\\text\{seed\}\}to design an execution plan𝒮′\\mathcal\{S\}^\{\\prime\}and an underlying narrative𝒩\\mathcal\{N\}, using failure histories \(ℱavoid\\mathcal\{F\}\_\{\\text\{avoid\}\}\) to bypass known execution bottlenecks\. In Stage II \(feedback\-driven execution, addressingℂ​1\\mathbb\{C\}1\), an execution orchestrator instantiates𝒮′\\mathcal\{S\}^\{\\prime\}in a simulated environment𝒞0\\mathcal\{C\}\_\{0\}to produce a trajectory with ground\-truth dependencies\. An error critic applies multi\-tier mitigation to find a valid instantiation acrossKmax=3K\_\{\\max\}\\\!=\\\!3attempts, while a query agent maps each turn to natural\-language queries without exposing function details\. In Stage III \(holistic semantic grounding, addressingℂ​3\\mathbb\{C\}3\), a rewrite agent renders the entire multi\-turn trajectory simultaneously based on𝒩\\mathcal\{N\}\. Unlike greedy turn\-by\-turn generation, this approach anchors all turns to a single goal, resolving semantic disjointedness and ensuring natural conversational flow\. In Stage IV \(critique and refinement\), variants undergo rule\-based checks and LLM scoring via a critique agent\. A feedback loop with the rewrite agent prunes logic flaws and fixes phrasing\. Finally, Stage V \(optional adversarial augmentation\) injects structural exceptions, such as missing tools or blurred parameters, to force clarification turns and improve out\-of\-distribution robustness\.

### 3\.4Dynamic Replay Buffer Management

To maintain a high\-fidelity training distribution, we implement a dual\-control lifecycle that balances*expansion flow*with*stock relevance*\.

#### Expansion flow control: staged injection\.

To prevent instability from abrupt distributional shifts—where a sudden influx of synthetic data destabilizes RL gradients—we adopt a staged injection protocol\. Synthesized variants are initially held in an asynchronous candidate queue and merged into the active replay pool strictly at epoch boundaries\. Specifically, tasks seeded during epochnnare staged and injected at the start of epochn\+1n\{\+\}1\. To ensure gradual integration, the per\-epoch injection volume is capped atβ⋅\|𝒟active\|\\beta\\cdot\|\\mathcal\{D\}\_\{\\text\{active\}\}\|; any excess is deferred to a persistent staging buffer for subsequent epochs\. Upon each injection, the sampling distribution is re\-initialized to ensure uniform coverage of the expanded pool\.

#### Active stock management: multi\-layer retirement\.

We maintain pool informativeness through a three\-layer retirement mechanism that tracks the shifting capability boundary\. The first layer, burn\-in filtering, discards new variants with initial rewards belowϵtrial\\epsilon\_\{\\text\{trial\}\}to prune tasks beyond the immediate exploration horizon\. The second layer, boundary\-drift eviction, removes tasks that have drifted into mastered \(r¯i\>αretire\+\\bar\{r\}\_\{i\}\>\\alpha^\{\+\}\_\{\\text\{retire\}\}\) or unsolvable \(r¯i<αretire−\\bar\{r\}\_\{i\}<\\alpha^\{\-\}\_\{\\text\{retire\}\}\) zones afternminn\_\{\\text\{min\}\}observations\. The third layer applies variance\-prioritized pruning: if the pool exceedsPmaxP\_\{\\max\}, samples are retired based on reward varianceϕ​\(r¯i\)=4​r¯i​\(1−r¯i\)\\phi\(\\bar\{r\}\_\{i\}\)=4\\bar\{r\}\_\{i\}\(1\-\\bar\{r\}\_\{i\}\), preserving the highest gradient signals \(ℂ​2\\mathbb\{C\}2\)\. We also prune generated variants that remain unsampled for an extended period to prevent stale data accumulation\. This dual\-control architecture ensures the replay buffer remains a high\-fidelity mirror of the capability boundary while maintaining the stability of the RL optimization loop\. We employ default hyperparameters across all settings to ensure a reliable and fair evaluation; details are provided in Appendix[J](https://arxiv.org/html/2606.19047#A10)\.

## 4Experiments

Our experiments are designed to answer five questions:\(Q1\)Does boundary\-targeted expansion outperform both fixed\-data RL and environment augmentation?\(Q2\)Does boundary\-expanded training generalize to OOD tasks?\(Q3\)How doesRODSdynamically expand the data space without catastrophic distribution shifts?\(Q4\)How data\-efficient isRODScompared to large\-scale offline synthesis?\(Q5\)Which components ofRODScontribute most to performance?

#### Benchmark\.

We evaluate on the multi\-turn subset of BFCL V3\(patil2025bfcl\), which includes 800 samples across four balanced splits:Base,Missing Function,Missing Parameter, andLong\-Context\. Followingtuneenv, we reserve 400 samples \(100 per split\) for training and use the remaining 400 for held\-in evaluation\. For OOD testing, we adopt the BFCL V4 multi\-turn tracks,τ2\\tau^\{2\}\-bench\(tau2bench\), and the ACEBench Agent split\(acebench\)as our held\-out test sets\. Detailed benchmark and evaluation details are provided in Appendix[O](https://arxiv.org/html/2606.19047#A15)\.

#### Training configuration\.

We train Qwen3\-4B\-Instruct using GRPO\(shao2024deepseekmath\)\(K=16K\\\!=\\\!16rollouts\) on8×8\\timesA100 GPUs via a three\-stage curriculum \(shared across baselines for fair comparison, details in Appendix[J](https://arxiv.org/html/2606.19047#A10)\)\. Stage transitions occur when validation performance plateaus \(changing<<1% over one full epoch\) and gradient norms converge, following the protocol oftuneenv\. The protocol isolates syntactic and logical acquisition: \(1\)Stage 1 \(Format\):Training on 100Basesamples using a format rewardRformatR\_\{\\text\{format\}\}to isolate syntactic acquisition \(XML, function names, arguments\) before task reasoning \(see Appendix[H](https://arxiv.org/html/2606.19047#A8)\); \(2\)Stage 2 \(Base Reasoning\):Continuing on theBasesplit with the Progress RewardRPR\_\{P\}to build a stable reasoning anchor without expansion; and \(3\)Stage 3 \(Full data \+ expansion\):Scaling to 400 samples across all splits\. ForRODS, the synthesis engine targets high\-variance boundary tasks, using the Stage 2 foundation to drive generalization\. To evaluate cross\-model generalizability, we also report results on Qwen2\.5\-7B\-Instruct and Llama\-3\.1\-8B\-Instruct\.

#### Data synthesis\.

TheRODSsynthesis pipeline uses Qwen3\-32B deployed via vLLM on a separate8×8\\timesA100 cluster running asynchronously alongside training\. It achieves a seed\-to\-injection latency of∼\\sim1 training step, introducing no idle time into the RL loop \(boundary thresholdsα−=0\.20,α\+=0\.85,Pmax=400\\alpha^\{\-\}\\\!=\\\!0\.20,\\alpha^\{\+\}\\\!=\\\!0\.85,P\_\{\\max\}\\\!=\\\!400\)\. Full configuration and detailed cost breakdown are provided in Appendix[J](https://arxiv.org/html/2606.19047#A10)and[N](https://arxiv.org/html/2606.19047#A14)\.

#### Baselines and references\.

We structure our comparisons into two tiers\.Tier 1: Controlled RL comparisons\(consistent 400\-sample seeds\) includesStatic dataset\(baseline trained only on the fixed seed set without generation\),EnvTuning\(tuneenv\)\(actionable environment enrichment\), andRODS\(our full dynamic boundary expansion system\)\.Tier 2: Data efficiency references\(varying scales\) includesFunReason\-MT\-4B\(xu2025funreasonmt\)\(20×20\\timesmore data\) and state\-of\-the\-art models like GPT\-4o and DeepSeek\-V3\.2\-Exp\.

### 4\.1Results \(Q1\): Boundary Expansion vs\. Fixed\-Data RL and Environment Augmentation

Table 1:In\-distribution performance on BFCL V3 multi\-turn\(Tier 1: controlled RL comparisons\)\. All RL methods share the same 400 training samples and GRPO setup\. FunReason\-MT\-4B \(Tier 2\) is included for data\-scaling reference\. For a detailed comparison with 20\+ models, see Appendix[Q](https://arxiv.org/html/2606.19047#A17)\.Red textindicates improvement over the base model\.Table[1](https://arxiv.org/html/2606.19047#S4.T1)presents the controlled comparison among Tier 1 methods\. All three methods share the same 400 training samples, the same GRPO configuration, and the same progress reward; the only variable is the strategy for addressing gradient signal depletion in Stage 3\.

#### Boundary expansion achieves the best results in our controlled setting\.

RODSattains the highest overall scores across the three model families in our experiments\. On the Qwen3\-4B\-Instruct base model,RODSimproves overall multi\-turn performance by\+33\.87%\(reaching 56\.00%\), surpassing both the “Static dataset” baseline \(50\.00%\) and the EnvTuning baseline \(50\.50%\)\. This suggests that dynamically synthesized boundary data provides complementary gradient signals beyond what fixed datasets or enriched environment feedback alone supply\. The gains are consistent across all four sub\-splits, indicating that boundary targeting benefits diverse task complexities rather than overfitting to a specific category\.RODSachieves comparable performance to large\-scale offline synthesis \(FunReason\-MT\-4B\) while using roughly20×20\\timesfewer trajectories\. Furthermore, it improves over fixed\-data RL and environment augmentation in our reported runs under the same 400\-sample controlled setup\. WhileRODSalso achieves competitive benchmark performance against larger models \(e\.g\., DeepSeek\-V3\.2\), we emphasize that the controlled comparisons within the same 400\-sample setup constitute the primary evidence for the mechanism’s efficacy\.

#### Data expansion vs\. environment augmentation: two orthogonal strategies\.

The comparison betweenRODSand EnvTuning is informative, as both address the same problem \(gradient sparsity under data scarcity\) but via opposite mechanisms\. EnvTuning enriches the*feedback signal*on existing data by providing corrective hints upon failure, effectively extracting more value from each fixed sample\.RODSinstead expands*the data distribution itself*by injecting new variants precisely at the capability boundary\.

The observed empirical advantage ofRODS\(e\.g\., a \+5\.50% absolute lead over EnvTuning on Qwen3\-4B\) suggests that expanding the data distribution at the capability boundary is more effective than deepening feedback on fixed samples in extreme data\-scarce RL settings\. The structural diversity of boundary\-targeted variants provides gradient signals that an exhausted static pool cannot\. BecauseRODSachieves this without any structural modification to the training environment’s feedback mechanism, it is environment\-agnostic and readily applicable to new API domains\. We investigate whether the two strategies are complementary in Appendix[I](https://arxiv.org/html/2606.19047#A9)\.

#### OOD generalization \(Q2\)\.

We evaluate on BFCL V4,τ2\\tau^\{2\}\-bench, and the ACEBench Agent split to test whether boundary\-expanded training yields generalizable reasoning or merely in\-distribution pattern matching\. The OOD improvements ofRODSover the base model \(detailed in Appendix[L](https://arxiv.org/html/2606.19047#A12)\) are consistent with our design hypothesis in Section[3\.3](https://arxiv.org/html/2606.19047#S3.SS3): by preserving API execution plans while heavily randomizing surface variables and environment states,*structural isomorphism*prevents the policy from overfitting to specific textual cues, forcing it to internalize abstract, generalizable multi\-turn reasoning patterns\.

### 4\.2Mechanism validation \(Q3\) via Data Space Evolution

![Refer to caption](https://arxiv.org/html/2606.19047v1/x4.png)

![Refer to caption](https://arxiv.org/html/2606.19047v1/x5.png)

Figure 3:TheRODSdynamic synthesis mechanism during stage 3 training\.\(Left\)Evolution of the data space partition\.RODSbreaks the static capacity limit \(400400tasks\) by continuously synthesizing active boundary data and evicting mastered tasks, effectively expanding the training curriculum without exploding memory\.\(Middle\)Reward trajectory of newly synthesized variants upon injection\. The continuous data generation strictly anchors the mean progress reward within the boundary zone \(\[0\.25,0\.75\]\[0\.25,0\.75\]\), confirming successful targeting of the capability frontier\.\(Right\)Empirical validation of the variance heuristic\. Across 4,800 per\-task measurements \(K=16K\\\!=\\\!16rollouts each\), rollout reward variance in the boundary zone is2\.02\.0–2\.2×2\.2\\timeshigher than in the low\-reward or high\-reward regions, confirming that gradient signal concentrates near the capability boundary\.#### Breaking the static capacity limit via boundary anchoring\.

Figure[3](https://arxiv.org/html/2606.19047#S4.F3)illustrates the internal mechanics driving the data efficiency reported in Table[1](https://arxiv.org/html/2606.19047#S4.T1)\. Rather than accumulating data indefinitely,RODStreats the active training pool as a sliding window over the capability space\. The left panel shows how the system breaches the 400\-task static limit by introducing new variants while retiring mastered ones, generating over 800 unique tasks in total while keeping the active pool bounded byPmaxP\_\{\\max\}\. The middle panel confirms that newly synthesized variants land inside the boundary zone \(RP∈\[0\.25,0\.75\]R\_\{P\}\\in\[0\.25,0\.75\]\)\. The right panel empirically validates the underlying heuristic: across 4,800 per\-task measurements, rollout reward variance in the boundary zone is2\.02\.0–2\.2×2\.2\\timeshigher than in the mastered or too\-hard regions, confirming that gradient signal concentrates near the capability boundary and thus alleviating gradient starvation \(ℂ​2\\mathbb\{C\}2\)\.

### 4\.3Data efficiency analysis \(Q4\): boundary targeting vs\. blind scaling

Our central claim is that*where*to synthesize data matters more than*how much*\. We evaluate this by \(a\) scaling the boundary\-targeted data pool inRODS, and \(b\) comparing its efficiency against FunReason\-MT\(xu2025funreasonmt\), a state\-of\-the\-art large\-scale offline synthesis pipeline\.

#### Scaling the maximum generated pool size \(PmaxP\_\{\\max\}\)\.

We varyPmax∈\{0,50,100,200,400\}P\_\{\\max\}\\\!\\in\\\!\\\{0,50,100,200,400\\\}on Qwen3\-4B\-Instruct while maintaining the standard three\-stage curriculum\.

![Refer to caption](https://arxiv.org/html/2606.19047v1/x6.png)Figure 4:Data scaling analysis\.Performance on BFCL V3 as a function of the generated pool capPmaxP\_\{\\max\}\.Pmax=0P\_\{\\max\}\\\!=\\\!0corresponds to the static baseline \(static pool of 400 items\)\. \(Left\) Performance across sub\-splits \(Base, Missing Functions, Missing Parameters, Long\-Context\)\. \(Right\) The overall average performance \(solid orange\) steadily improves asPmaxP\_\{\\max\}increases, with specific average scores annotated\.- •EvenPmax=50P\_\{\\max\}\\\!=\\\!50\(∼\\sim12% expansion\) yields a meaningful improvement over the static baseline, demonstrating the high marginal value of a small amount of targeted boundary data\.
- •Performance scales withPmaxP\_\{\\max\}but exhibits diminishing returns beyondPmax=200P\_\{\\max\}\\\!=\\\!200, as the boundary region of the 400 original samples becomes fully covered\.
- •This validatesPmax=400P\_\{\\max\}\\\!=\\\!400as a practical operating point and establishes that boundary data has a significantly higher per\-sample training value than uniformly sampled data\.

#### Comparison with large\-scale offline synthesis\.

To contextualize the data efficiency of boundary targeting, we compare against FunReason\-MT\-4B\(xu2025funreasonmt\)in Table[1](https://arxiv.org/html/2606.19047#S4.T1), which was trained on 17K offline trajectories\.RODSachieves a highly competitive 56\.00% overall utilizing an active training pool of∼\\sim800 samples \(400 original seeds \+ up toPmax=400P\_\{\\max\}=400generated variants\), matching FunReason\-MT on Long\-Context and significantly exceeding it on Missing Functions and Missing Parameters\. This represents an∼\\sim20×20\\timesreduction in data volume to achieve comparable or superior performance, demonstrating that boundary\-targeted synthesis is a highly data\-efficient alternative to massive offline corpora\.

We further isolate the value of boundary\-aware seed selection from the mere benefit of additional data volume via an ablation on random expansion \(Table[2](https://arxiv.org/html/2606.19047#S4.T2), row “w/ random seed selection”\)\.

### 4\.4Ablation study \(Q5\)

We ablate the three pillars ofRODS—boundary detection, synthesis pipeline, and lifecycle management—to quantify their individual contributions \(Table[2](https://arxiv.org/html/2606.19047#S4.T2)\)\. All ablations use Qwen3\-4B\-Instruct withPmax=400P\_\{\\max\}\\\!=\\\!400under the same three\-stage curriculum\. Detailed ablation configuration descriptions are provided in Appendix[R](https://arxiv.org/html/2606.19047#A18)\.

Table 2:Ablation study on BFCL V3\.Each row modifies one component ofRODSwhile keeping the rest intact\.Δ\\Delta: change relative to the full system\.ConfigurationBFCL V3 Multi TurnΔ\\DeltaAvg\.Avg\.BaseM\. FuncM\. ParamL\. CtxtRODS\(full system\)56\.0068\.0059\.0044\.0053\.00—*\(a\) Boundary detection*w/ random seed selection51\.2563\.5052\.5037\.0052\.00↓\\downarrow4\.75w/ binary acc instead of progress reward52\.7565\.0055\.0036\.5054\.50↓\\downarrow3\.25*\(b\) Synthesis pipeline*w/o coherence rewrite50\.8763\.0052\.0036\.0052\.50↓\\downarrow5\.13w/o narrative planning52\.3764\.5054\.0042\.0049\.00↓\\downarrow3\.63w/o feedback loop \(blind retry\)53\.8766\.0057\.0040\.5052\.00↓\\downarrow2\.13*\(c\) Lifecycle management*w/o retirement mechanism52\.6264\.5055\.0039\.0052\.00↓\\downarrow3\.38w/ static pool \(no dynamic refresh\)53\.1265\.5056\.0039\.5051\.50↓\\downarrow2\.88

#### Key findings\.

\(a\)Removing coherence rewrite \(−5\.13%\-5\.13\\%\) causes the largest overall drop, collapsing the Quality Judge pass rate from∼\\sim63% to∼\\sim12% and drastically reducing usable variants\. Among boundary detection ablations, random seed selection \(−4\.75%\-4\.75\\%\) confirms that boundary targeting—not merely additional data—drives improvement\. Replacing progress reward with binary accuracy \(−3\.25%\-3\.25\\%\) further validates the importance of continuous credit for boundary identification\.\(b\)The synthesis pipeline is robust to backbone choice: replacing Qwen3\-32B with GLM\-4\.5\-Air yields only−0\.75%\-0\.75\\%\(see Appendix[M](https://arxiv.org/html/2606.19047#A13)\)\.\(c\)Disabling retirement \(−3\.38%\-3\.38\\%\) confirms that mastered data accumulation dilutes gradient signal; continuous pool refresh is necessary\.

## 5Conclusion

Training multi\-turn tool\-use agents via RL faces a tension between data scarcity and signal relevance: static datasets lose informativeness as the agent improves\. This paper introducesRODS, a framework that recasts this bottleneck as a dynamic curriculum design problem\. Because progress reward variance peaks near the agent’s shifting capability boundary,RODSreuses this signal as a practical heuristic for boundary detection to sustain informative policy gradients throughout training\. Combined with a narrative\-driven, structurally isomorphic synthesis pipeline, our approach achieves competitive in\-distribution and OOD improvements using roughly20×20\\timesless data than massive offline pipelines\. These results suggest that boundary\-targeted synthesis achieves higher per\-sample training value than uniformly scaled static corpora, though at additional synthesis compute cost\.

Limitations & Future Work\.WhileRODSprovides a highly efficient curriculum, its current synthesis pipeline relies on deterministic simulation environments \(implemented via executable Python objects\) to verify execution correctness and provide feedback\. Adapting this framework to inherently opaque environments or remote Model Context Protocol \(MCP\) servers remains an area for refinement\. Future work will explore extending our simulation abstraction to robustly wrap and interact with stateful MCP endpoints, allowing the synthesis engine to safely capture input\-observation dynamics without direct access to the underlying internal state\. In addition, investigating multi\-backbone synthesis ensembles to inject diverse structural priors at the boundary presents a promising direction for scaling agentic capabilities\.

## References

## Appendix AFormulation of the Sequential Multi\-turn Decision Process

We formalize the multi\-turn tool\-use problem within the framework of a Partially Observable Markov Decision Process \(POMDP\)\(williams2007reinforcement\)\. In this context, a single episode represents an entire user task, comprising a series of predefined sequential instructions, referred to as turns\.

Formally, we represent the sequence of user instructions asq1,q2,…,qNq\_\{1\},q\_\{2\},\\dots,q\_\{N\}\. The episode initiates with the initial observationo0o\_\{0\}, which encapsulates the first instructionq1q\_\{1\}alongside the documentation of accessible tools\. To fulfill this instruction, a sequence of operations is executed\. At each timesteptt, an actionata\_\{t\}is sampled from a predefined action space𝒜\\mathcal\{A\}based on the policyπθ​\(at\|ot\)\\pi\_\{\\theta\}\(a\_\{t\}\|o\_\{t\}\)\. This action space consists of two primary categories:

- •Tool Invocation \(attoola\_\{t\}^\{\\text\{tool\}\}\):A formatted request to interact with one or multiple external APIs to acquire necessary context \(e\.g\.,<tool\_call\>\.\.\.</tool\_call\>\)\. Following execution, the environment yields a new observation containing the execution results\.
- •Task Resolution \(atanswera\_\{t\}^\{\\text\{answer\}\}\):A conversational response directed at the user \(e\.g\.,<answer\>\.\.\.</answer\>\)\. Emitting this action signifies the completion of the current sub\-task, triggering the environment to provide the subsequent user instruction\.

For any given turnii, the interaction proceeds as an intermediate trajectory of tool invocations, culminating when a task resolution is output\. Onceatanswera\_\{t\}^\{\\text\{answer\}\}is generated, the environment advances the state by embedding the next instructionqi\+1q\_\{i\+1\}into the subsequent observationot\+1o\_\{t\+1\}\. This iterative process continues until allNNinstructions are completed\.

A complete episode yields a full trajectoryτ=\(o0,a0,o1,a1,…,oT\)\\tau=\(o\_\{0\},a\_\{0\},o\_\{1\},a\_\{1\},\\dots,o\_\{T\}\), which terminates at timestepTTwhen the final response for the ultimate instructionqNq\_\{N\}is issued\. Importantly, it is only at this terminal stepTTthat asparse, binary rewardRT∈\{0,1\}R\_\{T\}\\in\\\{0,1\\\}is assigned, reflecting the overall success or failure of the entire task\. Such delayed and sparse feedback makes credit assignment and exploration during reinforcement learning particularly hard, a common obstacle in long\-horizon scenarios\.

Therefore, the primary objective is to optimize the policy parametersθ\\thetato maximize the expected terminal reward:

J​\(θ\)=𝔼τ∼πθ,P​\[RT\]J\(\\theta\)=\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\},P\}\[R\_\{T\}\]\(2\)

## Appendix BSystem Implementation and Hyperparameters

In this section, we detail the specific hyperparameters, engineering optimizations, and fault\-tolerance mechanisms used in our system implementation\.

### B\.1Hyperparameter Specifications

Dataset Partitioning Thresholds\.As introduced in Section 3\.1, the dataset is partitioned into three zones\. Empirically, we set the boundary thresholds as follows:

- •Boundary Zone:α−=0\.20\\alpha^\{\-\}=0\.20andα\+=0\.85\\alpha^\{\+\}=0\.85\. This captures tasks where the model exhibits partial but incomplete mastery, maximizing gradient variance\.
- •Mastered Zone:αretire\+=0\.95\\alpha^\{\+\}\_\{\\text\{retire\}\}=0\.95\.
- •Too Hard Zone:αretire−=0\.20\\alpha^\{\-\}\_\{\\text\{retire\}\}=0\.20\.

Injection and Capacity Limits\.To prevent distribution shock \(Section 3\.3\), the per\-epoch injection volume is strictly capped at20%20\\%of the active pool size\. Additionally, the active dataset is bounded by a maximum capacityPmax=400P\_\{\\max\}=400generated items, triggering the priority\-based eviction strategy when exceeded\. The original seed dataset is strictly preserved and exempt from retirement\.

### B\.2Data Injection Latency Analysis

Variants generated during training stepttmay not enter the dataset until stept\+Δt\+\\Delta\. Empirically, we observe a seed\-to\-variant synthesis delay averaging 1 step, and a total staging delay \(awaiting the next epoch boundary\) averaging 13 steps \(roughly 1–2 full epochs\)\. Within this asynchronous window, a data point’s pass@16 changes by less than0\.100\.10on average, ensuring it remains in its originally classified zone\. As a result, freshly generated variants remain highly relevant to the evolving model capability upon injection, validating the asynchronous design\.

### B\.3System Engineering and Concurrency

Correctness Guarantees: Simulation Environment Execution as Oracle\.The Synthesis Simulation Environment is implemented as a faithful replica of the training simulation environment\. If a variant executes successfully on the Simulation Environment during generation, it is mathematically guaranteed to execute without environment\-level errors during training\. This architectural choice eliminates the need for costly and error\-prone LLM\-based correctness judgments, replacing them with deterministic, reproducible data validation\.

Concurrency and Isolation\.To prevent resource contention during online synthesis, each training environment instance maintains strictly isolated model and environment states\. The asynchronous background generation module operates in a fully decoupled process space, utilizing filesystem\-based inter\-process communication with file\-level locking\. This design prevents shared\-memory race conditions and allows the generation daemon to scale independently of the primary reinforcement learning loop\.

Prompt Engineering:<reason\>vs\.<think\>\.To prevent modern reasoning models \(e\.g\., Qwen3, QwQ\) from defaulting to unconstrained native thinking modes that disrupt structured parsing, our LLM agents use custom<reason\>tags for Chain\-of\-Thought generation rather than standard reasoning tokens \(e\.g\.,<think\>\)\. Because<reason\>is not mapped to a special token in the tokenizer’s vocabulary, it functions purely as a structural prompt directive, ensuring stable and parseable JSON/XML outputs\.

### B\.4Crash Recovery and Fault Tolerance

To support reliable, long\-running RL experiments, the system is designed with full fault tolerance\. PromptTracker state is serialized totracker\.jsonafter each epoch boundary\. Generated variants are written to immutable append\-only logs \(e\.g\.,expanded\_epoch\_\*\.jsonl\)\. On daemon or trainer restart, the SeedManager reconstructs the full synthesized dataset and training state by:

1. 1\.Loading the latesttracker\.jsonto recover PromptTracker historical windows\.
2. 2\.Replaying allexpanded\_\*\.jsonlfiles to rebuild the list of generated data\.
3. 3\.Deterministically recomputing the data retirement logic from stored metrics\.

This guarantees zero data loss across restart boundaries and lets the curriculum resume where it was interrupted\.

### B\.5Motivating Analysis: Gradient Variance at the Capability Boundary

We present the analysis motivating our boundary\-targeting design heuristic\. Specifically, we establish the rationale for preferentially synthesizing new data from tasks where the model’s average progress reward is near the midpoint of its range \(μ≈0\.5\\mu\\approx 0\.5\)\. This heuristic is grounded in the relationship between reward variance and gradient signal strength\.

In GRPO, for a given promptxx, the model generatesKKrolloutsy1,…,yKy\_\{1\},\\dots,y\_\{K\}\. The corresponding binary rewards \(success = 1, failure = 0\) are denoted asri∈\{0,1\}r\_\{i\}\\in\\\{0,1\\\}\. While the Progress Reward \(RPR\_\{P\}\) used in our training is a multi\-valued dense signal rather than strictly binary, Popoviciu’s inequality states that the variance of any bounded variableX∈\[0,1\]X\\in\[0,1\]is upper\-bounded byμ​\(1−μ\)\\mu\(1\-\\mu\)\. This bound motivates our design heuristic: the gradient signal potential is highest nearμ=0\.5\\mu=0\.5, though the actual variance depends on the full reward distribution and may not saturate this bound\. The policy gradient is estimated as:

∇θJ​\(θ\)≈1K​∑i=1KA^i​∇θlog⁡πθ​\(yi\|x\)\\nabla\_\{\\theta\}J\(\\theta\)\\approx\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{K\}\\hat\{A\}\_\{i\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{i\}\|x\)\(3\)whereA^i\\hat\{A\}\_\{i\}is the z\-score normalized advantage:A^i=ri−μxσx\+ϵ\\hat\{A\}\_\{i\}=\\frac\{r\_\{i\}\-\\mu\_\{x\}\}\{\\sigma\_\{x\}\+\\epsilon\}\. Here,μx\\mu\_\{x\}is the empirical success rate \(pass@KK, denoted aspp\), andσx=p​\(1−p\)\\sigma\_\{x\}=\\sqrt\{p\(1\-p\)\}is the standard deviation\.

Variance maximization in the binary case\.For binary rewards, the normalized advantage for successful \(ri=1r\_\{i\}=1\) and failed \(ri=0r\_\{i\}=0\) rollouts can be exactly computed asA^\+=1−pp\\hat\{A\}^\{\+\}=\\sqrt\{\\frac\{1\-p\}\{p\}\}andA^−=−p1−p\\hat\{A\}^\{\-\}=\-\\sqrt\{\\frac\{p\}\{1\-p\}\}, respectively\. The total magnitude of the gradient signal \(signal density\) is proportional to the variance of the Bernoulli distribution:

Signal Density∝p​\(1−p\)\\text\{Signal Density\}\\propto p\(1\-p\)\(4\)When a task is too hard \(p→0p\\to 0\) or already mastered \(p→1p\\to 1\), the signal density vanishes because all rollouts yield identical rewards, resulting in zeroed\-out advantages\. For binary rewards, this signal density is maximized atp=0\.5p=0\.5, where the model generates a balanced mix of successes and failures, allowing GRPO to clearly contrast correct and incorrect behaviors\. For the continuous Progress RewardRP∈\[0,1\]R\_\{P\}\\in\[0,1\], the Popoviciu bound provides an analogous \(though not necessarily tight\) guideline: targetingμ≈0\.5\\mu\\approx 0\.5maximizes the*potential*for high variance, making it a practical proxy for the learning frontier\. Because GRPO inherently generates theseKKrollouts during the standard forward pass to compute the group advantage, we can identify these high\-varianceμ≈0\.5\\mu\\approx 0\.5tasks atzero additional inference cost\.

## Appendix CAgent Prompt Design

This section provides the complete prompt templates used by the multi\-agent synthesis pipeline\. Each agent is implemented as an LLM\-based component with carefully designed system and user prompts to ensure high\-quality data generation\.

### C\.1Planner Agent

The Planner Agent is responsible for generating a novel function call sequence that matches the structural complexity of the boundary seed task while introducing variation in the specific functions and scenario\.

Planner Agent User Prompt\# Task You are a function call planner for a multi\-turn tool\-calling benchmark\.You are given a seed task with its user queries and ground truth function call sequence\. Your goal is to select a function sequence from the available functions that tests SIMILAR capabilities as the seed – such as parameter extraction, multi\-step reasoning, cross\-turn dependency, etc\. – but using potentially DIFFERENT functions\.You may use both HIGH\-LEVEL functions \(which the system will automatically decompose into multiple bottom\-level calls\) and BOTTOM\-LEVEL functions \(executed directly\)\.\# Seed Task Classes:\{classes\_str\} User queries:\{queries\_text\} Ground truth function sequence:\{gt\_summary\}\# Available Functions \{func\_list\}\# Guidelines1\.Select functions that test similar skills to the seed task \(e\.g\., if the seed requires multi\-step parameter passing, your plan should also require it\)2\.HIGH\-LEVEL functions are preferred when available – they produce richer, multi\-step call sequences after decomposition3\.Each turn should have 1\-3 functions from the SAME class4\.For multi\-class seeds, alternate classes across turns \(e\.g\., Turn 1: ClassA, Turn 2: ClassB, Turn 3: ClassA\)5\.Output 2\-5 turns total6\.Ensure the sequence is logically coherent \(e\.g\., authenticate before posting, fill fuel before driving\)\# Output Format First, analyze the seed task and plan your approach inside<reason\></reason\>tags\. Then output a brief narrative scenario \(2\-3 sentences\) inside<narrative\>tags\. Finally output each turn inside a<turn\>tag\.IMPORTANT: Output 2\-5 turns total\. Do NOT output more than 5 turns\.IMPORTANT: ONLY use function names that appear in the Available Functions list above\.Format: <reason\> Your step\-by\-step analysis\.\.\. </reason\> <narrative\> A user named \[name\] wants to \[goal\]\.\.\. </narrative\> <turn\> ClassName: func1, func2, func3 </turn\> <turn\> ClassName: func4 </turn\>Example 1\(single class – VehicleControlAPI\):<turn\> VehicleControlAPI: fillFuelTankWithLiter </turn\> <turn\> VehicleControlAPI: activateParkingBrake, pressBrakePedal, startEngine </turn\> <turn\> VehicleControlAPI: estimate\_drive\_feasibility\_between\_city </turn\>Example 2\(multi\-class – TwitterAPI \+ TravelAPI\):<turn\> TravelAPI: get\_flight\_cost\_from\_cityA\_to\_cityB </turn\> <turn\> TravelAPI: book\_flight </turn\> <turn\> TwitterAPI: post\_tweet\_with\_my\_acount </turn\>Example 3\(multi\-class – GorillaFileSystem \+ MathAPI\):<turn\> GorillaFileSystem: find\_cat </turn\> <turn\> GorillaFileSystem: grep\_function\_name </turn\> <turn\> MathAPI: mean </turn\>Example 4\(single class – TradingBot, using high\-level functions\):<turn\> TradingBot: update\_market\_status\_with\_current\_time </turn\> <turn\> TradingBot: add\_to\_watchlist\_with\_company\_name </turn\> <turn\> TradingBot: get\_stock\_info\_with\_company\_name </turn\> <turn\> TradingBot: place\_order\_with\_market\_price </turn\>Now generate a function plan for the seed task above\.

### C\.2Config Patch Agent \(Error Critic\)

The Config Patch Agent analyzes execution failures and generates environment configuration patches to resolve state conflicts\. This agent implements the Environment Patches mechanism described in Stage II of the synthesis pipeline\.

Config Patch Agent System PromptSystem Prompt: You are an expert at diagnosing configuration issues in function\-calling systems\. You analyze why a function failed with a given config, and suggest minimal fixes\.

Config Patch Agent User PromptInput: A data generation pipeline failed with this error:Error type:\{error\_type\} Failed function:\{error\_function\} Detail:\{error\_detail\}The initial\_config used was:\{config\_str\}Task: Analyze WHY this function failed with this config, and suggest the MINIMUM config change\(s\) to fix the issue\.Common Fixes:•market\_status “Closed”→\\rightarrow“Open” for trading functions•authenticated: false→\\rightarrowtrue for functions requiring login•Add pending orders for cancel\_order•Ensure sufficient balance for transactions•fuelLevel too low→\\rightarrowincrease for driving functions\# Output Format First, analyze the error inside<reason\></reason\>tags\. Then output each field change as a<patch\>block:<reason\> Your analysis\.\.\. </reason\> <patch\> <class\>ClassName</class\> <field\>field\_name</field\> <value\>new\_value</value\> </patch\>For nested fields, use dot notation: e\.g\.,orders\.12345\.statusNow analyze the error above and output the necessary patches\.

### C\.3Coherence Rewrite Agent

The Coherence Rewrite Agent performs holistic semantic grounding \(Stage III\) by generating natural user queries that are semantically coherent across all turns, guided by the latent narrative\.

Coherence Rewrite Agent System PromptSystem Prompt: You are a helpful assistant that writes natural user queries for function\-calling conversations\.

Coherence Rewrite Agent User Prompt\# Task Rewrite user queries for a multi\-turn function\-calling conversation\.\# Scenario \{narrative\}\# Ground Truth Function Calls Per Turn \{turns\_for\_rewrite\}\# Instructions Generate one natural user query for EACH turn that:1\.Accurately describes what the GT function calls do \(without mentioning function names\)2\.Sounds like a real user talking to an AI assistant3\.Is semantically coherent with the narrative scenario4\.Does NOT contain special characters that could cause parsing issues5\.For turns with multiple function calls, the query should naturally imply all of them\# Output Format Output each query inside<query\>tags, one per turn:<query\>Turn 1 user query here</query\> <query\>Turn 2 user query here</query\> …IMPORTANT: Output exactly\{num\_turns\}queries, one for each turn\.

### C\.4Quality Judge Agent

The Quality Judge validates synthesized trajectories against strict quality criteria to ensure data integrity before injection into the training pool\.

Quality Judge Agent System PromptSystem Prompt: You are a strict quality judge for multi\-turn function\-calling data\. You evaluate whether a generated sample meets quality standards\.

Quality Judge Agent User PromptInput: Evaluate this multi\-turn function\-calling sample against quality criteria\.\# Sample Data \{sample\_summary\}\# Quality Criteria1\.Query\-GT Alignment: Does each user query accurately describe what the GT function calls do?2\.State Consistency: Do parameter values reflect the actual environment config?3\.Cross\-Turn Coherence: Is there logical state progression across turns?4\.Query Naturalness: Do queries sound like real user requests?5\.Structural Correctness: Are deliberate ambiguities \(e\.g\., missing parameters\) properly reflected?\# Automatic Rejection Patterns If the query contains any of these patterns, REJECT immediately:•“Thought Process”, “Construct Query”, “Step 1:”, “Step 2:”•Function names or parameter names mentioned explicitly•Technical jargon that a real user wouldn’t say\# Output Format First explain your reasoning inside<reason\></reason\>tags\. Then output your decision:<reason\> Your detailed analysis\.\.\. </reason\> <decision\>accept</decision\>or<decision\>reject</decision\> <fail\_reason\>Specific reason for rejection \(if rejected\)</fail\_reason\>

### C\.5Refine Classify Agent

When the Quality Judge rejects a sample, the Refine Classify Agent determines whether the issue can be fixed by rewriting the user query or if the ground truth itself is unfixable\.

Refine Classify Agent System PromptSystem Prompt: You are a precise diagnostic assistant\. Analyze the root cause of data quality issues\.

Refine Classify Agent User PromptInput: A Quality Judge rejected this multi\-turn function\-calling data sample\.Rejection reason:\{fail\_reason\}Data summary:\{data\_summary\}Task: Analyze the rejection reason and determine:•Is the problem in the USER QUERY \(wrong wording, mentions wrong values, unnatural phrasing, format issues\)?→\\rightarrowThese can be fixed by rewriting the query\.•Is the problem in the GT function calls \(wrong parameters, wrong function, wrong cross\-turn state, calling non\-existent resources\)?→\\rightarrowThese CANNOT be fixed by rewriting the query\.\# Output Format First explain your reasoning inside<reason\></reason\>tags, then output your answer\.<reason\> Your analysis of where the root cause is\.\.\. </reason\> <answer\>query\_fixable</answer\>or<answer\>gt\_unfixable</answer\>

### C\.6Refine Rewrite Agent

If the Refine Classify Agent determines the issue is query\-fixable, the Refine Rewrite Agent performs a targeted rewrite of the problematic user query\.

Refine Rewrite Agent System PromptSystem Prompt: You are a helpful assistant that rewrites user queries to be natural and accurate\.

Refine Rewrite Agent User PromptInput: A quality check found an issue with this user query in a function\-calling dataset\.Issue:\{fail\_reason\}Original query: “\{old\_query\}”Ground truth function calls for this turn:\{gt\_str\}Task: Rewrite the user query so that:1\.It naturally and accurately describes what the GT function calls actually do2\.It sounds like a real user talking to an AI assistant3\.It does NOT mention function names, parameter names, or technical details4\.It does NOT contain special characters that could cause parsing issues5\.It fixes the specific issue described above\# Output Format Output ONLY the rewritten query inside<answer\>tags:<answer\>your rewritten query here</answer\>

## Appendix DDeterministic Execution Pipeline Internals

The Stage II execution pipeline instantiates the abstract plan within a Python\-based sandbox environment that faithfully replicates the training environment’s API semantics\. The pipeline processes each turn sequentially through the following stages:

1. 1\.Function Sampling:Given the Planner’s output specifying which functions to call per turn, the pipeline samples concrete function instances from the available catalog\. HIGH\-LEVEL functions are automatically decomposed into sequences of BOTTOM\-LEVEL API calls\.
2. 2\.Parameter Generation:For each selected function, an LLM generates concrete parameter values conditioned on the function schema, the current environment state𝒞t\\mathcal\{C\}\_\{t\}, and any dependencies from previous turns\.
3. 3\.VM Execution:The parameterized function call is executed against the sandbox VM\. The VM maintains a complete environment state \(file systems, account balances, database entries, etc\.\) and returns execution results or error messages\.
4. 4\.Query Generation:A per\-class LLM prompt generates a natural user query that describes the function call’s intent without exposing function names or parameters\.
5. 5\.Query Verification:A verification LLM checks whether the generated query’s semantics align with the GT function calls, rejecting queries that are misaligned or contain data generation artifacts\.

If any stage fails, a structured error is recorded with the error type, failing function, turn number, and diagnostic detail\. This structured error feeds directly into the Config Patch Agent and the Planner re\-invocation described in the main text\.

## Appendix EError Taxonomy for Feedback\-Driven Re\-synthesis

To provide structured feedback to the Error Critic and Planner Agent, the execution pipeline classifies failures into the following categories\. Each error type triggers specific recovery strategies:

1. 1\.param\_gen\_failed:The LLM failed to generate valid parameters for a function call\. The Config Patch Agent may adjust environment state to make parameter generation feasible\.
2. 2\.decompose\_failed:A HIGH\-LEVEL function could not be decomposed into valid BOTTOM\-LEVEL calls\. The Planner is instructed to “Use BOTTOM\-LEVEL functions only\.”
3. 3\.func\_sample\_failed:No valid function could be sampled after multiple retries \(e\.g\., all candidates require unavailable prerequisites\)\. The Planner is told to “AVOID functions requiring authentication or specific prior state\.”
4. 4\.vm\_exec\_failed:The function call executed but the VM returned an error \(e\.g\., insufficient balance, closed market, missing resource\)\. This is the primary trigger for Config Patching\.
5. 5\.duplicate\_func:The same function call appeared multiple times in the same turn\. The pipeline rejects the sample\.
6. 6\.query\_gen\_failed:The query generation LLM failed to produce a valid<query\>tag after multiple retries\. The Planner is told to “Use simpler function combinations\.”
7. 7\.query\_verify\_failed:The generated query was rejected by the verification LLM as semantically misaligned with the GT\. The Planner is told to “Use 1 function per turn to simplify\.”
8. 8\.query\_verify\_no\_tag:The verification LLM did not return a parseable verdict tag\.
9. 9\.conversation\_construct\_failed:The multi\-turn conversation assembly failed \(e\.g\., cross\-turn dependency resolution error\)\.
10. 10\.no\_prompts:No prompt template exists for the specified class\.
11. 11\.no\_pattern:No valid execution pattern could be derived for the given class order\.
12. 12\.pipeline\_exception:An uncaught exception during pipeline execution\.

Only the first four error types \(param\_gen\_failed,decompose\_failed,func\_sample\_failed,vm\_exec\_failed\) trigger the Config Patch Agent, as these represent environment\-level issues resolvable through state modification\. Query\-level errors trigger Planner re\-invocation with action space constraints but do not invoke config patching\.

## Appendix FFeedback Loop Implementation Details

This section provides detailed implementation of the feedback\-driven re\-synthesis mechanism\.

Dual\-Feedback Loop\.When the execution engine encounters an error \(classified per Appendix[E](https://arxiv.org/html/2606.19047#A5)\), recovery proceeds through a dual\-feedback loop that simultaneously accumulates two types of corrective signals acrossKmax=3K\_\{\\max\}\\\!=\\\!3pipeline attempts:

1. 1\.Environment Config Patching:If the error type is patchable \(param\_gen\_failed,vm\_exec\_failed,func\_sample\_failed, ordecompose\_failed\), the Config Patch Agent \(Appendix[C\.2](https://arxiv.org/html/2606.19047#A3.SS2)\) analyzes the initial environment configuration and outputs structured XML patches to update the state\. Patches are accumulated across retries via recursive deep\-merge, where later patches override earlier ones for the same field but coexist for different fields\. A safety mechanism prevents non\-dict values from overwriting dict\-structured fields \(e\.g\., preventing a string summary from corrupting a file tree\)\.
2. 2\.Action Space Pruning:Simultaneously, the names of all functions involved in failures are extracted and added to a cumulative blocklist\. On re\-invocation, the Planner Agent receives: \(a\) the full failure history with structured error descriptions, \(b\) a list of specifically blocked function names, and \(c\) error\-type\-specific guidance \(e\.g\., “Use BOTTOM\-LEVEL functions only” for decomposition failures, “AVOID functions requiring authentication” for sampling failures\)\. The Planner is explicitly instructed to “Generate a COMPLETELY DIFFERENT plan using different functions\.”

Both corrective signals are applied jointly to the next pipeline attempt, progressively narrowing the search space until a valid instantiation is found\. If allKmaxK\_\{\\max\}attempts fail, the seed is discarded\.

## Appendix GQuality Judge and Refinement Loop

This section details the multi\-tier validation pipeline and the iterative refinement mechanism\.

Rule\-Based Validation \(Gate 1–3\)\.Before reaching the LLM Quality Judge, variants must pass three deterministic gates:

- •VM Re\-verification:All GT function calls are re\-executed against a fresh VM instance initialized with the variant’sinitial\_config, ensuring execution correctness\.
- •Tool Availability:Every function name in the GT must exist in the tools provided to the model\. Formiss\_funcvariants, this check accounts for tools provided in recovery turns\.
- •Parameter Complexity:List/tuple parameters are limited to≤5\\leq 5elements; string parameters to≤200\\leq 200characters\. This ensures the model can realistically reproduce the GT during training \(pass@16\>0\>0\)\.

LLM Quality Judge \(Gate 4\)\.The Quality Judge strictly evaluates synthesized trajectories against five semantic criteria: \(1\)Query–GT Alignment\(ground truth must exactly match the query’s intent—no more, no less\), \(2\)State Consistency\(parameter values must reflect the actual environment config, not incorrect values stated by the user\), \(3\)Cross\-Turn Coherence\(logical state progression across turns\), \(4\)Query Naturalness\(human\-like conversational flow without data generation artifacts\), and \(5\)Structural Correctness\(e\.g\., ensuring deliberate ambiguity in missing\-parameter scenarios\)\. Automatic failure is triggered by prompt\-leakage patterns \(e\.g\., “Thought Process”, “Construct Query”, raw JSON tool definitions in non\-recovery turns\)\. The judge outputs a<reason\>analysis followed by a<decision\>accept</decision\>or<decision\>reject</decision\>verdict\. Full prompt is provided in Appendix[C\.4](https://arxiv.org/html/2606.19047#A3.SS4)\.

Diagnostic Refinement Loop \(max 1 cycle\)\.If a trajectory is rejected, aFail Classifier\(Appendix[C\.5](https://arxiv.org/html/2606.19047#A3.SS5)\) analyzes the rejection reason and outputs one of two verdicts:

- •gt\_unfixable: The problem lies in the GT function calls \(e\.g\., wrong parameters, wrong function, state dependency violations\)\. The sample isimmediately dropped\.
- •query\_fixable: The problem is in the user query \(e\.g\., unnatural wording, misaligned description\)\. The sample is routed to aRefine Rewriter\(Appendix[C\.6](https://arxiv.org/html/2606.19047#A3.SS6)\) which performs a one\-shot targeted rewrite of the identified turn’s query\.

The rewritten variant is submitted to the Quality Judge for a second evaluation\. Trajectories that fail this second check are permanently dropped\. No recursive refinement is applied\.

## Appendix HReward Details

This section details the format rewardRformatR\_\{\\text\{format\}\}design used in Stage 1 training, which evaluates the structural correctness of tool calls\.

Format Reward Formulation \(Stage 1\)\.In Stage 1, the agent is optimized purely for formatting and valid API execution syntax\. We assign per\-turn penalty codes:−3\-3for XML format errors,−2\-2for tool schema errors \(e\.g\., invalid JSON\), and−1\-1for valid syntax but execution failure\. Valid executions receive0or11\. LetNNbe the total interaction rounds \(turns\), andn−kn\_\{\-k\}be the count of turns receiving code−k\-k\. The format and tool execution rewards are defined as:

rformat=max⁡\(0,N−n−3N\),rtool=\{n−1n−1\+n−2if​n−1\+n−2\>00otherwiser\_\{\\text\{format\}\}=\\max\\left\(0,\\frac\{N\-n\_\{\-3\}\}\{N\}\\right\),\\qquad r\_\{\\text\{tool\}\}=\\begin\{cases\}\\frac\{n\_\{\-1\}\}\{n\_\{\-1\}\+n\_\{\-2\}\}&\\text\{if \}n\_\{\-1\}\+n\_\{\-2\}\>0\\\\ 0&\\text\{otherwise\}\\end\{cases\}\(5\)An indicator function𝟙tool=𝕀​\[n−1\+n−2\>0\]\\mathbb\{1\}\_\{\\text\{tool\}\}=\\mathbb\{I\}\[n\_\{\-1\}\+n\_\{\-2\}\>0\]ensures at least one tool call was attempted\. The final Stage 1 reward is defined asRfinal=𝟙tool⋅\(rformat\+rtool\)∈\[0,2\]R\_\{\\text\{final\}\}=\\mathbb\{1\}\_\{\\text\{tool\}\}\\cdot\(r\_\{\\text\{format\}\}\+r\_\{\\text\{tool\}\}\)\\in\[0,2\]\. In Stages 2 and 3, the Progress Reward \(RPR\_\{P\}\) is computed simply as the fraction of successfully resolved turns\.

## Appendix ICombination Experiment: RODS \+ EnvTuning

To analyze whether boundary\-targeted synthesis \(RODS\) and environmental feedback augmentation \(EnvTuning\) provide complementary gradient signals, we plot their training progress rewards\. As shown in Table[1](https://arxiv.org/html/2606.19047#S4.T1), the static baseline without augmentation suffers from rapid reward saturation\. WhileRODSmaintains a continuously climbing progress reward by introducing boundary tasks, replacing static tasks alone still lacks granular execution hints\. By combining both methods, the agent not only receives high\-variance boundary tasks but also environment\-aided corrections, leading to improved training stability and sample efficiency\.

## Appendix JTraining Details

This section lists the full training hyperparameters for the RL pipeline\.

Table 3:System Hyperparameters\.Configuration details for RL training, data synthesis, and the dynamic replay buffer\.CategoryHyperparameterValueRL Training \(GRPO\)Actor Learning Rate1×10−61\\times 10^\{\-6\}KL Loss Coefficient \(β\\beta\)0\.010\.01Number of Rollouts \(KK\)1616PPO Mini\-batch Size512512Total Training Epochs55\(per stage\)Data SynthesisLLM BackendQwen3\-32BDecoding Temperature1\.01\.0Top\-pp0\.70\.7Max Pipeline / Planner Retries33/33Dynamic BufferBoundary Lower Bound \(α−\\alpha^\{\-\}\)0\.200\.20Boundary Upper Bound \(α\+\\alpha^\{\+\}\)0\.850\.85Generated Pool Cap \(PmaxP\_\{\\max\}\)400400Trial\-Period Observation Count11Trial\-Period Eviction Threshold0\.200\.20Retirement Mastered Threshold \(αretire\+\\alpha^\{\+\}\_\{\\text\{retire\}\}\)0\.950\.95Retirement Hard Threshold \(αretire−\\alpha^\{\-\}\_\{\\text\{retire\}\}\)0\.200\.20

## Appendix KTheoretical Extension to PPO

Although our empirical results focus on GRPO due to its memory efficiency, theRODScapability probe naturally extends to Proximal Policy Optimization \(PPO\)\. In PPO, the advantage functionAtA\_\{t\}is typically estimated via Generalized Advantage Estimation \(GAE\), driven by the TD\-errorδt=rt\+γ​V​\(st\+1\)−V​\(st\)\\delta\_\{t\}=r\_\{t\}\+\\gamma V\(s\_\{t\+1\}\)\-V\(s\_\{t\}\)\.

For binary or bounded task\-level rewards \(like our Progress RewardRPR\_\{P\}\), the variance of the TD\-error across multiple trajectory samples from the same prompt remains tightly coupled to the variance of the final reward\. When a task is fully mastered \(RP→1R\_\{P\}\\to 1\) or consistently failed \(RP→0R\_\{P\}\\to 0\), the Value networkV​\(s\)V\(s\)accurately predicts the outcome, leading to near\-zero TD\-errors \(δt≈0\\delta\_\{t\}\\approx 0\) and vanishing gradients\. Conversely, at the capability boundary where outcomes are highly uncertain \(p≈0\.5p\\approx 0\.5\), the Value network’s prediction error is maximized, resulting in high\-variance advantage estimates\.

Therefore, tracking the variance of the PPO advantage estimatesVar​\(A^\)\\text\{Var\}\(\\hat\{A\}\)or the raw reward variance inside the PPO rollout buffer serves identical functions to the GRPO variance probe, allowingRODSto dynamically identify and synthesize boundary tasks without architectural changes\.

## Appendix LOut\-of\-Distribution Generalization Results

This section provides the detailed performance metrics for out\-of\-distribution \(OOD\) generalization across the BFCL V4,τ2\\tau^\{2\}\-bench, and ACEBench Agent benchmarks\. The results confirm that boundary\-expanded training yields generalizable reasoning capabilities rather than mere in\-distribution pattern matching\.

Table 4:OOD generalization performance on BFCL V4,τ2\\tau^\{2\}\-bench, and ACEBench Agent benchmarks\. All results are compared against theLlama\-3\.1\-8B\-Instructbase model\. Models trained withRODS\(rows in blue\) show improvements on these OOD tasks\. Scores for xLAM on the Retail and Airline domains aregrayed outas it was trained on the originalτ\\tau\-bench, making them invalid for OOD evaluation\.
## Appendix MSynthesis LLM Robustness

A natural question is whether the performance gains ofRODSare attributable to the framework design or to the specific generative capacity of the synthesis LLM\. To investigate this, we replace the default synthesis backbone \(Qwen3\-32B\) with GLM\-4\.5\-Air—a generally stronger model—deployed via vLLM on8×8\\timesA100 GPUs, while keeping all other components—boundary detection, structural isomorphism, dynamic replay buffer, and training pipeline—strictly identical\. The generated pool cap remainsPmax=400P\_\{\\max\}\\\!=\\\!400\.

Table 5:Synthesis LLM robustness on BFCL V3 multi\-turn\.Replacing the synthesis backbone with a stronger model \(GLM\-4\.5\-Air\) yields nearly identical overall performance \(↓\\downarrow0\.75%\), confirming thatRODSis model\-agnostic: its gains derive from the framework design rather than from the specific synthesis LLM\.#### Analysis\.

The overall performance difference is 0\.75% \(56\.00%→\\to55\.25%\)\. GLM\-4\.5\-Air is a stronger model than Qwen3\-32B, yet switching to it does not yield further gains\. This suggests limited sensitivity to the synthesis backbone under our setup: schema\-guided planning, deterministic VM validation, and multi\-tier quality filtering together normalize the output regardless of the underlying LLM’s raw capability\. Put differently,RODS’s performance is determined by the*framework design*\(boundary detection, structural isomorphism, dynamic lifecycle\), not by the generative capacity of the synthesis model\.

At the sub\-split level, the two LLMs show mild distributional differences: GLM\-4\.5\-Air scores higher onMissing Parameter\(\+2\.00%\) andLong Context\(\+2\.00%\), while Qwen3\-32B leads onBase\(\+3\.00%\) andMissing Functions\(\+4\.00%\)\. These complementary biases suggest that different LLMs produce variants with subtly different structural characteristics, pointing to a potential direction of multi\-backbone synthesis ensembles\.

## Appendix NSynthesis Computational Cost

We report the computational overhead of theRODSsynthesis pipeline to contextualize its cost relative to the RL training loop\.

#### Hardware allocation\.

Training and synthesis run on separate GPU clusters in parallel:

- •RL training:8×8\\timesA100 \(80GB\) GPUs running GRPO with the Qwen3\-4B\-Instruct policy\.
- •Data synthesis:8×8\\timesA100 \(80GB\) GPUs hosting Qwen3\-32B via vLLM, with 64 parallel worker threads dispatching synthesis requests\.

The two systems communicate asynchronously via filesystem\-based queues \(seed emission from the trainer, variant ingestion at epoch boundaries\)\. The synthesis daemon does not block or slow the training loop\.

#### Wall\-clock time\.

To reach the reported performance of 56\.00% \(at training step∼\\sim600\), the system runs for approximately56 hourswall\-clock\. Since both clusters operate concurrently for the full duration, the total compute is:

- •Training:8×56=4488\\times 56=448GPU\-hours\.
- •Synthesis:8×56=4488\\times 56=448GPU\-hours\.
- •Total:∼\\sim896 GPU\-hours\(16×16\\timesA100 for∼\\sim56 hours\)\.

The synthesis overhead is thus1×1\\timesthe training cost in GPU\-hours\. However, the synthesis cluster runs a single vLLM instance with no gradient computation, so its actual FLOP consumption is lower than the training cluster\.

#### Per\-variant synthesis cost\.

On the successful path, generating a single base variant requires approximately9–15 LLM callsdepending on the number of turns: 1 Planner call, 1 query generation \+ 1 query verification per turn \(for a 3\-turn variant: 6 calls\), 1 coherence rewrite, and 1 quality judge evaluation\. Formiss\_funcandmiss\_paramvariants, 2–5 additional calls are needed for the adversarial transform and its verification\. The 64\-worker thread pool processes these calls concurrently across seeds, achieving high throughput despite the per\-variant multi\-stage pipeline\.

#### Synthesis latency\.

The measured seed\-to\-injection delay averages1 training step\(∼\\sim3\.7 minutes\)\. Since variant injection occurs at epoch boundaries and the daemon processes seeds continuously in the background, the synthesis pipeline does not introduce any idle time into the training loop\. Variants generated during epochnnare staged and injected at the start of epochn\+1n\{\+\}1, as described in Section 3\.3\.

#### Cost\-efficiency perspective\.

While the synthesis overhead doubles the GPU footprint relative to standard GRPO training, the resulting20×20\\timesdata efficiency gain \(matching 17K\-sample offline pipelines with an active training pool of∼\\sim800 samples\) represents a practical trade\-off\. Generating 17K high\-quality multi\-turn trajectories offline via similarly complex multi\-agent simulation pipelines \(such as APIGen\-MT or FunReason\-MT\) typically incurs massive upstream computational costs before training even begins\. By explicitly targeting only the high\-variance capability boundary,RODSnot only reduces the required data volume by an order of magnitude but also provides a favorable end\-to\-end data\-compute trade\-off relative to massive offline pipelines\.

## Appendix OBenchmark and Evaluation Details

We evaluate our agents using several multi\-turn tool\-use benchmarks to assess both in\-distribution learning and out\-of\-distribution \(OOD\) generalization\.

#### In\-Distribution Evaluation \(BFCL V3\)\.

The Berkeley Function Calling Leaderboard \(BFCL\) V3\(patil2025bfcl\)provides a reliable testbed for multi\-turn scenarios\. We utilize its 800\-sample multi\-turn subset, partitioned equally across four categories:

- •Base: Standard multi\-turn tasks with straightforward dependencies\.
- •Missing Function: Tasks requiring the agent to recognize missing capabilities and either gracefully decline or request alternative tools\.
- •Missing Parameter: Tasks lacking necessary arguments, requiring the agent to ask the user for clarification before proceeding\.
- •Long\-Context: Scenarios involving extended conversations where context must be maintained across many turns\.

We adopt the exact 400/400 train/test split established bytuneenvto ensure a fair comparison\. Evaluation is performed via the official BFCL abstract syntax tree \(AST\) matching evaluator\.

#### Out\-of\-Distribution Evaluation\.

To verify thatRODSinduces generalizable reasoning rather than mere pattern matching, we evaluate on benchmarks featuring unseen APIs and interaction modalities:

- •BFCL V4:We test on theWeb SearchandMemorytracks, representing dynamic information retrieval and long\-term state tracking not present in V3\.
- •τ2\\tau^\{2\}\-bench\(tau2bench\):A dual\-control conversational benchmark set in theRetail,Airline, andTelecomdomains, emphasizing highly constrained, real\-world business logic\.
- •ACEBench\(acebench\):We utilize theMulti\-turnandMulti\-stepsplits of the Agent track to test complex API topologies\.

All OOD evaluations strictly follow their respective official evaluation protocols and scoring scripts\.

## Appendix PSynthesized Data Examples

This section presents seed\-to\-variant pairs for each data type, demonstrating howRODSpreserves structural complexity while generating novel content\. For each example, we show the original seed \(left/top\) and the synthesized variant \(right/bottom\) side by side\.

### P\.1Base Type: VehicleControlAPI \(Seed→\\toVariant\)

Seed\(multi\_turn\_base\_63\): A 3\-turn task involving unit conversion, engine startup with safety checks, and distance estimation\.

Original SeedTurn 1:“I require assistance in determining the quantity of gasoline necessary for an extensive journey across California\. I currently anticipate needing around 166 liters\. How much is that in gallons?”Turn 2:“Prior to commencing the drive, kindly initiate the engine, ensuring all doors are securely closed and the parking brake is engaged\.”Turn 3:“Could you provide me with the approximate distance between San Francisco and Rivermist? Will I be able to get there?”

Synthesized Variant: Same skill structure \(unit conversion→\\tovehicle status \+ fuel→\\tomulti\-step distance estimation\), but different functions and parameters\.

Synthesized VariantTurn 1—*Unit Conversion*User:“I just checked my car’s fuel gauge and it shows 7\.8 liters remaining\. How many gallons is that?”GT:liter\_to\_gallon\(liter=7\.8\)Turn 2—*Vehicle Status \+ Fuel Operation*User:“Please show me my current fuel level and add 4 gallons to the tank for the trip\.”GT:displayCarStatus\(option=’fuel’\),fillFuelTank\(fuelAmount=4\.0\)Turn 3—*Multi\-step Distance \+ Feasibility*User:“I’m driving from Crescent Hollow to Autumnville – what’s the total distance, and can I complete the trip with 630 miles of fuel?”GT:get\_zipcode\_based\_on\_city\(city=’Crescent Hollow’\),get\_zipcode\_based\_on\_city\(city=’Autumnville’\),estimate\_distance\(cityA=’69238’, cityB=’51479’\),estimate\_drive\_feasibility\_by\_mileage\(distance=630\.0\)

Structural preservation:Both share the pattern*unit conversion \(1 call\)→\\tovehicle operation \(2 calls\)→\\todistance planning \(4 calls with dependency chain\)*\. The variant uses different city names, conversion direction \(liters→\\togallons vs\. gallons→\\toliters\), and vehicle operations \(fuel display vs\. engine startup\), forcing the model to generalize the abstract reasoning pattern\.

### P\.2Missing Function Type: GorillaFileSystem \(Seed→\\toVariant\)

Seed\(multi\_turn\_miss\_func\_38\): A file system task wherermis removed, requiring refusal and recovery\.

Original SeedTurn 1:“I’ve misplaced a vital document\. Assist in locating a file named ‘findings\_report’ within ‘SuperResearch’\. Could you remove it and the directory\.”Turn 2:*\[Functionrmremoved; agent must refuse\]*Turn 3:“What’s left in the current directory including the hidden files?”

Synthesized Variant: Different file operations,mkdirremoved instead ofrm, recovery in Turn 4\.

Synthesized VariantTurn 1:“I need to rename my JSON file ‘wqmmw\.json’ to ‘data\.csv’ for compatibility\.”GT:mv\(source=’wqmmw\.json’, destination=’data\.csv’\)Turn 2:“Please check how many lines are in ‘data\.csv’ so I can validate the data\.”GT:wc\(file\_name=’data\.csv’, mode=’l’\)Turn 3—*mkdirremoved from tool list*User:“I’d like to create a new folder called ‘docs’ to organize these files\.”GT:\[\]\(*agent must refuse*\)Turn 4—*Function restored; agent recovers*User:\[\{‘‘name’’: ‘‘mkdir’’, \.\.\.\}\]“Here’s a tool that might help\.”GT:mkdir\(dir\_name=’docs’\)

### P\.3Missing Parameter Type: TradingBot \+ MathAPI \(Seed→\\toVariant\)

Seed\(multi\_turn\_miss\_param\_144\): A cross\-class task where the user provides a vague computation request, requiring parameter clarification\.

Original SeedTurn 1:“After determining the current market status, retrieve the stock information for symbol ‘AAPL’\.”Turn 2:“Using the current details of a stock, calculate the average of price, trading volume, MA5, and MA20\.”*\[Parameters vague – which stock?\]*Turn 3:“The stock should be AAPL\.”*\[User provides clarification\]*

Synthesized Variant: Different stock, different vague reference \(“those two percentage changes”\), same clarification pattern\.

Synthesized VariantTurn 1—*Stock lookup \(TradingBot\)*User:“I’m looking at Synex Solutions’ stock – can you get their ticker symbol and the latest details?”GT:get\_symbol\_by\_name\(name=’Synex Solutions’\),get\_stock\_info\(symbol=’SYNX’\)Turn 2—*Vague query; concrete values omitted*User:“I need the average of those two percentage changes we just saw\.”GT:\[\]\(*agent must ask for clarification*\)Turn 3—*User provides missing numerical values*User:“They are−3\.4\-3\.4and−1\.0\-1\.0\.”GT:mean\(numbers=\[\-3\.4, \-1\.0\]\)

Key observation:In both seed and variant, the ambiguity arises from a vague back\-reference to prior tool output\. The variant changes the specific stock, the nature of the computation \(average of percentage changes vs\. average of multiple metrics\), and the exact missing values, while preserving the core skill: recognizing under\-specified parameters and requesting clarification before executing\.

## Appendix QFull Benchmark Results

Table 6:Full in\-distribution performance on BFCL V3 multi\-turn\(Tier 1: controlled RL comparisons\)\. All RL methods share the same 400 training samples and GRPO setup; only the data/environment strategy differs\.Red textindicates improvement over the base model\. The best result within each model group isbolded\.
## Appendix RAblation Configuration Details

This section provides detailed descriptions of each ablation condition in Table[2](https://arxiv.org/html/2606.19047#S4.T2)\.

#### \(a\) Boundary detection ablations\.

- •w/ random seed selection:Instead of selecting seeds from the boundary region \(𝒟boundary\\mathcal\{D\}\_\{\\text\{boundary\}\}\), we sample seeds uniformly at random from the entire training pool regardless of their reward\. This isolates the effect of boundary\-targeted capability tracking\.
- •w/ binary acc instead of progress reward:We replace the continuous Progress RewardRPR\_\{P\}with binary task accuracy \(1 if all turns correct, 0 otherwise\) for boundary detection\. This tests whether the fine\-grained partial credit ofRPR\_\{P\}is necessary for accurate boundary identification\.

#### \(b\) Synthesis pipeline ablations\.

- •w/o coherence rewrite:We skip Stage III \(holistic semantic grounding\)\. Per\-turn queries are generated independently without the Rewrite Agent’s narrative\-driven single\-pass rendering\.
- •w/o narrative planning:The Planner Agent generates a function sequence without an underlying narrative \(𝒩\\mathcal\{N\}\)\. This removes the cross\-turn thematic coherence that anchors all turns to a unified goal\.
- •w/o feedback loop \(blind retry\):We remove the Error Critic and Config Patch Agent\. When execution fails, the pipeline simply retries with a fresh random plan rather than accumulating corrective signals\.

#### \(c\) Lifecycle management ablations\.

- •w/o retirement mechanism:All three retirement layers \(L1–L3\) are disabled\. The pool only grows and is never pruned\.
- •w/ static pool \(no dynamic refresh\):We generate variants once at the beginning of Stage 3 and freeze the pool thereafter\. No new variants are synthesized as training progresses\.