Data Turnstile: A Scalable Open Framework for Function-Calling Data Generation

# A Scalable Open Framework for Function-Calling Data Generation
Source: [https://arxiv.org/html/2607.29250](https://arxiv.org/html/2607.29250)
Goutham Ramakrishnan Amazon AGI Bengaluru, India gorama@amazon\.com &Megha Sharma11footnotemark:1 Amazon AGI Bengaluru, India meghshar@amazon\.com

###### Abstract

Small language models \(SLMs\) are attractive for agentic deployment due to low latency, reduced cost, and on\-device privacy, yet they struggle with tool\-use tasks where training data is scarce and noisy\. Unlike larger models, SLMs cannot compensate for low\-quality supervision through sheer capacity, making data quality the critical bottleneck\. We present Data Turnstile, an open\-source framework that takes user\-defined API specifications and generates high\-quality synthetic training data for function calling\. Turnstile decomposes multi\-turn tool\-use interactions into constrained, stepwise generation with validation and error\-feedback loops, providing fine\-grained control over API diversity, conversation complexity, and output correctness\. We demonstrate effectiveness of domain adaptation with Turnstile data on two challenging function calling benchmarks\. On the BFCL single\-turn benchmark, a Qwen3\-0\.6B fine\-tuned on Turnstile data without chain\-of\-thought achieves 75\.9% overall accuracy \(versus 67\.4% for the base model with thinking enabled\), closing the gap with thinking\-enabled Qwen3\-1\.7B \(78\.4%\) and Qwen3\-4B \(79\.9%\) despite being 3×\\timesand 7×\\timessmaller respectively\. Onτ2\\tau^\{2\}\-bench, a multi\-turn agentic benchmark, Turnstile\-trained Qwen3\-1\.7B achieves 31\.1% pass^1 on the Telecom domain, improving 4\.7×\\timesover its 6\.6% base and surpassing Qwen2\.5\-32B\-Instruct \(27\.4%\), a model 19×\\timeslarger\. Turnstile\-trained Qwen3\-0\.6B achieves 24\.6%, improving 7×\\timesover its 3\.5% base and approaching the 32B model \(53×\\timeslarger\)\. We release Data Turnstile along with a dataset spanning 1,000\+ APIs and 100K\+ multi\-turn interactions\.

## 1Introduction

Function\-calling111We use*tool*,*function*, and*API*interchangeably\.language models power applications from customer service to coding assistants, but deploying them reliably requires expensive frontier models\[[23](https://arxiv.org/html/2607.29250#bib.bib36)\]\. Such models are typically large proprietary LLMs \(32B\+, often 100B\+\) and therefore deployed on the cloud\. In comparison, small language models \(SLMs,<<4B parameters\) offer practical benefits for edge deployment: lower inference cost, reduced latency, and improved data privacy\[[15](https://arxiv.org/html/2607.29250#bib.bib37)\]\. However, these models face specific challenges in terms of their agentic ability: they struggle to \(a\) natively reason about and execute complex agentic workflows, and \(b\) generalize from noisy or imperfect training data that larger models can tolerate\. Hence, improving function calling in SLMs is an active area of research\[[6](https://arxiv.org/html/2607.29250#bib.bib28)\]\.

LLMs typically excel at tool calling in a zero\-shot setting, or with in\-context learning through a few demonstrations\[[20](https://arxiv.org/html/2607.29250#bib.bib1),[16](https://arxiv.org/html/2607.29250#bib.bib18)\]\. For SLMs however, supervised fine\-tuning \(SFT\) on diverse, high\-quality data is the primary path to competent agentic behavior\[[13](https://arxiv.org/html/2607.29250#bib.bib26),[32](https://arxiv.org/html/2607.29250#bib.bib23)\]\. They must internalize conversational workflows, tool\-calling patterns, API schemas, and multi\-step reasoning directly from the training data\. Defects in training data are amplified disproportionately, as they cannot generalize from noisy data the way larger models can\[[21](https://arxiv.org/html/2607.29250#bib.bib29)\]\. This makes data quality the binding constraint for training SLMs\[[9](https://arxiv.org/html/2607.29250#bib.bib30),[14](https://arxiv.org/html/2607.29250#bib.bib20)\]and data generation methodology a first\-class research problem\.

![Refer to caption](https://arxiv.org/html/2607.29250v1/figures/hero_taubench.png)Figure 1:τ2\\tau^\{2\}\-bench multi\-turn results: Domain adaptation with Turnstile data helps SLMs beat zero\-shot 32B models\.#### Gaps\.

Existing approaches for generating synthetic tool\-use data face several limitations\. End\-to\-end generation methods such as ToolBench\[[27](https://arxiv.org/html/2607.29250#bib.bib17)\]typically prompt an LLM to produce entire multi\-turn conversations in a ‘single\-shot’, i\.e\. through a single call to the LLM\. This leads to quality degradation in longer and complex interactions, with unnatural conversations, malformed API calls, hallucinated parameter values and illogical API execution outputs\. Execution\-verified approaches such as APIGen\[[14](https://arxiv.org/html/2607.29250#bib.bib20)\]achieve higher quality but require functional API implementations, limiting to APIs with execution backends\. Most approaches also use proprietary frontier models for generation \(e\.g\. ToolBench uses ChatGPT\), making iteration and reproducibility difficult and expensive\.

#### Our Approach\.

We present*Data Turnstile*, a framework for generating high\-quality synthetic training data for tool\-use\. In contrast to single\-shot generation, Turnstile decomposes each interaction into a directed acyclic graph \(DAG\) of typed ‘roles’: user queries, reasoning traces, API calls, execution outputs, and assistant responses\. Each role is generated independently with focused constraints and validation, retrying with error feedback on failure\. This decomposition has two key benefits\. First, it imposes quality checks at each step to catch cascading errors\. Second, it reduces the per\-step generation task to a complexity that cheap, open\-weight models handle reliably, eliminating the need for proprietary LLMs\. This enables local execution and rapid iteration cycles at a fraction of the cost\. Our methodology enables generation of training data of varying complexity, from simple single\-turn user requests to nuanced, multi\-step agentic workflows, while maintaining structural validation throughout\.*We open\-source the framework, enabling any team to generate targeted tool\-use data for their own APIs\.*

We demonstrate effectiveness on two challenging benchmarks\. On the BFCL single\-turn function calling benchmark\[[17](https://arxiv.org/html/2607.29250#bib.bib22)\], a Qwen3\-0\.6B model trained on Turnstile data achieves 75\.9% accuracy \(base: 67\.4%\), within touching distance of the base Qwen3\-4B\-Instruct \(79\.9%\), a model almost 7×\\timeslarger\. Onτ2\\tau^\{2\}\-bench’s multi\-turn Telecom benchmark\[[4](https://arxiv.org/html/2607.29250#bib.bib24)\], the Qwen3 1\.7B and 4B trained on our data match or beat Qwen2\.5\-32B\-Instruct \(Figure[1](https://arxiv.org/html/2607.29250#S1.F1)\)\. Additionally, Turnstile\-trained SLMs achieve good performance even without chain\-of\-thought \(CoT\)\[[26](https://arxiv.org/html/2607.29250#bib.bib31)\]; avoiding the inference\-time reasoning overhead for latency\-sensitive deployments\.

#### Contributions\.

Our contributions are as follows:

- •We present Data Turnstile222[https://github\.com/amazon\-science/data\-turnstile](https://github.com/amazon-science/data-turnstile), a scalable open\-source framework for generating function\-calling data from custom APIs, using a step\-wise method controlling quality and diversity\.
- •We empirically show the efficacy of our approach, evaluating on BFCL andτ2\\tau^\{2\}\-bench across three model scales \(0\.6B \- 4B\) with ablations on CoT and tool\-call weighted SFT loss\.
- •

![Refer to caption](https://arxiv.org/html/2607.29250v1/figures/modified_st_diag.png)Figure 2:Data Turnstile Overview: role\-wise generation with per\-step validation and error\-feedback based retry\.

## 2Data Turnstile

Generating high quality function\-calling data is challenging: user requests must be realistic, API calls must be syntactically well\-formed, API parameters must be grounded \(hallucination\-free\), API execution outputs must be detailed and plausible, etc\. The complexity compounds further when there are multiple user turns and API calls involved, as in real\-world workflows\. LLMs are well\-known to solve complex tasks by decomposing them into tractable subproblems, whether implicitly through internal CoT reasoning, or explicitly through targeted decomposition\[[33](https://arxiv.org/html/2607.29250#bib.bib32)\]\. We took inspiration from the latter approach to build Data Turnstile; by breaking down the problem of generating tool\-use interactions into a set of independently constrained and validated generation steps\.

### 2\.1Template\-Based Generation

The core abstraction is an*interaction template*: a DAG𝒯=\(V,E,Θ\)\\mathcal\{T\}=\(V,E,\\Theta\)where each nodev∈Vv\\in Vcorresponds to a*role*, an atomic generation step\. Directed edgesEEencode dependencies between roles, whileΘ\\Thetadefines additional generation specifications and context \(API definitions, user personas, quality checks\)\. A template specifies the skeleton of an interaction by codifying its*structure*, without the content\. The interaction content itself varies with different parameter draws fromΘ\\Theta, meaning a single template can yield many structurally identical but content\-diverse interactions\.

We define five roles for function\-calling data:USER,API\_CALL,API\_OBS\(tool responses\),ASSISTANT, andTHINKING\(CoT reasoning\)\. Generation proceeds sequentially through the template, as shown in Figure[2](https://arxiv.org/html/2607.29250#S1.F2)\. At each step, we construct a prompt comprising: \(1\) the structural context of the full template, \(2\) the outputs of previously generated roles, and \(3\) generation context specific to the current role\. This means the language model has visibility into both what has been generated*and*the outline of what is expected next\. The DAG edges explicitly codify inter\-role dependencies, ensuring each role receives exactly the context it needs while allowing the LLM to plan ahead\. For instance, knowing that a specificAPI\_CALLwill follow, it can craft aUSERrequest which naturally elicits it\. After generation, the output is validated against role\-specific constraints\. If successful, generation proceeds to the next role, otherwise it is retried with error feedback or aborted altogether as required\. Multiple template instantiations are batched together for efficiency, with the generation state maintained independently for each\.

Quality and diversity are two key concerns for synthetic data\. Interaction templates provide the ideal abstraction forstructured diversitywith fine\-grained quality controls, which we discuss next\.

### 2\.2Data Quality

#### Structural validations\.

The interaction template deterministically enforces format compliance at two levels: \(a\) sequence\-level ordering \(API\_OBSmust always follow anAPI\_CALL;THINKINGmust precedeAPI\_CALLorASSISTANT\) and \(b\) per\-role validation \(API\_CALLmust conform to its schema definitions; everyAPI\_OBSmust be a well\-formed JSON with the required fields\)\.

#### Validate\-before\-Generate

Before generating each role, the LLM validates the previous role content against predefined error criteria, catching issues beyond structural validation, such as hallucinated parameters and implausible API observations\.

#### Error Feedback and Early Abort\.

When structural or pre\-generation validations fail for a role, it is retried up to a predefined budget, with additional context through an error feedback loop\. This allows the LLM to rectify previous flaws, and potentially salvage the generation cost of the interaction thus far \(instead of immediately discarding\)\. If issues are irrecoverable, the LLM has the option to trigger anearly abort\. Together, these mechanisms ensure only high\-quality interactions reach the final dataset\.

### 2\.3Data Diversity

#### Template\-level\.

Templates allow interactions with varying structural patterns \(turns, number of API calls, etc\.\), with the distribution controlled with weights\. For instance, we may want 50% single\-API single\-turn, 25% multi\-API single\-turn and 25% multi\-turn data\.

#### Parameter\-level\.

Each template instance is associated with a particularΘ\\Theta, the set of generation specifications\. These typically include the API names for everyAPI\_CALL, a user persona to guide the request, and other information to inform the generation \(likescenariosdiscussed in §[4\.1](https://arxiv.org/html/2607.29250#S4.SS1)\)\. The combinatorial product of templates and parameters ensuresstructured diversity, which cannot be achieved solely with temperature sampling\.

#### Dynamic Template Perturbations\.

We introduce dynamic perturbations to the interaction template for an additional degree of data diversity\. For example, tool execution failures can be introduced to elicit retry behavior from the model\. We can simulate the incomplete information case, where the model is forced to ask for additional clarifications from the user before executing the API call\. These perturbations are critical for generating more realistic data to train robust models that can recover from unexpected scenarios\.

### 2\.4Implementation

Turnstile is implemented as a modular Python package with four core components: \(a\)*Role*: Atomic units of interactions, encapsulating generation constraints and validations; \(b\)*Template*: DAG specifications composed of multiple*Roles*and their dependencies; \(c\)*Builder*: Stateful controllers that orchestrate the generation loop \(Figure[2](https://arxiv.org/html/2607.29250#S1.F2)\); \(d\)*API Library*: Pluggable wrappers over a set of API definitions\. The specifications of a generation run are controlled by a top\-level config, including template distribution, API library, teacher LLM, and other hyperparameters\. The LLM itself is served by vLLM\[[11](https://arxiv.org/html/2607.29250#bib.bib33)\]\. We built the initial framework from scratch with no AI assistance\. For the open\-source package, we used Qwen2\.5\-32B\-Instruct to assist with code clean\-up and documentation\.

#### Generation Efficiency\.

Per\-role generation produces approximately the same number of total output tokens as single\-shot generation\. The overhead is additional LLM calls \(one per role rather than one per interaction\) and prompt prefills\. In practice, this overhead is modest: \(1\) Each prefill is smaller than single\-shot generation prompts, as only context specific to the current generation step is provided\. \(2\) vLLM’s paged attention\[[11](https://arxiv.org/html/2607.29250#bib.bib33)\]ensures efficient KV\-cache re\-use across requests and steps\. \(3\) Continuous batching\[[31](https://arxiv.org/html/2607.29250#bib.bib34)\]and chunked prefill\[[1](https://arxiv.org/html/2607.29250#bib.bib35)\]amortizes the prefill cost across decoding steps\. \(4\) The error retry mechanism drastically improves generation success \(discussed in §[3\.5](https://arxiv.org/html/2607.29250#S3.SS5)\)\.

## 3BFCL: Single Turn Function\-Calling

The Berkeley Function Calling Leaderboard \(BFCL\)\[[17](https://arxiv.org/html/2607.29250#bib.bib22)\]is a popular benchmark for agentic models\. We focus on the single\-turn test sets, which contain six categories:*Simple*\(single tool, single API call expected\),*Multiple*\(many tools, single API call expected\),*Parallel*\(single tool, many API calls expected\),*Parallel Multiple*\(many tools, many API calls expected\),*Relevance*\(tools available but unnecessary\), and*Irrelevance*\(no applicable tool\)\. For each test interaction, tool definitions are provided in the prompt, and correctness is evaluated via AST parsing of function calls\.

### 3\.1Single Turn Data Generation with Turnstile

In this section, we describe how Turnstile \(§[2](https://arxiv.org/html/2607.29250#S2)\) is applied to generate single\-turn function\-calling data at scale, targeting general\-purpose tool\-use ability across arbitrary API definitions\.

#### Templates\.

We define the following templates that closely mimic the BFCL categories: single\-call \(one API invoked\), parallel\-single \(same API calledNNtimes with different arguments\), parallel\-multiple \(NNdifferent APIs invoked simultaneously\), and multi\-turn \(one API per turn over multiple turns\)\. Specific API names are assigned to template instances, by sampling from the global set of API definitions\. In each turn, reasoning traces are generated following the user request to assess the impact of CoT on function\-calling performance\. We implement a perturbation on the single\-call template for generating irrelevance data, by generating the user request pertinent to a tool that is not available to the model\.

#### APIs and Datasets\.

We generate four large\-scale synthetic datasets from different API sets: xLAM\[[32](https://arxiv.org/html/2607.29250#bib.bib23)\]and Glaive\[[2](https://arxiv.org/html/2607.29250#bib.bib14)\]\(definitions extracted from open\-source datasets\), Synthetic Domains \(synthetically generated APIs, details in §[3\.4](https://arxiv.org/html/2607.29250#S3.SS4)\), and BFCL \(definitions derived from the BFCL single\-turn test set\)\. Each source contains 1\-3K definitions, providing a large pool for generation at scale\. The dataset statistics are summarized in Table[1](https://arxiv.org/html/2607.29250#S3.T1)\. We used Qwen2\.5\-32B\-Instruct model as the generation LLM on a local p5\.48xlarge instance \(8 H100 GPUs\)\. Turnstile’s diversity levers produce datasets that are larger and measurably more diverse than raw open\-source data from the same APIs \(§[3\.5](https://arxiv.org/html/2607.29250#S3.SS5)\)\. On average, Turnstile datasets have more roles and API calls than the open\-source versions\. This is due to the multi\-turn multi\-API templates used for generation, along with additional CoT reasoning traces present in our data\.

Table 1:Key Dataset Statistics\. OS refers to open\-source datasets, without CoT reasoning\. All others are Turnstile generated and contain CoT traces\.Per Interaction: avg \(min\-max\)DomainSize\# APIsRolesAPI CallsTokensxLAM \(OS\)∼\\sim59K3,6022\.7 \(2\-10\)1\.6 \(1\-9\)240 \(74\-1\.8K\)Glaive \(OS\)∼\\sim78K1,0695\.9 \(2\-14\)1\.7 \(1\-2\)279 \(76\-2\.1K\)xLAM∼\\sim230K3,1787\.0 \(5\-10\)1\.7 \(1\-2\)520 \(73\-2\.5K\)Glaive∼\\sim199K1,06411\.2 \(6\-16\)2\.7 \(1\-4\)623 \(109\-1\.8K\)Synth\.∼\\sim100K1,0259\.9 \(5\-15\)2\.8 \(1\-4\)972 \(235\-6\.8K\)BFCL∼\\sim159K1,2609\.6 \(6\-26\)2\.0 \(1\-5\)540 \(110\-2\.9K\)Telecom∼\\sim36K1438\.9 \(17\-81\)4\.1 \(2\-13\)1\.6K \(649\-3\.1K\)
#### Distractors\.

During training, we augment each interaction with semantically similar distractor APIs alongside the ones present in the interaction, forcing the model to discriminate between plausible alternatives and select the appropriate tool\.

### 3\.2Experiment Setup

#### Models\.

We fine\-tune Qwen3\-0\.6B on different combinations of function\-calling data, comparing against Qwen3 base models\[[29](https://arxiv.org/html/2607.29250#bib.bib3)\]at three scales \(0\.6B, 1\.7B, 4B\) without fine\-tuning\. We evaluate the models in two inference modes:*think*\(explicit CoT before function calls\) and*no\-think*\(direct output\), to assess the performance gap\.

Table 2:BFCL single\-turn results on Qwen3 models\.Raw OS: unprocessed open\-source xLAM\+Glaive data\.Turnstile\-OS: Turnstile\-generated data from xLAM\+Glaive APIs\.Turnstile\-OOD:Turnstile\-OS \+ data from Synthetic Domain APIs\.Turnstile \(OOD\+ID\): OOD \+ adds data generated from BFCL’s own API definitions\.Bold= best overall;underline= best among SFT models\.OverallDetectionNon\-Live \(AST\)Live \(AST\)ModelAvgNon\-LiveLiveRel\.Irrel\.SimpleMultiParaPara\-MSimpleMultiParaPara\-MBase Models \(think eval\)Qwen3\-4B79\.987\.172\.688\.275\.378\.993\.589\.589\.580\.276\.162\.566\.7Qwen3\-1\.7B78\.481\.475\.352\.983\.372\.389\.580\.581\.074\.869\.843\.879\.2Qwen3\-0\.6B67\.472\.362\.552\.981\.063\.375\.074\.066\.560\.949\.362\.545\.8Base Models \(no\-think eval\)Qwen3\-4B71\.177\.664\.582\.452\.674\.990\.087\.584\.081\.469\.268\.866\.7Qwen3\-1\.7B67\.076\.357\.7100\.056\.272\.990\.580\.576\.070\.959\.843\.858\.3Qwen3\-0\.6B58\.263\.053\.476\.590\.657\.062\.554\.549\.538\.427\.512\.529\.2SFT\-0\.6B\-no\-think: Methodology ablation \(same xLAM\+Glaive APIs\)Raw\-OS55\.163\.946\.276\.535\.767\.284\.573\.065\.560\.546\.331\.345\.8Turnstile\-OS70\.475\.965\.088\.277\.570\.989\.075\.062\.059\.962\.562\.525\.0SFT\-0\.6B\-no\-think: Different API SourcesTurnstile\-OOD72\.978\.667\.276\.580\.272\.092\.075\.570\.560\.561\.550\.016\.7Turnstile\-OOD\+ID75\.981\.570\.570\.681\.872\.994\.076\.578\.068\.665\.943\.845\.8SFT\-0\.6B\-think \(trained in thinking mode; think eval\)Turnstile\-OOD69\.776\.46376\.577\.069\.991\.076\.065\.058\.156\.143\.816\.7Turnstile\-OOD\+ID72\.879\.865\.870\.681\.870\.894\.079\.575\.066\.762\.743\.829\.2

#### Experiments\.

We design our experiments to investigate single\-turn performance, and evaluate the following:

1. 1\.Impact of training on existing open\-source datasets \(xLAM\-OS and Glaive\-OS\)\. This is a baseline to compare Turnstile SFT against\.
2. 2\.Impact of out\-of\-distribution \(OOD\) Turnstile data, generated using xLAM, Glaive, and Synthetic Domains APIs444OOD APIs have just 2% overlap with BFCL evaluation APIs\., measuring whether high\-quality tool calling data improves performance on unseen evaluation APIs\.
3. 3\.Impact of in\-distribution \(ID\) Turnstile data, generated from BFCL’s APIs, to assess whether training on the target API schemas provides additional benefit\. This is particularly relevant for SLMs deployed on a fixed set of domain APIs\.

#### Training Details\.

All models were trained identically, with datasets sampled proportional to their volume\. We perform full fine\-tuning using AdamW with a learning rate of5×10−55\{\\times\}10^\{\-5\}, linear warmup over 50 steps, an effective batch size of 64, and a maximum sequence length of 4096\. We train in both no\-think and think modes; each model is evaluated in its corresponding training mode\.

### 3\.3Results

Table[2](https://arxiv.org/html/2607.29250#S3.T2)presents our main single\-turn results\. BFCL contains two splits: ‘Non\-live’ \(synthetic APIs\) and ‘Live’ \(real\-world APIs\)\.

#### Turnstile SFT vs\. base models

\(Rows: SFT no\-think vs\. Base think/no\-think\.\) Turnstile \(OOD\+ID\) no\-think achieves 75\.9%, surpassing the base 0\.6B in both modes: think \(67\.4%\) and no\-think \(58\.2%\)\. This is notable for SLM deployment since Turnstile SFT without CoT exceeds performance that the base model can only achieve with CoT overhead\. It also approaches the 3×\\timeslarger Qwen3\-1\.7B \(78\.4%\) and the 7×\\timeslarger Qwen3\-4B \(79\.9%\) base models with thinking\.

#### Impact of generation methodology

\(Rows: Turnstile\-OS vs Raw\-OS\) The most controlled comparison is between Raw\-OS and Turnstile\-OS, since both draw from the same xLAM and Glaive API schemas; only the generation methodology differs\. Turnstile\-OS achieves 70\.4% versus 55\.1% for Raw\-OS, a gain of \+15\.3 percentage points \(pp\) attributable to generation methodology alone\. The base 0\.6B already scores 67\.4% with thinking; open\-source SFT*degrades*this native ability to 55\.1%, whereas Turnstile lifts it well beyond without CoT overhead\. Per\-category analysis reveals two failure modes of open\-source data: irrelevance detection collapses from 81% \(in base 0\.6B\) to 35\.7% \(every training sample invokes a tool, so the model never learns to refuse\), and live accuracy drops to 46\.2%\. Turnstile addresses both through dynamic irrelevance injection \(recovering refusal to 77\.5%\) and template\-controlled structural diversity \(65\.0% on live APIs without training on those schemas\)\.

#### Impact of API diversity and target alignment

\(Rows: SFT no\-think, Turnstile\-OOD and OOD\+ID\.\) To study the effect of API diversity, we used the Synthetic Domains dataset generated with Turnstile\. Adding this to Turnstile\-OS \(forming Turnstile\-OOD\) yields a \+2\.5pp gain over Turnstile\-OS \(70\.4%→\\to72\.9%\), with improvements across non\-live and live splits\. The broader API coverage helps the model generalize to unseen evaluation schemas\. Further adding Turnstile data generated from BFCL’s own API schemas \(Turnstile\-OOD\+ID\) yields \+3pp \(72\.9%→\\to75\.9%\), distributed across non\-live \(\+2\.9%\) and live \(\+3\.3%\), with the largest gain on parallel\-multiple \(70\.5%→\\to78\.0%\)\. OOD data teaches the core function\-calling skill across diverse APIs, while ID data provides incremental alignment to the target schema conventions\. This validates Turnstile’s practical utility for deployment, where practitioners can generate targeted data for their own domain and use cases\.

#### Impact of CoT

\(Rows: SFT think vs\. SFT no\-think, Turnstile OOD\+ID\.\) No\-think Turnstile \(OOD\+ID\) outperforms its think counterpart \(75\.9% vs\. 72\.8%\), with the gap largest on live APIs \(70\.5% vs\. 65\.8%\) and parallel\-multiple patterns \(45\.8% vs\. 29\.2% live\)\. Base models show the opposite pattern, with all three dropping significantly without CoT \(0\.6B: 67\.4%→\\to58\.2%, 1\.7B: 78\.4%→\\to67\.0%, 4B: 79\.9%→\\to71\.1%\), indicating a learned dependence on reasoning for tool\-calling\. Turnstile SFT removes this dependency entirely\. We identify two failure modes of CoT in single\-turn settings:*irrelevance rationalization*\(reasoning into calling inapplicable APIs\) and*parameter overthinking*\(over\-interpreting ambiguous constraints on parallel calls\)\. A qualitative analysis is provided in Appendix[A](https://arxiv.org/html/2607.29250#A1)\.

### 3\.4Released Dataset: Synthetic Domains

We publicly release the Synthetic Domains dataset under the CC BY\-NC 4\.0 license\. It comprises∼\{\\sim\}100K Turnstile\-generated interactions over 1,025 APIs spanning 50\+ categories \(finance, weather, maps, e\-commerce, etc\.\)\. It covers single\-call, parallel, parallel\-multiple, and multi\-turn templates, with CoT reasoning traces\.

#### Standalone evaluation\.

Training Qwen3\-0\.6B on this dataset alone achieves 67\.4% on BFCL single\-turn, surpassing both the unprocessed open\-source baseline \(Raw\-OS, 55\.1%\) and the base 0\.6B no\-think model \(58\.2%\)\. This demonstrates that the released data is directly useful for improving SLM tool\-calling ability\.

#### Human assessment\.

Two annotators rated 100 random interactions on*correctness*,*naturalness*, and*groundedness*using a Likert scale of 1–5\. On average, we obtained scores of 4\.4, 3\.8, and 4\.5 respectively, indicating overall high quality of the synthetic data\.

### 3\.5Data Generation: Analysis

In this section, we discuss insights we gathered during the generation of the Synthetic Domains dataset\.

#### Impact of Error Retries\.

We started with a seed size of 125K template instantiations, to generate∼\\sim100K interactions \(∼\\sim84% success rate\)\. Of the successful interactions,∼\\sim22% encountered at least one error during the step\-wise generation, but eventually succeeded due to Turnstile’s*retry with error feedback*\. Therefore this mechanism is critical for maintaining higher success rates, and enabling local generation with smaller less powerful models \(like Qwen\-32B\)\. Interestingly, 87% of the errors flagged were through structural validations \(e\.g\. invalid API calls\) while the rest were through the per\-role LLM\-as\-Judge \(primarily catching hallucinations\)\.

Table 3:Diversity comparison between Turnstile\-generated data and open\-source data\. All metrics are in range \[0, 1\]; 1\.0 indicates maximum diversity\. Tools≥\\geq10 shows the percentage of tools qualifying for Arg Richness computation\.ToolBalanceCallSeq\.StructureArgRichnessTools≥\\geq10 \(%\)xLAM \(OS\)0\.920\.910\.850\.8471%Glaive \(OS\)0\.630\.670\.620\.6126%xLAM0\.990\.950\.910\.8999%Glaive0\.980\.940\.960\.89100%Synth\. Dom\.0\.990\.940\.950\.9899%
#### Failure rates vs complexity\.

Even with role\-wise generation in Turnstile, we find that the failure rates increase drastically with increase in the complexity of the interaction\. For single\-turn interactions, the failure rate increases with the number of function calls: 7%, 16%, 18% and 22% for 1, 2, 3 and 4 parallel calls respectively\. In multi\-turn interactions \(with one function call per turn\), we see failure rates of 7%, 9% and 13% for 1, 2 and 3 turns respectively\.

#### Diversity analysis\.

We quantify the diversity of Turnstile data against the open\-source datasets \(xLAM, Glaive\) using four metrics \(Table[3](https://arxiv.org/html/2607.29250#S3.T3), definitions in Appendix[B](https://arxiv.org/html/2607.29250#A2)\): \(a\)*Tool Balance*: the normalized entropy of API call frequencies, capturing how evenly tools are exercised; \(b\)*Call Sequence*: the normalized entropy over the distribution of per\-example tool\-call orderings; \(c\)*Structure*: 4\-gram normalized entropy over role sequences \(USER, THINK, API\_CALL, API\_OBS, ASST\), capturing variety in interaction patterns; \(d\)*Arg Richness*: argument\-value variety per tool\. It is the fraction of unique argument\-value combinations when samplingnncalls per tool \(repeated 50 times\), averaged across all qualifying tools\. We choosen=10n\{=\}10to retain sufficient qualifying tools for comparison in OS data\.

Turnstile data consistently outperforms open\-source data across all four metrics\. Tool Balance metric indicates near\-uniform API usage in Turnstile, while open\-source \(OS\) shows skewed call distributions\. The gaps on Call Sequence \(0\.94\-0\.95 vs 0\.67\-0\.91\) and Structure \(0\.91\-0\.96 vs 0\.62\-0\.85\) highlight the limited structural patterns in OS data\. Arg Richness shows a similar trend, with Turnstile achieving 0\.89\-0\.98 compared to 0\.61\-0\.84 for OS data\. This validates that our template\-based generation produces measurably more diverse data, even when drawing from the same API pools\.

![Refer to caption](https://arxiv.org/html/2607.29250v1/figures/mt_framework.png)Figure 3:Multi\-turn data generation forτ2\\tau^\{2\}\-bench Telecom: reusable*Issues*compose into*Scenarios*with dynamic perturbations\.

## 4τ2\\tau^\{2\}\-bench: Multi Turn Agentic Workflows

τ2\\tau^\{2\}\-bench\[[4](https://arxiv.org/html/2607.29250#bib.bib24)\]is a multi\-turn conversational benchmark that evaluates agentic tool use with user turns simulated by LLMs\. It tests policy\-adherent tool use, multi\-step reasoning, and error recovery in realistic customer\-service scenarios555We choseτ2\\tau^\{2\}\-bench over the BFCL multi\-turn benchmark, to demonstrate the capabilities of Turnstile in the setting of adherence to a predefined policy document\. It provides 4 evaluation domains: Banking, Retail, Airline and Telecom\. We focus on the Telecom domain for our experiments, comprising 114 tasks requiring diagnosis and resolution of various issues\. Evaluations report pass^1, the percent of successful tasks averaged across four trials\.

### 4\.1Policy\-Based Generation with Turnstile

There are three key differences between theτ2\\tau^\{2\}\-bench Telecom and single\-turn BFCL benchmark: \(1\) agent is expected to adhere to a policy document, \(2\) only 14 APIs in comparison to 1K\+, and \(3\) much longer interactions, with emphasis on conversational ability for facilitating user\-side debugging along with agent\-side tools\.

#### Issue and Scenario

We define two new concepts to codify and enable policy\-driven data generation\. An*Issue*minimally specifies a user problem, diagnosis steps/tools, and actions to fix it, typically consisting of 4\-8 generation*Roles*\. For example, the issue of ’Data Roaming Disabled’ encapsulates the relevant root cause, diagnosis steps and fixes\. A*Scenario*is a curated composition of one or more issues, with shared workflows \(such as authentication\) and additional generation context\. Figure[3](https://arxiv.org/html/2607.29250#S3.F3)shows two separate scenarios with the same problem symptom \(*‘Data not working’*\), but with one \(*‘device\_roaming\_only’*\) or two \(*‘device\_roaming\_only’*\+*‘device\_and\_account\_roaming’*\) issues as actual root causes\. In other words,*Issues*are building blocks that can be used to compose scenarios that simulate real\-world complexity\. We derive a total of 17 issues and 34 composed scenarios from the policy document to construct a diverse and representative set of workflows\.

#### Quality and Diversity

Multi\-turn templates are dynamically assembled from the scenario and its constituent issues\. We generate detailed reasoning traces to help the model understand the required assistant workflows\. For additional diversity, we use template perturbations for simulating API execution failures, asking for missing information, and user personas \(un\-cooperative, tech\-illiterate, etc\.\)\. Figure[3](https://arxiv.org/html/2607.29250#S3.F3)shows an example of the API execution failure, as applied to an interaction\. Additionally, we chain multiple scenarios together to create more complex interactions, e\.g\. by joining with an*‘Is there anything else I can help with?’*assistant turn\.

#### Dataset\.

We generated training data for the Telecom domain spanning the 14 APIs\. The dataset comprises∼\\sim35K interactions with an average of 39 roles per conversation and an average sequence length of 1\.6K tokens \(Table[1](https://arxiv.org/html/2607.29250#S3.T1)\)\. Note that this is substantially longer \(33–4×4\\times\) and more complex than the single\-turn data, as the model needs to maintain state and reason over extended context\.

Table 4:Multi\-turn results onτ2\\tau^\{2\}\-bench Telecom \(114 tasks, 4 trials, pass^1 \[95% CI\] \)\. SFT models trained on Turnstile Telecom data\. Weighted SFT applies 5×\\timesloss on tool\-call tokens\. No\-think models are trained/evaluated without CoT\.

### 4\.2Experimental Setup

We fine\-tune Qwen3 models at three scales \(0\.6B, 1\.7B, 4B\) on Turnstile\-generated Telecom data and evaluate against base models at the same scales, as well as Qwen3\-32B and Qwen2\.5\-32B\-Instruct which serve as upper\-bound references without any fine\-tuning\. We use Qwen2\.5\-32B\-Instruct as the user simulator LLM, as no instruction\-tuned Qwen3\-32B variant is publicly available\.

#### Training Details\.

We perform SFT using AdamW with a learning rate of5×10−55\{\\times\}10^\{\-5\}, batch size of 64, and a maximum sequence length of 8192\. DeepSpeed ZeRO Stage 2\[[19](https://arxiv.org/html/2607.29250#bib.bib2)\]is used for 1\.7B and 4B models\. All models are trained with CoT/thinking enabled until convergence, determined via checkpoint sweeps on validation set\. We also experiment with a*tool\-call weighted*variant that applies 5×\\timesloss weight \(multiplier selected via validation\) on the*API call*tokens\. This is to help the model to prioritize correct API invocations, which are a small but crucial fraction of total tokens\.

### 4\.3Results

Table[4](https://arxiv.org/html/2607.29250#S4.T4)and Figure[1](https://arxiv.org/html/2607.29250#S1.F1)summarize our multi\-turn results\.

#### Impact of Domain Adaptation\.

\(Columns: Base vs\. SFT\)\. All three models benefit from the SFT, with absolute gains of \+19\.5pp \(0\.6B\), \+20\.6pp \(1\.7B\), and \+25\.2pp \(4B\) over their respective baselines \(with non\-overlapping 95% CIs\)\. Relative improvements are larger for smaller models \(6\.5×\\timesfor 0\.6B vs\. 3\.2×\\timesfor 4B\), confirming the model capacity bottleneck\. Notably, Qwen3\-4B \(36\.6%\) surpasses Qwen2\.5\-32B\-Instruct \(27\.4%\) by \+9\.2pp despite being 8×\\timessmaller, and Qwen3\-1\.7B \(27\.2%\) achieves parity at 19×\\timessmaller\. Even the 0\.6B model \(23%\) outperforms Qwen3\-32B \(16\.2%\) by \+6\.8pp\. These gains come from 35K interactions generated entirely from a 14\-API policy specification with no human\-written examples, validating Turnstile’s practical value for domain adaptation\. One can specify their APIs and domain policy, generate data with Turnstile, and obtain a competent small model without manual data collection or annotation\.

#### Tool\-call Weighted SFT

\(Columns: SFT vs\. Tool\-call Weighted\)\. Tool\-call weighted SFT shows consistent gains of \+1\.6 to \+3\.9pp across all three scales\. While individual deltas fall within the 95% CIs, the positive impact suggests this direction is worth exploring\.

#### Impact of CoT

\(Columns: SFT vs No\-think SFT\) In contrast to the single\-turn finding, multi\-turn agentic tasks benefit substantially from CoT reasoning\. Without thinking, all models collapse by 11\-22pp\. Thinking enables three capabilities critical for multi\-turn diagnosis: \(1\)*hypothesis formation*\(reasoning about possible causes before acting\), \(2\)*evidence\-based progression*\(interpreting API results to confirm or reject hypotheses\), and \(3\)*multi\-issue discovery*\(after resolving one problem, reasoning that additional issues may exist\)\. Without this deliberative structure, models iterate blindly through available data, latch onto irrelevant signals, and terminate prematurely\. A qualitative comparison is provided in Appendix[A](https://arxiv.org/html/2607.29250#A1)\.

#### Comparison to Single\-Shot\.

We performed a controlled ablation to isolate the impact of our methodology, compared to single\-shot generation\. We generated∼\\sim35k interactions by mirroring the constraints provided to role\-wise generation and enforcing validation checks post\-hoc\. We found major quality issues in the generated data: mainly stemming from oracle information leakage and false success declarations\. With SFT on this data, the SLMs show major degradations, with 1\.7B and 4B models achieving just 3\.5% and 3\.7% respectively\. More details are provided in Appendix[C](https://arxiv.org/html/2607.29250#A3)\.

![Refer to caption](https://arxiv.org/html/2607.29250v1/x1.png)Figure 4:τ2\\tau^\{2\}\-bench Telecom: Error Analysis across different dimensions for Turnstile SFT models \(0\.6B, 1\.7B, 4B\)\.

### 4\.4Error Analysis: Discussion

We further analyze the model performance across model sizes on the 114 evaluation tasks, breaking it down by category \(Figure[4](https://arxiv.org/html/2607.29250#S4.F4)\)\.

#### Category\.

Complaints span three categories \(Mobile Data, Service, MMS\) with a clear difficulty hierarchy: models resolve 40\-60% of Mobile Data tasks but perform poorly on MMS, which requires more complex debugging\. We expect that richer multi\-issue scenarios for harder categories would improve performance further\. We could not invest further in this due to budget constraints\.

#### Complexity\.

Pass rate degrades monotonically with issue count per task, yet models generalize beyond training \(which contains at most 3\-issue scenarios\) to solve tasks with up to 5 issues\. The capacity gap widens with complexity: at 7 issues, 4B retains 18\.8% while 1\.7B and 0\.6B collapse to 3\.1% and 6\.3%\.τ2\\tau^\{2\}\-bench also evaluates across different user personas \(None, Easy and Hard\)\. They reveal a similar capacity threshold: 4B is robust to adversarial users \(37\.5% Hard vs 38\.8% None\), while 0\.6B drops sharply from 36\.3% to 13\.9%\. The models perform comparably with cooperative users\.

#### Termination Behavior\.

Termination reasons reveal differences in failure modes across model sizes\. The 4B model terminates normally in 88\.6% of cases, while 0\.6B gets stuck in hallucination loops \(44\.1%\) or hits the error limit due to malformed tool calls \(4\.8%\)\. This suggests 0\.6B lacks sufficient capacity for multi\-turn reasoning over long context; 1\.7B and 4B are relatively better suited\.

## 5Related Work

#### Synthetic Data for Tool\-Use\.

Training LLMs for function calling increasingly relies on synthetic data generated by stronger models\[[24](https://arxiv.org/html/2607.29250#bib.bib15)\]\. ToolBench\[[27](https://arxiv.org/html/2607.29250#bib.bib17)\]pioneered large\-scale generation using depth\-first decision tree reasoning across 16K real REST APIs with execution\-based validation, while APIGen\[[14](https://arxiv.org/html/2607.29250#bib.bib20)\]demonstrated that post\-hoc three\-stage verification \(format, execution, semantic\) on 60K examples outperforms larger unfiltered corpora\. The field has since expanded to multi\-turn settings: APIGen\-MT\[[18](https://arxiv.org/html/2607.29250#bib.bib21)\]uses blueprint\-to\-trajectory generation with LLM committee scoring, Magnet\[[30](https://arxiv.org/html/2607.29250#bib.bib13)\]and ToolFlow\[[25](https://arxiv.org/html/2607.29250#bib.bib12)\]employ graph\-based dependency modeling, and FunReason\-MT\[[28](https://arxiv.org/html/2607.29250#bib.bib10)\]introduces Environment\-API Graph Interactions with guided chain\-of\-thought\. ToolWeave\[[10](https://arxiv.org/html/2607.29250#bib.bib4)\]demonstrated that open\-weight models can serve as generation agents for controllable multi\-turn synthesis\. However, none of these works support generation from domain policy documents for deployment\-specific API sets, and most rely on organic LLM variation for diversity rather than providing controllable guarantees\. Turnstile adopts execution\-free validation with per\-role error\-feedback retry and explicit template\-controlled diversity\.

#### Open\-Source Data Generation\.

Despite rapid progress, reproducibility remains limited\. Only ToolBench releases usable pipeline code \(requiring OpenAI API plus live API backends\)\. FunReason\-MT releases only model weights without its synthesis pipeline\. Among open\-source tools, Distilabel\[[7](https://arxiv.org/html/2607.29250#bib.bib6)\]and the Azure Function\-Calling Data Synthesizer\[[3](https://arxiv.org/html/2607.29250#bib.bib11)\]reimplement APIGen\-style pipelines but are restricted to single\-turn generation\. CAMEL\-AI\[[12](https://arxiv.org/html/2607.29250#bib.bib5)\]offers multi\-agent data generation via role\-playing but lacks per\-turn structural validation\. To our knowledge, Turnstile is the first open\-source framework that jointly supports multi\-turn generation, per\-role quality validation with retry, template\-controlled diversity, and policy\-adherent generation from domain specifications\.

#### Benchmarks\.

We evaluate on BFCL v3\[[17](https://arxiv.org/html/2607.29250#bib.bib22)\]for single\-turn function calling andτ2\\tau^\{2\}\-bench\[[4](https://arxiv.org/html/2607.29250#bib.bib24)\]for multi\-turn agentic conversations with policy adherence\. Other benchmarks such as NESTFUL\[[5](https://arxiv.org/html/2607.29250#bib.bib9)\]\(nested compositional calls\), T\-Eval\[[8](https://arxiv.org/html/2607.29250#bib.bib8)\]\(multi\-step orchestration\), and GTA\[[22](https://arxiv.org/html/2607.29250#bib.bib7)\]\(executable tool chains\) highlight that multi\-turn, policy\-adherent tool use remains unsolved even for frontier models\.

## 6Conclusion

We presented Data Turnstile, a framework for generating high\-quality synthetic tool\-use data through step\-wise role generation and validation\. It supports both general function\-calling \(from custom API definitions\) and policy\-adherent generation \(from API schemas and policy documents\)\. Turnstile\-trained SLMs \(0\.6B\-4B\) match or surpass zero\-shot 32B models on multi\-turn agentic tasks, and a fine\-tuned 0\.6B achieves 75\.9% on BFCL \(approaching Qwen3\-4B at 79\.9%\), showing that data quality can compensate for model capacity\. We also find CoT reasoning is task\-dependent: it hurts single\-turn function calling through irrelevance rationalization and parameter overthinking, but is critical for multi\-turn workflows requiring sequential diagnosis\. We open\-source the framework and release a synthetic dataset with 1K\+ APIs and 100K\+ interactions\.

#### Broader Impact\.

Turnstile enables on\-premise generation with open\-weight models, avoiding costs and data sovereignty concerns of proprietary APIs, helping democratize synthetic data generation\. However, as with all synthetic data pipelines, the data may reflect biases present in the teacher model or contain harmful tool\-use patterns\. Our per\-step validation and template structure provide natural insertion points for responsible AI guardrails \(e\.g\., content filters, bias checks\) at generation time rather than post\-hoc\.

#### Limitations and Future Work\.

The quality of generated data depends on the teacher LLM capability; stronger models will produce better data\. Template design currently requires domain expertise, which we aim to address through semi\-automated template construction\. In future work, we plan to extend Turnstile to generate preference pairs for reinforcement learning and validate cross\-family transfer with SFT on non\-Qwen models\.

## References

- \[1\]A\. Agrawal, N\. Kedia, A\. Panwar, J\. Mohan, N\. Kwatra, B\. S\. Gulavani, A\. Tumanov, and R\. Ramjee\(2025\-08\)Efficient llm inference via chunked prefills\.SIGOPS Oper\. Syst\. Rev\.59\(1\),pp\. 9–16\.External Links:ISSN 0163\-5980,[Link](https://doi.org/10.1145/3759441.3759444),[Document](https://dx.doi.org/10.1145/3759441.3759444)Cited by:[§2\.4](https://arxiv.org/html/2607.29250#S2.SS4.SSS0.Px1.p1.1)\.
- \[2\]G\. AI\(2023\)Glaive function calling v2 dataset\.Hugging Face\.Note:[https://huggingface\.co/datasets/glaiveai/glaive\-function\-calling\-v2](https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2)Cited by:[§3\.1](https://arxiv.org/html/2607.29250#S3.SS1.SSS0.Px2.p1.1)\.
- \[3\]Azure\-Samples\(2025\)Data synthesizer for function calling\.Note:[https://github\.com/Azure\-Samples/function\-calling\-data\-synthesizer](https://github.com/Azure-Samples/function-calling-data-synthesizer)Accessed: 2026\-07\-22Cited by:[§5](https://arxiv.org/html/2607.29250#S5.SS0.SSS0.Px2.p1.1)\.
- \[4\]V\. Barres, H\. Dong, S\. Ray, X\. Si, and K\. Narasimhan\(2025\)τ2\\tau^\{2\}\-Bench: evaluating conversational agents in a dual\-control environment\.External Links:2506\.07982,[Link](https://arxiv.org/abs/2506.07982)Cited by:[§1](https://arxiv.org/html/2607.29250#S1.SS0.SSS0.Px2.p2.2),[§4](https://arxiv.org/html/2607.29250#S4.p1.1),[§5](https://arxiv.org/html/2607.29250#S5.SS0.SSS0.Px3.p1.1)\.
- \[5\]K\. Basu, I\. Abdelaziz, K\. Kate, M\. Agarwal, M\. Crouse, Y\. Rizk, K\. Bradford, A\. Munawar, S\. Kumaravel, S\. Goyal, X\. Wang, L\. A\. Lastras, and P\. Kapanipathi\(2025\-11\)NESTFUL: a benchmark for evaluating LLMs on nested sequences of API calls\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 33538–33547\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1702/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1702),ISBN 979\-8\-89176\-332\-6Cited by:[§5](https://arxiv.org/html/2607.29250#S5.SS0.SSS0.Px3.p1.1)\.
- \[6\]P\. Belcák, G\. Heinrich, S\. Diao, Y\. Fu, X\. Dong, S\. Muralidharan, Y\. Lin, and P\. Molchanov\(2025\)Small language models are the future of agentic ai\.ArXivabs/2506\.02153\.External Links:[Link](https://api.semanticscholar.org/CorpusID:279119702)Cited by:[§1](https://arxiv.org/html/2607.29250#S1.p1.1)\.
- \[7\]Á\. B\. D\. Canto, G\. M\. Blázquez, A\. P\. Lajarín, and D\. V\. Suero\(2024\)Distilabel: an ai feedback \(aif\) framework for building datasets with and for llms\.GitHub\.Note:[https://github\.com/argilla\-io/distilabel](https://github.com/argilla-io/distilabel)Cited by:[§5](https://arxiv.org/html/2607.29250#S5.SS0.SSS0.Px2.p1.1)\.
- \[8\]Z\. Chen, W\. Du, W\. Zhang, K\. Liu, J\. Liu, M\. Zheng, J\. Zhuo, S\. Zhang, D\. Lin, K\. Chen, and F\. Zhao\(2024\-08\)T\-eval: evaluating the tool utilization capability of large language models step by step\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 9510–9529\.External Links:[Link](https://aclanthology.org/2024.acl-long.515/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.515)Cited by:[§5](https://arxiv.org/html/2607.29250#S5.SS0.SSS0.Px3.p1.1)\.
- \[9\]S\. Gunasekar, Y\. Zhang, J\. Aneja, C\. C\. T\. Mendes, A\. D\. Giorno, S\. Gopi, M\. Javaheripi, P\. Kauffmann, G\. de Rosa, O\. Saarikivi, A\. Salim, S\. Shah, H\. S\. Behl, X\. Wang, S\. Bubeck, R\. Eldan, A\. T\. Kalai, Y\. T\. Lee, and Y\. Li\(2023\)Textbooks are all you need\.ArXivabs/2306\.11644\.External Links:[Link](https://api.semanticscholar.org/CorpusID:259203998)Cited by:[§1](https://arxiv.org/html/2607.29250#S1.p2.1)\.
- \[10\]D\. Khandelwal, G\. P\. Punnavajhala, G\. Bhargav, G\. Pandey, S\. Joshi, H\. Karanam, and D\. Raghu\(2026\)ToolWeave: structured synthesis of complex multi\-turn tool\-calling dialogues\.External Links:2605\.12521,[Link](https://arxiv.org/abs/2605.12521)Cited by:[§5](https://arxiv.org/html/2607.29250#S5.SS0.SSS0.Px1.p1.1)\.
- \[11\]W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica\(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles,Cited by:[§2\.4](https://arxiv.org/html/2607.29250#S2.SS4.SSS0.Px1.p1.1),[§2\.4](https://arxiv.org/html/2607.29250#S2.SS4.p1.1)\.
- \[12\]G\. Li, H\. A\. A\. K\. Hammoud, H\. Itani, D\. Khizbullin, and B\. Ghanem\(2023\)CAMEL: communicative agents for ”mind” exploration of large language model society\.InThirty\-seventh Conference on Neural Information Processing Systems,Cited by:[§5](https://arxiv.org/html/2607.29250#S5.SS0.SSS0.Px2.p1.1)\.
- \[13\]Q\. Lin, M\. Wen, Q\. Peng, G\. Nie, J\. Liao, J\. Wang, X\. Mo, J\. Zhou, C\. Cheng, Y\. Zhao, J\. Wang, and W\. Zhang\(2024\)Hammer: robust function\-calling for on\-device language models via function masking\.External Links:2410\.04587,[Link](https://arxiv.org/abs/2410.04587)Cited by:[§1](https://arxiv.org/html/2607.29250#S1.p2.1)\.
- \[14\]Z\. Liu, T\. Q\. Hoang, J\. Zhang, M\. Zhu, T\. Lan, S\. Kokane, J\. Tan, W\. Yao, Z\. Liu, Y\. Feng, R\. R\. N, L\. Yang, S\. Savarese, J\. C\. Niebles, H\. Wang, S\. Heinecke, and C\. Xiong\(2024\)APIGen: automated PIpeline for generating verifiable and diverse function\-calling datasets\.InThe Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=Jfg3vw2bjx)Cited by:[§1](https://arxiv.org/html/2607.29250#S1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.29250#S1.p2.1),[§5](https://arxiv.org/html/2607.29250#S5.SS0.SSS0.Px1.p1.1)\.
- \[15\]Z\. Lu, X\. Li, D\. Cai, R\. Yi, F\. Liu, W\. Liu, J\. Luan, X\. Zhang, N\. D\. Lane, and M\. Xu\(2025\-07\)Demystifying small language models for edge deployment\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 14747–14764\.External Links:[Link](https://aclanthology.org/2025.acl-long.718/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.718),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2607.29250#S1.p1.1)\.
- \[16\]S\. G\. Patil, T\. Zhang, X\. Wang, and J\. E\. Gonzalez\(2024\)Gorilla: large language model connected with massive APIs\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=tBRNC6YemY)Cited by:[§1](https://arxiv.org/html/2607.29250#S1.p2.1)\.
- \[17\]S\. G\. Patil, H\. Mao, C\. Cheng\-Jie Ji, F\. Yan, V\. Suresh, I\. Stoica, and J\. E\. Gonzalez\(2024\)The berkeley function calling leaderboard \(bfcl\): from tool use to agentic evaluation of large language models\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.29250#S1.SS0.SSS0.Px2.p2.2),[§3](https://arxiv.org/html/2607.29250#S3.p1.1),[§5](https://arxiv.org/html/2607.29250#S5.SS0.SSS0.Px3.p1.1)\.
- \[18\]A\. Prabhakar, Z\. Liu, M\. Zhu, J\. Zhang, T\. M\. Awalgaonkar, S\. Wang, Z\. Liu, H\. Chen, T\. Q\. Hoang, J\. C\. Niebles, S\. Heinecke, W\. Yao, H\. Wang, S\. Savarese, and C\. Xiong\(2026\)APIGen\-MT: agentic pipeline for multi\-turn data generation via simulated agent\-human interplay\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=qk6ORqQ4Cu)Cited by:[§5](https://arxiv.org/html/2607.29250#S5.SS0.SSS0.Px1.p1.1)\.
- \[19\]J\. Rasley, S\. Rajbhandari, O\. Ruwase, and Y\. He\(2020\)DeepSpeed: system optimizations enable training deep learning models with over 100 billion parameters\.InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,KDD ’20,New York, NY, USA,pp\. 3505–3506\.External Links:ISBN 9781450379984,[Link](https://doi.org/10.1145/3394486.3406703),[Document](https://dx.doi.org/10.1145/3394486.3406703)Cited by:[§4\.2](https://arxiv.org/html/2607.29250#S4.SS2.SSS0.Px1.p1.2)\.
- \[20\]T\. Schick, J\. Dwivedi\-Yu, R\. Dessi, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom\(2023\)Toolformer: language models can teach themselves to use tools\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=Yacmpz84TH)Cited by:[§1](https://arxiv.org/html/2607.29250#S1.p2.1)\.
- \[21\]B\. Sorscher, R\. Geirhos, S\. Shekhar, S\. Ganguli, and A\. S\. Morcos\(2022\)Beyond neural scaling laws: beating power law scaling via data pruning\.ArXivabs/2206\.14486\.External Links:[Link](https://api.semanticscholar.org/CorpusID:250113273)Cited by:[§1](https://arxiv.org/html/2607.29250#S1.p2.1)\.
- \[22\]J\. Wang, Z\. Ma, Y\. Li, S\. Zhang, C\. Chen, K\. Chen, and X\. Le\(2024\)GTA: a benchmark for general tool agents\.Advances in Neural Information Processing Systems,pp\. 75749–75790\.Cited by:[§5](https://arxiv.org/html/2607.29250#S5.SS0.SSS0.Px3.p1.1)\.
- \[23\]M\. Wang, Y\. Zhang, B\. Yu, B\. Hao, C\. Peng, Y\. Chen, W\. Zhou, J\. Gu, C\. Zhuang, R\. Guo, W\. Wang, and X\. Zhao\(2026\-02\)Function calling in large language models: industrial practices, challenges, and future directions\.ACM Comput\. Surv\.58\(9\)\.External Links:ISSN 0360\-0300,[Link](https://doi.org/10.1145/3788284),[Document](https://dx.doi.org/10.1145/3788284)Cited by:[§1](https://arxiv.org/html/2607.29250#S1.p1.1)\.
- \[24\]Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. Hajishirzi\(2022\)Self\-instruct: aligning language model with self generated instructions\.Cited by:[§5](https://arxiv.org/html/2607.29250#S5.SS0.SSS0.Px1.p1.1)\.
- \[25\]Z\. Wang, X\. Zeng, W\. Liu, L\. Li, Y\. Wang, L\. Shang, X\. Jiang, Q\. Liu, and K\. Wong\(2025\-04\)ToolFlow: boosting LLM tool\-calling through natural and coherent dialogue synthesis\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 4246–4263\.External Links:[Link](https://aclanthology.org/2025.naacl-long.214/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.214),ISBN 979\-8\-89176\-189\-6Cited by:[§5](https://arxiv.org/html/2607.29250#S5.SS0.SSS0.Px1.p1.1)\.
- \[26\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InProceedings of the 36th International Conference on Neural Information Processing Systems,NIPS ’22,Red Hook, NY, USA\.External Links:ISBN 9781713871088Cited by:[§1](https://arxiv.org/html/2607.29250#S1.SS0.SSS0.Px2.p2.2)\.
- \[27\]Q\. Xu, F\. Hong, B\. Li, C\. Hu, Z\. Chen, and J\. Zhang\(2023\)On the tool manipulation capability of open\-source large language models\.External Links:2305\.16504Cited by:[§1](https://arxiv.org/html/2607.29250#S1.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.29250#S5.SS0.SSS0.Px1.p1.1)\.
- \[28\]Z\. Xu, B\. Hao, Z\. Wang, Y\. Wen, X\. Xu, Y\. Liu, L\. Chen, D\. Wang, M\. Wang, T\. Zhao, Y\. Chen, C\. Peng, J\. Gu, L\. Gan, X\. Zhao, C\. Zhuang, and S\. Gu\(2025\)FunReason\-mt technical report: advanced data synthesis solution for real\-world multi\-turn tool\-use\.External Links:2510\.24645,[Link](https://arxiv.org/abs/2510.24645)Cited by:[§5](https://arxiv.org/html/2607.29250#S5.SS0.SSS0.Px1.p1.1)\.
- \[29\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu\(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§3\.2](https://arxiv.org/html/2607.29250#S3.SS2.SSS0.Px1.p1.1)\.
- \[30\]F\. Yin, Z\. Wang, I\. Hsu, J\. Yan, K\. Jiang, Y\. Chen, J\. Gu, L\. Le, K\. Chang, C\. Lee, H\. Palangi, and T\. Pfister\(2025\-07\)Magnet: multi\-turn tool\-use data synthesis and distillation via graph translation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 32600–32616\.External Links:[Link](https://aclanthology.org/2025.acl-long.1566/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1566),ISBN 979\-8\-89176\-251\-0Cited by:[§5](https://arxiv.org/html/2607.29250#S5.SS0.SSS0.Px1.p1.1)\.
- \[31\]G\. Yu, J\. S\. Jeong, G\. Kim, S\. Kim, and B\. Chun\(2022\-07\)Orca: a distributed serving system for Transformer\-Based generative models\.In16th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 22\),Carlsbad, CA,pp\. 521–538\.External Links:ISBN 978\-1\-939133\-28\-1,[Link](https://www.usenix.org/conference/osdi22/presentation/yu)Cited by:[§2\.4](https://arxiv.org/html/2607.29250#S2.SS4.SSS0.Px1.p1.1)\.
- \[32\]J\. Zhang, T\. Lan, M\. Zhu, Z\. Liu, T\. Hoang, S\. Kokane, W\. Yao, J\. Tan, Z\. Liu, Y\. Feng, J\. C\. Niebles, S\. Heinecke, H\. Wang, S\. Savarese, and C\. Xiong\(2025\-04\)XLAM: a family of large action models to empower AI agent systems\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 11583–11597\.External Links:[Link](https://aclanthology.org/2025.naacl-long.578/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.578),ISBN 979\-8\-89176\-189\-6Cited by:[§1](https://arxiv.org/html/2607.29250#S1.p2.1),[§3\.1](https://arxiv.org/html/2607.29250#S3.SS1.SSS0.Px2.p1.1)\.
- \[33\]D\. Zhou, N\. Schärli, L\. Hou, J\. Wei, N\. Scales, X\. Wang, D\. Schuurmans, C\. Cui, O\. Bousquet, Q\. V\. Le, and E\. H\. Chi\(2023\)Least\-to\-most prompting enables complex reasoning in large language models\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=WZH7099tgfM)Cited by:[§2](https://arxiv.org/html/2607.29250#S2.p1.1)\.

## Appendix AThink vs No\-Think: Qualitative Analysis

We present qualitative examples illustrating why CoT reasoning hurts single\-turn performance but helps multi\-turn agentic tasks\. We identify three single\-turn failure modes \(irrelevance rationalization, parameter overthinking, format drift\) and shows that multi\-turn tasks benefit from structured reasoning\. Detailed examples in Tables[5](https://arxiv.org/html/2607.29250#A1.T5),[6](https://arxiv.org/html/2607.29250#A1.T6)and Table[7](https://arxiv.org/html/2607.29250#A1.T7)\.

Table 5:Single\-turn: Irrelevance rationalization\. Think model reasons itself into calling an inapplicable API; no\-think model correctly refuses\.Table 6:Single\-turn: Parameter overthinking on a parallel\-multiple call\. Think model over\-interprets the user request and introduces a spurious constraint; no\-think model maps parameters directly\.Table 7:Multi\-turnτ2\\tau^\{2\}\-bench Telecom example \(task:data\_saver\_on\|data\_usage\_exceeded\)\. Customer C1001 has 3 lines; user’s line is L1002 \(phone 555\-123\-2002\)\. Think model succeeds via structured diagnosis; no\-think model fails by latching onto an irrelevant line\.Think Model \(1\.7B\)No\-Think Model \(1\.7B\)Diagnosis42 messages,Success14 messages,FailureUser“My mobile data either stops working or is very slow\. I need excellent internet speed\.”Auth“I need to authenticate the user\. I should ask for their phone number…”
get\_customer\_by\_phone\(555\-123\-2002\)→\\toC1001get\_customer\_by\_phone\(555\-123\-2002\)→\\toC1001Both authenticate correctlyLines“L1002’s phone matches the user’s number, confirming this is the correct line…”
get\_details\_by\_id\(L1001\)→\\tophone 2001
get\_details\_by\_id\(L1002\)→\\tophone2002
Stops here\.get\_details\_by\_id\(L1001\)→\\tophone 2001
get\_details\_by\_id\(L1002\)→\\tophone 2002
get\_details\_by\_id\(L1003\)→\\toSuspendedThink reasons L1002 matches user’s phone and stops\. No\-think blindly iterates all lines\.Next“Correct line identified\. I should guide through initial device checks…”→\\toproceeds to diagnose L1002“Your line L1003 is suspended due to a contract issue\. I can transfer you to a specialist…”→\\touser exits \(out of scope\)Think focuses on correct line\. No\-think sees “Suspended” on L1003 and assumes this explains the issue\.Diag\.“Data Saver is on, this could be causing slowness…”
check\_status\_bar\(\)→\\toData Saver ON
toggle\_data\_saver\_mode\(\)→\\toOFF—Hypothesis formation: thinks about possible cause before actingDiag\.“Still failing after Data Saver fix\. Data might be capped…”
run\_speed\_test\(\)→\\toNo Connection
get\_data\_usage\(L1002\)→\\to15\.1/15\.0 GB—Evidence\-based pivot: first fix insufficient, reasons about next hypothesisFix“Data limit exceeded, I need to refuel…”
refuel\_data\(C1001, L1002, 2\.0\)→\\to\+2 GB
run\_speed\_test\(\)→\\to220 Mbps—Multi\-issue discovery: resolves both root causes sequentially
## Appendix BDiversity Metrics: Formal Definitions

We define four metrics used in Table[3](https://arxiv.org/html/2607.29250#S3.T3)\. Let𝒟\\mathcal\{D\}denote a dataset of interactions\.

#### \(a\) Tool Balance\.

Letcic\_\{i\}be the number of calls to tooliiacross all interactions, andKKthe number of unique tools\. Tool Balance is:

Dtool=−∑i=1Kpi​log2⁡pilog2⁡K,where​pi=ci∑jcjD\_\{\\text\{tool\}\}=\\frac\{\-\\sum\_\{i=1\}^\{K\}p\_\{i\}\\log\_\{2\}p\_\{i\}\}\{\\log\_\{2\}K\},\\quad\\text\{where \}p\_\{i\}=\\frac\{c\_\{i\}\}\{\\sum\_\{j\}c\_\{j\}\}Score of 1\.0 means all tools are called equally often; lower values indicate skewed usage\.

#### \(b\) Call Sequence\.

Each interaction is represented as a sequence of tool\-call steps\. Consecutive API calls without an intervening observation are grouped as a parallel batch \(sorted alphabetically\)\. Letsjs\_\{j\}be the canonical tool\-call sequence for interactionjj, andKKthe number of unique sequences\. Call Sequence is the normalized entropy over the frequency distribution of these sequences:

Dseq=−∑k=1Kqk​log2⁡qklog2⁡K,where​qk=\|\{j:sj=k\}\|\|𝒟\|D\_\{\\text\{seq\}\}=\\frac\{\-\\sum\_\{k=1\}^\{K\}q\_\{k\}\\log\_\{2\}q\_\{k\}\}\{\\log\_\{2\}K\},\\quad\\text\{where \}q\_\{k\}=\\frac\{\|\\\{j:s\_\{j\}=k\\\}\|\}\{\|\\mathcal\{D\}\|\}Score of 1\.0 means every interaction has a unique tool\-call ordering\.

#### \(c\) Structure\.

Extract the role sequence for each interaction \(excluding SYSTEM\), e\.g\., \[USER, THINK, API\_CALL, API\_OBS, ASST\]\. Compute all 4\-grams across all interactions\. Structure is the normalized entropy over the 4\-gram frequency distribution:

Dstruct=−∑k=1K4rk​log2⁡rklog2⁡K4D\_\{\\text\{struct\}\}=\\frac\{\-\\sum\_\{k=1\}^\{K\_\{4\}\}r\_\{k\}\\log\_\{2\}r\_\{k\}\}\{\\log\_\{2\}K\_\{4\}\}whererkr\_\{k\}is the frequency of thekk\-th unique 4\-gram andK4K\_\{4\}is the number of unique 4\-grams\. High values indicate diverse interaction structures; low values indicate repetitive patterns\.

#### \(d\) Arg Richness\.

For each toolttwith at leastnncalls, samplenncalls uniformly without replacement\. Canonicalize each call’s arguments as a sorted tuple of \(key, lowercase\-value\) pairs\. Compute uniqueness as the fraction of distinct canonical tuples in the sample\. RepeatRRtimes and average\. Macro\-average across all qualifying tools:

ArgRich=1\|T\|​∑t∈T1R​∑r=1R\|unique​\(sampler​\(t,n\)\)\|n\\text\{ArgRich\}=\\frac\{1\}\{\|T\|\}\\sum\_\{t\\in T\}\\frac\{1\}\{R\}\\sum\_\{r=1\}^\{R\}\\frac\{\|\\text\{unique\}\(\\text\{sample\}\_\{r\}\(t,n\)\)\|\}\{n\}whereT=\{t:\|calls​\(t\)\|≥n\}T=\\\{t:\|\\text\{calls\}\(t\)\|\\geq n\\\}\. We usen=10n\{=\}10,R=50R\{=\}50\. A score of 1\.0 means every sampled call to a tool has a unique set of argument values; lower values indicate repetitive parameterization\.

## Appendix CAblation Details: Single\-Shot Generation

To isolate the contribution of role\-wise decomposition, we implement a single\-shot generation baseline that produces an entire multi\-turn interaction in one LLM call using the same teacher model \(Qwen2\.5\-32B\-Instruct\) and the same scenario distribution forτ2\\tau^\{2\}\-bench\.

#### Generation Details

The single\-shot prompt provides the LLM with: \(1\) role format rules \(identical to role\-wise\), \(2\) all 14 backend API definitions, \(3\) the backend vs\. device\-side tool distinction, and \(4\) a*role sequence outline*derived from the sampled scenario—listing the expected sequence of roles \(e\.g\.,USER: complaint→\\toTHINKING→\\toAPI\_CALL: get\_customer\_by\_phone→\\toAPI\_OBS→\\to…→\\toASST: resolution\)\. This outline mirrors the structure that the role\-wise template enforces, giving the single\-shot baseline equivalent structural guidance\. Scenario\-specific data \(customer identifiers, expected API observation values, device report values\) is provided separately, with instructions to use observation values only inAPI\_OBSroles and to respect information asymmetry and causality between user and assistant roles\.

We generated∼\\sim35k interactions \(matching the role\-wise volume\) using the same vLLM server configuration and generation parameters\. We apply the same structural validators used for role\-wise data: JSON parsing, role sequence validity, API call syntax and schema checking\. No retries are performed, interactions either pass or are discarded\. We observed a 89\.4% pass rate for the structural validations\. The primary failure modes are invalid API observations \(non\-JSON or empty objects\), missing THINKING before action roles, and truncated interactions\. This compares to 96\.4% for role\-wise generation with retries \(88\.9% without retries\)\.

#### Evaluation and Quality Checks\.

We trained the 1\.7B and 4B models on the single\-shot generated data \- both models fared very poorly on the Telecom benchmark, achieving just 3\.5% and 3\.7% respectively\. An analysis of the errors revealed two catastrophic failure modes: \(i\) failed authentication and diagnosis workflows due to hallucinated API parameters, and \(ii\) model gets stuck in irrecoverable action\-reaction loops\.

Manual quality review of the training data confirms that structural validity does not imply semantic correctness\. In a sample of 10 structurally\-valid interactions, we find: \(1\) oracle information leakage in 60% of cases: the assistant uses customer identifiers before any API call reveals them, \(2\) false success declarations where the claimed fix would not satisfy the benchmark’s environment predicate, and \(3\) shallow reasoning traces that narrate actions rather than demonstrating decision\-making\. These defects are structurally undetectable but directly harmful to downstream training\. Turnstile helps minimize such pitfalls by: \(1\) role\-wise generation provides a tractable task to the LLM, with focused constraints, \(2\) the validate\-before\-generate methodology helps detect hallucinations early and prevent cascading errors\.