当更优的轮次不造就更优的智能体：诊断次轮指标与工作流成功率之间的差距

# When Better Turns Do Not Make Better Agents:Diagnosing the Gap Between Next-Turn Metrics and Workflow Success
Source: [https://arxiv.org/html/2609.21187](https://arxiv.org/html/2609.21187)
Xue\-Yong FuAffiliation:Gundeep Singh, Karol Chang, Kevin Sanders, Shi Zong, Tania Habib,Affiliation:Julien Bouvier Tremblay, Shayna Gardiner, Harsh Saini, Matthias Lee,Affiliation:Elena Khasanova, Quinten McNamara, Shashi Bhushan TNAffiliation:\(Bold in Author Names Denotes Equal Contribution\)Affiliation:Dialpad Inc\.

###### Abstract

Agent models are frequently evaluated one decision at a time, where the model predicts the next action based on the gold interaction history, which is scored against a reference\. We investigate whether improvement under this protocol is predictive of improved autonomous workflow execution\. We study pre\-SFT and supervised fine\-tuned \(SFT\) Qwen3 models at 4B and 14B parameters and Gemma 3 models at 4B and 12B parameters on multi\-turn customer\-support workflows\. We find that SFT consistently improves text\-turn success, and that overall next\-turn success increases for every model under gold\-history evaluation\. However, these improvements do not transfer to autonomous workflow execution\. Tool\-specific gains also vary across metrics and models\. None of the four SFT models succeeds under holistic workflow evaluation, with strict trajectory completion reaching at most 10\.4% workflow success\. Our results show that next\-turn evaluation is not a reliable proxy for workflow success, motivating separate reporting of text quality, local action correctness, tool execution, and end\-to\-end task completion\.

## 1Introduction

Language\-model agents deployed in production settings must produce more than just plausible responses\. For instance, a workflow agent needs to generate a factually correct answer, follow the domain policy, invoke tools with state\-dependent arguments\([Schick et al\., 2023](https://arxiv.org/html/2609.21187#bib.bib4);[Qin et al\., 2024](https://arxiv.org/html/2609.21187#bib.bib5)\), gather missing information, incorporate tool results, and recover from failed actions\. An error early in an interaction changes the context for every later decision, so agent quality is fundamentally a property of a trajectory, not the average quality of its turns, a premise underlying recent interactive agent benchmarks\([Yao et al\., 2024](https://arxiv.org/html/2609.21187#bib.bib12);[Barres et al\., 2025](https://arxiv.org/html/2609.21187#bib.bib24)\)\. Nevertheless, next\-turn evaluation, where the gold history precedes every decision and compares the model’s prediction with a reference using a lexical or semantic similarity metric\([Lin, 2004](https://arxiv.org/html/2609.21187#bib.bib27);[Zhang et al\., 2020](https://arxiv.org/html/2609.21187#bib.bib28);[Zha et al\., 2023](https://arxiv.org/html/2609.21187#bib.bib29)\), remains a dominant protocol in practice since it is cheap, reproducible, and verifiable\.

However, next\-turn evaluation has two important limitations\. First, because each prediction is based on the correct history, earlier mistakes cannot affect later decisions\. It therefore measures how well a model responds from a correct state, rather than whether it can build and maintain that state during an interaction\. Second, reference\-based metrics may reward an incorrect response that resembles the reference while penalizing a valid response expressed differently\. This concern has also been noted by[Alkhouli et al\. \(2025\)](https://arxiv.org/html/2609.21187#bib.bib11)in their CONFETTI benchmark, in which they mention that the gold trajectory could “artificially inflate” later\-turn performance through in\-context learning\. Nonetheless, they did not compare the turn\-level evaluation in CONFETTI with end\-to\-end autonomous execution, where the model must continue from its own previous decisions\.

To this end, our study directly measures this gap using multi\-turn customer\-support workflows in the task\-oriented dialogue evaluation setting\([Qin et al\., 2023](https://arxiv.org/html/2609.21187#bib.bib10);[Budzianowski et al\., 2018](https://arxiv.org/html/2609.21187#bib.bib7)\)\. Our research question is whether improvements in gold\-history next\-action scores predict improvements in end\-to\-end workflow success\. We compare pre\-SFT and supervised fine\-tuned \(SFT\)\([Wei et al\., 2022](https://arxiv.org/html/2609.21187#bib.bib1);[Ouyang et al\., 2022](https://arxiv.org/html/2609.21187#bib.bib2)\)Qwen3 models\([Yang et al\., 2025](https://arxiv.org/html/2609.21187#bib.bib26)\)at 4B and 14B parameters and Gemma 3 models\([Gemma Team, 2025](https://arxiv.org/html/2609.21187#bib.bib30)\)at 4B and 12B\. Under gold\-history evaluation, SFT improves text quality and overall turn\-level performance across all model scales\. However, when models must complete workflows using their own previous outputs, nearly all trajectories requiring tool use led to failure\. Thus, the same SFT models that appear effective under turn\-level evaluation are found to be ineffective in end\-to\-end workflow execution\.

In this paper, we investigate three research questions : \(i\) whether turn\-level improvements predict end\-to\-end workflow success, \(ii\) whether aggregate scores conceal differences between text generation and tool execution, and \(iii\) how early errors affect later decisions\. Across multi\-turn workflows, we find that SFT consistently improves gold\-history turn\-level performance but rarely enables successful execution of tool\-requiring workflows\. Separating response generation from tool use further shows that improvements in one capability can hide failures in the other, while trajectory analysis reveals error propagation that gold\-history evaluation cannot capture\. These findings motivate a minimal reporting framework that combines turn\-level scores with end\-to\-end success and trajectory\-level error analysis\.

## 2Related Work

Text\-generation and reference\-based metrics\.Reference\-based metrics such as ROUGE measure lexical overlap[Lin \(2004\)](https://arxiv.org/html/2609.21187#bib.bib27), while BERTScore and AlignScore measure embedding similarity and source–candidate factual alignment, respectively\([Zhang et al\., 2020](https://arxiv.org/html/2609.21187#bib.bib28);[Zha et al\., 2023](https://arxiv.org/html/2609.21187#bib.bib29)\)\. These are well suited to isolated response turns but do not directly measure policy compliance or tool execution\. LLM judges can recognize a valid response that differs from the reference, but their conclusions remain sensitive to the rubric and context supplied to the judge\([Laskar et al\., 2025a](https://arxiv.org/html/2609.21187#bib.bib16);[Gu et al\., 2026](https://arxiv.org/html/2609.21187#bib.bib22)\)\.

Tool\-using language agents\.A separate line of work studies models that select and invoke external tools, typically evaluating tool generalization across a broad catalog of functions\([Schick et al\., 2023](https://arxiv.org/html/2609.21187#bib.bib4);[Yao et al\., 2023](https://arxiv.org/html/2609.21187#bib.bib3);[Li et al\., 2023](https://arxiv.org/html/2609.21187#bib.bib6);[Qin et al\., 2024](https://arxiv.org/html/2609.21187#bib.bib5);[Laskar et al\., 2026](https://arxiv.org/html/2609.21187#bib.bib15)\), with CONFETTI extending this to conversational, turn\-level function calling\([Alkhouli et al\., 2025](https://arxiv.org/html/2609.21187#bib.bib11)\)\. Our setting instead fixes the tool catalog and governing policy per workflow in advance and asks whether an agent can execute the full multi\-turn trajectory, including recovering from errors\.

Task\-oriented dialogue and interactive agent benchmarks\.Task\-oriented dialogue has long studied goal completion through multi\-turn interaction and API\-grounded slot filling\([Qin et al\., 2023](https://arxiv.org/html/2609.21187#bib.bib10);[Budzianowski et al\., 2018](https://arxiv.org/html/2609.21187#bib.bib7);[Rastogi et al\., 2020](https://arxiv.org/html/2609.21187#bib.bib8)\)\. Recent benchmarks extend evaluation to interactive environments[Liu et al\. \(2024\)](https://arxiv.org/html/2609.21187#bib.bib9), including policy\-constrained tool–agent–user interactions inτ\\tau\-bench and dual\-control interaction inτ2\\tau^\{2\}\-bench[Yao et al\. \(2024\)](https://arxiv.org/html/2609.21187#bib.bib12);[Barres et al\. \(2025\)](https://arxiv.org/html/2609.21187#bib.bib24)\.

We complement these benchmarks by holding workflows and models fixed while varying only the evaluation protocol, isolating how gold\-history, strict tool\-matching, and closed\-loop evaluation support different conclusions for identical outputs\.

Supervised fine\-tuning for agent behavior\.Supervised fine\-tuning on instruction and dialogue data is a standard method for adapting models to follow instructions and use tools\([Wei et al\., 2022](https://arxiv.org/html/2609.21187#bib.bib1);[Ouyang et al\., 2022](https://arxiv.org/html/2609.21187#bib.bib2);[Wang et al\., 2023](https://arxiv.org/html/2609.21187#bib.bib25);[Qu et al\., 2025](https://arxiv.org/html/2609.21187#bib.bib21)\), and compact open\-weight models are an increasingly attractive target for this adaptation in cost\-sensitive deployments\([Fu et al\., 2024](https://arxiv.org/html/2609.21187#bib.bib23)\)\. We treat SFT as a fixed, realistic intervention and ask whether its gains, measured under a gold\-history protocol, transfer to closed\-loop execution of the same workflows\.

![Refer to caption](https://arxiv.org/html/2609.21187v1/protocol.png)Figure 1:An Overview of our Evaluation Protocol
## 3Experimental Setup

### 3\.1Data and Models

Training data\.We collect a proprietary dataset from Dialpad111[https://www\.dialpad\.com/](https://www.dialpad.com/)covering 130 customer\-support workflows that are constructed from business conversations[Fu et al\. \(2022\)](https://arxiv.org/html/2609.21187#bib.bib18);[Laskar et al\. \(2023\)](https://arxiv.org/html/2609.21187#bib.bib17);[Khasanova et al\. \(2025\)](https://arxiv.org/html/2609.21187#bib.bib14);[Laskar et al\. \(2025b\)](https://arxiv.org/html/2609.21187#bib.bib13)\. Each workflow is represented by the following: a domain policy, a set of available tool schemas, and one or more user goals\. Following prior work on tool\-use scenario generation and simulated users\([Li et al\., 2023](https://arxiv.org/html/2609.21187#bib.bib6);[Qin et al\., 2024](https://arxiv.org/html/2609.21187#bib.bib5);[Yao et al\., 2024](https://arxiv.org/html/2609.21187#bib.bib12)\), we construct scenarios that combine these elements and generate multi\-turn conversations with two separate instances of GPT\-5[Singh et al\. \(2025\)](https://arxiv.org/html/2609.21187#bib.bib19): one acts as the user and the other as the workflow agent\. The resulting trajectories contain user messages, assistant responses, tool calls, and tool results\.

At first, we generated 1800 conversations\. Then, each generated conversation is independently checked by Claude\-4\.5\-Opus222[https://www\.anthropic\.com/news/claude\-opus\-4\-5](https://www.anthropic.com/news/claude-opus-4-5)and Gemini\-2\.5\-Pro[Comanici et al\. \(2025\)](https://arxiv.org/html/2609.21187#bib.bib20)for logical consistency, missing workflow steps, and policy compliance; a conversation is retained only when both judges approve it\. This filtering removes 773 conversations and leaves 1,027 validated conversations\. We then convert every assistant decision into a next\-action example: the input contains the policy, tools, and preceding history, and the target is either a natural\-language response or a structured tool call\. The final training split contains 5,834 examples \(4,093 text responses and 1,741 tool calls\), with 664 additional validation examples \(466 text and 198 tool\)\. These generated conversations contain 13 turns on average\.

Evaluation data\.The held\-out evaluation split contains 84 validated conversations, yielding 542 next\-action examples: 376 natural\-language responses and 166 tool calls\. Conversations contain 2–18 assistant decisions \(median 6\)\. The split is held out at the conversation level, and no training conversation snippet is reused in evaluation\. Gold\-history metrics score the 542 examples independently; closed\-loop evaluation instead assesses each model across the 84 complete conversations by leveraging their state\-dependent tool results\.

Models\.We compare public Qwen3 models\([Yang et al\., 2025](https://arxiv.org/html/2609.21187#bib.bib26)\)at 4B and 14B parameters and Gemma 3 instruction\-tuned models\([Gemma Team, 2025](https://arxiv.org/html/2609.21187#bib.bib30)\)at 4B and 12B \(Pre\-SFT\) with the corresponding models after full supervised fine\-tuning \(SFT\)\. Every model uses the same training and validation examples and the same five\-epoch recipe\.

### 3\.2Evaluation Protocols

We evaluate every model under five protocols, each targeting a different capability along the path from producing a plausible utterance to completing an autonomous workflow \(see Figure[1](https://arxiv.org/html/2609.21187#S2.F1)\)\.

\(i\) Text similarity\.For every natural\-language decision, the model receives the gold history preceding it, the policy, and the tool definitions, and we compare its response with the reference using the ROUGE\-1 metric\.

\(ii\) Gold\-history turn success\.Gemini\-2\.5\-Pro at temperature zero judges all model\-generated responses \(542 turns, covering both text\-based responses and tool calls\) based on the given policy, tool definitions, gold history, and reference action\. The judge model is required to return a binary success label by assessing action correctness while adhering to the domain policy\.

\(iii\) Strict tool correctness\.For the 166 reference tool\-call decisions, we write a deterministic parsing script that requires an exact function\-name and normalized\-argument match between the prediction and reference\. We report exact accuracy and argument F1, with no partial credit for a plausible but non\-matching action\.

\(iv\) Closed\-loop replay\.We replay all 84 conversations using each model’s own generated assistant history rather than gold history, with a deterministic user simulator supplying user turns\. We use a deterministic parser to evaluate the tool calls\. A correctly matched tool call receives the state\-dependent result, while an invalid call receives an error and up to two retries\. The workflow succeeds only if the model reaches the end of the conversation without any unresolved tool errors\.

\(v\) Holistic workflow judgment\.Gemini\-2\.5\-Pro judges every closed\-loop replay for full user\-goal completion where successful workflows do not have any policy violation and no tool errors\.

Table 1:Results across the evaluation protocols for Qwen3 \(4B/14B\) and Gemma 3 \(4B/12B\) under Zero\-Shot \(ZS\) and Supervised Fine\-Tuning \(SFT\) settings\.

## 4Results

Table[1](https://arxiv.org/html/2609.21187#S3.T1)reports performance across the evaluation protocol, from gold\-history turn\-level evaluation to end\-to\-end workflow execution\. Overall, SFT substantially improves text\-based responses, but these gains transfer weakly to tool execution and rarely yield completed workflows\.

### 4\.1SFT improves text\-based turn\-level performance

Under gold\-history evaluation, SFT improves performance across all model families and scales\. Averaged across the four models, ROUGE\-1 increases by 24\.4 points, from 24\.8 to 49\.2, while LLM\-judged text\-turn success rises by 25\.2 points, from 33\.2% to 58\.4%\. Consequently, all\-turn success improves by 19\.3 points, from 28\.5% to 47\.7%\. These consistent gains show that SFT helps models better predict the expected next response when given the correct interaction history\.

### 4\.2Turn\-level gains fail to extend to tool use

Tool\-specific improvements are substantially smaller\. LLM\-judged tool\-turn success increases by only 5\.9 points on average, from 17\.8% to 23\.7%, compared with 25\.2 points for text turns\. Exact call accuracy increases from 3\.5% to 8\.5%, and argument F1 from 6\.7% to 14\.3%\. These gains are concentrated in Qwen3\-4B and Qwen3\-14B\. Both Gemma 3 models show little or negative improvement; for example, Gemma 3\-4B declines from 3\.0% to 0\.0% in exact call accuracy and from 5\.1% to 0\.5% in argument F1\. Thus, turn\-level scores based on gold history can obscure poor tool use: frequent text turns improve strongly after SFT and raise the overall score even when exact tool execution remains poor or deteriorates\.

### 4\.3Gold\-history overestimates workflow success

The gap widens when models execute workflows using their own prior outputs\. Before SFT, no model completes a tool\-requiring workflow\. After SFT, Qwen3\-4B and Qwen3\-14B complete only 3 and 8 workflows, respectively, while neither Gemma 3 model completes any\. The best result, from Qwen3\-14B, is only 10\.4% workflow completion\. Moreover, deterministic completion does not imply overall interaction success\. The holistic judge identifies no successful tool\-requiring workflow for any model, before or after SFT\. A workflow may complete its tool calls while still failing due to missing information, incorrect responses, policy violations, or improper use of tool results\.

### 4\.4Evaluation protocol changes conclusion

The two protocols yield different conclusions about the same SFT interventions\. Gold\-history evaluation indicates broad improvement: every model achieves higher ROUGE\-1, text\-turn success, and all\-turn success after SFT\. End\-to\-end execution instead shows that nearly all tool\-requiring workflows still fail, with none succeeding under holistic evaluation\. This discrepancy arises because gold\-history evaluation restores the correct state before each decision\. It tests whether a model can predict the next action from a correct history, but not whether it can construct and maintain that history through its own decisions\. End\-to\-end execution exposes this limitation by allowing early errors to propagate\. Turn\-level evaluation is therefore informative but measures a narrower capability than workflow execution\. Reliable agent evaluation should report text quality, tool correctness, workflow completion, and holistic trajectory success separately, rather than treating aggregate next\-turn performance as evidence of end\-to\-end capability\.

## 5Conclusion

Across four model pairs from two families, SFT clearly improves reference\-matched text and next\-action prediction under gold\-history evaluation\. However, tool\-call correctness varies across metrics, model sizes, and families, while end\-to\-end completion remains low as holistic evaluation finds no successful tool\-requiring workflow at any scale\. These results show that strong next\-turn performance does not necessarily indicate that an agent can complete a workflow using its own interaction history\. Future work should validate this finding across additional domains, model families, and interactive environments, while developing training methods that explicitly target error recovery, state maintenance, and end\-to\-end task completion\. More broadly, agent evaluations should combine turn\-level metrics with tool correctness and workflow\-level success to provide a more complete account of agent capability\.

## Limitations

Our evaluation covers two model families and one proprietary customer\-support domain, limiting generalizability\. Closed\-loop replay requires exact reference tool calls and uses deterministic gold user turns, potentially rejecting valid alternatives and underrepresenting real interactions\.

## Ethics Statement

All conversations are synthetic and contain no customer data or personally identifiable information\. To help facilitate future work, sanitized prompt templates are provided in the Appendix \(see Section[A](https://arxiv.org/html/2609.21187#A1)and Section[B](https://arxiv.org/html/2609.21187#A2)\)\.

## References

- Alkhouliet al\.\(2025\)T\. Alkhouli, K\. Margatina, J\. Gung, R\. Shu, C\. Zaghi, M\. Sunkara, and Y\. ZhangCONFETTI: conversational function\-calling evaluation through turn\-level interactions\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 7993–8006\.Cited by:[§1](https://arxiv.org/html/2609.21187#S1.p2.1),[§2](https://arxiv.org/html/2609.21187#S2.p2.1)\.
- Barreset al\.\(2025\)V\. Barres, H\. Dong, S\. Ray, X\. Si, and K\. Narasimhanτ2\\tau^\{2\}\-Bench: evaluating conversational agents in a dual\-control environment\.arXiv preprint arXiv:2506\.07982\.Cited by:[§1](https://arxiv.org/html/2609.21187#S1.p1.1),[§2](https://arxiv.org/html/2609.21187#S2.p3.1)\.
- Budzianowskiet al\.\(2018\)P\. Budzianowski, T\. Wen, B\. Tseng, I\. Casanueva, S\. Ultes, O\. Ramadan, and M\. GašićMultiWOZ: a large\-scale multi\-domain wizard\-of\-oz dataset for task\-oriented dialogue modelling\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 5016–5026\.Cited by:[§1](https://arxiv.org/html/2609.21187#S1.p3.1),[§2](https://arxiv.org/html/2609.21187#S2.p3.1)\.
- Comaniciet al\.\(2025\)G\. Comanici, E\. Bieber, M\. Schaekermann, I\. Pasupat, N\. Sachdeva, I\. Dhillon, M\. Blistein, O\. Ram, D\. Zhang, E\. Rosen,et al\.Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.arXiv preprint arXiv:2507\.06261\.Cited by:[§3\.1](https://arxiv.org/html/2609.21187#S3.SS1.p2.1)\.
- Fuet al\.\(2022\)X\. Fu, C\. Chen, M\. T\. R\. Laskar, S\. B\. Tn, and S\. Corston\-OliverAn effective, performant named entity recognition system for noisy business telephone conversation transcripts\.InProceedings of the Eighth Workshop on Noisy User\-generated Text \(W\-NUT 2022\),pp\. 96–100\.Cited by:[§3\.1](https://arxiv.org/html/2609.21187#S3.SS1.p1.1)\.
- Fuet al\.\(2024\)X\. Fu, M\. T\. R\. Laskar, E\. Khasanova, C\. Chen, and S\. B\. TnTiny titans: can smaller large language models punch above their weight in the real world for meeting summarization?\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 6: Industry Track\),pp\. 387–394\.Cited by:[§2](https://arxiv.org/html/2609.21187#S2.p5.1)\.
- Gemma Team \(2025\)Gemma TeamGemma 3\.arXiv preprint arXiv:2503\.19786\.External Links:[Link](https://arxiv.org/abs/2503.19786)Cited by:[§1](https://arxiv.org/html/2609.21187#S1.p3.1),[§3\.1](https://arxiv.org/html/2609.21187#S3.SS1.p4.1)\.
- Guet al\.\(2026\)J\. Gu, X\. Jiang, Z\. Shi, H\. Tan, X\. Zhai, C\. Xu, W\. Li, Y\. Shen, S\. Ma, H\. Liu,et al\.A survey on llm\-as\-a\-judge\.The Innovation7\(6\)\.Cited by:[§2](https://arxiv.org/html/2609.21187#S2.p1.1)\.
- Khasanovaet al\.\(2025\)E\. Khasanova, H\. Saini, M\. T\. R\. Laskar, X\. Fu, C\. Chen, and S\. B\. TnDacip\-rc: domain adaptive continual instruction pre\-training via reading comprehension on business conversations\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track,pp\. 1867–1877\.Cited by:[§3\.1](https://arxiv.org/html/2609.21187#S3.SS1.p1.1)\.
- Laskaret al\.\(2023\)M\. T\. R\. Laskar, C\. Chen, X\. Fu, M\. Azizi, S\. Bhushan, and S\. Corston\-OliverAI coach assist: an automated approach for call recommendation in contact centers for agent coaching\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 5: Industry Track\),pp\. 599–607\.Cited by:[§3\.1](https://arxiv.org/html/2609.21187#S3.SS1.p1.1)\.
- Laskaret al\.\(2026\)M\. T\. R\. Laskar, X\. Fu, S\. S\. Sarfjoo, Q\. McNamara, J\. Robertson, and S\. B\. TNFrom text to voice: a reproducible and verifiable framework for evaluating tool calling llm agents\.arXiv preprint arXiv:2605\.15104\.Cited by:[§2](https://arxiv.org/html/2609.21187#S2.p2.1)\.
- Laskaret al\.\(2025a\)M\. T\. R\. Laskar, M\. S\. Islam, R\. Mahbub, A\. Masry, M\. Rahman, A\. Bhuiyan, M\. T\. Nayeem, S\. Joty, E\. Hoque, and J\. X\. HuangJudging the judges: can large vision\-language models fairly evaluate chart comprehension and reasoning?\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 6: Industry Track\),pp\. 1203–1216\.Cited by:[§2](https://arxiv.org/html/2609.21187#S2.p1.1)\.
- Laskaret al\.\(2025b\)M\. T\. R\. Laskar, J\. B\. Tremblay, X\. Fu, C\. Chen, and S\. B\. TnAI knowledge assist: an automated approach for the creation of knowledge bases for conversational ai agents\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track,pp\. 1856–1866\.Cited by:[§3\.1](https://arxiv.org/html/2609.21187#S3.SS1.p1.1)\.
- Liet al\.\(2023\)M\. Li, Y\. Zhao, B\. Yu, F\. Song, H\. Li, H\. Yu, Z\. Li, F\. Huang, and Y\. LiApi\-bank: a comprehensive benchmark for tool\-augmented llms\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 3102–3116\.Cited by:[§2](https://arxiv.org/html/2609.21187#S2.p2.1),[§3\.1](https://arxiv.org/html/2609.21187#S3.SS1.p1.1)\.
- Lin \(2004\)C\. LinROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,pp\. 74–81\.Cited by:[§1](https://arxiv.org/html/2609.21187#S1.p1.1),[§2](https://arxiv.org/html/2609.21187#S2.p1.1)\.
- Liuet al\.\(2024\)X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang, S\. Zhang, X\. Deng, A\. Zeng, Z\. Du, C\. Zhang, S\. Shen, T\. Zhang, Y\. Su, H\. Sun, M\. Huang, Y\. Dong, and J\. TangAgentBench: evaluating LLMs as agents\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2609.21187#S2.p3.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2609.21187#S1.p3.1),[§2](https://arxiv.org/html/2609.21187#S2.p5.1)\.
- Qinet al\.\(2023\)L\. Qin, W\. Pan, Q\. Chen, L\. Liao, Z\. Yu, Y\. Zhang, W\. Che, and M\. LiEnd\-to\-end task\-oriented dialogue: a survey of tasks, methods, and future directions\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 5925–5941\.Cited by:[§1](https://arxiv.org/html/2609.21187#S1.p3.1),[§2](https://arxiv.org/html/2609.21187#S2.p3.1)\.
- Qinet al\.\(2024\)Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian,et al\.Toolllm: facilitating large language models to master 16000\+ real\-world apis\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 9695–9717\.Cited by:[§1](https://arxiv.org/html/2609.21187#S1.p1.1),[§2](https://arxiv.org/html/2609.21187#S2.p2.1),[§3\.1](https://arxiv.org/html/2609.21187#S3.SS1.p1.1)\.
- Quet al\.\(2025\)C\. Qu, S\. Dai, X\. Wei, H\. Cai, S\. Wang, D\. Yin, J\. Xu, and J\. WenTool learning with large language models: a survey\.Frontiers of Computer Science19\(8\),pp\. 198343\.Cited by:[§2](https://arxiv.org/html/2609.21187#S2.p5.1)\.
- Rastogiet al\.\(2020\)A\. Rastogi, X\. Zang, S\. Sunkara, R\. Gupta, and P\. KhaitanTowards scalable multi\-domain conversational agents: the schema\-guided dialogue dataset\.InProceedings of the AAAI Conference on Artificial Intelligence,pp\. 8689–8696\.Cited by:[§2](https://arxiv.org/html/2609.21187#S2.p3.1)\.
- Schicket al\.\(2023\)T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, L\. Zettlemoyer, N\. Cancedda, and T\. ScialomToolformer: language models can teach themselves to use tools\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2609.21187#S1.p1.1),[§2](https://arxiv.org/html/2609.21187#S2.p2.1)\.
- Singhet al\.\(2025\)A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram,et al\.Openai gpt\-5 system card\.arXiv preprint arXiv:2601\.03267\.Cited by:[§3\.1](https://arxiv.org/html/2609.21187#S3.SS1.p1.1)\.
- Wanget al\.\(2023\)Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. HajishirziSelf\-instruct: aligning language models with self\-generated instructions\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 13484–13508\.Cited by:[§2](https://arxiv.org/html/2609.21187#S2.p5.1)\.
- Weiet al\.\(2022\)J\. Wei, M\. Bosma, V\. Y\. Zhao, K\. Guu, A\. W\. Yu, B\. Lester, N\. Du, A\. M\. Dai, and Q\. V\. LeFinetuned language models are zero\-shot learners\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2609.21187#S1.p3.1),[§2](https://arxiv.org/html/2609.21187#S2.p5.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2609.21187#S1.p3.1),[§3\.1](https://arxiv.org/html/2609.21187#S3.SS1.p4.1)\.
- Yaoet al\.\(2024\)S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhanτ\\tau\-Bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv preprint arXiv:2406\.12045\.Cited by:[§1](https://arxiv.org/html/2609.21187#S1.p1.1),[§2](https://arxiv.org/html/2609.21187#S2.p3.1),[§3\.1](https://arxiv.org/html/2609.21187#S3.SS1.p1.1)\.
- Yaoet al\.\(2023\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. CaoReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2609.21187#S2.p2.1)\.
- Zhaet al\.\(2023\)Y\. Zha, Y\. Yang, R\. Li, and Z\. HuAlignScore: evaluating factual consistency with a unified alignment function\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 11328–11348\.Cited by:[§1](https://arxiv.org/html/2609.21187#S1.p1.1),[§2](https://arxiv.org/html/2609.21187#S2.p1.1)\.
- Zhanget al\.\(2020\)T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. ArtziBERTScore: evaluating text generation with BERT\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2609.21187#S1.p1.1),[§2](https://arxiv.org/html/2609.21187#S2.p1.1)\.

## Appendix ASanitized Data\-Generation Prompts

The following prompts preserve the roles, inputs, and decisions used in the pipeline while replacing proprietary policies, tools, and company information with placeholders\.

### A\.1Scenario Generation

Given the domain policy and available tools, create arealistic customer\-support scenario for this workflow\.Specify:\- the user’s objective and relevant initial state;\- the desired final state;\- information the user initially knows;\- the expected tool operations; and\- observable criteria for successful completion\.The scenario must be solvable using only the policy andtools supplied below\. Use synthetic identities and data\.DOMAIN POLICY:<SANITIZED\_DOMAIN\_POLICY\>AVAILABLE TOOLS:<SANITIZED\_TOOL\_DEFINITIONS\>

### A\.2GPT\-5 User Simulator

Act as the customer described below\. Remain in characterand pursue the assigned objective naturally\.\- Reveal information gradually and only when appropriate\.\- Answer the agent’s questions consistently with thescenario state and persona\.\- Do not mention tools, policies, prompts, or simulation\.\- End when the goal is met or cannot be completed\.SCENARIO AND OBJECTIVE:<SANITIZED\_SCENARIO\>SYNTHETIC IDENTITY AND PERSONA:<SYNTHETIC\_USER\_PROFILE\>CONVERSATION SO FAR:<DIALOGUE\_HISTORY\>Return only the next user message\.

### A\.3GPT\-5 Workflow Agent

Act as a customer\-support agent operating under thedomain policy below\.\- Follow the workflow and all policy constraints\.\- Maintain the conversation state and request missinginformation when needed\.\- Call an available tool when the workflow requires it\.\- Use tool results before making state\-dependent claims\.\- Handle invalid requests and tool errors safely\.Return either the next natural\-language response or astructured tool call\.DOMAIN POLICY:<SANITIZED\_DOMAIN\_POLICY\>AVAILABLE TOOLS:<SANITIZED\_TOOL\_DEFINITIONS\>CONVERSATION SO FAR:<DIALOGUE\_HISTORY\_WITH\_TOOL\_RESULTS\>

### A\.4Conversation Validation

This prompt is run independently with each validation judge\. A conversation is retained only when both judges approve it\.

Review the complete synthetic conversation for use as aworkflow\-training example\.Approve it only if the agent follows the supplied policy,uses tools appropriately, remains consistent with toolresults, and reaches a sensible outcome for the scenario\.Reject it for a missing required step, contradiction,unsupported claim, invalid tool use, policy violation, orinconsistent simulated\-user behavior\.DOMAIN POLICY AND TOOLS:<SANITIZED\_POLICY\_AND\_TOOLS\>SCENARIO:<SANITIZED\_SCENARIO\>GENERATED CONVERSATION:<SYNTHETIC\_CONVERSATION\>Return JSON only:\{"decision": "APPROVE or REJECT","reasons": \["brief reason"\]\}

## Appendix BSanitized Evaluation\-Judge Prompts

These prompts show the information supplied to the judges and the binary decisions reported in the paper\. Auxiliary diagnostic fields used during analysis are omitted\.

### B\.1Gold\-History Turn Judge

Evaluate one assistant decision in a multi\-turncustomer\-support workflow\. Judge only what should happenat this turn; do not require final workflow completion atan intermediate turn\.Use the policy, tools, correct history, and referenceaction to determine whether the prediction is anacceptable next action\. Alternative wording and safe,semantically equivalent actions are allowed\. A tool callmust select an appropriate function and use argumentsgrounded in the supplied history\.DOMAIN POLICY:<SANITIZED\_DOMAIN\_POLICY\>AVAILABLE TOOLS:<SANITIZED\_TOOL\_DEFINITIONS\>CORRECT HISTORY BEFORE THIS TURN:<GOLD\_HISTORY\>REFERENCE ACTION:<REFERENCE\_ACTION\>MODEL PREDICTION:<MODEL\_OUTPUT\>Return JSON only:\{"turn\_success": true or false,"failure\_category": "none or primary failure","rationale": "brief evidence\-based explanation"\}

### B\.2Holistic Workflow Judge

Evaluate whether the candidate replay completes theuser’s workflow under the supplied domain policy\.Use the reference trajectory to understand the scenario,required state changes, and acceptable final outcome\. Donot reward wording similarity\. Safe alternative wordingor an equivalent valid path is allowed, but requiredverification, policy constraints, and consequential toolactions must be satisfied\.Accepted tool attempts received the recorded tool result\.Rejected attempts were invalid at that workflow state\. Areplay that ends before the user goal is resolved is notsuccessful\.DOMAIN POLICY AND TOOLS:<SANITIZED\_POLICY\_AND\_TOOLS\>REFERENCE TRAJECTORY:<SANITIZED\_REFERENCE\_TRAJECTORY\>CANDIDATE CLOSED\-LOOP REPLAY:<SANITIZED\_CANDIDATE\_REPLAY\>Return JSON only:\{"success": true or false,"failure\_category": "none or primary failure","rationale": "brief evidence\-based explanation"\}