MADS：面向鲁棒对话智能体的多智能体对话模拟

# MADS: Multi-Agent Dialog Simulation for Robust Conversational Agents
Source: [https://arxiv.org/html/2610.12757](https://arxiv.org/html/2610.12757)
Chakraborty Zernik Kulkarni Arora Lin Chiu Di Bari Gupta Stolcke

###### Abstract

Task\-oriented conversational agents remain fragile under realistic conditions, especially when users exhibit non\-cooperative behavior\. Existing function\-calling benchmarks underrepresent adversarial conversation trajectories, thereby limiting the training resources available for developing robust agents\. We presentMADS \(Multi\-Agent Dialog Simulation\), a synthetic data generation pipeline that produces structured, diverse dialogue trajectories\. We inject adversarial conversation, producing validated conversation trajectories that capture realistic, unhappy paths\. Using MADS, we construct MADS\-Bench: a benchmark spanning six domains, 55 personas, and 1095 conversations, with approximately 66% of conversations containing failure\-prone conversation scenarios\. We evaluate the robustness of the trained models using a two\-stage training approach that combines supervised fine\-tuning \(SFT\) and reinforcement learning with verifiable rewards \(RLVR\)\.

###### keywords

conversational agents, LLMs, dialog management, synthetic data, human\-computer interaction

††address:1University of California, Santa Cruz
2Uniphore††email:arkajyoti\.chakraborty, neha\.gupta, justin\.chiu, alessandro\.dibari, andreas\.stolcke@uniphore\.com## 1Introduction

Task\-oriented dialog systems are a central component of modern enterprise conversational AI, powering applications such as customer support, transaction handling, and automated service workflows\. Despite significant advances in natural language understanding and generation driven by large language models \(LLMs\), the dialog management layer in many deployed systems remains largely unchanged\. Most enterprise\-grade systems continue to rely on rigid, finite\-state dialogue managers or pre\-scripted call flows that enforce fixed interaction paths\. While such architectures offer predictability and control, they struggle to handle deviations from expected user behavior, evolving goals, or incomplete information, often leading to brittle interactions and increased user friction\. This limitation is particularly acute in spoken dialogue systems, where users expect natural, flexible interaction and cannot easily navigate rigid menu\-driven flows\.

In contrast, human agents exhibit a high degree of flexibility in task\-oriented conversations\. They maintain rich conversational context, infer missing information when possible, and dynamically decide whether to ask clarifying questions or consult external resources\. Human agents can also revise previous assumptions as new information emerges, enabling graceful recovery from errors or unexpected user inputs\. Bridging this gap between rigid automated systems and adaptive human behavior remains a core challenge in conversational AI\.

Recent work on LLM\-based agents has begun to explore mechanisms for integrating reasoning, action, and interaction with external tools\. Chain\-of\-thought prompting\[[1](https://arxiv.org/html/2610.12757#bib.bib1)\]demonstrated that explicitly eliciting intermediate reasoning steps improves performance on complex tasks\. Building on this idea, ReAct\[[2](https://arxiv.org/html/2610.12757#bib.bib7)\]introduced a framework that interleaves natural\-language reasoning with environment actions, enabling LLMs to plan and execute multi\-step tasks without task\-specific fine\-tuning\. ReSpAct\[[3](https://arxiv.org/html/2610.12757#bib.bib6)\]further extended this paradigm by incorporating an explicit conversational channel, allowing agents to ask clarifying questions and incorporate user feedback during execution\. More recently, Pre\-Act\[[4](https://arxiv.org/html/2610.12757#bib.bib3)\]showed that internalizing multi\-step planning within the model through fine\-tuning can substantially improve action selection and task completion across both conversational and non\-conversational domains\.

While these methods demonstrate promising capabilities across agent reasoning and planning, applying them directly to customer\-enterprise dialog management presents several open challenges\. First, customer\-enterprise interactions often require agents to decide not only what action to take, but whether to act autonomously or defer to the user for clarification\. Existing function\-calling benchmarks\[[5](https://arxiv.org/html/2610.12757#bib.bib4),[6](https://arxiv.org/html/2610.12757#bib.bib5),[7](https://arxiv.org/html/2610.12757#bib.bib8),[8](https://arxiv.org/html/2610.12757#bib.bib10),[9](https://arxiv.org/html/2610.12757#bib.bib20)\]do not model failure\-prone conversation trajectories and evaluate tool\-use across normal, ambiguous, and agentic settings\. The lack of robust training data, which comprises real\-world scenarios such as user goal changes, policy violations, or infeasible requests, in multi\-turn dialogue trajectories\. This leaves a resource gap for training and evaluating agents robustness under realistic, non\-cooperative conversational conditions\.

To address this gap, we proposeMADS \(Multi\-Agent Dialog Simulatation\)a synthetic data generation pipeline based on a multi\-agent LLM simulation framework\. Specifically, we target the absence of training resources that systematically capture non\-cooperative user behavior, tool failures, and infeasible requests within validated multi\-turn dialogue trajectories\.

The pipeline comprises of three interacting agents: a user agent that simulates realistic user behavior, a system agent that performs dialog reasoning and action selection, and a tool agent that emulates external APIs and environmental feedback\. By explicitly modeling these roles, the pipeline produces synthetic multi\-turn dialog traces with structured reasoning and action supervision, including both successful and failure\-prone scenarios categorized into unhappy\-path and impossible scenarios\.

Using MADS, we constructMADS\-Benchspanning six domains, comprising 72 tools and 55 personas, with unhappy\-path scenarios forming the majority of tasks\. We then use this benchmark both to fine\-tune open\-source models and to evaluate them using a two\-level evaluation paradigm as proposed in Pre\-Act\[[4](https://arxiv.org/html/2610.12757#bib.bib3)\]\.

The contributions of this work are summarized as follows:

- •We introduce MADS, a multi\-agent dialog simulation that generates structured multi\-turn dialogues with explicit reasoning, planning, and tool invocation\. The pipeline enables systematic injection of conversational and backend adversities, thereby enabling the construction of a high\-quality synthetic function\-calling dataset across varying levels of scenario complexity\.
- •Leveraging MADS, we constructed MADS\-Bench, a synthetic multi\-turn function\-calling conversational benchmark spanning normal, unhappy, and impossible scenarios across multiple domains\. The dataset incorporates Pre\-Act\[[4](https://arxiv.org/html/2610.12757#bib.bib3)\]style reasoning and planning, making it efficient across various post\-training methods in developing advanced conversational agents\.
- •We designed a taxonomy that decomposes task\-oriented dialogue complexity into conversational adversities and system execution adversities\. By explicitly modeling these failure modes, MADS\-Bench evaluates agents across complex scenarios that are not evaluated by existing function\-calling benchmarks\.
- •We conduct controlled empirical evaluations using Qwen\-2\.5 models \(7B and 32B\) and compare training on MADS\-Bench with APIGen\[[5](https://arxiv.org/html/2610.12757#bib.bib4)\]\. Models are optimized with supervised fine\-tuning \(SFT\), followed by reinforcement learning with verifiable rewards \(RLVR\), to evaluate the impact of training data distribution and post\-training stages on agent robustness\.

![Refer to caption](https://arxiv.org/html/2610.12757v1/example_impossible_flow.png)Figure 1:Illustration of an impossible scenario\. Success requires explaining the unsuitability and proposing an alternative date, not scheduling the event\.
## 2Related Work

### 2\.1Benchmarks and Datasets

Several benchmarks evaluate LLMs’ performance in function calling and tool use across varying levels of complexity\. The Berkeley Function\-Calling Leaderboard \(BFCL\)\[[10](https://arxiv.org/html/2610.12757#bib.bib14)\]provides a large\-scale collection of real\-world API schemas linked to natural\-language instructions, emphasizing schema grounding, argument inference, and relevance detection, with later versions extending evaluation to multi\-turn and multi\-step settings requiring API state tracking across turns\. ComplexFuncBench\[[11](https://arxiv.org/html/2610.12757#bib.bib9)\]targets constrained, multi\-step function calling under long\-context conditions, requiring implicit parameter reasoning across sequences of up to 128k tokens\. ToolBench\[[8](https://arxiv.org/html/2610.12757#bib.bib10)\]assesses pipeline\-level tool use across heterogeneous APIs, covering multi\-step planning, cross\-tool coordination, and error recovery\. More recent benchmarks, such asτ\\tau\-bench\[[12](https://arxiv.org/html/2610.12757#bib.bib18)\]andτ2\\tau^\{2\}\-bench\[[13](https://arxiv.org/html/2610.12757#bib.bib19)\], further evaluate the behavior of conversation interaction agents with simulated users and tools\.τ\\tau\-bench simulates dynamic conversations between a user and a tool\-augmented agent across common task domains and introduces evaluation metrics to assess agent reliability across trials\.τ2\\tau^\{2\}\-bench extends previous work to a dual\-control environment in which both the agent and the user use tools within a shared state space, examining user\-agent coordination\. However, these benchmarks evaluate models under cooperative user behavior and do not systematically expose agents to non\-cooperative dynamics such as goal changes, tool failures, or infeasible requests\.

![Refer to caption](https://arxiv.org/html/2610.12757v1/example_unhappy_flow.jpg)Figure 2:Illustration of an unhappy\-path scenario\. The injected behavior is partial information: the user cannot provide a service ID or confirmation code\. The agent must recover by invokingfind\_servicebefore rescheduling\.In contrast to human\-annotated task\-oriented dialogue corpora such as MultiWOZ\[[14](https://arxiv.org/html/2610.12757#bib.bib16)\]and the Schema\-Guided Dialogue dataset\[[15](https://arxiv.org/html/2610.12757#bib.bib17)\], which provide naturalistic multi\-turn conversations, but at a limited scale and without explicit tool\-use supervision\. MADS\-Bench addresses this gap by providing training and evaluation data specifically designed for unhappy\-path conversational scenarios involving agent reasoning and planning\. By providing structured reasoning, the dataset steers models to learn underlying multi\-step planning for navigating a conversational scenario\. The structure of the dataset is beneficial for supervised post\-training and RL post\-training, making the agents robust by enabling reasoning across conversation turns\.

### 2\.2Synthetic Data for Tool\-Augmented Dialogue

Generating synthetic training data for tool\-augmented dialogue has emerged as a practical alternative to costly human annotation\. ToolFlow\[[16](https://arxiv.org/html/2610.12757#bib.bib15)\]proposes a multi\-agent pipeline using graph\-based tool sampling and planned\-generation strategies to synthesize coherent multi\-turn tool\-calling dialogues, demonstrating that SFT on 8,000 synthetic dialogues can match GPT\-4\-level tool\-calling performance\. xLAM\[[17](https://arxiv.org/html/2610.12757#bib.bib11)\]unifies diverse agentic datasets through a scalable synthesis pipeline, yielding strong generalization across function\-calling benchmarks\. While these approaches prioritize dialogue coherence and tool coverage, they generate predominantly cooperative scenarios\. MADS complements this line of work by explicitly modeling non\-cooperative user behavior through controllable behavior injection, producing training data that covers both cooperative and unhappy\-path dynamics within a validated multi\-turn simulation framework\.

### 2\.3Reinforcement Learning for Tool\-integrated Reasoning

Supervised fine\-tuning on curated tool\-use trajectories remains the dominant paradigm for adapting LLMs to function calling\. Reinforcement learning \(RL\) offers a complementary approach for optimizing grounding and correctness beyond what imitation learning achieves\.

ToolRL\[[18](https://arxiv.org/html/2610.12757#bib.bib12)\]demonstrates that RL with verifiable rewards, using tool execution outcomes as training signal, substantially improves tool selection and argument generation\. StepTool\[[19](https://arxiv.org/html/2610.12757#bib.bib13)\]extends this by using step\-grained reward shaping, assigning rewards at each tool interaction based on invocation correctness and task contribution, rather than rewarding only outcomes\. Our training pipeline combines SFT for structural competence with GRPO\-based RLVR for grounding and correctness\. Unlike StepTool, our rewards are computed against gold reference actions rather than live execution outcomes, making the approach applicable to simulated environments without API access\.

## 3MADS\-Bench

Figure 3:Multi\-Agent Dialog Simulation \(MADS\) pipeline for generating validated tool\-augmented dialogue trajectories\. Scenario configurations specifying task objectives, user personas, tool definitions, and injected adversarial behaviors are used to initialize a three\-agent simulation consisting of a user agent, assistant agent, and tool agent\.### 3\.1MADS Pipeline

We propose a Multi\-Agent Dialog Simulation \(MADS\) pipeline that produces synthetic conversations with intermediate reasoning and planning traces\. Each conversation represents a user attempting to complete a task with the help of an LLM agent that uses external tools throughout the interaction to solve task guided tasks\.

The proposed pipeline comprises of three distinct LLM\-based agents that interact to generate realistic dialogues between human and Tool augmented\-AI assistants\.

Assistant AgentThe assistant being trained or evaluated\. At each step, it receives the conversation history and available tool definitions\. Then it produces a structured response that includes reasoning \(<think\>\), an optional plan \(<plan\>\), and an action \(<action\>\)\. Actions are either tool invocations \(type="tool"\) or user\-facing utterances \(type="say"\)\.

User AgentA simulated user agent is instantiated with a person, task objectives, known slot values, and optional injected behaviors\. The user agent acts as the human and generates appropriate responses to the assistant and terminates the conversation by outputting an explicit\[DONE\]token when the task is completed or abandoned\.

Tool AgentThe tool agent receives tool definitions, scenario\-specific seed data, and injected behaviors, allowing it to produce realistic outputs, including edge cases such as time\-out conditions, empty results, and manage termination conditions\.

Each agent maintains its own distinct view of the conversation history\. The assistant agent observes all steps, including tool calls and results; the user agent observes only user\-facing exchanges; and the tool agent observes only tool invocations and their context\. A central runner orchestrates turn\-taking and manages termination conditions; the assistant agents interact with the user agents to execute dialogue traces and can access the tool agent for tool calls and tool responses\.

Multi\-Stage EvaluatorOnce the synthetic conversation traces are generated, they undergo a multi\-stage automatic data validation combining rule\-based checks with LLM\-as\-judge evaluators\. The generated conversation samples go through a checklist of four validation stages; a conversation enters the final dataset only if all the following four evaluation checks succeed:

- •Syntax EvaluatorVerifies structural correctness: each AssistantAgent step must contain a<think\>block; the first step of each turn must include a<plan\>; and each turn must end with exactly one<action type="say"\>\. Tool calls are validated against their schemas for correct names, required parameters, and value types\.
- •Faithfulness EvaluatorDetects hallucination\. For tool calls, the judge verifies that parameter values are grounded in user intent\. Forsayactions, it checks that the assistant’s utterances accurately reflect tool results rather than introducing unsupported information\.
- •Role Confusion EvaluatorIdentifies cases where the UserAgent broke character and began acting like an assistant\. We filter conversations where the user asks questions to help the assistant or offers assistance\.
- •Task Success Evaluator111The prompt for the Judge Model was engineered across various iterations over a small subset of the generated dataset\.Evaluated by an LLM judge that receives the task objective, slot constraints, and preferences, success criteria, and conversation transcript\. The judge determines whether the assistant has achieved the user’s goal, while respecting the distinction between hard constraints and soft preferences\. In cases where a scenario is impossible, success requires appropriate explanations and alternatives\.

### 3\.2Dataset Construction

Using MADS, we generate a synthetic dataset spanning multiple task\-oriented domains, including online shopping, home services, and enterprise support workflows\. Dialogs are generated using diverse user personas and scenario templates to encourage variability in language, goals, and interaction patterns\.

We designed scenarios across six task\-oriented domains: banking assistance, online shopping, calendar assistance, restaurant booking, travel assistance, and home services\. Each scenario is defined by a JSON configuration containing:

- •Task specification: An objective \(e\.g\., “Cancel an order that was placed”\), slot values divided into hard constraints and soft preferences, and success criteria defining task completion\.
- •Tool definitions: Domain\-specific APIs with typed parameters and return schemas\. Across all domains, we defined 72 unique tools\.
- •Seed data: Realistic backend state \(e\.g\., order records, flight options, account information\) that grounds the ToolAgent\.

The distinction between user slot\-value constraints and preferences enables nuanced evaluation: constraints must be satisfied for the task to be judged successful, whereas preferences may be relaxed at the user’s discretion\. To increase dialogue diversity in our dataset, we created a catalog of 55 personas derived from the PersonaHub dataset\[[20](https://arxiv.org/html/2610.12757#bib.bib2)\], spanning different demographics, occupations, and communication styles\. Each persona includes a name, age, hometown, occupation, biographical background, a description of writing style, and sample messages\. For example, one persona is a geology research assistant who communicates in short, action\-oriented fragments, while another is a history professor who uses formal, structured sentences\.

Personas are assigned to scenarios based on domain relevance\. When a simulation is run, the UserAgent’s prompt incorporates the full persona specification to encourage stylistically consistent responses throughout the conversation\.

### 3\.3Taxonomy of Unhappy Paths

Real\-world dialogues frequently involve complications: users provide incomplete information, change their minds, encounter system errors, or request impossible outcomes\. To ensure our dataset covers such phenomena, we implemented an injected\-behavior mechanism\.

Conversational adversities\.These capture dynamic user behaviors that stress the agent’s planning and clarification strategy\.

- •Partial information: The user provides incomplete or underspecified task details\.
- •Incremental constraint revelation: Preferences and constraints are disclosed gradually over several turns\.
- •Goal revision: The user modifies their objective or constraints during the conversation\.
- •Adversarial tone: The user expresses frustration, skepticism, or challenges system suggestions\.

Execution\-level adversities\.These model realistic tool or environment failures that require recovery or principled refusal\.

- •Backend failure: Simulated server\-side errors during tool invocation\.
- •Empty results: Tool queries return no valid candidates\.
- •Timeout: Tool calls exceed execution limits\.
- •Policy constraint: Requests are denied due to domain\-specific business rules \(e\.g\., expired refund window\)\.

For scenarios in which the user’s objective is literally unattainable given the scenario configuration, we mark the task with animpossibleflag and specify afallback\_behaviorthat describes how the assistant should gracefully decline while offering alternatives\. Approximately 66% of our final dataset contains unhappy\-path scenarios\.

### 3\.4Dataset Statistics

Table 1:MADS\-Bench dataset summaryMADS\-Bench comprises 189 validated conversations spanning 1,095 turns across 87 unique scenarios in six task domains: Banking, Calendar, Home Services, Online Shopping, Restaurant Booking, and Travel\. A summary of the dataset statistics is provided in Table[1](https://arxiv.org/html/2610.12757#S3.T1)\. The dataset comprises 106 training conversations and 83 test conversations\. As shown in figure[4](https://arxiv.org/html/2610.12757#S3.F4),[5](https://arxiv.org/html/2610.12757#S4.F5), the Home Services domain is held out entirely from training and was used for evaluating the cross\-domain generalization across the different stages of fine\-tuning as described in section[4](https://arxiv.org/html/2610.12757#S4)\. Scenario complexity is categorized into Normal, Unhappy path, and Impossible trajectories\. The distribution of the scenario types across the training and test split is illustrated in Figure[4](https://arxiv.org/html/2610.12757#S3.F4),[5](https://arxiv.org/html/2610.12757#S4.F5)\. The training consists of a higher proportion of unhappy path scenarios, reflecting the model to be trained on unhappy paths and evaluating the model’s robustness\.

### 3\.5Evaluation

Inspired from Pre\-Act\[[4](https://arxiv.org/html/2610.12757#bib.bib3)\], we evaluated the performance of models across two levels of evaluation\. First, we compute the turn level evaluation \(L\-1\), in which each conversation turn is evaluated against the ground\-truth\. Further, we evaluated the agents trajectory level success rate \(L\-2\), assessing multi\-turn reasoning and planning across the entire conversation trajectory\.

#### 3\.5\.1Turn Level Evaluation \(L\-1\)

In turn level evaluation, we compare the model’s performance for each conversation turn against the ground\-truth response\. For inference on the test split, we used the following two methods Pre\-Act\[[4](https://arxiv.org/html/2610.12757#bib.bib3)\]and ReAct\[[2](https://arxiv.org/html/2610.12757#bib.bib7)\]\. For each user query, the model must decide between two actions either to make a tool call and generate response based on the tool’s response, or to have a conversation with the user to clarify the user query\. We compute the following metrics:

Action ClassificationWe measure recall for the binary action decision \(tool call vs\. answer\), which captures the model’s ability to correctly identify the correct action to perform given the user query and conversation context\.

Tool Call AccuracyWhen the ground\-truth action requires a tool call, we evaluate the tool invocation accuracy, whether the right tool was invoked, and whether the tool parameters matched, as well as whether all the required parameters were provided with the correct parameter values retrieved from the conversation context\.

Conversation QualityFor turns where the ground\-truth action is a direct user\-facing response, we compute semantic similarity between the predicted and reference utterances usingsentence\-transformers/all\-mpnet\-base\-v2222[https://huggingface\.co/sentence\-transformers/all\-mpnet\-base\-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2)\. Cosine similarity between embeddings captures semantic alignment beyond surface\-level token overlap\.

![Refer to caption](https://arxiv.org/html/2610.12757v1/download1.png)Figure 4:MADS\-Bench training split distribution by domain \(left\) and scenario complexity \(right\)\. Banking dominates the training set \(62\.3%\), while unhappy\-path scenarios account for 42\.5% of training conversations, reflecting the dataset’s emphasis on non\-cooperative user behaviors\.
#### 3\.5\.2Trajectory Level Evaluation \(L\-2\)

We evaluate the models at the trajectory level by scoring whether a model can sustain correct multi\-turn decision\-making and action sequence over an entire conversation, rather than matching isolated turns\. Concretely, we run the model on the held\-out MADS test set and obtain a sequence of predicted assistant outputs, each containing a structured<action\>block\. We then group predictions into full conversations byscenario\_idand conversation boundaries, and score each conversation in temporal order using a strict “stop\-on\-first\-error” criterion\. A turn is marked correct if the predicted action type matches the reference \(i\.e\.,toolvs\.say\); for tool actions, the predicted tool name must match the reference tool name under case\-insensitive exact matching with a small edit\-distance tolerance\. If a turn is incorrect, scoring terminates immediately for that conversation\. The conversation score is defined as the fraction of turns that are correct up to the first error, i\.e\.,correct/attempted\\texttt\{correct\}/\\texttt\{attempted\}with attempted equal to the index of the first incorrect turn\. The reported L2 success rate is the mean conversation score, which penalizes early failures and rewards agents that maintain consistent grounded behavior over longer turn conversations\.

## 4Experiments

In this section, we demonstrate the effectiveness of MADS\-Bench through a series of evaluations across different training stages\. We frame the task of agentic dialogue output as a sequential decision\-making problem where, at each step, the model receives the conversation history and must produce a single structured output containing reasoning \(<think\>\), a plan \(<plan\>\), and an action \(<action\>\)\. Actions are either tool invocations or user\-facing utterances\. We adopted a two\-stage training pipeline consisting of supervised fine\-tuning \(SFT\) followed by reinforcement learning with verifiable rewards \(RLVR\) inspired by\[[18](https://arxiv.org/html/2610.12757#bib.bib12),[21](https://arxiv.org/html/2610.12757#bib.bib21)\]\.

### 4\.1Training Setup

Supervised Fine\-Tuning \(SFT\)During the SFT stage, models are trained on the full MADS\-Bench training split, consisting of 106 conversation trajectories, using a learning rate of2×10−42\\times 10^\{\-4\}, batch size 4, with gradient accumulation over 4 steps\. The objective is next\-token prediction over structured outputs containing reasoning \(<think\>\), planning \(<plan\>\), and action tokens \(<action\>\)\.

Reinforcement Learning with Verifiable Rewards \(RLVR\)RLVR training is performed on a subset of 50 sampled conversations from the training set\. The RL stage uses a learning rate of1×10−51\\times 10^\{\-5\}, batch size 1 with gradient accumulation of 4 steps, and is trained for 5 epoch\. During training, the model generates multiple candidate responses \(K=8K=8generations\) for each rollout\. We adopt the Group Relative Policy Optimization \(GRPO\) objective, following prior work\[[18](https://arxiv.org/html/2610.12757#bib.bib12)\], which reports stronger performance compared to PPO in ablation studies\.

### 4\.2Supervised Fine\-Tuning

While supervised fine\-tuning \(SFT\) substantially improved the model’s ability to follow the required format structure and increased tool\-use frequency, it did not eliminate several critical failure modes that emerged repeatedly during trajectory\-level evaluation\. Most notably, the assistant often:

- •Hallucinated task completion without invoking any tools\.
- •Generated ungrounded or fabricated tool arguments, including nonexistent IDs and incorrect temporal values\.
- •Ignored or contradicted tool outputs, e\.g\., promising a refund when the tool explicitly returned a zero\-refund result\.

These errors are not rare formatting mistakes but fundamental reasoning failures that compromise safety, robustness, and correctness in agentic systems\. Crucially, they arise precisely in situations where the environment provides verifiable signals, tool schemas, argument types, tool outputs, and scenario constraints that a correct agent should incorporate\. Therefore, we perform reinforcement learning post\-training within our training pipeline to optimize the model for action correctness, accurate tool invoking and the semantic similarity of user\-facing responses using signals that the environment can objectively verify\.

![Refer to caption](https://arxiv.org/html/2610.12757v1/download2.png)Figure 5:MADS\-Bench test split distribution by domain \(left\) and scenario complexity \(right\)\. Home Services appears exclusively in the test set, serving as a held\-out domain for cross\-domain generalization evaluation\. The test set is predominantly composed of normal scenarios \(71\.1%\), with unhappy\-path and impossible scenarios comprising the remaining 28\.9%\.
### 4\.3Reinforcement Learning with Verifiable Rewards \(RLVR\)

RLVR shifts the training objective from imitating demonstrations to behaving correctly according to tool outputs, argument validity, and action criteria\.

To incorporate RL into our agent training pipeline, we adopt a GRPO policy gradient, but adapted to the design constraints of our Pre\-Act simulation conversation\. RLVR is applied on top of the SFT\-trained model and operates by generating full rollouts, rewarding each assistant action against verifiable conversation state, and using the reward signals to update the policy toward action grounded and tool\-dependent behavior\.

Our RLVR setup uses three verifiable reward components that directly reflect the reward functions implemented in our code: \(1\) an action\-type reward, \(2\) a tool\-match reward, and \(3\) a final\-answer F1 reward\. Each component is computed deterministically from the model’s generated<action\>block and a gold reference action extracted using the same parsing logic\. No heuristic scoring or human annotation is used\.

- •Action RewardThis reward measures whether the assistant predicted the correct high\-level action type \(toolvs\.say\)\. The implementation compares the predicted action type with the gold action type extracted from the reference output\.
- •Tool Accuracy RewardThis reward is computed only when both the gold and predicted actions are tool invocations\. It checks two conditions: \(a\) whether the predicted tool name matches the gold tool name, and \(b\) whether the predicted parameter dictionary exactly matches the gold parameters\.
- •Semantic RewardThis reward is evaluated only when both the predicted and gold actions are of typesay\. Instead of relying on token\-level overlap, we compute the reward using a sentence\-transformer model that outputs a semantic similarity score between the predicted natural language response and the gold reference response\. The predicted and gold utterances are embedded into a shared vector space, and their cosine similarity is used as the reward value\. Formally: Rsem=cos⁡\(embed⁡\(predicted\),embed⁡\(gold\)\)\.R\_\{\\text\{sem\}\}=\\cos\\bigl\(\\mathrm\{embed\}\(\\text\{predicted\}\),\\ \\mathrm\{embed\}\(\\text\{gold\}\)\\bigr\)\.If either action is not asayaction, the reward defaults to0\.0\. Since cosine similarity is bounded, we haveRsem∈\[0,1\]R\_\{\\text\{sem\}\}\\in\[0,1\]\.
- •Total RewardFor each sample, the final reward passed to the RL optimizer is the sum of all three components:Rtotal=Ract\+Rtool\+RsemR\_\{\\text\{total\}\}=R\_\{\\text\{act\}\}\+R\_\{\\text\{tool\}\}\+R\_\{\\text\{sem\}\}\. Given the individual ranges,Rtotal∈\[0,3\]R\_\{\\text\{total\}\}\\in\[0,3\]\.

This formulation ensures that rewards remain bounded, sparse, and fully verifiable, aligning with the design principles of RL with verifiable signals\.

### 4\.4Cross\-Dataset Generalization: APIGen Baseline

To evaluate whether MADS\-Bench provides a training signal beyond what is available in existing function\-calling datasets, we train a baseline model exclusively on data converted from APIGen MT\[[5](https://arxiv.org/html/2610.12757#bib.bib4)\]and evaluate it on our held\-out MADS\-Bench test set\.

Data Conversion\.APIGen\-MT conversations usehuman,gpt,function\_call, andobservationturns without explicit reasoning or planning traces\. We convert 198 APIGen\-MT conversations into our format via an automated pipeline: tool definitions are parsed from the dataset’s JSON schema, turn roles are remapped to our message structure, and<think\>and<plan\>blocks are synthesized for each assistant turn using GPT\-4o\-mini conditioned on the conversation history and the original assistant response\. Converted conversations undergo the same multi\-stage validation pipeline as MADS\-Bench, yielding 750 training samples\.

Training\.We fine\-tune Qwen2\.5\-7B\-Instruct exclusively on the converted APIGen data, using the same QLoRA configuration as described, allowing a direct comparison of the training data source while controlling for model architecture and fine\-tuning procedure\.

Evaluation\.The APIGen\-finetuned model is evaluated on our MADS\-Bench test set at the L1 level, using the same metrics and evaluation infrastructure as all other models\.

## 5Results and Analysis

### 5\.1L\-1 Evaluation

We evaluate turn\-level behavior using structured action metrics, including action recall, tool\-name F1, parameter match accuracy, final\-answer F1, and semantic similarity\. L1 evaluation isolates individual decision points and measures whether the model selects the correct action type, invokes the appropriate tool with correct arguments, and produces grounded user\-facing responses\. We analyze two orthogonal factors: \(i\) Inference time reasoning prompts \(Pre\-Act vs\. ReAct\) Table \([2](https://arxiv.org/html/2610.12757#S5.T2)\) and \(ii\) Impact of RLVR \(Table[3](https://arxiv.org/html/2610.12757#S5.T3)\)\.

Table 2:L1 \(turn\-level\) evaluation on the MADS test set comparingsupervision sources\.Approachdenotes the inference\-time prompting format \(Pre\-Act vs\. ReAct\)\.Supervisionindicates the dataset used for SFT \(APIGen vs\. MADS\)\. All models are trained with a Pre\-Act system prompt; ReAct is applied only at inference time\.Table 3:L1 \(turn\-level\) evaluation on the MADS test set comparingtraining stages on MADS\. Stage denotes the optimization stage \(SFT→\\rightarrowSFT\+RLVR\)\. Approach denotes the inference\-time prompting format \(Pre\-Act vs\. ReAct\)\. All models are trained with a Pre\-Act system prompt; ReAct is applied only at inference time\.Inference\-Time Reasoning Prompting \(Pre\-Act vs\. ReAct\)\.We further analyze the interaction between training supervision and prompting strategy\. Pre\-Act prompting generally yields higher action recall under MADS supervision, particularly for smaller models, suggesting that structured planning during training aligns naturally with Pre\-Act inference\.

However, ReAct prompting exhibits competitive or stronger tool F1 in some settings \(e\.g\., Qwen\-7B SFT\+RLVR\), indicating that step\-wise reasoning may better leverage RL\-refined policies\. These results suggest that prompting strategy interacts with supervision type and model capacity, and that RLVR improves robustness under both prompting paradigms\.

Effect of RLVRWe next examine the impact of RLVR refinement on models that have already been fine\-tuned with MADS\-bench supervision referred in table[3](https://arxiv.org/html/2610.12757#S5.T3)\. RLVR consistently improves structured grounding metrics across both model sizes, with particularly strong gains in tool\-level correctness\. RLVR improves verifiable components of decision\-making, particularly tool\-name selection and parameter matching, more reliably than semantic answer quality\. Larger models benefit more consistently from reward signals, suggesting the emergent capacity to internalize verifiable reward structures\. Taken together, the L1 results indicate that while supervised learning establishes basic formatting and tool\-use capability, RLVR is necessary to reduce hallucinated actions and fabricated arguments\.

### 5\.2L\-2 Evaluation

We evaluate models at the trajectory level, measuring whether they can sustain correct multi\-turn decision\-making throughout the conversation\. Unlike L1, which isolates individual turns, L2 captures the compounding effects of early errors, tool misuse, and recovery failures\. We report the mean conversation success rate on the held\-out MADS test set\.

Overall Task Success Rate across Training stages

Table[4](https://arxiv.org/html/2610.12757#S5.T4)summarizes the overall trajectory\-level task success rate across the model size and training stage\. Across models, the RLVR stage consistently improves performance over the SFT stage, indicating improved prediction of the correct set of actions and invocation of the appropriate tool throughout the entire conversation trajectory\. The results suggest that while SFT is important for guiding the model toward a structured action pattern, RLVR plays a key role in improving overall correctness across the entire conversation trajectory\.

For Qwen2\.5\-32B, SFT already achieves a stronger baseline success rate of 44\.08%, reflecting the larger model’s improved capacity for structured reasoning and action sequence\. RLVR further increases performance to 53\.96%, yielding a consistent improvement\. The smaller relative gain suggests that larger models internalize structured supervision more effectively during SFT, leaving less headroom for reinforcement refinement\. Nonetheless, RLVR still improves long\-horizon action sequence and reduces failure cascades\.

Table 4:L2 \(trajectory\-level\) task success on the MADS\-bench test set\. Training Stage denotes optimization on MADS \(SFT vs\. RLVR\)\.Cross\-Domain Generalization

To understand the effect of domain supervision, Table[5](https://arxiv.org/html/2610.12757#S5.T5)compares the performance of models trained on APIGen and MADS\-Bench across domains\. For the APIGen models trained on the retail and airline domains, and the MADS\-Bench models trained across the five domains in the training set\. We kept Home Services as the held\-out domain to compare the performance of different training stages in improving the success rate across held\-out domains\. Interestingly, we observe that APIGen\-trained models occasionally achieve higher trajectory success than MADS\-trained models under supervised fine\-tuning\. This behavior arises because APIGen primarily contains cooperative task trajectories with deterministic tool execution patterns, which encourage stable but shallow decision policies\. In contrast, MADS\-Bench intentionally introduces adversarial conversational dynamics\. While these scenarios improve model robustness during the RLVR stage of post\-training, they don’t significantly improve performance alone in the SFT stage\. The RLVR stage consistently improves the performance across all the domains, importantly the same trend is seen across both the model sizes in the held out home services domain indicating RLVR stage improves the cross\-domain generalization when applied as an additional step after supervised post training\.

Table 5:L2 task success rate \(%\) by domain\. Models are evaluated on the MADS\-Bench test set\. APIGen models are trained with SFT only\. MADS\-Bench models are trained with SFT and further refined with RLVR\. The Home Services domain was excluded from training to evaluate cross\-domain generalization\. Travel contains a single test instance\.Scenario Complexity Analysis

Table[6](https://arxiv.org/html/2610.12757#S5.T6)shows task success rates broken down by task complexity, i\.e\., whether it can be completed “normally” \(without problems\), via an “unhappy path”, or encounters an “impossible” scenario\. In cooperative normal scenarios, models trained by RLVR achieve the strongest performance, suggesting that RLVR helps follow the correct sequence of steps and reduces failure cases\. Using SFT only, APIGen training is actually better than MADS\-Bench training, given that the dataset is focused on “normal” conversations\. However, RLVR improves both the “normal” and “unhappy path” cases dramatically, and outperforms the APIgen baseline\. The RLVR improvement is largest for the small model, but still substantial for the large \(32B\) model\. We also find that RLVR degrades the performance for “impossible” conversations\. This could simply be due to a lack of training data, since our limited RL fine\-tuning set has only five instances of impossible cases\.

In terms of model size, we note that the 32B model performs overall better than the 7B version, but that the advantage is especially large for the harder conversations \(unhappy and impossible\)\. It makes sense that larger models have better reasoning ability, allowing them to deal better with more complex tasks\.

Table 6:Task success \(%\) by scenario complexity for different model configurations \(training data and objective\)\. Unhappy\-path scenarios involve non\-cooperative user behaviors, such as goal changes and tool failures\. Impossible scenarios require the agent to recognize task infeasibility and respond gracefully\. APIGen models are trained only with SFT\. MADS models are trained with SFT and further refined with RLVR\. All models are evaluated on the MADS\-Bench test set\.

## 6Conclusions

Real\-world task\-oriented conversational systems must operate under conditions that diverge substantially from the cooperative settings typically represented in existing function\-calling benchmarks\. Users frequently provide incomplete information, revise goals during interaction, or issue requests that cannot be satisfied due to system constraints or policy limitations\. Despite the rapid progress of LLM\-based agents, the lack of training resources that systematically capture such failure\-prone conversational dynamics remains a key bottleneck for building robust dialog managers\.

In this work, we introduced MADS, a multi\-agent dialog simulation pipeline designed to generate structured multi\-turn conversations that include reasoning, planning, and tool\-invocation supervision\. By orchestrating interactions between simulated user, assistant, and tool agents, the pipeline enables controllable injection of both conversational adversities \(e\.g\., partial information, goal revisions\) and execution\-level failures \(e\.g\., backend errors, policy constraints\)\. Using this simulator, we constructed MADS\-Bench, a multi\-domain benchmark spanning six task\-oriented domains, 72 tools, and diverse user personas, with a large proportion of scenarios representing unhappy\-path or impossible interactions\.

Empirical results demonstrate that training on MADS\-Bench improves the robustness of tool\-augmented agents beyond that achievable with cooperative datasets alone\. While supervised fine\-tuning establishes structural competence for reasoning and tool usage, reinforcement learning with verifiable rewards \(RLVR\) significantly improves action correctness and trajectory\-level task success across both small and large models\. These findings suggest that verifiable reward signals derived from tool\-grounded environments are critical for stabilizing long\-horizon decision making in conversational agents\.

Future work may extend this pipeline along several directions\. First, scaling the simulator to larger tool ecosystems and more diverse adversarial behaviors could further improve the coverage of realistic dialog conditions\. Second, integrating real tool execution environments or sandboxed APIs may enable richer reward signals and more faithful agent evaluation\. Finally, extending the simulator to multimodal or speech\-to\-speech systems would enable the study of robustness in next\-generation conversational assistants operating in real world human–AI interaction scenarios\.

Overall, MADS provides a scalable approach to generating structured training data that exposes conversational agents to realistic failure modes, thereby enabling the development of more robust and reliable task oriented conversational agents\.

## 7Acknowledgments

We appreciate the efforts of the developers of the Qwen 2\.5 model family and the maintainers of the open‑source libraries that supported our pipeline\.

We are also appreciative to Interspeech for providing a platform to share our work with the speech and language processing community\.

## References

- \[1\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. Le, and D\. Zhou\(2023\)Chain\-of\-thought prompting elicits reasoning in large language models\.External Links:2201\.11903,[Link](https://arxiv.org/abs/2201.11903)Cited by:[§1](https://arxiv.org/html/2610.12757#S1.p3.1)\.
- \[2\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao\(2022\)ReAct: synergizing reasoning and acting in language models\.arXiv preprint arXiv:2210\.03629\.External Links:[Link](https://arxiv.org/abs/2210.03629)Cited by:[§1](https://arxiv.org/html/2610.12757#S1.p3.1),[§3\.5\.1](https://arxiv.org/html/2610.12757#S3.SS5.SSS1.p1.1)\.
- \[3\]V\. Dongre, X\. Yang, E\. C\. Acikgoz, S\. Dey, G\. Tur, and D\. Hakkani\-Tur\(2025\)ReSpAct: harmonizing reasoning, speaking, and acting towards building large language model\-based conversational ai agents\.Proceedings of the 15th International Workshop on Spoken Dialogue Systems Technology,pp\. 72–102\.External Links:[Link](https://aclanthology.org/2025.iwsds-1.7/)Cited by:[§1](https://arxiv.org/html/2610.12757#S1.p3.1)\.
- \[4\]M\. Rawat, A\. Gupta, R\. Goomer, A\. Di Bari, N\. Gupta, and R\. Pieraccini\(2024\)Pre\-Act: multi\-step planning and reasoning improves acting in llm agents\.arXiv preprint arXiv:2505\.09970\.External Links:[Link](https://arxiv.org/abs/2505.09970)Cited by:[2nd item](https://arxiv.org/html/2610.12757#S1.I1.i2.p1.1),[§1](https://arxiv.org/html/2610.12757#S1.p3.1),[§1](https://arxiv.org/html/2610.12757#S1.p7.1),[§3\.5\.1](https://arxiv.org/html/2610.12757#S3.SS5.SSS1.p1.1),[§3\.5](https://arxiv.org/html/2610.12757#S3.SS5.p1.1)\.
- \[5\]Z\. Liu, T\. Hoang, J\. Zhang, M\. Zhu, T\. Lan, S\. Kokane, J\. Tan, W\. Yao, Z\. Liu, Y\. Feng,et al\.\(2024\)APIGen: automated pipeline for generating verifiable and diverse function\-calling datasets\.arXiv preprint arXiv:2406\.18518\.External Links:[Link](https://arxiv.org/abs/2406.18518)Cited by:[4th item](https://arxiv.org/html/2610.12757#S1.I1.i4.p1.1),[§1](https://arxiv.org/html/2610.12757#S1.p4.1),[§4\.4](https://arxiv.org/html/2610.12757#S4.SS4.p1.1)\.
- \[6\]C\. Chen, X\. Hao, W\. Liu, X\. Huang, X\. Zeng, S\. Yu, D\. Li, Y\. Huang, X\. Liu, W\. Xinzhi, and W\. Liu\(2025\)ACEBench: a comprehensive evaluation of llm tool usage\.Findings of the Association for Computational Linguistics: EMNLP 2025,pp\. 12970–12998\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.697/)Cited by:[§1](https://arxiv.org/html/2610.12757#S1.p4.1)\.
- \[7\]S\. G\. Patil, H\. Mao, C\. C\. Ji, F\. Yan, V\. Suresh, I\. Stoica, and J\. E\. Gonzalez\(2025\)The Berkeley function calling leaderboard \(BFCL\): from tool use to agentic evaluation of large language models\.InForty\-second International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2610.12757#S1.p4.1)\.
- \[8\]Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian,et al\.\(2024\)ToolLLM: facilitating large language models to master 16000\+ real\-world APIs\.InProceedings of the International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2610.12757#S1.p4.1),[§2\.1](https://arxiv.org/html/2610.12757#S2.SS1.p1.1)\.
- \[9\]M\. Li, Y\. Zhao, B\. Yu, F\. Song, H\. Li, H\. Yu, Z\. Li, F\. Huang, and Y\. Li\(2023\)API\-bank: a comprehensive benchmark for tool\-augmented llms\.External Links:2304\.08244,[Link](https://arxiv.org/abs/2304.08244)Cited by:[§1](https://arxiv.org/html/2610.12757#S1.p4.1)\.
- \[10\]F\. Yan, H\. Mao, C\. C\. Ji, and S\. G\. Patil\(2024\)BFCL V3: multi\-turn and multi\-step function calling\.External Links:[Link](https://gorilla.cs.berkeley.edu/blogs/13_bfcl_v3_multi_turn.html)Cited by:[§2\.1](https://arxiv.org/html/2610.12757#S2.SS1.p1.1)\.
- \[11\]L\. Zhong, Z\. Du, X\. Zhang, H\. Hu, and J\. Tang\(2025\)ComplexFuncBench: exploring multi\-step and constrained function calling under long\-context scenario\.arXiv preprint arXiv:2501\.10132\.External Links:[Link](https://arxiv.org/abs/2501.10132)Cited by:[§2\.1](https://arxiv.org/html/2610.12757#S2.SS1.p1.1)\.
- \[12\]S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhan\(2024\)τ\\tau\-Bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.External Links:2406\.12045,[Link](https://arxiv.org/abs/2406.12045)Cited by:[§2\.1](https://arxiv.org/html/2610.12757#S2.SS1.p1.1)\.
- \[13\]V\. Barres, H\. Dong, S\. Ray, X\. Si, and K\. Narasimhan\(2025\)τ2\\tau^\{2\}\-Bench: evaluating conversational agents in a dual\-control environment\.External Links:2506\.07982,[Link](https://arxiv.org/abs/2506.07982)Cited by:[§2\.1](https://arxiv.org/html/2610.12757#S2.SS1.p1.1)\.
- \[14\]P\. Budzianowski, T\. Wen, B\. Tseng, I\. Casanueva, S\. Ultes, O\. Ramadan, and M\. Gašić\(2018\)MultiWOZ: a large\-scale multi\-domain Wizard\-of\-Oz dataset for task\-oriented dialogue modelling\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,Brussels, Belgium,pp\. 5016–5026\.External Links:[Link](https://aclanthology.org/D18-1547/)Cited by:[§2\.1](https://arxiv.org/html/2610.12757#S2.SS1.p2.1)\.
- \[15\]A\. Rastogi, X\. Zang, S\. Sunkara, R\. Gupta, and P\. Khaitan\(2020\)Towards scalable multi\-domain conversational agents: the schema\-guided dialogue dataset\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 8689–8696\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/6394)Cited by:[§2\.1](https://arxiv.org/html/2610.12757#S2.SS1.p2.1)\.
- \[16\]Z\. Wang, X\. Zeng, W\. Liu, L\. Li, Y\. Wang, L\. Shang, X\. Jiang, Q\. Liu, and K\. Wong\(2025\)ToolFlow: boosting LLM tool\-calling through natural and coherent dialogue synthesis\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Albuquerque, New Mexico,pp\. 4246–4263\.External Links:[Link](https://aclanthology.org/2025.naacl-long.214/)Cited by:[§2\.2](https://arxiv.org/html/2610.12757#S2.SS2.p1.1)\.
- \[17\]J\. Zhang, T\. Lan, M\. Zhu, Z\. Liu, T\. Hoang, S\. Kokane, W\. Yao, J\. Tan, A\. Prabhakar, H\. Chen,et al\.\(2024\)xLAM: a family of large action models to empower AI agent systems\.arXiv preprint arXiv:2409\.03215\.External Links:[Link](https://arxiv.org/abs/2409.03215)Cited by:[§2\.2](https://arxiv.org/html/2610.12757#S2.SS2.p1.1)\.
- \[18\]C\. Qian, E\. C\. Acikgoz, Q\. He, H\. Wang, X\. Chen, D\. Hakkani\-Tur, G\. Tur, and H\. Ji\(2025\)ToolRL: reward is all tool learning needs\.arXiv preprint arXiv:2504\.13958\.External Links:[Link](https://arxiv.org/abs/2504.13958)Cited by:[§2\.3](https://arxiv.org/html/2610.12757#S2.SS3.p2.1),[§4\.1](https://arxiv.org/html/2610.12757#S4.SS1.p2.1),[§4](https://arxiv.org/html/2610.12757#S4.p1.1)\.
- \[19\]Y\. Yu, Z\. Wang, W\. Ma, Z\. Guo, J\. Zhan, S\. Wang, C\. Wu, Z\. Guo, and M\. Zhang\(2024\)StepTool: enhancing multi\-step tool usage in LLMs via step\-grained reinforcement learning\.arXiv preprint arXiv:2410\.07745\.External Links:[Link](https://arxiv.org/abs/2410.07745)Cited by:[§2\.3](https://arxiv.org/html/2610.12757#S2.SS3.p2.1)\.
- \[20\]T\. Ge, X\. Chan, X\. Wang, D\. Yu, H\. Mi, and D\. Yu\(2024\)Scaling synthetic data creation with 1,000,000,000 personas\.arXiv preprint arXiv:2406\.20094\.External Links:[Link](https://arxiv.org/abs/2406.20094)Cited by:[§3\.2](https://arxiv.org/html/2610.12757#S3.SS2.p4.1)\.
- \[21\]M\. Rawat, A\. Chakraborty, N\. Gupta, and R\. Pieraccini\(2025\)When actions teach you to think: reasoning\-action synergy via reinforcement learning in conversational agents\.External Links:2512\.11277,[Link](https://arxiv.org/abs/2512.11277)Cited by:[§4](https://arxiv.org/html/2610.12757#S4.p1.1)\.