Agent Seer: Synthesizing Scenarios from Specification Understanding

# Agent Seer: Synthesizing Scenarios from Specification Understanding
Source: [https://arxiv.org/html/2608.26133](https://arxiv.org/html/2608.26133)
###### Abstract

Evaluating AI agents that use external tools requires realistic test scenarios that capture how practitioners compose tools and iterate across conversation turns\. Constructing such scenarios by hand demands deep domain expertise, does not scale across tool ecosystems, and produces static benchmarks that cannot track evolving APIs\. We observe that tool specifications—function names, natural\-language descriptions, and typed parameter schemas—already encode sufficient semantic information to synthesize realistic evaluation scenarios without manual curation or live tool execution\.Agent Seerbuilds off this latent information: from a single Model Context Protocol \(MCP\) specification, with no examples, no live tool access, and no domain\-specific tuning\. This pipeline enriches raw schemas, generates graded scenarios with synthetic tool outputs, and expands them into mock\-data\-grounded multi\-turn dialogues that exhibit strong tool\-calling correctness and conversational coherence\. Evaluation quality is measured by applying this pipeline on seven MCP specifications spanning diverse domains and tool\-suite sizes and measuring the tool\-calling correctness and conversational coherence\. The pipeline achieves strong quality across all domains, with complete tool coverage on small and medium specifications\. Two findings emerge within this analysis: parameter schema complexity is the strongest correlate of quality variation—tool\-suite size plays a smaller, orthogonal role—and argument value accuracy is the dominant failure mode among imperfect scenarios, a sub\-dimension invisible to coarse\-grained name\-match metrics\.

Agent Seer: Synthesizing Scenarios from Specification Understanding

Harish KarumuriAppleMahesh VemulaAppleDavid Lopes PegnaApple

## 1Introduction

The deployment of large language model \(LLM\)\-based agents capable of autonomously calling external tools is increasingly common in enterprise software environments\. Agents backed by calendar APIs, project management systems, communications platforms, and internal databases are being used to automate workflows that previously required human intervention\. Despite rapid progress in agent capabilities, the evaluation of such systems remains laborious and fragile\.

Three problems limit the current state of agent evaluation:

#### The curation bottleneck\.

Realistic evaluation scenarios must connect user intent to specific tool calls, fill parameters with plausible values, and capture how tools chain across turns\. Hand\-authored benchmarks\(Froger et al\.,[2026](https://arxiv.org/html/2608.26133#bib.bib8); Liu et al\.,[2023a](https://arxiv.org/html/2608.26133#bib.bib16); Yao et al\.,[2025](https://arxiv.org/html/2608.26133#bib.bib34)\)produce high\-fidelity scenarios but their coverage is bounded by curation effort; scaling such efforts across the combinatorial space of tool pathways is impractical by hand\.

#### The static benchmark problem\.

A fixed benchmark ceases to reflect reality as tool APIs evolve\. An agent that scores highly on a snapshot benchmark may be evaluated against tool descriptions that no longer match the production environment\.

#### The multi\-turn evaluation gap\.

Conversational agents require evaluation across interaction sequences, not just single\-turn prompts\. Generating multi\-turn scenarios where follow\-up turns react to specific tool outputs—rather than repeating generic instructions—is particularly difficult without access to real tool responses\.

Together these define thecold\-start evaluation problem: producing realistic evaluation data for a tool suite that has none\. The shortage is sharpest for new, private, or rapidly evolving APIs, and persists across the long tail of enterprise tool suites that wrap or extend publicly described systems\. We address it by observing that tool specifications—function names, natural\-language descriptions, and typed parameter schemas—already encode much of the semantic information needed to synthesize evaluation data: enough for an LLM to infer plausible workflows, fill parameters with realistic values, synthesize what tool responses would look like, and construct multi\-turn dialogues grounded in that synthetic data\. The bottleneck shifts fromhuman curationtostructured extraction\.

Agent Seer realizes this as a four\-stage pipeline—semantic interpretation, scenario generation, mock output synthesis, and multi\-turn expansion—that converts raw MCP tool specifications into self\-contained evaluation harnesses\. Each stage consumes validated structured outputs from the previous, so schema violations are caught at boundaries rather than propagating\. The resulting harnesses are decoupled from any execution backend: any MCP\-compatible framework can consume them to run and grade agents\.

The primary contributions are:

1. 1\.Aspec\-to\-harness generation pipelinethat produces complete evaluation scenarios—expected tool sequences, calibrated mock tool outputs, and data\-grounded multi\-turn dialogues—from MCP tool specifications alone, without live tool execution or manual annotation\.
2. 2\.Astructured harness artifact format\(Section[3\.5](https://arxiv.org/html/2608.26133#S3.SS5)\) with held\-out oracles and mock outputs, enabling any MCP\-compatible evaluation framework to run agents against the generated scenarios as self\-contained artifacts\.
3. 3\.Empirical characterizationof generation quality and tool coverage across seven MCP specifications spanning diverse enterprise domains, showing that quality variation is more strongly correlated with parameter schema complexity than with tool\-suite size, and that complete coverage is achievable on small and medium specifications\.
4. 4\.Afailure\-mode characterizationidentifying argument sub\-dimension cascading as the dominant generation failure mechanism—a failure class invisible to coarse\-grained name\-match metrics—with domain\-specific signatures characterizing where and why the pipeline falls short\.

## 2Background and Related Work

Evaluating tool\-calling agents requires evaluation data—scenarios, expected tool sequences, and representative tool outputs\. For established public APIs this can be hand\-curated or mined from usage logs, but for new, private, or rapidly evolving tool suites no such data exists, and what does may not cover niche or challenging use cases\. Even when curation is feasible, identifying the user goals that matter and connecting them to tool sequences that reflect how practitioners compose tools, handle partial results, and iterate across turns requires deep domain expertise\. Agent Seer addresses this cold\-start gap throughexecution\-free harness synthesis from a structured tool specification\.

### 2\.1Agent Benchmarks

Agent evaluation has progressed from single\-function prediction\(Qin et al\.,[2024](https://arxiv.org/html/2608.26133#bib.bib27); Patil et al\.,[2024](https://arxiv.org/html/2608.26133#bib.bib25)\)through multi\-step tool use over curated API corpora\(Qin et al\.,[2024](https://arxiv.org/html/2608.26133#bib.bib27); Lu et al\.,[2025](https://arxiv.org/html/2608.26133#bib.bib19); Chen et al\.,[2024](https://arxiv.org/html/2608.26133#bib.bib4)\)to multi\-turn, policy\-grounded evaluation with simulated users\(Yao et al\.,[2025](https://arxiv.org/html/2608.26133#bib.bib34); Barres et al\.,[2025](https://arxiv.org/html/2608.26133#bib.bib2); Wang et al\.,[2024](https://arxiv.org/html/2608.26133#bib.bib30)\)\. General benchmarks such as GAIA\(Froger et al\.,[2026](https://arxiv.org/html/2608.26133#bib.bib8)\), AgentBench\(Liu et al\.,[2023a](https://arxiv.org/html/2608.26133#bib.bib16)\), and WorkArena\(Drouin et al\.,[2024](https://arxiv.org/html/2608.26133#bib.bib6)\)provide rich evaluation environments, while recent MCP\-centric suites—MCPVerse\(Lei et al\.,[2025](https://arxiv.org/html/2608.26133#bib.bib13)\), MCP\-AgentBench\(Guo et al\.,[2026](https://arxiv.org/html/2608.26133#bib.bib9)\), Toolathlon\(Li et al\.,[2026](https://arxiv.org/html/2608.26133#bib.bib14)\)—reflect the growing adoption of MCP\(Anthropic,[2024](https://arxiv.org/html/2608.26133#bib.bib1)\)as a standard tool integration layer\. All of these benchmarks are constructed through manual curation or require live tool access, and remain static once released, leaving the cold\-start problem unaddressed\.

### 2\.2Synthetic Data Generation

Prior work on generating synthetic data falls into three clusters, distinguished by their reliance on live tool execution\.

#### Training trajectories from live execution\.

The largest cluster generates fine\-tuning trajectories via real tool execution: APIGen\(Liu et al\.,[2024](https://arxiv.org/html/2608.26133#bib.bib18); Prabhakar et al\.,[2025](https://arxiv.org/html/2608.26133#bib.bib26)\)uses execution\-based verification, TOUCAN\(Xu et al\.,[2025](https://arxiv.org/html/2608.26133#bib.bib32)\)scales to 1\.5M trajectories from live MCP servers, GEM\(Xu et al\.,[2026](https://arxiv.org/html/2608.26133#bib.bib33)\)mines trajectories from text corpora, and Agent World Model\(Wang et al\.,[2026](https://arxiv.org/html/2608.26133#bib.bib31)\)synthesizes RL environments\. Most require live tool invocation\.

#### Simulated tool environments\.

A second cluster replaces live tools with simulated environments—for training: Simia\(Li et al\.,[2025](https://arxiv.org/html/2608.26133#bib.bib15)\), Gecko\(Zhang et al\.,[2026](https://arxiv.org/html/2608.26133#bib.bib36)\), LOGIGEN\(Zeng et al\.,[2026](https://arxiv.org/html/2608.26133#bib.bib35)\); for evaluation:τ\\tau\-bench\(Yao et al\.,[2025](https://arxiv.org/html/2608.26133#bib.bib34)\),τ2\\tau^\{2\}\-bench\(Barres et al\.,[2025](https://arxiv.org/html/2608.26133#bib.bib2)\), and ToolSandbox\(Lu et al\.,[2025](https://arxiv.org/html/2608.26133#bib.bib19)\)\.

#### Spec\-only generation\.

An emerging cluster generates synthetic data purely from specifications\. DiGiT\-TC\(Crouse et al\.,[2026](https://arxiv.org/html/2608.26133#bib.bib5)\)back\-translates tool\-call sequences into user requests for fine\-tuning data; FuncBenchGen\(Maekawa et al\.,[2026](https://arxiv.org/html/2608.26133#bib.bib20)\)defines contamination\-free task benchmarks via DAG\-modelled call dependencies\.

### 2\.3Evaluation Methodology

The LLM\-as\-judge paradigm is well established\(Liu et al\.,[2023b](https://arxiv.org/html/2608.26133#bib.bib17); Kim et al\.,[2024](https://arxiv.org/html/2608.26133#bib.bib12); Es et al\.,[2024](https://arxiv.org/html/2608.26133#bib.bib7)\)\. For tool\-use evaluation, prior work uses exact match\(Qin et al\.,[2024](https://arxiv.org/html/2608.26133#bib.bib27)\), execution success rate\(Qin et al\.,[2024](https://arxiv.org/html/2608.26133#bib.bib27)\), and single\-call binary AST matching\(Patil et al\.,[2025a](https://arxiv.org/html/2608.26133#bib.bib23)\)\. Agent GPA\(Jia et al\.,[2026](https://arxiv.org/html/2608.26133#bib.bib11)\)decomposes traces into Goal\-Plan\-Action stages but evaluates on live task environments\. FuncBenchGen\(Maekawa et al\.,[2026](https://arxiv.org/html/2608.26133#bib.bib20)\)finds that models systematically propagate stale or incorrect arguments across chained calls despite syntactic validity—a multi\-step failure mode that coarse\-grained name\-match metrics miss entirely, motivating the per\-argument sub\-dimension decomposition we apply here\. We draw on this methodology to assess generation quality along two complementary dimensions \(tool\-calling correctness and conversational coherence\), applying LLM\-as\-judge scoring with structured multi\-dimensional prompts at each turn\.

### 2\.4Positioning

Agent Seer takes only tool specifications as input \(like spec\-only systems\) but uses an LLM to synthesize plausible tool responses rather than requiring a runtime environment \(like simulated environments for evaluation\)\. The contribution is not the prompt\-driven generation mechanism—shared with APIGen, TOUCAN, and others—but thestructure imposed on it: a four\-stage pipeline with validated structured outputs at each boundary, producing harnesses decoupled from any execution backend\. Table[17](https://arxiv.org/html/2608.26133#A3.T17)in Appendix[C](https://arxiv.org/html/2608.26133#A3)situates these contributions against prior literature\.

## 3Pipeline Stages

The pipeline has four stages \(schema flow in Figure[2](https://arxiv.org/html/2608.26133#A6.F2), Appendix[F](https://arxiv.org/html/2608.26133#A6)\)\. Each consumes validated structured outputs from the previous stage; schema constraints ensure malformed artifacts are caught at boundaries rather than propagating downstream\.

### 3\.1Tool Interpretation

The first stage converts raw MCP tool specifications into semantically\-enriched descriptions\. For each tool, the module issues a structured prompt to an LLM requesting four semantic fields: a functional description, required parameters with semantic roles, primary use case, and organizational context \(prompt templates in Appendix[D](https://arxiv.org/html/2608.26133#A4); output schemas in Appendix[F](https://arxiv.org/html/2608.26133#A6)\)\. This connects terse API documentation to richer scenarios\.

### 3\.2Scenario Generation

The scenario generation module produces realistic enterprise workflow scenarios at two complexity tiers\.Simple scenariostarget everyday operational tasks: single\-domain, short tool call chains\.Complex scenariostarget novel, multi\-domain workflows combining tools in sophisticated ways\. Both tiers are implemented through prompt engineering rather than structural constraints\.

Each generated scenario includes a title, user\-facing instruction, ordered list of expected tool calls with parameter values, novelty explanation, and a natural follow\-up question\. The output schema embedsstructured reasoning tracefields: each tool call carries aquick\_explanationjustifying why the call is made, and each scenario includes anovelty\_reasonexplaining its evaluation value\. These fields force the LLM to reason about tool selection and workflow composition during generation\.

### 3\.3Mock Output Generation

For each function call in the sequence, the module produces a synthetic tool output\. The pipeline accepts an optional example outputs field, making it a spectrum: fully unsupervised from specifications alone, but able to incorporate available traces to improve fidelity\. When examples are provided, the generator matches their structure; when absent, generation relies on the tool description alone\. Each mock output carries a grounding tier \(high/medium/low\) recording the availability of reference material\.

### 3\.4Multi\-Turn Scenario Expansion

The multi\-turn expansion stage takes a scenario and its mock outputs and emits a list of conversational turns \(prompt template in Appendix[D](https://arxiv.org/html/2608.26133#A4); output schema in Figure[2](https://arxiv.org/html/2608.26133#A6.F2)\); the output is a list, but the prompt does not prescribe how many turns to produce\.

Splits target natural phase boundaries so the resulting turns exercise two multi\-turn tool\-calling patterns formalized by BFCL v3\(Patil et al\.,[2025b](https://arxiv.org/html/2608.26133#bib.bib24)\):multi\-stepsequences, where each call depends on the output of the previous one, andmulti\-hoppatterns, where independent calls gather information that must be synthesized\. Splitting at phase boundaries \(rather than at arbitrary points\) preserves these dependency structures across turns\. When the expansion yields only a single turn, it is discarded under the heuristic that the scenario lacked enough substance to split\.

When the expansion succeeds, follow\-up prompts reference concrete values from synthetic outputs—entity names, counts, status codes—rather than abstract task descriptions, producingdata\-groundedmulti\-turn dialogues\.

### 3\.5Harness Artifact Format

The four stages produce a self\-contained evaluation harness \(Table[1](https://arxiv.org/html/2608.26133#S3.T1)\)\. A downstream framework presents the prompt, feeds mock outputs as tool responses, and scores the agent’s emitted calls against the scenario workflow as the held\-out oracle—enabling evaluation on a previously unseen tool suite without live access\.

Table 1:Fields of a generated evaluation harness\.

## 4Generation Quality Assessment

We assess generation quality using LLM\-as\-judge scoring\(Liu et al\.,[2023b](https://arxiv.org/html/2608.26133#bib.bib17)\)along two complementary quality dimensions: tool\-calling correctness \(TC\) and conversational coherence \(Coh\)\. The framework also supports oracle\-grounded evaluation when reference data is available; we use the unsupervised path exclusively here, as the generated scenarios serve as both output and oracle\.

### 4\.1Tool\-Calling Scoring

The tool\-calling evaluator independently scores four dimensions, decomposing the coarse\-grained pass/fail signal used by most prior work\(Qin et al\.,[2024](https://arxiv.org/html/2608.26133#bib.bib27)\)into orthogonal axes\. An LLM judge scores each sub\-dimension on 0–10; scores are normalized to 0–1 before aggregation\.

Tool usage correctnesscaptures necessity\(Huang et al\.,[2023](https://arxiv.org/html/2608.26133#bib.bib10)\)—whether a tool was warranted at all—with overuse recorded as diagnostic but excluded from the aggregate, since necessity dominates appropriateness\.

Tool selection correctnessaverages correctness, specificity, and completeness of the chosen tools\.

Tool ordering correctnessaverages sequence logic, dependency handling, and execution efficiency\. It is marked not applicable when only one tool is called, and is then excluded from the turn\-level aggregate\.

Tool argument correctnessaverages six sub\-dimensions \(completeness, name, value, type, format, relevancy\)\. Cascading penalties are enforced through prompt instructions to the judge \(Table[18](https://arxiv.org/html/2608.26133#A5.T18), footnote\): a wrong parameter name or missing required parameter zeros out value, type, and format; a wrong value cascades to type, format, and relevancy\. A single critical error therefore collapses the argument mean\.

The four dimensions combine via arithmetic mean per turn; conversation scores are the arithmetic mean of turn scores\. Per\-MCP rankings and the schema\-complexity correlation are robust to harmonic\-mean and minimum aggregation \(Appendix[B](https://arxiv.org/html/2608.26133#A2)\); the full prompt is in Appendix[E](https://arxiv.org/html/2608.26133#A5)\.

### 4\.2Coherence Scoring

The coherence evaluator assesses five sub\-aspects—logical flow, completeness, conciseness, topic relevance, and context retention—scored on a 1–3 scale, normalized to 0–1, and aggregated via arithmetic mean \(full prompt in Appendix[E](https://arxiv.org/html/2608.26133#A5)\)\.

## 5Experimental Evaluation

We evaluate Agent Seer on seven publicly available MCP specifications, analyzing the quality, coverage, and failure modes of generated evaluation scenarios\.

### 5\.1Experimental Setup

#### MCP specifications\.

Seven open\-source MCP server specifications span diverse domains, tool counts, and schema complexity \(Table[2](https://arxiv.org/html/2608.26133#S5.T2)\)\. Sources: the official MCP reference server repository\(Model Context Protocol,[2024](https://arxiv.org/html/2608.26133#bib.bib21)\)and the MCP server registry\(Model Context Protocol,[2025](https://arxiv.org/html/2608.26133#bib.bib22)\)\.

Table 2:MCP specifications used for evaluation\.p¯\\bar\{p\}= mean parameters per tool\.Schemacharacterizes the dominant parameter structure:Flat= simple key\-value or primitive parameters;Nested obj/DSL= parameters containing nested objects or domain\-specific query languages;Deep opt\.= deeply nested schemas with many optional fields;Flat state= flat parameters but stateful sequential semantics;Mixed= combination of flat and structured parameters across tools\.
#### Generation model\.

All scenarios were generated using Gemini 2\.5 Flash Lite with structured output mode enabled for schema\-constrained generation at each pipeline stage\. Pipeline stages used temperature 0\.7\. Structured\-output validation failures triggered up to three retries before the record was discarded\.

#### Evaluation method\.

Generated scenarios were scored using Gemini 2\.5 Flash at temperature 0 \(for deterministic scoring\) as the LLM judge, following the procedure described in Section[4](https://arxiv.org/html/2608.26133#S4): \(1\)Tool Calling: LLM\-as\-judge scoring across four dimensions combined via arithmetic mean, with cascading penalties on argument sub\-scores enforced through prompt instructions to the judge \(scale 0–1\); \(2\)Coherence: five sub\-dimensions on a 1–3 raw scale, normalized to 0–1 and aggregated via arithmetic mean\. The full corpus was additionally re\-scored with an out\-of\-family judge \(Qwen3\.5\-122B\-A10B\-FP8, Alibaba\) to probe judge robustness; diagnostics appear in Appendix[A\.8](https://arxiv.org/html/2608.26133#A1.SS8)\.

#### Scale\.

The pipeline generated 337 scenarios across the seven MCPs \(Table[6](https://arxiv.org/html/2608.26133#A1.T6)in Appendix[A](https://arxiv.org/html/2608.26133#A1)\), yielding 391 evaluation records\. Multi\-turn expansion succeeded for 54 scenarios \(16\.0% overall\), heavily skewed toward complex scenarios \(30\.8% expansion rate vs\. 2\.8% for simple\), as the expansion stage requires sufficient workflow substance to generate meaningful follow\-up turns\.

### 5\.2Quality Results

#### Overall quality\.

The pipeline achieves a mean unsupervised tool\-calling score of 0\.911 \(95% bootstrap CI\[0\.897,0\.925\]\[0\.897,0\.925\]; median 0\.979\) and mean coherence of 0\.855 \(95% CI\[0\.838,0\.872\]\[0\.838,0\.872\]; median 0\.933\)\. 31\.7% of records achieve a perfect tool\-calling score, while only 2\.3% score below 0\.5\. The distribution is concentrated in the upper range, reflecting consistent generation quality across specifications \(see Figure[3](https://arxiv.org/html/2608.26133#A7.F3)in Appendix[G](https://arxiv.org/html/2608.26133#A7)\)\.

#### Cross\-family co\-evaluation\.

To check that these scores reflect the generated data rather than a single judge family, the corpus was re\-scored with an out\-of\-family judge \(Qwen3\.5\-122B\-A10B\-FP8, Alibaba; the primary judge is Google’s Gemini 2\.5 Flash\)\. Tool\-calling agrees at every grain: no mean shift \(Δ​μTC≈0\\Delta\\mu\_\{\\textrm\{TC\}\}\{\\approx\}0, 95% CI \[−0\.009\-0\.009,\+0\.008\+0\.008\]\), record\-level pairedr=0\.79r\{=\}0\.79overn=384n\{=\}384paired records, per\-MCP CIs overlap for all seven MCPs, and the MCP ranking is preserved \(ρ=0\.86\\rho\{=\}0\.86\)\. The failure\-mode taxonomy replicates bilaterally: argument value\-accuracy is the dominant sub\-failure under both judges by a 4–5×\\timesmargin \(Gemini 240 records, Qwen35 252; Table[14](https://arxiv.org/html/2608.26133#A1.T14)\)\. The judges diverge on the absolute level of coherence \(Δ​μCoh≈−0\.16\\Delta\\mu\_\{\\textrm\{Coh\}\}\{\\approx\}\-0\.16, pairedr=0\.42r\{=\}0\.42\)\. Coherence levels are therefore reported as judge\-dependent; MCP\-level coherence rank\-preservation is partial \(ρ=0\.46\\rho\{=\}0\.46\), and the worst\-coherence MCP differs across judges \(Git under Gemini, Illustrator under Qwen 3\.5\)\. Full diagnostics appear in Appendix[A\.8](https://arxiv.org/html/2608.26133#A1.SS8)\.

#### Quality by MCP specification\.

Table[3](https://arxiv.org/html/2608.26133#S5.T3)reports per\-MCP scores\.

Table 3:Unsupervised scores by MCP\. TC = tool calling, Coh = coherence, Simp\. = simple scenario TC,w¯\\bar\{w\}= mean workflow length \(tools per scenario\)\. 95% intervals are percentile bootstrap \(B=10,000B=10\{,\}000\)\. All seven MCPs exceed 0\.85 overall; all exceed 0\.91 on simple scenarios\.Tool count and parameter schema complexity play distinct, orthogonal roles in quality variation across this sample\.At the per\-MCP grain \(n=7n=7\), tool count correlates positively but modestly with mean unsupervised tool\-calling \(r=\+0\.40r=\+0\.40\), while parameter schema complexity correlates negatively—average parameters per tool \(r=−0\.60r=\-0\.60\) and optional parameter fraction \(r=−0\.66r=\-0\.66\)\. The two effects operate on different axes of an MCP and do not cancel: Selenium \(56 tools\) scores 0\.935 while Filesystem \(14 tools\) scores 0\.876, but Git \(33 tools, 11\.2 average parameters\) is the lowest at 0\.857\.

To strengthen the inferential basis of these per\-MCP observations, we additionally disaggregate to the tool level\. Across the 222 unique in\-spec tools appearing in any generated scenario, parameter count and optional\-fraction both correlate negatively with mean unsupervised TC \(r=−0\.29r=\-0\.29and−0\.30\-0\.30, bothp<0\.001p<0\.001\), confirming the per\-MCP direction at a substantially larger sample\. The same schema features also correlate negatively with mean coherence at the tool level \(r=−0\.41r=\-0\.41and−0\.34\-0\.34, bothp<0\.001p<0\.001\)\. This second finding—not detectable at the per\-MCP grain—suggests complex parameters degrade not only tool\-calling correctness but also the agent’s ability to communicate cleanly around those tools\.

Coherence tells a different story: Slack achieves the highest coherence \(0\.938\) because its scenarios follow structured messaging patterns that produce natural conversational flow, while Git has the lowest \(0\.757\) because version control workflows involve complex multi\-step operations with technical context\. The two dimensions remain weakly correlated at the unit of evaluation: record\-levelr=\+0\.23r=\+0\.23\(n=381n=381\) and tool\-levelr=\+0\.16r=\+0\.16\(n=222n=222,p=0\.016p=0\.016\)\. The cross\-MCP correlation between MCP means is higher \(r=\+0\.57r=\+0\.57,n=7n=7\) but reflects aggregation effects rather than a stronger underlying relationship\. TC and coherence therefore provide largely independent diagnostic signals across grains\.

#### Complexity breakdown\.

Complex scenarios degrade by−\-7\.3pp relative to simple ones in tool calling \(0\.949 \[95% CI 0\.932, 0\.965\]→\\to0\.877 \[0\.855, 0\.897\]\) and−\-5\.3pp in coherence \(0\.883 \[0\.857, 0\.908\]→\\to0\.830 \[0\.807, 0\.853\]\); the simple/complex CIs are non\-overlapping in both dimensions\. The effect varies by domain \(Table[7](https://arxiv.org/html/2608.26133#A1.T7)in Appendix[A](https://arxiv.org/html/2608.26133#A1)\): Elasticsearch shows the largest degradation \(−\-12\.2pp\), followed by Git \(−\-11\.1pp, from 0\.910 to 0\.799\)\. Selenium is stable \(0\.932→\\to0\.935\), suggesting that its long sequential workflows are no harder to compose at higher complexity\.

### 5\.3Failure Mode Analysis

#### Tool\-calling dimension failures\.

Table[4](https://arxiv.org/html/2608.26133#S5.T4)reports per\-dimension failure rates\. Usage is near\-perfect; selection is correct on 77% of records \(4% zeros, reflecting the cost of choosing among semantically similar tools\); ordering fails on 10% of multi\-tool scenarios\.Argument correctness is the dominant challenge: only 42% of records score perfectly and 57% score partial, driven by value\-accuracy errors that degrade scores without collapsing them\.

Table 4:Tool\-calling dimension failure rates\.†\\daggerOrdering is evaluated only when multiple tools are called \(n=181n\\\!=\\\!181\)\.
#### Argument failure patterns\.

Value accuracydominates argument failures \(223 records\), followed byrelevancy\(44\),format\(35\),type\(31\),completeness\(16\), andname accuracy\(11\)\. Counts attribute each failing record to its lowest\-scored sub\-dimension \(single assignment\); the judge\-comparison appendix \(Table[14](https://arxiv.org/html/2608.26133#A1.T14)\) uses the looser criterion of any sub\-dimension scoring below 1\.0, which yields larger per\-sub\-dimension totals\. The pipeline reliably generates correct parameter names and types but struggles with precise values, particularly for optional parameters with ambiguous semantics\.

#### Failure concentration by MCP\.

Failures are sparse and unevenly distributed: Git shows the highest record\-level failure rate, followed by Filesystem and Redis; the other four MCPs are near\-zero\. Per\-domain breakdowns and aggregation\-rule sensitivity appear in Appendices[A\.6](https://arxiv.org/html/2608.26133#A1.SS6)and[B](https://arxiv.org/html/2608.26133#A2)\.

#### Illustrative failure: Redis argument ambiguity\.

A representative pattern occurs in Redis, where several commands carry optional parameters that the pipeline omits when context implies them\. Thesettool, for example, accepts an optionalexpirationfield that is frequently skipped when the scenario implies time\-bounded storage\. The pipeline consistently generates the correct function name, key, and value arguments but omits this expiry parameter, which the argument scoring framework flags as acompletenesssub\-dimension failure; a coarse name\-match metric would score these records as fully correct\. This pattern—correct tool selection with subtly incomplete argument specification—is the primary driver of Redis failures and illustrates why argument correctness must be decomposed below the function\-call level: without the sub\-dimension breakdown, the dominant failure mode in this domain is invisible\.

#### Illustrative failures: Git’s two mechanisms\.

Git failures decompose into two distinct mechanisms operating at different scenario complexities\. The first, observable on simple scenarios, istool\-name hallucination: in three Git scenarios the pipeline emits real Git CLI commands that are not in the MCP specification \(fetch,revert,filter\-repo\), a pretraining\-knowledge leak past the spec\. These three records account for half of Git’s six records with TC<<0\.5 and for the gap between Git’s simple\-scenario TC \(0\.910\) and the other six MCPs \(0\.93–0\.99 simple\)\. Across the full corpus, hallucinated calls are 0\.336% of all tool invocations \(3 of 893\), entirely concentrated in Git\.

The second mechanism, observable on complex scenarios, isparameter overload—localized to specific high\-parameter tools rather than the full MCP\. Git tools average 11\.2 parameters \(3×\\timesthe next highest\), 95% optional, and therefparameter appears in seven tools with different semantics \(“commits starting from” inlog, “compare against” indiff, “show object at” inshow\)\. The pipeline selects the right tool \(selection score 0\.802\) but generates incorrect parameter values, producing a mean argument score of 0\.780—the lowest of any MCP\. The bottom\-fifteen tools by unsupervised TC across the corpus include five Git tools \(blame,add,commit,log,show\), all high in parameter density\. Complex Git scenarios degrade further \(arguments 0\.726\) as multi\-tool workflows compound per\-call parameter errors\.

#### Coherence taxonomy and cross\-dimension pattern\.

The most frequent coherence issues aremissing information or shallow response\(373 records\),off\-topic\(130\),non\-sequitur\(86\), andself\-contradiction or hallucination\(85\); the full taxonomy is in Table[10](https://arxiv.org/html/2608.26133#A1.T10)\. Record\-level TC and Coh remain weakly correlated \(r=0\.23r\{=\}0\.23\), and the “correct tools, poor coherence” quadrant \(23% of records at a 0\.75 threshold\) spans all MCPs rather than concentrating in one domain—coherence shortfalls are a general pipeline property\.

### 5\.4Domain Cluster Analysis

Grouping MCPs by target system type lets structural differences within each cluster act as a natural experiment\.Data Store\(Redis, Elasticsearch\) pairs near\-identical profiles and yields near\-identical quality \(Δ\\Delta0\.036 TC\)\.Developer/File\(Filesystem, Git\) spans a 6×\\timesparameter\-density gap \(1\.8 vs\. 11\.2\) but onlyΔ\\Delta0\.019 in TC, with the cost localized to argument correctness \(Git 0\.780, Filesystem 0\.894\) while strong selection and ordering compensate\.Application/UI\(Illustrator, Selenium\) is the most semantically distant pair: Selenium \(0\.935 TC, 56 tools\) outperforms Illustrator \(0\.898 TC, 64 tools\) because its low parameter density enables reliable argument generation in long sequential chains\.

### 5\.5Coverage and Diversity

#### Tool coverage\.

Table[5](https://arxiv.org/html/2608.26133#S5.T5)reports the fraction of available tools exercised\. Within the small\-to\-medium range \(14–56 tools\), every tool in Redis, Selenium, Git, Elasticsearch, Slack, and Filesystem appears in at least one generated scenario\. The only specification beyond that range, Illustrator \(64 tools\), reaches 56%—the only evidence of a coverage ceiling in this experiment\.

Table 5:Tool coverage and usage uniformity\. Gini coefficient measures inequality \(lower = more uniform\)\.
#### Usage uniformity\.

The Gini coefficient over each MCP’s tool\-usage frequency distribution ranges from 0\.146 \(Redis\) to 0\.340 \(Git\)—near\-uniform sampling\. Selenium is the outlier \(0\.676\), with long sequential workflows \(mean 9\.7 calls\) concentrating usage on core navigation and interaction tools\. Across all MCPs, the pipeline yields 781 unique co\-occurrence pairs from 138 multi\-tool scenarios and 108 unique scenario categories \(Appendix[A\.5](https://arxiv.org/html/2608.26133#A1.SS5)\)\.

### 5\.6Summary of Findings

Across 337 scenarios on seven MCPs, the pipeline achieves mean TC 0\.911 and mean coherence 0\.855 with complete tool coverage on small and medium specifications\. The principal observations:

1. 1\.Parameter schema complexity is the strongest correlate of quality variation in this sample; tool\-suite size plays a smaller, orthogonal role\. Schema correlations are negative at both grains \(per\-MCPr=−0\.60r=\-0\.60/−0\.66\-0\.66on TC; tool\-levelr=−0\.29r=\-0\.29/−0\.30\-0\.30on TC and−0\.41\-0\.41/−0\.34\-0\.34on coherence,p<0\.001p<0\.001throughout\); tool count correlates positively but modestly with TC \(r=\+0\.40r=\+0\.40per\-MCP\)\.
2. 2\.All seven MCPs exceed 0\.91 on simple scenarios—including Git despite 11\.2 avg params per tool—and complex scenarios degrade by 7\.3pp on average\. Domain clusters confirm the pattern: same\-profile pairs yield similar TC; differing\-profile pairs stay close when parameter density dominates\.
3. 3\.TC and coherence provide largely independent diagnostic signals \(record\-levelr=\+0\.23r=\+0\.23, tool\-levelr=\+0\.16r=\+0\.16\), with characteristic inversions \(Slack high coherence, low TC; Selenium best ordering\)\.
4. 4\.Argument correctness is the primary challenge: 57% of records score partial on arguments, value\-accuracy dominant; on Git, failures decompose into pretraining\-leak tool\-name hallucination on simple scenarios and parameter\-overload localized to a handful of high\-parameter tools on complex scenarios\.
5. 5\.Headline tool\-calling and the dominant argument\-value\-accuracy failure mode are robust to an out\-of\-family judge swap \(pairedr≈0\.79r\{\\approx\}0\.79on TC, MCP rankρ=0\.86\\rho\{=\}0\.86\); absolute coherence levels and MCP\-level coherence rank\-preservation are judge\-dependent \(Appendix[A\.8](https://arxiv.org/html/2608.26133#A1.SS8)\)\.

## 6Conclusion

We have presented Agent Seer, a four\-stage pipeline that converts MCP tool specifications into complete evaluation harnesses—graded scenarios, mock tool outputs, and multi\-turn dialogues—without live tool execution or manual annotation\. Across seven structurally diverse specifications, the pipeline reaches full tool coverage on small and medium MCPs and surfaces consistent diagnostic patterns: parameter schema complexity is the strongest correlate of quality variation in this sample, and argument value accuracy is the dominant remaining sub\-failure\. These findings are grounded inn=7n=7specifications and should be read as observations within the experiment rather than universal claims; the durable contribution is the methodology, which produces harnesses as reusable artifacts that feed live tool environments\(Yao et al\.,[2025](https://arxiv.org/html/2608.26133#bib.bib34)\)or simulated agents\(Li et al\.,[2025](https://arxiv.org/html/2608.26133#bib.bib15)\)—closing the cold\-start evaluation gap for MCP\-compatible tool suites\.

## Limitations

Ground truth reliability\.The most significant limitation is reliance on LLM\-generated ground truth, which introduces systematic biases from the generating model\. The framework is best understood as a proxy evaluation tool that identifies broad capability gaps and relative performance differences\.

Cross\-call referential integrity\.Mock outputs for sequential calls are currently generated independently, meaning IDs or values may not align across dependent calls\. A shared state dictionary across a workflow would address this\.

Coverage ceiling\.At 64 tools \(Illustrator\), coverage drops to 56%\. Targeted generation strategies—such as iterative generation with coverage\-aware tool sampling—would be needed for larger specifications\.

Specification scope\.Cross\-domain harness generation and analysis spanning multiple MCP specifications in a single workflow is structurally straightforward within the existing pipeline as well, but is outside the scope of this evaluation and remains a rich direction for future work\. Extension to other spec formats \(OpenAPI, gRPC, function\-calling schemas\) is similarly structurally straightforward—the same name \+ typed\-parameter scaffolding is present—and is a natural next step for evaluating the framework’s generality\.

Complexity stratification\.The simple/complex distinction uses prompt framing rather than structural enforcement\. Post\-generation filters based on structural complexity metrics would improve stratification\.

Experimental scope\.Seven MCP specifications and a single generation model \(Gemini 2\.5 Flash Lite\) cannot establish universal claims\. Multi\-turn evaluation records are limited \(n=54n\\\!=\\\!54\), with expansion heavily skewed toward complex scenarios \(30\.8% expansion rate\) versus simple scenarios \(2\.8%\), reflecting the stage’s dependence on sufficient workflow substance to generate meaningful follow\-up turns\. This constrains statistical power for multi\-turn findings\.

LLM\-as\-judge circularity\.Both generation and quality verification rely on LLMs, raising the concern that scenarios that “look good to an LLM” score well regardless of actual quality\. We address this in two ways\. First, an evaluator–generator capability gap: the judge \(Gemini 2\.5 Flash\) has surplus capacity over the weaker generator \(Gemini 2\.5 Flash Lite\), consistent with teacher–student evaluation paradigms; empirically, Section[5](https://arxiv.org/html/2608.26133#S5)shows the judge discriminates meaningful, domain\-specific failure patterns \(argument cascading, semantic ambiguity in Redis\) rather than producing uniformly high scores\. Second, an out\-of\-family replication: re\-scoring the full 391\-record corpus withQwen3\.5\-122B\-A10B\-FP8\(Alibaba family\) yields paired Pearsonr≈0\.79r\{\\approx\}0\.79on tool\-calling \(n=384n\{=\}384\) with no mean shift, preserves the MCP ranking \(ρ=0\.86\\rho\{=\}0\.86\), and reproduces the dominant argument value\-accuracy failure mode bilaterally \(Gemini 240 records, Qwen 3\.5 252; 4–5×\\timesmargin over the next sub\-dim under both judges\)\. Coherence shows a systematic stricter\-judge shift \(Δ​μ≈−0\.16\\Delta\\mu\{\\approx\}\-0\.16\) with moderate record\-level correlation \(r=0\.42r\{=\}0\.42\); absolute coherence levels and MCP\-level coherence rank\-preservation should therefore be read as judge\-dependent\. Full agreement tables, per\-MCP bootstrap CIs under both judges, and the Bland–Altman analysis appear in Appendix[A\.8](https://arxiv.org/html/2608.26133#A1.SS8)\. A systematic human evaluation study correlating framework scores with human quality judgments remains an important direction for future work\.

## References

- Anthropic \(2024\)Anthropic\. 2024\.Model context protocol\.[https://modelcontextprotocol\.io](https://modelcontextprotocol.io/)\.
- Barres et al\. \(2025\)Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan\. 2025\.[τ2\\tau^\{2\}\-bench: Evaluating conversational agents in a dual\-control environment](https://arxiv.org/abs/2506.07982)\.*Preprint*, arXiv:2506\.07982\.
- Castellani et al\. \(2025\)Tommaso Castellani, Naimeng Ye, Daksh Mittal, Thomson Yen, and Hongseok Namkoong\. 2025\.[SynthTools: A framework for scaling synthetic tools for agent development](https://arxiv.org/abs/2511.09572)\.*Preprint*, arXiv:2511\.09572\.
- Chen et al\. \(2024\)Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, and Feng Zhao\. 2024\.[T\-eval: Evaluating the tool utilization capability of large language models step by step](https://doi.org/10.18653/v1/2024.acl-long.515)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 9510–9529, Bangkok, Thailand\. Association for Computational Linguistics\.
- Crouse et al\. \(2026\)Maxwell Crouse, Ibrahim Abdelaziz, Kshitij Fadnis, Siva Sankalp Patel, Kinjal Basu, Chulaka Gunasekara, Sadhana Kumaravel, Asim Munawar, and Pavan Kapanipathi\. 2026\.[Simulating complex multi\-turn tool calling interactions in stateless execution environments](https://arxiv.org/abs/2601.19914)\.*Preprint*, arXiv:2601\.19914\.
- Drouin et al\. \(2024\)Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H\. Laradji, Manuel Del Verme, Tom Marty, David Vazquez, Nicolas Chapados, and Alexandre Lacoste\. 2024\.[WorkArena: How capable are web agents at solving common knowledge work tasks?](https://proceedings.mlr.press/v235/drouin24a.html)In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pages 11642–11662\. PMLR\.
- Es et al\. \(2024\)Shahul Es, Jithin James, Luis Espinosa\-Anke, and Steven Schockaert\. 2024\.RAGAS: Automated evaluation of retrieval augmented generation\.In*Proceedings of the 18th European Chapter of the Association for Computational Linguistics \(System Demonstrations\)*\.ArXiv:2309\.15217\.
- Froger et al\. \(2026\)Romain Froger, Pierre Andrews, Matteo Bettini, Amar Budhiraja, Ricardo Silveira Cabral, Virginie Do, Emilien Garreau, Jean\-Baptiste Gaya, Hugo Laurençon, Maxime Lecanu, Kunal Malkan, Dheeraj Mekala, Pierre Ménard, Gerard Moreno\-Torres Bertran, Ulyana Piterbarg, Mikhail Plekhanov, Mathieu Rita, Andrey Rusakov, Vladislav Vorotilov, and 5 others\. 2026\.[Gaia2: Benchmarking LLM agents on dynamic and asynchronous environments](https://openreview.net/forum?id=9gw03JpKK4)\.In*Proceedings of the Fourteenth International Conference on Learning Representations*\.
- Guo et al\. \(2026\)Zikang Guo, Benfeng Xu, Chiwei Zhu, Wentao Hong, Xiaorui Wang, and Zhendong Mao\. 2026\.[MCP\-AgentBench: Evaluating real\-world language agent performance with MCP\-mediated tools](https://doi.org/10.1609/aaai.v40i37.40347)\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 40, pages 30888–30896\.
- Huang et al\. \(2023\)Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun\. 2023\.[MetaTool benchmark for large language models: Deciding whether to use tools and which to use](https://arxiv.org/abs/2310.03128)\.*Preprint*, arXiv:2310\.03128\.
- Jia et al\. \(2026\)Allison Sihan Jia, Daniel Huang, Nikhil Vytla, Seung Won Wilson Yoo, Nirvika Choudhury, Shayak Sen, John C\. Mitchell, and Anupam Datta\. 2026\.[What is your agent’s gpa? a framework for evaluating agent goal\-plan\-action alignment](https://arxiv.org/abs/2510.08847)\.*arXiv:2510\.08847*\.
- Kim et al\. \(2024\)Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo\. 2024\.[Prometheus 2: An open source language model specialized in evaluating other language models](https://doi.org/10.18653/v1/2024.emnlp-main.248)\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 4334–4353, Miami, Florida, USA\. Association for Computational Linguistics\.
- Lei et al\. \(2025\)Fei Lei, Yibo Yang, Wenxiu Sun, and Dahua Lin\. 2025\.[MCPVerse: An expansive, real\-world benchmark for agentic tool use](https://arxiv.org/abs/2508.16260)\.*Preprint*, arXiv:2508\.16260\.
- Li et al\. \(2026\)Junlong Li, Wenshuo Zhao, Jian Zhao, Weihao Zeng, Haoze Wu, Xiaochen Wang, Rui Ge, Yuxuan Cao, Yuzhen Huang, Wei Liu, Junteng Liu, Zhaochen Su, Yiyang Guo, Fan Zhou, Lueyang Zhang, Juan Michelini, Xingyao Wang, Xiang Yue, Shuyan Zhou, and 2 others\. 2026\.[The tool decathlon: Benchmarking language agents for diverse, realistic, and long\-horizon task execution](https://arxiv.org/abs/2510.25726)\.*arXiv:2510\.25726*\.
- Li et al\. \(2025\)Yuetai Li, Huseyin A Inan, Xiang Yue, Wei\-Ning Chen, Lukas Wutschitz, Janardhan Kulkarni, Radha Poovendran, Robert Sim, and Saravan Rajmohan\. 2025\.[Simulating environments with reasoning models for agent training](https://arxiv.org/abs/2511.01824)\.*Preprint*, arXiv:2511\.01824\.
- Liu et al\. \(2023a\)Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, and 3 others\. 2023a\.[AgentBench: Evaluating LLMs as agents](https://arxiv.org/abs/2308.03688)\.*Preprint*, arXiv:2308\.03688\.
- Liu et al\. \(2023b\)Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu\. 2023b\.[G\-eval: NLG evaluation using gpt\-4 with better human alignment](https://doi.org/10.18653/v1/2023.emnlp-main.153)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 2511–2522, Singapore\. Association for Computational Linguistics\.
- Liu et al\. \(2024\)Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh Murthy, Liangwei Yang, Silvio Savarese, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, and Caiming Xiong\. 2024\.[APIGen: Automated pipeline for generating verifiable and diverse function\-calling datasets](https://doi.org/10.52202/079017-1725)\.In*Advances in Neural Information Processing Systems*, volume 37, pages 54463–54482\.
- Lu et al\. \(2025\)Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang\. 2025\.[ToolSandbox: A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities](https://doi.org/10.18653/v1/2025.findings-naacl.65)\.In*Findings of the Association for Computational Linguistics: NAACL 2025*, pages 1160–1183, Albuquerque, New Mexico\. Association for Computational Linguistics\.
- Maekawa et al\. \(2026\)Seiji Maekawa, Jackson Hassell, Pouya Pezeshkpour, Tom Mitchell, and Estevam Hruschka\. 2026\.[Towards reliable benchmarking: A contamination free, controllable evaluation framework for multi\-step LLM function calling](https://arxiv.org/abs/2509.26553)\.In*The Fourteenth International Conference on Learning Representations*\.
- Model Context Protocol \(2024\)Model Context Protocol\. 2024\.Reference MCP servers\.GitHub,[https://github\.com/modelcontextprotocol/servers](https://github.com/modelcontextprotocol/servers)\.Official reference and example MCP server implementations\. Accessed: 2026\-03\-27\.
- Model Context Protocol \(2025\)Model Context Protocol\. 2025\.MCP server registry\.[https://registry\.modelcontextprotocol\.io](https://registry.modelcontextprotocol.io/)\.Official registry of MCP servers\. Accessed: 2026\-03\-27\.
- Patil et al\. \(2025a\)Shishir G\. Patil, Huanzhi Mao, Charlie Cheng\-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E\. Gonzalez\. 2025a\.The Berkeley Function Calling Leaderboard \(BFCL\): From tool use to agentic evaluation of large language models\.In*Forty\-second International Conference on Machine Learning*\.
- Patil et al\. \(2025b\)Shishir G\. Patil, Huanzhi Mao, Charlie Cheng\-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E\. Gonzalez\. 2025b\.The berkeley function calling leaderboard \(BFCL\): From tool use to agentic evaluation of large language models\.In*Forty\-second International Conference on Machine Learning*\.
- Patil et al\. \(2024\)Shishir G\. Patil, Tianjun Zhang, Xin Wang, and Joseph E\. Gonzalez\. 2024\.Gorilla: Large language model connected with massive apis\.In*Advances in Neural Information Processing Systems*\.
- Prabhakar et al\. \(2025\)Akshara Prabhakar, Zuxin Liu, Weiran Yao, Jianguo Zhang, Ming Zhu, Shiyu Wang, Zhiwei Liu, Tulika Awalgaonkar, Haolin Chen, Thai Hoang, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Silvio Savarese, and Caiming Xiong\. 2025\.[APIGen\-MT: Agentic pipeline for multi\-turn data generation via simulated agent\-human interplay](https://openreview.net/forum?id=qk6ORqQ4Cu)\.In*Advances in Neural Information Processing Systems*\.
- Qin et al\. \(2024\)Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun\. 2024\.[ToolLLM: Facilitating large language models to master 16000\+ real\-world APIs](https://openreview.net/forum?id=dHng2O0Jjr)\.In*The Twelfth International Conference on Learning Representations*\.
- Ren et al\. \(2025\)Zhenzhen Ren, Xinpeng Zhang, Zhenxing Qian, Yan Gao, Yu Shi, Shuxin Zheng, and Jiyan He\. 2025\.GTM: Simulating the world of tools for AI agents\.*arXiv:2512\.04535*\.
- Verma et al\. \(2025\)Abhigya Verma, Seganrasan Subramanian, Nandhakumar Kandasamy, and Naman Gupta\. 2025\.[FABRIC: Framework for agent\-based realistic intelligence creation](https://arxiv.org/abs/2510.17995)\.*Preprint*, arXiv:2510\.17995\.
- Wang et al\. \(2024\)Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji\. 2024\.[Mint: Evaluating llms in multi\-turn interaction with tools and language feedback](https://arxiv.org/abs/2309.10691)\.In*Proceedings of the Twelfth International Conference on Learning Representations*\.
- Wang et al\. \(2026\)Zhaoyang Wang, Canwen Xu, Boyi Liu, Yite Wang, Siwei Han, Zhewei Yao, Huaxiu Yao, and Yuxiong He\. 2026\.[Agent world model: Infinity synthetic environments for agentic reinforcement learning](https://arxiv.org/abs/2602.10090)\.*Preprint*, arXiv:2602\.10090\.
- Xu et al\. \(2025\)Zhangchen Xu, Adriana Meza Soria, Shawn Tan, Anurag Roy, Ashish Sunil Agrawal, Radha Poovendran, and Rameswar Panda\. 2025\.[Toucan: Synthesizing 1\.5m tool\-agentic data from real\-world mcp environments](https://arxiv.org/abs/2510.01179)\.*Preprint*, arXiv:2510\.01179\.
- Xu et al\. \(2026\)Zhihao Xu, Rumei Li, Jiahuan Li, Rongxiang Weng, Jingang Wang, Xunliang Cai, and Xiting Wang\. 2026\.[Unlocking implicit experience: Synthesizing tool\-use trajectories from text](https://doi.org/10.48550/arXiv.2601.10355)\.*arXiv:2601\.10355*\.
- Yao et al\. \(2025\)Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan\. 2025\.[τ\\tau\-bench: A benchmark for tool\-agent\-user interaction in real\-world domains](https://openreview.net/forum?id=roNSXZpUDN)\.In*Proceedings of the Thirteenth International Conference on Learning Representations*\.
- Zeng et al\. \(2026\)Yucheng Zeng, Weipeng Lu, Linyun Liu, Shupeng Li, Zitian Qu, Chenghao Zhu, Shaofei Li, Zhengdong Tan, Mengyue Liu, Haotian Zhao, Zhe Zhou, and Jianmin Wu\. 2026\.[Logigen: Logic\-driven generation of verifiable agentic tasks](https://arxiv.org/abs/2603.00540)\.*arXiv:2603\.00540*\.
- Zhang et al\. \(2026\)Zeyu Zhang, Guohao Li, Zhenchang Xing, Alexandros Apostolopoulos, Yu Lin Lee, and Liang Zheng\. 2026\.[Gecko: A simulation environment with stateful feedback for refining agent tool calls](https://arxiv.org/abs/2602.19218)\.*arXiv:2602\.19218*\.

## Appendix AExtended Experimental Results

### A\.1Evaluation Scale

Table[6](https://arxiv.org/html/2608.26133#A1.T6)breaks down the 337 generated scenarios and 391 evaluation records by MCP specification\. Records exceed scenarios because multi\-turn scenarios contribute one record per turn \(single\-turn entry plus one multi\-turn entry per additional turn\)\.

Table 6:Evaluation scale by MCP specification\.
### A\.2Complexity Breakdown

Table[7](https://arxiv.org/html/2608.26133#A1.T7)reports mean tool\-calling scores split by complexity tier \(simple vs\. complex\) for each MCP\. TheΔ\\Deltacolumn shows the score change from simple to complex; a negative value indicates quality degradation under higher complexity\. Elasticsearch shows the largest degradation \(−\-12\.2pp\), followed by Git \(−\-11\.1pp\)\. Selenium is notably stable, scoring slightly higher on complex scenarios\.

Table 7:Tool\-calling scores by complexity tier and MCP\.
### A\.3Coherence Sub\-Dimensions

Table[8](https://arxiv.org/html/2608.26133#A1.T8)reports mean scores for each of the five coherence sub\-dimensions on the raw 1–3 scale used by the judge\. Conciseness is near\-ceiling across all domains \(mean 2\.90, std 0\.33\), indicating the pipeline reliably produces appropriately scoped responses without unnecessary elaboration\. Completeness is the primary bottleneck \(mean 2\.04\), driven by responses that address the user’s request but omit contextual detail that a practitioner would expect—the most actionable target for future pipeline improvements\.

Table 8:Coherence sub\-dimension scores \(raw 1–3 scale\)\.
### A\.4Grounding Tiers

Table[9](https://arxiv.org/html/2608.26133#A1.T9)defines the grounding tiers assigned to each mock output at Stage 3\. The tier records how much reference material was available to the LLM when generating the synthetic tool response:highwhen concrete example outputs were present in the tool specification,mediumwhen similar tools provided transferable examples, andlowwhen generation relied on the parameter schema alone\. In the current experiments, all seven MCP specifications lack example outputs, so all 871 mock calls are annotated aslow\. The annotation is carried through to the harness artifact for downstream filtering when specs with richer documentation are used\.

Table 9:Grounding tiers for mock output generation\.
### A\.5Workflow Composition and Category Diversity

Selenium generates the longest workflows \(mean 9\.71 calls, max 17\) reflecting the sequential nature of browser automation tasks\. Illustrator workflows are also long \(mean 3\.33, max 12\), while Redis \(mean 1\.27\) and Slack \(mean 1\.39\) tend toward atomic single\-tool scenarios\. Category count \(108 unique categories across 337 scenarios; 3\.1 scenarios per category\) scales with scenario count rather than tool count\.

### A\.6Failure Details

#### Failure concentration by MCP\.

Git exhibits the highest failure rate: 7% of records score below 0\.5, followed by Filesystem \(5%\) and Redis \(1%\); four MCPs have zero failures\. Git’s failures are driven by argument correctness for tools with complex parameter schemas\.

#### Argument failure patterns\.

Among argument failures,value accuracyis the dominant sub\-dimension \(223 records\), followed byrelevancy\(44\),format compliance\(35\),type compliance\(31\),completeness\(16\), andname accuracy\(11\)\.

Table 10:Coherence issue taxonomy grouped from evaluator notes\.

### A\.7Per\-MCP Dimension Scores

Table 11:Mean TC dimension scores by MCP \(overall\)\. Selenium’s ordering \(0\.989\) is the highest of any MCP, despite having the longest workflows\. Git’s arguments \(0\.780\) are weakest, consistent with its high parameter density\.
### A\.8Out\-of\-Family Judge Replication

To assess whether reported scores are artifacts of within\-family judge circularity, the full 391\-record evaluation corpus was re\-scored with an out\-of\-family judge: Qwen3\.5\-122B\-A10B\-FP8 \(Alibaba family; vLLM\-hosted, FP8 quantization\)\. Each record’s existing chat transcript was replayed against the alternate judge with only the evaluator model swapped; the generation pipeline and rubric are unchanged\. After joining on the composite key \(MCP,conversation\_id\), 384 records have paired tool\-calling scores and 380 have paired coherence scores under both judges; all paired statistics below are computed on these sets\.

#### Per\-MCP means with bootstrap CIs\.

Table[12](https://arxiv.org/html/2608.26133#A1.T12)reports per\-MCP means and 95% percentile bootstrap CIs \(B=2,000B\{=\}2\{,\}000\) under both judges\. Tool\-calling CIs overlap for every MCP\. Coherence CIs are disjoint for all seven MCPs \(the Qwen 3\.5 judge is systematically lower\), and the identity of the worst\-coherence MCP differs across judges \(Git under Gemini, Illustrator under Qwen 3\.5\)\.

Table 12:Per\-MCP means under the Gemini judge \(gemini\-2\.5\-flash\-lite\) and the Qwen 3\.5 judge \(Qwen3\.5\-122B\-A10B\-FP8\)\.nncounts come from the \(MCP,conversation\_id\) join, which preserves all records; Gemini means match those reported in Table[3](https://arxiv.org/html/2608.26133#S5.T3)\. 95% bootstrap CIs \(omitted for space\) overlap for every MCP on TC; coherence CIs are disjoint for all 7 MCPs\. The bottom\-coherence MCP differs across judges \(Git under Gemini, Illustrator under Qwen 3\.5\)\.MCP\-level Spearman rank correlations across the seven MCPs areρ=0\.86\\rho\{=\}0\.86for tool\-calling andρ=0\.46\\rho\{=\}0\.46for coherence\. The top tool\-calling MCP \(Redis\) is preserved under both judges; the bottom\-coherence MCP differs \(Git under Gemini, Illustrator under Qwen 3\.5\)\.

#### Record\-level agreement\.

Table[13](https://arxiv.org/html/2608.26133#A1.T13)reports paired\-record agreement on the aggregate per\-aspect unsupervised score \(agg\_unsupervised\_score, the metric reported throughout the paper\)\. Tool\-calling agreement is strong; coherence is moderate\.

Table 13:Record\-level agreement between judges, computed on records keyed by \(MCP,conversation\_id\) to avoid scenario\-id collisions across MCPs\. Bootstrap 95% CIs \(B=2,000B\{=\}2\{,\}000\) omitted for space\. MAE and the±0\.1\\pm 0\.1agreement rate reproduce the within\-tolerance qualitative pattern of the original analysis\.
#### Systematic shift\.

The mean signed difference \(Qwen 3\.5−\-Gemini\) isΔ​μTC≈0\\Delta\\mu\_\{\\textrm\{TC\}\}\{\\approx\}0\(95% CI\[−0\.009,\+0\.008\]\[\-0\.009,\+0\.008\]; balanced sign\-test\) andΔ​μCoh≈−0\.16\\Delta\\mu\_\{\\textrm\{Coh\}\}\{\\approx\}\-0\.16\(95% CI\[−0\.173,−0\.138\]\[\-0\.173,\-0\.138\]; the majority of non\-tied records score lower under the Qwen 3\.5 judge\)\. Tool\-calling shows no systematic bias; coherence is systematically stricter under the out\-of\-family judge\. Without a third judge or a human anchor, neither judge can be designated “correct” on coherence; absolute coherence levels are therefore reported as judge\-dependent\.

#### Failure\-mode taxonomy replication\.

Table[14](https://arxiv.org/html/2608.26133#A1.T14)reports record counts where each argument sub\-dimension scored below 1\.0 under each judge\. Both judges identifyvalue\_accuracyas the dominant argument failure mode by a 4–5×\\timesmargin over the next\-highest sub\-dimension, replicating the central qualitative finding of Section[5](https://arxiv.org/html/2608.26133#S5); counts are within 5% across judges\.

Table 14:Argument sub\-dimension failure counts \(records with score<1\.0<1\.0\) under each judge\. Value\-accuracy dominance is preserved bilaterally; counts agree within 5% across judges\.
#### Visual diagnostics\.

Figure[1](https://arxiv.org/html/2608.26133#A1.F1)shows scatter \(top row\) and Bland–Altman \(bottom row\) plots for tool\-calling and coherence\. The TC scatter clusters tightly alongy=xy\{=\}xand the TC Bland–Altman limits\-of\-agreement bracket zero symmetrically; the coherence scatter shows a visible offset belowy=xy\{=\}xand the coherence Bland–Altman LoA is shifted≈−0\.14\{\\approx\}\{\-\}0\.14below zero with wider spread\.

![Refer to caption](https://arxiv.org/html/2608.26133v1/x1.png)Figure 1:Record\-level agreement between the Gemini judge \(gemini\-2\.5\-flash\-lite\) and the Qwen 3\.5 judge \(Qwen3\.5\-122B\-A10B\-FP8\)\. Top: scatter \(Qwen 3\.5 vs\. Gemini\); the TC cloud \(left\) lies ony=xy\{=\}x, the coherence cloud \(right\) is offset below\. Bottom: Bland–Altman; TC differences are centered on zero, coherence differences are shifted≈−0\.138\{\\approx\}\{\-\}0\.138\.
#### Interpretation\.

Per\-MCP TC means, the MCP TC ranking, and the dominant argument\-value\-accuracy failure mode are robust to the judge swap\. Absolute coherence levels and MCP\-level coherence rank\-preservation \(ρCoh=0\.46\\rho\_\{\\textrm\{Coh\}\}\{=\}0\.46\) are judge\-dependent\. The worst\-coherence MCP differs across judges \(Git under Gemini, Illustrator under Qwen 3\.5\), reflecting the broader judge\-dependence of coherence scoring rather than a contradiction in the underlying data\.

Table 15:Schema complexity profile\.\|T\|\|T\|= tools,p¯\\bar\{p\}= mean params/tool,pmaxp\_\{\\max\}= max params, Opt% = optional parameter ratio,w¯\\bar\{w\}= mean workflow length\. Per\-MCP correlations with mean TC: average parameters per toolr=−0\.60r=\-0\.60, optional ratior=−0\.66r=\-0\.66, tool countr=\+0\.40r=\+0\.40\(positive but smaller\)\. Tool\-level disaggregation \(n=222n=222\) confirms the schema\-complexity direction withp<0\.001p<0\.001: average parametersr=−0\.29r=\-0\.29, optional ratior=−0\.30r=\-0\.30, both relative to mean unsupervised TC\.

## Appendix BAggregation Sensitivity

The reported tool\-calling scores combine the four top\-level dimensions via arithmetic mean\. Table[16](https://arxiv.org/html/2608.26133#A2.T16)re\-aggregates the same per\-turn dimension scores under harmonic mean and minimum\-across\-dimensions \(n=385n=385; six coherence\-only records omitted\)\. Corpus\-wide means shift substantially \(0\.911 / 0\.851 / 0\.781\), but per\-MCP rankings are highly stable: Spearmanρ=0\.964\\rho=0\.964\(Arith\. vs Harm\.\),0\.9640\.964\(Arith\. vs Min\.\),0\.9290\.929\(Harm\. vs Min\.\)\. The only swap is Filesystem↔\\leftrightarrowGit near the bottom; the top\-3 ordering is identical under all three rules\. The schema\-complexity correlation \(Section[5](https://arxiv.org/html/2608.26133#S5)\) holds in direction throughout: Pearsonr=−0\.60r=\-0\.60\(Arith\.\),−0\.48\-0\.48\(Harm\.\),−0\.50\-0\.50\(Min\.\) across the seven MCPs, and localizes to the argument sub\-dimension \(r=−0\.86r=\-0\.86versus\|r\|≤0\.46\|r\|\\leq 0\.46for the other three\)\. Argument correctness is also the dominant failure mode at the input level \(60\.5% of 448 turns score below 1\.0, vs\. 29\.1% for ordering, 28\.6% for selection, 1\.3% for usage\)—a turn\-level fact unaffected by the outer aggregation choice\.

Table 16:Per\-MCP TC means under arithmetic, harmonic, and minimum aggregation across the four dimensions \(n=385n=385\)\.
## Appendix CNovelty Summary

Table[17](https://arxiv.org/html/2608.26133#A3.T17)situates the contributions against prior literature across two gaps: the data generation gap \(rows 1–3\) and the metric decomposition gap \(rows 4–6\)\.

Table 17:Contributions vs\. prior literature, organized by thematic gap\.
## Appendix DPipeline Prompts

This section reproduces the instructional content of the LLM prompts used at each pipeline stage\. JSON output templates and tool\-spec payloads are abbreviated for space\.

### D\.1Stage 1: Tool Interpretation

The tool interpreter receives a single MCP tool specification and produces a structured semantic explanation, requested as a JSON object with five named fields\.

```
I have a tool that can be called by an agent,
and I could use help understanding what it
does and what it is helpful for.

Tool info:
‘‘‘json
{tool_info}
‘‘‘

I need a json in the following format that
can help me thoroughly understand what the
tool is capable of, especially in an
enterprise context. Keep the explanations
grounded within the tool info.

{
  "tool_name": <Tool name here, as given>,
  "what_it_does": <Complete explanation of
      the tool’s functionality and what it
      aims to do>,
  "what_it_needs": <What parameters the tool
      needs and how they should be formatted>,
  "why_its_used": <Reasons an agent would
      call this tool; potential use cases>,
  "enterprise_context": <Tags for what aspect
      of an enterprise this could help with>
}
```

### D\.2Stage 2: Scenario Generation

Two prompts generate scenarios at different complexity levels from the enriched tool summaries\. Both target an “agentic chatbot for enterprise use cases” and request scenarios organized by category, each containing an exactagent\_workflowof function calls with parameters\.

#### Simple scenarios\.

```
I’m building an agentic chatbot for enterprise
use cases. Based on the available tool
capabilities below, generate realistic,
straightforward, and commonplace scenarios
organized by category that showcase how
employees would use this chatbot for everyday
tasks. These examples would not require too
many tool calls -- they’ll be smaller and
more precise.

Available Tool Capabilities:
{tool_summary}

For each scenario, include the exact function
calls the agent would make using the available
tools.

[JSON format omitted: categories[].scenarios[]
 with title, prompt, agent_workflow[], novelty
 _reason, agent_followup]

Make sure:
 1. Use actual tool names from the available
    capabilities
 2. Function names and parameters are
    structured separately with realistic
    values adhering to the parameter schema
 3. Workflows show logical progression
 4. Scenarios are practical and commonly
    encountered
 5. Agent workflows do the necessary context
    management & tool calls to identify how
    parameters are selected
 6. Provide meaningful agent_followup content
    that makes sense within the context of
    the scenario.
```

#### Complex scenarios\.

The complex prompt mirrors the simple prompt but replaces “straightforward, and commonplace” with “novel, and complex”, asks for advanced and creative tool usage, and adds two “Make sure” items:each scenario demonstrates complex, multi\-step processesandcreative combinations of tools that unlock new capabilities\.

#### Coverage hint and follow\-up\.

A coverage suffix appended to the initial prompt instructs:“IMPORTANT: Ensure broad coverage across ALL available tools\. Every tool listed above should appear in at least one scenario’s agent\_workflow\. There are\{N\}tools total — design scenarios that collectively exercise all of them\.”If tools remain uncovered after the first round, a follow\-up prompt requests additional scenarios for the named uncovered tools, allowing combination with previously covered tools in multi\-step workflows\.

### D\.3Stage 3: Mock Output Generation

```
You are a Mock Tool Output Generator for
synthetic agent workflow data. Your task is
to generate realistic mock tool outputs that
complete synthetic scenarios.

You will be given an initial prompt (and
maybe a description of what the aim of the
prompt is & why the scenario is of interest).
Then, you will be given an agent workflow.
This will detail:
  (1) A function call w/ parameters and
  (2) A quick explanation of what the tool
      does.

Finally, you will have some example function
calls with their respective outputs OR a JSON
schema object describing the output
structure. Use this to guide formatting for
the final mock tool output.

[JSON output: mock_workflow[] with function
 _name, parameters, quick_explanation,
 mock_output, confidence; plus expected
 _response that references specific mock data]

### CONFIDENCE LEVEL GUIDELINES
"high":   concrete example for THIS specific
          function was provided
"medium": no example for this function, but
          similar functions have examples
"low":    no example output provided for
          this function

### CRITICAL INSTRUCTIONS
 1. Concrete Data: replace placeholders (e.g.
    "{user_id}" or "XYZ") with realistic,
    specific values.
 2. Realism & Diversity: reflect how a real
    system would respond; incorporate diverse
    names, global locations, and varied data
    points.
 3. Formatting: strictly adhere to provided
    reference examples or JSON schema.
 4. Expected response references concrete
    mock data (names, IDs, counts, statuses,
    dates) and reflects the full workflow,
    not just the last call.
```

### D\.4Stage 4: Multi\-Turn Expansion

The multi\-turn expansion prompt is constructed in parts: a preamble, scenario and tool\-result context, key principles, task instructions, worked examples of good vs\. bad turn splitting, quality guidelines, and a JSON output template\. The principle set differs betweenmock\-data\-groundedmode \(when synthetic outputs exist\) andstandardmode\.

```
You are a Natural Conversation Flow Analyzer
for enterprise agent interactions. Your task
is to take a scenario and intelligently break
it into natural conversation turns that
reflect how real employees would interact
with an agent.

# Key Principles (mock-data-grounded mode)
 1. Use Available Tool Results to inform
    realistic follow-up questions and workflow
    decisions.
 2. Data-Driven Breakpoints from the actual
    data returned by tools.
 3. Realistic User Reactions to the specific
    data shown.
 4. Progressive Data Exploration: each turn
    builds on prior tool outputs.
 5. Split at Phase Boundaries when distinct
    phases exist (e.g., information-gathering
    then acting on it).
 6. Group Related Operations Within a Phase:
    batch repeated operations (e.g., 3
    lookups) into a SINGLE turn.
 7. Self-Contained Turns: agent_followup
    reports concrete results, NOT narration of
    future actions.
 8. Complete the Full Workflow: every tool
    call from the initial workflow appears in
    exactly one turn.
 9. CRITICAL CONSTRAINT - Use Only Existing
    Functions: same function names and
    parameter structures as the initial
    workflow.

[Worked examples follow: BAD over-granular
 splitting, BAD no-splitting, GOOD batched-
 within-phases. JSON format omitted: turns[]
 with title, prompt, agent_workflow[]
 including mock_output and confidence,
 novelty_reason, agent_followup]
```

Instandardmode \(no mock outputs available\), the principles drop the data\-driven breakpoint and tool\-result references, and the JSON template omits themock\_outputandconfidencefields\. The phase\-boundary, batching, self\-contained\-turn, and function\-reuse constraints are preserved across both modes\.

## Appendix EEvaluation Prompts

The quality assessment framework \(Section[4](https://arxiv.org/html/2608.26133#S4)\) uses two LLM\-as\-judge prompts: one for tool\-calling correctness and one for conversational coherence\. Both operate in unsupervised mode \(no reference answer\)\. Tables[18](https://arxiv.org/html/2608.26133#A5.T18)and[19](https://arxiv.org/html/2608.26133#A5.T19)detail the aspects measured, their definitions, and scoring scales\.

### E\.1Tool\-Calling Evaluation

The tool\-calling prompt evaluates across four dimensions, each with scored sub\-dimensions on a 0–10 scale\. Sub\-scores are normalized to 0–1; the selection, ordering, and argument dimension scores are the arithmetic mean of their sub\-scores, while the usage dimension score is taken directly from its necessity sub\-score\. The four top\-level dimension scores are then combined via arithmetic mean\. Cascading penalties on argument sub\-scores are enforced through prompt instructions to the judge \(Table[18](https://arxiv.org/html/2608.26133#A5.T18), footnote\)\.

DimensionSub\-dimensionDefinitionScaleUsageNecessityWas a tool actually needed, or could the assistant answer directly?0–10Overuse detectionAre there redundant or unnecessary tool calls?0–10SelectionCorrectnessDo the selected tools match the task described by the user?0–10SpecificityWas the most specific tool chosen when alternatives exist?0–10CompletenessAre all tools needed to fully address the query called?0–10OrderingSequence logicIs the execution order logical; do later calls build on earlier ones?0–10Dependency handlingAre inter\-tool dependencies respected \(output→\\rightarrowinput\)?0–10Execution efficiencyCould reordering improve efficiency?0–10ArgumentsCompletenessAre all required parameters provided?0–10Name accuracyDo parameter names match schemas exactly \(case\-sensitive\)?0–10Value accuracyAre values correct and grounded in the user query or prior tool outputs?0–10Type complianceDo parameter values match expected data types?0–10Format complianceDo values follow expected formats \(dates, enums, patterns\)?0–10RelevancyAre there any extra or invalid parameters not in the schema?0–10Table 18:Tool\-calling evaluation dimensions and sub\-dimensions\. Ordering is marked not applicable for single tool calls and excluded from aggregation\.Cascading rules\(enforced via prompt instructions to the judge\): when the judge identifies a wrong parameter name \(score≤\\leq2\), it is instructed to assign near\-zero scores to the dependent argument sub\-dimensions \(value, type, format\); a missing required parameter triggers the same cascade; a wrong value \(score≤\\leq3\) cascades to type, format, and relevancy\. Values from prior tool outputs in chained calls are not penalized\.
### E\.2Coherence Evaluation

The coherence prompt evaluates across five dimensions on a 1–3 scale, normalized to 0–1 and aggregated via arithmetic mean\. Each dimension checks for specific failure manifestations\.

Table 19:Coherence evaluation dimensions\. Each dimension is scored on a 1–3 scale:Good\(3\) = no manifestations detected;Adequate\(2\) = 1–2 minor manifestations;Poor\(1\) = 3\+ manifestations or critical failures\. Context retention is optional \(excluded when no conversation history exists\)\.

## Appendix FStructured Output Schemas

ToolInfoname: strdescription: strinput\_schema: dictannotations: dictInputToolExplanationtool\_name: strwhat\_it\_does: strwhy\_its\_used: strwhat\_it\_needs: strenterprise\_context: strScenariotitle: strprompt: stragent\_workflow:list\[AgentCall\]novelty\_reason: stragent\_followup: strStage 1: InterpretStage 2: GenerateAgentCallfunction\_name: strparameters: dictquick\_explanation: strMockOutputextends AgentCallmock\_output: str\|dictconfidence: Enumhigh\|medium\|lowMockWorkflowmock\_workflow:list\[MockOutput\]expected\_response: strMockMultiTurn\-Scenarioturns: list\[Turn\]each turn:prompt: stragent\_workflow:list\[MockOutput\]agent\_followup: strStage 3: MockStage 4: ExpandFigure 2:Structured output schemas across the four pipeline stages\. Solid arrows indicate data flow; dashed arrows indicate schema inheritance \(MockOutputextendsAgentCall, which is referenced byScenario\)\.Each pipeline stage uses Pydantic models as structured output schemas, constraining the LLM to produce well\-formed JSON at every step\.

## Appendix GScore Distributions and Per\-Domain Analysis

### G\.1Overall Score Distributions

Figure[3](https://arxiv.org/html/2608.26133#A7.F3)shows the distribution of tool\-calling and coherence scores across all 391 evaluation records\. The tool\-calling distribution is concentrated in the upper range \(31\.7% perfect scores, 2\.3% below 0\.5\)\. The coherence distribution is left\-skewed with a mode near 0\.9\.

![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/tool_calling_distribution_kde.png)\(a\)Tool\-calling score distribution with KDE\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/coherence_distribution_kde.png)\(b\)Coherence score distribution with KDE\.

Figure 3:Score distributions across all 391 evaluation records\.
### G\.2Per\-Domain Analysis

Figures[4](https://arxiv.org/html/2608.26133#A7.F4)–[10](https://arxiv.org/html/2608.26133#A7.F10)show the scenario category breakdown \(simple vs\. complex\), workflow length distribution, tool co\-occurrence graph, and sequential adjacency graph for each MCP specification\. Co\-occurrence graphs show which tools appear in the same scenario \(undirected\); adjacency graphs show toolA→A\\totoolBBtransition patterns within ordered workflows \(directed\)\.

![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/elasticsearch/elasticsearch_category.png)\(a\)Scenario categories\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/elasticsearch/elasticsearch_length.png)\(b\)Workflow length distribution\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/elasticsearch/elasticsearch_exploration.png)\(c\)Tool co\-occurrence graph\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/elasticsearch/elasticsearch_adjacency.png)\(d\)Sequential adjacency graph\.

Figure 4:Elasticsearch \(20 tools\): scenario categories, workflow lengths \(mean 1\.9 calls\), tool co\-occurrence, and sequential adjacency\.![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/illustrator/illustrator_category.png)\(a\)Scenario categories\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/illustrator/illustrator_length.png)\(b\)Workflow length distribution\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/illustrator/illustrator_exploration.png)\(c\)Tool co\-occurrence graph\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/illustrator/illustrator_adjacency.png)\(d\)Sequential adjacency graph\.

Figure 5:Illustrator \(64 tools\): scenario categories, workflow lengths \(mean 3\.3 calls, max 12\), tool co\-occurrence, and sequential adjacency\.![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/redis/redis_category.png)\(a\)Scenario categories\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/redis/redis_length.png)\(b\)Workflow length distribution\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/redis/redis_exploration.png)\(c\)Tool co\-occurrence graph\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/redis/redis_adjacency.png)\(d\)Sequential adjacency graph\.

Figure 6:Redis \(47 tools\): scenario categories, workflow lengths \(mean 1\.3 calls\), tool co\-occurrence, and sequential adjacency\.![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/slack/slack_category.png)\(a\)Scenario categories\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/slack/slack_length.png)\(b\)Workflow length distribution\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/slack/slack_exploration.png)\(c\)Tool co\-occurrence graph\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/slack/slack_adjacency.png)\(d\)Sequential adjacency graph\.

Figure 7:Slack \(16 tools\): scenario categories, workflow lengths \(mean 1\.4 calls\), tool co\-occurrence, and sequential adjacency\.![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/filesystem/filesystem_category.png)\(a\)Scenario categories\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/filesystem/filesystem_length.png)\(b\)Workflow length distribution\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/filesystem/filesystem_exploration.png)\(c\)Tool co\-occurrence graph\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/filesystem/filesystem_adjacency.png)\(d\)Sequential adjacency graph\.

Figure 8:Filesystem \(14 tools\): scenario categories, workflow lengths \(mean 2\.0 calls\), tool co\-occurrence, and sequential adjacency\.![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/git/git_category.png)\(a\)Scenario categories\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/git/git_length.png)\(b\)Workflow length distribution\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/git/git_exploration.png)\(c\)Tool co\-occurrence graph\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/git/git_adjacency.png)\(d\)Sequential adjacency graph\.

Figure 9:Git \(33 tools\): scenario categories, workflow lengths \(mean 2\.2 calls\), tool co\-occurrence, and sequential adjacency\.![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/selenium/selenium_category.png)\(a\)Scenario categories\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/selenium/selenium_length.png)\(b\)Workflow length distribution\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/selenium/selenium_exploration.png)\(c\)Tool co\-occurrence graph\.
![Refer to caption](https://arxiv.org/html/2608.26133v1/plots/selenium/selenium_adjacency.png)\(d\)Sequential adjacency graph\.

Figure 10:Selenium \(56 tools\): scenario categories, workflow lengths \(mean 9\.7 calls, max 17\), tool co\-occurrence, and sequential adjacency\.