AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling

# AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling
Source: [https://arxiv.org/html/2608.26623](https://arxiv.org/html/2608.26623)
Abhigya VermaAffiliation:Amit Kumar SahaAffiliation:Seganrasan SubramanianAffiliation:Sai Harshitha AluruAffiliation:ServiceNow AIAffiliation:Hyderabad, India

###### Abstract

LLM judges are widely used to evaluate agentic tool\-calling systems, yet their reliability on structured, dependency\-driven workflows remains largely unexamined\. We present AgentJudgeBench, the first benchmark to systematically study LLM\-as\-a\-judge reliability for agentic tool\-calling over workflow DAGs, as distinct from the broader LLM\-as\-a\-judge task of open\-ended text or preference evaluation\. The benchmark comprises 3,808 instances spanning six DAG topologies and three difficulty tiers, evaluated with five generators \(3B–70B open\-weight models and GPT\-5\.4\) and six judges \(20B to frontier scale\) under paired with\- and without\-ground\-truth conditions\. Judge alignment degrades monotonically with task difficulty, 1\.5×\\timesfaster without ground truth, and on hard queries without ground truth all six judges converge to a narrow 77–82% band regardless of scale, revealing a structural ceiling, driven primarily by task difficulty though its height is partly prompt\-dependent for weaker generators, that model capacity alone cannot overcome\. Ground\-truth exposure is not uniformly beneficial: it reduces alignment for GPT\-5\.4 \(1\.5 pp\) and Gemini\-2\.5\-Pro \(3\.9 pp\), consistent with over\-anchoring\. Among mitigation strategies, chain\-of\-thought reasoning and judge temperature both have negligible effect, while structured evaluation rubrics improve alignment by up to 6\.5 pp but do not generalise uniformly across judge\-generator pairs\. With ground truth, QwQ\-32B best matches the programmatic reference, while a human validation study identifies GPT\-OSS\-120B as the most human\-aligned judge; without it, frontier judges lead only marginally within the shared ceiling\. These results expose fundamental limitations of current LLM judges and yield practical guidelines for reliable evaluation in agentic systems\. The code and dataset are available at[https://github\.com/ServiceNow/SyGra/tree/scratch/agent\_judge\_bench/tasks/agentic\_bfcl\_judge\_eval](https://github.com/ServiceNow/SyGra/tree/scratch/agent_judge_bench/tasks/agentic_bfcl_judge_eval)and[https://huggingface\.co/datasets/ServiceNow\-AI/AgentJudgeBench](https://huggingface.co/datasets/ServiceNow-AI/AgentJudgeBench)\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2608.26623v1/overall_flow.png)Figure 1:Overall pipeline of AgentJudgeBench\. Each BFCL\-style Agentic record undergoes difficulty\-controlled rewriting, generator inference, and parallel scoring by a programmatic judge and LLM judges \(with/without GT\), followed by alignment with the programmatic reference\.Using LLMs as automated judges has become standard practice for evaluating model outputs\([Zheng et al\., 2023](https://arxiv.org/html/2608.26623#bib.bib1);[Tan et al\., 2025](https://arxiv.org/html/2608.26623#bib.bib15);[Li et al\., 2024b](https://arxiv.org/html/2608.26623#bib.bib16)\)\. On text\-centric tasks – dialogue, summarization, instruction following – judge reliability is well characterized, with documented biases and known failure modes\([Zheng et al\., 2023](https://arxiv.org/html/2608.26623#bib.bib1);[Wang et al\., 2023a](https://arxiv.org/html/2608.26623#bib.bib22)\)\. As LLMs are increasingly deployed as autonomous agents that invoke tools and orchestrate multi\-step workflows, the judge paradigm has been extended naturally to agentic tool\-calling\([Qin et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib23);[Guo et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib24);[Guo et al\., 2025](https://arxiv.org/html/2608.26623#bib.bib25)\)\. This extension, however, has proceeded without a basic calibration check:*how reliable are LLM judges in this structured setting?*

Agentic tool\-calling differs from text evaluation in ways that matter for judge reliability\. Correctness is not a matter of fluency or preference: it requires selecting the right tools from a typed schema, supplying well\-formed arguments, ordering calls to respect execution dependencies, and covering all parts of the user’s intent\. A plan can fail in four orthogonal ways \(tool selection, parameter structure, sequence accuracy, query coverage\) that do not correlate cleanly – a judge reliable at one may be blind to another, e\.g\., calibrated on simple sequential tasks yet fail on complex fan\-in or diamond workflows where parallel branches must converge\. When no ground\-truth execution trace is available, the common deployment scenario, the judge must instead reconstruct correctness from the query and tool schemas alone, a fundamentally harder inference problem\. Existing benchmarks that deploy LLM judges for tool\-calling report only aggregate pass\-rate agreement and vary none of these dimensions\([Qin et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib23);[Guo et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib24);[Guo et al\., 2025](https://arxiv.org/html/2608.26623#bib.bib25)\), leaving practitioners without a principled basis for choosing a judge, setting its configuration, or interpreting its outputs\.

We introduceAgentJudgeBenchto close this gap\. The benchmark comprises 3,808 BFCL\-style records\([Patil et al\., 2025](https://arxiv.org/html/2608.26623#bib.bib11)\)spanning six DAG111A Directed Acyclic Graph \(DAG\) encodes execution dependencies between tool calls: nodes are tool invocations and directed edges indicate that one call must complete before another can begin, with the acyclicity constraint preventing circular dependencies\.topologies at three controlled difficulty tiers, each with a programmatically verified ground\-truth trace\. Five generators – four open\-weight \(3B–70B\) and one frontier \(GPT\-5\.4\) – produce tool\-calling outputs that are scored by six LLM judges \(ranging from 20B open\-weight to frontier closed systems\) under paired with\- and without\-GT conditions across four structural metrics \(tool selection, parameter structure, sequence accuracy, query coverage\), yielding321,648321\{,\}648paired with\-GT/without\-GT evaluations \(Appendix[D](https://arxiv.org/html/2608.26623#A4)\)\. We organize our analysis around six research questions: which metrics and topologies are hardest \(RQ1\), how much judges agree with each other \(RQ2\), why without\-GT alignment converges on hard queries \(RQ3\), and how sensitive alignment is to judge temperature \(RQ4\), chain\-of\-thought reasoning \(RQ5\), and prompt format \(RQ6\)\.

The results are non\-obvious\. Judge alignment degrades monotonically with query difficulty,1\.5×1\.5\\timesfaster without ground truth than with it, and all six judges converge to a7777–82%82\\%ceiling on hard queries without a reference, regardless of model capacity; a follow\-up ablation \(§[4\.2](https://arxiv.org/html/2608.26623#S4.SS2)\) confirms task difficulty as the primary driver for capable generators, though its height is somewhat prompt\-dependent for weaker ones\. GT exposure is not universally beneficial: two frontier judges \(GPT\-5\.4, Gemini\-2\.5\-Pro\) are*less*aligned when shown the reference, consistent with over\-anchoring rather than independent judgement\. Chain\-of\-thought adds at most0\.30\.3pp across 24 paired comparisons, and temperature has negligible effect \(≤0\.25\{\\leq\}0\.25pp spread, two judge\-generator pairings\)\. Structured per\-metric prompt rubrics add\+4\.8\+4\.8–\+6\.5\+6\.5pp over free\-form on one pairing, the largest lever we test, but a second pairing shows a smaller effect that reverses on hard queries, so we treat prompt format as impactful but judge/generator\-dependent rather than universally dominant \(§[4\.2](https://arxiv.org/html/2608.26623#S4.SS2)\)\. Inter\-judge agreement is moderate \(κ≈0\.42\\kappa\\approx 0\.42with GT\), with systematic, capacity\-correlated patterns not visible in aggregate scores\.

Scope\.We study LLM\-as\-a\-judge specifically for agentic tool\-calling, not the broader LLM\-as\-a\-judge literature; DAG\-structured tool\-use data, synthetic dependency graphs, and programmatic trajectory scoring each have close prior work \(§2\)\. Our contribution is the reliability*protocol*that combines them – paired with\-GT/without\-GT conditions, controlled difficulty, and per\-metric decomposition – applied to measure judge reliability rather than agent capability\.

Contributions\.\(1\)A dataset of 3,808 records spanning six DAG topologies and three difficulty tiers with programmatically verified ground\-truth traces, to be released publicly\.\(2\)A four\-metric evaluation framework with a paired with\-GT/without\-GT protocol and bootstrap confidence intervals\.\(3\)A systematic six\-RQ empirical study yielding actionable guidance: with ground truth, QwQ\-32B best matches the programmatic reference, while GPT\-OSS\-120B is the most human\-aligned judge in our validation study \(Appendix[G](https://arxiv.org/html/2608.26623#A7)\); without ground truth, frontier judges lead narrowly, but the convergence ceiling limits the practical difference\.

## 2Related Work

Table[30](https://arxiv.org/html/2608.26623#A21.T30)\(Appendix[V](https://arxiv.org/html/2608.26623#A22)\) surveys prior work across three related areas; we explain the key distinctions below\.

### 2\.1Agentic Data and Tool\-Calling Benchmarks

The dominant paradigm for evaluating tool\-calling agents uses either environment\-based execution feedback\([Zhou et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib7);[Drouin et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib8);[Trivedi et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib9);[Yao et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib14)\)or deterministic scoring against annotated trajectories\([Deng et al\., 2023](https://arxiv.org/html/2608.26623#bib.bib3);[Xu et al\., 2025a](https://arxiv.org/html/2608.26623#bib.bib10);[Patil et al\., 2025](https://arxiv.org/html/2608.26623#bib.bib11)\)\.τ\\tau\-bench\([Yao et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib14)\)stresses agents against simulated users and proposespass^kreliability rather than per\-turn correctness; like this line generally, it scores agents but does not interrogate the judge producing those scores\. BFCL\([Patil et al\., 2025](https://arxiv.org/html/2608.26623#bib.bib11)\)is the closest structural ancestor to our data format, isolating single\-turn function\-calling with typed JSON schemas and AST\-based ground\-truth comparison; we adopt the same schema but extend it to multi\-step DAG\-structured workflows at three controlled difficulty tiers\. The most structurally similar work is FuncBenchGen\([Maekawa et al\., 2025](https://arxiv.org/html/2608.26623#bib.bib13)\), which frames multi\-step calling as DAG traversal with controllable complexity\. The key distinction is purpose: FuncBenchGen trains and evaluates*generator*agents, whereas AgentJudgeBench measures*judge*reliability on those generators’ outputs via a paired with\-GT/without\-GT protocol and per\-metric decomposition that FuncBenchGen does not provide\. TaskBench\([Shen et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib12)\)shares the tool\-dependency graph framing but focuses on decomposition quality rather than judge alignment\. Synthetic data pipelines\([Wang et al\., 2023b](https://arxiv.org/html/2608.26623#bib.bib2);[Xu et al\., 2025b](https://arxiv.org/html/2608.26623#bib.bib5);[Cui et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib6)\)inform our generation approach but target model training rather than evaluation benchmarking\.

### 2\.2LLM\-as\-Judge

[Zheng et al\. \(2023\)](https://arxiv.org/html/2608.26623#bib.bib1)established the LLM\-as\-judge paradigm on MT\-Bench, showing GPT\-4’s strong agreement with human preferences on open\-ended dialogue while identifying positional preference, verbosity sensitivity, and self\-enhancement as systematic biases\. Subsequent work either extends the evaluation surface – JudgeBench\([Tan et al\., 2025](https://arxiv.org/html/2608.26623#bib.bib15)\)and Arena\-Hard\-Auto\([Li et al\., 2024b](https://arxiv.org/html/2608.26623#bib.bib16)\)move to hard, verifiable response pairs – or builds dedicated judge models\([Wang et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib17);[Li et al\., 2024a](https://arxiv.org/html/2608.26623#bib.bib18);[Zhu et al\., 2023](https://arxiv.org/html/2608.26623#bib.bib19);[Kim et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib20)\)\. JudgeLM\([Zhu et al\., 2023](https://arxiv.org/html/2608.26623#bib.bib19)\)fine\-tunes 7B–33B judges on GPT\-4\-distilled verdicts and identifies position, knowledge, and format biases that mirror those reported for prompt\-based judges; Prometheus 2\([Kim et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib20)\)adds rubric\-conditioned direct assessment with open weights\. Both target text\-quality scoring rather than structural tool\-calling correctness, but their bias taxonomies inform our prompt\-format ablation \(§[4\.2](https://arxiv.org/html/2608.26623#S4.SS2)\)\. The jury\-of\-judges paradigm\([Verga et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib21)\)shows that ensembling reduces individual judge biases\. Crucially, this line targets text\-centric tasks where judge quality reduces to preference alignment over fluent output\. AgentJudgeBench occupies a different regime: correctness is structural, the verdict space is\{0,0\.5,1\}\\\{0,0\.5,1\\\}rather than a preference ranking, and failure modes are multi\-dimensional and interdependent\.

### 2\.3LLM Judges for Tool\-Calling

LLM judges have been deployed for tool\-calling evaluation in several recent benchmarks, but always as an implementation detail rather than the subject of study\. ToolLLM\([Qin et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib23)\)uses ChatGPT to compute pass rates over API trajectories, reporting 87\.1% human agreement in aggregate; StableToolBench\([Guo et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib24)\)and MCP\-AgentBench\([Guo et al\., 2025](https://arxiv.org/html/2608.26623#bib.bib25)\)adopt similar approaches with updated judge models\. GeoBenchX\([Krechetova and Kochedykov, 2025](https://arxiv.org/html/2608.26623#bib.bib26)\)assembles a three\-judge panel achieving 88–96% agreement, and Agent\-as\-a\-Judge\([Zhuge et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib27)\)and Auto\-Eval Judge\([Bhonsle et al\., 2025](https://arxiv.org/html/2608.26623#bib.bib28)\)propose more structured evaluation frameworks\. In each case, judge reliability is reported as a single aggregate figure on a small sample, with no variation of task complexity, difficulty, or ground\-truth availability\. ToolSandbox\([Lu et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib29)\)notably questions LLM judge reliability directly but replaces judges with programmatic evaluation rather than characterising their failure modes\. AgentJudgeBench takes the complementary stance: keep the judge paradigm and measure it systematically, decomposing reliability by metric, topology, difficulty, and condition across321,648321\{,\}648completed evaluations \(Appendix[D](https://arxiv.org/html/2608.26623#A4)\)\. A detailed feature comparison across all seven related systems appears in Appendix[L](https://arxiv.org/html/2608.26623#A12)\.

## 3Methodology

Figure[1](https://arxiv.org/html/2608.26623#S1.F1)gives an end\-to\-end view of the pipeline\. Each record is expanded into three difficulty variants; every generatorg∈𝒢g\\in\\mathcal\{G\}produces tool\-call predictions; a deterministic programmatic judge scores each prediction to yield a reference vector𝐩r\\mathbf\{p\}\_\{r\}; and every LLM judgej∈𝒥j\\in\\mathcal\{J\}produces paired verdictsℓj,rGT\\boldsymbol\{\\ell\}^\{\\text\{GT\}\}\_\{j,r\},ℓj,rwithout GT\\boldsymbol\{\\ell\}^\{\\text\{without GT\}\}\_\{j,r\}, compared against𝐩r\\mathbf\{p\}\_\{r\}via Eq\.[5a](https://arxiv.org/html/2608.26623#S3.E5.1)\. The full pipeline is implemented as computational graphs onSyGra\([Pradhan et al\., 2025](https://arxiv.org/html/2608.26623#bib.bib30)\), an open\-source graph\-oriented synthetic\-data\-generation framework\.

### 3\.1Data Generation

The benchmark’s BFCL\-style agentic data is constructed through a multi\-stage synthetic pipeline that generates complete agentic records from minimal seed inputs\.

Why synthetic\.We generate records synthetically rather than mining real enterprise traces because the two are not interchangeable for this study’s purpose: a controlled reliability study needs, for every record, a ground\-truth trace we can certify as correct; systematic coverage across six DAG topologies and three difficulty tiers rather than whatever distribution occurs in logs; and enough scale and domain diversity \(15 enterprise domains\) to isolate structural effects from domain idiosyncrasy\. Real enterprise traces satisfying all three properties are difficult to source, requiring proprietary internal tool schemas, live or replayable execution environments, and an independent way to certify trace correctness – rarely available outside a single organisation’s internal systems\. Generating synthetically lets us guarantee a verified reference trace for every one of the3,8083\{,\}808records and control topology and difficulty independently, which is what makes the paired with\-GT/without\-GT protocol \(§[3\.2](https://arxiv.org/html/2608.26623#S3.SS2)\) possible; the Limitations section discusses what this trades off \(domain drift, interactive multi\-turn execution\) and the validation evidence for this design choice\.

Record construction\.Given an enterprise domain label \(e\.g\., IT service management, contract lifecycle management, energy grid operations\), the pipeline generates use\-case scenarios, synthesizes a typed tool inventory as function signatures with constrained return types \(str,bool,list,dict,None\), and produces executable pseudocode linking all tool dependencies\. A natural language user utterance is then paired with function input/output specifications and formalized into a JSON schema with typed arguments, required fields, and output definitions; finally, an ordered execution trace captures sequential and parallel tool calls, their inputs, outputs, and step descriptions\.

DAG topologies\.Records are organized into six topologies derived from common dependency patterns in real enterprise agentic workflows –linear,fan\-out,fan\-in,diamond,optional enrichment, andloop\-like– from single\-path execution to iterative loops\.

Difficulty tiers\.Each record is expanded into three levels \(easy,medium,hard\) by increasing query ambiguity while holding the task structure and GT trace constant\. Rewrite validation \(Appendix[N](https://arxiv.org/html/2608.26623#A14)\) confirms the medium→\\tohard step on93\.9%93\.9\\%of records; the easy→\\tomedium step is unanimously validated on only58\.1%58\.1\\%, and in roughly41\.9%41\.9\\%of cases medium is better characterised as a paraphrase than a strict difficulty increase\. The primary difficulty\-degradation evidence therefore comes from the medium→\\tohard step; medium data should be read as a robustness check rather than a calibrated mid\-point \(see Limitations\)\.

Validation\.GT traces pass a two\-level quality gate: each record is first structurally verified via JSON\-schema validation, argument type checking, and trace consistency checks, then each \(utterance, tool\-call\) pair is programmatically checked for argument sufficiency, grounding alignment, and query naturalness, with failing records regenerated\. No model from𝒥\\mathcal\{J\}, or any other LLM, is involved in either gate check, ruling out self\-reinforcing bias\. A 120\-record human annotation study on hard\-difficulty records \(Appendix[G](https://arxiv.org/html/2608.26623#A7)\), stratified across all six topologies, validates the scorer at92\.7%92\.7\\%metric\-level agreement with independent human judgement \(scope and limitations in the Limitations section\)\. The final corpus comprises3,8083\{,\}808records spanning 15 enterprise domains with 8–19 tools per record, across six topologies and three difficulty levels\.

### 3\.2Evaluation

As shown in Figure[2](https://arxiv.org/html/2608.26623#S3.F2), our evaluation is layered: a deterministic*programmatic judge*computes a reference score on every record, an*LLM judge*produces a per\-metric verdict on the same record, and an*alignment aggregator*compares the two\. Each layer is defined below; symbol definitions appear in Appendix[K](https://arxiv.org/html/2608.26623#A11), and all prompts are reproduced verbatim in Appendix[W](https://arxiv.org/html/2608.26623#A23)\.

Programmatic Judge

Why a deterministic reference\.The programmatic judge, not a human or LLM annotator, serves as the reference signal against which every LLM judge \(and, transitively, the generator\) is measured: using another LLM would reintroduce the same reliability question one level up, and a human\-in\-the\-loop reference cannot scale to the321,648321\{,\}648evaluations in the full factorial grid, whereas a deterministic, rule\-based scorer gives every cell an identical, reproducible reference\. We validate rather than assume this choice: ground\-truth traces pass a two\-level programmatic quality gate with no LLM involved \(below\), and a 120\-record independent human study \(Appendix[G](https://arxiv.org/html/2608.26623#A7)\) directly checks where the scorer’s notion of correctness agrees and disagrees with human judgement\. LetG=\(g1,…,g\|G\|\)G=\(g\_\{1\},\\dots,g\_\{\|G\|\}\)denote the generator’s predicted ordered sequence of tool calls andE=\(e1,…,e\|E\|\)E=\(e\_\{1\},\\dots,e\_\{\|E\|\}\)the GT ordered sequence\. Letname⁡\(⋅\)\\mathrm\{name\}\(\\cdot\)return a tool’s identifier,args⁡\(⋅\)\\mathrm\{args\}\(\\cdot\)its argument\-key set, and let𝒢=\{name⁡\(gi\)\}\\mathcal\{G\}=\\\{\\mathrm\{name\}\(g\_\{i\}\)\\\},ℰ=\{name⁡\(ei\)\}\\mathcal\{E\}=\\\{\\mathrm\{name\}\(e\_\{i\}\)\\\}denote the unordered sets of tool identifiers\. The programmatic judge emits four independent per\-record scores in\[0,1\]\[0,1\]\. Throughout the paper we call each of these four scores – tool selection, parameter structure, sequence accuracy, query coverage – a*metric*: the term denotes one axis of tool\-calling correctness scored independently by both the programmatic judge and the LLM judge, not a distinct evaluation instrument or benchmark\-level metric\.

Tool\-selection accuracyptoolp^\{\\text\{tool\}\}and sequence accuracypseqp^\{\\text\{seq\}\}\.ptoolp^\{\\text\{tool\}\}penalises set\-level mismatch;pseqp^\{\\text\{seq\}\}measures position\-by\-position identity, normalised by expected length:

ptool\\displaystyle p^\{\\text\{tool\}\}=max⁡\(0,1−\|𝒢∖ℰ\|\+\|ℰ∖𝒢\|max⁡\(\|ℰ\|,1\)\)\\displaystyle=\\max\\\!\\left\(0,\\,1\-\\frac\{\|\\mathcal\{G\}\\setminus\\mathcal\{E\}\|\+\|\\mathcal\{E\}\\setminus\\mathcal\{G\}\|\}\{\\max\(\|\\mathcal\{E\}\|,1\)\}\\right\)\(1a\)pseq\\displaystyle p^\{\\text\{seq\}\}=1max⁡\(\|E\|,1\)∑i=1min⁡\(\|G\|,\|E\|\)\[name\(gi\)=name\(ei\)\]\\displaystyle=\\frac\{1\}\{\\max\(\|E\|,1\)\}\\sum\_\{i=1\}^\{\\min\(\|G\|,\|E\|\)\}\\mathbb\{1\}\\\!\\left\[\\mathrm\{name\}\(g\_\{i\}\)=\\mathrm\{name\}\(e\_\{i\}\)\\right\]\(1b\)
withptool=pseq=1p^\{\\text\{tool\}\}=p^\{\\text\{seq\}\}=1when the respective sequences are empty\.

Parameter\-structure accuracypparamp^\{\\text\{param\}\}\.For each predicted callg∈Gg\\in GletAg=args⁡\(g\)A\_\{g\}=\\mathrm\{args\}\(g\)and letAg∗A^\{\*\}\_\{g\}be the expected\-argument key set of the call inEEwith the same tool identifier \(undefined if the predicted tool is not inℰ\\mathcal\{E\}\)\. Define per\-call structural scores

s⁡\(g\)=\{1\.0if​Ag=Ag∗\(all keys present, no extras\)0\.5if​Ag∗⊆Agand​Ag∖Ag∗≠∅\(extras only\)0\.0if​Ag∗⊈Agor​Ag∗​undefined\\displaystyle s\(g\)=\\begin\{cases\}1\.0&\\text\{if \}A\_\{g\}=A^\{\*\}\_\{g\}\\\\\[\-1\.0pt\] &\\quad\\text\{\(all keys present, no extras\)\}\\\\\[4\.0pt\] 0\.5&\\text\{if \}A^\{\*\}\_\{g\}\\subseteq A\_\{g\}\\\\\[\-1\.0pt\] &\\quad\\text\{and \}A\_\{g\}\\setminus A^\{\*\}\_\{g\}\\neq\\emptyset\\\\\[\-1\.0pt\] &\\quad\\text\{\(extras only\)\}\\\\\[4\.0pt\] 0\.0&\\text\{if \}A^\{\*\}\_\{g\}\\not\\subseteq A\_\{g\}\\\\\[\-1\.0pt\] &\\quad\\text\{or \}A^\{\*\}\_\{g\}\\ \\text\{undefined\}\\end\{cases\}\(2\)and aggregate by mean:pparam=\(∑g∈Gs⁡\(g\)\)/max⁡\(\|G\|,1\)p^\{\\text\{param\}\}=\\bigl\(\\sum\_\{g\\in G\}s\(g\)\\bigr\)/\\max\(\|G\|,1\), withpparam=1p^\{\\text\{param\}\}=1whenGGis empty\.

Query\-coverage accuracypcovp^\{\\text\{cov\}\}\.The fraction of expected tool identifiers covered by the prediction, ignoring extras:

pcov=\{1if​ℰ=∅,0if​𝒢∩ℰ=∅,\|𝒢∩ℰ\|/\|ℰ\|otherwise\.p^\{\\text\{cov\}\}\\;=\\;\\begin\{cases\}1&\\text\{if \}\\mathcal\{E\}=\\emptyset,\\\\ 0&\\text\{if \}\\mathcal\{G\}\\cap\\mathcal\{E\}=\\emptyset,\\\\ \\lvert\\mathcal\{G\}\\cap\\mathcal\{E\}\\rvert/\\lvert\\mathcal\{E\}\\rvert&\\text\{otherwise\}\.\\end\{cases\}\(3\)Record\-level aggregate\.Each record receives a four\-element programmatic vector𝐩r=\(prtool,prparam,prseq,prcov\)\\mathbf\{p\}\_\{r\}=\(p^\{\\text\{tool\}\}\_\{r\},p^\{\\text\{param\}\}\_\{r\},p^\{\\text\{seq\}\}\_\{r\},p^\{\\text\{cov\}\}\_\{r\}\)\. Where a scalar is needed, specifically in the generator\-accuracy reporting of Table[2](https://arxiv.org/html/2608.26623#S4.T2), we use the unweighted meanp¯r=1M​∑m=1Mprm\\bar\{p\}\_\{r\}=\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}p^\{m\}\_\{r\}, whereM=4M=4is the number of metrics; elsewhere the four components are kept separate and averaged independently\. We adopt equal weights since the four metrics target orthogonal aspects of tool\-call correctness with no principled basis for preferring one; Section[4\.2](https://arxiv.org/html/2608.26623#S4.SS2)gives per\-metric breakdowns practitioners can reweight for a given application \(e\.g\., upweighting sequence accuracy for strict\-ordering workflows\)\.

Figure 2:Evaluation pipeline of AgentJudgeBench\. Records expand into three difficulty tiers, LLM judges \(with/without GT\) and programmatic judge score the generated tool\-call predictions\.Empirically, the judge ranking is stable under alternative metric weightings: Spearmanρ≥0\.83\\rho\\geq 0\.83222Spearman’sρ\\rhois a rank correlation coefficient measuring the monotone association between two orderings;ρ=1\\rho=1indicates identical rankings,ρ=0\\rho=0no association, andρ=−1\\rho=\-1perfectly reversed rankings\.between the equal\-weight ranking and five alternative schemes \(seq×2\{\\times\}2, param×2\{\\times\}2, cov×0\.5\{\\times\}0\.5, seq×2\+\{\\times\}2\{\+\}param×2\{\\times\}2\), confirming that the equal\-weight aggregate is an adequate proxy for any practitioner\-specific weighting\.

LLM JudgeThe same generator outputs scored by the programmatic judge are independently evaluated by every LLM judgej∈𝒥j\\in\\mathcal\{J\}\. Each judge receives a structured prompt containing the original user query, the full set of available tool schemas, and the generator’s predicted tool\-call sequence, and is asked to produce a single JSON object scoring each of the four metrics \(tool selection, parameter structure, sequence accuracy, query coverage\) on a\{0,0\.5,1\}\\\{0,0\.5,1\\\}scale, along with a one\-sentence justification per metric and an overall assessment\.

Two prompt variants exist, corresponding to the conditionsc∈\{with\-GT,without\-GT\}c\\in\\\{\\text\{with\-GT\},\\text\{without\-GT\}\\\}: thewith\-GTprompt additionally exposes the GT tool\-call sequence as a reference block, while thewithout\-GTprompt omits it entirely, requiring the judge to assess correctness from the query and tool schemas alone\. Judge decoding is held constant across all\(g,j,d,c\)\(g,j,d,c\)configurations to isolate the effect of our independent variables\. The full prompt bodies appear in Appendix[W](https://arxiv.org/html/2608.26623#A23)\.

Alignment Against the Programmatic ReferenceGiven a recordrr, conditioncc, judgejj, and metricm∈\{tool,param,seq,cov\}m\\in\\\{\\text\{tool\},\\text\{param\},\\text\{seq\},\\text\{cov\}\\\}, letℓj,r,cm∈\{0,0\.5,1\}\\ell^\{m\}\_\{j,r,c\}\\in\\\{0,0\.5,1\\\}denote the judge’s verdict andprmp^\{m\}\_\{r\}the programmatic value\. The per\-metric match score and record\-level aggregate are:

μj,r,cm\\displaystyle\\mu^\{m\}\_\{j,r,c\}=1−\|prm−ℓj,r,cm\|\\displaystyle=1\-\\left\|p^\{m\}\_\{r\}\-\\ell^\{m\}\_\{j,r,c\}\\right\|\(4a\)μj,r,c\\displaystyle\\mu\_\{j,r,c\}=1M​∑m=1Mμj,r,cm\\displaystyle=\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}\\mu^\{m\}\_\{j,r,c\}\(4b\)exact match on all metrics yieldsμ=1\\mu=1and maximally divergent verdict yields00\. The configuration\-level alignment percentage and GT lift are:

align⁡\(j,g,d,c\)\\displaystyle\\mathrm\{align\}\(j,g,d,c\)=100Ng,d​∑r=1Ng,dμj,r,c\\displaystyle=\\frac\{100\}\{N\_\{g,d\}\}\\sum\_\{r=1\}^\{N\_\{g,d\}\}\\mu\_\{j,r,c\}\(5a\)lift⁡\(j\)\\displaystyle\\mathrm\{lift\}\(j\)=align¯GT​\(j\)−align¯without GT​\(j\)\\displaystyle=\\overline\{\\mathrm\{align\}\}\_\{\\text\{GT\}\}\(j\)\-\\overline\{\\mathrm\{align\}\}\_\{\\text\{without GT\}\}\(j\)\(5b\)where the overline denotes the mean over all 12\(g,d\)\(g,d\)configurations\. Perfect reproduction of the programmatic vector yieldsalign=100%\\mathrm\{align\}=100\\%; an independent judge attains50%50\\%in expectation\.

In the experiments, we report both aggregate alignmentalign⁡\(j,g,d,c\)\\mathrm\{align\}\(j,g,d,c\)and per\-metric breakdowns to identify which dimensions of tool\-calling correctness judges find most difficult to assess\.

## 4Experiments

We use AgentJudgeBench to evaluate the extent to which LLM\-judge alignment depends on the generator, query difficulty, and ground\-truth availability\. Protocol details \(prompts, scoring, conditions\) are in Section[3](https://arxiv.org/html/2608.26623#S3); model identifiers are in Appendix[M](https://arxiv.org/html/2608.26623#A13)and Appendix[P](https://arxiv.org/html/2608.26623#A16)\.

### 4\.1Experimental Setup

We instantiate a fully\-crossed factorial over three factors: generatorg∈𝒢g\\in\\mathcal\{G\}\(five models, 3B to frontier\), judgej∈𝒥j\\in\\mathcal\{J\}\(six LLMs\), and difficultyd∈\{easy,medium,hard\}d\\in\\\{\\text\{easy\},\\text\{medium\},\\text\{hard\}\\\}\. Each cell is observed under both with GT and without GT conditions, yielding 90 factorial cells and321,648321\{,\}648valid \(generator, judge, difficulty, record\) tuples\. Exact per\-cell counts are in Appendix[D](https://arxiv.org/html/2608.26623#A4)\. We reportalign⁡\(j,g,d,c\)\\mathrm\{align\}\(j,g,d,c\)\(Eq\.[5a](https://arxiv.org/html/2608.26623#S3.E5.1)\) and, where relevant, GT lift \(Eq\.[5b](https://arxiv.org/html/2608.26623#S3.E5.2)\)\.

### 4\.2Results and Analysis

Table[2](https://arxiv.org/html/2608.26623#S4.T2)reports every configuration in the factorial design\. Each generator occupies a sub\-block with six rows corresponding to the judges in𝒥\\mathcal\{J\}, plus a seventh row for Prometheus\-2\([Kim et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib20)\), a judge\-specialised model added as a baseline across all five generators and both GT conditions \(Limitations\), shown for reference and excluded from the six\-judge statistics below\. Columns pair each difficulty level with its with\-GT and without\-GT conditions; bold entries mark the highest alignment within each \(difficulty, condition\) column of each sub\-block\.

Finding 1: Monotone difficulty degradation\.All 30 \(generator, judge\) pairs exhibit strictly monotone alignment degradation from easy to hard under both conditions \(Table[2](https://arxiv.org/html/2608.26623#S4.T2)\), without\-GT degradation roughly1\.5×1\.5\\timeslarger than with\-GT\. On hard without\-GT records, all six judges converge to a narrow7777–82%82\\%band across four of five generators \(including frontier GPT\-5\.4\), indicating a task\-level rather than judge\-level ceiling; degradation curves are in Appendix[F](https://arxiv.org/html/2608.26623#A6)\.

Table 1:Full C3 results on Llama\-3\.3\-70B generator\. All values are mean alignment \(%\) over 3,764–3,771 records per cell\.cGT: corrupted\-GT condition\.Gemini: Gemini 2\.5 Pro\.ΔcGT\-GT\\Delta\_\{\\text\{cGT\-GT\}\}: corrupted\-GT minus standard GT\.ΔcGT\-noGT\\Delta\_\{\\text\{cGT\-noGT\}\}: corrupted\-GT minus without GT\.Scope of Finding 1\.The medium→\\tohard drop \(≈5\{\\approx\}5–77pp with GT\) is the primary evidence for difficulty\-driven degradation; hard rewrites are unanimously validated on93\.9%93\.9\\%of records \(Appendix[N](https://arxiv.org/html/2608.26623#A14)\), while the easy→\\tomedium step \(unanimous on58\.1%58\.1\\%\) partially reflects paraphrase\-robustness\.

Finding 2: Ground\-truth exposure is not monotonically beneficial\.GT lift \(Eq\.[5b](https://arxiv.org/html/2608.26623#S3.E5.2)\) is positive for QwQ\-32B and GPT\-OSS\-120B, but negative for GPT\-5\.4 and Gemini\-2\.5\-Pro \(non\-overlapping bootstrap CIs; Appendix[E](https://arxiv.org/html/2608.26623#A5)\)\. Figure[3](https://arxiv.org/html/2608.26623#S4.F3)shows the effect concentrates in sequence accuracy: frontier judges anchor to the GT trace’s ordering and penalise functionally equivalent but structurally deviant sequences \(case studies in Appendix[H](https://arxiv.org/html/2608.26623#A8)\)\.

Table 2:LLM–judge alignment \(%\) with the programmatic reference across generators, judges, difficulty, and GT condition\. Bold = column\-wise maximum within each generator block among the six main judges\. Prom2 = Prometheus\-2, a judge\-specialised baseline \(Limitations\); shown for reference and not counted toward bold column\-wise maxima or six\-judge statistics\.C3 control\.Replacing the reference with a wrong GT from a different record confirms pure anchoring \(Table[1](https://arxiv.org/html/2608.26623#S4.T1)\): Gemini\-2\.5\-Pro aligns identically under standard and corrupted GT, while QwQ\-32B tracks without\-GT within0\.20\.2pp\. Full results are in Appendix[J](https://arxiv.org/html/2608.26623#A10)\.

Figure 3:Per\-metric alignment \(with and without GT\) across the six main judges, plus Prometheus\-2 \(judge\-specialised baseline; Limitations\), shown for reference\.22footnotetext:Cohen’sκ\\kappameasures inter\-rater agreement corrected for chance: \(00=chance,11=perfect;0\.20\.2–0\.40\.4“fair,”0\.40\.4–0\.60\.6“moderate,”≥0\.6\{\\geq\}0\.6“substantial”\) agreement\. Allκ\\kappavalues here fall in the fair\-to\-substantial range \(0\.2250\.225–0\.6060\.606\), consistent with the structural complexity of the four\-metric scoring task\.Finding 3: Best judge depends on configuration\.QwQ\-32B leads GT alignment in 10 of 15 \(generator, difficulty\) cells but is never the top without\-GT judge; Gemini\-2\.5\-Pro and GPT\-5\.4 lead without\-GT on stronger and weaker generators, respectively\. See Appendix[A](https://arxiv.org/html/2608.26623#A1)for a deployment decision table\. The following research questions explain the mechanisms behind Findings 1–3\.

RQ1: Which metrics and DAG topologies are hardest for judges?

Figure[3](https://arxiv.org/html/2608.26623#S4.F3)breaks alignment into per\-metric components: with GT, sequence accuracy is the weakest dimension with substantial inter\-judge spread, while without GT all judges compress to a narrow band, reflecting the1\.01\.0\-default rubric\. DAG topology imposes a judge\-independent difficulty ordering \(*fan\-out*easiest,*loop\-like*/*fan\-in*hardest\); full breakdowns are in Appendices[Q](https://arxiv.org/html/2608.26623#A17)–[S](https://arxiv.org/html/2608.26623#A19)\.

RQ2: How much do judges agree with each other?

Table[5](https://arxiv.org/html/2608.26623#footnote5)reports pairwise judge agreement\. Mean agreement is79\.1%79\.1\\%\(κ=0\.419\\kappa=0\.419\) with GT and92\.6%92\.6\\%\(κ=0\.559\\kappa=0\.559\) without GT; the higher without\-GT value reflects prompt\-driven verdict compression rather than genuine consensus\. The highest agreement is QwQ\-32B×\\timesGPT\-OSS\-120B \(89\.4%89\.4\\%,κ=0\.606\\kappa=0\.606\), and the lowest is Claude Sonnet 4\.5×\\timesGPT\-OSS\-20B \(70\.4%70\.4\\%,κ=0\.225\\kappa=0\.225\)\. Under without\-GT, pairwiseκ\\kappadecreases monotonically with judge tier separation \(ρ=−0\.825\\rho=\-0\.825333Spearman’sρ\\rhomeasures the rank correlation between the pairwise judge tier gap \(by parameter count\) and theirκ\\kappaagreement score;\|ρ\|\>0\.7\|\\rho\|\>0\.7indicates a strong monotone relation\.,p<0\.01p<0\.01\); this relationship vanishes under GT \(ρ=−0\.171\\rho=\-0\.171\), where GT acts as a shared anchor\. Disagreement concentrates at the0\.50\.5partial\-credit boundary \(9494–97%97\\%of off\-diagonal entries\); verdict\-level confusion matrices are in Appendix[T](https://arxiv.org/html/2608.26623#A20)and a binary scale ablation is in Appendix[T](https://arxiv.org/html/2608.26623#A20.SSx1)\. Prometheus\-2 agrees with every judge at close to chance level under both conditions \(4242–52%52\\%exact,κ=0\.01\\kappa=0\.01–0\.070\.07; Table[5](https://arxiv.org/html/2608.26623#footnote5)\), well below the fair\-to\-substantial range spanned by the six main judges, indicating it forms its own bloc rather than a lower\-alignment member of the general\-purpose cluster \(Limitations\)\.

Table 3:Pairwise inter\-judge agreement under the with GT condition\.Upper triangle: exact metric\-level agreement \(%\);Lower triangle: Cohen’sκ\\kappa55footnotemark:5\. Prom2 \(Prometheus\-2, judge\-specialised baseline\) row/column added for reference; excluded from the six\-judge mean agreement statistics discussed in the text\.A soft jury of all six judges matches but does not exceed the top individual judge \(82\.5%82\.5\\%with GT hard\)\([Verga et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib21)\); ensemble and individual rankings against human annotators are in Appendix[G](https://arxiv.org/html/2608.26623#A7)\.

RQ3: Why does without\-GT alignment converge on hard queries?

The7777–82%82\\%hard without\-GT convergence \(Table[2](https://arxiv.org/html/2608.26623#S4.T2)\) has three consistent explanations: \(1\) the without\-GT prompt defaults to1\.01\.0, suppressing hard\-query discrimination; \(2\) generator error peaks on hard records, compressing all judges identically; and \(3\) per\-metric compression is uniform \(Table[24](https://arxiv.org/html/2608.26623#A17.T24)\)\. The six\-judge ensemble achieves79\.5%79\.5\\%, matching the best individual judge \(GPT\-5\.4,79\.8%79\.8\\%\) to within0\.40\.4pp, confirming correlated, structural failure rather than independent per\-judge noise\.

Table 4:Hard without GT alignment \(%\) under the standard 1\.0\-default and the recalibrated 0\.5\-default prompts, averaged across six judges per generator\.Δ\\Delta= \(0\.5\-default\)−\-\(1\.0\-default\)\.C2 ablation: 0\.5\-default without\-GT prompt\.Table[4](https://arxiv.org/html/2608.26623#S4.T4)compares hard without\-GT alignment under the standard1\.01\.0\-default and a recalibrated0\.50\.5\-default prompt\. For the three strongest generators the delta is≤\+1\.0\{\\leq\}\{\+1\.0\}pp, confirming structural task difficulty as the primary ceiling driver\. For weaker generators \(\+4\.1\+4\.1–\+5\.6\+5\.6pp\), the1\.01\.0\-default over\-credits incorrect outputs; practitioners evaluating low\-quality generators should consider the0\.50\.5\-default prompt\.

RQ4: Does judge temperature affect alignment?

Figure[4](https://arxiv.org/html/2608.26623#S4.F4)shows that Qwen3\-32B alignment is insensitive to temperature acrossT∈\{0\.3,0\.7,1\.0\}T\\in\\\{0\.3,0\.7,1\.0\\\}, with a maximum spread of0\.60\.6pp across all \(difficulty, condition\) cells; structural pattern\-matching dominates \(full table in Appendix[R](https://arxiv.org/html/2608.26623#A18)\)\. A second \(judge, generator\) pairing \(GPT\-OSS\-120B on Llama\-3\.1\-8B\-Instruct\) confirms this, with a maximum spread of0\.250\.25pp across all three difficulty tiers, even tighter than the original pairing, indicating that temperature insensitivity is not an artefact of the single test\-bed cell\.

Figure 4:Judge temperature sensitivity\.Qwen3\-32B alignment atT∈\{0\.3,0\.7,1\.0\}T\\in\\\{0\.3,0\.7,1\.0\\\}on Llama\-3\.3\-70B\. All three difficulty curves are near\-flat; maximum spread≤0\.6\\leq 0\.6pp\.Figure 5:Prompt structure vs\. alignment\.Structured JSON prompt vs\. free\-form for Qwen3\-32B on Llama\-3\.3\-70B\. The per\-metric rubric adds\+4\.8\+4\.8\-\+6\.5\+6\.5pp with GT across all difficulty levels\.RQ5: Does chain\-of\-thought reasoning in the judge improve alignment?

Table[5](https://arxiv.org/html/2608.26623#S4.T5)reports QwQ\-32B alignment with thinking on vs\. off across four open\-weight generators and three difficulty levels \(24 paired cells\)\. CoT reasoning contributes negligibly, with a mean GT gap of\+0\.11\+0\.11pp and a maximum per\-cell difference of0\.30\.3pp; no cell exceeds0\.30\.3pp on either condition\. QwQ\-32B’s advantage in the main results therefore reflects training distribution rather than inference\-time compute\. The full per\-generator grid is reproduced in Appendix[R](https://arxiv.org/html/2608.26623#A18)\.

Table 5:Chain\-of\-thought reasoning in the QwQ\-32B judge: alignment \(%\) with thinking on vs\. off across four open\-weight generators\.Δ\\Delta= thinking\-on−\-thinking\-off \(pp\)\.RQ6: Does prompt output format affect alignment?

Figure[5](https://arxiv.org/html/2608.26623#S4.F5)shows that the structured per\-metric JSON prompt \(verbatim in Appendix[W\.2](https://arxiv.org/html/2608.26623#A23.SS2)/[W\.3](https://arxiv.org/html/2608.26623#A23.SS3)\) outperforms a free\-form variant \(Appendix[X](https://arxiv.org/html/2608.26623#A24)\) by\+4\.8\+4\.8–\+6\.5\+6\.5pp with GT across all difficulty levels on the original \(Qwen3\-32B, Llama\-3\.3\-70B\) pairing, the largest lever we test among the configuration choices in this study\. A second pairing \(QwQ\-32B on SmolLM3\-3B\) partially replicates this: structured format still wins on easy \(\+3\.9\+3\.9pp\) and medium \(\+2\.4\+2\.4pp\), but the effect shrinks relative to the first pairing and*reverses*on hard \(−0\.8\-0\.8pp, free\-form marginally ahead\)\. We therefore do not treat prompt format as a uniformly dominant, difficulty\-independent lever: the effect is real and judge/generator\-dependent, largest on easier queries, and not guaranteed to generalise in direction on hard queries for every pairing\. Full tables for both pairings are in Appendix[R](https://arxiv.org/html/2608.26623#A18)\.

## 5Conclusion

We introduceAgentJudgeBench, a benchmark measuring LLM\-judge reliability on structured, dependency\-driven tool\-calling, where existing LLM\-as\-judge work offers little calibration, via three design choices: difficulty\-stratified rewrites \(easy/medium/hard, preserving the ground\-truth trace\), per\-record DAG\-topology annotation across six patterns, and a paired with\-GT/without\-GT evaluation protocol\. Across321,648321\{,\}648completed evaluations, four patterns emerge\. First, alignment degrades monotonically with difficulty, approximately1\.5×1\.5\\timesfaster without ground truth than with it\. Second, hard without\-GT alignment converges to a7777–82%82\\%band regardless of judge capacity, a task\-level ceiling confirmed by the C2 recalibrated\-prompt ablation \(≤\+1\.0\{\\leq\}\{\+1\.0\}pp for capable generators\)\. Third, GT exposure is counterproductive for frontier judges \(Gemini\-2\.5\-Pro:−3\.9\-3\.9pp; GPT\-5\.4:−1\.5\-1\.5pp\), consistent with over\-anchoring\. Fourth, prompt structure is the largest configuration lever \(\+4\.8\+4\.8–\+6\.5\+6\.5pp\) but is judge/generator\-dependent, while CoT reasoning and temperature are negligible \(§[4\.2](https://arxiv.org/html/2608.26623#S4.SS2)\)\.With ground truth, QwQ\-32B agrees most closely with the programmatic reference and GPT\-OSS\-120B with human judgement; without it, frontier judges lead only narrowly\.Limitations and deployment guidance are in the Limitations section and Appendix[A](https://arxiv.org/html/2608.26623#A1)\.

## Supplementary Material

The Hugging Face dataset \(link, page 1\) contains all3,8083\{,\}808records, generator outputs, and all seven judges’ with\-GT/without\-GT verdicts underlying every table and figure\. The SyGra code release \(link, page 1\) contains the pipeline implementation – data generation, difficulty rewriting, the programmatic judge \(Eqs\.[1a](https://arxiv.org/html/2608.26623#S3.E1.1)\-[5a](https://arxiv.org/html/2608.26623#S3.E5.1)\), and the LLM\-judge runner with full prompt templates \(Appendix[W](https://arxiv.org/html/2608.26623#A23)\) – and a reproducibility script for the main results, under a permissive open\-source license\.

## Limitations

Four of five generators are open\-weight; GPT\-5\.4 is a non\-reproducible Azure snapshot \(Appendix[P](https://arxiv.org/html/2608.26623#A16)\) that also serves as judge and rewrite meta\-judge\. All primary claims replicate on the open\-weight generators independently of it; as a generator, GPT\-5\.4’s self\-bias when judged stays within range on other generators except sequence accuracy \(\+0\.172\+0\.172vs\.≤\+0\.008\\leq\+0\.008\), a post\-hoc correlational signal we cannot rule out and, being non\-reproducible, cannot independently re\-verify \(Appendix[B](https://arxiv.org/html/2608.26623#A2)\)\.

We use a deterministic programmatic scorer, rather than a human or LLM annotator, as the reference signal, since only it scales to321,648321\{,\}648evaluations without reintroducing the reliability question under study\. A 120\-record single\-annotator human study \(Appendix[G](https://arxiv.org/html/2608.26623#A7)\) validates it at92\.592\.5–98\.3%98\.3\\%agreement on three of four metrics but only82\.5%82\.5\\%on parameter structure, where the scorer penalises schema\-valid extra keys that annotators accept \(99of2121tool\-selection disagreements reflect an analogous tool\-redundancy gap\); this does not drive our findings \(ranking shifts≤0\.09\\leq 0\.09pp under a maximally generous correction\), though headline*absolute*numbers still reflect the current scoring rule, so we report both programmatic and human\-verdict rankings for best\-judge claims\. Extended discussion is in Appendix[B](https://arxiv.org/html/2608.26623#A2)\.

Records are synthetically generated rather than mined from enterprise traces, since a verified per\-record reference – required by our paired GT/no\-GT protocol – is not obtainable at this scale from proprietary systems; a two\-level quality gate and the same human study \(92\.7%92\.7\\%agreement\) validate the pipeline, but domain drift remains an open concern \(Appendix[B](https://arxiv.org/html/2608.26623#A2)\)\. Our six main judges are also all general\-purpose: adding Prometheus\-2\([Kim et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib20)\)as a judge\-specialised baseline shows it does not cluster with them \(2020–3030pp lower alignment; Table[5](https://arxiv.org/html/2608.26623#footnote5)\) and it is excluded from six\-judge statistics \(Appendix[B](https://arxiv.org/html/2608.26623#A2)\)\.

The RQ4/RQ6 ablations were each extended to a second \(judge, generator\) pairing: temperature insensitivity replicates tightly, but the prompt\-format advantage only partially replicates and reverses on hard queries, so we treat format as judge/generator\-dependent rather than uniformly dominant\. Difficulty rewriting is unanimously validated on58\.1%58\.1\\%of easy→\\tomedium and93\.9%93\.9\\%of medium→\\tohard records, so medium should be read as a robustness check rather than a fully calibrated tier\. The without\-GT ceiling is also partly prompt\-dependent: a C2 ablation \(Table[4](https://arxiv.org/html/2608.26623#S4.T4)\) shifts it≤\+1\.0\\leq\{\+1\.0\}pp under an alternative0\.50\.5\-default prompt for the three strongest generators but\+4\.1\+4\.1–\+5\.6\+5\.6pp for the two weakest, so task difficulty is the primary ceiling driver only for capable generators \(Appendix[B](https://arxiv.org/html/2608.26623#A2)\)\.

Finally, we study judge reliability at evaluation time only and do not test whether these failure modes carry over as a training signal \(e\.g\., a reward model or model\-selection gate\); we reason through three findings under this framing – GT\-exposure over\-anchoring, the without\-GT verdict\-compression ceiling, and correlated cross\-judge failure – and flag it as concrete future work in Appendix[B](https://arxiv.org/html/2608.26623#A2)\.

## Ethical Considerations

Over\-reliance on automated evaluation\.Our findings reveal systematic failure modes in LLM judges, including the 77–82% without\-GT ceiling and over\-anchoring in frontier models; practitioners unaware of these limitations risk certifying incorrect tool\-calling outputs as correct, particularly in safety\-critical domains \(e\.g\., energy grid, healthcare\) represented in our dataset\. We mitigate this via deployment guidance \(Appendix[A](https://arxiv.org/html/2608.26623#A1)\) and by recommending against sole reliance on any single judge\.

Evaluation monoculture\.All six judges converge to similar failure patterns on hard queries, suggesting shared training\-distribution biases; over\-reliance on our rankings could reinforce a monoculture where the same blind spots propagate across pipelines, so we encourage complementing LLM judges with programmatic scoring and human review\.

Dual use, synthetic data, and compute\.The detailed failure\-mode analysis \(over\-anchoring, the 1\.0\-default bias, prompt\-format sensitivity\) could be exploited to game judge\-based evaluation; we release all prompts, scorer code, and raw outputs to enable countermeasures\. All records are synthetically generated – avoiding real\-user privacy concerns but inheriting the generation pipeline’s biases and possibly under\-representing non\-English or marginalised workflows\. The full evaluation \(321,648 paired judge calls\) is compute\-intensive; our decision guide \(Appendix[A](https://arxiv.org/html/2608.26623#A1)\) reduces unnecessary evaluation by recommending specific judges per scenario\.

## Acknowledgements

We are especially grateful to Sai Rajeswar for consistently supporting research efforts like this one, and for generously taking the time to review the paper and help improve it whenever needed\.

Generative AI tools \(Claude\) were used for language polishing and proofreading of author\-written text; all research ideas, experiments, analysis, and writing are the authors’ own\.

## References

- R\. Bhonsle, R\. Dutta, S\. Vavilapalli, H\. Seth, A\. Jaye, Y\. Chang, M\. Rungta, E\. A\. Boateng, S\. Hasan, E\. Nosakhare, and S\. SrinivasanAuto\-Eval Judge: towards a general agentic framework for task completion evaluation\.arXiv preprint arXiv:2508\.05508\.Cited by:[Table 17](https://arxiv.org/html/2608.26623#A12.T17.2.1.7.1),[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.29.1),[§2\.3](https://arxiv.org/html/2608.26623#S2.SS3.p1.1)\.
- Cuiet al\.\(2024\)G\. Cui, L\. Yuan, N\. Ding, G\. Yao, W\. Zhu, Y\. Ni, G\. Xie, Z\. Liu, and M\. SunUltraFeedback: boosting language models with scaled AI feedback\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Cited by:[§2\.1](https://arxiv.org/html/2608.26623#S2.SS1.p1.1)\.
- Denget al\.\(2023\)X\. Deng, Y\. Gu, B\. Zheng, S\. Chen, S\. Stevens, B\. Wang, H\. Sun, and Y\. SuMind2Web: towards a generalist agent for the web\.InAdvances in Neural Information Processing Systems,Cited by:[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.4.1),[§2\.1](https://arxiv.org/html/2608.26623#S2.SS1.p1.1)\.
- Drouinet al\.\(2024\)A\. Drouin, M\. Gasse, M\. Caccia, I\. H\. Laradji, M\. D\. Verme, T\. Marty, L\. Boisvert, M\. Thakkar, Q\. Cappart, D\. Vazquez, N\. Chapados, and A\. LacosteWorkArena: how capable are web agents at solving common knowledge work tasks?\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),pp\. 11642–11662\.Cited by:[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.7.1),[§2\.1](https://arxiv.org/html/2608.26623#S2.SS1.p1.1)\.
- Guoet al\.\(2024\)Z\. Guo, S\. Cheng, H\. Wang, S\. Liang, Y\. Qin, P\. Li, Z\. Liu, M\. Sun, and Y\. LiuStableToolBench: towards stable large\-scale benchmarking on tool learning of LLMs\.arXiv preprint arXiv:2403\.07714\.Cited by:[Table 17](https://arxiv.org/html/2608.26623#A12.T17.2.1.3.1),[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.24.1),[§1](https://arxiv.org/html/2608.26623#S1.p1.1),[§1](https://arxiv.org/html/2608.26623#S1.p2.1),[§2\.3](https://arxiv.org/html/2608.26623#S2.SS3.p1.1)\.
- Guoet al\.\(2025\)Z\. Guo, B\. Xu, C\. Zhu, W\. Hong, X\. Wang, and Z\. MaoMCP\-AgentBench: evaluating real\-world language agent performance with MCP\-mediated tools\.arXiv preprint arXiv:2509\.09734\.Cited by:[Table 17](https://arxiv.org/html/2608.26623#A12.T17.2.1.4.1),[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.26.1),[§1](https://arxiv.org/html/2608.26623#S1.p1.1),[§1](https://arxiv.org/html/2608.26623#S1.p2.1),[§2\.3](https://arxiv.org/html/2608.26623#S2.SS3.p1.1)\.
- Kimet al\.\(2024\)S\. Kim, J\. Suk, S\. Longpre, B\. Y\. Lin, J\. Shin, S\. Welleck, G\. Neubig, M\. Lee, K\. Lee, and M\. SeoPrometheus 2: an open source language model specialized in evaluating other language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),External Links:2405\.01535,[Link](https://arxiv.org/abs/2405.01535)Cited by:[Appendix B](https://arxiv.org/html/2608.26623#A2.p5.1),[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.21.1),[§W\.4](https://arxiv.org/html/2608.26623#A23.SS4.p1.1),[§2\.2](https://arxiv.org/html/2608.26623#S2.SS2.p1.1),[§4\.2](https://arxiv.org/html/2608.26623#S4.SS2.p1.1),[Limitations](https://arxiv.org/html/2608.26623#Sx2.p3.1)\.
- Krechetova and Kochedykov \(2025\)V\. Krechetova and D\. KochedykovGeoBenchX: benchmarking LLMs in agent solving multistep geospatial tasks\.InProceedings of the 1st ACM SIGSPATIAL International Workshop on Generative and Agentic AI for Multi\-Modality Space\-Time Intelligence,Cited by:[Table 17](https://arxiv.org/html/2608.26623#A12.T17.2.1.5.1),[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.27.1),[§2\.3](https://arxiv.org/html/2608.26623#S2.SS3.p1.1)\.
- Liet al\.\(2024a\)J\. Li, S\. Sun, W\. Yuan, R\. Fan, H\. Zhao, and P\. LiuGenerative judge for evaluating alignment\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.19.1),[§2\.2](https://arxiv.org/html/2608.26623#S2.SS2.p1.1)\.
- Liet al\.\(2024b\)T\. Li, W\. Chiang, E\. Frick, L\. Dunlap, T\. Wu, B\. Zhu, J\. E\. Gonzalez, and I\. StoicaFrom crowdsourced data to high\-quality benchmarks: Arena\-Hard and BenchBuilder pipeline\.arXiv preprint arXiv:2406\.11939\.Cited by:[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.17.1),[§1](https://arxiv.org/html/2608.26623#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.26623#S2.SS2.p1.1)\.
- Liuet al\.\(2024\)X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang, S\. Zhang, X\. Deng, A\. Zeng, Z\. Du, C\. Zhang, S\. Shen, T\. Zhang, Y\. Su, H\. Sun, M\. Huang, Y\. Dong, and J\. TangAgentBench: evaluating LLMs as agents\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.5.1)\.
- Luet al\.\(2024\)J\. Lu, T\. Holleis, Y\. Zhang, B\. Aumayer, F\. Nan, F\. Bai, S\. Ma, S\. Ma, M\. Li, G\. Yin, Z\. Wang, and R\. PangToolSandbox: a stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities\.arXiv preprint arXiv:2408\.04682\.Cited by:[Table 17](https://arxiv.org/html/2608.26623#A12.T17.2.1.8.1),[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.25.1),[§2\.3](https://arxiv.org/html/2608.26623#S2.SS3.p1.1)\.
- Maekawaet al\.\(2025\)S\. Maekawa, J\. Hassell, P\. Pezeshkpour, T\. Mitchell, and E\. HruschkaTowards reliable benchmarking: a contamination\-free, controllable evaluation framework for multi\-step LLM function calling\.arXiv preprint arXiv:2509\.26553\.Cited by:[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.12.1),[§2\.1](https://arxiv.org/html/2608.26623#S2.SS1.p1.1)\.
- Patilet al\.\(2025\)S\. G\. Patil, H\. Mao, C\. C\. Ji, F\. Yan, V\. Suresh, I\. Stoica, and J\. E\. GonzalezThe Berkeley Function Calling Leaderboard \(BFCL\): from tool use to agentic evaluation of large language models\.InProceedings of the 42nd International Conference on Machine Learning \(ICML\),Cited by:[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.11.1),[§1](https://arxiv.org/html/2608.26623#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.26623#S2.SS1.p1.1)\.
- Pradhanet al\.\(2025\)B\. Pradhan, S\. Dasgupta, A\. K\. Saha, O\. Anustoop, S\. Puttagunta, V\. Mittal, and S\. GopalSyGra: an open\-source graph\-oriented synthetic data generation pipeline\.arXiv preprint arXiv:2508\.15432\.External Links:[Link](https://github.com/ServiceNow/SyGra)Cited by:[§3](https://arxiv.org/html/2608.26623#S3.p1.1)\.
- Qinet al\.\(2024\)Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian, S\. Zhao, R\. Tian, R\. Xie, J\. Zhou, M\. Gerstein, D\. Li, Z\. Liu, and M\. SunToolLLM: facilitating large language models to master 16000\+ real\-world APIs\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[Table 17](https://arxiv.org/html/2608.26623#A12.T17.2.1.2.1),[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.23.1),[§1](https://arxiv.org/html/2608.26623#S1.p1.1),[§1](https://arxiv.org/html/2608.26623#S1.p2.1),[§2\.3](https://arxiv.org/html/2608.26623#S2.SS3.p1.1)\.
- Shenet al\.\(2024\)Y\. Shen, K\. Song, X\. Tan, W\. Zhang, K\. Ren, S\. Yuan, W\. Lu, D\. Li, and Y\. ZhuangTaskBench: benchmarking large language models for task automation\.InAdvances in Neural Information Processing Systems,Cited by:[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.9.1),[§2\.1](https://arxiv.org/html/2608.26623#S2.SS1.p1.1)\.
- Tanet al\.\(2025\)S\. Tan, S\. Zhuang, K\. Montgomery, W\. Y\. Tang, A\. Cuadron, C\. Wang, R\. A\. Popa, and I\. StoicaJudgeBench: a benchmark for evaluating LLM\-based judges\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),Cited by:[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.16.1),[§1](https://arxiv.org/html/2608.26623#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.26623#S2.SS2.p1.1)\.
- Trivediet al\.\(2024\)H\. Trivedi, T\. Khot, M\. Hartmann, R\. Manku, V\. Dong, E\. Li, S\. Gupta, A\. Sabharwal, and N\. BalasubramanianAppWorld: a controllable world of apps and people for benchmarking interactive coding agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Cited by:[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.8.1),[§2\.1](https://arxiv.org/html/2608.26623#S2.SS1.p1.1)\.
- Vergaet al\.\(2024\)P\. Verga, S\. Hofstätter, S\. Althammer, Y\. Su, A\. Piktus, A\. Arkhangorodsky, M\. Xu, N\. White, and P\. LewisReplacing judges with juries: evaluating LLM generations with a panel of diverse models\.arXiv preprint arXiv:2404\.18796\.Cited by:[§2\.2](https://arxiv.org/html/2608.26623#S2.SS2.p1.1),[§4\.2](https://arxiv.org/html/2608.26623#S4.SS2.p11.1)\.
- Wanget al\.\(2023a\)P\. Wang, L\. Li, L\. Chen, D\. Zhu, B\. Lin, Y\. Cao, Q\. Liu, T\. Liu, and Z\. SuiLarge language models are not fair evaluators\.arXiv preprint arXiv:2305\.17926\.Cited by:[§1](https://arxiv.org/html/2608.26623#S1.p1.1)\.
- Wanget al\.\(2024\)Y\. Wang, Z\. Yu, Z\. Zeng, L\. Yang, C\. Wang, H\. Chen, C\. Jiang, R\. Xie, J\. Wang, X\. Xie, W\. Ye, S\. Zhang, and Y\. ZhangPandaLM: an automatic evaluation benchmark for LLM instruction tuning optimization\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.18.1),[§2\.2](https://arxiv.org/html/2608.26623#S2.SS2.p1.1)\.
- Wanget al\.\(2023b\)Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. HajishirziSelf\-instruct: aligning language models with self\-generated instructions\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 13484–13508\.Cited by:[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.3.1),[§2\.1](https://arxiv.org/html/2608.26623#S2.SS1.p1.1)\.
- Xuet al\.\(2025a\)Y\. Xu, D\. Lu, Z\. Shen, J\. Wang, Z\. Wang, Y\. Mao, C\. Xiong, and T\. YuAgentTrek: agent trajectory synthesis via guiding replay with web tutorials\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),Cited by:[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.10.1),[§2\.1](https://arxiv.org/html/2608.26623#S2.SS1.p1.1)\.
- Xuet al\.\(2025b\)Z\. Xu, F\. Jiang, L\. Niu, Y\. Deng, R\. Poovendran, Y\. Choi, and B\. Y\. LinMagpie: alignment data synthesis from scratch by prompting aligned LLMs with nothing\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),Cited by:[§2\.1](https://arxiv.org/html/2608.26623#S2.SS1.p1.1)\.
- Yaoet al\.\(2024\)S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhanτ\\tau\-bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv preprint arXiv:2406\.12045\.Cited by:[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.13.1),[§2\.1](https://arxiv.org/html/2608.26623#S2.SS1.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems,Cited by:[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.15.1),[§1](https://arxiv.org/html/2608.26623#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.26623#S2.SS2.p1.1)\.
- Zhouet al\.\(2024\)S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. NeubigWebArena: a realistic web environment for building autonomous agents\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.6.1),[§2\.1](https://arxiv.org/html/2608.26623#S2.SS1.p1.1)\.
- Zhuet al\.\(2023\)L\. Zhu, X\. Wang, and X\. WangJudgeLM: fine\-tuned large language models are scalable judges\.arXiv preprint arXiv:2310\.17631\.Cited by:[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.20.1),[§2\.2](https://arxiv.org/html/2608.26623#S2.SS2.p1.1)\.
- Zhugeet al\.\(2024\)M\. Zhuge, C\. Zhao, D\. Ashley, W\. Wang, D\. Khizbullin, Y\. Xiong, Z\. Liu, E\. Chang, R\. Krishnamoorthi, Y\. Tian, Y\. Shi, V\. Chandra, and J\. SchmidhuberAgent\-as\-a\-judge: evaluate agents with agents\.arXiv preprint arXiv:2410\.10934\.Cited by:[Table 17](https://arxiv.org/html/2608.26623#A12.T17.2.1.6.1),[Table 30](https://arxiv.org/html/2608.26623#A21.T30.2.1.28.1),[§2\.3](https://arxiv.org/html/2608.26623#S2.SS3.p1.1)\.

## Appendix APractitioner Decision Guide

In most deployment scenarios ground\-truth tool\-call sequences are unavailable at inference time; the without GT judge is therefore the practical baseline\. Table[6](https://arxiv.org/html/2608.26623#A1.T6)reports the best\-performing judge for each deployment scenario, derived directly from Table[2](https://arxiv.org/html/2608.26623#S4.T2)\. Rankings are computed as an*unweighted*mean of per\-cell alignment over all five generators per \(condition, difficulty\) combination; count\-weighted averaging shifts results by≤0\.3\\leq 0\.3pp\.

Table 6:Best judge per deployment scenario\. The without GT rows reflect the common deployment setting where ground\-truth tool calls are unavailable\. GT = ground\-truth tool calls available; without GT = judge uses only query and tool schemas\. Alignment \(%\) is unweighted mean over five generators\.Takeaways\.\(1\)No ground truth\(the common case\): Gemini\-2\.5\-Pro and GPT\-5\.4 lead narrowly \(≤1\\leq 1pp\), but the without GT convergence ceiling \(§[4\.2](https://arxiv.org/html/2608.26623#S4.SS2)\) makes judge choice less consequential on hard queries: the practical difference between any two judges is≤2\\leq 2pp\. \(2\)Ground truth available: QwQ\-32B dominates all difficulty levels; GPT\-OSS\-120B is the best open\-weight alternative without reasoning\-model inference overhead\. \(3\)Single judge across all conditions: QwQ\-32B offers the best mean GT alignment \(89\.1%\) with competitive without GT performance \(85\.6%\)\. \(4\)Binary pass/fail pipelines: Practitioners scoring verdicts on a binary pass/fail scale rather than\{0,0\.5,1\}\\\{0,0\.5,1\\\}should prefer GPT\-OSS\-20B over QwQ\-32B\. Under binary verdict collapse, judge rankings differ substantially from the three\-point scale \(Spearmanρ=0\.03\\rho=0\.03; Table[29](https://arxiv.org/html/2608.26623#A20.T29), Appendix[T](https://arxiv.org/html/2608.26623#A20.SSx1)\): judges with high 0\.5\-verdict rates \(GPT\-OSS\-20B:26\.5%26\.5\\%; GPT\-5\.4:28\.2%28\.2\\%\) gain a systematic advantage because every 0\.5 verdict is remapped toward the programmatic direction, inflating their binary alignment\. QwQ\-32B’s structural advantage over GPT\-OSS\-20B diminishes or reverses under binary scoring\. \(5\)Human\-intuition alignment: Practitioners whose primary concern is agreement with human annotators rather than with the programmatic scorer should prefer GPT\-OSS\-120B, which tops both the programmatic and human\-verdict orderings \(Δ=−2\.8\\Delta=\-2\.8pp vs\. human; Table[13](https://arxiv.org/html/2608.26623#A7.T13), Appendix[G](https://arxiv.org/html/2608.26623#A7)\)\.

## Appendix BExtended Limitations Discussion

This appendix expands, point by point, the summary given in the main\-text Limitations section\.

Programmatic reference vs\. human correctness\.We adopt a deterministic programmatic scorer, rather than a human or LLM annotator, as the reference signal, since it uniquely scales to321,648321\{,\}648evaluations without reintroducing the judge\-reliability question under study\. A 120\-record human study \(Appendix[G](https://arxiv.org/html/2608.26623#A7)\), each record scored by a single annotator, validates this choice at92\.592\.5–98\.3%98\.3\\%agreement on three of four metrics but only82\.5%82\.5\\%on parameter structure, where the scorer penalises schema\-valid extra argument keys that annotators accept; the92\.7%92\.7\\%figure and human\-verdict rankings \(including the QwQ\-32B 1st→\\to4th reversal\) therefore reflect one calibrated annotator, not a consensus reference, and a multi\-annotator replication is left to future work\. This gap does not drive our findings: excluding parameter structure changes the aggregate ranking by≤0\.2\\leq 0\.2pp \(Table[14](https://arxiv.org/html/2608.26623#A7.T14)\), and a maximally generous upper\-bound correction moves every judge’s alignment by≤0\.09\\leq 0\.09pp with the ranking unchanged \(Table[15](https://arxiv.org/html/2608.26623#A7.T15)\)—though every headline*absolute*number still incorporates the current, imperfectly\-validated scoring rule, so we report both programmatic and human\-verdict rankings wherever a best\-judge claim is made\.

Tool redundancy\.The scorer also does not model tool redundancy: a generator reaching an equivalent result via a structurally different tool \(e\.g\., a dedicated aggregation tool vs\. composing two simpler ones\) can be penalised byptoolp^\{\\text\{tool\}\}andpseqp^\{\\text\{seq\}\}even though the substitution is conceptually correct\. The human study already surfaces a version of this \(99of2121tool\-selection disagreements involve semantically equivalent tools or defensible extra calls; Appendix[G](https://arxiv.org/html/2608.26623#A7)\); quantifying corpus\-wide prevalence and building a schema\-aware tool\-equivalence correction are left to future work\.

Synthetic data and real\-trace validity\.Records are synthetically generated rather than mined from enterprise traces, since a verified reference for every record – required by our paired GT/no\-GT protocol – is not obtainable at this scale from proprietary systems\. A two\-level quality gate and the same human study \(92\.7%92\.7\\%agreement\) validate the pipeline, but domain drift from real deployments remains an open validity concern, and a real\-trace comparison is blocked on corpus access\. All prompts are scoped to single\-shot, stateless planning; headline findings are comparative rather than absolute\-magnitude, which limits but does not eliminate exposure to this gap\.

Judge\-specialised baseline\.Our six main judges are all general\-purpose; to test generalisation to judge\-specialised models we added Prometheus\-2\([Kim et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib20)\)across all five generators and both GT conditions \(Table[2](https://arxiv.org/html/2608.26623#S4.T2), row Prom2\)\. It does not cluster with the general\-purpose judges under either condition \(2020–3030pp lower alignment, near\-chance pairwise agreement; Table[5](https://arxiv.org/html/2608.26623#footnote5)\), concentrated on parameter structure and query coverage \(Table[24](https://arxiv.org/html/2608.26623#A17.T24)\), and shows a negative GT lift opposite to most judges – consistent with, not contradicting, its own published reference\-free result against*human*judgement\([Kim et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib20), Appendix F\), a different comparison axis\. It is excluded from all six\-judge statistics and does not affect any headline finding; a broader judge\-specialised roster is left to future work\.

GPT\-5\.4’s triple role\.GPT\-5\.4 simultaneously serves as generator, judge, and rewrite meta\-judge\. Its self\-bias on its own generations falls within its range on other generators \(Appendix[O](https://arxiv.org/html/2608.26623#A15)\), except for sequence accuracy, where it over\-credits itself \(\+0\.172\+0\.172vs\.≤\+0\.008\\leq\+0\.008elsewhere\); we find no aggregate self\-preference but cannot rule out this metric\-localised effect\. This is a post\-hoc correlational analysis of the existing grid, not a controlled ablation isolating GPT\-5\.4’s role, and is further compounded by GPT\-5\.4 being a non\-reproducible Azure snapshot with no fixed version string \(Appendix[P](https://arxiv.org/html/2608.26623#A16)\): unlike other effects bounded in this section, this signal cannot be independently re\-verified\.

Ablation coverage\.The RQ4 \(temperature\) and RQ6 \(prompt format\) ablations were each extended to a second \(judge, generator\) pairing: temperature insensitivity replicates more tightly, while the prompt\-format advantage replicates only partially and reverses on hard queries, so we treat it as judge/generator\-dependent rather than a uniformly dominant lever\. A full grid across all judges and generators is left to future work\.

Difficulty\-tier calibration\.Difficulty rewriting is unanimously validated on58\.1%58\.1\\%of easy→\\tomedium and93\.9%93\.9\\%of medium→\\tohard records; the latter is our primary difficulty\-degradation evidence, and the “three difficulty tiers” framing should be read with medium as a robustness check rather than a fully independent, calibrated tier\.

Without\-GT ceiling and prompt default\.The without\-GT ceiling is partly attributable to the prompt’s default\-to\-1\.01\.0rubric: the C2 ablation \(Table[4](https://arxiv.org/html/2608.26623#S4.T4)\) shows it shifts by only≤\+1\.0\{\\leq\}\{\+1\.0\}pp under an alternative0\.50\.5\-default for the three strongest generators, but by\+4\.1\+4\.1and\+5\.6\+5\.6pp for the two weakest \(Llama\-3\.1\-8B, SmolLM3\-3B\); task difficulty remains the primary driver for capable generators, but the ceiling is somewhat prompt\-dependent on weaker ones, a nuance the abstract’s “structural ceiling” framing compresses\. Headline figures use unweighted topology averages over an intentionally imbalanced corpus; count\-weighted averaging shifts figures by≤1\.5\\leq 1\.5pp with no ranking change \(Appendix[S](https://arxiv.org/html/2608.26623#A19)\)\.

Evaluation\-time vs\. training\-time risk\.We study judge reliability at evaluation time and do not test whether these failure modes carry over when the same judges are used as a training signal \(e\.g\., a reward model in RLHF/DPO fine\-tuning of a tool\-calling agent, or an automatic gate for model selection\) – an evaluation\-time error changes a single reported number, while a training\-time error changes the objective an agent is optimized against\. Three findings would shift this risk differently: GT\-exposure over\-anchoring \(Finding 2\) would only affect pipelines that feed the reference trace into the reward model, pushing the agent toward mimicking surface form rather than penalising missing steps \(case studies in Appendix[H](https://arxiv.org/html/2608.26623#A8)\); the without\-GT verdict\-compression ceiling \(§[4\.2](https://arxiv.org/html/2608.26623#S4.SS2)\) is the higher\-risk case, since a reward model built on such a judge would supply near\-constant reward on hard queries regardless of actual correctness, weakening the gradient exactly where the agent most needs correction; and the correlated failure across judges \(six\-judge ensemble matching, not exceeding, the best individual judge; §[4\.2](https://arxiv.org/html/2608.26623#S4.SS2)\) implies this risk would not be mitigated by ensembling our evaluated judges, since they fail on the same records for the same structural reason\. Testing this directly would require training runs with judge\-derived rewards under controlled bias conditions, outside this paper’s scope; we flag it as a concrete, motivated direction for future work\.

## Appendix CDataset Statistics

This appendix reports the structural composition of the 3,808 AgentJudgeBench records, covering topology distribution, tool inventory size, parameter depth, and expected call\-sequence length\. All statistics are computed over the unique\-record pool \(i\.e\., collapsing the three difficulty rewrites of each record into one, since they share the same tool schemas and ground\-truth trace\)\. Figures[6](https://arxiv.org/html/2608.26623#A3.F6)and[7](https://arxiv.org/html/2608.26623#A3.F7)visualise the distributions; Tables[7](https://arxiv.org/html/2608.26623#A3.T7)and[8](https://arxiv.org/html/2608.26623#A3.T8)report the precise counts\.

#### Topology distribution \(Table[7](https://arxiv.org/html/2608.26623#A3.T7), Figure[6](https://arxiv.org/html/2608.26623#A3.F6)\)\.

Records are not uniformly distributed across topologies\. Fan\-in \(27\.5%\) and optional enrichment \(21\.6%\) are the most prevalent, reflecting the frequency of multi\-source aggregation and conditional enrichment patterns in the 15 enterprise seed domains\. Loop\-like records are the rarest \(5\.9%\) because iterative workflows are less commonly expressed as single\-turn tool\-call sequences in the generation pipeline\. Linear records \(21\.3%\) serve as the structural baseline: they require no dependency tracking and are included to anchor the difficulty gradient\. The imbalance is intentional: it mirrors the relative prevalence of each pattern in enterprise agentic workloads rather than imposing artificial uniformity\.

Figure 6:DAG topology distribution across the 3,808 unique records \(left: proportional pie, right: absolute counts\)\. Fan\-in accounts for over a quarter of all records; loop\-like is the rarest at 5\.9%\.Table 7:Record counts by DAG topology\. Counts reflect unique records; each record has three difficulty variants \(easy / medium / hard\), giving3,808×3=11,4243\{,\}808\\times 3=11\{,\}424total rows in the full dataset\.
#### Tool inventory and parameter depth \(Table[8](https://arxiv.org/html/2608.26623#A3.T8), Figure[7](https://arxiv.org/html/2608.26623#A3.F7)\)\.

Each record exposes a pool of 8\-19 available tools \(mean 12\.9\), of which the ground\-truth trace invokes 2\-5 \(mean 3\.3\)\. Tools are deliberately over\-provisioned: the ground truth uses on average only 25% of the available pool, which forces the generator and judge to perform genuine tool selection rather than selecting by elimination\. Each tool carries 0\-19 typed parameters \(mean 2\.4, median 2\); the small median reflects the prevalence of single\-argument utility functions in the inventory, while the long tail \(up to 19 parameters\) comes from complex configuration and validation tools\. The parameter minimum of zero corresponds to no\-argument sentinel tools used in optional\-enrichment and loop\-like patterns\. These structural properties collectively ensure that each of the four evaluation metrics \(tool selection, parameter structure, sequence accuracy, and query coverage\) is non\-trivially exercised across the record pool\.

Figure 7:Frequency distributions of \(left\) available tools per record, \(centre\) parameters per tool, and \(right\) expected tool calls per record\. Dashed vertical lines mark the mean of each distribution\. Tool inventory is tightly concentrated around 13 \(range 8\-19\); parameter counts are right\-skewed with a median of 2; expected call\-sequence length ranges from 2 to 5\.Table 8:Structural statistics of the 3,808 AgentJudgeBench records\. All quantities are computed over unique records \(collapsing difficulty variants\)\.*Available tools*is the size of the tool inventory exposed to the generator per record\.*Parameters per tool*counts typed argument fields in each tool’s JSON schema\.*Expected tool calls*is the length of the ground\-truth execution trace\.

## Appendix DEvaluation Corpus Statistics

Total:53,608×6​judges×2​conditions=643,29653\{,\}608\\times 6\\ \\mathrm\{judges\}\\times 2\\ \\mathrm\{conditions\}=643\{,\}296evaluation instances \(theoretical max\);321,648321\{,\}648unique \(generator, judge, difficulty, record\) tuples, each evaluated under both with GT and without GT conditions\.

Table 9:Number of records with valid LLM\-judge verdicts per generator and difficulty tier\.*Theoretical max*per generator is3,808×3=11,4243\{,\}808\\times 3=11\{,\}424\(three difficulty variants of each base record\)\. Judged records are identical across all six LLM judges for a given \(generator, difficulty\) cell\. Multiplying the*Total*column by66judges×2\\times~2conditions yields the full evaluation instance count per generator\.Table[9](https://arxiv.org/html/2608.26623#A4.T9)reports the exact number of records that received valid LLM\-judge verdicts per \(generator, difficulty\) cell, after excluding records where the generator produced an unparseable or empty tool\-call sequence\. The base dataset contains3,8083\{,\}808unique records; each is rewritten into three difficulty variants, giving a theoretical maximum of3,808×3=11,4243\{,\}808\\times 3=11\{,\}424generator inputs per generator\. The final column shows the success rate relative to this theoretical maximum\.

Sources of attrition\.The overall success rate is93\.8%93\.8\\%\(53,60853\{,\}608of57,12057\{,\}120\)\. Attrition arises from two distinct sources at different pipeline stages:

- •Unparseable generator output\(≈5\.7%\{\\approx\}5\.7\\%of inputs, weighted average\): the generator produces a response that cannot be decoded as a valid JSON tool\-call list\. This is the dominant source of attrition, concentrated on SmolLM3\-3B \(−15\.6%\-15\.6\\%\) and Qwen3\-32B \(−10\.4%\-10\.4\\%\), and more prevalent at medium difficulty for SmolLM3\-3B \(medium rate79\.8%79\.8\\%vs\.86\.6%86\.6\\%on easy/hard\)\. These failures reflect intrinsic model capability gaps on the structured output format; re\-generating these records would produce the same failure pattern and was not pursued\.
- •Persistent LLM\-judge null verdicts\(≈3\.3%\{\\approx\}3\.3\\%of main\-grid judge calls\): after the initial run, records with nulloverall llm alignment percentagewere identified after retrying twice \(two independent rerun rounds\. Records that remained null after both retries \(≈10,498\{\\approx\}10\{,\}498\(generator, judge, difficulty, record\) tuples in the main grid\) were permanently excluded\. These persistent nulls are concentrated on the longest GPT\-5\.4 and Llama\-3\.3\-70B generator outputs, which push near the context limits of certain judge endpoints \(particularly Claude Sonnet 4\.5\), causing consistent response truncation or malformed JSON\. A third retry round was not run because \(a\) two retries had already demonstrated a<5%\{<\}5\\%recovery rate for persistently null records, making further attempts cost\-prohibitive, and \(b\) the persistent nulls are distributed uniformly across DAG topologies and difficulty levels, giving no reason to expect systematic bias in the excluded records\.

## Appendix EBootstrap Confidence Intervals for Table[2](https://arxiv.org/html/2608.26623#S4.T2)

All alignment percentages in Table[2](https://arxiv.org/html/2608.26623#S4.T2)are means overNg,dN\_\{g,d\}per\-record scores\. To quantify uncertainty, we compute 95% bootstrap confidence intervals \(n=2,000n\{=\}2\{,\}000stratified resamples\) for every \(generator, judge, difficulty, condition\) cell\. Table[10](https://arxiv.org/html/2608.26623#A5.T10)reports the full CI matrix for the Llama\-3\.3\-70B generator\. The same bootstrap analysis applied to all five generators yields CIs≤0\.3\\leq 0\.3pp half\-width throughout \(the remaining matrices are in the supplementary code release\), confirming that the point estimates in Table[2](https://arxiv.org/html/2608.26623#S4.T2)are reliable and that every monotone difficulty effect is statistically robust\. CIs for the GT lift values \(Eq\.[5b](https://arxiv.org/html/2608.26623#S3.E5.2)\) appear in Section[4\.2](https://arxiv.org/html/2608.26623#S4.SS2); all six lift CIs are non\-overlapping across the positive\-vs\-negative divide\.

Table 10:Bootstrap 95% confidence intervals for alignment \(%\) on the Llama\-3\.3\-70B\-Instruct generator\. Format: mean \[lo, hi\]\.
## Appendix FDifficulty Degradation and Generator Accuracy

Figure[8](https://arxiv.org/html/2608.26623#A6.F8)shows mean judge alignment as a function of query difficulty, averaged across all six judges and five generators\. Table[11](https://arxiv.org/html/2608.26623#A6.T11)reports generator programmatic accuracyp¯\\bar\{p\}\(%\) on AgentJudgeBench, averaged across the four metrics and six DAG topologies\.

Figure 8:Mean judge alignment as a function of query difficulty, averaged across all six judges and five generators\. The without GT degradation slope is roughly1\.5×1\.5\\timessteeper than the GT slope across all generators\.Table 11:Generator programmatic accuracyp¯\\bar\{p\}\(%\) on AgentJudgeBench, averaged across the four metrics and six DAG topologies\. Generators are ranked by hard\-difficulty accuracy\.
## Appendix GProgrammatic Judge Validation

This appendix reports a human annotation study validating the programmatic scorer against independent human judgement\. We sampled 120*hard*\-difficulty records stratified across all six DAG topologies \(20 per topology\)\. For each record, annotators were shown the user query, available tool schemas, generated tool calls, ground\-truth expected tool calls, and the programmatic scores\. For each of the four metrics they were asked to*agree*or*disagree*with the programmatic score and provide a brief justification, yielding120×4=480120\\times 4=480metric\-level verdicts\. Records were stratified to over\-represent cases where at least one programmatic score is below1\.01\.0\(14 per topology\) alongside perfect\-score records \(6 per topology\), ensuring annotators encountered the full range of difficulty\. Annotators were not shown the paper’s hypotheses or the LLM judge outputs prior to annotation\.Each record was scored by a single annotator; we did not collect a second, independent judgement per record and so cannot report an inter\-annotator agreement statistic for the human labels themselves\.The92\.7%92\.7\\%and human\-verdict\-ranking figures throughout this appendix should be read as validating the programmatic scorer against one calibrated annotator’s judgement, not against a consensus reference; a multi\-annotator replication with an inter\-annotator agreement check is left to future work \(Limitations\)\.

Table 12:Human annotator agreement with the programmatic scorer across 120 hard\-difficulty records \(n=480n\{=\}480metric\-level verdicts\), stratified at 20 records per DAG topology\.The92\.7%92\.7\\%overall agreement across480480metric\-level verdicts confirms the programmatic scorer as a reliable reference signal for the alignment metric of Section[3\.2](https://arxiv.org/html/2608.26623#S3.SS2)\. Agreement is not uniform across metrics:*parameter structure*is the weakest axis \(82\.5%82\.5\\%,2121disagreements\), while*sequence accuracy*and*query coverage*are near\-perfect \(97\.5%97\.5\\%and98\.3%98\.3\\%\)\. Inspection of the2121parameter\-structure disagreements reveals two error modes\. The dominant one \(≈16\{\\approx\}16of2121cases\) is a*schema\-vs\.\-GT mismatch*: the model includes additional parameter keys that are valid per the available\-tool JSON schema but absent from the specific ground\-truth trace; the programmatic scorer penalises these extras \(per Eq\.[2](https://arxiv.org/html/2608.26623#S3.E2)\), whereas human annotators treat them as correct given the schema\. The remaining55disagreements involve structurally matching parameters that the scorer under\-penalises due to partial\-match rounding\. This pattern is precisely the semantic\-equivalence gap described in the Limitations section, and it bounds the scope of any mis\-alignment attributable to scorer noise:74\.1%74\.1\\%of metric\-level mismatches between the programmatic and LLM judges fall on tool selection, sequence accuracy, and coverage – dimensions where structural and semantic correctness largely coincide\.

On the tool\-selection axis \(99disagreements\), two failure modes emerge: semantic equivalence between tools with different names \(e\.g\.,validate\_jurisdictional\_limitsvs\.assess\_jurisdictional\_limits\), and reasonable extra tools that the scorer penalises as out\-of\-reference\. The per\-DAG pattern is consistent with known topology difficulty:*optional\_enrichment*has the lowest agreement \(85\.0%85\.0\\%\), reflecting the inherent ambiguity in judging whether optional branches were correctly included or omitted, the same ambiguity that makes this topology moderately hard in the main judge evaluation \(Table[28](https://arxiv.org/html/2608.26623#A19.T28)\)\.

#### Judge alignment against human verdicts\.

The 120 annotated records also allow measuring how closely each LLM judge’s with GT verdict matches human judgement, where records with annotator disagreement \(n=35n\{=\}35\) are scored using the human\-corrected value rather than the programmatic score\. Table[13](https://arxiv.org/html/2608.26623#A7.T13)compares each judge’s programmatic\-reference alignment \(primary metric\) against their human\-verdict alignment on the same 120 hard records\.

Table 13:Judge alignment \(%, with GT\) on 120 hard records: vs\. programmatic reference and vs\. human verdicts \(35 corrected\)\.Δ\\Delta= human−\-programmatic\. Smaller\|Δ\|\|\\Delta\|is better\.All six judges are systematically closer to the programmatic scorer than to human annotators \(meanΔ=−6\.9\\Delta=\-6\.9pp\)\. GPT\-OSS\-120B is the most human\-aligned judge \(Δ=−2\.8\\Delta=\-2\.8pp\), while Gemini\-2\.5\-Pro diverges most \(Δ=−10\.2\\Delta=\-10\.2pp\)\. The negativeΔ\\Deltareflects that human annotators are more lenient on hard examples \(particularly on sequence accuracy and parameter structure\), while both the programmatic scorer and LLM judges apply stricter structural matching\. The judge rankings under human\-verdict alignment differ meaningfully from programmatic\-reference rankings \(Spearmanρ=0\.26\\rho=0\.26\): QwQ\-32B, which leads the programmatic leaderboard, drops to 4th under human alignment, while GPT\-OSS\-120B tops both orderings\. This suggests that the programmatic scorer’s strictness inflates QwQ’s apparent advantage; GPT\-OSS\-120B is the most robust judge across both evaluation bases\. Practitioners preferring alignment with human intuition should favour GPT\-OSS\-120B, regardless of whether programmatic or human verdicts are used as the reference\.

#### Is the ranking driven by the parameter\-structure gap?

The result above raises a natural follow\-up: since parameter structure is the one metric where the programmatic scorer measurably diverges from human judgement \(§[3\.2](https://arxiv.org/html/2608.26623#S3.SS2), above\), does QwQ\-32B’s programmatic\-reference lead depend on how much weight that metric carries? We recompute the with\-GT judge ranking \(full321,648321\{,\}648\-evaluation grid, not just the 120\-record human subset\) under the equal\-weight default, parameter structure downweighted by half, and parameter structure excluded entirely\.

Table 14:With\-GT judge ranking \(programmatic reference\) under alternative weightings of the parameter\-structure metric\. The full six\-judge ranking is identical across all three weightings\.The full six\-judge ranking is unchanged under all three weightings, and QwQ\-32B’s score shifts by at most0\.20\.2pp between the equal\-weight and parameter\-excluded columns\. The programmatic\-reference ranking is therefore not an artefact of how parameter structure is weighted within the programmatic scorer\. This is a different question from the one immediately above: reweighting the four metrics within the same \(programmatic\) reference does not change the ranking, but*replacing*the reference with human verdicts on the 120\-record subset does \(QwQ\-32B 1st→\\to4th\)\. The two results together localize the issue precisely: the programmatic scorer’s four metrics are not internally biasing the ranking against each other, but the programmatic scorer as a whole does disagree with human judgement on some records, and that disagreement is concentrated on parameter structure specifically\.

#### Bounding the Eq\. 2 rescoring fix\.

A distinct, related question is what would happen if the parameter\-structure scoring rule itself were corrected, crediting schema\-valid extra argument keys as1\.01\.0rather than the current0\.50\.5, rather than just reweighting the existing scores as above\. A full fix requires checking each extra key against the tool’s schema, which we leave to future work \(main text, “Programmatic scorer as reference”\), but its impact is boundable now: we recompute the programmatic scorer’s parameter\-structure score under the maximally generous upper bound, crediting*every*currently\-penalised extra\-key case as valid, not only the schema\-valid subset a real fix would credit, across the full53,61953\{,\}619\-record, five\-generator corpus\. Only0\.83%0\.83\\%of all186,654186\{,\}654generated tool\-calls exhibit an extra\-key\-only pattern at all, and the scorer’s own mean parameter\-structure accuracy moves by\+0\.49\+0\.49pp \(90\.5%→91\.0%90\.5\\%\\to 91\.0\\%\)\. Table[15](https://arxiv.org/html/2608.26623#A7.T15)propagates this upper bound into each judge’s with\-GT alignment, holding every judge’s own verdicts fixed\.

Table 15:With\-GT alignment \(%\) under the current parameter\-structure rule vs\. the maximally generous upper\-bound correction, record\-weighted across the full corpus\. Judges’ own verdicts are held fixed; only the programmatic reference’s parameter\-structure score changes\. Ranking order is identical to the current one in both columns\.Every judge moves by at most0\.090\.09pp, and the ranking order \(QwQ\-32B\>\>GPT\-OSS\-120B\>\>Claude Sonnet 4\.5\>\>GPT\-5\.4\>\>GPT\-OSS\-20B\>\>Gemini\-2\.5\-Pro\) is identical to the current one\. Since this is the most generous correction possible \(the true schema\-aware fix can only credit a subset of what this upper bound credits\), the real fix’s effect is bounded above by these deltas\. The Eq\. 2 rescoring remains a committed future\-work item, but the ranking\-relevant risk it poses is now a measured sub\-0\.1\-pp bound rather than an open question\.

## Appendix HOver\-Anchoring Case Studies

This appendix presents three representative records from the 1,322 over\-anchoring instances identified on Gemini\-2\.5\-Pro hard\-difficulty outputs \(records whereprm<1\.0p^\{m\}\_\{r\}<1\.0,ℓGTm\>prm\\ell^\{m\}\_\{\\text\{GT\}\}\>p^\{m\}\_\{r\}, andℓwithout GTm≤prm\+0\.1\\ell^\{m\}\_\{\\text\{without GT\}\}\\leq p^\{m\}\_\{r\}\+0\.1on at least one metric\)\. Cases 1–2 document the dominant*sequence\-accuracy anchor*mechanism; Case 3 documents the secondary*coverage anchor*\. In each case the judge’s with\-GT justification contains explicit reference to the GT trace \(“sequence is logical given the reference”\), whereas the without\-GT justification independently identifies the structural flaw the programmatic judge also penalises\. For each case, thered\-shaded box is the with\-GT verdict \(over\-anchored\) and thegreen\-shaded box is the without\-GT verdict \(independently correct diagnosis\); within each box,redmarks the reasoning that over\-anchors to the reference, andgreenmarks the reasoning that catches the actual structural flaw\. GPT\-5\.4 shows a comparable negative GT lift overall \(−1\.5\-1\.5pp, Finding 2\), but we have not extracted matching qualitative case studies for it here; the mechanism below is demonstrated for Gemini\-2\.5\-Pro only, and its generalisation to GPT\-5\.4 is not independently verified\.

### Case 1: Sequence Anchor, Microbial\-Growth Linear DAG

Query & Generated CallsQuery:*“Got this cheese with 250 CFU/g initial bugs, left at 20°C for 6\.5 hours, need the growth rate, then a safe storage temperature for 120 hours, then the shelf life at that temp\.”*Generated calls:calculate\_microbial\_growth\_rate→\\toestimate\_optimal\_storage\_temperature\(2 calls; ground truth requires a third:predict\_shelf\_life\)\.Programmatic score:seq=0\.67=0\.67\(partial match; missing third call\)\.

Gemini\-2\.5\-Pro: With GT seq=1\.0=1\.0*“The sequence of the selected tools is logical, with the growth rate calculation correctly preceding the temperature estimation\.”*The judge evaluates the two\-call prefix against the GT prefix and deems it correct,ignoring the missing third step\.

Gemini\-2\.5\-Pro: Without GT seq=0\.5=0\.5*“The plan includes a redundant, identical tool call, which is a minor structural flaw in the sequence\.”*The judgeinvents a different critiquerather than catching the missing call, but does not award full credit\.

Mechanism: GT exposure causes the judge to evaluate only the presented steps against the reference rather than checking for completeness\.

### Case 2: Sequence Anchor, Cloud\-Budget Linear DAG

Query & Generated CallsQuery:*“Cloud bill hit $4,800 over 15 days vs\. $250/day normal\. 10 days left, $8,000 budget\. What’s going on and will we blow the budget?”*Generated calls:calculate\_budget\_deviation\_risk\(predicted\_trend=1\.93\)→\\toforecast\_budget\_usage\. The first call consumes apredicted\_trendargument that has not been computed by any prior step\.Programmatic score:seq=0\.33=0\.33\.

Gemini\-2\.5\-Pro: With GT seq=1\.0=1\.0*“The two selected tools are independent and can be executed in any order\.”*The GT reference shows two calls and the judge rationalises independence; it doesnot notice the unresolved input dependency\.

Gemini\-2\.5\-Pro: Without GT seq=0\.0=0\.0*“The first tool call depends on apredicted\_trendinput that is not available from the query or generated by a prior step\.”*Without the GT to anchor to, the judgecorrectly identifies the data\-dependency violation\.

Mechanism: GT anchoring suppresses dependency\-checking; the judge substitutes a plausibility heuristic \(“independent tools can run in any order”\) for structural verification\.

### Case 3: Coverage Anchor, Power\-Grid Linear DAG

Query & Generated CallsQuery:*“We’re pulling 8,450 kW of 12,000 kW, expecting \+1,200 kW, running hot for 5\.3 h vs\. 4 h threshold\. Is everything okay?”*Generated calls:Three analysis tools \(load\-duration risk, grid\-fatigue risk, capacity headroom\)\. Ground truth uses two; the extra call isestimate\_grid\_fatigue\_risk\.Programmatic score:cov=0\.67=0\.67\(extra call penalised\)\.

Gemini\-2\.5\-Pro: With GT cov=1\.0=1\.0*“The plan selects tools that address all numerical data points provided in the user’s query\.”*Seeing the GT calls, the judgecredits the prediction for covering the queryrather than checking against the reference\.

Gemini\-2\.5\-Pro: Without GT cov=0\.5=0\.5*“The plan analyzes all data points but lacks a final synthesis tool to answer whether to be worried\.”*Without GT, the judge applies anindependent completeness checkand finds partial coverage\.

Mechanism: GT presence triggers credit\-by\-association; the judge attributes coverage to the prediction by observing that the GT is complete, rather than evaluating the prediction independently\.

## Appendix IWorked Example

This appendix walks through one complete record end to end – query, available tools, generator output, ground truth, programmatic score, and LLM\-judge verdict – to make the abstract quantities in §[3](https://arxiv.org/html/2608.26623#S3)concrete\. Unlike the over\-anchoring cases above, this record is*not*a judge failure: the LLM judge and the programmatic scorer disagree because they resolve a genuine structural ambiguity differently, so the judge verdict below is shown in a neutral colour rather than as right or wrong\.

QueryHard, Fan\-In DAG*“Hey, so I bought something for $184\.73 yesterday morning around 9:45 UTC on the 18th of September 2023, and I’m wondering if that’s weird for me since I usually do like 5 transactions a day and spend around $312\.50 daily on average, can you tell me how abnormal this looks overall?”*

Available ToolsExcerpt, 14 totalcalculate\_temporal\_transaction\_deviation,calculate\_amount\_based\_deviation,compute\_composite\_behavior\_anomaly\_score, plus 11 distractor tools \(e\.g\.,detect\_device\_or\_location\_shift,get\_user\_behavior\_context\) not relevant to this query\.

Ground\-Truth Tool\-Call Sequence\(1\)calculate\_temporal\_transaction\_deviation, \(2\)calculate\_amount\_based\_deviation, \(3\)compute\_composite\_behavior\_anomaly\_score: the first two calls are independent and must both complete before the third \(fan\-in\), which combines their outputs\.

Generator Output: Llama\-3\.3\-70B\-InstructThe same three tools, in the order \(1\)calculate\_amount\_based\_deviation, \(2\)calculate\_temporal\_transaction\_deviation, \(3\)compute\_composite\_behavior\_anomaly\_score: the first two calls are swapped relative to the reference, and the third call is emitted immediately with symbolic placeholder arguments \(\{% output of function with id 2 %\}\) rather than waiting for the first two to resolve\.

Programmatic Judge tool=1\.0=1\.0, param=1\.0=1\.0, seq=0\.33=0\.33, cov=1\.0=1\.0tool=1\.0=1\.0\(correct set, no extras\); param=1\.0=1\.0\(all required argument names present – placeholder values do not count against this metric by design, §[3\.2](https://arxiv.org/html/2608.26623#S3.SS2)\);seq=0\.33=0\.33\(position\-by\-position match against the 3\-call reference: only position 3 agrees\); cov=1\.0=1\.0\(both source aspects of the query – amount and timing – are addressed\)\.

LLM Judge: QwQ\-32B, With GT tool=1\.0=1\.0, param=1\.0=1\.0, seq=1\.0=1\.0, cov=1\.0=1\.0*“Logical order: calculate individual deviations first, then combine them into a composite score\.”*The without\-GT verdict reaches the identical scores with an equivalent justification\.

Reading the divergence\.The LLM judge and the programmatic scorer agree on three of four metrics and disagree sharply on sequence accuracy \(match score0\.330\.33, the sole source of this record’s83\.3%83\.3\\%alignment rather than100%100\\%\)\. Both readings are defensible under different notions of “correct sequencing”: the judge treats the plan as a coherent two\-then\-one dependency structure regardless of which independent call comes first, while the programmatic scorer enforces the exact reference ordering position by position\. This is a concrete instance of the structural\-vs\-semantic gap discussed in Limitations: the disagreement is not a scorer bug, but a genuine ambiguity in how strictly “sequence accuracy” should be defined for fan\-in topologies where sibling branches are interchangeable\.

## Appendix JC3 Corrupted\-GT Control: Full Results

Table[1](https://arxiv.org/html/2608.26623#S4.T1)reports per\-difficulty alignment for both judges under standard GT, without GT, and the corrupted\-GT \(C3\) condition on the Llama\-3\.3\-70B generator\. The C3 condition replaces the judge’s reference with a randomly sampled GT from a different record of the same DAG topology\. The key finding is the judge\-level split: Gemini\-2\.5\-Pro with corrupted GT matches its standard GT alignment within00\-1\.81\.8pp across all difficulties, while QwQ\-32B with corrupted GT matches its without GT alignment within0\.20\.2pp\. This confirms the mechanism described in Section[4\.2](https://arxiv.org/html/2608.26623#S4.SS2): Gemini anchors to any reference block regardless of content; QwQ exercises independent reasoning when the reference is incoherent\.

## Appendix KNotation

Table[16](https://arxiv.org/html/2608.26623#A11.T16)summarises the symbols used throughout the paper, in the order in which they appear\.

Table 16:Notation used in the paper\.
## Appendix LJudge LLMs Comparison

Table[17](https://arxiv.org/html/2608.26623#A12.T17)compares AgentJudgeBench against the seven existing systems that deploy LLM judges for agentic tool\-calling evaluation, across four dimensions: per\-metric decomposition, difficulty variation, ground\-truth ablation, and use of a deterministic programmatic reference\. No prior system addresses more than one of these dimensions; AgentJudgeBench is the first to provide all four\.

Table 17:Comparison of systems that deploy LLM judges for agentic tool\-calling evaluation\.Judge Model: LLM used as judge\.Per\-Metric: whether evaluation is decomposed into fine\-grained dimensions\.Difficulty: whether tasks span multiple difficulty tiers\.GT Abl\.: whether the effect of ground\-truth availability is studied\.Prog\. Ref\.: whether a deterministic evaluator is used as a bias\-free baseline\.
## Appendix MGenerator Models

Table[18](https://arxiv.org/html/2608.26623#A13.T18)lists the five generator models used to produce tool\-calling outputs\. Models span four capability tiers – small open\-source \(3B\), mid\-scale open \(8B\), large open \(32B–70B\), and frontier closed \(GPT\-5\.4\) – ensuring the judge evaluation covers a representative range of output quality\. All generators are decoded at temperature00with the model’s native function\-calling prompt\.

Short nameHF / Server identifierSizeLlama\-3\.3\-70B\-Instructmeta\-llama/Llama\-3\.3\-70B\-Instruct70BQwen3\-32BQwen/Qwen3\-32B32BLlama\-3\.1\-8B\-Instructmeta\-llama/Llama\-3\.1\-8B\-Instruct8BSmolLM3\-3BHuggingFaceTB/SmolLM3\-3B3BGPT\-5\.4†Azure OpenAI /gpt\-5\.4, api\-ver2025\-04\-01\-preview, accessed Apr 2026\-

†“GPT\-5\.4” is the internal designation for the GPT\-5 preview variant deployed on Azure OpenAI as of April 2026; not an officially published model version string\. Raw per\-record outputs are included in the supplementary data release\.

Table 18:Generator models used to produce tool\-calling outputs\. All generators use temperature=0=0with their native function\-calling prompt\.
## Appendix NRewriting\-Preservation Validation Study

This appendix reports the full results of the meta\-judge validation study summarised in §[3\.1](https://arxiv.org/html/2608.26623#S3.SS1)\.*Goal:*verify, independently of the programmatic judge, that our difficulty\-controlled rewrites \(a\) preserve the ground\-truth tool\-call sequence and \(b\) are strictly harder \(less explicit\) than their predecessor\.

Table 19:Per\-judge “yes”\-rate on the rewrite\-validation criteria overn=198n=198stratified triplets\.Table 20:Unanimity of the three meta\-judges on each criterion \(n=198n=198\)\.#### Protocol\.

We stratify\-sampled198198triplets \(3333per DAG topology\) from the3,8083\{,\}808record pool and asked each of three frontier LLM meta\-judges \(Claude Sonnet 4\.5, GPT\-5\.4, Gemini\-2\.5\-Pro\) to emit a single JSON verdict per triplet, answering four Boolean questions: \(i\) does the medium query admit the same ground\-truth tool calls as the easy query?; \(ii\) same for hard?; \(iii\) is the medium rewrite strictly harder than easy \(less explicit in at least one of parameter names, numeric values, or tool intents\)?; \(iv\) is hard strictly harder than medium? All three meta\-judges receive identical prompt scaffolding and decoding parameters\. The meta\-judges are*not*shown any generator’s tool\-call prediction; they reason only over the three query variants, the shared available\-tools schema, and the shared ground\-truth tool\-call sequence\.

#### Per\-judge verdict rates\.

Table[19](https://arxiv.org/html/2608.26623#A14.T19)reports the fraction of triplets for which each meta\-judge answered “yes” to each question\. Task\-preservation rates are uniformly high \(77\.8%77\.8\\%\-100%100\\%across all judges and both rewrite pairs\); the strict\-hardening rates are also high for hard\-versus\-medium \(≥95\.5%\\geq 95\.5\\%\) but lower and more judge\-dependent for medium\-versus\-easy, driven by GPT\-5\.4’s stricter interpretation of what constitutes a genuine loss of explicitness\.

#### Unanimous agreement\.

Table[20](https://arxiv.org/html/2608.26623#A14.T20)reports, per criterion, the number of triplets on which the three meta\-judges unanimously agreed \(all\-yes or all\-no\) versus gave mixed verdicts\. Crucially,*no triplet receives a unanimous “no” verdict*on either task\-preservation question, and every single triplet is unanimously judged as hard\-strictly\-harder\-than\-medium\. The only criterion with sub\-60%60\\%unanimity is medium\-strictly\-harder\-than\-easy, consistent with the per\-judge analysis above: annotators disagree at the margin on whether medium constitutes a genuine hardening as opposed to a paraphrase\.

†“GPT\-5\.4” is the internal designation for the GPT\-5 preview variant deployed on Azure OpenAI as of April 2026; it is not an officially published model version string\.

Table 21:LLM judge configurations evaluated in this paper\. All judges consume the same prompt scaffold \(Appendix[W](https://arxiv.org/html/2608.26623#A23)\) and produce the same JSON output schema\.
#### Interpretation\.

The validation supports the stronger of our two design claims \(task preservation\) and partially supports the weaker one \(strict hardening\)\. The task\-preservation result means that any alignment degradation observed in Section[4](https://arxiv.org/html/2608.26623#S4)between easy and hard difficulty conditions is*not*attributable to the rewrites silently changing the underlying task: even the most conservative meta\-judge \(GPT\-5\.4, which tends to reject rewrites for minor interpretative drift\) finds task\-preservation intact on77\.8%77\.8\\%of hard rewrites, and the unanimous\-no rate is zero\. The strict\-hardening result for medium\-vs\-easy is a weaker claim; on that axis our rewriting process achieves a hardening GPT\-5\.4 would accept only about two\-thirds of the time, suggesting that a subset of our medium rewrites are better characterised as paraphrases than as genuine hardenings\. This is an informative finding on its own: it suggests the paper’s “easy→\\rightarrowmedium→\\rightarrowhard” degradation curves may partially conflate paraphrase\-robustness with genuine difficulty\-robustness, and is a natural target for a follow\-up rewriting\-pipeline revision\.

#### Robustness to dropping the non\-reproducible meta\-judge\.

GPT\-5\.4 is a non\-reproducible snapshot \(Limitations\) and also serves as a generator and LLM judge elsewhere in the pipeline, raising the question of whether the rewrite\-validation rates above are dependent on it\. We recompute unanimity using only the two reproducible meta\-judges, Claude Sonnet 4\.5 and Gemini\-2\.5\-Pro\.

Table 22:Unanimous\-agreement rate on each rewrite\-validation criterion, Claude Sonnet 4\.5 \+ Gemini\-2\.5\-Pro only \(n=198n=198\), versus the published three\-meta\-judge rate\.Δ=\\Delta=2\-judge−\-3\-judge\.All four deltas are positive: dropping GPT\-5\.4*raises*the unanimous\-agreement rate on every criterion, most sharply on hard\-preserves\-tool\-calls \(\+20\.7\+20\.7pp\) and medium\-harder\-than\-easy \(\+27\.3\+27\.3pp\)\. This is the opposite of what a self\-serving meta\-judge would produce: GPT\-5\.4 is consistently the*most conservative*of the three meta\-judges \(Table[19](https://arxiv.org/html/2608.26623#A14.T19)\), not one inflating agreement to validate its own downstream role\. The published three\-judge rates are therefore a lower bound driven by GPT\-5\.4’s stricter interpretation, not evidence that the difficulty design is unreliable; the two independently\-reproducible meta\-judges alone would support a substantially stronger validation claim\. We report the more conservative three\-judge figures throughout the main text\.

## Appendix OGPT\-5\.4 Self\-Preference Check

GPT\-5\.4 also acts as both a generator and an LLM judge in the main grid, raising a second, distinct concern from the meta\-judge robustness check above: does GPT\-5\.4\-as\-judge over\-credit GPT\-5\.4\-as\-generator? We test this directly using the existing321,648321\{,\}648\-evaluation grid \(no new inference\)\. For each \(generator, metric\) pair we compute the signed biasllm−prog\\mathrm\{llm\}\-\\mathrm\{prog\}averaged over all records and difficulty tiers, comparing GPT\-5\.4\-as\-judge’s bias on its own generations against its bias on the four other generators, with GPT\-OSS\-120B as a generator\-agnostic control judge\.

Table 23:Judge=GPT\-5\.4 bias \(llm​\_​accuracy−programmatic​\_​accuracy\\mathrm\{llm\\\_accuracy\}\-\\mathrm\{programmatic\\\_accuracy\}\) by generator, averaged over all difficulty tiers\.∗GPT\-5\.4 judging its own generations\.GPT\-5\.4’s aggregate self\-bias \(\+0\.012\+0\.012\) falls inside the range spanned by its bias on the four other generators \(−0\.024\-0\.024to\+0\.034\+0\.034\) and is closest to its bias on SmolLM3\-3B; it is not an aggregate outlier\. The control judge, GPT\-OSS\-120B, shows a comparable generator\-independent bias on GPT\-5\.4’s outputs \(\+0\.072\+0\.072\) relative to its own cross\-generator range \(\+0\.050\+0\.050to\+0\.097\+0\.097\), confirming GPT\-5\.4’s generations are not receiving unusual treatment from an unrelated judge either\. The one exception is*sequence accuracy*: GPT\-5\.4\-as\-judge over\-credits its own sequence\-accuracy by\+0\.172\+0\.172, versus at most\+0\.008\+0\.008for any other generator on that same metric under the same judge – a metric\-localised signal we report rather than average away\. We do not find evidence of aggregate self\-preference, but we cannot rule out a sequence\-accuracy\-specific effect with this design; a controlled ablation swapping GPT\-5\.4 out of one role at a time \(Limitations\) would be needed to isolate the mechanism\.

## Appendix PLLM Judge Configurations

Table[21](https://arxiv.org/html/2608.26623#A14.T21)lists the six LLM judge configurations evaluated in this paper\. The set includes large open models \(20B\-120B\), a reasoning\-enabled open model \(QwQ\-32B\), and frontier closed models \(GPT\-5\.4, Claude Sonnet 4\.5, Gemini\-2\.5\-Pro\)\. QwQ\-32B is invoked withenable\_thinking=true\(chain\-of\-thought enabled\); all other judges use greedy or near\-greedy decoding\.

Decoding temperature\.vLLM\-hosted open judges \(QwQ\-32B, GPT\-OSS\-20B, GPT\-OSS\-120B\) use temperature=0\.15=0\.15\. Frontier judges \(GPT\-5\.4, Claude Sonnet 4\.5, Gemini\-2\.5\-Pro\) were accessed via provider APIs that did not support temperature=0=0at time of evaluation; they use the provider’s recommended default \(≤1\.0\\leq 1\.0\)\. This asymmetry is an implementation constraint, not a design choice\. To assess its impact: our temperature sensitivity study \(§[4\.2](https://arxiv.org/html/2608.26623#S4.SS2)\) shows alignment varies by at most0\.60\.6pp acrossT∈\{0\.3,0\.7,1\.0\}T\\in\\\{0\.3,0\.7,1\.0\\\}, indicating that the inter\-condition temperature gap is unlikely to materially confound the cross\-judge comparisons\. A broader evaluation with harmonised temperatures across all judges is planned for a future revision\.

## Appendix QPer\-Metric Breakdown

Table[24](https://arxiv.org/html/2608.26623#A17.T24)reports per\-metric alignment under both conditions, averaged across all 12\(g,d\)\(g,d\)configurations\. Under with GT, sequence accuracy shows the widest inter\-judge spread: Gemini\-2\.5\-Pro scores64\.2%64\.2\\%while GPT\-OSS\-20B reaches86\.4%86\.4\\%, a gap of over 22 pp\. Tool selection reveals a scale effect: GPT\-OSS\-20B drops to70\.7%70\.7\\%, nearly 18 pp below GPT\-OSS\-120B \(88\.7%88\.7\\%\)\. Parameter structure and query coverage are uniformly high \(8686\-94%94\\%\)\. Under without\-GT, all variation collapses: the widest spread on any metric is3\.03\.0pp, confirming that the “default to1\.01\.0” rubric erases capability differences\.

Table 24:Per\-metric match scoreμj,r,cm×100\\mu^\{m\}\_\{j,r,c\}\\times 100, averaged across all\(g,d\)\(g,d\)configurations\. Bold marks the best value per column among the six main judges\. Prometheus\-2 \(all five generators, both GT conditions\) is shown for reference below the rule and excluded from bold column\-wise maxima; under GT it trails the six\-judge range by 20\-45 pp on parameter structure and query coverage specifically, while remaining roughly competitive on tool selection and sequence\. Without GT, the same pattern holds \(parameter structure and query coverage remain its weakest axes\) but every metric moves in the same direction as its own overall alignment \(Table[2](https://arxiv.org/html/2608.26623#S4.T2)\): tool selection and sequence trade off oppositely, with sequence and query coverage rising and tool selection falling relative to GT, consistent with a judge that, lacking a reference to anchor against, defaults to crediting plausible\-looking coverage and ordering while penalising tool selection more inconsistently\.Table 25:Judge temperature sensitivity: per\-difficulty alignment \(%\) for Qwen3\-32B on Llama\-3\.3\-70B\.Table 26:Chain\-of\-thought reasoning study: full per\-generator alignment \(%\) for QwQ\-32B with thinking on vs\. off\.Δ\\Deltarows show thinking\-on minus thinking\-off in percentage points\.
## Appendix RSupplementary Numerical Tables

### Judge Temperature: Per\-Difficulty Results

Table[25](https://arxiv.org/html/2608.26623#A17.T25)provides the full per\-difficulty numerical results for the temperature sensitivity study \(Section[4\.2](https://arxiv.org/html/2608.26623#S4.SS2)\)\. Alignment variance across temperatures is≤0\.6\\leq 0\.6pp on every \(difficulty, condition\) slice, confirming that Qwen3\-32B judge behaviour is insensitive to sampling stochasticity\.

Table 27:Judge temperature sensitivity, second pairing: with\-GT alignment \(%\) for GPT\-OSS\-120B on Llama\-3\.1\-8B\-Instruct\. Maximum spread \(0\.250\.25pp\) is even tighter than the original \(Qwen3\-32B, Llama\-3\.3\-70B\) pairing in Table[25](https://arxiv.org/html/2608.26623#A17.T25)\(≤0\.6\\leq 0\.6pp\), confirming temperature insensitivity generalises beyond the original test\-bed cell\.
### Chain\-of\-Thought Reasoning: Full Per\-Generator Grid

Table[26](https://arxiv.org/html/2608.26623#A17.T26)reports the full4×3×24\\times 3\\times 2grid for the QwQ\-32B reasoning study \(Section[4\.2](https://arxiv.org/html/2608.26623#S4.SS2)\)\. No cell shows a difference exceeding0\.30\.3pp\. The pattern is uniform across generator quality tiers: even on SmolLM3\-3B, where generator errors are most frequent and reasoning might be expected to help the judge distinguish correct from incorrect calls, the thinking trace adds nothing\.

## Appendix SPer\-Topology Breakdown

Table[28](https://arxiv.org/html/2608.26623#A19.T28)reports per\-topology alignment under with GT\. QwQ\-32B leads on five of six topologies; GPT\-OSS\-120B leads on fan\_out \(93\.9%93\.9\\%\) and is a close second elsewhere\. Gemini\-2\.5\-Pro is consistently weakest, trailing QwQ\-32B by55\-88pp\. The gap between the easiest \(fan\_out,∼\\sim93%\) and hardest \(fan\_in,∼\\sim83%\) topologies is roughly 10 pp, comparable to the easy\-to\-hard difficulty degradation in Table[2](https://arxiv.org/html/2608.26623#S4.T2)\. Fan\-out’s advantage is intuitive: each parallel branch can be verified independently, whereas fan\-in and loop\-like topologies require tracking cross\-branch dependencies\.

Table 28:Judge alignment \(with GT, %\) by DAG topology\. Bold marks the best judge per topology\.![Refer to caption](https://arxiv.org/html/2608.26623v1/figures/topo_heatmap.png)Figure 9:Judge alignment \(%\) by DAG topology \(with GT\), visualised as a heatmap\. Fan\-out consistently achieves the highest alignment across all judges; fan\-in and loop\-like are the hardest\. The ordering is judge\-independent, indicating that DAG structural complexity is an intrinsic difficulty signal\.
## Appendix TInter\-Judge Confusion Matrices

Table[10](https://arxiv.org/html/2608.26623#A20.F10)reports per\-verdict counts for the highest\-agreement pair \(QwQ\-32B×\\timesGPT\-OSS\-120B,κ=0\.606\\kappa=0\.606\) and the lowest\-agreement pair \(Claude Sonnet 4\.5×\\timesGPT\-OSS\-20B,κ=0\.225\\kappa=0\.225\) under with GT\. Disagreement concentrates almost entirely at the partial\-credit boundary:94\.7%94\.7\\%and96\.8%96\.8\\%of off\-diagonal entries involve at least one0\.50\.5verdict\. The dominant off\-diagonal cell for the low\-agreement pair is Claude Sonnet 4\.5=1=1/ GPT\-OSS\-20B=0\.5=0\.5\(18\.2%18\.2\\%of verdicts\), indicating GPT\-OSS\-20B is systematically more conservative\. Under without GT, both pairs converge as both judges default to1\.01\.0: Claude Sonnet 4\.5×\\timesOSS\-20B agreement rises from70\.3%70\.3\\%to90\.1%90\.1\\%\.

QwQ\-32B×\\timesGPT\-OSS\-120B \(κ=0\.61\\kappa=0\.61, Agreement 87\.9%\)

Claude Sonnet 4\.5×\\timesGPT\-OSS\-20B \(κ=0\.23\\kappa=0\.23, Agreement 70\.3%\)

Figure 10:Verdict\-level confusion matrices \(% ofNN\)\. Rows = judge 1, columns = judge 2\. Diagonal entries are shaded\.### Binary vs\. \{0, 0\.5, 1\} Verdict Scale

Table[29](https://arxiv.org/html/2608.26623#A20.T29)compares judge rankings under the original three\-level\{0,0\.5,1\}\\\{0,0\.5,1\\\}verdict scale vs\. a binary collapse in which each0\.50\.5verdict is remapped to00if the programmatic reference is below0\.50\.5, and11otherwise\. Columns report mean GT alignment across all 15 \(generator, difficulty\) cells per judge, and the per\-judge rate of issuing0\.50\.5verdicts\.

Table 29:Binary vs\.\{0,0\.5,1\}\\\{0,0\.5,1\\\}verdict scale: mean GT alignment \(%\) and judge rankings averaged over 15 \(generator, difficulty\) cells\.Δ\\Delta= Binary−\-Standard\.The binary collapse does not inflate all scores uniformly: judges with high0\.50\.5rates \(GPT\-5\.4:28\.2%28\.2\\%, GPT\-OSS\-20B:26\.5%26\.5\\%\) gain disproportionately \(\+11\.9\+11\.9pp and\+13\.0\+13\.0pp respectively\)\. Under binary remapping, the0\.50\.5verdict is always counted as correct because it is mapped to the programmatic direction by construction\. Judges that hedge on uncertain records therefore receive artificially inflated binary alignment, reshuffling the ranking from QwQ\-32B\>\>GPT\-OSS\-120B to GPT\-OSS\-20B\>\>GPT\-5\.4\. The Spearman rank correlation between standard and binary ranking isρ=0\.03\\rho=0\.03\(p=0\.96p=0\.96\), indicating the rankings are essentially uncorrelated\. This confirms that the0\.50\.5verdict is not random noise: it encodes directional uncertainty that is systematically correlated with judge capacity and lost under binary collapse\. We recommend retaining the three\-level scale\.

## Appendix UJudge Score Stochasticity and Prompt Sensitivity

NameYearDescriptionAgentic Data & Tool\-Calling BenchmarksSelf\-Instruct\([Wang et al\., 2023b](https://arxiv.org/html/2608.26623#bib.bib2)\)2022LLM\-only instruction synthesis; seed for synthetic data pipelines\.Mind2Web\([Deng et al\., 2023](https://arxiv.org/html/2608.26623#bib.bib3)\)2023Human\-annotated user trajectories over real websites\.AgentBench\([Liu et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib4)\)2023Multi\-domain agentic benchmark across reasoning and interaction settings\.WebArena\([Zhou et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib7)\)2024Realistic web environment with functional sites and long\-horizon tasks\.WorkArena\([Drouin et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib8)\)2024Evaluation on production websites for task\-solving agents\.AppWorld\([Trivedi et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib9)\)2024Multi\-app environment for interactive code\-based agent workflows\.TaskBench\([Shen et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib12)\)2024Tool\-graph\-based benchmark for task decomposition and tool selection\.AgentTrek\([Xu et al\., 2025a](https://arxiv.org/html/2608.26623#bib.bib10)\)2025Converts web tutorials into executable agent trajectories\.BFCL\([Patil et al\., 2025](https://arxiv.org/html/2608.26623#bib.bib11)\)2025Leaderboard for single\-turn function\-calling accuracy; AST\-based scoring\.FuncBenchGen\([Maekawa et al\., 2025](https://arxiv.org/html/2608.26623#bib.bib13)\)2025DAG\-based synthetic function\-calling with controllable complexity\.τ\\tau\-bench\([Yao et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib14)\)2024Tool\-agent\-user interaction with simulated users andpass^kreliability metric\.LLM\-as\-JudgeMT\-Bench\([Zheng et al\., 2023](https://arxiv.org/html/2608.26623#bib.bib1)\)2023Established the LLM\-as\-judge paradigm; identified positional, verbosity, and self\-enhancement biases\.JudgeBench\([Tan et al\., 2025](https://arxiv.org/html/2608.26623#bib.bib15)\)2024Judge evaluation on hard pairs with verifiable ground truth\.Arena\-Hard\-Auto\([Li et al\., 2024b](https://arxiv.org/html/2608.26623#bib.bib16)\)2024Automated pairwise judge evaluation; high human agreement\.PandaLM\([Wang et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib17)\)2024Dedicated judge model trained for pairwise comparison\.Auto\-J\([Li et al\., 2024a](https://arxiv.org/html/2608.26623#bib.bib18)\)2024Generalist judge model trained on diverse evaluation criteria\.JudgeLM\([Zhu et al\., 2023](https://arxiv.org/html/2608.26623#bib.bib19)\)2023Fine\-tuned 7B–33B judges; characterises position, knowledge, and format biases\.Prometheus 2\([Kim et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib20)\)2024Open\-weight evaluator with rubric\-conditioned direct and pairwise assessment\.LLM Judges for Tool\-Calling EvaluationToolEval\([Qin et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib23)\)2023ChatGPT as judge for pass rate on API trajectories\.StableToolBench\([Guo et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib24)\)2024GPT\-4\-turbo as evaluator in a virtualized API environment\.ToolSandbox\([Lu et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib29)\)2024Questioned LLM judge reliability; replaced with milestone\-based programmatic scoring\.MCP\-AgentBench\([Guo et al\., 2025](https://arxiv.org/html/2608.26623#bib.bib25)\)2025Hybrid rule\-based and LLM judge for task\-completion scoring\.GeoBenchX\([Krechetova and Kochedykov, 2025](https://arxiv.org/html/2608.26623#bib.bib26)\)2025Three\-judge panel for geospatial tool\-use evaluation\.Agent\-as\-a\-Judge\([Zhuge et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib27)\)2024Agent evaluates another agent on DAG\-structured development tasks\.Auto\-Eval Judge\([Bhonsle et al\., 2025](https://arxiv.org/html/2608.26623#bib.bib28)\)2025Modular framework decomposing evaluation into checklist questions\.

Table 30:Representative works across agentic benchmarks, LLM\-as\-judge methods, and LLM judges for tool\-calling evaluation\.### Repeated\-run variability

A natural concern for any LLM\-based evaluation pipeline is whether the judge’s score for a given record is stable across independent runs with the same prompt\. All frontier judges \(GPT\-5\.4, Claude Sonnet 4\.5, Gemini\-2\.5\-Pro\) were called at near\-default temperatures \(≤1\.0\\leq 1\.0, as noted in Appendix[P](https://arxiv.org/html/2608.26623#A16)\), meaning a small amount of stochasticity is inherent to each call\. Our temperature sensitivity study \(Section[4\.2](https://arxiv.org/html/2608.26623#S4.SS2)and Appendix[R](https://arxiv.org/html/2608.26623#A18)\) indirectly characterises this: Qwen3\-32B alignment varies by at most0\.60\.6pp acrossT∈\{0\.3,0\.7,1\.0\}T\\in\\\{0\.3,0\.7,1\.0\\\}on 3,771 records, providing an upper bound on within\-judge run\-to\-run variance\. We therefore expect configuration\-level alignment estimates \(averaged overNg,d≥3,764N\_\{g,d\}\\geq 3\{,\}764records\) to be highly stable; a0\.60\.6pp spread at record level contracts to≪0\.1\{\\ll\}0\.1pp at the configuration mean by the central limit theorem\.

### Prompt variation sensitivity

Our prompt ablation \(Section[4\.2](https://arxiv.org/html/2608.26623#S4.SS2), Figure[5](https://arxiv.org/html/2608.26623#S4.F5)\) shows that switching from the structured per\-metric JSON rubric to a free\-form one\-sentence instruction drops GT alignment by4\.84\.8–6\.56\.5pp for Qwen3\-32B on the Llama\-3\.3\-70B generator\. This\>5\>5pp gap on the original pairing dwarfs the within\-prompt stochasticity bound and confirms that prompt structure is a substantial source of judge\-score variability on that pairing, not sampling noise\. Practitioners adapting these prompts should expect similar sensitivity: minor wording changes \(e\.g\., removing the “default to 1\.0” instruction\) can shift alignment by11–55pp, as demonstrated by the C2 ablation \(Table[4](https://arxiv.org/html/2608.26623#S4.T4)\)\.

We tested whether the format effect generalises on a second pairing, QwQ\-32B on SmolLM3\-3B \(Table[31](https://arxiv.org/html/2608.26623#A21.T31)\)\. The direction replicates on easy \(\+3\.9\+3\.9pp\) and medium \(\+2\.4\+2\.4pp\) but is smaller than on the original pairing, and*reverses*on hard \(−0\.8\-0\.8pp: free\-form marginally ahead\)\. We therefore revise our characterisation: prompt format is not a uniformly dominant lever independent of judge, generator, or difficulty: it is a real and sometimes large effect, but its magnitude and even its direction on hard queries depend on the specific pairing \(see Limitations\)\. Standalone prompt texts for all four variants used in this study are provided in Appendix[W](https://arxiv.org/html/2608.26623#A23)\.

Table 31:Prompt format ablation, second pairing: with\-GT alignment \(%\) for QwQ\-32B on SmolLM3\-3B, structured per\-metric prompt vs\. free\-form\. Compare to the original \(Qwen3\-32B, Llama\-3\.3\-70B\) pairing’s\+4\.8\+4\.8\-\+6\.5\+6\.5pp \(Figure[5](https://arxiv.org/html/2608.26623#S4.F5)\): the effect is smaller here and reverses sign on hard queries\.

## Appendix VRelated Work Survey

A rigorous per\-judge repeated\-run study is planned for a future revision: running each judge twice on a stratified subset and computing per\-record verdict\-flip rates\. Such a study would directly quantify the fraction of borderline verdicts driven by stochasticity rather than systematic judge disagreement, and would allow score variance to be separated from the inter\-judge disagreement reported in Table[5](https://arxiv.org/html/2608.26623#footnote5)\.

## Appendix WPrompts

This appendix reproduces, verbatim, every prompt used in the evaluation pipeline: \(i\) the*generator*prompt \(Appendix[W\.1](https://arxiv.org/html/2608.26623#A23.SS1)\) that instructs each generatorg∈𝒢g\\in\\mathcal\{G\}to emit a tool\-call sequence, \(ii\) the*with GT judge*prompt \(Appendix[W\.2](https://arxiv.org/html/2608.26623#A23.SS2)\) sent to every LLM judgej∈𝒥j\\in\\mathcal\{J\}under the GT condition, \(iii\) the*without GT judge*prompt \(Appendix[W\.3](https://arxiv.org/html/2608.26623#A23.SS3)\), identical to \(ii\) save for the omission of theExpected Tool Callsblock, and \(iv\) the*Prometheus\-2*judge prompt \(Appendix[W\.4](https://arxiv.org/html/2608.26623#A23.SS4)\), following the model’s official absolute\-grading template\. Decoding parameters are held constant across all prompts and all\(g,j,d,c\)\(g,j,d,c\)configurations\.

### W\.1Generator prompt\.

SystemYou are a tool\-call generation model participating in a BENCHMARK EVALUATION\. This is a SIMULATION\. The tools listed are HYPOTHETICAL\. You are NOT being asked to actually execute anything\. You are being evaluated on your ability to select and format the correct tool calls given a query and a tool schema\. Your ONLY job: Given the user message and available tools, output the correct JSON array of tool calls\. Nothing else\. Rules: \- Output ONLY a raw JSON array\. No text, no markdown, no explanation\. \- If no tools apply, output: \[\] \- Use EXACTLY the format specified below\. \{system\_prompt\} Available Tools: \{available\_tools\}

User\{user\_message\}

### W\.2Judge prompt: with ground truth \(GT condition\)\.

UserYou are a STRICT STRUCTURAL EVALUATOR for AI agent tool\-calling plans\. IMPORTANT CONTEXT \(NON\-NEGOTIABLE\): \- The evaluated model is SINGLE\-SHOT and STATELESS\. \- The model does NOT observe tool execution or tool outputs\. \- The model ONLY plans tool calls\. \- ALL evaluation MUST be STATIC and STRUCTURAL\. \-Assume the programmatic judge is the ground truth\. ABSOLUTELY DO NOT: \- Evaluate correctness of parameter VALUES \- Compare generated values to expected values \- Infer correctness from tool output logic \- Penalize placeholder, symbolic, or dummy values \- Penalize derived\-value mismatches \(e\.g\., 0\.75 vs 0\.005\) ONLY evaluate: \- Tool presence \- Tool names \- Parameter NAMES \- Parameter STRUCTURE \(dict shape\) \- Logical ordering of calls \- Coverage of user intent If parameter NAMES match and the tool intent is satisfied, the parameter structure MUST be treated as correct\. \- Evaluation Metrics \(PROGRAMMATIC\-ALIGNED\) You must compute FOUR scores\. Scores should default to1\.0 unless a clear structural violation exists\. 1\. TOOL SELECTION ACCURACY\.Evaluate only: required tools present, no clearly irrelevant tools\. Ignore tool grouping and execution feasibility\. Scoring: 1\.0 all required/no irrelevant; 0\.5 missing or clearly irrelevant; 0\.0 plan fundamentally incorrect\. 2\. PARAMETER STRUCTURE ACCURACY \(MOST IMPORTANT\)\.Evaluate strictly: presence of required parameter names; arguments are a dictionary; parameter names logically belong to the tool\. Do NOT evaluate parameter values, numeric ranges, threshold correctness, dependency resolution, or output\-derived values\. Rules: required parameter names present⇒\\Rightarrowstructurally correct; optional parameters never reduce score; placeholder or symbolic values are always valid\. Scoring: 1\.0 correct names on all tools; 0\.5 minor missing/extra; 0\.0 required names missing for most tools\. 3\. SEQUENCE & DEPENDENCY ACCURACY\.Does the order reflect logical planning? Are prerequisites placed earlier? Assume symbolic dependency handling is valid\. Scoring: 1\.0 logical; 0\.5 minor ordering issue, intent preserved; 0\.0 illogical or contradictory\. 4\. QUERY COVERAGE ACCURACY\.Does the plan cover all parts of the user request? Are all sub\-goals planned for? Scoring: 1\.0 full coverage; 0\.5 partial; 0\.0 major intent missed\. \- OUTPUT FORMAT \(STRICT\)\.Return exactly one JSON object with the four metric fields, each containing \{"accuracy": <float\>, "justification": "<1 sentence\>"\}, plus an "overall\_assessment" single\-sentence summary\. Be conservative\. Prefer 1\.0 unless aclear structural violationexists\. Judge structure, not semantics\. QUERY: \{user\_message\} AVAILABLE TOOLS: \{available\_tools\} GENERATED TOOL CALLS: \{generated\_tool\_calls\} EXPECTED TOOL CALLS: \{expected\_responses\}

### W\.3Judge prompt: without ground truth \(without\-GT condition\)\.

UserYou are a STRICT STRUCTURAL EVALUATOR for AI agent tool\-calling plans\. IMPORTANT CONTEXT \(NON\-NEGOTIABLE\): \- The evaluated model is SINGLE\-SHOT and STATELESS\. \- The model does NOT observe tool execution or tool outputs\. \- The model ONLY plans tool calls\. \- ALL evaluation MUST be STATIC and STRUCTURAL\. \- There is NO expected tool plan available\. ABSOLUTELY DO NOT: \- Evaluate correctness of parameter VALUES \- Infer correctness from tool output logic \- Penalize placeholder, symbolic, or dummy values \- Penalize derived\-value mismatches \- Penalize missing runtime\-dependent values ONLY evaluate using: \- The user QUERY \- AVAILABLE TOOLS schema \- GENERATED TOOL CALLS Treat the AVAILABLE TOOLS as the ONLY source of truth\. \- Evaluation Metrics \(STRUCTURAL ONLY\) Compute FOUR scores\. Scores MUST default to1\.0 unless a clear structural violation exists\. 1\. TOOL SELECTION ACCURACY\.Are selected tools relevant to the query? Tools that clearly do not belong? Rules: if a tool plausibly helps answer the query, treat as valid; only penalize clearly irrelevant tools\. Scoring: 1\.0 all relevant; 0\.5 one questionable/missing; 0\.0 selection unrelated\. 2\. PARAMETER STRUCTURE ACCURACY \(MOST IMPORTANT\)\.Evaluate strictly: arguments exist and are dictionaries; parameter names exist in the tool schema; parameter names logically belong to the tool\. Do NOT evaluate parameter values, types, ranges, thresholds, or symbolic references\. Rules: matching names⇒\\Rightarrowcorrect; optional parameters never reduce score; extra parameters only penalize if clearly invalid\. Scoring: 1\.0 valid across all tools; 0\.5 minor issues; 0\.0 most tools incorrect\. 3\. SEQUENCE & DEPENDENCY ACCURACY\.Is ordering logically consistent with the query? Are analysis steps before decision/recommendation steps? Assume symbolic dependencies are valid\. Scoring: 1\.0 logical; 0\.5 minor issue; 0\.0 illogical\. 4\. QUERY COVERAGE ACCURACY\.Does the plan attempt to address all parts of the query? Are key sub\-goals represented? Scoring: 1\.0 full; 0\.5 partial; 0\.0 major intent missed\. \- OUTPUT FORMAT \(STRICT\)\.Return exactly one JSON object with the four metric fields, each containing \{"accuracy": <float\>, "justification": "<1 sentence\>"\}, plus an "overall\_assessment" single\-sentence summary\. Be conservative\. Prefer 1\.0 unless aclear structural violationexists\. Judge structure, not execution or correctness\. QUERY: \{user\_message\} AVAILABLE TOOLS: \{available\_tools\} GENERATED TOOL CALLS: \{generated\_tool\_calls\}

### W\.4Judge prompt: Prometheus\-2 \(judge\-specialised baseline\)\.

Prometheus\-2\([Kim et al\., 2024](https://arxiv.org/html/2608.26623#bib.bib20)\)is fine\-tuned for single\-metric absolute grading and cannot score four metrics in one call; we follow the model’s official absolute\-grading template and system message, issuing four independent calls per record \(one per metric, differing only in the score rubric block\), then merge the four verdicts into the same judge\-response schema every other judge produces\.

SystemYou are a fair judge assistant tasked with providing clear, objective feedback based on specific criteria, ensuring each assessment reflects the absolute standards set for performance\.

User\#\#\#Task Description: An instruction \(an agentic tool\-calling query with its available tool schemas\), a response to evaluate \(a proposed tool\-call plan\), a reference answer that gets a score of 5, and a score rubric representing an evaluation criteria are given\. 1\. Write a detailed feedback that assesses the quality of the response strictly based on the given score rubric, not evaluating in general\. 2\. After writing a feedback, write a score that is an integer between 1 and 5\. You should refer to the score rubric\. 3\. The output format should look as follows: "Feedback: \(write a feedback for criteria\) \[RESULT\] \(an integer number between 1 and 5\)" 4\. Please do not generate any other opening, closing, and explanations\. \#\#\#The instruction to evaluate: Given the user query and available tool schemas below, select and structure the correct tool call\(s\) to satisfy the query\. QUERY: \{user\_message\} AVAILABLE TOOLS: \{available\_tools\} \#\#\#Response to evaluate: \{generated\_tool\_calls\} \#\#\#Reference Answer \(Score 5\): \{expected\_responses\} \#\#\#Score Rubrics: \{one of four metric\-specific rubrics: tool selection, parameter structure, sequence & dependency, or query coverage \-\- each a bracketed criterion followed by five score descriptions, structurally identical to the with GT judge rubric in Appendix[W\.2](https://arxiv.org/html/2608.26623#A23.SS2)\} \#\#\#Feedback:

## Appendix XFree\-Form Judge Prompt \(A5\)

The following prompt is used in the A5 prompt\-format ablation \(§[4\.2](https://arxiv.org/html/2608.26623#S4.SS2)\)\. It omits explicit per\-metric definitions and scoring rubrics, asking the judge to reason freely and return a single holistic verdict per metric without anchoring instructions\.

UserYou are evaluating an AI agent’s tool\-calling plan\. Given the user query, the available tools, and the generated tool calls, assess how well the agent performed\. Consider whether it selected appropriate tools, structured parameters correctly, ordered calls logically, and covered the user’s intent\. Return a JSON object with four keys: tool\_selection\_accuracy, parameter\_structure\_accuracy, sequence\_accuracy, and query\_coverage\_accuracy\. Each key should map to an object with "accuracy" \(a float between 0 and 1\) and "justification" \(one sentence\)\. Also include an "overall\_assessment" field\. QUERY: \{user\_message\} AVAILABLE TOOLS: \{available\_tools\} GENERATED TOOL CALLS: \{generated\_tool\_calls\} EXPECTED TOOL CALLS: \{expected\_responses\}