An Empirical Study of Automating Agent Evaluation

# An Empirical Study of Automating Agent Evaluation
Source: [https://arxiv.org/abs/2605.11378](https://arxiv.org/abs/2605.11378)
Authors:[Kang Zhou](https://arxiv.org/search/cs?searchtype=author&query=Zhou,+K),[Sangmin Woo](https://arxiv.org/search/cs?searchtype=author&query=Woo,+S),[Haibo Ding](https://arxiv.org/search/cs?searchtype=author&query=Ding,+H),[Kiran Ramnath](https://arxiv.org/search/cs?searchtype=author&query=Ramnath,+K),[Subramanian Chidambaram](https://arxiv.org/search/cs?searchtype=author&query=Chidambaram,+S),[Aosong Feng](https://arxiv.org/search/cs?searchtype=author&query=Feng,+A),[Vinayak Arannil](https://arxiv.org/search/cs?searchtype=author&query=Arannil,+V),[Muhyun Kim](https://arxiv.org/search/cs?searchtype=author&query=Kim,+M),[Ishan Singh](https://arxiv.org/search/cs?searchtype=author&query=Singh,+I),[Darren Wang](https://arxiv.org/search/cs?searchtype=author&query=Wang,+D),[Zhichao Xu](https://arxiv.org/search/cs?searchtype=author&query=Xu,+Z),[Megha Gandhi](https://arxiv.org/search/cs?searchtype=author&query=Gandhi,+M),[Nirmal Prabhu](https://arxiv.org/search/cs?searchtype=author&query=Prabhu,+N),[Soumya Smruti Mishra](https://arxiv.org/search/cs?searchtype=author&query=Mishra,+S+S),[Vivek Singh](https://arxiv.org/search/cs?searchtype=author&query=Singh,+V),[Gouri Pandeshwar](https://arxiv.org/search/cs?searchtype=author&query=Pandeshwar,+G),[Lin Lee Cheong](https://arxiv.org/search/cs?searchtype=author&query=Cheong,+L+L)

[View PDF](https://arxiv.org/pdf/2605.11378)

> Abstract:Agent evaluation requires assessing complex multi\-step behaviors involving tool use and intermediate reasoning, making it costly and expertise\-intensive\. A natural question arises: can frontier coding assistants reliably automate this evaluation process? Our study shows that simply prompting coding assistants is insufficient for this task\. Without domain\-specific evaluation knowledge, frontier coding assistants achieve only a 30% execution success rate and produce over\-engineered evaluations averaging 12\+ metrics per agent, indicating that strong coding ability does not automatically translate to reliable agent evaluation\. We introduce EvalAgent, an AI assistant that automates the end\-to\-end agent evaluation pipeline\. EvalAgent encodes evaluation domain expertise as evaluation skills \(procedural instructions, reusable code and templates, and dynamically retrieved API documentation\) that compose into a trace\-based pipeline producing complete evaluation artifacts including metrics, executable code, and reports\. To systematically assess generated evaluations, we introduce a meta\-evaluation framework alongside AgentEvalBench, a benchmark comprising 20 agents, each paired with evaluation requirements and test scenarios\. We further propose the Eval@1 metric to measure whether generated evaluation code both executes and yields meaningful results on the first run\. Our experiments show that EvalAgent produces focused evaluations, improving Eval@1 from 17\.5% to 65%, and achieving 79\.5% human expert preference over baseline approaches\. Further ablation studies show that evaluation skills are critical for handling complex evaluation: removing them causes Eval@1 to drop significantly from 65% to 30%\.

## Submission history

From: Sangmin Woo \[[view email](https://arxiv.org/show-email/e217f31a/2605.11378)\] **\[v1\]**Tue, 12 May 2026 01:06:34 UTC \(13,103 KB\)