APIEval-20
Summary
APIEval-20 is an open benchmark designed to evaluate AI agents' capabilities in testing APIs.
Similar Articles
An Empirical Study of Automating Agent Evaluation
This paper introduces EvalAgent, a system that automates the evaluation of AI agents by encoding domain-specific expertise, addressing the limitations of standard coding assistants in this task. It also presents AgentEvalBench, a benchmark for testing evaluation pipelines, and demonstrates significant improvements in evaluation reliability.
Agents' Last Exam
Introduces Agents' Last Exam (ALE), a benchmark for evaluating AI agents on long-horizon, economically valuable real-world tasks across 13 industry clusters with over 1000 tasks, revealing a large gap between benchmark performance and practical deployment.
AgentX - AI Agent evaluation framework
AgentX is an AI agent evaluation framework that helps pinpoint issues and fix them with one click.
It's impossible to test your own agent. I tried and failed.
A developer's personal account of the difficulty in objectively evaluating their own AI agent's performance, highlighting the pitfalls of self-testing and the value of unexpected, real-world benchmarks.
ProgramBench (5 minute read)
ProgramBench is a new benchmark that evaluates AI agents' ability to reconstruct complete software projects from compiled binaries and documentation without access to source code or decompilation tools.