Paper page - A^2E : An End-to-End Agent Auditing Engine

Source: https://huggingface.co/papers/2608.07346

Abstract

A2E is an end-to-end evaluation engine for agent harnesses that uses a standardized task protocol and execution traces to assess capabilities across efficiency, tool use, planning, and error recovery.

With the rapid advancement oflarge language models(LLMs), harnesses have become essential infrastructure for deploying agents across a wide range of domains. The fast-evolving harness ecosystem has also made rigorous capability evaluation increasingly important. However, efficiently building an end-to-end, systematic, and comprehensive evaluation pipeline remains a significant challenge. To address this challenge, we introduce A^2E (Agent Auditing Engine), an end-to-end evaluation engine designed foragent harnesses. A^2E leverages our newly proposedAgent Task Protocol(ATP) to enable the rapid integration of evaluation tasks with different harnesses. Through an automatically instrumentedMonitor, it captures and generates standardizedexecution tracesduring experiments. In the Evaluation stage, A^2E systematically assesses harness capabilities using a suite ofmultidimensional metrics. Compared with correctness alone, these metrics provide a more fine-grained characterization of differences among harnesses inexecution efficiency,tool use,task planning, anderror recovery. Experiments conducted with A^2E further reveal that model-harness combinations exhibit substantial performance variation across different types of tasks, and that no single combination consistently outperforms all others across every task. These findings not only demonstrate the necessity of systematic evaluation but also provide useful guidance for the co-evolving of models and harnesses. Our code is available at https://github.com/datamllab/A2E.

View arXiv pageView PDFGitHub20Add to collection

Get this paper in your agent:

hf papers read 2608\.07346

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.07346 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.07346 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.07346 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.