@gneubig: Agentic evaluations are sloooow, and expeeeensive. Can we instead get a good idea of how well an LLM will do on agentic…
Summary
PACE introduces a proxy method to estimate LLM agentic capabilities using cheaper single-turn tasks, reducing the cost and time of full agentic evaluations.
View Cached Full Text
Cached at: 07/06/26, 10:22 PM
Agentic evaluations are sloooow, and expeeeensive.
Can we instead get a good idea of how well an LLM will do on agentic tasks by running it solely on single-turn tasks?
Our method PACE comes up with an effective strategy for doing so!
Yueqi Song (@yueqi_song): 🚀Excited to release PACE: A Proxy for Agentic Capability Evaluation!
Evaluating LLM agents on benchmarks like SWE-Bench and GAIA is expensive, slow, and infrastructure-heavy, often costing $$$ and taking hours or days per model.
❓But do we always need to run full agentic
Similar Articles
PACE: A Proxy for Agentic Capability Evaluation
This paper introduces PACE, a framework that predicts expensive LLM agent benchmark scores using a small subset of cheaper non-agentic evaluation instances, achieving high accuracy at less than 1% of the cost.
GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents
GAUGE is a reusable offline protocol that measures the validity of using LLM-as-a-Judge in evaluating task-oriented agents, revealing a satisfaction–success gap and proposing a calibration method to improve accuracy.
Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA
This paper tests the assumption that LLMs judge better than they generate in in-context QA, finding generation accuracy exceeds self-evaluation on most benchmarks, with evaluation attending less to context. The findings challenge core assumptions in self-evaluation pipelines.
AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling
AgentJudgeBench is a benchmark for evaluating the reliability of LLM judges in agentic tool-calling scenarios, revealing limitations like structural ceilings and the nuanced impact of ground truth on alignment.
Are most LLM eval tools still too prompt-focused?
The author questions whether current LLM evaluation tools are too focused on isolated prompts rather than full workflows and agent interactions, noting that step-by-step accuracy can mask overall behavioral drift in production.