Now evaluating agent capability is really too expensive!

Running an agentic benchmark like SWE-Bench or GAIA often requires complex environments, tool chains, long rollout times, and can cost hundreds or even thousands of dollars.

This PACE paper from Carnegie Mellon University asks a very practical question:

Is it possible to roughly know a model’s agent capability without having to run the full agent evaluation every time?

Its approach is quite clever.

Many agentic tasks look complex, but they still rely on some atomic capabilities: instruction following, planning, code generation, tool calling, information retrieval, multimodal understanding, error recovery, verification and testing, etc.

What PACE does is to select a small subset of questions from a large pool of cheap non-agentic benchmarks that best predict performance on agentic benchmarks.

Then it uses the results of these small tests to predict the model’s performance on large evaluations like SWE-Bench and GAIA.

The paper finally constructs PACE-Bench.

Experimental results are quite strong: covering 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks, with prediction error below 4%, Spearman correlation above 0.80, pairwise model ranking accuracy around 85%, and cost less than 1% of a full agent evaluation.

What I find most interesting is that it moves agent evaluation from “running the full task every time” to “estimating overall performance using cheaper capability slices.”

This is very useful for model selection, model routing, and agent harness tuning.

Because in real development, it’s hard to run the expensive agent benchmark completely every time we change a model, prompt, tool, or memory.

In the future, agent evaluation might be split into two layers:

One layer is a small number of high-cost real tasks for final confirmation; The other layer is a large number of low-cost proxy evaluations for daily iteration and model screening.

The more complex the agent, the more engineering the evaluation itself requires.

arXiv: https://arxiv.org/abs/2607.02032

Pace: A Proxy for Agentic Capability Evaluation

Source: https://arxiv.org/html/2607.02032 Yueqi Song1, Lintang Sutawika1, Jiarui Liu1, Lindia Tjuatja1, Jiayi Geng1, Yunze Xiao1, Daniel Lee2, Aditya Bharat Soni1, Vincent Lo1, Xiang Yue1, Graham Neubig1 1Carnegie Mellon University2Salesforce AI Research {yueqis, gneubig}@cs.cmu.edu neulab/paceneulab/pace-bench

Abstract

Evaluating large language model (LLM) agents on benchmarks like SWE-Bench and GAIA can be expensive, time-consuming, and requires complex infrastructure. A single evaluation can cost thousands of dollars and take days to complete. In contrast, non-agentic LLM benchmarks that test individual capabilities (e.g., reasoning, code generation, instruction following) are fast and cheap to run. In this paper, we investigate whether performance on expensive agentic benchmarks can be accurately predicted by the performance on a small, carefully selected subset of atomic evaluation instances. We introducePace, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations whose aggregate scores most reliably predict model performances on agentic benchmarks. Given a pool of candidate instances spanning atomic capabilities (instruction following, planning, tool calling, etc.),Pacefits a regression that maps a model’s scores on a compact subset of source instances to its score on the target agentic benchmark. The subset itself is curated by combining two complementary instance-selection strategies, target-relevance local selection and globally informative global selection. We applyPaceto the 4 target agentic benchmarks in this paper, which yieldsPace-Bench, the concrete proxy benchmark that we evaluate in the paper. Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show thatPace-Benchpredicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under4%4\%, Spearman correlation above0.800.80, and pairwise model-ranking accuracy around85%85\%, all at much less than1%1\%of the full agentic evaluation cost. We further analyze the selected proxy instances, revealing which skills each agentic benchmark uniquely demands.Paceenables practitioners to obtain reliable estimates of agentic performance during model development, selection, and routing, without the overhead of full agent evaluation.

Refer to captionFigure 1:Cost-versus-quality tradeoffofPace(blue) and sub-sampling target agentic evals (red), averaged across four datasets.Left:mean absolute error.Middle:Spearman correlation.Right:pairwise model-ranking accuracy. At every budget below saturation,Pacedominates sub-sampling agentic evals on all three metrics, matching quality at roughly1/1001/100of the cost.## 1Introduction

Tracking the progress of large language models (LLMs) capabilities has long relied on fast and inexpensive benchmarks that evaluate models’ individual capabilities such as knowledge retrieval(Hendryckset al.,2021a (https://arxiv.org/html/2607.02032#bib.bib1); Wanget al.,2024b (https://arxiv.org/html/2607.02032#bib.bib2)), mathematical reasoning(Hendryckset al.,2021b (https://arxiv.org/html/2607.02032#bib.bib3)), instruction following(Zhouet al.,2023 (https://arxiv.org/html/2607.02032#bib.bib4)), code generation(Jainet al.,2024 (https://arxiv.org/html/2607.02032#bib.bib5)), and more. Because such benchmarks consist of short, self-contained instances that can be scored with a single model invocation, these instances are cheap to run, easy to reproduce, and widely used for informing model development(Lianget al.,2023 (https://arxiv.org/html/2607.02032#bib.bib10); Bidermanet al.,2024 (https://arxiv.org/html/2607.02032#bib.bib69)).

As language models are increasingly deployed as agents, however, this evaluation paradigm breaks down. Agentic benchmarks such as SWE-Bench(Jimenezet al.,2024 (https://arxiv.org/html/2607.02032#bib.bib12)), GAIA(Mialonet al.,2024 (https://arxiv.org/html/2607.02032#bib.bib13)), and WebArena(Zhouet al.,2024 (https://arxiv.org/html/2607.02032#bib.bib14))require models to operate over long horizons, interact with tools or environments, and recover from errors(Wanget al.,2024a (https://arxiv.org/html/2607.02032#bib.bib11)), often requiring complex infrastructure, long rollout times, and substantial API costs. Even evaluation of a single model under a single agent harness could cost thousands of dollars and take hours or days of setup and execution. These burdens often force researchers to evaluate models less frequently, report results on limited subsets, and make rigorous agent evaluation disproportionately accessible only to well-resourced groups.

Despite the complexity of agentic tasks, success of LLMs on agentic benchmarks depends on model abilities like instruction following, planning, tool use, and reasoning, which are already measured by fast and inexpensive non-agentic benchmarks(Sumerset al.,2023 (https://arxiv.org/html/2607.02032#bib.bib17); Xiet al.,2025 (https://arxiv.org/html/2607.02032#bib.bib18)). However, researchers still run full agent evaluations to compare models to track progress, suggesting that the predictive connection between model performance on non-agentic and agentic tasks is not yet well understood. Thus, we ask:can non-agentic benchmarks serve as a reliable and low-cost proxy for agentic benchmarks?

To answer this question, we proposePace(Proxy forAgenticCapabilityEvaluation), a simple yet effective framework that selects a compact subset of non-agentic benchmark instances whose aggregate scores could best predict the target agentic benchmark performances across models.Pacedraws its candidate pool of non-agentic evaluation instances from existing benchmarks that broadly cover skills that intuitively seem important for agentic tasks, such as instruction following, tool calling, multimodal understanding, etc. We formulate the construction ofPaceas a budget-constrained subset selection problem. Given a fixed budget ofCCproxy instances,Paceuses a calibration set of models with known scores on both the candidate pool and a target agentic benchmark to identify whichCCinstances are the most predictive of the target benchmark. Concretely,Pacefits a least-squares regression that maps each model’s per-instance scores on theCCselected source instances to its target benchmark mean, with the calibration models supplying training data; bootstrap resampling over the target instances stabilizes the regression weights against label noise. TheCCinstances themselves are produced by combining two complementary criteria, a target-relevance local signal (rank-correlation with target labels) and a globally-informative global signal (SVD leverage in the source matrix).

Our approach differs from prior benchmark compression and subset-selection methods(Perlitzet al.,2024 (https://arxiv.org/html/2607.02032#bib.bib19); Poloet al.,2024 (https://arxiv.org/html/2607.02032#bib.bib20)), which aim to reduce costs within a single target benchmark, and from approaches that recast agent tasks into alternative formats (e.g., multiple choice questions)(Qinet al.,2025 (https://arxiv.org/html/2607.02032#bib.bib21)). Instead, we seek to predict model performances on a target agentic benchmark using a compact subset drawn from a separate candidate pool of inexpensive evaluation instances, with no modification to how the target benchmark is scored.

To demonstrate the empirical effectiveness ofPace, we evaluatePaceacross 14 models, 4 agentic benchmarks (GAIA(Mialonet al.,2024 (https://arxiv.org/html/2607.02032#bib.bib13)), SWE-Bench Multimodal(Yanget al.,2025 (https://arxiv.org/html/2607.02032#bib.bib26)), SWE-Bench Verified(Jimenezet al.,2024 (https://arxiv.org/html/2607.02032#bib.bib12)), SWT-Bench(Mündleret al.,2024 (https://arxiv.org/html/2607.02032#bib.bib27))), and 19 source non-agentic benchmarks spanning 11 capabilities of LLMs.

Figure 1 (https://arxiv.org/html/2607.02032#S0.F1)summarizes our answer: a small, well-chosen subset of non-agentic instances tracks agentic performance closely, at less than1100\frac{1}{100}of the cost of either a full agent evaluation or a random subset of the target benchmark itself. Concretely, using a proxy of just 100 instances,Paceachieves strong predictive performance. At equal prediction quality,Pacerequires roughly100×100\timesless cost in dollars than a random target-sampling baseline. Our main findings are as follows:

• •Generalization to Unseen Models:Instances selected withPaceon a training set of models generalize to a held-out set of models not seen during selection. This setting is relevant to deployment, where the goal is to predict a new model’s agentic performance, without committing to full agentic evaluation. Specifically, across the 4 benchmarks,Pacepredicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under4%4\%, Spearman correlation above0.800.80, and pairwise model-ranking accuracy around85%85\%, all much less than1%1\%of the full agentic evaluation cost.
• •Predictable Cost-Accuracy Tradeoff:We show that the cost-accuracy tradeoff is smooth and highly predictable. As shown inFigure 1 (https://arxiv.org/html/2607.02032#S0.F1), prediction quality broadly improves with the proxy budget and then saturates, with diminishing returns past a few hundred instances; even small budgets are already highly competitive. This lets practitioners choose evaluation budgets that match their specific resource constraints.
• •Interpretability of Capabilities:By examining the number of selected instances from each benchmark and each model capability, we identify which model capabilities most strongly affect each target agentic task, providing interpretable evidence for the capability structure that underlies successes and failures in agentic tasks.

2Background

BenchmarkIFLCAERPlanCodeIRCSTCReasMMVerSetup#InstCostAgentic BenchmarksGAIA∙\bullet∙\bullet∙\bullet∙\bullet∙\bullet∙\bullet∙\bullet∙\bullet∙\bullet∙\bulletH165$ 0.38SWE-Bench Multimodal∙\bullet∙\bullet∙\bullet∙\bullet∙\bullet∙\bullet∙\bullet∙\bullet∙\bullet∙\bulletH102$ 1.89\rowcolorgray!15SWE-Bench Verified∙\bullet∙\bullet∙\bullet∙\bullet∙\bullet∙\bullet∙\bullet∙\bullet∙\bulletH500$ 1.19SWT-Bench∙\bullet∙\bullet∙\bullet∙\bullet∙\bullet∙\bullet∙\bullet∙\bulletH430$ 0.98Non-Agentic BenchmarksACPBench∙\bullet∙\bullet∙\bullet∙\bulletL1,040$ 0.009\rowcolorgray!15AIME 2025∙\bullet∙\bulletL30$ 0.015BEIR (NFCorpus)∙\bullet∙\bulletL323$ 0.121\rowcolorgray!15BFCL∙\bullet∙\bullet∙\bullet∙\bulletL5,343$ 0.008DebugBench∙\bullet∙\bullet∙\bullet∙\bullet∙\bulletM4,253$ 0.005\rowcolorgray!15GPQA∙\bullet∙\bulletL2,384$ 0.009HumanEval∙\bullet∙\bullet∙\bullet∙\bulletL164$ 0.004\rowcolorgray!15IFEval∙\bullet∙\bulletL541$ 0.006InFoBench∙\bullet∙\bulletL500$ 0.009\rowcolorgray!15LIFBench∙\bullet∙\bullet∙\bulletL2,766$ 0.118LiveCodeBench∙\bullet∙\bullet∙\bullet∙\bullet∙\bullet∙\bulletM2,870$ 0.051\rowcolorgray!15LogiQA∙\bullet∙\bulletL1,302$ 0.007MBPP∙\bullet∙\bullet∙\bullet∙\bulletL500$ 0.004\rowcolorgray!15MMLU∙\bullet∙\bulletL28,084$ 0.006MMMU∙\bullet∙\bullet∙\bulletL900$ 0.012\rowcolorgray!15PlanBench∙\bullet∙\bullet∙\bullet∙\bulletL4,000$ 0.007RepoBench-R∙\bullet∙\bullet∙\bullet∙\bullet∙\bulletM2,010$ 0.013\rowcolorgray!15VisualPuzzles∙\bullet∙\bullet∙\bulletL1,168$ 0.018VisualWebBench∙\bullet∙\bullet∙\bullet∙\bulletL1,536$ 0.007

Table 1:Overview of existing agentic and non-agentic benchmarks, with capability coverage, setup complexity, number of instances, and estimated per-instance evaluation cost for Claude Sonnet 4.5.IF=Instruction Following,LCA=Long Context Aggregation,ER=Error Recovery,Plan=Planning,Code=Code Generation,IR=Information Retrieval,CS=Code Search,TC=Tool Calling,Reas=Reasoning,MM=Multimodal Understanding,Ver=Verification and Test. We classify a setup effort for each benchmark within 3 classifications ofHigh(requiring setting up evaluation environments),Medium(medium effort setup) orLow(only needing API calls).### 2.1From Static to Agentic Benchmarks

Standard LLM evaluation has historically relied on atomic, single-turn, and largely static benchmarks that are inexpensive to run and easy to reproduce. In contrast, evaluating LLM-based agents requires models to act over longer horizons, interact with tools or environments, and recover from intermediate errors. The agentic benchmarks often require sandboxes, browsers, repositories, or custom runtime environments, and can be sensitive to environmental noise, harness design, and external dependencies(Fanet al.,2025 (https://arxiv.org/html/2607.02032#bib.bib36)). What makes prediction plausible across these two evaluation protocols is that they require and evaluate models on overlapping underlying capabilities. We organize these into 11 categories: instruction following, long context aggregation, error recovery, planning, code generation, information retrieval, code search, tool calling, reasoning, multimodal understanding, and verification and test222Appendix B (https://arxiv.org/html/2607.02032#A2)includes citations and descriptions of these benchmarks and capabilities..Table 1 (https://arxiv.org/html/2607.02032#S2.T1)maps each non-agentic and agentic benchmark to the capabilities it requires, alongside its evaluation cost and setup requirements. SWE-Bench(Jimenezet al.,2024 (https://arxiv.org/html/2607.02032#bib.bib12)), for instance, requires models to possess capabilities like planning, code generation, and verification, all of which are also measured by cheaper non-agentic benchmarks, suggesting its agentic performance is in principle predictable from non-agentic signals.

Recent work suggests that complex model-level behavior can be partially predicted from simpler evaluation signals. Collaborative Performance Prediction (CPP)(Zhanget al.,2024 (https://arxiv.org/html/2607.02032#bib.bib37))uses matrix factorization to estimate missing model-task outcomes, ONEBench(Ghoshet al.,2025 (https://arxiv.org/html/2607.02032#bib.bib38))studies sample-level unification across open-ended capabilities, and several meta-analyses have shown that aggregated static benchmark scores can predict human-preference-based Elo(Spangheret al.,2025 (https://arxiv.org/html/2607.02032#bib.bib39); Ramaswamyet al.,2025 (https://arxiv.org/html/2607.02032#bib.bib40)). Although human preference and agentic execution are substantively different outcomes, this literature supports the br