生产环境中的高效基准测试：一项演化LLM代理研究

# Efficient Benchmarking in Production: A Study of an Evolving LLM Agent
Source: [https://arxiv.org/html/2609.21267](https://arxiv.org/html/2609.21267)
Yining She††thanks:Work done while at Meta\.Lei LinAffiliation:MetaEmail:[llin22@meta\.com](mailto:)

###### Abstract

Production LLM agents are evaluated repeatedly as they evolve, but full agent benchmarks are costly to rerun\. We study efficient recurring evaluation for a production analytics agent serving tens of thousands of monthly active users and report first\-hand deployment experience\. Using 574 historical runs of the production benchmark, split chronologically into calibration and held\-out periods, we compare random sampling, historical caching, fixed representative subsets, and IRT\-based adaptive testing\. The results show that multidimensional 2PL adaptive testing achieves the best overall score fidelity: executing 200 questions, 38\.5% of a full run, yields 1\.03 pp of MAE\. We nevertheless deployed difficulty\-stratified fixed subsets because of their operational simplicity, and show they transfer without recalibration to five other agent families and remain stable across calibration windows as short as one day\. Drawing on this deployment experience, we report practical recommendations for recurring production\-agent evaluation\.

## 1Introduction

Figure 1:Overview of the study\.\(a\)Historical evaluation runs of a production analytics agent, split by time into calibration and held\-out test runs\.\(b\)The four partial\-evaluation methods compared: each executes a subsetS⊆𝒬S\\subseteq\\mathcal\{Q\}and reports an estimates^\\hat\{s\}of the pass ratessof a complete run\.\(c\)Test on the 287 held\-out runs: pass\-rate MAE against the complete run \(lower is better\) and Spearman and Kendall rank correlation \(higher is better\)\.Production LLM agents require repeated evaluation as their models, prompts, tools, and surrounding systems evolve\. Each benchmark item may involve a multi\-step trajectory with repeated model calls, tool use, stateful interaction, and task\-specific grading; stochasticity may also require repeated trials\([Kapoor et al\., 2026](https://arxiv.org/html/2609.21267#bib.bib10);[Yao et al\., 2024](https://arxiv.org/html/2609.21267#bib.bib9);[Jimenez et al\., 2024](https://arxiv.org/html/2609.21267#bib.bib7)\)\. Our work is motivated by this challenge in the development of a deployed analytics agent, which serves tens of thousands of monthly active users\. Its development and monitoring generate tens of thousands of evaluation runs\. Benchmark evaluation is a major component of this workload\. Each run of a central 519\-question benchmark takes approximately three hours\. This makes cost\- and time\-efficient measurement important for continuous development\.

Prior work reduces LLM evaluation cost through subset selection, score reconstruction, and adaptive testing\([Vivek et al\., 2024](https://arxiv.org/html/2609.21267#bib.bib2);[Polo et al\., 2024](https://arxiv.org/html/2609.21267#bib.bib1);[Perlitz et al\., 2024](https://arxiv.org/html/2609.21267#bib.bib4);[Kipnis et al\., 2025](https://arxiv.org/html/2609.21267#bib.bib3);[Yuan et al\., 2025](https://arxiv.org/html/2609.21267#bib.bib18);[Truong et al\., 2025](https://arxiv.org/html/2609.21267#bib.bib5)\)\. These methods are primarily calibrated on responses of distinct models evaluated on public benchmarks\. Less is known about recurring evaluation of an evolving production agent, where temporally ordered outcomes from earlier runs can inform later evaluations\.

We therefore ask how faithfully partial evaluation can stand in for the complete benchmark throughout continuous agent development, along two axes:score fidelity, the tradeoff between the number of questions executed and error in the estimated full\-benchmark score, andranking fidelity, how well rankings among evaluation runs are preserved by each method\.

We compare random sampling with three uses of historical evaluation data: caching reuses stable outcomes, fixed subsets execute representative questions, and adaptive testing selects each next question based on responses in the current run\([Polo et al\., 2024](https://arxiv.org/html/2609.21267#bib.bib1);[Truong et al\., 2025](https://arxiv.org/html/2609.21267#bib.bib5)\)\. We use real historical data from our production agent: 574 benchmark runs collected over 52 days, split into 287 calibration and 287 held\-out test runs \(Fig\.[1](https://arxiv.org/html/2609.21267#S1.F1)\)\.

Our results show that partial evaluation can closely recover full\-benchmark performance with substantially fewer executions\. Multidimensional 2PL adaptive testing provides the strongest score fidelity: executing 200 of 519 questions \(38\.5% of a complete run\) yields 1\.03 percentage points \(pp\) of pass\-rate MAE\. Partial evaluation also preserves rankings: at 300 questions, the best adaptive configuration reaches 0\.992 Spearman correlation with full\-benchmark rankings\. Difficulty\-stratified fixed subsets give the strongest non\-adaptive results, and we deployed them for their operational simplicity\. We further validate this choice through cross\-agent transfer and calibration\-window sensitivity\.

Our contributions are as follows:

- •We study recurring evaluation of an evolving production agent using 574 historical benchmark runs and a temporally held\-out protocol\.
- •We compare the score\- and ranking\-fidelity tradeoffs of random sampling, historical caching, fixed subsets, and adaptive testing\.
- •We analyze method design choices and validate difficulty\-stratified fixed subsets through transfer to five agent families and sensitivity to calibration history\.
- •We derive practical recommendations from these results and our deployment experience in a production evaluation workflow\.

## 2Problem Setup

### 2\.1Recurring Production\-Agent Evaluation

As a production agent’s models, prompts, tools, and surrounding systems change, developers need to repeatedly evaluate its configurations on a human\-curated benchmark\. Each question receives a binary pass/fail verdict, which is aggregated into a run\-level pass rate used to compare configurations\.

Let the complete benchmark containNNquestions,𝒬=\{q1,…,qN\}\\mathcal\{Q\}=\\\{q\_\{1\},\\ldots,q\_\{N\}\\\}\. For evaluation runrr, letyr​i∈\{0,1\}y\_\{ri\}\\in\\\{0,1\\\}denote the verdict on questionqiq\_\{i\}, where one indicates a pass\. The complete vector of question\-level verdicts is

𝐲r=\(yr​1,…,yr​N\),\\mathbf\{y\}\_\{r\}=\(y\_\{r1\},\\ldots,y\_\{rN\}\),\(1\)and the complete\-evaluation pass rate is

sr=1N​∑i=1Nyr​i\.s\_\{r\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}y\_\{ri\}\.\(2\)We treatsrs\_\{r\}as the reference score an efficient evaluation should recover\.

### 2\.2Partial Evaluation

To reduce computational cost and latency, an efficient evaluation methodmmexecutes a subsetSr\(m\)⊆𝒬S\_\{r\}^\{\(m\)\}\\subseteq\\mathcal\{Q\}and estimates the complete pass rate ass^r\(m\)\\hat\{s\}\_\{r\}^\{\(m\)\}\. Letℋr\\mathcal\{H\}\_\{r\}denote question\-level verdicts available from earlier runs\. The estimator is

s^r\(m\)=fm​\(\{yr​i:qi∈Sr\(m\)\},ℋr\),\\hat\{s\}\_\{r\}^\{\(m\)\}=f\_\{m\}\\\!\\left\(\\\{y\_\{ri\}:q\_\{i\}\\in S\_\{r\}^\{\(m\)\}\\\},\\mathcal\{H\}\_\{r\}\\right\),\(3\)wherefmf\_\{m\}may use current\-run verdicts and history\. Different methods may select different questions and compute their reported pass rates differently\. Section[3](https://arxiv.org/html/2609.21267#S3)specifies each method’s selection rule and estimator\.

We measure efficiency by the fraction of benchmark questions executed:

ExecutionFractionr\(m\)=\|Sr\(m\)\|/N\.\\operatorname\{ExecutionFraction\}\_\{r\}^\{\(m\)\}=\|S\_\{r\}^\{\(m\)\}\|~/~\{N\}\.\(4\)

### 2\.3Evaluation Objectives

Our first objective is*score fidelity*: an efficient evaluation should report a pass rate close to the complete evaluation\. For a setℛ\\mathcal\{R\}of evaluation runs, we summarize score error using

MAE\(m\)=1\|ℛ\|​∑r∈ℛ\|s^r\(m\)−sr\|\.\\operatorname\{MAE\}^\{\(m\)\}=\\frac\{1\}\{\|\\mathcal\{R\}\|\}\\sum\_\{r\\in\\mathcal\{R\}\}\\left\|\\hat\{s\}\_\{r\}^\{\(m\)\}\-s\_\{r\}\\right\|\.\(5\)
Our second objective is*ranking fidelity*\. Developers use run rankings to compare candidate changes and investigate regressions, so partial evaluation should preserve the complete\-evaluation ordering\. We measure agreement between\{s^r\(m\)\}r∈ℛ\\\{\\hat\{s\}\_\{r\}^\{\(m\)\}\\\}\_\{r\\in\\mathcal\{R\}\}and\{sr\}r∈ℛ\\\{s\_\{r\}\\\}\_\{r\\in\\mathcal\{R\}\}using Spearman correlation and Kendall’s Tau\.

## 3Methods

We compare random sampling, historical caching, fixed subsets, and adaptive testing\. Each defines an executed setSr\(m\)S\_\{r\}^\{\(m\)\}and an estimator of the complete pass rate\.

### 3\.1Random Sampling

Random sampling drawskkquestions uniformly without replacement and reports their unweighted pass rate:

s^rrand=1k​∑qi∈Srrandyr​i\.\\hat\{s\}\_\{r\}^\{\\mathrm\{rand\}\}=\\frac\{1\}\{k\}\\sum\_\{q\_\{i\}\\in S\_\{r\}^\{\\mathrm\{rand\}\}\}y\_\{ri\}\.\(6\)

### 3\.2Historical Outcome Caching

Questions that consistently pass or fail across recent runs are less likely to change their outcomes after a modest agent update\. Historical caching leverages this stability by reusing expected verdicts for selected questions instead of executing them again\. From the available historyℋr\\mathcal\{H\}\_\{r\}, we select a look\-back window𝒲r\\mathcal\{W\}\_\{r\}and compute the historical pass rate of each questionqiq\_\{i\}:

y¯r​i=1\|𝒲r\|​∑u∈𝒲ryu​i\.\\bar\{y\}\_\{ri\}=\\frac\{1\}\{\|\\mathcal\{W\}\_\{r\}\|\}\\sum\_\{u\\in\\mathcal\{W\}\_\{r\}\}y\_\{ui\}\.\(7\)A question is cache eligible when

min⁡\(y¯r​i,1−y¯r​i\)≤ϵ,\\min\(\\bar\{y\}\_\{ri\},1\-\\bar\{y\}\_\{ri\}\)\\leq\\epsilon,\(8\)whereϵ∈\[0,0\.5\]\\epsilon\\in\[0,0\.5\]controls the required stability\. Eligible questions receive their recent majority verdict,y~r​i=𝟏\[y¯r​i≥0\.5\]\\tilde\{y\}\_\{ri\}=\\mathbf\{1\}\[\\bar\{y\}\_\{ri\}\\geq 0\.5\], while all others are executed\. IfKrK\_\{r\}is the cached set, the method executesSr=𝒬∖KrS\_\{r\}=\\mathcal\{Q\}\\setminus K\_\{r\}and reports

s^rcache=1N​\(∑qi∈Sryr​i\+∑qi∈Kry~r​i\)\.\\hat\{s\}\_\{r\}^\{\\mathrm\{cache\}\}=\\frac\{1\}\{N\}\\left\(\\sum\_\{q\_\{i\}\\in S\_\{r\}\}y\_\{ri\}\+\\sum\_\{q\_\{i\}\\in K\_\{r\}\}\\tilde\{y\}\_\{ri\}\\right\)\.\(9\)

### 3\.3Fixed\-Subset Evaluation

Inspired by[Polo et al\. \(2024\)](https://arxiv.org/html/2609.21267#bib.bib1), fixed\-subset evaluation uses historical outcomes to select a representative set that is reused across later runs\. We compare three selection strategies and two score estimators to separate subset construction from score reconstruction\.

#### 3\.3\.1IRT Models

Several selectors and estimators use item response theory \(IRT\) models fitted to the historical response matrix\. A multidimensional two\-parameter logistic \(2PL\) model assigns each run an ability vector𝜽r\\boldsymbol\{\\theta\}\_\{r\}, each question a discrimination vector𝜶i\\boldsymbol\{\\alpha\}\_\{i\}, and each question a scalar difficultyβi\\beta\_\{i\}\. The model predicts the probability of a pass as

pr​i=Pr⁡\(yr​i=1∣𝜽r,𝜶i,βi\)=σ⁡\(𝜶i⊤​𝜽r−βi\),p\_\{ri\}=\\Pr\(y\_\{ri\}=1\\mid\\boldsymbol\{\\theta\}\_\{r\},\\boldsymbol\{\\alpha\}\_\{i\},\\beta\_\{i\}\)=\\sigma\(\\boldsymbol\{\\alpha\}\_\{i\}^\{\\top\}\\boldsymbol\{\\theta\}\_\{r\}\-\\beta\_\{i\}\),\(10\)whereσ\\sigmais the logistic sigmoid\. We estimate parameters from historical verdicts by maximum a posteriori estimation with Gaussian priors\. The Rasch, or one\-parameter logistic \(1PL\), model is the unidimensional special case withαi=1\\alpha\_\{i\}=1:

pr​i=σ⁡\(θr−βi\)\.p\_\{ri\}=\\sigma\(\\theta\_\{r\}\-\\beta\_\{i\}\)\.\(11\)

#### 3\.3\.2Subset Selection

We use difficulty stratification as a direct, interpretable construction that selects questions across the estimated difficulty range\. For comparison, we also evaluate historical\-response and IRT\-feature clustering, adapting the two clustering selectors of[Polo et al\. \(2024\)](https://arxiv.org/html/2609.21267#bib.bib1)\.

##### Difficulty stratification\.

We fit the Rasch model \(Eq\.[11](https://arxiv.org/html/2609.21267#S3.E11)\) on the calibration runs, sort questions by difficultyβi\\beta\_\{i\}, and divide them intokkapproximately equal strata\. From each stratum, we select the question closest to its median difficulty, and weight it by the stratum’s fraction of the benchmark\.

##### Historical\-response clustering\.

We represent each question by a vector recording its pass/fail outcomes across historical runs and apply K\-means clustering withkkclusters\. The question closest to each cluster centroid is selected, and its weight is the fraction of benchmark questions in that cluster\.

##### IRT\-feature clustering\.

We instead represent each question by its fitted 2PL item parameters,\[𝜶i;βi\]\[\\boldsymbol\{\\alpha\}\_\{i\};\\beta\_\{i\}\], and apply the same K\-means selection and weighting\.

#### 3\.3\.3Score Estimation

Following[Polo et al\. \(2024\)](https://arxiv.org/html/2609.21267#bib.bib1), we evaluate each selector with two score estimators\. The weighted subset estimator uses only the selected questions, while generalized p\-IRT estimator \(gp\-IRT\) reconstructs the unobserved questions using the fitted IRT model\.

##### Weighted subset estimator\.

This estimator treats each selected question as the representative of its group\. For fixed subsetSS, letwiw\_\{i\}be the cluster or stratum weight of questionqiq\_\{i\}, with∑qi∈Swi=1\\sum\_\{q\_\{i\}\\in S\}w\_\{i\}=1\. The weighted estimate is

s^rweighted=∑qi∈Swi​yr​i\.\\hat\{s\}\_\{r\}^\{\\mathrm\{weighted\}\}=\\sum\_\{q\_\{i\}\\in S\}w\_\{i\}y\_\{ri\}\.\(12\)

##### Generalized p\-IRT estimator\.

The gp\-IRT estimator combines the weighted subset estimate with an IRT\-based reconstruction of the entire benchmark, balancing sampling variation against error from the fitted IRT model\. The IRT\-based reconstruction is p\-IRT\. It first estimates the current run’s ability𝜽^r\\hat\{\\boldsymbol\{\\theta\}\}\_\{r\}from its responses inSS, holding fitted item parameters fixed, and then replaces each unobserved verdict with its predicted pass probability:

s^rpIRT=1N​\(∑qi∈Syr​i\+∑qi∉Spi​\(𝜽^r\)\)\.\\hat\{s\}\_\{r\}^\{\\mathrm\{pIRT\}\}=\\frac\{1\}\{N\}\\left\(\\sum\_\{q\_\{i\}\\in S\}y\_\{ri\}\+\\sum\_\{q\_\{i\}\\notin S\}p\_\{i\}\(\\hat\{\\boldsymbol\{\\theta\}\}\_\{r\}\)\\right\)\.\(13\)In our experiments, difficulty stratification uses Rasch parameters for selection and reconstruction, while the clustering variants use the fitted multidimensional 2PL model\.

The gp\-IRT estimator blends the weighted and p\-IRT estimates:

s^rgpIRT=λ​s^rweighted\+\(1−λ\)​s^rpIRT,\\hat\{s\}\_\{r\}^\{\\mathrm\{gpIRT\}\}=\\lambda\\hat\{s\}\_\{r\}^\{\\mathrm\{weighted\}\}\+\(1\-\\lambda\)\\hat\{s\}\_\{r\}^\{\\mathrm\{pIRT\}\},\(14\)where the blending weightλ\\lambdais:

λ=b^2σ^2/\(4​\|S\|\)\+b^2\.\\lambda=\\frac\{\\hat\{b\}^\{2\}\}\{\\hat\{\\sigma\}^\{2\}/\(4\|S\|\)\+\\hat\{b\}^\{2\}\}\.\(15\)The weight favors the direct estimate when estimated IRT reconstruction biasb^2\\hat\{b\}^\{2\}is large relative to weighted\-estimator varianceσ^2/\(4​\|S\|\)\\hat\{\\sigma\}^\{2\}/\(4\|S\|\)\. We estimateσ^2\\hat\{\\sigma\}^\{2\}by averaging within\-run question\-outcome variance over historical runs andb^2\\hat\{b\}^\{2\}as squared IRT reconstruction MAE on held\-out historical runs\. Following the default anchor\-estimator adjustment of[Polo et al\. \(2024\)](https://arxiv.org/html/2609.21267#bib.bib1), we divideσ^2\\hat\{\\sigma\}^\{2\}by four, equivalently halving the estimated standard deviation\.

### 3\.4IRT\-Based Adaptive Testing

Adaptive testing selects each next question using responses observed in the current run\. The intuition is to always pick the questions that are most informative for the current evaluation\. We adapt the Rasch/Fisher\-information procedure of[Truong et al\. \(2025\)](https://arxiv.org/html/2609.21267#bib.bib5)and additionally evaluate a multidimensional 2PL extension with D\-optimal selection\.

We first calibrate item difficulties\{βi\}i=1N\\\{\\beta\_\{i\}\\\}\_\{i=1\}^\{N\}from historical runs and initialize a new run’s ability atθ^r\(0\)=0\\hat\{\\theta\}\_\{r\}^\{\(0\)\}=0\. At steptt, the predicted pass probability for each unobserved question is

pi\(t\)=σ⁡\(θ^r\(t\)−βi\)\.p\_\{i\}^\{\(t\)\}=\\sigma\(\\hat\{\\theta\}\_\{r\}^\{\(t\)\}\-\\beta\_\{i\}\)\.\(16\)Under the Rasch model, the Fisher information supplied by questionqiq\_\{i\}is

ℐi​\(θ^r\(t\)\)=pi\(t\)​\(1−pi\(t\)\)\.\\mathcal\{I\}\_\{i\}\(\\hat\{\\theta\}\_\{r\}^\{\(t\)\}\)=p\_\{i\}^\{\(t\)\}\(1\-p\_\{i\}^\{\(t\)\}\)\.\(17\)At each step, the method executes the unobserved question with maximum information and re\-estimatesθ^r\\hat\{\\theta\}\_\{r\}from all observed verdicts, stopping afterkkquestions\.

The multidimensional 2PL variant estimates ability vector𝜽^r\\hat\{\\boldsymbol\{\\theta\}\}\_\{r\}and uses D\-optimality to reduce uncertainty across its coordinates\. Given the accumulated Fisher information matrix𝐈t\\mathbf\{I\}\_\{t\}, it selects the question maximizingpi\(t\)​\(1−pi\(t\)\)​𝜶i⊤​𝐈t−1​𝜶ip\_\{i\}^\{\(t\)\}\(1\-p\_\{i\}^\{\(t\)\}\)\\boldsymbol\{\\alpha\}\_\{i\}^\{\\top\}\\mathbf\{I\}\_\{t\}^\{\-1\}\\boldsymbol\{\\alpha\}\_\{i\}, then updates the ability vector and information matrix from all observed verdicts\.

Because Fisher selection targets informative rather than representative questions, the raw pass rate of the selected questions need not approximate the complete\-benchmark pass rate\. We therefore estimate the complete pass rate using the p\-IRT reconstruction defined above \(Eq\.[13](https://arxiv.org/html/2609.21267#S3.E13)\)

## 4Experimental Setup

### 4\.1Production Evaluation Data

The evaluated system is a production analytics agent designed to answer natural\-language questions about data\. It combines language\-model reasoning with tools for retrieving and analyzing information, and is evaluated repeatedly as its models, prompts, tools, and surrounding system change\.

The benchmark consists of human\-curated questions about data written by internal analysts; neither the questions nor the recorded evaluation outcomes contain user data or personal information\. Each recorded outcome is a binary pass/fail verdict produced by an automated grader\. We retain evaluation runs with valid outcomes for at least 80% of the benchmark and compute each run’s reference pass rate over its valid outcomes\. This yields 287 calibration runs from Day 1 through Day 28, and 287 test runs from Day 29 through Day 52 \(Figure[1](https://arxiv.org/html/2609.21267#S1.F1)\)\.

### 4\.2Temporal Protocol and Method Configurations

##### Random sampling\.

We evaluatek∈\{10,20,30,50,75,100,150,200,250,300,350,400\}k\\in\\\{10,\\allowbreak 20,\\allowbreak 30,\\allowbreak 50,\\allowbreak 75,\\allowbreak 100,\\allowbreak 150,\\allowbreak 200,\\allowbreak 250,\\allowbreak 300,\\allowbreak 350,\\allowbreak 400\\\}, averaging results over 20 random seeds\.

##### Historical caching\.

Caching uses a rolling seven\-day window of prior routine runs, requires at least five historical trials per eligible question, and is rebuilt daily for reuse on that day’s evaluations\. We sweepϵ\\epsilonfrom 0\.00 to 0\.30 in increments of 0\.01, caching a question when Eq\.[8](https://arxiv.org/html/2609.21267#S3.E8)is satisfied\.

##### Fixed\-subset evaluation\.

We construct each subset once from all calibration runs at the same computational budgetskkused for random sampling and apply it unchanged to every test run\. IRT models for selection and p\-IRT reconstruction are also fitted on all calibration runs\. To estimate the gp\-IRT bias termb^2\\hat\{b\}^\{2\}, we split calibration runs in half, fit an auxiliary IRT model on the first half, and measure reconstruction error on the second by revealing half of each run’s available outcomes\. For every multidimensional 2PL fit, we selectd∈\{2,5,10,15\}d\\in\\\{2,5,10,15\\\}by log likelihood on a fixed 15% sample of observed calibration cells\.

##### Adaptive testing\.

We evaluate adaptive testing at the samekk\. Item parameters are fitted on all calibration runs and held fixed during question selection for each test run\. We compare Rasch adaptive testing with the multidimensional 2PL variant to assess whether modeling multiple latent abilities improves the accuracy–execution tradeoff\.

## 5Results

### 5\.1Score Fidelity

Figure[2](https://arxiv.org/html/2609.21267#S5.F2)compares historical caching, difficulty\-stratified fixed subsets with weighted and gp\-IRT estimation, and Rasch adaptive testing on 287 held\-out runs111Appendix report complete values in tables\.\. We examine the remaining design choices within each method family in Section[5\.3](https://arxiv.org/html/2609.21267#S5.SS3)\.

![Refer to caption](https://arxiv.org/html/2609.21267v1/figures/main_mae_efficiency.png)Figure 2:Pass\-rate MAE v\.s\. execution fraction\. Fixed\-budget methods usek/519k/519\. Caching points report mean execution fractions across cache thresholds\.Among the methods evaluated at fixed computational budgetskk, difficulty\-stratified subsets lead at small budgets: atk=100k\{=\}100\(19\.3% of the benchmark\), gp\-IRT obtains 2\.65pp MAE, compared with 2\.94 for random sampling and 2\.92 for adaptive testing\. Adaptive testing leads fromk=200k\{=\}200\(38\.5%\), where its MAE is 1\.37 pp versus 1\.40 for fixed\-subset gp\-IRT and 1\.79 for random sampling\. Atk=300k\{=\}300\(57\.8%\), the respective errors are 0\.79, 0\.99, and 1\.18 pp\. Weighted and gp\-IRT subset estimates are similar across these budgets\.

Caching determines execution through its eligibility threshold, yielding mean execution fractions from 20\.3% to 81\.6% and MAE from 7\.36 to 0\.81 pp\. More aggressive caching reduces execution but increases error, and its curve lies above the other methods over their shared range\. For example, caching obtains 1\.54 pp MAE at 69\.1% execution, while adaptive testing, fixed\-subset gp\-IRT, and random sampling obtain 0\.60, 0\.73, and 0\.93 pp atk=350k\{=\}350\(67\.4%\)\.

##### Takeaway\.

Difficulty\-stratified fixed subsets perform best at small budgets, while adaptive testing achieves the lowest MAE fromk=200k\{=\}200onward\. Historical caching is less accurate than the other methods at comparable execution levels\.

### 5\.2Ranking Fidelity

Fig\.[3](https://arxiv.org/html/2609.21267#S5.F3)reports ranking fidelity for the same configurations\. Both correlations increase with the computational budget, but the differences among methods are more pronounced for Kendall’s Tau\. Atk=100k\{=\}100\(19\.3%\), fixed\-subset gp\-IRT obtains 0\.930 Spearman and 0\.770 Kendall, compared with 0\.923&0\.760 for adaptive testing and 0\.911&0\.751 for random sampling\.

![Refer to caption](https://arxiv.org/html/2609.21267v1/figures/main_spearman_efficiency.png)

\(a\) Spearman correlation

![Refer to caption](https://arxiv.org/html/2609.21267v1/figures/main_kendall_efficiency.png)

\(b\) Kendall’s Tau

Figure 3:Ranking fidelity v\.s\. execution fraction on 287 held\-out runs\. Higher is better\.Adaptive testing becomes stronger as the computational budget increases\. Atk=200k\{=\}200\(38\.5%\), adaptive testing and fixed\-subset gp\-IRT tie in Spearman \(0\.972\), while adaptive testing has higher Kendall \(0\.873 v\.s\. 0\.859\)\. Atk=300k\{=\}300\(57\.8%\), adaptive testing reaches 0\.986 Spearman and 0\.920 Kendall, compared with 0\.981&0\.887 for fixed\-subset gp\-IRT and 0\.975&0\.875 for random sampling across budgets\.

Caching preserves rankings much better than its score MAE suggests\. At 20\.3% execution it obtains 0\.905 Spearman and 0\.733 Kendall\. At 79\.0%, it reaches 0\.996 and 0\.967, compared with 0\.995 and 0\.962 for adaptive testing atk=400k\{=\}400\(77\.1%\)\.

##### Takeaway\.

Difficulty\-stratified subsets give the strongest ranking fidelity at small budgets, while adaptive testing leads at larger budgets\. Historical caching preserves run rankings substantially better than its score\-estimation error alone would suggest\.

### 5\.3Design Choices within Method Families

The score and ranking fidelity comparisons in Sections[5\.1](https://arxiv.org/html/2609.21267#S5.SS1)and[5\.2](https://arxiv.org/html/2609.21267#S5.SS2)yield similar method orderings\. We therefore focus on MAE in the remaining analyses and report rank correlations in the appendix\.

#### 5\.3\.1Fixed\-Subset Selection and Estimation

Figure[4](https://arxiv.org/html/2609.21267#S5.F4)separates the effect of subset selection from score estimation\. Difficulty stratification has the lowest MAE across most budgets and leads under both estimators fromk=150k\{=\}150\(28\.9%\)\. With gp\-IRT, historical\-response K\-means leads fromk=30k\{=\}30\(5\.8%\) throughk=100k\{=\}100\(19\.3%\) and generally outperforms IRT\-feature K\-means\. Fromk=200k\{=\}200\(38\.5%\), both clustering selectors perform worse than random sampling under either estimator, while difficulty stratification remains better\.

Figure 4:Fixed\-subset MAE on 287 held\-out runs\.gp\-IRT benefits the clustering selectors more than difficulty stratification, whose weighted and gp\-IRT estimates are consistently similar\. Ranking correlation results largely mirror MAE \(Fig\.[8](https://arxiv.org/html/2609.21267#A1.F8)&[9](https://arxiv.org/html/2609.21267#A1.F9)\)\.

The weak performance of IRT\-feature K\-means contrasts with[Polo et al\. \(2024\)](https://arxiv.org/html/2609.21267#bib.bib1), which found IRT\-based selection consistently effective\. Two differences between the experimental settings may explain the conflicts\. First, our calibration data of recurring configurations of one production agent may contain fewer distinct response patterns than diverse LLM populations used by[Polo et al\. \(2024\)](https://arxiv.org/html/2609.21267#bib.bib1), making multidimensional item parameters less reliable and therefore clusters less representative\. Difficulty stratification requires only a scalar difficulty ordering and directly preserves coverage across that range\. Second, our subsets are much larger fractions of the benchmark:k=100k\{=\}100\(19\.3%\) versus 100 of roughly 14,000 MMLU questions in[Polo et al\. \(2024\)](https://arxiv.org/html/2609.21267#bib.bib1)\. Random sampling becomes more accurate at these larger sampling fractions, leaving less scope for clustering to improve upon it\.

##### Takeaway\.

Difficulty stratification performs best overall, while the more complex selectors provide no consistent gain and fall below random sampling at moderate computational budgets\.

#### 5\.3\.2Adaptive IRT Model

Figure[5](https://arxiv.org/html/2609.21267#S5.F5)compares Rasch and multidimensional 2PL adaptive testing\. Multidimensional 2PL adaptive testing has lower MAE than Rasch and random sampling at everykk, with its largest advantage atk=100k\{=\}100\(19\.3%\): 1\.97 pp versus 2\.92 and 2\.94 pp, respectively\. Its advantage narrows as execution increases\. Spearman and Kendall show the same ordering \(Figure[10](https://arxiv.org/html/2609.21267#A1.F10)\)\.

Figure 5:Adaptive Testing MAE on the held\-out runs\.Rasch represents each run on one ability axis, whereas multidimensional 2PL allows question outcomes to depend on different ability coordinates through their discrimination vectors\. D\-optimal selection can therefore choose questions that inform ability coordinates not well covered by the questions already executed\. This benefit is largest at small computational budgets\. As execution increases, p\-IRT replaces more predictions with observations, reducing the models’ difference\.

##### Takeaway\.

A more expressive IRT model can substantially improve adaptive testing\.

## 6Practical Validation of Fixed\-Subset

The preceding comparisons show that difficulty\-stratified fixed subsets provide strong score and ranking fidelity\. Although multidimensional adaptive testing achieves lower error, difficulty stratification is operationally simpler: the subset is constructed once and requires neither sequential selection nor run\-specific ability updates\. Executing the same questions in every run also supports direct question\-level comparisons and diagnosis as the production agent evolves\. These properties motivated its use in our production workflow\.

We therefore validate two properties important for recurring evaluation on difficulty\-stratified fixed subset only: whether a subset calibrated on one agent transfers to other agent families, and how its fidelity changes with the calibration window\.

### 6\.1Cross\-Agent Transfer

We apply the difficulty\-stratified subsets and Rasch and gp\-IRT parameters calibrated on the original agent, without recalibration, to 299 runs from five additional agent families\. These distinct systems differ in their agent frameworks, execution harnesses, tool sets, and model configurations\. No outcomes from these families are used to select questions or fit the estimators\. We compare against uniform random sampling over the same computational budgets, averaged over 20 seeds\.

On the pooled transferred runs, gp\-IRT has lower MAE than random sampling at 11 of 12 budgets \(Figure[6](https://arxiv.org/html/2609.21267#S6.F6)\)\. Atk=200k\{=\}200\(38\.5%\), random sampling, weighted estimation, and gp\-IRT obtain 1\.86, 1\.49, and 1\.47 pp MAE, respectively; gp\-IRT also remains close to its 1\.40 pp MAE on the original 287 held\-out runs\. Both fixed\-subset estimators also exceed random sampling in Spearman and Kendall correlation at every budget \(Figure[11](https://arxiv.org/html/2609.21267#A1.F11)\)\.

Figure 6:MAE on 287 original\-agent and 299 transferred runs from five families\. The subset and estimators use only original\-agent calibration data\.The advantage is not uniform across every family and computational budget\. Nevertheless, gp\-IRT has lower MAE than random sampling for a majority of budgets within each family\. The appendix reports complete pooled and family\-level results\.

This transfer indicates that the IRT model captures intrinsic aspects of question difficulty that generalize across agent systems, even when calibrated on a single agent\.

##### Takeaway\.

Without recalibration, the difficulty\-stratified subset outperforms random sampling on the pooled transferred runs at 11 of 12 budgets and remains close to its original\-agent fidelity, although gains vary across families\.

### 6\.2Calibration\-Window Sensitivity

We next examine how the amount and time span of calibration data affect the fixed subset\. We fix the calibration end day at Day 28 and advance its start day in three\-day increments from Day 1 to Day 28, yielding ten nested windows containing 287 to 14 runs \(Table[12](https://arxiv.org/html/2609.21267#A1.T12)\)\. For each window, we refit Rasch and gp\-IRT, reconstruct the difficulty\-stratified subsets fork∈\{100,200,300,400\}k\\in\\\{100,200,300,400\\\}, and evaluate them on the same 287 held\-out runs\.

MAE does not increase monotonically as the calibration window shortens \(Fig\.[7](https://arxiv.org/html/2609.21267#S6.F7)\)\. Variation is greatest atk=100k\{=\}100\. Atk=200k\{=\}200, gp\-IRT MAE ranges from 1\.35\-1\.59 pp: the one\-day window \(14 runs\) obtains 1\.41 pp, compared with 1\.40 pp for the full four\-week window\. Weighted and gp\-IRT estimates remain similar, while Spearman and Kendall show the same non\-monotonic pattern \(Fig\.[15\(a\)](https://arxiv.org/html/2609.21267#A1.F15.sf1)&[15\(b\)](https://arxiv.org/html/2609.21267#A1.F15.sf2)\)\.

Figure 7:MAE across different calibration windows on 287 held\-out runs\.This stability may reflect the relatively mature development stage of the agent and limited day\-to\-day change during the study\. Recent runs may therefore suffice to estimate the difficulty ordering used for stratification and may better match the immediately following test period, while older runs add limited new information\.

##### Takeaway\.

Fixed\-subset fidelity is stable across the studied calibration windows, with no consistent benefit from retaining the full 4\-week history\.

## 7Practical Recommendations

Production deployment requires considering operational properties alongside statistical fidelity\. Historical caching retains the full question set, but requires recent outcomes and monitoring for stale verdicts\. Fixed subsets provide predictable execution and common questions across runs\. Adaptive testing can tailor selection to each run, but requires sequential orchestration, run\-specific state, and repeated ability updates\. These operational differences make the appropriate method dependent on the surrounding evaluation workflow\.

For our production workflow, we deployed difficulty\-stratified fixed subsets withk∈\{100,200,300,400\}k\\in\\\{100,200,300,400\\\}, allowing users to choose an execution–fidelity tradeoff\. Fixed subsets reveal the workload before execution and preserve question\-level comparisons for diagnosing behavioral changes\. They also avoid additional sequential serving logic in an evaluation system designed to execute questions in parallel\. The subset transferred without recalibration to five additional agent families and showed no monotonic loss of fidelity across the calibration windows\. These results support reuse across the systems and period studied\.

As the agent evolves, we recommend recalibrating after material changes to its models, prompts, tools, or execution system, or after sustained shifts in question\-level outcomes\. Developers should also periodically run the full benchmark and compare its pass rate with the fixed\-subset estimate to measure the error introduced during ongoing monitoring\.

## 8Related Work

##### Efficient Evaluation\.

Prior work reduces evaluation cost by estimating complete\-benchmark results from item subsets\. Fixed\-form methods use cross\-model correlations, IRT, clustering, or benchmark\-level optimization\([Vivek et al\., 2024](https://arxiv.org/html/2609.21267#bib.bib2);[Polo et al\., 2024](https://arxiv.org/html/2609.21267#bib.bib1);[Perlitz et al\., 2024](https://arxiv.org/html/2609.21267#bib.bib4);[Kipnis et al\., 2025](https://arxiv.org/html/2609.21267#bib.bib3);[Fogliato et al\., 2024](https://arxiv.org/html/2609.21267#bib.bib17)\)\. Other methods tailor subsets to the target model, optimize ranking preservation, or jointly select observed items and predict unobserved outcomes\([Yuan et al\., 2025](https://arxiv.org/html/2609.21267#bib.bib18);[Saranathan et al\., 2025](https://arxiv.org/html/2609.21267#bib.bib19);[Li et al\., 2025](https://arxiv.org/html/2609.21267#bib.bib20)\)\. Adaptive evaluation, building on computerized adaptive testing\([Lord, 1980](https://arxiv.org/html/2609.21267#bib.bib6)\), selects questions by their expected information about the current system\([Truong et al\., 2025](https://arxiv.org/html/2609.21267#bib.bib5)\)\. These methods are typically calibrated on response matrices of distinct language models\. We instead compare historical caching, reusable fixed subsets, and adaptive testing on temporally ordered evaluations of one evolving production agent\.

##### Evaluating LLM Agents

Agent evaluation is especially expensive because each item may require a full trajectory involving repository interaction, browsing, tool calls, or multi\-turn interaction\([Jimenez et al\., 2024](https://arxiv.org/html/2609.21267#bib.bib7);[Mialon et al\., 2024](https://arxiv.org/html/2609.21267#bib.bib8);[Yao et al\., 2024](https://arxiv.org/html/2609.21267#bib.bib9)\)\. Such work develops benchmarks and infrastructure for comparing agents, while our setting concerns recurring evaluations generated as one agent evolves\.

##### Continuous Evaluation of Evolving Systems

Our setting also relates to regression\-test minimization, selection, and prioritization for evolving software\([Rothermel and Harrold, 1997](https://arxiv.org/html/2609.21267#bib.bib11);[Yoo and Harman, 2012](https://arxiv.org/html/2609.21267#bib.bib12)\)\. Continuous\-integration methods use code changes and historical executions to allocate testing resources\([Kim and Porter, 2002](https://arxiv.org/html/2609.21267#bib.bib13);[Elbaum et al\., 2014](https://arxiv.org/html/2609.21267#bib.bib14);[Machalica et al\., 2019](https://arxiv.org/html/2609.21267#bib.bib15)\), while non\-deterministic agent outcomes resemble flaky tests\([Luo et al\., 2014](https://arxiv.org/html/2609.21267#bib.bib16)\)\. Conventional regression testing seeks faults using signals such as coverage or change impact, but changes to an LLM agent do not directly identify affected benchmark questions\. We therefore use historical question\-level outcomes and a partial current run to estimate the complete benchmark score and run ranking\.

## Limitations

Our data come from one organization and one production benchmark\. Although the difficulty\-stratified subset transferred to five additional agent families, this evidence covers only fixed subsets and does not establish generalization to other benchmarks, historical caching, or adaptive testing\. Future work should extend the analysis using data from a more diverse range of agents\.

We evaluate pass\-rate MAE and aggregate ranking fidelity using Spearman and Kendall correlations, which do not directly measure regression detection or release\-gate decisions at operational thresholds\. Future work should add threshold\-based analyses that measure missed regressions, false alarms, and agreement with decisions based on the full benchmark\.

We do not release the benchmark questions, the question\-level outcome matrix, or the implementation\. The methods we compare are fully specified in Section 3 and are reproducible on any benchmark with recorded per\-item outcomes\.

## References

- Elbaumet al\.\(2014\)S\. Elbaum, G\. Rothermel, and J\. PenixTechniques for improving regression testing in continuous integration development environments\.InProceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering,pp\. 235–245\.Cited by:[§8](https://arxiv.org/html/2609.21267#S8.SS0.SSS0.Px3.p1.1)\.
- Fogliatoet al\.\(2024\)R\. Fogliato, P\. Patil, N\. Akpinar, and M\. MonfortPrecise model benchmarking with only a few observations\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 9563–9575\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.536/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.536)Cited by:[§8](https://arxiv.org/html/2609.21267#S8.SS0.SSS0.Px1.p1.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. NarasimhanSwe\-bench: can language models resolve real\-world github issues?\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 54107–54157\.Cited by:[§1](https://arxiv.org/html/2609.21267#S1.p1.1),[§8](https://arxiv.org/html/2609.21267#S8.SS0.SSS0.Px2.p1.1)\.
- Kapooret al\.\(2026\)S\. Kapoor, B\. Stroebl, P\. Kirgis, N\. Nadgir, Z\. Siegel, B\. Wei, T\. Xue, Z\. Chen, F\. Chen, S\. Utpala,et al\.Holistic agent leaderboard: the missing infrastructure for ai agent evaluation\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 98778–98849\.Cited by:[§1](https://arxiv.org/html/2609.21267#S1.p1.1)\.
- Kim and Porter \(2002\)J\. Kim and A\. PorterA history\-based test prioritization technique for regression testing in resource constrained environments\.InProceedings of the 24th international conference on software engineering,pp\. 119–129\.Cited by:[§8](https://arxiv.org/html/2609.21267#S8.SS0.SSS0.Px3.p1.1)\.
- Kipniset al\.\(2025\)A\. Kipnis, K\. Voudouris, L\. Schulze Buschoff, and E\. SchulzMetabench\-a sparse benchmark of reasoning and knowledge in large language models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 31734–31770\.Cited by:[§1](https://arxiv.org/html/2609.21267#S1.p2.1),[§8](https://arxiv.org/html/2609.21267#S8.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2025\)Y\. Li, J\. Ma, M\. Ballesteros, Y\. Benajiba, and G\. HorwoodActive evaluation acquisition for efficient LLM benchmarking\.InProceedings of the 42nd International Conference on Machine Learning,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267,pp\. 35581–35602\.External Links:[Link](https://proceedings.mlr.press/v267/li25bp.html)Cited by:[§8](https://arxiv.org/html/2609.21267#S8.SS0.SSS0.Px1.p1.1)\.
- Lord \(1980\)F\. M\. LordApplications of item response theory to practical testing problems\.Routledge\.Cited by:[§8](https://arxiv.org/html/2609.21267#S8.SS0.SSS0.Px1.p1.1)\.
- Luoet al\.\(2014\)Q\. Luo, F\. Hariri, L\. Eloussi, and D\. MarinovAn empirical analysis of flaky tests\.InProceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering,pp\. 643–653\.Cited by:[§8](https://arxiv.org/html/2609.21267#S8.SS0.SSS0.Px3.p1.1)\.
- Machalicaet al\.\(2019\)M\. Machalica, A\. Samylkin, M\. Porth, and S\. ChandraPredictive test selection\.In2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice \(ICSE\-SEIP\),pp\. 91–100\.Cited by:[§8](https://arxiv.org/html/2609.21267#S8.SS0.SSS0.Px3.p1.1)\.
- Mialonet al\.\(2024\)G\. Mialon, C\. Fourrier, T\. Wolf, Y\. LeCun, and T\. ScialomGaia: a benchmark for general ai assistants\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 9025–9049\.Cited by:[§8](https://arxiv.org/html/2609.21267#S8.SS0.SSS0.Px2.p1.1)\.
- Perlitzet al\.\(2024\)Y\. Perlitz, E\. Bandel, A\. Gera, O\. Arviv, L\. E\. Dor, E\. Shnarch, N\. Slonim, M\. Shmueli\-Scheuer, and L\. ChoshenEfficient benchmarking \(of language models\)\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 2519–2536\.Cited by:[§1](https://arxiv.org/html/2609.21267#S1.p2.1),[§8](https://arxiv.org/html/2609.21267#S8.SS0.SSS0.Px1.p1.1)\.
- Poloet al\.\(2024\)F\. M\. Polo, L\. Weber, L\. Choshen, Y\. Sun, G\. Xu, and M\. YurochkinTinyBenchmarks: evaluating llms with fewer examples\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[§1](https://arxiv.org/html/2609.21267#S1.p2.1),[§1](https://arxiv.org/html/2609.21267#S1.p4.1),[§3\.3\.2](https://arxiv.org/html/2609.21267#S3.SS3.SSS2.p1.1),[§3\.3\.3](https://arxiv.org/html/2609.21267#S3.SS3.SSS3.Px2.p2.3),[§3\.3\.3](https://arxiv.org/html/2609.21267#S3.SS3.SSS3.p1.1),[§3\.3](https://arxiv.org/html/2609.21267#S3.SS3.p1.1),[§5\.3\.1](https://arxiv.org/html/2609.21267#S5.SS3.SSS1.p3.1),[§8](https://arxiv.org/html/2609.21267#S8.SS0.SSS0.Px1.p1.1)\.
- Rothermel and Harrold \(1997\)G\. Rothermel and M\. J\. HarroldA safe, efficient regression test selection technique\.ACM Transactions on Software Engineering and Methodology \(TOSEM\)6\(2\),pp\. 173–210\.Cited by:[§8](https://arxiv.org/html/2609.21267#S8.SS0.SSS0.Px3.p1.1)\.
- Saranathanet al\.\(2025\)G\. Saranathan, C\. Xu, M\. P\. Alam, T\. Kumar, M\. Foltin, S\. Y\. Wong, and S\. BhattacharyaSubLIME: subset selection via rank correlation prediction for data\-efficient llm evaluation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 30572–30593\.Cited by:[§8](https://arxiv.org/html/2609.21267#S8.SS0.SSS0.Px1.p1.1)\.
- Truonget al\.\(2025\)S\. T\. Truong, Y\. Tu, P\. Liang, B\. Li, and S\. KoyejoReliable and efficient amortized model\-based evaluation\.InProceedings of the 42nd International Conference on Machine Learning,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267,pp\. 60238–60265\.External Links:[Link](https://proceedings.mlr.press/v267/truong25c.html)Cited by:[§1](https://arxiv.org/html/2609.21267#S1.p2.1),[§1](https://arxiv.org/html/2609.21267#S1.p4.1),[§3\.4](https://arxiv.org/html/2609.21267#S3.SS4.p1.1),[§8](https://arxiv.org/html/2609.21267#S8.SS0.SSS0.Px1.p1.1)\.
- Viveket al\.\(2024\)R\. Vivek, K\. Ethayarajh, D\. Yang, and D\. KielaAnchor points: benchmarking models with much fewer examples\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1576–1601\.Cited by:[§1](https://arxiv.org/html/2609.21267#S1.p2.1),[§8](https://arxiv.org/html/2609.21267#S8.SS0.SSS0.Px1.p1.1)\.
- Yaoet al\.\(2024\)S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhanτ\\tau\-Bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.External Links:2406\.12045,[Link](https://arxiv.org/abs/2406.12045)Cited by:[§1](https://arxiv.org/html/2609.21267#S1.p1.1),[§8](https://arxiv.org/html/2609.21267#S8.SS0.SSS0.Px2.p1.1)\.
- Yoo and Harman \(2012\)S\. Yoo and M\. HarmanRegression testing minimization, selection and prioritization: a survey\.Software testing, verification and reliability22\(2\),pp\. 67–120\.Cited by:[§8](https://arxiv.org/html/2609.21267#S8.SS0.SSS0.Px3.p1.1)\.
- Yuanet al\.\(2025\)P\. Yuan, Y\. Zhang, S\. Feng, Y\. Li, X\. Wang, J\. Shi, C\. Tan, B\. Pan, Y\. Hu, and K\. LiBeyond one\-size\-fits\-all: tailored benchmarks for efficient evaluation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15591–15615\.Cited by:[§1](https://arxiv.org/html/2609.21267#S1.p2.1),[§8](https://arxiv.org/html/2609.21267#S8.SS0.SSS0.Px1.p1.1)\.

## Appendix AAdditional Results

This appendix reports the complete numerical results underlying the paper’s plots\. The guide below follows the order of the corresponding main\-text analyses and points to the relevant figures and tables\. In matched\-budget method comparisons, boldface marks the best displayed value and underlining marks the second\-best; table panel headings state the metric direction\.

### A\.1Guide to the Complete Results

##### Headline method comparison\.

The exact values behind the principal accuracy–efficiency comparison is in Table[1](https://arxiv.org/html/2609.21267#A1.T1); it reports executed questions, execution fraction, pass\-rate MAE, Spearman correlation, and Kendall’sτ\\taufor every method and operating point in Figures[2](https://arxiv.org/html/2609.21267#S5.F2)and[3](https://arxiv.org/html/2609.21267#S5.F3)\. Table[2](https://arxiv.org/html/2609.21267#A1.T2)gives the corresponding threshold\-level results for historical caching\. Together, these tables provide the complete numerical record for the two headline plots\.

##### Fixed\-subset design results\.

Figures[8](https://arxiv.org/html/2609.21267#A1.F8)and[9](https://arxiv.org/html/2609.21267#A1.F9)give the full ranking\-fidelity curves for the selection\-by\-estimation comparison whose MAE results appear in Figure[4](https://arxiv.org/html/2609.21267#S5.F4)\. Table[3](https://arxiv.org/html/2609.21267#A1.T3)then collects all three metrics for every selector, estimator, and budget\.

##### Adaptive\-model design results\.

Figure[10](https://arxiv.org/html/2609.21267#A1.F10)supplies the Spearman and Kendall views corresponding to the MAE comparison in Figure[5](https://arxiv.org/html/2609.21267#S5.F5)\. Table[4](https://arxiv.org/html/2609.21267#A1.T4)reports the exact MAE and rank correlations for Rasch adaptive testing, multidimensional 2PL adaptive testing, and random sampling at every evaluated budget\.

##### Cross\-agent transfer results\.

Table[5](https://arxiv.org/html/2609.21267#A1.T5)first defines the five held\-out families and their run counts\. For the aggregate transfer result, Figure[11](https://arxiv.org/html/2609.21267#A1.F11)and Table[6](https://arxiv.org/html/2609.21267#A1.T6)provide the pooled ranking curves and all exact pooled metrics\. Figures[12](https://arxiv.org/html/2609.21267#A1.F12)–[14](https://arxiv.org/html/2609.21267#A1.F14)and Tables[7](https://arxiv.org/html/2609.21267#A1.T7)–[11](https://arxiv.org/html/2609.21267#A1.T11)report the same results separately for each agent family\.

##### Calibration\-window sensitivity\.

Figure[15](https://arxiv.org/html/2609.21267#A1.F15)gives the Spearman and Kendall sensitivity curves that complement the MAE results in Figure[7](https://arxiv.org/html/2609.21267#S6.F7)\. Table[12](https://arxiv.org/html/2609.21267#A1.T12)reports every window’s start day, number of calibration runs, subset size, and three fidelity metrics\.

Table 1:Complete fixed\-budget headline results on 287 held\-out runs\. Within each budget and metric, bold marks the best displayed value and underlining marks the second\-best; emphasis is descriptive and does not denote statistical significance\.Table 2:Complete historical\-caching threshold sweep on 287 held\-out runs\. Rows are ordered by mean execution fraction\. MAE is in percentage points\.Figure 8:Spearman ranking fidelity for the fixed\-subset selection\-by\-estimation comparison\. Higher values are better\. All methods use the same 287 held\-out evaluation runs and the same question budgets; random sampling is averaged over 20 seeds\.Figure 9:Kendall’sτ\\tauranking fidelity for the fixed\-subset selection\-by\-estimation comparison\. Higher values are better\. The test population, budgets, and random\-sampling seeds match Figure[8](https://arxiv.org/html/2609.21267#A1.F8)\.Table 3:Complete fixed\-subset selection and estimation results on 287 held\-out runs\. Bold and underlining mark the best and second\-best displayed values at each budget, respectively\.\(a\)Spearman correlation\.\(b\)Kendall’sτ\\tau\.
Figure 10:Ranking fidelity for the adaptive\-model comparison\. Higher values are better\. Both panels use the same 287 held\-out evaluation runs and question budgets as Figure[5](https://arxiv.org/html/2609.21267#S5.F5); random sampling is averaged over 20 seeds\.Table 4:Complete adaptive\-model comparison on 287 held\-out runs\. Bold and underlining mark the best and second\-best displayed values at each budget, respectively\.Table 5:Breakdown of transferred evaluation runs by anonymized agent family\. The labels match the family\-level figures and tables\.\(a\)Spearman correlation\.\(b\)Kendall’sτ\\tau\.
Figure 11:Pooled cross\-agent ranking fidelity for a difficulty\-stratified gp\-IRT subset calibrated on the focal family and transferred unchanged to 299 runs from five held\-out families\. Higher values are better\. Random sampling is averaged over 20 seeds\.Table 6:Complete pooled cross\-agent transfer results\. Original denotes the 287 calibration\-agent test runs; Transfer denotes 299 pooled runs from five additional families\. Emphasis is computed separately within the original and transferred populations\.Figure 12:Cross\-agent pass\-rate MAE by anonymized family for the transferred difficulty\-stratified gp\-IRT subset and 20\-seed random sampling\. Lower is better\. Each panel compares methods on the same held\-out runs within that family\.Figure 13:Cross\-agent Spearman ranking fidelity by anonymized family\. Higher is better\. Correlations are computed within family, and random sampling is averaged over 20 seeds\.Figure 14:Cross\-agent Kendall’sτ\\tauranking fidelity by anonymized family\. Higher is better\. Correlations are computed within family, and random sampling is averaged over 20 seeds\.Table 7:Complete transfer results for anonymized Family A\. Bold and underlining mark the best and second\-best displayed values at each budget, respectively\.Table 8:Complete transfer results for anonymized Family B\. Bold and underlining mark the best and second\-best displayed values at each budget, respectively\.Table 9:Complete transfer results for anonymized Family C\. Bold and underlining mark the best and second\-best displayed values at each budget, respectively\.Table 10:Complete transfer results for anonymized Family D\. Bold and underlining mark the best and second\-best displayed values at each budget, respectively\.Table 11:Complete transfer results for anonymized Family E\. Bold and underlining mark the best and second\-best displayed values at each budget, respectively\.\(a\)Spearman correlation\.\(b\)Kendall’sτ\\tau\.
Figure 15:Ranking\-fidelity sensitivity of difficulty\-stratified fixed subsets to nested calibration windows ending Day 28\. Higher values are better\. Windows vary jointly in historical range and calibration\-run count; all curves use the same 287 held\-out evaluation runs\.Table 12:Complete calibration\-window sensitivity results\. All windows end Day 28 and use the same 287 held\-out runs\. MAE is in percentage points\. Values are not emphasized because the windows form a sensitivity sweep rather than prespecified competing methods\.