Beyond "AI Helps Humans": Decision-Targeted Evaluation Design for Human-Agent Teams in the Agentic Era

# Beyond “AI Helps Humans”: Decision-Targeted Evaluation Design for Human–Agent Teams in the Agentic Era
Source: [https://arxiv.org/html/2609.05527](https://arxiv.org/html/2609.05527)
\\workshoptitle

Human\-AI Coevolution: Measuring Human\-Agent Teams in the Agentic Era

Hamed KhosraviAffiliation:H\. Milton Stewart School of Industrial and Systems EngineeringAffiliation:Georgia Institute of TechnologyAffiliation:Atlanta, GA 30332Email:[hkhosravi7@gatech\.edu](mailto:)Xiaoming HuoAffiliation:H\. Milton Stewart School of Industrial and Systems EngineeringAffiliation:Georgia Institute of TechnologyAffiliation:Atlanta, GA 30332Email:[huo@gatech\.edu](mailto:)

###### Abstract

Wherever a coding agent works under engineer supervision, or a clinical model assists a radiologist, the deployment question is whether to keep the human–AI workflow or replace it with the human alone or the agent alone\. The human–AI workflow is worth keeping only if it beats both of those alternatives\. Yet once it is deployed, neither alternative outcome is observed: recovering one means replaying the task with the human alone or with the agent alone, and every replay costs expert time or compute\. Under a fixed replay budget, the design question is therefore which tasks should be more likely to receive a human\-only replay, and which an agent\-only replay\. Existing methods do not directly target this decision\. Agent benchmarks do not choose which missing baseline to measure, variance\-based sampling ignores which of the two comparisons is closer to failing, and Bayesian information methods focus on learning model parameters instead of making the deployment decision\. We proposeTEAM\-Design, a rule that gives every task two replay probabilities, one per baseline\. It raises a probability where the missing baseline outcome is hard to predict from what is already known about the task and where that comparison is harder to establish, and lowers it where replay is expensive\. We prove that the rule solves this budgeted design problem, and that drawing the replays at random from recorded probabilities still controls the chance of wrongly declaring that the human–AI workflow beats both\. We reanalyze 6 clinical settings, where no human–AI workflow beats both alternatives, and a coding benchmark, where one does, then evaluateTEAM\-Designon synthetic designs and on a semi\-synthetic design built from a real chest X\-ray reader study\.TEAM\-Designworks best when one of the two comparisons is clearly harder to settle than the other, and can do worse than variance\-based allocation when the two are similarly difficult\.

## 1Introduction

#### The deployment question\.

Human and AI systems increasingly work together\. In software, coding agents write code, use tools, and run tests while engineers review and redirect their work\. Anthropic reports that its employees use Claude frequently yet can fully delegate only00–20%20\\%of their work\[[1](https://arxiv.org/html/2609.05527#bib.bib1)\], and a fintech team deploying Claude Code is moving toward developers reviewing the agent’s pull requests\[[2](https://arxiv.org/html/2609.05527#bib.bib2)\]\. In medicine, AI models assist clinicians who remain responsible for the final decision\. In both settings, the organization observes the outcome of the human–AI workflow it actually runs, in which the human and the agent work the same task together\. What it does not observe is how the same task would have turned out with the human alone or the agent alone\. Hence the deployment question we study: is the human–AI workflow worth keeping if either alternative could be used instead?

#### The missing baselines\.

Answering this requires knowing how the same task would have turned out under each alternative, but neither outcome is recorded\. An agent\-only replay uses model calls, compute, and tools\. A human\-only replay may require a senior engineer to redo the task independently, or a clinician to read the case without AI assistance\. Running both alternatives on every task can cost more than the evaluation itself is worth, which leaves a practical problem\.

> *Given a fixed replay budget, which tasks should be more likely to receive a human\-only replay, and which an agent\-only replay, so that we can determine whether the human–AI workflow beats both alternatives?*

We do not choose a fixed subset of tasks in advance\. Each task receives two replay probabilities, one per baseline, and the two replay decisions are drawn separately, so a task may receive neither replay, one, or both\. A higher probability means that baseline is replayed more often among tasks carrying similar information\.

#### The two settings\.

In coding, human and agent replays differ greatly in cost\. In medicine, some studies have clinicians evaluate the same cases with and without AI support\[[22](https://arxiv.org/html/2609.05527#bib.bib22),[23](https://arxiv.org/html/2609.05527#bib.bib23)\], so we use the clinical data to study the two comparisons and coding to motivate unequal replay costs\. Some coding benchmarks report one overall score for each of the human alone, the agent alone, and the human–AI workflow\[[10](https://arxiv.org/html/2609.05527#bib.bib10),[11](https://arxiv.org/html/2609.05527#bib.bib11)\]\. Those are averages across tasks, not three outcomes recorded on each individual task, so no task has a gain to estimate and selective replay cannot be tested on them\.

#### Related work\.

Prior work provides pieces of this problem, but not the full evaluation design\. A framework already compares the human alone, human with AI, and AI alone\[[3](https://arxiv.org/html/2609.05527#bib.bib3)\], but it does not decide which tasks should receive a human\-only or agent\-only replay\. Work on agent evaluation shows that accuracy alone can miss evaluation cost and deployment validity\[[4](https://arxiv.org/html/2609.05527#bib.bib4),[5](https://arxiv.org/html/2609.05527#bib.bib5)\]\. Active learning and experimental design choose data for a downstream decision\[[6](https://arxiv.org/html/2609.05527#bib.bib6),[7](https://arxiv.org/html/2609.05527#bib.bib7)\], and off\-policy methods design collection for estimating policy performance\[[8](https://arxiv.org/html/2609.05527#bib.bib8),[9](https://arxiv.org/html/2609.05527#bib.bib9)\]\. Classical variance\-based methods, including Neyman allocation and cost\-aware sampling, spend measurements where outcomes are uncertain and hold back where measurement is costly\[[16](https://arxiv.org/html/2609.05527#bib.bib16),[17](https://arxiv.org/html/2609.05527#bib.bib17)\], but they do not account for which of our two comparisons is closer to failing\. Ranking and best\-system selection methods also allocate measurements efficiently\[[12](https://arxiv.org/html/2609.05527#bib.bib12),[13](https://arxiv.org/html/2609.05527#bib.bib13),[14](https://arxiv.org/html/2609.05527#bib.bib14),[15](https://arxiv.org/html/2609.05527#bib.bib15)\], but they typically choose among outcomes that can be measured directly\. Sizing a second stage from a first\-stage estimate is classical two\-stage sampling\[[19](https://arxiv.org/html/2609.05527#bib.bib19)\]\. Here, both baseline outcomes are missing and costly to obtain, and the human–AI workflow must beat both of them\.

#### Contributions\.

We make four contributions\.

- •A design that targets the deployment decision\.We give the first evaluation design that chooses which missing baseline to measure on which task, so that a fixed replay budget goes as far as possible toward settling whether the human–AI workflow beats both alternatives\.TEAM\-Designassigns every task a human\-only and an agent\-only replay probability in closed form, buying more replay where the missing baseline is hard to predict and where that comparison is harder to establish, and less where replay is expensive\. The quantities it needs are learned once from a small*pilot*set of tasks on which the workflow, the human alone, and the agent alone have all been run\.
- •The rule is optimal, not heuristic\.We show that this design problem is a convex minimax program: because the workflow must beat both alternatives, the objective maximizes over the two comparisons to select whichever needs more evidence, then minimizes over the replay probabilities to make that one as easy as possible to resolve\. We prove that the square\-root rule ofTEAM\-Designis its exact solution\.
- •The decision stays valid\.We prove that choosing the replay probabilities from the pilot and then drawing the replays at random from the recorded probabilities still controls the chance of wrongly declaring that the workflow beats both, and that requiring both bounds to clear needs no multiplicity correction\.
- •The problem is real, and the rule works\.Across 6 clinical study settings, assisted clinicians perform better than clinicians alone but worse than the model alone, so none of these human–AI workflows beats both alternatives\. A coding benchmark designed for collaboration shows the opposite pattern\. On a semi\-synthetic design built from a real chest X\-ray study,TEAM\-Designgives the lowest estimation error for the gain against the harder baseline at every budget below full replay\. Its advantage is strongest when one comparison is clearly harder than the other, and it can perform worse when the two are similarly difficult\.

#### Paper organization\.

Section[2](https://arxiv.org/html/2609.05527#S2)defines the evaluation problem\. Section[3](https://arxiv.org/html/2609.05527#S3)developsTEAM\-Design\. Section[4](https://arxiv.org/html/2609.05527#S4)gives the theoretical guarantees\. Section[5](https://arxiv.org/html/2609.05527#S5)evaluates the method and studies when it helps\.

## 2Problem formulation

A task can be handled three ways: by the human and the agent together, by the human alone, or by the agent alone\. Only the first is deployed, and*the workflow*always refers to that joint configuration, never to a human\-only or agent\-only one; the other two are its*baselines*\. For a task from the deployment population we observe informationW∈𝒲W\\in\\mathcal\{W\}, such as tool calls, test failures, or runtime, together with the workflow’s outcomeTT\. The baseline outcomesYHY\_\{H\}andYAY\_\{A\}are missing unless the task is replayed\.

The workflow’s average gain over each baseline is

ΔH=𝔼⁡\[T−YH\],ΔA=𝔼⁡\[T−YA\],\\Delta\_\{H\}=\\mathbb\{E\}\[T\-Y\_\{H\}\],\\qquad\\Delta\_\{A\}=\\mathbb\{E\}\[T\-Y\_\{A\}\],\(1\)where𝔼⁡\[⋅\]\\mathbb\{E\}\[\\cdot\]averages over tasks\. We writej∈\{H,A\}j\\in\\\{H,A\\\}for either baseline, soΔj\>0\\Delta\_\{j\}\>0means the workflow beats baselinejjon average\. Letδj≥0\\delta\_\{j\}\\geq 0be the improvement required over that baseline\. The workflow is kept only when

ΔH\>δHandΔA\>δA;\\Delta\_\{H\}\>\\delta\_\{H\}\\quad\\text\{and\}\\quad\\Delta\_\{A\}\>\\delta\_\{A\};\(2\)settingδH=δA=0\\delta\_\{H\}=\\delta\_\{A\}=0asks only whether it beats both alternatives\.

Now take an evaluation pool ofnnindependent tasks, subscripted byii\. We observe\(Wi,Ti\)\(W\_\{i\},T\_\{i\}\), whileYi​jY\_\{ij\}is missing unless baselinejjis replayed at costcj​\(Wi\)c\_\{j\}\(W\_\{i\}\)\. Since we cannot replay both baselines on every task, we choose a pair of*replay rules*\. A rule is a function of the task information,qj:𝒲→\[qj,min,1\]q\_\{j\}:\\mathcal\{W\}\\to\[q\_\{j,\\min\},1\], so a task with informationWWhas baselinejjreplayed with probabilityqj​\(W\)q\_\{j\}\(W\), andRj\|W∼Bernoulli⁡\{qj​\(W\)\}R\_\{j\}\\mid W\\sim\\mathrm\{Bernoulli\}\\\{q\_\{j\}\(W\)\\\}indicates that it happens\. The two draws are separate, so a task may receive neither replay, one, or both\.

LetBBbe the average replay budget per task, so that

𝔼⁡\[cH​\(W\)​qH​\(W\)\+cA​\(W\)​qA​\(W\)\]≤B,qj​\(W\)∈\[qj,min,1\]\.\\mathbb\{E\}\\\!\\left\[c\_\{H\}\(W\)q\_\{H\}\(W\)\+c\_\{A\}\(W\)q\_\{A\}\(W\)\\right\]\\leq B,\\qquad q\_\{j\}\(W\)\\in\[q\_\{j,\\min\},1\]\.\(3\)The floorqj,min\>0q\_\{j,\\min\}\>0leaves every task some chance of revealing each baseline\.

Writeσj2​\(qj\)/n\\sigma\_\{j\}^\{2\}\(q\_\{j\}\)/nfor the approximate variance of the gain estimate built from the replayed tasks\. Replay is what makes this variance a design choice: baselinejjis observed only on the tasks where it is replayed, so raisingqj​\(W\)q\_\{j\}\(W\)supplies more of the outcomes that inform comparisonjjand lowersσj2\\sigma\_\{j\}^\{2\}\. The variance therefore depends on the rule as a whole and not on any single probability, and \([8](https://arxiv.org/html/2609.05527#S3.E8)\) gives the exact form\. A comparison is hard when that estimate is noisy or when the true gain sits close to the improvement required of it: forΔj\>δj\\Delta\_\{j\}\>\\delta\_\{j\}, the number of tasks needed to establish comparisonjjgrows withσj2​\(qj\)/\(Δj−δj\)2\\sigma\_\{j\}^\{2\}\(q\_\{j\}\)/\(\\Delta\_\{j\}\-\\delta\_\{j\}\)^\{2\}, which we take as its difficulty\. Both comparisons must pass, so improving an already easy one does not resolve the decision\. The harder comparison therefore sets the evidence required, and we choose both replay rules to make it as easy as possible:

minqH​\(⋅\),qA​\(⋅\)⁡maxj∈\{H,A\}​σj2​\(qj\)\(Δj−δj\)2subject to \([3](https://arxiv.org/html/2609.05527#S2.E3)\)\.\\min\_\{q\_\{H\}\(\\cdot\),\\,q\_\{A\}\(\\cdot\)\}\\max\_\{j\\in\\\{H,A\\\}\}\\frac\{\\sigma\_\{j\}^\{2\}\(q\_\{j\}\)\}\{\(\\Delta\_\{j\}\-\\delta\_\{j\}\)^\{2\}\}\\qquad\\text\{subject to \\eqref\{eq:budget\}\.\}\(4\)The variables in \([4](https://arxiv.org/html/2609.05527#S2.E4)\) are the two rules themselves, one probability for every value of the task information, and \([3](https://arxiv.org/html/2609.05527#S2.E3)\) defines the feasible set they are drawn from\. Comparisonjjdepends only on its own ruleqjq\_\{j\}; the two are coupled through the shared budget \([3](https://arxiv.org/html/2609.05527#S2.E3)\) alone, so replay spent on one comparison is replay the other cannot have\. BothΔj\\Delta\_\{j\}andσj2​\(qj\)\\sigma\_\{j\}^\{2\}\(q\_\{j\}\)depend on the missing baseline outcomes, so neither is known in advance\. Section[3](https://arxiv.org/html/2609.05527#S3)introducesTEAM\-Design, which estimates them from a pilot and uses them to set the replay probabilities\.

## 3TEAM\-Design

Section[2](https://arxiv.org/html/2609.05527#S2)asks for two replay rules,qHq\_\{H\}andqAq\_\{A\}\. The best rules depend on quantities we do not know: how close each comparison sits to its threshold, and where the missing baseline outcomes are hard to predict\.TEAM\-Designestimates both from a small separate pilot, then uses them to set the rules for the evaluation pool\.

#### Learning from a pilot\.

A*pilot*is a set ofnpn\_\{\\mathrm\{p\}\}tasks, separate from the evaluation pool, on which all three configurations are run, so we observeTT,YHY\_\{H\}, andYAY\_\{A\}together; write𝔼~p\\widetilde\{\\mathbb\{E\}\}\_\{\\mathrm\{p\}\}for an average over them\. Decoration carries the source from here on: a tilde marks a quantity computed on the pilot and used to plan the replay, a hat marks one computed on the evaluation pool and reported, and a plain symbol is the population quantity they estimate\. Running all three configurations is expensive, so the pilot is small, and its cost sits outside the budgetBBof \([3](https://arxiv.org/html/2609.05527#S2.E3)\), which governs the evaluation pool\.

With both baselines observed, the pilot estimates the gain directly asΔ~j=𝔼~p​\[T−Yj\]\\widetilde\{\\Delta\}\_\{j\}=\\widetilde\{\\mathbb\{E\}\}\_\{\\mathrm\{p\}\}\[T\-Y\_\{j\}\], which is \([1](https://arxiv.org/html/2609.05527#S2.E1)\) with the pilot average in place of the deployment average\.TEAM\-DesignusesΔ~j\\widetilde\{\\Delta\}\_\{j\}only to plan the replay; the reported answer comes from the evaluation pool\.

The pilot also shows which missing outcomes are hard to predict fromWW\. Letm~j\\widetilde\{m\}\_\{j\}be a predictor of the baseline outcomeYjY\_\{j\}fitted on the pilot, and write

sj​\(W\)=𝔼⁡\[\{Yj−m~j​\(W\)\}2\|W\]s\_\{j\}\(W\)=\\mathbb\{E\}\\\!\\left\[\\\{Y\_\{j\}\-\\widetilde\{m\}\_\{j\}\(W\)\\\}^\{2\}\\;\\middle\|\\;W\\right\]\(5\)for its expected squared error on tasks with informationWW, the*prediction difficulty*of baselinejjthere\. Conditionally on the pilot,m~j\\widetilde\{m\}\_\{j\}is a fixed function, sosjs\_\{j\}is a population quantity\. A largersj​\(W\)s\_\{j\}\(W\)means the baseline is harder to predict, so replaying it reveals more\. Writes~j​\(W\)\\widetilde\{s\}\_\{j\}\(W\)for the pilot estimate\. Appendix[B\.1](https://arxiv.org/html/2609.05527#A2.SS1.SSS0.Px8)gives both models and the held\-out fitting that keepss~j\\widetilde\{s\}\_\{j\}from making a baseline look more predictable than it is\.

#### Estimating the gains after selective replay\.

Take thennevaluation tasks, disjoint from the pilot and indexed byi=1,…,ni=1,\\ldots,n\. We observe\(Wi,Ti\)\(W\_\{i\},T\_\{i\}\), andYi​jY\_\{ij\}only when baselinejjis replayed, which happens with probabilityqj​\(Wi\)q\_\{j\}\(W\_\{i\}\)and is recorded byRi​j∈\{0,1\}R\_\{ij\}\\in\\\{0,1\\\}\. For each task, define a scoreψi​j\\psi\_\{ij\}that estimates the gain over baselinejjfrom that task alone,

ψi​j=Ti−m~j​\(Wi\)−Ri​jqj​\(Wi\)​\{Yi​j−m~j​\(Wi\)\}\.\\psi\_\{ij\}=T\_\{i\}\-\\widetilde\{m\}\_\{j\}\(W\_\{i\}\)\-\\frac\{R\_\{ij\}\}\{q\_\{j\}\(W\_\{i\}\)\}\\left\\\{Y\_\{ij\}\-\\widetilde\{m\}\_\{j\}\(W\_\{i\}\)\\right\\\}\.\(6\)The first term uses the predicted baseline in place of the missing one; when the task is replayed, the second corrects that prediction by the observed error\. The factor1/qj​\(Wi\)1/q\_\{j\}\(W\_\{i\}\)accounts for how rarely corrections are seen: atqj​\(Wi\)=0\.2q\_\{j\}\(W\_\{i\}\)=0\.2only one task in five revealsYi​jY\_\{ij\}, so an observed correction carries weight55\. WhenRi​j=0R\_\{ij\}=0the correction vanishes and the missingYi​jY\_\{ij\}is never needed\.

Averaging these scores gives the estimated gain over baselinejj,

Δ^j=1n​∑i=1nψi​j\.\\widehat\{\\Delta\}\_\{j\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\psi\_\{ij\}\.\(7\)Because the replay probability is known,𝔼⁡\[Ri​j/qj​\(Wi\)∣Wi\]=1\\mathbb\{E\}\[R\_\{ij\}/q\_\{j\}\(W\_\{i\}\)\\mid W\_\{i\}\]=1and𝔼⁡\[Δ^j\]=Δj\\mathbb\{E\}\[\\widehat\{\\Delta\}\_\{j\}\]=\\Delta\_\{j\}however poorly the pilot predicts; a poor predictor only makes the estimate noisier\.

#### Choosing the replay probabilities\.

The rule needs three things: how much variance replay can remove, how close each comparison sits to its threshold, and what a replay costs\. We take the first two in turn, then put them together\.

Writeσj2​\(qj\)\\sigma\_\{j\}^\{2\}\(q\_\{j\}\)for the variance of one task’s score under ruleqjq\_\{j\}, so thatΔ^j\\widehat\{\\Delta\}\_\{j\}has varianceσj2​\(qj\)/n\\sigma\_\{j\}^\{2\}\(q\_\{j\}\)/n\. It splits into a part replay cannot touch and a part it can:

σj2​\(qj\)=κj\+𝔼⁡\[sj​\(W\)qj​\(W\)\]\.\\sigma\_\{j\}^\{2\}\(q\_\{j\}\)=\\kappa\_\{j\}\+\\mathbb\{E\}\\\!\\left\[\\frac\{s\_\{j\}\(W\)\}\{q\_\{j\}\(W\)\}\\right\]\.\(8\)Appendix[A\.2](https://arxiv.org/html/2609.05527#A1.SS2)derives this\. The correction in \([6](https://arxiv.org/html/2609.05527#S3.E6)\) has mean zero given the task, because replay is drawn fromWWalone, so it does not covary withT−YjT\-Y\_\{j\}; its own variance carries a factor\{1−qj​\(W\)\}/qj​\(W\)\\\{1\-q\_\{j\}\(W\)\\\}/q\_\{j\}\(W\)\. The first term,κj=Var⁡\(T−Yj\)−𝔼⁡\[sj​\(W\)\]\\kappa\_\{j\}=\\operatorname\{Var\}\(T\-Y\_\{j\}\)\-\\mathbb\{E\}\[s\_\{j\}\(W\)\]withVar\\operatorname\{Var\}the population variance, holds noqjq\_\{j\}and is therefore fixed whatever we buy;κ~j\\widetilde\{\\kappa\}\_\{j\}denotes its pilot estimate, formed by the same expression with pilot averages\. In the second, a task with informationWWcontributes its prediction difficultysj​\(W\)s\_\{j\}\(W\)of \([5](https://arxiv.org/html/2609.05527#S3.E5)\) divided by its replay probability, so replay buys the most where the missing baseline is hardest to predict\. Settingqj≡1q\_\{j\}\\equiv 1returnsσj2=Var⁡\(T−Yj\)\\sigma\_\{j\}^\{2\}=\\operatorname\{Var\}\(T\-Y\_\{j\}\), the variance when no outcome is missing, which is what the−𝔼​\[sj​\(W\)\]\-\\mathbb\{E\}\[s\_\{j\}\(W\)\]inκj\\kappa\_\{j\}is for\.

The design also needs to know how close comparisonjjsits to its thresholdδj\\delta\_\{j\}\. The true gap is unknown, so we use the pilot estimate, held away from zero to stop a noisy estimate that lands near its threshold from taking almost the entire budget:

d~j2=\{max⁡\(Δ~j−δj,0\)\}2\+τ2,\\widetilde\{d\}\_\{j\}^\{2\}=\\left\\\{\\max\(\\widetilde\{\\Delta\}\_\{j\}\-\\delta\_\{j\},0\)\\right\\\}^\{2\}\+\\tau^\{2\},\(9\)whereτ\>0\\tau\>0is a small constant fixed in advance and the maximum treats the gap as zero when the pilot gain falls below its threshold\. Thisd~j2\\widetilde\{d\}\_\{j\}^\{2\}stands in for the unknown\(Δj−δj\)2\(\\Delta\_\{j\}\-\\delta\_\{j\}\)^\{2\}of Section[2](https://arxiv.org/html/2609.05527#S2), and Section[5\.4](https://arxiv.org/html/2609.05527#S5.SS4)reports thatτ\\tauworks best on the scale of the pilot’s uncertainty\.

Difficulty is variance over squared gap, so the estimated difficulty of comparisonjjis \([8](https://arxiv.org/html/2609.05527#S3.E8)\) divided byd~j2\\widetilde\{d\}\_\{j\}^\{2\}, withκ~j\\widetilde\{\\kappa\}\_\{j\}ands~j\\widetilde\{s\}\_\{j\}in place of the population quantities\. Two multipliers turn the constrained problem \([4](https://arxiv.org/html/2609.05527#S2.E4)\) into one that can be solved task by task:λH,λA≥0\\lambda\_\{H\},\\lambda\_\{A\}\\geq 0withλH\+λA=1\\lambda\_\{H\}\+\\lambda\_\{A\}=1price the two comparisons, andη\>0\\eta\>0prices the budget\. A task with informationWWthen contributes

λjd~j2​s~j​\(W\)qj​\(W\)\+η​cj​\(W\)​qj​\(W\),\\frac\{\\lambda\_\{j\}\}\{\\widetilde\{d\}\_\{j\}^\{2\}\}\\,\\frac\{\\widetilde\{s\}\_\{j\}\(W\)\}\{q\_\{j\}\(W\)\}\\;\+\\;\\eta\\,c\_\{j\}\(W\)\\,q\_\{j\}\(W\),difficulty on the left and cost on the right\. The first term falls like1/qj​\(W\)1/q\_\{j\}\(W\)and the second grows in proportion toqj​\(W\)q\_\{j\}\(W\), so the sum is smallest where the two balance, at

qj⋆​\(W\)=clip\[qj,min,1\]⁡\[λj​s~j​\(W\)η​d~j2​cj​\(W\)\]\.q\_\{j\}^\{\\star\}\(W\)=\\operatorname\{clip\}\_\{\[q\_\{j,\\min\},1\]\}\\left\[\\sqrt\{\\frac\{\\lambda\_\{j\}\\,\\widetilde\{s\}\_\{j\}\(W\)\}\{\\eta\\,\\widetilde\{d\}\_\{j\}^\{2\}\\,c\_\{j\}\(W\)\}\}\\right\]\.\(10\)The clipping,clip\[a,b\]⁡\(x\)=min⁡\{max⁡\(x,a\),b\}\\operatorname\{clip\}\_\{\[a,b\]\}\(x\)=\\min\\\{\\max\(x,a\),b\\\}, keeps every replay probability in\[qj,min,1\]\[q\_\{j,\\min\},1\]\. Theorem[4\.2](https://arxiv.org/html/2609.05527#S4.Thmtheorem2)shows that this balance point is the exact solution of \([4](https://arxiv.org/html/2609.05527#S2.E4)\), not an approximation to it\.

Equation \([10](https://arxiv.org/html/2609.05527#S3.E10)\) shows where the budget goes\. Within a baseline, the quantities that vary from task to task decide which tasks are replayed: replay rises wheres~j​\(W\)\\widetilde\{s\}\_\{j\}\(W\)is large, because that outcome is harder to predict, and falls wherecj​\(W\)c\_\{j\}\(W\)is large, because replay is more expensive\. Across the two baselines,λj\\lambda\_\{j\}andd~j2\\widetilde\{d\}\_\{j\}^\{2\}shift the budget toward the comparison that needs more evidence\. OnlyλH\\lambda\_\{H\}andη\\etaare left to set\. For a givenλH\\lambda\_\{H\}, withλA=1−λH\\lambda\_\{A\}=1\-\\lambda\_\{H\}, the multiplierη\\etais adjusted until the two rules together spend exactly the budgetBBof \([3](https://arxiv.org/html/2609.05527#S2.E3)\)\.TEAM\-DesignscansλH\\lambda\_\{H\}over\[0,1\]\[0,1\], computes both estimated difficulties at each value, and keeps the pair of rules whose larger difficulty is smallest\. Both are one\-dimensional searches, so no solver is needed\.

#### Making the final decision\.

For each evaluation taskii, writeqi​j⋆=qj⋆​\(Wi\)q\_\{ij\}^\{\\star\}=q\_\{j\}^\{\\star\}\(W\_\{i\}\), drawRi​HR\_\{iH\}andRi​AR\_\{iA\}separately fromBernoulli⁡\(qi​H⋆\)\\mathrm\{Bernoulli\}\(q\_\{iH\}^\{\\star\}\)andBernoulli⁡\(qi​A⋆\)\\mathrm\{Bernoulli\}\(q\_\{iA\}^\{\\star\}\), and record both probabilities\. Equation \([7](https://arxiv.org/html/2609.05527#S3.E7)\) then givesΔ^H\\widehat\{\\Delta\}\_\{H\}andΔ^A\\widehat\{\\Delta\}\_\{A\}\. Lettingσ^j2\\widehat\{\\sigma\}\_\{j\}^\{2\}be the empirical variance of the scoresψ1​j,…,ψn​j\\psi\_\{1j\},\\ldots,\\psi\_\{nj\}, the one\-sided lower confidence bound isLj=Δ^j−z1−α​σ^j/nL\_\{j\}=\\widehat\{\\Delta\}\_\{j\}\-z\_\{1\-\\alpha\}\\widehat\{\\sigma\}\_\{j\}/\\sqrt\{n\}, wherez1−αz\_\{1\-\\alpha\}is the\(1−α\)\(1\-\\alpha\)quantile of the standard normal distribution andα=0\.05\\alpha=0\.05\.

The human–AI workflow is declared to beat both alternatives only when

LH\>δHandLA\>δA\.L\_\{H\}\>\\delta\_\{H\}\\quad\\text\{and\}\\quad L\_\{A\}\>\\delta\_\{A\}\.\(11\)Requiring both bounds to clear their thresholds needs no multiplicity correction, for the reason given with Theorem[4\.3](https://arxiv.org/html/2609.05527#S4.Thmtheorem3)\. Algorithm[1](https://arxiv.org/html/2609.05527#alg1)states the procedure end to end, and Section[4](https://arxiv.org/html/2609.05527#S4)establishes that it remains valid when the rules are chosen from the pilot\.

Algorithm 1TEAM\-Designpseudocode1:Independent pilot of

npn\_\{\\mathrm\{p\}\}fully observed tasks; evaluation pool

\{\(Wi,Ti\)\}i=1n\\\{\(W\_\{i\},T\_\{i\}\)\\\}\_\{i=1\}^\{n\}; replay costs

cj​\(Wi\)c\_\{j\}\(W\_\{i\}\); budget

BB; floors

qj,minq\_\{j,\\min\}; stabilizer

τ\\tau; thresholds

δH,δA\\delta\_\{H\},\\delta\_\{A\}\.

2:From the pilot, estimate

Δ~j\\widetilde\{\\Delta\}\_\{j\},

m~j​\(W\)\\widetilde\{m\}\_\{j\}\(W\),

s~j​\(W\)\\widetilde\{s\}\_\{j\}\(W\), and

κ~j\\widetilde\{\\kappa\}\_\{j\}\(Appendix[A\.2](https://arxiv.org/html/2609.05527#A1.SS2)\), for

j∈\{H,A\}j\\in\\\{H,A\\\}\.

3:Compute the stabilized gap

d~j2\\widetilde\{d\}\_\{j\}^\{2\}from \([9](https://arxiv.org/html/2609.05527#S3.E9)\)\.

4:Search over

λH∈\[0,1\]\\lambda\_\{H\}\\in\[0,1\], with

λA=1−λH\\lambda\_\{A\}=1\-\\lambda\_\{H\}, and for each bisect

η\\etato meet the budget

BB, giving

qi​j⋆q\_\{ij\}^\{\\star\}from \([10](https://arxiv.org/html/2609.05527#S3.E10)\)\. Appendix[B\.1](https://arxiv.org/html/2609.05527#A2.SS1.SSS0.Px8)gives the budget range over which such an

η\\etaexists\.

5:Keep the

λH\\lambda\_\{H\}that minimizes the larger of the two estimated difficulties\.

6:For each task

ii, draw

Ri​H∼Bernoulli⁡\(qi​H⋆\)R\_\{iH\}\\sim\\mathrm\{Bernoulli\}\(q\_\{iH\}^\{\\star\}\)and

Ri​A∼Bernoulli⁡\(qi​A⋆\)R\_\{iA\}\\sim\\mathrm\{Bernoulli\}\(q\_\{iA\}^\{\\star\}\)separately, replay baseline

jjonly when

Ri​j=1R\_\{ij\}=1, and record every

qi​j⋆q\_\{ij\}^\{\\star\}\.

7:Compute

Δ^H,Δ^A\\widehat\{\\Delta\}\_\{H\},\\widehat\{\\Delta\}\_\{A\}from \([7](https://arxiv.org/html/2609.05527#S3.E7)\) and report the decision \([11](https://arxiv.org/html/2609.05527#S3.E11)\)\.

## 4Theoretical guarantees

Section[3](https://arxiv.org/html/2609.05527#S3)gave a practical rule for assigning replay probabilities\. Three results justify it\. First, we show why the harder comparison determines the approximate number of evaluation tasks needed\. Second, we show that the square\-root rule of \([10](https://arxiv.org/html/2609.05527#S3.E10)\) solves the minimax problem of Section[2](https://arxiv.org/html/2609.05527#S2)\. Third, we show that choosing the replay probabilities from the pilot and then drawing replays at random still controls the chance of wrongly declaring that the workflow beats both baselines\.

###### Proposition 4\.1\(The harder comparison determines evaluation effort\)\.

Assume both gains exceed their required thresholds and the two gain estimates are asymptotically normal\. To obtain one\-sided level\-α\\alphatests with joint power at least1−β1\-\\beta, it is sufficient, to first order, that

n≥\{z1−α\+z1−β/2\}2​maxj∈\{H,A\}​σj2​\(qj\)\(Δj−δj\)2\.n\\geq\\\{z\_\{1\-\\alpha\}\+z\_\{1\-\\beta/2\}\\\}^\{2\}\\max\_\{j\\in\\\{H,A\\\}\}\\frac\{\\sigma\_\{j\}^\{2\}\(q\_\{j\}\)\}\{\(\\Delta\_\{j\}\-\\delta\_\{j\}\)^\{2\}\}\.\(12\)Therefore, minimizing the objective in Section[2](https://arxiv.org/html/2609.05527#S2)minimizes this first\-order sample\-size requirement\.

The result explains the maximum in our design objective: the comparison requiring more evidence determines the evaluation effort\. Appendix[A\.1](https://arxiv.org/html/2609.05527#A1.SS1)proves it by asking each comparison to succeed with probability1−β/21\-\\beta/2, so that both hold with probability at least1−β1\-\\beta\. The bound says how many tasks are enough, not exactly how often both bounds will clear at a given budget, and Section[5\.4](https://arxiv.org/html/2609.05527#S5.SS4)measures the difference\.

###### Theorem 4\.2\(TEAM\-Designsolves the replay problem\)\.

SupposeΔj\>δj\\Delta\_\{j\}\>\\delta\_\{j\}forj∈\{H,A\}j\\in\\\{H,A\\\}, the replay costscj​\(W\)c\_\{j\}\(W\)are nonnegative and finite, and the budget is feasible\. On tasks withcj​\(W\)=0c\_\{j\}\(W\)=0takeqj⋆​\(W\)=1q\_\{j\}^\{\\star\}\(W\)=1\. Wherecj​\(W\)\>0c\_\{j\}\(W\)\>0, an optimal replay rule for the cost\-constrained design problem of Section[2](https://arxiv.org/html/2609.05527#S2)has the form

qj⋆​\(W\)=clip\[qj,min,1\]⁡\[λj​sj​\(W\)η​\(Δj−δj\)2​cj​\(W\)\],q\_\{j\}^\{\\star\}\(W\)=\\operatorname\{clip\}\_\{\[q\_\{j,\\min\},1\]\}\\left\[\\sqrt\{\\frac\{\\lambda\_\{j\}s\_\{j\}\(W\)\}\{\\eta\(\\Delta\_\{j\}\-\\delta\_\{j\}\)^\{2\}c\_\{j\}\(W\)\}\}\\right\],\(13\)for multipliersλH,λA≥0\\lambda\_\{H\},\\lambda\_\{A\}\\geq 0withλH\+λA=1\\lambda\_\{H\}\+\\lambda\_\{A\}=1and a budget multiplierη\>0\\eta\>0when the budget binds\. Complementary slackness gives positive weight only to a comparison that attains the maximum in the design objective, so both multipliers are positive where the budget lets the two comparisons equalize\.

Replacing the unknown quantities by the pilot estimates gives theTEAM\-Designrule in \([10](https://arxiv.org/html/2609.05527#S3.E10)\)\.

The square\-root rule is therefore not a heuristic but the exact solution; Appendix[A\.3](https://arxiv.org/html/2609.05527#A1.SS3)proves it from the KKT conditions of the convex program\.

###### Theorem 4\.3\(Valid inference after pilot\-based replay\)\.

Write𝒫\\mathcal\{P\}for the pilot\. Suppose𝒫\\mathcal\{P\}is independent of the final evaluation pool, the replay probabilities are fixed before the missing baseline outcomes are observed, andqj​\(W\)≥qj,min\>0q\_\{j\}\(W\)\\geq q\_\{j,\\min\}\>0\. Assume the tasks are independent and identically distributed, with outcomes bounded and‖m~j‖∞<∞\\\|\\widetilde\{m\}\_\{j\}\\\|\_\{\\infty\}<\\inftyfor almost every𝒫\\mathcal\{P\}\. Assume also that, given𝒫\\mathcal\{P\}, the population multiplier search has a unique solutionq∘q^\{\\circ\}\(Appendix[A\.4](https://arxiv.org/html/2609.05527#A1.SS4)\)\. Then, conditionally on𝒫\\mathcal\{P\},

n​\(Δ^H−ΔHΔ^A−ΔA\)\|𝒫→𝑑N⁡\{0,Σ𝒫\},\\sqrt\{n\}\\begin\{pmatrix\}\\widehat\{\\Delta\}\_\{H\}\-\\Delta\_\{H\}\\\\ \\widehat\{\\Delta\}\_\{A\}\-\\Delta\_\{A\}\\end\{pmatrix\}\\;\\Big\|\\;\\mathcal\{P\}\\xrightarrow\{d\}N\\\{0,\\Sigma\_\{\\mathcal\{P\}\}\\\},\(14\)withΣ𝒫\\Sigma\_\{\\mathcal\{P\}\}the covariance at the limiting designq∘q^\{\\circ\}that the pilot induces\. Consequently the decision rule \([11](https://arxiv.org/html/2609.05527#S3.E11)\) has conditional asymptotic false\-claim probability at mostα\\alphawhenever at least one true gain fails its threshold; averaging over𝒫\\mathcal\{P\}gives the same bound unconditionally\.

The reason is that the pilot fixes the replay probabilities before any evaluation outcome is observed, and the estimator records those probabilities when correcting for selective replay\. The pilot therefore sets efficiency, not validity: a poor pilot gives a poor design and hence wider intervals, but leaves the error rate untouched\. A false claim needs both bounds to clear while one comparison truly fails, so no multiplicity adjustment is needed; formally, this is an intersection–union test\[[20](https://arxiv.org/html/2609.05527#bib.bib20),[21](https://arxiv.org/html/2609.05527#bib.bib21)\]\.

## 5Results

We first ask whether the two\-baseline deployment question matters in real data\. We then testTEAM\-Designitself: Section[5\.2](https://arxiv.org/html/2609.05527#S5.SS2)uses simulation, where the true gains are known and we control which baseline outcomes are revealed, and Section[5\.3](https://arxiv.org/html/2609.05527#S5.SS3)keeps the dependence and variability of real chest X\-ray data\. Section[5\.4](https://arxiv.org/html/2609.05527#S5.SS4)asks when the method helps and when it does not\. Within each replication every method sees the same pilot, evaluation pool, and potential outcomes, following the protocol set out in Appendix[B](https://arxiv.org/html/2609.05527#A2)\. Unless stated otherwise, we useδH=δA=0\\delta\_\{H\}=\\delta\_\{A\}=0\.

### 5\.1Do clinical human–AI workflows beat both alternatives?

#### Intuition\.

The two\-comparison decision matters only if comparing the human–AI workflow with the human alone can give a different answer from comparing it with the agent alone\. We first ask whether this happens in real clinical data\.

#### Setting\.

We reanalyze two studies in which clinicians evaluate the same cases with and without AI support\[[22](https://arxiv.org/html/2609.05527#bib.bib22),[23](https://arxiv.org/html/2609.05527#bib.bib23)\]\. The model prediction is also recorded, so each case provides outcomes for the assisted clinician, the clinician alone, and the model alone\. Each outcome is a Brier reward\[[18](https://arxiv.org/html/2609.05527#bib.bib18)\]: if a clinician or model states probabilityppfor a case whose true label isy∈\{0,1\}y\\in\\\{0,1\\\}, the reward is1−\(p−y\)21\-\(p\-y\)^\{2\}, so higher is better\.

#### Finding\.

Across 6 clinical settings and 8,723 reads, each a single clinician judging a single case, the assisted clinician performs better than the clinician alone in every setting, but worse than the model alone in every setting\. In no setting does the assisted clinician beat both alternatives \(0 of 6\)\. Table[1](https://arxiv.org/html/2609.05527#S5.T1)gives the two gains per setting with reader\-clustered standard errors, and marks which lower bounds clear zero\. Because the same clinicians evaluate some cases more than once, seeing a case in one condition may affect a later judgment of it\.

Table 1:Matched human–AI reader studies judged against*both*baselines, with reader\-clustered standard errors in parentheses\.Boldmarks a one\-sided 95% lower bound above zero\. In no setting does the assisted clinician beat both alternatives\.
#### Contrast\.

A coding benchmark orders the three configurations the other way, with pass rates of 0\.67% for the agent alone and 18\.89% for the human alone against 31\.11% for the workflow\[[10](https://arxiv.org/html/2609.05527#bib.bib10)\], though only in aggregate, without the task\-level pairing our two comparisons need\. Improvement over the human alone does not settle the deployment question\.

### 5\.2Does decision\-targeted replay help when one comparison is harder?

#### Intuition\.

TEAM\-Designis intended to help when one of the two comparisons requires substantially more evidence than the other\. Replay does not change the true gains, only how precisely we see them\. We test whether directing replay toward the harder comparison lets both gains clear their thresholds more often at the same cost\.

#### Setting\.

Each replication draws a fully observed pilot and a separate evaluation pool, then hides the pool’s human\-only and agent\-only outcomes\.TEAM\-Designsets the two replay probabilities, and only the drawn replays are revealed\. Tasks fall into two trace groups of differing prediction difficulty, and five scenarios vary how close the two gains sit to their thresholds and how much each replay costs \(Appendix[B\.1](https://arxiv.org/html/2609.05527#A2.SS1)\)\. Two comparators run here: Neyman allocation, which uses prediction uncertainty and cost but not distance from the threshold, and uniform replay, which becomes an equal cost split in the unequal\-cost scenario\. Table[2](https://arxiv.org/html/2609.05527#S5.T2)adds a surrogate oracle, the same rule given the true gains in place of pilot estimates\. The chest X\-ray study below adds two more,*AI\-uncertainty*replay, which buys where the model is least certain, and*constantqq*, which replays every case at one rate\. Appendix[B\.2](https://arxiv.org/html/2609.05527#A2.SS2)defines each\. We report how often both lower confidence bounds clear zero\.

#### Finding\.

When one comparison sits much closer to its threshold,TEAM\-Designconcludes that the workflow beats both baselines in 77\.1% of replications against 65\.1% for Neyman at the same cost\. It gets there by raising the replay probabilities for the harder comparison, and settles near an equal split when neither is harder \(Figure[1](https://arxiv.org/html/2609.05527#S5.F1)\)\. The advantage survives replay costs that differ by five to one, and reverses when the two comparisons are equally hard, where Neyman does better because there is no imbalance to exploit\. At the decision boundaryTEAM\-Design’s false\-claim rate stays below 5%, even when the pilot is deliberately misled\. Table[2](https://arxiv.org/html/2609.05527#S5.T2)gives all five scenarios with Monte Carlo standard errors\.

Figure 1:How the replay budget divides between the two baselines, aggregated from the task\-level probabilitiesqi​Hq\_\{iH\}andqi​Aq\_\{iA\}\.TEAM\-Designraises the replay probabilities for the harder comparison when one clearly requires more evidence, and stays closer to an equal split otherwise\.Table 2:Beats\-both decision rate: replications \(%\) with both lower bounds above zero; under the boundary null this is a false\-claim rate at a nominal 5%\. Monte Carlo errors in parentheses,R=20,000R=20\{,\}000per row\. The uniform column is an equal cost split in the unequal\-cost row\. The oracle allocates from the true gains, a reference design and not a deployable rule\.

### 5\.3Validation on real clinical structure

#### Intuition\.

In the real studies the model alone beats the assisted clinician by margins far larger than their standard errors, so the answer is the same however the budget is spent\. A design only matters when the gains sit close to their thresholds\. We therefore keep the real cases, costs, and readers, and shift only the average outcomes so the gains land there\.

#### Setting\.

The shift makes the agent comparison the harder one\. Here the model’s output costs nothing and only clinician reads are purchased, so this study does not exercise the cost trade\-off between the two baselines; Section[5\.2](https://arxiv.org/html/2609.05527#S5.SS2)remains the direct test of that\. A quarter of the cases are held out as the pilot and the rest form the evaluation pool\. Because the split is by case, the same readers appear in both, so Theorem[4\.3](https://arxiv.org/html/2609.05527#S4.Thmtheorem3)does not apply exactly here\. Appendix[B\.1](https://arxiv.org/html/2609.05527#A2.SS1.SSS0.Px7)states which quantities keep their measured values and which are shifted\.

#### Finding\.

TEAM\-Designhas the lowest root mean squared error for estimating the gain against the harder baseline at every budget below full replay \(Figure[2](https://arxiv.org/html/2609.05527#S5.F2)\); at one fifth of full replay it reaches10\.4×10−310\.4\\times 10^\{\-3\}against15\.7×10−315\.7\\times 10^\{\-3\}for Neyman, where that gain is itself about10×10−310\\times 10^\{\-3\}\. Lower error does not always mean both comparisons pass more often: the true agent gain sits so close to its threshold that a noisier estimate crosses it more often by chance, which is why we compare on error here\. Appendix[B\.4](https://arxiv.org/html/2609.05527#A2.SS4.SSS0.Px2)reports a floor sweep that leaves the pattern unchanged\.

Figure 2:Root mean squared error on the harder comparison as a function of the replay budget, on the structure of the chest X\-ray study\. All methods coincide at full replay\.

### 5\.4Scope conditions

Two things bound whereTEAM\-Designhelps\. The first is imbalance: it needs a clearly harder comparison and enough budget to shift measurements to it, so when the two are similarly difficult nothing is left to exploit and Neyman does better, though a larger pilot narrows the gap \(Figure[3](https://arxiv.org/html/2609.05527#S5.F3)a\)\.

The second is the objective, which measures large\-sample difficulty and not the exact chance that both bounds clear at a fixed budget; a noisy pilot that puts one comparison very close to its threshold can therefore draw too much replay\. The stabilizerτ\\tauholds that in check and works best on the scale of the pilot’s uncertainty \(Figure[3](https://arxiv.org/html/2609.05527#S5.F3)b\)\.TEAM\-Designis therefore a decision\-targeted design for unequal comparisons, not a uniformly better replay rule\.

Figure 3:Scope ofTEAM\-Design\.\(a\)When the two comparisons are similarly difficult,TEAM\-Designtrails Neyman, with the gap shrinking as the pilot grows\.\(b\)Performance against the stabilizerτ\\tau, with the pilot standard error marked\.Appendix[B\.3](https://arxiv.org/html/2609.05527#A2.SS3)measures the gap against Bayesian designs aimed at the decision, and Appendix[B\.4](https://arxiv.org/html/2609.05527#A2.SS4)reports how the results respond to the stabilizer, the replay floor, the budget, and a non\-zeroδj\\delta\_\{j\}\.

## 6Conclusion

Keeping a human–AI workflow in production means beating both alternatives at once, and neither is observed once the workflow runs\. Measuring either costs expert time, so we made what to buy part of the design\.TEAM\-Designassigns each task a human\-only and an agent\-only replay probability, giving more measurement to the comparison that needs more evidence while accounting for prediction difficulty and replay cost\. In the clinical data the assisted clinician beats the clinician alone but not the model alone, so evaluating only the human comparison can give the wrong deployment conclusion\.

#### Limitations\.

The rule helps where one comparison is clearly harder, not where the two are balanced, and its objective bounds joint power to first order instead of maximizing it at a fixed budget\. The clinical evidence is matched reader studies, not deployments\.

#### Reproducibility\.

## References

- \[1\]Anthropic\. December 2025\.How AI Is Transforming Work at Anthropic\.[https://www\.anthropic\.com/research/how\-ai\-is\-transforming\-work\-at\-anthropic](https://www.anthropic.com/research/how-ai-is-transforming-work-at-anthropic)\.Accessed 30 August 2026\.
- \[2\]Anthropic\.CRED accelerates fintech development workflows with Claude\.Customer case study, undated\.[https://claude\.com/customers/cred](https://claude.com/customers/cred)\.Accessed 30 August 2026\.
- \[3\]Eli Ben\-Michael, D\. James Greiner, Melody Huang, Kosuke Imai, Zhichao Jiang, and Sooahn Shin\. 2025\.Does AI help humans make better decisions? A statistical evaluation framework for experimental and observational studies\.*Proceedings of the National Academy of Sciences*122, 38 \(2025\), e2505106122\.
- \[4\]Sayash Kapoor, Benedikt Stroebl, Zachary S\. Siegel, Nitya Nadgir, and Arvind Narayanan\. 2025\.AI Agents That Matter\.*Transactions on Machine Learning Research*\.
- \[5\]Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy, and Mykel J\. Kochenderfer\. 2024\.BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices\.In*Advances in Neural Information Processing Systems 37*\.
- \[6\]Louis Filstroff, Iiris Sundin, Petrus Mikkola, Aleksei Tiulpin, Juuso Kylmäoja, and Samuel Kaski\. 2024\.Targeted Active Learning for Bayesian Decision\-Making\.*Transactions on Machine Learning Research*\.
- \[7\]Tom Rossa, Angus Phillips, and Tom Rainforth\. 2026\.Action\-BED: Task\-Driven Bayesian Experimental Design with Singly Intractable Objectives\.Preprint, arXiv:2606\.23662\.
- \[8\]Yu\-Xiang Wang, Alekh Agarwal, and Miroslav Dudík\. 2017\.Optimal and Adaptive Off\-policy Evaluation in Contextual Bandits\.In*Proceedings of the 34th International Conference on Machine Learning*, 3589–3597\.
- \[9\]Connor Douglas, Joel Persson, and Foster Provost\. 2026\.Logging Policy Design for Off\-Policy Evaluation\.Preprint, arXiv:2605\.15108\.
- \[10\]Hanjun Luo, Chiming Ni, Jiaheng Wen, Zhimu Huang, Yiran Wang, Bingduo Liao, Sylvia Chung, Yingbin Jin, Xinfeng Li, Wenyuan Xu, XiaoFeng Wang, and Hanan Salam\. 2026\.CentaurEval: Benchmarking Human\-in\-the\-Loop Value in Agentic Coding\.In*Proceedings of the 43rd International Conference on Machine Learning*\.
- \[11\]Carlos E\. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan\. 2024\.SWE\-bench: Can Language Models Resolve Real\-World GitHub Issues?In*The Twelfth International Conference on Learning Representations*\.
- \[12\]Robert E\. Bechhofer\. 1954\.A Single\-Sample Multiple Decision Procedure for Ranking Means of Normal Populations with Known Variances\.*The Annals of Mathematical Statistics*25, 1 \(1954\), 16–39\.
- \[13\]Chun\-Hung Chen, Jianwu Lin, Enver Yücesan, and Stephen E\. Chick\. 2000\.Simulation Budget Allocation for Further Enhancing the Efficiency of Ordinal Optimization\.*Discrete Event Dynamic Systems*10, 3 \(2000\), 251–270\.
- \[14\]Seong\-Hee Kim and Barry L\. Nelson\. 2006\.Selecting the Best System\.In*Simulation*, Handbooks in Operations Research and Management Science, Vol\. 13\. Elsevier, 501–534\.
- \[15\]Jean\-Yves Audibert, Sébastien Bubeck, and Rémi Munos\. 2010\.Best Arm Identification in Multi\-Armed Bandits\.In*Proceedings of the 23rd Conference on Learning Theory \(COLT\)*\. Omnipress, 41–53\.
- \[16\]Anastasios N\. Angelopoulos, Jacob Eisenstein, Jonathan Berant, Alekh Agarwal, and Adam Fisch\. 2025\.Cost\-Optimal Active AI Model Evaluation\.*arXiv preprint arXiv:2506\.07949*\.
- \[17\]Peter B\. Gilbert, Xuesong Yu, and Andrea Rotnitzky\. 2014\.Optimal Auxiliary\-Covariate Based Two\-Phase Sampling Design for Semiparametric Efficient Estimation of a Mean or Mean Difference, with Application to Clinical Trials\.*Statistics in Medicine*33, 6 \(2014\), 901–917\.
- \[18\]Glenn W\. Brier\. 1950\.Verification of Forecasts Expressed in Terms of Probability\.*Monthly Weather Review*78, 1 \(1950\), 1–3\.
- \[19\]Charles Stein\. 1945\.A Two\-Sample Test for a Linear Hypothesis Whose Power Is Independent of the Variance\.*The Annals of Mathematical Statistics*16, 3 \(1945\), 243–258\.
- \[20\]Roger L\. Berger\. 1982\.Multiparameter Hypothesis Testing and Acceptance Sampling\.*Technometrics*24, 4 \(1982\), 295–300\.
- \[21\]Eugene M\. Laska and Morris J\. Meisner\. 1989\.Testing Whether an Identified Treatment Is Best\.*Biometrics*45, 4 \(1989\), 1139–1151\.
- \[22\]Alex Moehring, Manasi Kutwal, Ray Huang, et al\. 2025\.A Dataset for Understanding Radiologist–Artificial Intelligence Collaboration\.*Scientific Data*12 \(2025\), 739\.
- \[23\]Tirtha Chanda, Katja Hauser, Sarah Hobelsberger, et al\. 2024\.Dermatologist\-like Explainable AI Enhances Trust and Confidence in Diagnosing Melanoma\.*Nature Communications*15 \(2024\), 524\.

## Appendix AProofs

### A\.1Proof of Proposition[4\.1](https://arxiv.org/html/2609.05527#S4.Thmtheorem1)

For comparisonjj, asymptotic normality gives

n​\(Δ^j−Δj\)σj​\(qj\)→𝑑N⁡\(0,1\)\.\\frac\{\\sqrt\{n\}\(\\widehat\{\\Delta\}\_\{j\}\-\\Delta\_\{j\}\)\}\{\\sigma\_\{j\}\(q\_\{j\}\)\}\\xrightarrow\{\\ d\\ \}N\(0,1\)\.\(15\)A one\-sided level\-α\\alphaWald test rejectsH0​j:Δj≤δjH\_\{0j\}:\\Delta\_\{j\}\\leq\\delta\_\{j\}when

Δ^j−δj\>z1−α​σj​\(qj\)n\.\\widehat\{\\Delta\}\_\{j\}\-\\delta\_\{j\}\>z\_\{1\-\\alpha\}\\frac\{\\sigma\_\{j\}\(q\_\{j\}\)\}\{\\sqrt\{n\}\}\.\(16\)Under a fixed positive alternative its asymptotic power is

Φ⁡\(n​Δj−δjσj​\(qj\)−z1−α\)\.\\Phi\\\!\\left\(\\sqrt\{n\}\\frac\{\\Delta\_\{j\}\-\\delta\_\{j\}\}\{\\sigma\_\{j\}\(q\_\{j\}\)\}\-z\_\{1\-\\alpha\}\\right\)\.\(17\)Requiring this to be at least1−β/21\-\\beta/2gives

n​Δj−δjσj​\(qj\)≥z1−α\+z1−β/2\.\\sqrt\{n\}\\frac\{\\Delta\_\{j\}\-\\delta\_\{j\}\}\{\\sigma\_\{j\}\(q\_\{j\}\)\}\\geq z\_\{1\-\\alpha\}\+z\_\{1\-\\beta/2\}\.\(18\)Squaring and requiring the inequality for both comparisons yields \([12](https://arxiv.org/html/2609.05527#S4.E12)\)\.□\\square

### A\.2Variance decomposition for Equation \([8](https://arxiv.org/html/2609.05527#S3.E8)\)

Write the score of \([6](https://arxiv.org/html/2609.05527#S3.E6)\) for a generic task, dropping the indexii, as

ψj=\(T−Yj\)\+\(1−Rjqj​\(W\)\)​\{Yj−m~j​\(W\)\}\.\\psi\_\{j\}=\(T\-Y\_\{j\}\)\+\\left\(1\-\\frac\{R\_\{j\}\}\{q\_\{j\}\(W\)\}\\right\)\\\{Y\_\{j\}\-\\widetilde\{m\}\_\{j\}\(W\)\\\}\.\(19\)Conditional on the pilot𝒫\\mathcal\{P\}, replay is randomized using the recorded task informationWWalone, soRj⟂\(T,Yj\)\|\(W,𝒫\)R\_\{j\}\\perp\(T,Y\_\{j\}\)\\mid\(W,\\mathcal\{P\}\)and𝔼\[Rj∣W,T,Yj,𝒫\]=qj\(W\)\\mathbb\{E\}\[R\_\{j\}\\mid W,T,Y\_\{j\},\\mathcal\{P\}\]=q\_\{j\}\(W\)\. The second term therefore has mean zero conditional on\(W,T,Yj\)\(W,T,Y\_\{j\}\), so it is uncorrelated with the first\. Its conditional variance is\{Yj−m~j​\(W\)\}2​Var⁡\{Rj/qj​\(W\)∣W\}=\{Yj−m~j​\(W\)\}2​\{1−qj​\(W\)\}/qj​\(W\)\\\{Y\_\{j\}\-\\widetilde\{m\}\_\{j\}\(W\)\\\}^\{2\}\\operatorname\{Var\}\\\{R\_\{j\}/q\_\{j\}\(W\)\\mid W\\\}=\\\{Y\_\{j\}\-\\widetilde\{m\}\_\{j\}\(W\)\\\}^\{2\}\\\{1\-q\_\{j\}\(W\)\\\}/q\_\{j\}\(W\)\. Taking expectations,

σj2​\(qj\)=Var⁡\(T−Yj\)\+𝔼⁡\[sj​\(W\)qj​\(W\)\]−𝔼⁡\[sj​\(W\)\],\\sigma\_\{j\}^\{2\}\(q\_\{j\}\)=\\operatorname\{Var\}\(T\-Y\_\{j\}\)\+\\mathbb\{E\}\\left\[\\frac\{s\_\{j\}\(W\)\}\{q\_\{j\}\(W\)\}\\right\]\-\\mathbb\{E\}\[s\_\{j\}\(W\)\],\(20\)which is \([8](https://arxiv.org/html/2609.05527#S3.E8)\) withκj=Var⁡\(T−Yj\)−𝔼⁡\[sj​\(W\)\]\\kappa\_\{j\}=\\operatorname\{Var\}\(T\-Y\_\{j\}\)\-\\mathbb\{E\}\[s\_\{j\}\(W\)\]\. No assumption relatingYjY\_\{j\}toTTgivenWWis used, so the decomposition holds under the correlated outcomes of Appendix[B\.1](https://arxiv.org/html/2609.05527#A2.SS1)\. The pilot estimateκ~j\\widetilde\{\\kappa\}\_\{j\}of Section[3](https://arxiv.org/html/2609.05527#S3)targets this quantity by replacing both population moments with their pilot counterparts,κ~j=Var~p​\(T−Yj\)−𝔼~p​\[s~j​\(W\)\]\\widetilde\{\\kappa\}\_\{j\}=\\widetilde\{\\operatorname\{Var\}\}\_\{\\mathrm\{p\}\}\(T\-Y\_\{j\}\)\-\\widetilde\{\\mathbb\{E\}\}\_\{\\mathrm\{p\}\}\[\\widetilde\{s\}\_\{j\}\(W\)\], whereVar~p\\widetilde\{\\operatorname\{Var\}\}\_\{\\mathrm\{p\}\}is the pilot sample variance, which the pilot can form because all three outcomes are observed there\. The simplerVar⁡\{T−m~j​\(W\)\}\\operatorname\{Var\}\\\{T\-\\widetilde\{m\}\_\{j\}\(W\)\\\}agrees with it only for a well\-specified predictor\.

□\\square

### A\.3Proof of Theorem[4\.2](https://arxiv.org/html/2609.05527#S4.Thmtheorem2)

Forj∈\{H,A\}j\\in\\\{H,A\\\}write

uj​\(W\)=sj​\(W\)\(Δj−δj\)2,bj=κj\(Δj−δj\)2,u\_\{j\}\(W\)=\\frac\{s\_\{j\}\(W\)\}\{\(\\Delta\_\{j\}\-\\delta\_\{j\}\)^\{2\}\},\\qquad b\_\{j\}=\\frac\{\\kappa\_\{j\}\}\{\(\\Delta\_\{j\}\-\\delta\_\{j\}\)^\{2\}\},so that by \([8](https://arxiv.org/html/2609.05527#S3.E8)\) comparisonjj’s difficulty isbj\+𝔼⁡\[uj​\(W\)/qj​\(W\)\]b\_\{j\}\+\\mathbb\{E\}\[u\_\{j\}\(W\)/q\_\{j\}\(W\)\]and the epigraph constraint that it be at mostttreads

bj\+𝔼⁡\[uj​\(W\)qj​\(W\)\]≤t\.b\_\{j\}\+\\mathbb\{E\}\\left\[\\frac\{u\_\{j\}\(W\)\}\{q\_\{j\}\(W\)\}\\right\]\\leq t\.\(21\)Introduce multipliersλH,λA≥0\\lambda\_\{H\},\\lambda\_\{A\}\\geq 0for the comparison constraints andη≥0\\eta\\geq 0for the budget, positive when it binds\. The Lagrangian terms involvingttandqqare

ℒ=t\+∑jλj​\{bj\+𝔼⁡\(ujqj\)−t\}\+η⁡\{𝔼⁡\(cH​qH\+cA​qA\)−B\}\.\\mathcal\{L\}=t\+\\sum\_\{j\}\\lambda\_\{j\}\\left\\\{b\_\{j\}\+\\mathbb\{E\}\\left\(\\frac\{u\_\{j\}\}\{q\_\{j\}\}\\right\)\-t\\right\\\}\+\\eta\\\{\\mathbb\{E\}\(c\_\{H\}q\_\{H\}\+c\_\{A\}q\_\{A\}\)\-B\\\}\.\(22\)Stationarity inttgivesλH\+λA=1\\lambda\_\{H\}\+\\lambda\_\{A\}=1\. At an interior point,

−λj​uj​\(W\)qj​\(W\)2\+η​cj​\(W\)=0,\-\\frac\{\\lambda\_\{j\}u\_\{j\}\(W\)\}\{q\_\{j\}\(W\)^\{2\}\}\+\\eta c\_\{j\}\(W\)=0,\(23\)so

qj​\(W\)=λj​uj​\(W\)η​cj​\(W\),q\_\{j\}\(W\)=\\sqrt\{\\frac\{\\lambda\_\{j\}u\_\{j\}\(W\)\}\{\\eta c\_\{j\}\(W\)\}\},\(24\)which is \([13](https://arxiv.org/html/2609.05527#S4.E13)\)\. The box constraints clip this expression to\[qj,min,1\]\[q\_\{j,\\min\},1\]\. Convexity ofu/qu/qonq\>0q\>0makes the KKT conditions sufficient, and Slater’s condition, which the feasible budget range of Appendix[B\.1](https://arxiv.org/html/2609.05527#A2.SS1.SSS0.Px8)supplies, makes them necessary, so an optimal rule takes this form\. Wherecj​\(W\)=0c\_\{j\}\(W\)=0the per\-task term isλj​uj​\(W\)/qj​\(W\)\\lambda\_\{j\}u\_\{j\}\(W\)/q\_\{j\}\(W\)alone, which is strictly decreasing, so the box constraint carriesqj​\(W\)q\_\{j\}\(W\)to one and no stationary point is required\. Complementary slackness yields the limiting\-comparison statement\. Replacingsj​\(W\)s\_\{j\}\(W\)and\(Δj−δj\)2\(\\Delta\_\{j\}\-\\delta\_\{j\}\)^\{2\}by their pilot counterpartss~j​\(W\)\\widetilde\{s\}\_\{j\}\(W\)andd~j2\\widetilde\{d\}\_\{j\}^\{2\}gives the implementable rule \([10](https://arxiv.org/html/2609.05527#S3.E10)\)\.□\\square

### A\.4Proof of Theorem[4\.3](https://arxiv.org/html/2609.05527#S4.Thmtheorem3)

Writeℱn\\mathcal\{F\}\_\{n\}for theσ\\sigma\-field generated by the pilot, the pool covariatesW1:nW\_\{1:n\}and the costs\. By hypothesisqHq\_\{H\}andqAq\_\{A\}areℱn\\mathcal\{F\}\_\{n\}\-measurable and no baseline outcome or replay indicator enters them\.

#### The conditional mean\.

Givenℱn\\mathcal\{F\}\_\{n\}the indicators are independent across tasks and comparisons, withRi​j∼Bernoulli⁡\{qj​\(Wi\)\}R\_\{ij\}\\sim\\mathrm\{Bernoulli\}\\\{q\_\{j\}\(W\_\{i\}\)\\\}independent of\(Ti,Yi​H,Yi​A\)\(T\_\{i\},Y\_\{iH\},Y\_\{iA\}\)givenWiW\_\{i\}, so the correction term of \([6](https://arxiv.org/html/2609.05527#S3.E6)\) has conditional mean zero and

𝔼⁡\[ψi​j∣ℱn\]=μj​\(Wi\),μj​\(W\)=𝔼⁡\[T−Yj∣W\]\.\\mathbb\{E\}\[\\psi\_\{ij\}\\mid\\mathcal\{F\}\_\{n\}\]=\\mu\_\{j\}\(W\_\{i\}\),\\qquad\\mu\_\{j\}\(W\)=\\mathbb\{E\}\[T\-Y\_\{j\}\\mid W\]\.\(25\)The conditioning centers each score at the*conditional*gain, not atΔj\\Delta\_\{j\}, and the gap between them is carried by the second term below\.

#### A decomposition that isolates the design\.

n​\(Δ^j−Δj\)=1n​∑i\{ψi​j−μj​\(Wi\)\}⏟An​j\+1n​∑i\{μj​\(Wi\)−Δj\}⏟Bn​j\.\\sqrt\{n\}\\,\(\\widehat\{\\Delta\}\_\{j\}\-\\Delta\_\{j\}\)=\\underbrace\{\\frac\{1\}\{\\sqrt\{n\}\}\\sum\_\{i\}\\\{\\psi\_\{ij\}\-\\mu\_\{j\}\(W\_\{i\}\)\\\}\}\_\{A\_\{nj\}\}\+\\underbrace\{\\frac\{1\}\{\\sqrt\{n\}\}\\sum\_\{i\}\\\{\\mu\_\{j\}\(W\_\{i\}\)\-\\Delta\_\{j\}\\\}\}\_\{B\_\{nj\}\}\.\(26\)Every dependence onqjq\_\{j\}sits inAn​jA\_\{nj\}\. The termBn​jB\_\{nj\}is a normalized sum of independent mean\-zero variables that the design cannot touch\.

#### The design\-dependent term\.

Conditionally onℱn\\mathcal\{F\}\_\{n\}the summands ofAn​jA\_\{nj\}are independent with mean zero by \([25](https://arxiv.org/html/2609.05527#A1.E25)\)\. Outcomes are bounded, the realized pilot gives‖m~j‖∞<∞\\\|\\widetilde\{m\}\_\{j\}\\\|\_\{\\infty\}<\\infty, andqj≥qj,min\>0q\_\{j\}\\geq q\_\{j,\\min\}\>0, so the summands are bounded by a fixed multiple of1/qj,min1/q\_\{j,\\min\}and the Lindeberg condition for the triangular array holds\. Their average conditional variance is

1n​∑i\[Var⁡\(T−Yj∣Wi\)\+sj​\(Wi\)​\{1qj​\(Wi\)−1\}\]\.\\frac\{1\}\{n\}\\sum\_\{i\}\\left\[\\operatorname\{Var\}\(T\-Y\_\{j\}\\mid W\_\{i\}\)\+s\_\{j\}\(W\_\{i\}\)\\Big\\\{\\frac\{1\}\{q\_\{j\}\(W\_\{i\}\)\}\-1\\Big\\\}\\right\]\.\(27\)

#### The design converges given the pilot\.

Fix𝒫\\mathcal\{P\}\. Conditional on the pilot,s~j\\widetilde\{s\}\_\{j\},d~j2\\widetilde\{d\}\_\{j\}^\{2\}andκ~j\\widetilde\{\\kappa\}\_\{j\}are fixed, and the remaining ingredients of the estimated difficulty of comparisonjjare pool averages\. Because the replay probabilities are bounded below byqj,min\>0q\_\{j,\\min\}\>0and the multiplier search runs over a compact set, the empirical comparison objectives converge uniformly in probability to their conditional population counterparts\. Under the assumed uniqueness of the population minimizer, the argmin theorem gives consistency of the selected multipliers and, writingqn​jq\_\{nj\}for the rule the pool of sizennselects,qn​j→qj∘q\_\{nj\}\\to q\_\{j\}^\{\\circ\}in probability, uniformly inWWby continuity of \([10](https://arxiv.org/html/2609.05527#S3.E10)\), whereqj∘q\_\{j\}^\{\\circ\}is𝒫\\mathcal\{P\}\-measurable\. The displayed average therefore converges in probability to𝔼⁡\[Var⁡\(T−Yj∣W\)\]\+𝔼⁡\[sj/qj∘\]−𝔼⁡\[sj\]\\mathbb\{E\}\[\\operatorname\{Var\}\(T\-Y\_\{j\}\\mid W\)\]\+\\mathbb\{E\}\[s\_\{j\}/q\_\{j\}^\{\\circ\}\]\-\\mathbb\{E\}\[s\_\{j\}\]\.

#### The limit\.

Work with the pairsAn=\(An​H,An​A\)A\_\{n\}=\(A\_\{nH\},A\_\{nA\}\)andBn=\(Bn​H,Bn​A\)B\_\{n\}=\(B\_\{nH\},B\_\{nA\}\), so that the covariance between the two comparisons is carried along\. The conditional multivariate Lindeberg–Feller theorem gives𝔼⁡\[exp⁡\(i​t⊤​An\)∣ℱn\]→exp⁡\(−12​t⊤​ΣA​t\)\\mathbb\{E\}\[\\exp\(it^\{\\top\}A\_\{n\}\)\\mid\\mathcal\{F\}\_\{n\}\]\\to\\exp\(\-\\tfrac\{1\}\{2\}t^\{\\top\}\\Sigma\_\{A\}t\)in probability\. SinceBnB\_\{n\}isℱn\\mathcal\{F\}\_\{n\}\-measurable,

𝔼⁡\[ei​t⊤​An\+i​u⊤​Bn\]=𝔼⁡\[ei​u⊤​Bn​𝔼​\{ei​t⊤​An∣ℱn\}\],\\mathbb\{E\}\\\!\\left\[e^\{it^\{\\top\}A\_\{n\}\+iu^\{\\top\}B\_\{n\}\}\\right\]=\\mathbb\{E\}\\\!\\left\[e^\{iu^\{\\top\}B\_\{n\}\}\\,\\mathbb\{E\}\\\{e^\{it^\{\\top\}A\_\{n\}\}\\mid\\mathcal\{F\}\_\{n\}\\\}\\right\],whose inner factor converges to a nonrandom limit, while the ordinary multivariate central limit theorem givesBn⇒N⁡\(0,ΣB\)B\_\{n\}\\Rightarrow N\(0,\\Sigma\_\{B\}\)\. The joint characteristic function therefore converges to the product of two Gaussian characteristic functions, soAnA\_\{n\}andBnB\_\{n\}are asymptotically independent given the pilot andAn\+Bn⇒N⁡\(0,ΣA\+ΣB\)A\_\{n\}\+B\_\{n\}\\Rightarrow N\(0,\\Sigma\_\{A\}\+\\Sigma\_\{B\}\), with variance

𝔼⁡\[Var⁡\(T−Yj∣W\)\]\+Var⁡\{μj​\(W\)\}\+𝔼⁡\[sjqj∘\]−𝔼⁡\[sj\]=κj\+𝔼⁡\[sjqj∘\],\\mathbb\{E\}\[\\operatorname\{Var\}\(T\-Y\_\{j\}\\mid W\)\]\+\\operatorname\{Var\}\\\{\\mu\_\{j\}\(W\)\\\}\+\\mathbb\{E\}\\\!\\left\[\\frac\{s\_\{j\}\}\{q\_\{j\}^\{\\circ\}\}\\right\]\-\\mathbb\{E\}\[s\_\{j\}\]=\\kappa\_\{j\}\+\\mathbb\{E\}\\\!\\left\[\\frac\{s\_\{j\}\}\{q\_\{j\}^\{\\circ\}\}\\right\],\(28\)by the variance decomposition ofT−YjT\-Y\_\{j\}overWW\. That is \([8](https://arxiv.org/html/2609.05527#S3.E8)\) at the limiting design, so the limit isN⁡\{0,Σ⁡\(qH∘,qA∘\)\}N\\\{0,\\Sigma\(q\_\{H\}^\{\\circ\},q\_\{A\}^\{\\circ\}\)\\\}\. The empirical covariance of the paired scores is consistent by the same boundedness\.

#### The reported rate\.

IfΔj≤δj\\Delta\_\{j\}\\leq\\delta\_\{j\}for somejj, then\{LH\>δH\}∩\{LA\>δA\}⊆\{Lj\>δj\}\\\{L\_\{H\}\>\\delta\_\{H\}\\\}\\cap\\\{L\_\{A\}\>\\delta\_\{A\}\\\}\\subseteq\\\{L\_\{j\}\>\\delta\_\{j\}\\\}, and the right\-hand event is the rejection of a true one\-sided component null, whose asymptotic probability is at mostα\\alpha\.□\\square

## Appendix BSimulation and empirical study design

Section[5](https://arxiv.org/html/2609.05527#S5)reports three constructed studies and one reanalysis of published reader data\. This appendix gives the construction of the three constructed studies, the information available to every competing rule, and the protocol under which the sweeps were run\. The reanalysis needs no construction, since it uses the published reader studies as they stand, and Section[5\.1](https://arxiv.org/html/2609.05527#S5.SS1)states how its outcomes are scored\.

### B\.1Generating the evaluation data

Measuring how much error a replay rule leaves requires knowing the true gains, which no real study reports\. We therefore generate tasks whose gains are fixed in advance\. The scenarios below differ only in how unequal the two comparisons are, since that imbalance is what the design is built to exploit\.

#### Unit and estimand\.

One unit is one software task\. Each task has a trace stratumW∈\{0,1\}W\\in\\\{0,1\\\}, a deployed human–AI workflow outcomeTT, a human\-only baseline outcomeYHY\_\{H\}, and an agent\-only baseline outcomeYAY\_\{A\}\. The primary outcome is binary task success\. The estimands are exactly the two population gains \([1](https://arxiv.org/html/2609.05527#S2.E1)\), and no scenario changes the estimand\.

#### Primary two\-stratum generator\.

HereW∼Bernoulli⁡\(1/2\)W\\sim\\mathrm\{Bernoulli\}\(1/2\), and conditional onWWthe three outcomesT,YH,YAT,Y\_\{H\},Y\_\{A\}are drawn independently as Bernoulli variables with stratum\-specific success probabilities\. The primary one\-bottleneck specification, the harder\-comparison scenario of Table[2](https://arxiv.org/html/2609.05527#S5.T2), is

P⁡\(T=1∣W\)=\(0\.70,0\.78\),P⁡\(YH=1∣W\)=\(0\.55,0\.85\),P⁡\(YA=1∣W\)=\(0\.60,0\.56\)\.P\(T=1\\mid W\)=\(0\.70,0\.78\),\\;P\(Y\_\{H\}=1\\mid W\)=\(0\.55,0\.85\),\\;P\(Y\_\{A\}=1\\mid W\)=\(0\.60,0\.56\)\.\(29\)This gives a small workflow\-versus\-human gain and a substantially larger workflow\-versus\-agent gain while making human\-only residual uncertainty strongly stratum dependent\. These values are design parameters, not empirical findings\.

#### Balanced\-comparison specification\.

This specification keepsTTandYHY\_\{H\}as in Equation \([29](https://arxiv.org/html/2609.05527#A2.E29)\) and replaces the agent baseline by

P⁡\(YA=1∣W\)=\(0\.72,0\.68\),P\(Y\_\{A\}=1\\mid W\)=\(0\.72,0\.68\),\(30\)so that the two population gains are of comparable size\. This is the pre\-specified balanced\-comparison check, since a method built around a unique limiting comparison should have little structural advantage here\.

#### Boundary\-null specification\.

A one\-sided test claims a win most readily when the true gain sits exactly at its threshold, and a false claim needs only one of the two comparisons to fail\. The least favorable case is therefore a null on one comparison with the other left comfortably positive, which is what this specification builds\. It sets

P⁡\(T=1∣W\)=\(0\.66,0\.74\)P\(T=1\\mid W\)=\(0\.66,0\.74\)\(31\)with the originalYHY\_\{H\}andYAY\_\{A\}probabilities, making the workflow\-versus\-human gain exactly zero while the workflow\-versus\-agent gain remains positive\. This is the primary false\-claim experiment\.

#### Correlation sensitivity\.

The independent\-Bernoulli generator is transparent but does not create residual correlation between potential outcomes after conditioning onWW\. Every primary scenario is therefore repeated with a Gaussian\-copula generator, which draws\(ZT,ZH,ZA\)\(Z\_\{T\},Z\_\{H\},Z\_\{A\}\)from a trivariate normal with pairwise correlationρ∈\{0\.25,0\.50,0\.75\}\\rho\\in\\\{0\.25,0\.50,0\.75\\\}and thresholds each coordinate to reproduce the same Bernoulli marginals\. This tests whether the allocation result depends on conditional independence\.

#### Unequal\-cost scenario\.

A human\-only replay and an agent\-only replay rarely cost the same\. Recovering the human baseline can take a senior engineer or a clinician the better part of an hour, while recovering the agent baseline costs compute, so a rule blind to that difference will spend its budget on whichever evidence is cheap and not on whichever evidence the decision needs\. This scenario prices the two apart\. It keeps the primary bottleneck generator and sets the cost ratio

cH/cA=5,c\_\{H\}/c\_\{A\}=5,\(32\)with the expected total cost fixed at one cost unit per evaluation task\.TEAM\-Design, cost\-aware Neyman, and equal cost splitting are compared at matched*expected*cost, and the primary result is the beats\-both decision rate\. The secondary results are the number of human\-only replays, the number of agent\-only replays, the gain standard errors, and the cost per successful beats\-both decision\.

The ratio is also swept overcH/cA∈\{2,5,10\}c\_\{H\}/c\_\{A\}\\in\\\{2,5,10\\\}\. The main text reports only the pre\-specified5:15\{:\}1setting, and the full curve is Table[5](https://arxiv.org/html/2609.05527#A2.T5)\.

#### Real clinical structure\.

A generator supplies known gains but invents the correlation between readers and cases\. The chest X\-ray study supplies that correlation but places its gains far enough above their thresholds that no replay rule can change the answer\. This experiment keeps the study’s covariates, costs, reader structure and correlations, and shifts only the arm means\. Every measured feature of the chest X\-ray study is retained, namely the covariates, the per\-case radiologist times that supply the replay costs, the reader and case structure used for clustering, the residual heteroskedasticity, and the within\-case correlation\. Only the arm means are shifted, which changes no second moment\. The reported regime makes the agent comparison the limiting one at\(ΔH,ΔA\)=\(0\.08,0\.01\)\(\\Delta\_\{H\},\\Delta\_\{A\}\)=\(0\.08,0\.01\), and the balanced regime\(0\.04,0\.04\)\(0\.04,0\.04\)and a boundary regime\(0\.05,0\.00\)\(0\.05,0\.00\)are run alongside it\.

A quarter of cases are held out as the fully observed pilot and the rest form the evaluation pool, about656656reads\. Splitting by case and not by reader leaves readers common to both, so the independence of Theorem[4\.3](https://arxiv.org/html/2609.05527#S4.Thmtheorem3)holds across cases and not across readers, which is why every standard error here is reader\-clustered\. Its i\.i\.d\. hypothesis is therefore met at the case level and not at the read level, so this experiment sits outside the theorem’s exact assumptions and its false\-claim rates are an empirical check and not an instance of the guarantee\. Replay cost is the measured per\-case radiologist time, so a budget of1\.01\.0is the cost of replaying every case\. The sweep runs0\.150\.15to1\.01\.0and the pilot is not charged against it, being already collected\. The probability floor isqj,min=0\.05q\_\{j,\\min\}=0\.05and each point is200200replications\. The stabilizer is set separately for each comparison to the pilot’s own standard error of that gain, which is the scale Section[5\.4](https://arxiv.org/html/2609.05527#S5.SS4)finds works best\.

Five rules are compared\.TEAM\-Designis the square\-root rule of \([10](https://arxiv.org/html/2609.05527#S3.E10)\), which prices prediction difficulty against replay cost and tilts the budget toward the comparison that is harder to settle\.*Neyman*weights by residual variance and cost but ignores the gains\.*Uniform*uses one constant probability for both baselines at the same expected cost\.*Constantqq*uses one constant probability at the same expected*count*of replays\.*AI\-uncertainty*weights by the model’s own predictive uncertainty on the case, the heuristic a practitioner would reach for first\. The reported quantity is the root mean squared error of the limiting gain\.

The agent\-limited and balanced regimes are chosen to differ only in how unequal the comparisons are\. The agent\-limited regime sets the agent gain at0\.010\.01against a human gain of0\.080\.08, so the agent comparison needs far more evidence, and the balanced regime sets both at0\.040\.04\. Replay costs are the study’s own measured times, an assisted read at117117s against an unaided read at176176s\.

The agent\-limited regime sits at the edge of what the data can resolve\. At full replay, where every case is bought and no design choice remains, the one\-sided bound on the agent gain needs16×10−316\\times 10^\{\-3\}while the true gain is10×10−310\\times 10^\{\-3\}, so both bounds clear in only9\.09\.0% of replications and every rule scores alike\.

#### Protocol\.

Theorem[4\.3](https://arxiv.org/html/2609.05527#S4.Thmtheorem3)requires the pilot to share no task with the evaluation pool and every replay probability to be fixed before any baseline outcome is observed\. Neither condition leaves a trace in the results when it is broken\. Table[3](https://arxiv.org/html/2609.05527#A2.T3)gives the sizes and the budgets, and the paragraphs below give the protocol under which each replication was run\.

Table 3:Settings for the three constructed studies\. The synthetic and semi\-synthetic sizes count tasks and reads, and the Gaussian sizes count measurements\.
#### Monte Carlo protocol\.

The bottleneck, balanced, boundary\-null, and cost\-ratio scenarios use 20,000 replications, and the correlation and size sweeps use 2,000\. Unit tests confirm that every method meets the same expected or realized budget, that on fully observed data the estimator recovers the known simulated gains, and that setting allqj=1q\_\{j\}=1reproduces the complete\-data estimator\. Each replication saves one row holding the true and estimated gains, their standard errors, replay counts and cost, allocation shares, and the final beats\-both decision, and every reported table derives from those rows\.

All experiments run on a single CPU machine with no accelerator\. The full replay grid takes about twelve minutes and the matched\-validity comparison about six, with the clinical reanalyses and the semi\-synthetic sweep a few minutes each\. Monte Carlo standard errors accompany every reported probability\. For a reported ratep^\\widehat\{p\}fromRRreplications, use

MCSE⁡\(p^\)=p^​\(1−p^\)/R\.\\mathrm\{MCSE\}\(\\widehat\{p\}\)=\\sqrt\{\\widehat\{p\}\(1\-\\widehat\{p\}\)/R\}\.\(33\)Differences within roughly two Monte Carlo standard errors are not treated as meaningful\.

Every method within a replication sees the same pilot sample and the same fully generated evaluation potential outcomes, so only the replay indicators differ\. This paired design reduces Monte Carlo noise in the comparisons between methods\. The replay probabilityqi​jq\_\{ij\}is recorded for every baseline, purchased or not\.

#### Nuisance fitting\.

WhereWWis a binary stratum,m~j\\widetilde\{m\}\_\{j\}ands~j\\widetilde\{s\}\_\{j\}are the within\-stratum mean and mean squared deviation, so no model is fitted\. On the chest X\-ray data,WWholds four recorded features, and bothm~j\\widetilde\{m\}\_\{j\}ands~j\\widetilde\{s\}\_\{j\}are gradient\-boosted regression trees with depth 3, 120 iterations, and learning rate 0\.08\. The residuals used to builds~j\\widetilde\{s\}\_\{j\}come from a five\-fold cross\-fit on the pilot, so they are out of fold and do not make the baseline look more predictable than it is\. Them~j\\widetilde\{m\}\_\{j\}applied to the evaluation pool is trained on the whole pilot, which needs no cross\-fitting because the two sets are disjoint\. Predicted squared errors are floored at10−610^\{\-6\}\.

#### Sizes and budget\.

The pilot is independent and fully observed, and supplies the baseline means and residual second moments\. The primary sizes are

np=500,n=3000\.n\_\{\\mathrm\{p\}\}=500,\\qquad n=3000\.\(34\)Sensitivity analyses usenp∈\{250,500,1000\}n\_\{\\mathrm\{p\}\}\\in\\\{250,500,1000\\\}andn∈\{1000,3000,10000\}n\\in\\\{1000,3000,10000\\\}\. The main replay budget is0\.40\.4baseline measurements per evaluation task with probability floorqj,min=0\.02q\_\{j,\\min\}=0\.02and stabilizationτ=0\.03\\tau=0\.03\.

A budget\-exhaustingη\\etain \([10](https://arxiv.org/html/2609.05527#S3.E10)\) exists wheneverBBlies between𝔼⁡\[cH​\(W\)​qH,min\+cA​\(W\)​qA,min\]\\mathbb\{E\}\[c\_\{H\}\(W\)q\_\{H,\\min\}\+c\_\{A\}\(W\)q\_\{A,\\min\}\]and𝔼⁡\[cH​\(W\)\+cA​\(W\)\]\\mathbb\{E\}\[c\_\{H\}\(W\)\+c\_\{A\}\(W\)\]\. Below the lower end the floors alone overspend, and above the upper end the budget does not bind and everyqj≡1q\_\{j\}\\equiv 1\.

#### Boundary\-null and pilot misspecification\.

The boundary\-null design is run for at least 5,000 Monte Carlo replications because the target probability sits near 0\.05\. The reported quantities are the false\-claim rate with an exact or Wilson 95% Monte Carlo interval, together with empirical bias and one\-sided coverage for each comparison separately\.

For pilot misspecification, the pilot stratum pattern is deliberately reversed before acquisition\. For both baselines the estimated residual second moments forW=0W=0andW=1W=1are exchanged, as are the fitted baseline means across strata, while the final outcomes and the recorded replay probabilities are left untouched\. This intervention is designed to make the replay allocation inefficient while leaving Theorem[4\.3](https://arxiv.org/html/2609.05527#S4.Thmtheorem3)’s target unchanged\.

### B\.2Comparators and metrics

The replay rule of \([10](https://arxiv.org/html/2609.05527#S3.E10)\) raises a task’s replay probability where the missing outcome is hard to predict, lowers it where the replay is expensive, and shifts the budget toward whichever comparison sits closer to its threshold\. The baselines below differ in which of those three inputs they are allowed\. Neyman sees prediction difficulty and cost but not the distance to the threshold, uniform replay sees none of them, and the surrogate oracle sees all three with true values in place of pilot estimates\. An improvement over Neyman therefore points to the distance to the threshold, and the gap to the oracle measures what pilot estimation costs\.

Every method receives the same pilot and budget\.

1. 1\.TEAM\-Design\.Algorithm[1](https://arxiv.org/html/2609.05527#alg1), including stabilizationτ\\tauand the probability floor\.
2. 2\.Uniform replay\.Constant replay probabilities chosen to meet the same expected number of baseline replays or the same expected cost\.
3. 3\.Variance\-only Neyman\.Allocationqj​\(W\)∝s~j​\(W\)/cj​\(W\)q\_\{j\}\(W\)\\propto\\sqrt\{\\widetilde\{s\}\_\{j\}\(W\)/c\_\{j\}\(W\)\}subject to the same floor, ceiling, and budget\. This baseline uses heteroskedasticity and cost but not the estimated gain\.
4. 4\.Parameter\-EIG\.In the Gaussian study, the next measurement is the one maximizing expected information gain about the two gains under symmetric priors\.
5. 5\.Decision\-targeted Bayesian\.A Bayesian acquisition utility tied directly to the downstream beats\-both decision and not to generic parameter entropy\. The primary utility is the expected reduction in posterior 0–1 decision loss for the event\{ΔH\>0,ΔA\>0\}\\\{\\Delta\_\{H\}\>0,\\Delta\_\{A\}\>0\\\}, and its sensitivity to the prior is measured across all three specifications\.
6. 6\.Surrogate oracle\.The trueΔj\\Delta\_\{j\}and true residual second moments in place of the pilot estimates in theTEAM\-Designobjective\. This is not a deployable competitor but the same rule given true inputs, so the difference isolates what pilot estimation costs on the criterion\. Because the criterion is a first\-order surrogate for joint power, the oracle can trailTEAM\-Designon the decision rate where it starves the easier comparison to its floor\.

Under positive alternatives the reported quantity is

P⁡\(LH\>0​and​LA\>0\),P\(L\_\{H\}\>0\\ \\text\{and\}\\ L\_\{A\}\>0\),\(35\)whereLH,LAL\_\{H\},L\_\{A\}are one\-sided 95% lower bounds computed from the paired score covariance\. Under a boundary null, the same quantity is the false\-claim rate\. Alongside it we record gain bias, gain standard error, replay count by baseline, total replay cost, fraction of budget spent on the limiting comparison, and gap to the oracle allocation\.

### B\.3Gaussian acquisition study

#### Intuition\.

Symmetric parameter\-information acquisition keeps splitting measurements evenly however unequal the two comparisons become\. A useful test needs a Bayesian comparator whose utility is the beats\-both decision itself, and it needs every method held to the same false\-claim rate, because a rule that reads its sample sizes off the data and then reports a fixed\-sample bound inflates the type\-I error rate\.

#### Setting\.

Two independent comparisons are observed as

Xj​k∼N⁡\(Δj,1\),j∈\{H,A\},X\_\{jk\}\\sim N\(\\Delta\_\{j\},1\),\\qquad j\\in\\\{H,A\\\},\(36\)under a fixed total budgetnH\+nA=nn\_\{H\}\+n\_\{A\}=n, over the grid

\(ΔH,ΔA\)∈\{\(0\.02,0\.02\),\(0\.02,0\.05\),\(0\.02,0\.10\),\(0\.05,0\.10\)\},\(\\Delta\_\{H\},\\Delta\_\{A\}\)\\in\\\{\(0\.02,0\.02\),\(0\.02,0\.05\),\(0\.02,0\.10\),\(0\.05,0\.10\)\\\},\(37\)whose first cell is balanced and whose others place the two comparisons at different distances apart\. The headline comparison sits at\(0\.02,0\.10\)\(0\.02,0\.10\), the widest of them, with a budget of 20,000 measurements and the pilot charged against it\. Four rules commit to an allocation before measuring, namelyTEAM\-Design, a decision\-targeted Bayesian batch design, parameter\-EIG under symmetric priors, and equal allocation, while a fifth interleaves acquisition with estimation\. The batch design maximizes the expected posterior probability that both held\-out bounds clear zero, so it evaluates the Bayesian utility exactly, and as the pilot sharpens it converges on the split 0\.907 that maximizes true joint power\. At the pilot size used here it allocates 0\.77\. We report power, the false\-claim rate at the least\-favorable nullΔH=0\\Delta\_\{H\}=0, and size\-corrected power, which calibrates each critical value to an exact 5% false\-claim rate\. Run under a correctly centered prior, a diffuse one, and one deliberately misspecified, the decision\-targeted rule averages 77\.9%, 77\.6% and 77\.6% across the four cells, so how its prior is set barely matters\.

#### Finding\.

Table[4](https://arxiv.org/html/2609.05527#A2.T4)reports the study\. Among rules that commit before measuring,TEAM\-Designreaches 76\.3% against 75\.2% for the Bayesian batch design, 59\.5% for parameter\-EIG and 60\.1% for equal allocation, every one of them within Monte Carlo error of the nominal false\-claim rate\. Parameter\-EIG splits at 0\.50, the even split that symmetric parameter\-information acquisition implies however unequal the comparisons become\. The stabilizer is what keepsTEAM\-Designoff the minimax corner, moving the split on true inputs from 0\.962 to 0\.893 against a joint\-power optimum of 0\.907, and lifting power from 70\.2% to 81\.1% where the best attainable is 81\.3%\. From the pilot,TEAM\-Designrealizes 0\.83\.

The measurements needed to reach a power target follow the same pattern\. At a target of0\.700\.70, where the two gains are equal at0\.020\.02, the rule needs 52,000 against 40,000 for parameter\-EIG, and where they are furthest apart it needs 20,000 against 26,000\. At the higher target of0\.800\.80the balanced cell levels at 52,000 apiece and the imbalanced one widens to 26,000 against 40,000\. The two intermediate cells tie at both targets, except at\(0\.05,0\.10\)\(0\.05,0\.10\), whereTEAM\-Designneeds 8,000 against 6,000 at the higher target\.

Table 4:Gaussian acquisition at\(ΔH,ΔA\)=\(0\.02,0\.10\)\(\\Delta\_\{H\},\\Delta\_\{A\}\)=\(0\.02,0\.10\)with a shared pilot charged to a common budget\. Every method spends the same total\.*False claim*is measured at the least\-favorable nullΔH=0\\Delta\_\{H\}=0\.*Size\-corrected power*calibrates each critical value so all methods have exactly 5% false\-claim rate; it needs the null distribution, so it upper\-bounds any implementable correction\. Every rule commits to an allocation before measuring, so every fixed\-sample bound is exact\.R=60,000R=60,000at the null andR=30,000R=30,000at the alternative\.A rule that interleaves measuring with estimating scores higher still, but it reads its sample sizes off the data and then reports a fixed\-sample bound, so it runs above the nominal false\-claim rate and is not comparable\. Table[4](https://arxiv.org/html/2609.05527#A2.T4)therefore reports only rules that commit before measuring, together with the oracle\. Where replays must be scheduled in advance, the closed\-form rule does at least as well as a Bayesian design that has to integrate over the pilot posterior\.

### B\.4Sensitivity analyses

The pilot supplies the predictors and the variance estimates, but it does not supply everything the design needs\. Each remaining input is varied here while the others are held at the values used throughout, so that a reader can see which of them the conclusion depends on\. Three come with the evaluation and cannot be chosen, namely the replay cost ratio, the correlation between potential outcomes, and the two sample sizes\. Table[5](https://arxiv.org/html/2609.05527#A2.T5)sweeps those three\. Raising the cost ratio narrows the advantage steadily and by ten to one has all but closed it, since a budget buying fewer human replays leaves less room to shift any of them\. Correlation leaves the pattern intact, and up toρ=0\.75\\rho=0\.75TEAM\-Designstill leads where one comparison is harder and still trails where the two are balanced\. Sample size decides whether an advantage can exist at all\. In the smallest evaluation pool no rule resolves either comparison and the four agree, in the largest every rule succeeds and they converge again, and the gain lives between those ends\. Enlarging the pilot liftsTEAM\-Design, Neyman and the oracle together and leavesTEAM\-Design’s lead intact\. Uniform replay does not use the pilot to allocate, and its small movements are within Monte Carlo error\.

Table 5:Sensitivity sweeps\. Beats\-both decision rate with Monte Carlo standard errors in parentheses\. The uniform column is an equal cost split in the unequal\-cost rows\. Cost\-ratio rows use 20,000 replications and the 5:1 row repeats Table[2](https://arxiv.org/html/2609.05527#S5.T2), while the correlation and size sweeps use 2,000\. Thenpilot=500n\_\{\\mathrm\{pilot\}\}=500andneval=3,000n\_\{\\mathrm\{eval\}\}=3\{,\}000rows are the primary setting run once in each sweep\. sweeps use 2,000\.The four below are the opposite\. Each is picked before the run, namely the improvement required of each baseline, the replay floor, the stabilizer, and the replay budget\. Each is varied with the other three held at their primary values, so any change in the result is attributable to that quantity alone\. The floor sweep serves a second purpose, distinguishing two explanations for the semi\-synthetic result of Section[5\.3](https://arxiv.org/html/2609.05527#S5.SS3)\.

#### Minimum required improvement\.

Two choices in \([2](https://arxiv.org/html/2609.05527#S2.E2)\) belong to the organization, and both are one\-line changes that leave the allocation theory intact\. A minimum required improvement enters throughd~j2\\widetilde\{d\}\_\{j\}^\{2\}in \([9](https://arxiv.org/html/2609.05527#S3.E9)\), and atδH=δA=0\.02\\delta\_\{H\}=\\delta\_\{A\}=0\.02the decision rate falls to 30\.9% forTEAM\-Designagainst 25\.4% for Neyman, a paired gain of 5\.5 points at 14\.2 standard errors\.

#### Probability floor\.

Section[5\.3](https://arxiv.org/html/2609.05527#S5.SS3)leaves two explanations open for whyTEAM\-Designwins on the gain against the harder baseline and not on the decision\. Either the true agent gain is too small for any budget to resolve, orTEAM\-Designstarves the easier comparison so badly that its bound fails and takes the joint decision down with it\. The probability floorqj,minq\_\{j,\\min\}of \([3](https://arxiv.org/html/2609.05527#S2.E3)\) separates the two, because raising it forbids the starvation and changes nothing else\. We sweep it over\{0\.05,0\.10,0\.20,0\.30,0\.40,0\.50\}\\\{0\.05,0\.10,0\.20,0\.30,0\.40,0\.50\\\}applied identically to every rule, on the agent\-limited regime at the same replication count as Section[5\.3](https://arxiv.org/html/2609.05527#S5.SS3)\.

Raising the floor does stop the starvation\.TEAM\-Design’s detection of the human gain at a fifth of full replay climbs from 77\.5% at the shipped floor of0\.050\.05to 98\.0% at0\.200\.20, and reaches 100% above that\. Were starvation the cause, the decision rate should climb with it\. It stays flat\. Across the six floors and the six budgets below full replayTEAM\-Designleads Neyman in 10 of the 36 comparisons and trails in 25, the remaining pair being a tie\. A one\-sided sign test on the 35 untied pairs puts the advantage with Neyman atp=0\.008p=0\.008\. At the highest floors it is the floor and no longer the rule that fixes the allocation, its realized share falling to the0\.400\.40set by the cost ratio, and there even the precision advantage inverts\. The starvation was real, removing it changes nothing, and the resolution account of Section[5\.3](https://arxiv.org/html/2609.05527#S5.SS3)is what stands\.

#### Stabilizer\.

At the Gaussian configuration of Appendix[B\.3](https://arxiv.org/html/2609.05527#A2.SS3), size\-corrected power againstτ\\tau\(Figure[3](https://arxiv.org/html/2609.05527#S5.F3)b\) runs 59\.5% atτ=0\.0025\\tau=0\.0025, rising to 75\.3% atτ=0\.03\\tau=0\.03and falling to 70\.0% atτ=0\.1\\tau=0\.1\. The pilot takes a tenth of that budget,1,0001\{,\}000measurements per comparison at unit variance, so the standard error of a single gain is1/1000≈0\.0321/\\sqrt\{1000\}\\approx 0\.032, and the optimum sits there\. Smallτ\\taumakes the allocation track pilot noise, and largeτ\\tauerases the bottleneck signal\.

#### Budget\.

Sweeping the total Gaussian budget,TEAM\-Designfirst matches parameter\-EIG at 8,000 measurements, moving from 32\.9% against 31\.3% there to 76\.6% against 58\.8% at 20,000\. Below the crossover neither gain can be resolved and the first\-order argument behind Proposition[4\.1](https://arxiv.org/html/2609.05527#S4.Thmtheorem1)does not apply\.

## Appendix CBroader impacts

Requiring a workflow to beat both alternatives raises the bar for deployment, and Section[5\.1](https://arxiv.org/html/2609.05527#S5.SS1)shows the bar is not always met, since the assisted clinician improves on the clinician alone in all six settings there and on the model alone in none\. Holding evaluations to the harder standard should discourage deployments that add cost or risk without adding value\. The converse risk is that a design targeting one decision invites reading it as more than it is\. Certification here is only as complete as the outcome measure and the thresholds it is given, so a workflow that beats both alternatives on task success may still be unsuitable on safety, equity, workload, or under distribution shift, none of which this design measures\. The method also assumes the deployed workflow can be replayed under each alternative; where that is impossible, the question it answers cannot be asked\.