READY or Not: Reliable Enterprise Agent Deployment

# READY or Not: Reliable Enterprise Agent Deployment
Source: [https://arxiv.org/html/2609.02095](https://arxiv.org/html/2609.02095)
Bryan Zhu\*Affiliation:Scale AIJingxuan Fan\*Affiliation:Scale AIGeorge PuAffiliation:Scale AISoham Dinesh TiwariAffiliation:Scale AISoham DanAffiliation:Scale AIRyan YoungAffiliation:Scale AIYuan \(Christy\) LiAffiliation:Scale AIYuang YaoAffiliation:Scale AIApaar ShankerAffiliation:Scale AIMinglai YangAffiliation:Scale AIDaniel Yue ZhangAffiliation:Scale AIYunzhong HeAffiliation:Scale AIYing LiuAffiliation:Scale AIChenguang WangAffiliation:Scale AIAffiliation:University of California, Santa CruzZhijun YinAffiliation:Scale AIAffiliation:Vanderbilt University Medical CenterYuan \(Emily\) XueAffiliation:Scale AIAffiliation:Co\-first authors\.

###### Abstract

An AI agent can perform well on benchmarks and still be unsuitable for deployment\. Existing AI\-agent benchmarks primarily measure whether an agent can complete realistic professional work, whereas enterprise deployment requires asking a different question: whether an agent can meet a required reliability level, under an acceptable level of human oversight, and at a tolerable cost\. We introduce*Reliable Enterprise Agent Deployment*\(READY\), an evaluation framework for qualifying AI agents for deployment on concrete enterprise workflows\. READY preserves each workflow’s own definition of successful execution while applying a common deployment\-qualification procedure\. Given an agent, a workflow, and a class of candidate oversight policies, READY measures the reliability and operating cost of the human\-AI system, selects the minimum\-cost policy that satisfies a specified reliability target, and statistically qualifies the selected policy on held\-out cases\. The*deployment profile*characterizes the operating point supported by the evidence, including reliability, human\-oversight burden, and cost\. READY is implemented as an open testbed that decouples workflow specification, execution, evaluation, and deployment qualification, and runs on existing agent\-evaluation infrastructure\. In an end\-to\-end clinical\-audit case study spanning 16 agent systems and 750 cases, READY reveals deployment differences hidden by autonomous benchmark performance: two systems separated by only 0\.3 percentage points in autonomous accuracy \(72\.8% vs\. 72\.5%\) require 39\.2% versus 29\.6% human review, respectively, to qualify at the same 76% reliability target under the evaluated oversight policy\. Systems with nearly identical autonomous performance can support substantially different reliability–oversight tradeoffs\. Accordingly, READY shifts enterprise agent evaluation from asking only*how well can the agent perform the work?*to asking*under what conditions, and at what cost, can it be reliably deployed?*By making those conditions explicit and statistically testable, READY provides a practical basis for comparing agent systems, setting oversight requirements, and making evidence\-based deployment decisions\.

## 1Introduction

Enterprises are starting to hand real professional work to AI agents that reason over domain knowledge, use tools, and carry out multi\-step tasks\. A growing number of benchmarks evaluate these capabilities on professional workflows\[[1](https://arxiv.org/html/2609.02095#bib.bib26),[2](https://arxiv.org/html/2609.02095#bib.bib27),[3](https://arxiv.org/html/2609.02095#bib.bib28)\], scoring whether an agent can complete the task and how good the result is\. These benchmarks mainly answer an important question about*capability*\. However, whether to deploy an agent is a different question\. An organization rarely needs an agent to work unattended\. Rather, it deploys a system in which some work may be handled autonomously while other work is reviewed, corrected, approved, or taken over by humans\. A more relevant question is not how often the agent succeeds on its own, but whether the human–AI system around it can achieve the reliability a particular workflow requires, how much human oversight that takes, and what the operating policy costs\. An agent that is right on 80% of cases is not, by that fact alone, ready or unready to deploy\. What matters is whether the wrong 20% can be identified and sent to a human, and how much that review adds to the cost of running the system\.

We introduce*Reliable Enterprise Agent Deployment*\(READY\), a framework for turning workflow\-level evaluation evidence into a deployment qualification\. READY takes as input an agent, an enterprise workflow, a set of representative cases, and a class of candidate oversight policies\. Each workflow retains its own definition of successful execution, including outcome correctness, process requirements, evidence grounding, policy adherence, and other requirements appropriate to that form of work\. READY then evaluates the human–AI system under candidate oversight policies and asks which policy can satisfy a specified reliability target at minimum operating cost\.

> *Professional\-work benchmarks ask how well an agent can do the work autonomously\. READY asks under what conditions, and at what cost, an organization can reliably deploy it\.*

At its core, READY casts deployment qualification as a constrained optimization problem over a class of oversight policies\. For each workflow, the evaluator maps a multidimensional set of deployment\-relevant criteria into a measure of successful execution, while the oversight policy specifies the points at which human intervention may occur\. Agent executions on representative cases provide the empirical evidence for estimating the reliability and operating cost induced by each candidate policy\. READY then searches the policy class for the lowest\-cost operating policy that satisfies a specified reliability target and any additional deployment constraints\. The selected policy is frozen and statistically qualified on held\-out cases not used during policy selection\. The resulting*deployment profile*characterizes the reliability, oversight burden, cost, and workflow\-specific risk of the qualified human–AI configuration\.

We use a retrospective clinical\-audit workflow\[[4](https://arxiv.org/html/2609.02095#bib.bib75)\]as a running case study to illustrate READY end to end\. In this setting, an agent receives a longitudinal patient record and a clinical audit question, such as whether an ICU patient developed acute kidney injury within 48 hours of a CT scan\. It must identify the relevant evidence in the record, apply the governing clinical standard, and return a verdict together with a stated confidence\. A capability benchmark stops at the verdict and scores it\. However, deployment requires an additional decision: which completed audits can be accepted automatically and which should be escalated to a clinician\. That oversight policy, rather than raw verdict accuracy alone, determines the reliability of the human–AI system, the review burden it imposes, and its operating cost\.

Across 16 agent systems and 750 cases, the deployment profiles reveal differences that autonomous benchmark performance alone does not\. For example, two systems separated by only 0\.3 percentage points in autonomous accuracy \(GPT\-5\.4’s 72\.8% versus Sonnet 5’s 72\.5%\) require 39\.2% versus 29\.6% human review, respectively, to qualify at the same 76% reliability target under the evaluated oversight policy\. More generally, the reliability–oversight frontier shows that systems with similar autonomous performance can support substantially different operating points\. On a conventional leaderboard, the two systems appear nearly identical, yet their deployment costs differ substantially\.

#### What READY adds\.

READY complements rather than replaces existing agent evaluations\. Capability and work\-product benchmarks\[[1](https://arxiv.org/html/2609.02095#bib.bib26),[2](https://arxiv.org/html/2609.02095#bib.bib27),[3](https://arxiv.org/html/2609.02095#bib.bib28)\]measure the quality of agent execution but do not determine an operating point for a deployed human–AI system\. The*passk*metric ofτ\\tau\-bench\[[5](https://arxiv.org/html/2609.02095#bib.bib71)\]measures consistency across repeated attempts but does not incorporate human oversight or operating cost\. Cost\-aware routing and cascades\[[6](https://arxiv.org/html/2609.02095#bib.bib77),[7](https://arxiv.org/html/2609.02095#bib.bib4)\]optimize quality–cost tradeoffs by routing among models, while selective prediction and learning\-to\-defer study when a model should abstain or pass a case to a human\[[8](https://arxiv.org/html/2609.02095#bib.bib123)\]\. READY brings these ideas into a workflow\-level deployment setting: it evaluates the reliability and cost of the complete human–AI system, optimizes an oversight policy against an explicit reliability requirement, and statistically qualifies the selected operating point on held\-out evidence\. It reuses the evidence produced by existing workflow evaluations instead of displacing them\.

#### An open, extensible testbed\.

To support heterogeneous enterprise workflows, READY separates workflow\-specific definitions of successful work from a common deployment\-qualification procedure\. We implement READY as an open, extensible testbed built on existing agent\-evaluation infrastructure\[[9](https://arxiv.org/html/2609.02095#bib.bib24)\]\. Workflows may differ in their task populations, execution environments, evaluation criteria, and forms of oversight while sharing the same qualification abstraction\. We provide a seed workflow to demonstrate complete end\-to\-end use of this interface, and new workflows, cases, environments, and evaluators can be contributed while retaining their own standards of correct work\.

Our contributions are as follows\.

1. 1\.A deployment\-centered evaluation objective\.We formulate agent deployment as evaluation of a human–AI system under an oversight policy, moving from the question of how well an agent performs autonomously to which policy can meet a workflow\-specific reliability requirement and at what operating cost \(Section[2](https://arxiv.org/html/2609.02095#S2)\)\.
2. 2\.A method for optimizing and statistically qualifying oversight\.READY selects a minimum\-cost oversight policy subject to a reliability target and qualifies the frozen policy on held\-out data\. The deployment profile characterizes the reliability–oversight–cost operating point supported by the evidence, not autonomous capability alone \(Section[3](https://arxiv.org/html/2609.02095#S3)\)\.
3. 3\.An open evaluation interface and end\-to\-end empirical instantiation\.We provide an extensible interface that separates workflow evaluation from deployment qualification and instantiate it on a retrospective clinical\-audit workflow\[[4](https://arxiv.org/html/2609.02095#bib.bib75)\]across 16 agent systems and 750 cases, demonstrating that similar autonomous performance can correspond to substantially different reliability–oversight tradeoffs \(Sections[4](https://arxiv.org/html/2609.02095#S4)–[5](https://arxiv.org/html/2609.02095#S5)\)\.

## 2Design Overview

We now formalize the READY deployment\-qualification problem, which asks whether an agent can be deployed reliably and economically on a concrete enterprise workflow\. For a given workflow and agent, READY evaluates the human–AI system induced by an*oversight policy*,π\\pi\. The policy specifies when the agent may operate autonomously and when human intervention may be introduced\. Depending on the workflow, intervention may occur after a completed result or during execution through review, approval, clarification, correction, retry, escalation, or takeover\. At a high level, READY separates three phases:

Workflow Evaluation⟶\\;\\longrightarrow\\;Policy Optimization⟶\\;\\longrightarrow\\;Deployment Qualification\.

Workflow evaluation defines and measures what constitutes successful execution for a particular form of professional work\. Policy optimization uses this evidence to select, from a specified class of oversight policies, the lowest\-cost policy that satisfies the deployment requirements\. Deployment qualification then freezes the selected policy and statistically evaluates it on held\-out evidence\. The output of this process is a*deployment profile*describing the reliability–oversight–cost operating point supported by the evidence under the stated deployment assumptions\. Figure[1](https://arxiv.org/html/2609.02095#S2.F1)summarizes the end\-to\-end process\.

Figure 1:*Reliable Enterprise Agent Deployment*\(READY\) takes as input a specification defining the workflow, representative task instances, execution environment, evaluation semantics, oversight policy class, and deployment requirements\. Within the testbed, multiple agentic systems can be evaluated against the same specification\. For each system, READY proceeds through three phases: \(1\) agent execution and workflow evaluation generate instance\-level execution evidence; \(2\) oversight policy optimization selects the lowest\-cost policy satisfying the specified reliability and risk constraints; and \(3\) statistical qualification freezes the selected policy and evaluates it on held\-out cases\. The output is a deployment profile for each evaluated system, characterizing the reliability, human\-oversight burden, operating cost, and workflow\-specific risk supported by the evidence under the stated deployment assumptions\.### 2\.1Evaluation Setting and Reliable Deployment Objective

A workflow,ww, denotes a reusable type of enterprise work\. LetEwE\_\{w\}denote the execution environment associated with workflowww, including the tools, data interfaces, systems, and interaction capabilities available during execution\. A task instance,xx, specifies one concrete case on which the workflow is performed\. It contains the inputs made available to the agent, together with the reference material and ground truth required to evaluate that instance\. Let𝒟w\\mathcal\{D\}\_\{w\}denote the population distribution over task instances of the workflow\. An evaluation set,SwS\_\{w\}, is a finite sample of instances,

Sw=\{xi\}i=1Nw,xi​∼iid​𝒟w\.S\_\{w\}=\\\{x\_\{i\}\\\}\_\{i=1\}^\{N\_\{w\}\},\\qquad x\_\{i\}\\overset\{\\text\{iid\}\}\{\\sim\}\\mathcal\{D\}\_\{w\}\.\(1\)
Letmmdenote the evaluated agent\. Executingmmon task instance,xix\_\{i\}, in environment,EwE\_\{w\}, under oversight policy,π\\pi, produces a trajectory:

Ziπ=Execute⁡\(m,xi,Ew,π\)\.Z\_\{i\}^\{\\pi\}=\\operatorname\{Execute\}\(m,x\_\{i\},E\_\{w\};\\pi\)\.\(2\)
The trajectory contains the final work product and the observable evidence needed to evaluate the work, including, when relevant, retrieved evidence, tool interactions, intermediate artifacts, interventions, state changes, and resource use\. If the agent, environment, or human oversight is stochastic,ZiπZ\_\{i\}^\{\\pi\}and all derived quantities are random variables under the corresponding deployment process\.

Each workflow defines a workflow\-specific evaluator,gwg\_\{w\}, that maps a trajectory to a vector of measurements:

𝐪iπ=gw​\(Ziπ\),\\mathbf\{q\}\_\{i\}^\{\\pi\}=g\_\{w\}\\\!\\left\(Z\_\{i\}^\{\\pi\}\\right\),\(3\)
where𝐪iπ\\mathbf\{q\}\_\{i\}^\{\\pi\}may characterize both the final outcome and relevant aspects of the execution process, such as evidence use, adherence to required procedures, or compliance with applicable policies\. The workflow also defines a pre\-specified*workflow\-success predicate*,ϕw\\phi\_\{w\}:

uiπ=ϕw​\(𝐪iπ\)∈\{0,1\},u\_\{i\}^\{\\pi\}=\\phi\_\{w\}\(\\mathbf\{q\}\_\{i\}^\{\\pi\}\)\\in\\\{0,1\\\},\(4\)
whereuiπ=1u\_\{i\}^\{\\pi\}=1indicates that final outcome and any required process or policy conditions are satisfied on instanceii\. The vector𝐪iπ\\mathbf\{q\}\_\{i\}^\{\\pi\}retains multidimensional diagnostic information about the execution, whileuiπu\_\{i\}^\{\\pi\}is an instance\-level execution outcome used to define deployment reliability\.

This separation distinguishes measurement from qualification:gwg\_\{w\}determines what aspects of the work are measured, whileϕw\\phi\_\{w\}specifies which combination of those measurements constitutes a successful execution\. For professional workflows, the success predicate may require a conjunction of outcome correctness with selected process, grounding, or policy requirements, not final\-answer correctness alone:

Ziπ→gw𝐪iπ⏟workflow measurements→ϕwuiπ⏟instance\-level successZ\_\{i\}^\{\\pi\}\\;\\xrightarrow\{\\;g\_\{w\}\\;\}\\;\\underbrace\{\\mathbf\{q\}\_\{i\}^\{\\pi\}\}\_\{\\text\{workflow measurements\}\}\\;\\xrightarrow\{\\;\\phi\_\{w\}\\;\}\\;\\underbrace\{u\_\{i\}^\{\\pi\}\}\_\{\\text\{instance\-level success\}\}\(5\)
For agentmmon workflowwwunder oversight policyπ\\pi, we define deployment reliability as

Rm,w​\(π\)=𝔼x∼𝒟w​\[uπ​\(x\)\]\.R\_\{m,w\}\(\\pi\)=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\_\{w\}\}\\left\[u^\{\\pi\}\(x\)\\right\]\.\(6\)
LetKw​\(Zπ\)K\_\{w\}\(Z^\{\\pi\}\)denote the workflow\-specific operating cost associated with an execution under policyπ\\pi, including agent execution cost and the cost of human oversight\. The expected operating cost is

Cm,w​\(π\)=𝔼x∼𝒟w​\[Kw​\(Zπ\)\]\.C\_\{m,w\}\(\\pi\)=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\_\{w\}\}\\left\[K\_\{w\}\(Z^\{\\pi\}\)\\right\]\.\(7\)
READY then casts oversight\-policy selection as a constrained optimization problem over a candidate policy classΠw\\Pi\_\{w\}\. Given a target reliabilityYY, and optionally a workflow\-specific risk toleranceBwB\_\{w\}, define the feasible set

ℱm,w\(Y,Bw\)=\{π∈Πw:Rm,w\(π\)≥Y,ρw\(Lw\(Zπ\)\)≤Bw\},\\mathcal\{F\}\_\{m,w\}\(Y,B\_\{w\}\)=\\left\\\{\\pi\\in\\Pi\_\{w\}:R\_\{m,w\}\(\\pi\)\\geq Y,\\;\\rho\_\{w\}\\\!\\left\(L\_\{w\}\(Z^\{\\pi\}\)\\right\)\\leq B\_\{w\}\\right\\\},\(8\)
whereLwL\_\{w\}is a workflow\-specific deployment loss andρw\\rho\_\{w\}is a corresponding risk functional\. The risk constraint is optional; when mean reliability is sufficient, it is inactive, equivalentlyBw=\+∞B\_\{w\}=\+\\infty\. For workflows in which the severity or concentration of failures matters,ρw\\rho\_\{w\}may instead represent a chance constraint or a tail\-risk measure such as Conditional Value at Risk \(CVaR\)\.

Ifℱm,w​\(Y,Bw\)=∅\\mathcal\{F\}\_\{m,w\}\(Y,B\_\{w\}\)=\\varnothing, no policy in the evaluated policy class supports the requested deployment operating point\. Otherwise, READY selects the lowest\-cost feasible policy:

πm,w⋆​\(Y,Bw\)∈arg⁡minπ∈ℱm,w​\(Y,Bw\)​Cm,w​\(π\)\.\\boxed\{\\pi^\{\\star\}\_\{m,w\}\(Y,B\_\{w\}\)\\in\\arg\\min\_\{\\pi\\in\\mathcal\{F\}\_\{m,w\}\(Y,B\_\{w\}\)\}C\_\{m,w\}\(\\pi\)\.\}\(9\)
Importantly, Equation[9](https://arxiv.org/html/2609.02095#S2.E9)defines*policy selection*, not statistical qualification\. The policy is selected using development evidence, then frozen and evaluated on held\-out qualification data to determine whether the requested reliability and any additional deployment constraints are statistically supported\. Section[3](https://arxiv.org/html/2609.02095#S3)develops the estimation of reliability, operating cost, risk, and statistical qualification in detail\.

### 2\.2End\-to\-End READY Flow

Figure[1](https://arxiv.org/html/2609.02095#S2.F1)organizes READY around its inputs, three evaluation phases, and resulting deployment profiles\. The input to READY is a specification defining the workflow, its task instances, associated execution environment, oversight policy class and evaluation criteria, including both the workflow\-specific evaluator and the instance\-level success criterion\. Within the READY testbed, a set of agentic systems is evaluated against this specification\. For each system, READY executes and evaluates the workflow, optimizes an oversight policy, and statistically qualifies the selected policy on held\-out cases\. The output is a deployment profile for each evaluated system\.

#### Inputs: specification\.

READY begins from declarative specifications that define the deployment question to be evaluated\. The*workflow specification*identifies the workflowww, task population𝒟w\\mathcal\{D\}\_\{w\}, sampled evaluation instancesSwS\_\{w\}, and execution environmentEwE\_\{w\}\. The*evaluation specification*defines the workflow evaluatorgwg\_\{w\}, the workflow\-success predicateϕw\\phi\_\{w\}, and the observable signals or state available to oversight policies\. The*deployment specification*defines the candidate policy classΠw\\Pi\_\{w\}, target reliabilityYY, human\-oversight model, operating\-cost model, and any workflow\-specific risk tolerance\. Keeping these specifications separate allows the same workflow evidence to be interpreted under different deployment requirements\.

#### Phase 1: agent execution and workflow evaluation\.

READY can evaluate multiple agent systemsmmagainst the same deployment specification, producing comparable deployment profiles for each\. For each evaluated systemmmand task instancexix\_\{i\}, READY executes the agent in the workflow environmentEwE\_\{w\}and records the trajectoryZiZ\_\{i\}, including the final work product, tool interactions, retrieved evidence, intermediate artifacts, interventions, and resource use\. The workflow evaluator maps this trajectory into workflow\-specific measurements,Zi→gw𝐪iZ\_\{i\}\\xrightarrow\{g\_\{w\}\}\\mathbf\{q\}\_\{i\}, and the success predicate maps those measurements into an instance\-level success outcome,𝐪i→ϕwui\\mathbf\{q\}\_\{i\}\\xrightarrow\{\\phi\_\{w\}\}u\_\{i\}\. The resulting evidence also includes policy\-observable signals and cost measurements needed to evaluate candidate oversight policies\.

#### Phase 2: oversight policy optimization\.

Using development evidence, READY estimates the reliability, operating cost, and any workflow\-specific risk induced by candidate policies inΠw\\Pi\_\{w\}\. It then selects the lowest\-cost policy satisfying the specified deployment constraints:

π⋆∈arg⁡minπ∈Πw⁡C⁡\(π\)subject toR⁡\(π\)≥Y,\\pi^\{\\star\}\\in\\arg\\min\_\{\\pi\\in\\Pi\_\{w\}\}C\(\\pi\)\\quad\\text\{subject to\}\\quad R\(\\pi\)\\geq Y,together with any additional risk constraint\. When oversight is trajectory\-invariant, such as terminal accept\-or\-escalate review, candidate policies can be evaluated by replaying saved execution evidence\. When oversight is trajectory\-dependent, intervention changes what happens next, candidate policies must be executed or simulated in the runtime\.

#### Phase 3: deployment qualification\.

Policy optimization alone does not establish deployability\. After the policyπ⋆\\pi^\{\\star\}is selected, READY freezes it and evaluates the human–AI system on held\-out qualification cases not used during policy selection\. Deployment qualification is granted only when the held\-out evidence statistically supports the specified reliability target and any additional deployment constraints\.

#### Output: deployment profiles\.

The output of READY is a set of*deployment profiles*, one for each evaluated agentic system\. A deployment profile reports the qualified policy, qualification status, achieved reliability and statistical confidence, human\-oversight burden, operating cost, workflow\-specific risk, and the scope and assumptions under which the profile holds\. The deployment profile connects benchmark evidence to an operational decision: whether, and under what oversight policy, the agent can be deployed at the required level of reliability and cost\.

Section[3](https://arxiv.org/html/2609.02095#S3)develops the qualification methodology in detail, while Section[4](https://arxiv.org/html/2609.02095#S4)describes the evaluation runtime and open interfaces used to implement this architecture across enterprise workflows\.

### 2\.3Running Example: Retrospective Clinical Audit

Throughout the paper, we use a retrospective clinical\-audit workflow from CliniCARE\-Bench\[[4](https://arxiv.org/html/2609.02095#bib.bib75)\]as a running example to instantiate the READY framework\.

In CliniCARE\-Bench, for each patient\-specific case, an agent receives a concise clinical\-audit question and access to a longitudinal patient record\. It must determine what evidence is needed, retrieve and reconcile that evidence, apply the relevant clinical or operational standard, and produce an auditable adjudication\.

In READY notation, retrospective clinical audit defines the workflowww\. A task instancexix\_\{i\}consists of a clinical\-audit question paired with the relevant patient record and, where applicable, a particular encounter or clinical event\. The environmentEwE\_\{w\}provides patient\-scoped access to structured and free\-text EHR data through clinically meaningful retrieval tools, together with facilities for explicit computation and access to governing policy documents\.

Executing agentmmon casexix\_\{i\}produces a complete evaluation run\. The final report contains one of four verdicts—*Yes*,*No*,*Indeterminate: Lack of Data*, or*Indeterminate: Medically Ambiguous*—together with supporting patient\- and policy\-evidence citations and a stated confidencesis\_\{i\}\. The recorded trajectoryZiZ\_\{i\}additionally contains observable investigation evidence such as tool calls, retrieved information, computations, and intermediate artifacts\.

For example, one scenario asks whether an ICU patient developed acute kidney injury within 48 hours after a CT scan\. The agent must establish baseline creatinine using a specified hierarchy, identify the CT execution time, inspect the relevant post\-CT creatinine trajectory, check chronic dialysis and ESRD \(end\-stage renal disease\) status, apply the KDIGO\[[10](https://arxiv.org/html/2609.02095#bib.bib68)\]creatinine criteria, and cite the record evidence supporting the resulting verdict\. The evaluation needs to inspect whether the agent retrieved the necessary evidence, used the appropriate temporal window and clinical rule, and avoided unsupported shortcuts\. The workflow illustrates why professional\-work evaluation may need to measure more than final\-answer correctness alone\.

The workflow evaluator,gwg\_\{w\}, measures multiple aspects of the observable execution, including verdict correctness, appropriate abstention, evidence grounding, policy grounding, and adherence to scenario\-specific investigation requirements\. The workflow\-success predicate,ϕw\\phi\_\{w\}, specifies which of these measurements are required for an execution to count as successful\. For example, an illustrative process\-aware success definition may take the form

ui=𝟏​\{verdict correct∧no disqualifying process defect\}\.u\_\{i\}=\\mathbf\{1\}\\left\\\{\\text\{verdict correct\}\\;\\land\\;\\text\{no disqualifying process defect\}\\right\\\}\.\(10\)
The clinical\-audit case study uses a terminal accept\-or\-escalate policy class\. Here, the agent’s stated confidencesis\_\{i\}is recorded as the policy\-observable signal, and the oversight policy is defined based on how that signal is used to route a completed adjudication between autonomous acceptance and human review\. Confidence is therefore a property of this particular policy instantiation, not a requirement of the general READY framework\.

Because the routing decision occurs only after the agent has completed the audit, changing the policy does not change the underlying agent trajectoryZiZ\_\{i\}\. The execution evidence is trajectory\-invariant, allowing multiple candidate oversight policies to be evaluated from the same saved runs\.

Given a reliability target and deployment assumptions about human\-review effectiveness and operating cost, READY uses this evidence to select an oversight policy and then statistically qualify the frozen operating point\. The clinical\-audit workflow provides a concrete instance of the trajectory\-invariant policy\-evaluation setting developed more fully in Section[3](https://arxiv.org/html/2609.02095#S3)\.

## 3Oversight Policy Optimization and Deployment Qualification

Section[2](https://arxiv.org/html/2609.02095#S2)formulated reliable deployment as a constrained optimization problem over oversight policies\. We now develop the quantities, policy\-optimization procedures, and statistical qualification methods needed to operationalize that formulation\.

The key distinction is between*agent performance*and*deployed\-system reliability*\. An oversight policy determines how autonomous agent execution is combined with human intervention\. Its value depends not only on how often the agent succeeds on its own, but also on how effectively the policy uses observable workflow information to introduce oversight, how effective that oversight is, and what it costs\.

We first develop the trajectory\-invariant terminal accept\-or\-escalate setting, where many candidate policies can be evaluated from the same saved execution evidence\. We then describe statistical qualification of a selected policy on held\-out cases and extend the formulation to trajectory\-dependent oversight, where intervention changes the execution trajectory itself\.

### 3\.1Qualification Quantities

For agentmm, workflowww, and oversight policyπ\\pi, letuπ∈\{0,1\}u^\{\\pi\}\\in\\\{0,1\\\}denote the workflow\-specific success event defined in Section[2\.1](https://arxiv.org/html/2609.02095#S2.SS1)\. The eventuπu^\{\\pi\}records whether an individual workflow execution under policyπ\\pisatisfies the pre\-specified success criterion\.

We define the reliability of the human–AI system as

Rm,w​\(π\)=𝔼x∼𝒟w​\[uπ​\(x\)\],R\_\{m,w\}\(\\pi\)=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\_\{w\}\}\\left\[u^\{\\pi\}\(x\)\\right\],\(11\)whereuπ​\(x\)∈\{0,1\}u^\{\\pi\}\(x\)\\in\\\{0,1\\\}denotes whether execution on workflow instancexxsatisfies the workflow\-specific success criterion\. The expectation is over workflow instancesxxdrawn from the population represented by𝒟w\\mathcal\{D\}\_\{w\}and, when execution is stochastic, over the induced randomness in agent execution, the environment, oversight process, and interacting actors\. Sinceuπ​\(x\)u^\{\\pi\}\(x\)is binary,Rm,w​\(π\)R\_\{m,w\}\(\\pi\)is the probability that an execution under policyπ\\pisatisfies the workflow\-specific success criterion on a randomly drawn workflow instance\. Deployment qualification is an aggregate statistical claim about the human–AI system induced byπ\\pi, not a property assigned to any individual task instance\.

LetKw​\(Zπ\)K\_\{w\}\(Z^\{\\pi\}\)denote the workflow\-specific operating cost associated with an execution under policyπ\\pi\. This may include agent execution as well as human review and intervention\. The expected operating cost is

Cm,w​\(π\)=𝔼⁡\[Kw​\(Zπ\)\]\.C\_\{m,w\}\(\\pi\)=\\mathbb\{E\}\\\!\\left\[K\_\{w\}\(Z^\{\\pi\}\)\\right\]\.\(12\)
Conceptually, this cost can be decomposed into agent and human components, where both terms may depend onπ\\pi, as intervention may increase human effort while changing the subsequent agent execution\.

Cm,w​\(π\)=Cagent​\(m,w,π\)\+Chuman​\(m,w,π\),C\_\{m,w\}\(\\pi\)=C\_\{\\mathrm\{agent\}\}\(m,w,\\pi\)\+C\_\{\\mathrm\{human\}\}\(m,w,\\pi\),\(13\)
Equations[11](https://arxiv.org/html/2609.02095#S3.E11)and[12](https://arxiv.org/html/2609.02095#S3.E12)define the two canonical quantities in the deployment objective of Equation[9](https://arxiv.org/html/2609.02095#S2.E9)\. They are properties of the deployed human–AI configuration under a specified policy, not intrinsic properties of the agent alone\.

### 3\.2Terminal Accept\-or\-Escalate Oversight

We first consider the trajectory\-invariant terminal accept\-or\-escalate setting used in our clinical\-audit case study\. In this setting, the agent completes its work before an oversight decision is made\. The completed execution provides a result together with an observable routing signalsis\_\{i\}; the policy then determines whether the result is accepted autonomously or routed to a declared human\-review path\.

To evaluate such policies, READY first executes the agent on representative workflow instances under autonomous execution\. For each task instancexix\_\{i\}, this produces a saved trajectoryZiZ\_\{i\}, an observable routing signalsis\_\{i\}, and an autonomous\-success indicatorui∈\{0,1\}u\_\{i\}\\in\\\{0,1\\\}\.

Here,\(Zi,si,ui\)\(Z\_\{i\},s\_\{i\},u\_\{i\}\)are evaluation data used for policy qualification; at deployment time, the routing decision is made from observable information such assis\_\{i\}, whileuiu\_\{i\}is generally unknown\.

Because the terminal routing decision occurs only afterZiZ\_\{i\}has been generated, the trajectory is invariant to the candidate routing policy\. READY can evaluate many accept\-or\-escalate policies by replaying the same saved evaluation runs instead of re\-executing the agent for every candidate policy\.

For a scalar routing signalsis\_\{i\}, consider a threshold policyπτ\\pi\_\{\\tau\}that accepts the completed result autonomously whensi≥τs\_\{i\}\\geq\\tauand routes it to human review otherwise\. We assume that larger values ofsis\_\{i\}indicate greater likelihood of successful autonomous execution, so thatsis\_\{i\}provides a meaningful ordering of cases for selective routing\. The formal routing assumption is given in Appendix[A\.1](https://arxiv.org/html/2609.02095#A1.SS1)\. The autonomous coverage under thresholdτ\\tauis

c⁡\(τ\)=Pr⁡\(s≥τ\),c\(\\tau\)=\\Pr\(s\\geq\\tau\),\(14\)
and its corresponding human\-review rate is

e⁡\(τ\)=1−c⁡\(τ\)\.e\(\\tau\)=1\-c\(\\tau\)\.\(15\)
Coverage is useful in this terminal setting because it is simply the complement of review burden; it is not required as a universal READY deployment metric\. Among autonomously accepted cases, define

a⁡\(τ\)=Pr⁡\(u=1∣s≥τ\),a\(\\tau\)=\\Pr\(u=1\\mid s\\geq\\tau\),\(16\)
the success probability of work accepted without review\.

Letah​\(τ\)a\_\{h\}\(\\tau\)denote the probability that a case routed by policyπτ\\pi\_\{\\tau\}is successfully resolved by the declared human\-review path\. The reliability of the deployed human–AI system is then

R⁡\(τ\)=c⁡\(τ\)​a​\(τ\)\+\(1−c⁡\(τ\)\)​ah​\(τ\)\.R\(\\tau\)=c\(\\tau\)a\(\\tau\)\+\\bigl\(1\-c\(\\tau\)\\bigr\)a\_\{h\}\(\\tau\)\.\(17\)
The first term corresponds to work accepted autonomously; the second corresponds to work routed to human review\.

The review\-path performanceah​\(τ\)a\_\{h\}\(\\tau\)can be either measured from the review process or introduced as a deployment assumption\. It is important to note that because routing deliberately selects a non\-random subset of cases,ah​\(τ\)a\_\{h\}\(\\tau\)may vary with the policy: cases routed under a more selective policy may be systematically more difficult than cases routed under another policy\. When a constant review\-success assumptionaha\_\{h\}is available, Equation[17](https://arxiv.org/html/2609.02095#S3.E17)reduces to

R⁡\(τ\)=c⁡\(τ\)​a​\(τ\)\+\(1−c⁡\(τ\)\)​ah\.R\(\\tau\)=c\(\\tau\)a\(\\tau\)\+\\bigl\(1\-c\(\\tau\)\\bigr\)a\_\{h\}\.\(18\)
This formulation makes clear why autonomous benchmark performance does not determine deployment reliability\. Two agents with similar autonomous success rates can support substantially different human–AI operating points because the oversight policies available to them may differ in how effectively they identify cases requiring intervention\.

### 3\.3Cost at a Target Reliability

In terminal accept\-or\-escalate deployment, the agent has already executed before the routing decision is made\. Letkmk\_\{m\}denote its expected execution cost per case and letkhk\_\{h\}denote the incremental cost of the declared human\-review path\. Under a constant\-cost approximation,

C⁡\(τ\)=km\+kh​e​\(τ\)=km\+kh​\(1−c⁡\(τ\)\)\.C\(\\tau\)=k\_\{m\}\+k\_\{h\}\\,e\(\\tau\)=k\_\{m\}\+k\_\{h\}\\bigl\(1\-c\(\\tau\)\\bigr\)\.\(19\)
For target reliabilityYY, the population\-level policy optimization problem is

τ⋆​\(Y\)=arg⁡minτ⁡C⁡\(τ\)subject toR⁡\(τ\)≥Y,\\tau^\{\\star\}\(Y\)=\\arg\\min\_\{\\tau\}C\(\\tau\)\\qquad\\text\{subject to\}\\qquad R\(\\tau\)\\geq Y,\(20\)
together with any additional workflow\-specific risk constraint\.

Equation[20](https://arxiv.org/html/2609.02095#S3.E20)defines the ideal population operating point\. In practice,R⁡\(τ\)R\(\\tau\)andC⁡\(τ\)C\(\\tau\)are unknown and policy selection is performed using development evidence\. LetR^dev​\(τ\)\\widehat\{R\}\_\{\\mathrm\{dev\}\}\(\\tau\)andC^dev​\(τ\)\\widehat\{C\}\_\{\\mathrm\{dev\}\}\(\\tau\)denote the corresponding development\-set estimates\. A generic development\-stage selection rule takes the form

τ^∈arg⁡minτ​C^dev​\(τ\)subject toR^dev​\(τ\)≥Ysel,\\widehat\{\\tau\}\\in\\arg\\min\_\{\\tau\}\\widehat\{C\}\_\{\\mathrm\{dev\}\}\(\\tau\)\\qquad\\text\{subject to\}\\qquad\\widehat\{R\}\_\{\\mathrm\{dev\}\}\(\\tau\)\\geq Y\_\{\\mathrm\{sel\}\},\(21\)
whereYselY\_\{\\mathrm\{sel\}\}is a pre\-specified development\-stage selection criterion\. It may equal the deployment targetYYor incorporate a pre\-specified margin or other conservative selection rule\.

READY does not require a particular optimization algorithm or development\-stage selection rule\. What is required for qualification is that all policy choices be made without using the held\-out qualification cases\. The selected policyπ^\\widehat\{\\pi\}is subsequently frozen and evaluated against the actual deployment targetYYas described in Section[3\.5](https://arxiv.org/html/2609.02095#S3.SS5)\.

Whenkmk\_\{m\}andkh\>0k\_\{h\}\>0are constant, minimizing operating cost is equivalent to minimizing human\-review burden, or equivalently maximizing autonomous coverage, among feasible policies\. Writing

e⋆​\(Y\)=e⁡\(τ⋆​\(Y\)\),e^\{\\star\}\(Y\)=e\\\!\\left\(\\tau^\{\\star\}\(Y\)\\right\),\(22\)
the population cost of reliable deployment is

C⋆​\(Y\)=km\+kh​e⋆​\(Y\)\.C^\{\\star\}\(Y\)=k\_\{m\}\+k\_\{h\}e^\{\\star\}\(Y\)\.\(23\)
This provides an economically meaningful basis for comparing systems: how much operating cost is required for each system to satisfy the same workflow\-specific reliability requirement\.

The constant\-cost model is useful for exposition but is not required by READY\. Review effort may vary substantially across cases, and different policies may trigger different forms of review, correction, or escalation\. When case\-level costs are available, READY estimatesC⁡\(π\)C\(\\pi\)directly from the costs induced by the candidate policy instead of assigning a single constantkhk\_\{h\}to every reviewed case\.

### 3\.4Quality of the Routing Signal

For terminal accept\-or\-escalate policies, the value of oversight depends in part on whether the routing signal identifies cases for which autonomous execution is likely to fail\. A useful signal should rank higher\-risk cases below lower\-risk cases, allowing a more selective policy to remove failures from autonomous handling preferentially\.

A natural diagnostic is the selective\-risk curve,

r⁡\(τ\)=Pr⁡\(u=0∣s≥τ\),r\(\\tau\)=\\Pr\(u=0\\mid s\\geq\\tau\),\(24\)or equivalentlyr⁡\(c\)r\(c\)when operating points are indexed by autonomous coverage\. Sweeping the routing threshold traces the risk–coverage relationship: a stronger routing signal achieves lower selective risk at the same autonomous coverage\.

We summarize this curve using the area under the risk–coverage curve \(AURC\),

AURC=∫01r⁡\(c\)​𝑑c,\\mathrm\{AURC\}=\\int\_\{0\}^\{1\}r\(c\)\\,dc,\(25\)with lower values indicating a more favorable risk–coverage tradeoff\.

AURC is influenced both by routing quality and by the base autonomous success ratea0=Pr⁡\(u=1\)a\_\{0\}=\\Pr\(u=1\)\. We also report theAUROCof the routing signalssfor predicting execution successuu\. AUROC measures ranking discrimination and, unlike AURC, is not directly determined by the prevalence of successful executions\.

These metrics are diagnostics for the terminal routing policy class rather than universal READY qualification quantities\. Their practical importance is ultimately determined by the operating policies they support: at a fixed reliability target, better routing can reduce the amount of human intervention, and the operating cost, required for qualification\.

### 3\.5Statistical Qualification

Optimizing an oversight policy on development evidence does not establish that the selected policy will satisfy its deployment requirements on future cases\. READY separates*policy selection*from*statistical policy qualification*\.

The evaluation sampleSwS\_\{w\}is partitioned by disjoint index setsIdev,Iqual⊆\[Nw\]I\_\{\\mathrm\{dev\}\},I\_\{\\mathrm\{qual\}\}\\subseteq\[N\_\{w\}\]into a development setSdev=\{xi:i∈Idev\}S\_\{\\mathrm\{dev\}\}=\\\{x\_\{i\}:i\\in I\_\{\\mathrm\{dev\}\}\\\}and a held\-out qualification setSqual=\{xi:i∈Iqual\}S\_\{\\mathrm\{qual\}\}=\\\{x\_\{i\}:i\\in I\_\{\\mathrm\{qual\}\}\\\}\. All choices that determine the policy—including routing rule, threshold, optimization procedure, and any development\-stage margin—are made usingSdevS\_\{\\mathrm\{dev\}\}\. The selected policyπ^\\widehat\{\\pi\}is then frozen before the qualification outcomes are examined\.

For terminal accept\-or\-escalate oversight, letτ^\\widehat\{\\tau\}denote the threshold selected on the development set\. For each qualification caseii, define

di=𝟏\{si≥τ^\},d\_\{i\}=\\mathbf\{1\}\\\{s\_\{i\}\\geq\\widehat\{\\tau\}\\\},\(26\)
wheredi=1d\_\{i\}=1denotes autonomous acceptance anddi=0d\_\{i\}=0denotes human review\. The qualification set is used only to estimate the operating characteristics of this fixed policy, not to search for a better threshold\.

#### Measured human\-review outcomes\.

When the outcome of the declared human\-review path is observed for each escalated case, lethi∈\{0,1\}h\_\{i\}\\in\\\{0,1\\\}denote whether that review path produces a successful result\. The realized success of the deployed system on qualification caseiiis

vi=di​ui\+\(1−di\)​hi,v\_\{i\}=d\_\{i\}u\_\{i\}\+\(1\-d\_\{i\}\)h\_\{i\},\(27\)
and the empirical deployment reliability is

R^​\(π^\)=1Nqual​∑i∈Iqualvi,Nqual=\|Iqual\|\.\\widehat\{R\}\(\\widehat\{\\pi\}\)=\\frac\{1\}\{N\_\{\\mathrm\{qual\}\}\}\\sum\_\{i\\in I\_\{\\mathrm\{qual\}\}\}v\_\{i\},\\qquad N\_\{\\mathrm\{qual\}\}=\|I\_\{\\mathrm\{qual\}\}\|\.\(28\)
In the terminal setting,viv\_\{i\}is the realized policy\-level success outcome corresponding to the general success eventuiπu\_\{i\}^\{\\pi\}in Section[3\.1](https://arxiv.org/html/2609.02095#S3.SS1);uiu\_\{i\}here denotes the success of the autonomous execution before routing\.

When every policy\-level outcome is observed and qualification cases are independent,viv\_\{i\}is binary and a one\-sided binomial lower confidence bound can be computed directly\. At confidence level1−α1\-\\alpha, READY qualifies the frozen policy only if

RLCB1−α​\(π^\)≥Y\.R\_\{\\mathrm\{LCB\}\}^\{\\,1\-\\alpha\}\(\\widehat\{\\pi\}\)\\geq Y\.\(29\)
A point estimate above the target is not sufficient: the held\-out evidence must support a reliability lower bound that itself clears the deployment requirement\.

#### Human\-review performance as a deployment assumption\.

In retrospective benchmark analysis, the declared human\-review path may not be executed for every escalated case\. Its effectiveness can instead be introduced explicitly as a deployment assumption\. Suppose, for example, that every routed case is assumed to be successfully resolved with probabilityaha\_\{h\}\. For each qualification case define the policy\-level conditional success value

yi​\(ah\)=di​ui\+\(1−di\)​ah∈\[0,1\]\.y\_\{i\}\(a\_\{h\}\)=d\_\{i\}u\_\{i\}\+\(1\-d\_\{i\}\)a\_\{h\}\\in\[0,1\]\.\(30\)
Under the constant\-review assumption, the population reliability of the fixed policy is

R⁡\(π^,ah\)=𝔼⁡\[yi​\(ah\)\],R\(\\widehat\{\\pi\};a\_\{h\}\)=\\mathbb\{E\}\\left\[y\_\{i\}\(a\_\{h\}\)\\right\],\(31\)
This quantity is no longer a simple binomial proportion, because reviewed cases contribute an assumed success probability rather than an observed binary outcome\. Writingnaccn\_\{\\mathrm\{acc\}\}for the number of accepted qualification cases andp^acc\\widehat\{p\}\_\{\\mathrm\{acc\}\}for the observed success rate among them, the empirical estimate can be written as

R^​\(π^,ah\)=naccNqual​p^acc\+Nqual−naccNqual​ah,\\widehat\{R\}\(\\widehat\{\\pi\};a\_\{h\}\)=\\frac\{n\_\{\\mathrm\{acc\}\}\}\{N\_\{\\mathrm\{qual\}\}\}\\,\\widehat\{p\}\_\{\\mathrm\{acc\}\}\+\\frac\{N\_\{\\mathrm\{qual\}\}\-n\_\{\\mathrm\{acc\}\}\}\{N\_\{\\mathrm\{qual\}\}\}\\,a\_\{h\},\(32\)
in which onlyp^acc\\widehat\{p\}\_\{\\mathrm\{acc\}\}is estimated from observed outcomes\.

For statistical qualification, however, both accepted\-case success and the fraction of cases routed by the frozen policy vary across samples from the workflow population\. READY constructs a one\-sided lower confidence bound directly for the mean of the bounded case\-level quantities\{yi​\(ah\)\}i∈Iqual\\\{y\_\{i\}\(a\_\{h\}\)\\\}\_\{i\\in I\_\{\\mathrm\{qual\}\}\}rather than treating the observed routing fraction as fixed\. Denote this bound by

RLCB1−α​\(π^,ah\)\.R\_\{\\mathrm\{LCB\}\}^\{\\,1\-\\alpha\}\(\\widehat\{\\pi\};a\_\{h\}\)\.\(33\)
The policy qualifies conditionally on the declared human\-review assumption only if

RLCB1−α​\(π^,ah\)≥Y\.R\_\{\\mathrm\{LCB\}\}^\{\\,1\-\\alpha\}\(\\widehat\{\\pi\};a\_\{h\}\)\\geq Y\.\(34\)
The lower bound may be computed using a pre\-specified valid procedure for the mean of bounded case\-level outcomes\. When cases are clustered, the qualification procedure operates at the corresponding cluster level as described below\.

No statistical confidence is attributed to the assumed term\. The qualification statement is explicitly conditional, of the form “qualified at targetYYunder the declared review assumptionaha\_\{h\}\.” For policy families whose contributions do not admit this decomposition, the same conditional bound is obtained by resampling qualification cases and re\-evaluating the fixed policy on each resample\. Ifaha\_\{h\}is itself estimated from separate human\-review data, uncertainty in that estimate must also be propagated rather than treatingaha\_\{h\}as known exactly; a conservative qualification replacesaha\_\{h\}with a one\-sided lower confidence boundah,LCBa\_\{h,\\mathrm\{LCB\}\}in Equation[34](https://arxiv.org/html/2609.02095#S3.E34)\.

#### Break\-even review assumption\.

Because qualification reliabilityRLCB1−α​\(π^,ah\)R\_\{\\mathrm\{LCB\}\}^\{\\,1\-\\alpha\}\(\\widehat\{\\pi\};a\_\{h\}\)is nondecreasing inaha\_\{h\}, the dependence on the human\-review assumption can be summarized by a single quantity: the weakest review effectiveness under which the frozen policy still qualifies:

ahmin​\(π^,Y\)=inf\{ah∈\[0,1\]:RLCB1−α​\(π^,ah\)≥Y\}\.a\_\{h\}^\{\\min\}\(\\widehat\{\\pi\},Y\)=\\inf\\left\\\{a\_\{h\}\\in\[0,1\]:R\_\{\\mathrm\{LCB\}\}^\{\\,1\-\\alpha\}\(\\widehat\{\\pi\};a\_\{h\}\)\\geq Y\\right\\\}\.\(35\)
The deployment profile can then state an auditable requirement on the human process: the selected policy qualifies at targetYYprovided the declared review path achieves effectiveness at leastahmina\_\{h\}^\{\\min\}\.

If noah∈\[0,1\]a\_\{h\}\\in\[0,1\]satisfies the criterion, the policy is reported as not qualified at targetYYunder any constant review\-success assumption\.

#### Sampling unit\.

The statistical sampling unit must match the unit over which the deployment claim is intended to generalize\. Multiple stochastic executions of the same task instance are not treated as independent new workflow cases\. When repeated runs are present, they are carried together at the task\-instance level\.

Similarly, when cases are naturally clustered by patient, customer, account, site, or another higher\-level unit, qualification must preserve that dependence structure, for example by constructing confidence intervals or resampling at the cluster level\.

A policy that fails the statistical criterion is reported as*not qualified at targetYY*or as having*insufficient evidence*; it is not retuned using the qualification set\. Any change to the policyπ^\\widehat\{\\pi\}after examining qualification outcomes requires a new held\-out qualification evaluation\.

Unless otherwise stated, READY qualification claims are*pointwise*for a pre\-specified deployment target and frozen policy\. Simultaneous statistical claims across multiple targets, policies, or operating points require an appropriate simultaneous\-inference procedure\.

### 3\.6Risk\-Sensitive Qualification

Reliability treats every unsuccessful execution as the same binary event\. For workflows in which failures differ materially in consequence, READY may additionally define a workflow\-specific deployment loss

Lw​\(Zπ\)≥0L\_\{w\}\(Z^\{\\pi\}\)\\geq 0\(36\)
and impose the constraint

ρw​\(Lw​\(Zπ\)\)≤Bw,\\rho\_\{w\}\\\!\\left\(L\_\{w\}\(Z^\{\\pi\}\)\\right\)\\leq B\_\{w\},\(37\)
whereBwB\_\{w\}is the declared risk tolerance andρw\\rho\_\{w\}is a workflow\-specific risk functional\. Examples include a bound on the probability of a specified severe event or a tail\-risk measure such as Conditional Value at Risk \(CVaR\)\.

This constraint is optional\. The common READY objective remains reliable deployment at minimum operating cost; an additional risk constraint is introduced when binary success alone does not adequately represent the consequences of failure\. Estimation and statistical qualification of risk\-sensitive constraints are discussed further in Appendix[A\.2](https://arxiv.org/html/2609.02095#A1.SS2)\.

### 3\.7Trajectory\-Dependent Oversight

The terminal formulation above is especially convenient because the routing decision does not change the saved agent trajectory\. More general enterprise workflows allow intervention during execution\.

Lethth\_\{t\}denote the observable workflow history at decision pointtt, and let

π⁡\(ot∣ht\)\\pi\(o\_\{t\}\\mid h\_\{t\}\)\(38\)
denote an oversight policy over intervention decisionsoto\_\{t\}, such as approval, correction or takeover\. Executing the workflow underπ\\piinduces a policy\-dependent trajectory

Ziπ∼P\(⋅∣m,xi,Ew,π\)\.Z\_\{i\}^\{\\pi\}\\sim P\(\\,\\cdot\\mid m,x\_\{i\},E\_\{w\},\\pi\\,\)\.\(39\)
The qualification objective remains the one defined in Equation[9](https://arxiv.org/html/2609.02095#S2.E9)\. The difference is how the evidence needed to estimate that objective is generated\.

For trajectory\-invariant terminal policies, candidate policies can be evaluated by replaying saved trajectories under different routing rules\. For a trajectory\-dependent policy, changingπ\\pichanges what the agent, oversight actor, or interacting user subsequently observes and does\. Reliability, cost, and risk must instead be estimated by executing or appropriately simulating the candidate policy in the evaluation runtime\.

The phases introduced in Section[2\.2](https://arxiv.org/html/2609.02095#S2.SS2)should therefore be understood as a logical evaluation flow rather than necessarily a one\-pass computation\. For trajectory\-dependent oversight, policy optimization may repeatedly invoke agent execution and workflow evaluation while searching over candidate policies\.

Policy\-in\-the\-loop evaluation may change both sides of the deployment tradeoff\. Intervention can improve the probability of successful completion, but can also change agent execution length, human effort, recovery behavior, and the distribution of subsequent outcomes\. These effects are captured by the same general quantitiesRm,w​\(π\)R\_\{m,w\}\(\\pi\),Cm,w​\(π\)C\_\{m,w\}\(\\pi\), and, when applicable,ρw​\(Lw​\(Zπ\)\)\\rho\_\{w\}\(L\_\{w\}\(Z^\{\\pi\}\)\)\.

This setting is naturally connected to simulation\-based and black\-box optimization, where objective and constraint values need not admit closed\-form expressions and can instead be estimated through stochastic simulation or system execution\[[11](https://arxiv.org/html/2609.02095#bib.bib132),[12](https://arxiv.org/html/2609.02095#bib.bib133)\]\. READY does not prescribe an optimization algorithm\. The policy representation and optimization method may be workflow\-dependent\. What is shared across workflows is the deployment objective and qualification procedure: candidate policies are evaluated in terms of the reliability, cost, and risk of the resulting human–AI system, a policy is selected using development evidence, and the frozen policy is qualified on held\-out evidence\.

#### Empirical instantiation\.

Section[5](https://arxiv.org/html/2609.02095#S5)applies the qualification methodology above to CliniCARE\-Bench\. The analysis is conducted on the benchmark’s full four\-way adjudication space using a signal\-based accept\-or\-escalate policy that is agnostic to the class labels, with sensitivity analyses in Appendix[B](https://arxiv.org/html/2609.02095#A2)and a cost\-of\-conflation control in Appendix[B\.5](https://arxiv.org/html/2609.02095#A2.SS5)\.

## 4READY Evaluation Runtime and Open Testbed

Sections[2](https://arxiv.org/html/2609.02095#S2)and[3](https://arxiv.org/html/2609.02095#S3)define what READY qualifies and how an oversight policy is selected\. This section describes the execution and interface testbed that produces the evidence required by that methodology\.

READY does not introduce a new general\-purpose agent\-evaluation harness\. Its reference implementation uses Inspect AI\[[9](https://arxiv.org/html/2609.02095#bib.bib24)\]for task execution, agent integration, sandboxed environments, scoring, intervention, and evaluation logging\. READY adds the workflow\-level semantics and deployment\-qualification layer needed to translate executions into a reliability–oversight–cost deployment profile\.

### 4\.1Runtime Layering and Reference Execution Substrate

The implementation follows the three\-layer architecture in Figure[1](https://arxiv.org/html/2609.02095#S2.F1)\. Non\-executing*specification artifacts*define the workflow, evaluation semantics, and deployment assumptions\. The*evaluation runtime*instantiates these specifications and executes the agent, workflow environment, and any required oversight or user actors\. The resulting execution evidence is evaluated and passed to the qualification methodology in Section[3](https://arxiv.org/html/2609.02095#S3)\.

READY standardizes the semantics and evidence that cross these boundaries without requiring every evaluation to share a task representation, agent implementation, environment technology, or grading mechanism, so existing evaluation infrastructure can be reused\.

In the Inspect reference implementation, an executable evaluation is represented by an InspectTask, which combines a collection ofSamples with an agent orSolver, one or moreScorers, and the required runtime configuration\. READY introduces the workflow and deployment semantics above these execution objects\. The correspondence is an implementation mapping, not a one\-to\-one semantic equivalence\.

### 4\.2Specification Interfaces

The runtime consumes three logically distinct classes of specification artifacts corresponding to Figure[1](https://arxiv.org/html/2609.02095#S2.F1)\.

#### Workflow specification\.

The workflow specification identifies the reusable workflow,ww, the task\-instance population,𝒟w\\mathcal\{D\}\_\{w\}, from which the evaluation sample,SwS\_\{w\}, is drawn, and the environment specification,EwE\_\{w\}\. It defines the work to be performed and the population for which the evaluation provides evidence\. A concrete task instance,xix\_\{i\}, supplies the case\-specific inputs for one execution\.

READY does not require a second executable task format\. In the Inspect reference implementation, a READY task instance is represented by an InspectSample, and a collection of task instances is represented by an InspectDataset\. The enclosing InspectTaskspecifies how those samples are executed and evaluated\. This distinction is important: an READY*task instance*corresponds to an InspectSample, not to an InspectTask\.

#### Evaluation specification\.

The evaluation specification defines the workflow evaluator,gwg\_\{w\}, the workflow\-success predicate,ϕw\\phi\_\{w\}, and any routing signal exposed to an oversight policy\.

It specifies the transformationZiπ→gw𝐪iπ→ϕwuiπZ\_\{i\}^\{\\pi\}\\xrightarrow\{g\_\{w\}\}\\mathbf\{q\}\_\{i\}^\{\\pi\}\\xrightarrow\{\\phi\_\{w\}\}u\_\{i\}^\{\\pi\}of Equation[5](https://arxiv.org/html/2609.02095#S2.E5)\.

The mechanisms used to compute𝐪iπ\\mathbf\{q\}\_\{i\}^\{\\pi\}may be deterministic, human\-annotated, model\-assisted, or composed from multiple evaluation mechanisms\. In Inspect, these computations can be implemented through one or moreScorers, including externally computed scores when appropriate\. READY’s evaluator,gwg\_\{w\}, denotes the workflow\-level semantics of what is measured; an InspectScoreris one executable mechanism for producing those measurements\. The success predicate,ϕw\\phi\_\{w\}, remains an READY\-level definition that determines which measurements constitute acceptable execution for deployment qualification\.

#### Deployment assumptions\.

Deployment assumptions remain separate from measured evaluation evidence\. They include the reliability target,YY, the human\-oversight model, operating\-cost assumptions, and any workflow\-specific risk tolerance\. Different organizations may therefore interpret the same measured execution evidence under different deployment requirements\.

The development and held\-out qualification partitions required by Section[3\.5](https://arxiv.org/html/2609.02095#S3.SS5)are versioned with the READY evaluation configuration\. Instances used to select an oversight policy must remain separate from the instances used to qualify the frozen policy\.

Table[1](https://arxiv.org/html/2609.02095#S4.T1)summarizes the minimal logical interface required of an READY\-compatible runtime and its reference realization in Inspect\.

Table 1:Minimal logical interface for an READY\-compatible evaluation runtime\.READY defines the semantics required for deployment qualification; the rightmost column shows their reference realization in Inspect AI\. Other execution substrates may implement the same interface using different native abstractions\.

### 4\.3Execution, Actors, and Policy Evaluation

The evaluation runtime instantiates the workflow environment and executes agentmmon task instancexix\_\{i\}under oversight policyπ\\pi\. The runtime is responsible for making the workflow\-specified tools, data, systems, and interaction interfaces available while preserving the declared evaluation conditions\.

Inspect provides the reference execution mechanisms for this layer\. Agents may be implemented through native Inspect agents or solvers, or integrated from external agent frameworks through the Inspect Agent Bridge\. Workflow\-specific files and computational environments can be attached to individual samples and executed in controlled sandboxes\. These implementation choices do not change the READY semantics of the workflow or trajectory\.

For workflows requiring interaction, READY distinguishes two additional actor roles\. An*oversight actor*represents an organization\-side reviewer or operator who may approve, clarify, correct, reject, escalate, or take over work\. A*user actor*represents an external user or counterparty interacting with the deployed system\. Either actor may be human or simulated, but its identity, configuration, and available actions must be part of the evaluation configuration\.

Inspect approval policies and agent\-intervention mechanisms provide reference implementations for several forms of oversight\. For example, approval policies can require selected tool calls to be approved, rejected, or escalated, while agent intervention permits a human operator to interrupt or redirect an executing agent\. These mechanisms do not define the READY policy classΠw\\Pi\_\{w\}; they are runtime primitives through which particular policies may be implemented\. Other forms of interaction may be realized through custom agents, solvers, or workflow\-specific simulators\.

The runtime supports both policy\-evaluation procedures introduced in Section[2\.2](https://arxiv.org/html/2609.02095#S2.SS2)\.

For*trajectory\-invariant*policies, the relevant agent execution is independent of the terminal routing decision\. The runtime can therefore execute the agent once, preserve the resulting evidence, and evaluate multiple candidate policies by replaying the same completed executions\. The accept\-or\-escalate analysis in Section[3\.2](https://arxiv.org/html/2609.02095#S3.SS2)uses this path\.

For*trajectory\-dependent*policies, intervention changes what happens next\. The candidate policy must participate in execution, and the runtime records the resulting policy\-dependent trajectoryZiπZ\_\{i\}^\{\\pi\}\. Reliability, cost, and risk for that policy must be estimated from executions or simulations performed under the policy itself\.

### 4\.4Evaluation Evidence and Workflow Evaluation

Each execution produces a sample\-level evaluation record containing the observable evidence required by the workflow evaluator and qualification analysis\. READY does not introduce a separate universal trajectory format\. Instead,ZiπZ\_\{i\}^\{\\pi\}denotes the workflow\-relevant observable execution trajectory represented by the underlying runtime record\.

In the Inspect reference implementation, evaluation logs preserve sample\-level outputs, model and tool interactions, scoring information, metadata, usage information, and the execution transcript\. For bridged agents, model interactions are likewise captured in the Inspect transcript\. When human or automated interventions occur during execution, those interventions should remain part of the retained evidence\.

The workflow evaluator appliesgwg\_\{w\}to this record to produce𝐪iπ\\mathbf\{q\}\_\{i\}^\{\\pi\}, and the pre\-specified predicateϕw\\phi\_\{w\}derives the corresponding success eventuiπu\_\{i\}^\{\\pi\}\. Diagnostic measurements remain available independently of this binary success event\. For example, a workflow may retain outcome correctness, evidence grounding, process adherence, appropriate abstention, and resource use even when only a subset of those measurements entersϕw\\phi\_\{w\}\.

Inspect’s scoring interface supports multiple workflow measurements without requiring them to be collapsed into a single benchmark score\. Where the saved execution record contains sufficient evidence, an updated evaluator can also be applied to an existing evaluation log without re\-executing the agent\.

For workflows requiring repeated stochastic execution, the same logical task instance may be executed for multiple epochs\. READY retains the underlying task\-instance identity and epoch\-level evidence so that repeated runs are not treated as independent task instances during statistical qualification\.

#### Running example: CliniCARE\-Bench\.

For CliniCARE\-Bench, each case is represented as an evaluation sample whose execution record contains the completed clinical adjudication, supporting patient\- and policy\-evidence citations, the case\-level routing signal, and the observable investigation trajectory, including retrieved evidence, tool calls, computations, and intermediate artifacts\. The CliniCARE evaluator derives verdict and process measurements from this record and applies the workflow\-specific success predicate before the resulting evidence is passed to READY qualification\.

### 4\.5Qualification Handoff and Deployment Profile

The boundary between workflow evaluation and deployment qualification is an evidence interface rather than a new task evaluator\. The qualification layer consumes measured execution evidence together with declared deployment assumptions and applies the methodology in Section[3](https://arxiv.org/html/2609.02095#S3)\.

For the trajectory\-invariant accept\-or\-escalate case, the minimal qualification record for task instanceiicontains the workflow\-defined success eventuiu\_\{i\}, the observable routing signalsis\_\{i\}, and the execution or resource measurements required by the cost model\. It also retains the identifiers needed to associate the record with its task instance, statistical sampling unit, evaluated system, and versioned evaluation configuration\. These quantities can be extracted from Inspect scores, metadata, usage measurements, and evaluation logs into the READY qualification dataset\.

#### Versioned qualification record\.

The qualification record has a stable, versioned schema so that a deployment claim can be reproduced and audited\. Each record carries aschema\_version; alineageblock that version\-stamps the workflow, task population, evaluated system, environment, evaluatorgwg\_\{w\}, success predicateϕw\\phi\_\{w\}, and routing\-signal definition; the per\-instancemeasurements\(uiu\_\{i\},sis\_\{i\}, retained diagnostics𝐪i\\mathbf\{q\}\_\{i\}, and cost\); and the identifiers of the task instance and statistical sampling unit\. Deployment assumptions are recorded separately with the profile, not in the per\-instance record, so the same records support multiple deployment analyses\.

The development and held\-out partitions carry an identical schema; only thepartitionfield differs\. Deployment assumptions \(YY,aha\_\{h\}, review cost\) are attached to the resulting profile, keeping measured evidence separate from the assumptions under which it is interpreted\.

Section[3\.5](https://arxiv.org/html/2609.02095#S3.SS5)then selects the oversight policy on development data and qualifies the frozen operating point on held\-out evidence\. The qualification computation is intentionally separate from Inspect’s task scoring: Inspect records what happened and how the workflow evaluator scored it, while READY determines which deployment policy is supported by that evidence\.

For trajectory\-dependent policies, the same logical handoff applies, but the measurements correspond to trajectories generated under the candidate policyπ\\pi, and the retained evidence includes the policy and actor configuration needed to interpretZiπZ\_\{i\}^\{\\pi\},uiπu\_\{i\}^\{\\pi\}, and its associated operating cost\.

The output is the*deployment profile*defined in Section[2\.2](https://arxiv.org/html/2609.02095#S2.SS2): a statistically supported operating point conditional on the workflow, evaluated system, oversight policy, and declared deployment assumptions, rather than a context\-free model score\.

### 4\.6Contributing and Maintaining Evaluations

New enterprise workflows can be added to READY without changing the qualification methodology or the reference execution infrastructure\. A contribution supplies an executable workflow evaluation together with the specification artifacts of Section[4\.2](https://arxiv.org/html/2609.02095#S4.SS2): the workflow identity and represented population, the evaluation semanticsgwg\_\{w\}andϕw\\phi\_\{w\}, any routing signal, the development and held\-out partitions, and the measurements needed for cost analysis\. The same interface serves public benchmarks and enterprise\-internal evaluations; READY does not require proprietary data or protected references to be made public, only that the workflow, evaluation, and reported deployment claim remain explicit\.

#### Validation\.

Interface conformance establishes interoperability; validation checks whether the resulting evidence supports a meaningful deployment claim\. A reviewer independent of the contributor checks four aspects against contributor\-supplied evidence:*workflow validity*\(a recognizable form of enterprise work sampled from the stated population\),*evaluation validity*\(gwg\_\{w\}andϕw\\phi\_\{w\}reproduce success labels within a pre\-registered tolerance from observable evidence\),*execution validity*\(reproducible dry runs with no hidden\-reference access or unintended shortcuts\), and*qualification validity*\(a frozen development/held\-out partition with policy selection never touching held\-out data\)\. Verdicts are recorded with the evaluation lineage so that acceptance is reproducible rather than a matter of reviewer opinion\.

#### Versioned evaluation lineage\.

A READY result is tied to a versioned*evaluation lineage*rather than just a benchmark or model name\. The lineage pins the workflow and task population, agent and model configuration, execution environment, evaluatorgwg\_\{w\}and predicateϕw\\phi\_\{w\}, routing signal, and the evidence used for qualification, through concrete build identifiers such as commit hashes, dependency and image digests, and the seeds governing sampling and any split\. A qualification claim then has the form

evaluation lineage\+deployment assumptions⟶deployment profile,\\text\{evaluation lineage\}\\;\+\\;\\text\{deployment assumptions\}\\;\\longrightarrow\\;\\text\{deployment profile\},separating the evidence that was measured from the assumptions under which it is interpreted\. Corrections and later releases create new evaluation records rather than silently reinterpreting earlier profiles\.

#### Maintenance and requalification\.

Because lineage components are versioned separately, different changes have different consequences:

- •Assumptions change→\\rightarrow*recompute*\. When only deployment assumptions change \(target reliability, review model, or cost\), the agent evidence is unchanged and READY recomputes the oversight policy and profile from existing records\.
- •Evaluation changes→\\rightarrow*re\-evaluate*when evidence is sufficient\. Whengwg\_\{w\}orϕw\\phi\_\{w\}changes, preserved trajectories may be re\-scored, but only when each trajectory records every input the revised evaluator reads; otherwise the case is re\-executed\.
- •Trajectory or population changes→\\rightarrow*re\-execute*\. Material changes to the agent, harness, environment, tools, or represented population can alter the trajectory or the claim’s scope, so the agent is rerun and passed through the same qualification procedure as a new evaluation\.

READY thus treats deployment qualification not as a permanent property of an agent but as an evidence\-backed claim attached to a specific lineage and deployment context\.

## 5Empirical Evaluation: Deployment Qualification on CliniCARE\-Bench

We use CliniCARE\-Bench to illustrate the deployment\-qualification methodology of Section[3](https://arxiv.org/html/2609.02095#S3)\. The goal is not to reproduce the full CliniCARE benchmark, but to measure the quantities that matter for READY: whether a routing signal supports selective autonomy, how much human oversight a given reliability target demands, and whether systems with similar autonomous performance differ in deployment value\. We organize the section around three questions\.

1. 1\.Does the routing signal separate more reliable from less reliable autonomous executions? \(Section[5\.3](https://arxiv.org/html/2609.02095#S5.SS3)\)
2. 2\.How does the required human\-review burden grow as the reliability target increases? \(Section[5\.4](https://arxiv.org/html/2609.02095#S5.SS4)\)
3. 3\.Can systems with similar autonomous performance support materially different qualified operating points? \(Sections[5\.5](https://arxiv.org/html/2609.02095#S5.SS5)–[5\.6](https://arxiv.org/html/2609.02095#S5.SS6)\)

### 5\.1Experimental Setup

#### Adapting CliniCARE\-Bench to READY\.

Adapting this existing dataset required no change to its clinical content, only expressing it through the interface of Section[4](https://arxiv.org/html/2609.02095#S4): each adjudication case becomes a task instancexix\_\{i\}\(an InspectSample\); the patient\-scoped EHR retrieval tools and sandbox become the environmentEwE\_\{w\}; the four\-way verdict grader becomes the evaluatorgwg\_\{w\}with success predicateϕw\\phi\_\{w\}; the report’s stated confidence becomes the routing signalsis\_\{i\}; and the fixed case split becomes the development and qualification partitions\. Deployment assumptions \(YY,aha\_\{h\}, cost\) stay separate, so the same evidence supports the sensitivity analyses of Appendix[B](https://arxiv.org/html/2609.02095#A2)\. The four reference verdicts are

𝒴=\{Yes,No,Indeterminate:LackofData,Indeterminate:MedicallyAmbiguous\}\.\\mathcal\{Y\}=\\\{\\mathrm\{Yes\},\\mathrm\{No\},\\mathrm\{Indeterminate\{:\}\\ Lack\\ of\\ Data\},\\mathrm\{Indeterminate\{:\}\\ Medically\\ Ambiguous\}\\\}\.\(40\)
As established in Section[2\.3](https://arxiv.org/html/2609.02095#S2.SS3), the two indeterminate outcomes are task\-level clinical judgments—a correct indeterminate verdict is itself a successful execution—and are distinct from the READY decision to escalate a completed case for review\. Keeping them separate lets abstention be handled inside the common READY optimization without treating every indeterminate output as a failure or a forced escalation\.

#### Success indicators\.

Letyi⋆∈𝒴y\_\{i\}^\{\\star\}\\in\\mathcal\{Y\}denote the reference adjudication,y^i\\widehat\{y\}\_\{i\}the agent adjudication, andsis\_\{i\}the routing signal\. The primary four\-way success indicator is

ui=𝟏\{y^i=yi⋆\}\.u\_\{i\}=\\mathbf\{1\}\\\{\\widehat\{y\}\_\{i\}=y\_\{i\}^\{\\star\}\\\}\.\(41\)
and a process\-aware variant, used as a robustness check in Appendix[B](https://arxiv.org/html/2609.02095#A2), additionally requires the absence of a disqualifying process defect,

ui,df=𝟏\{y^i=yi⋆∧no disqualifying process defect\},u\_\{i,\\mathrm\{df\}\}=\\mathbf\{1\}\\left\\\{\\widehat\{y\}\_\{i\}=y\_\{i\}^\{\\star\}\\;\\land\\;\\text\{no disqualifying process defect\}\\right\\\},\(42\)

#### Systems, cases, and labels\.

We evaluate1616systems on all750750cases, giving12,00012\{,\}000system–case runs\. Reference verdicts follow the natural cohort distribution rather than a balanced one:Yes352352\(46\.9%46\.9\\%\),No248248\(33\.1%33\.1\\%\),Indeterminate: Lack of Data123123\(16\.4%16\.4\\%\), andIndeterminate: Medically Ambiguous2727\(3\.6%3\.6\\%\)\.

#### Routing signal\.

The signalsis\_\{i\}is the confidence each system states in its own report \(required as a percentage\), which we rescale to\[0,1\]\[0,1\]and observe across the full range\. It is present for11,99211\{,\}992of12,00012\{,\}000runs; the eight exceptions, seven stating no confidence and one producing no report, are assignedsi=0s\_\{i\}=0and escalated first\. Systems differ sharply in signal resolution, from55distinct values \(Gemini\-3\.1\-Pro\) to3636\(GPT\-5\.4\-mini\)\. This granularity is itself deployment\-relevant: a system with few distinct values has correspondingly few reachable operating points\.

#### Development and qualification split\.

Cases are partitioned once, uniformly at random with a fixed seed, into\|Sdev\|=375\|S\_\{\\mathrm\{dev\}\}\|=375and\|Squal\|=375\|S\_\{\\mathrm\{qual\}\}\|=375\. The split is over*cases*, not runs, so every system is developed and qualified on an identical cohort and all cross\-system comparisons are paired\. The draw preserves verdict proportions without stratification \(SdevS\_\{\\mathrm\{dev\}\}:172/126/64/13172/126/64/13;SqualS\_\{\\mathrm\{qual\}\}:180/122/59/14180/122/59/14\)\.

#### Human\-review assumption\.

Because CliniCARE does not execute the declared review path for every routed case, we treat human\-review successaha\_\{h\}as a deployment assumption, with primary value

and sensitivity to alternatives in Appendix[B](https://arxiv.org/html/2609.02095#A2)\. All qualification statements are therefore conditional on this assumption, and we also report the break\-even valueahmina\_\{h\}^\{\\min\}\(Equation[35](https://arxiv.org/html/2609.02095#S3.E35)\), the weakest review success under which a policy still qualifies\. The assumption also imposes a ceiling: because deployment reliability is a convex combination of autonomous and reviewed outcomes and selective success1−r⁡\(τ\)1\-r\(\\tau\)stays belowaha\_\{h\}at every coverage level we observe, no policy can exceed reliabilityah=0\.90a\_\{h\}=0\.90\. Targets above0\.900\.90are unreachable here not because of agent capability but because of the assumed quality of the review path, a structural point a capability\-only benchmark cannot express\.

### 5\.2Escalation Policy

We evaluate a single terminal accept\-or\-escalate policy that acts only on the routing signal, selected onSdevS\_\{\\mathrm\{dev\}\}and frozen before qualification\. Because the workflow is treated as a four\-way classification with success indicatoruiu\_\{i\}, the policy is deliberately agnostic to what the classes*mean*: class semantics enter only through the success predicateϕw\\phi\_\{w\}, never through the escalation rule\. The same threshold applies to all four predicted adjudications,

πτ​\(si\)=\{accept,si≥τ,review,si<τ\.\\pi\_\{\\tau\}\(s\_\{i\}\)=\\begin\{cases\}\\mathrm\{accept\},&s\_\{i\}\\geq\\tau,\\\\ \\mathrm\{review\},&s\_\{i\}<\\tau\.\\end\{cases\}\(44\)
Fittingτ\\tauto hitYYdirectly on a sampled development set is prone to selection bias and tends to fail the stricter held\-out lower\-bound test\. We instead optimize against a margin\-adjusted target,

τ⋆​\(Y\)=arg⁡minτ⁡C⁡\(τ\)subject toR^dev​\(τ,ah\)≥Y\+δ,\\tau^\{\\star\}\(Y\)=\\arg\\min\_\{\\tau\}C\(\\tau\)\\qquad\\text\{subject to\}\\qquad\\widehat\{R\}\_\{\\mathrm\{dev\}\}\(\\tau;a\_\{h\}\)\\geq Y\+\\delta,\(45\)
whereR^dev​\(τ,ah\)\\widehat\{R\}\_\{\\mathrm\{dev\}\}\(\\tau;a\_\{h\}\)is the development\-partition reliability estimate under the declared review assumption\. All CliniCARE policies use a fixed marginδ=0\.05\\delta=0\.05\.

### 5\.3Autonomous Performance and Routing Quality

We first ask whethersis\_\{i\}carries information about autonomous success beyond the aggregate task score\. For each system we sweep the threshold \(Equation[44](https://arxiv.org/html/2609.02095#S5.E44)\) over the development data and compute the autonomous coveragec⁡\(τ\)c\(\\tau\)and selective riskr⁡\(τ\)r\(\\tau\)of Section[3\.4](https://arxiv.org/html/2609.02095#S3.SS4)\. Figure[2](https://arxiv.org/html/2609.02095#S5.F2)shows the resulting risk–coverage curves\. An informative signal should lower selective risk as coverage falls, because the first cases removed from autonomous handling should be disproportionately failures, which is broadly what we observe\. Each curve begins at the system’s lowest reachable coverage; the Gemini models return100%100\\%confidence on up to73%73\\%of cases, so even their most selective reachable policy still leaves most runs autonomous\.

Figure 2:Risk–coverage behavior of the stated\-confidence routing signal on the development partition\.Sweeping the routing threshold traces the tradeoff between autonomous coverage and failure risk among autonomously accepted cases; lower curves indicate more effective selective routing\. Faint curves show all 16 systems; highlighted are the running\-example pair, Claude Sonnet 5 and GPT\-5\.4, whose autonomous accuracies differ by only 0\.3 percentage points\. Markers on the highlighted curves denote their reachable operating points: only a threshold at a tie\-group boundary of a system’s stated confidences can be selected\. The shaded band marks coverage levels supported by fewer than 19 accepted cases, where the selective\-risk estimate is unstable\.Table[2](https://arxiv.org/html/2609.02095#S5.T2)summarizes autonomous success and routing quality\. Across the1616systems, base accuracya0a\_\{0\}and AUROC are essentially uncorrelated \(Pearsonρ=−0\.12\\rho=\-0\.12\): a system can be accurate yet rank its own outputs poorly, or the reverse\. Qwen\-3\.7\-Plus is the clearest case, pairing one of the lowest base accuracies with the best AUROC\. AURC, by contrast, tracks base accuracy closely \(ρ=−0\.80\\rho=\-0\.80witha0a\_\{0\}\), which is exactly why we report both—AURC captures the absolute risk–coverage tradeoff a deployment faces, while AUROC isolates ranking quality from the base rate\.

Table 2:Autonomous performance and routing quality on the development partition\.n=375n=375cases per system\.a0a\_\{0\}is four\-way verdict accuracy under full autonomy, with no review\. AURC summarises the risk–coverage curve of Figure[2](https://arxiv.org/html/2609.02095#S5.F2)and is the quantity the deployment frontier is built from; lower is better\. AUROC measures ranking quality independent of baseline accuracy; higher is better\. Succ@50% is selective success over the half of cases each system is most confident in\.
### 5\.4Reliability–Oversight Frontier

Routing quality becomes operational when translated into the oversight required to meet a reliability target\. For each targetYYwe solve Equation[45](https://arxiv.org/html/2609.02095#S5.E45)onSdevS\_\{\\mathrm\{dev\}\}to select the highest\-coverage threshold meeting the requirement, freeze it, and apply it to the held\-outSqualS\_\{\\mathrm\{qual\}\}\. Figure[3](https://arxiv.org/html/2609.02095#S5.F3)plots the resulting human\-review burden against target reliability\. This READY deployment frontier shows how much work must be routed to review as the organization demands more reliable operation, and it lets systems be compared at a common deployment requirement rather than a common coverage level: at a fixed target, the system needing less review supports the more efficient policy under the same review assumption\.

Figure 3:Reliability–oversight frontier on the held\-out qualification partition, at review assumptionah=0\.90a\_\{h\}=0\.90\. The same routing evidence is converted into the minimum human\-review burden required to satisfy increasing reliability targetsYY\. Policies are selected on the development partition atY\+δY\+\\deltaand frozen before qualification\. Faint curves show all 16 systems; highlighted are the running\-example pair: atY=0\.76Y=0\.76, Claude Sonnet 5 qualifies while routing 29\.6% of cases to human review against GPT\-5\.4’s 39\.2%, despite near\-identical autonomous accuracy \(72\.5% vs\. 72\.8%\)\. Becauseah=0\.90a\_\{h\}=0\.90and selective success is below it at every observed coverage level, no policy can exceed reliability0\.900\.90; every system’s review burden exceeds90%90\\%byY=0\.85Y=0\.85\.
### 5\.5Held\-Out Qualification and Deployment Comparison

READY separates a development\-set operating point from a statistically qualified deployment claim: we evaluate each frozen policy onSqualS\_\{\\mathrm\{qual\}\}and compute the reliability estimateR^\\widehat\{R\}together with its one\-sided95%95\\%lower confidence bound \(LCB\) from Section[3\.5](https://arxiv.org/html/2609.02095#S3.SS5)\. Following Equation[34](https://arxiv.org/html/2609.02095#S3.E34), the LCB attaches exact Clopper–Pearson confidence to the observed accepted component and*no*confidence to the assumed review termaha\_\{h\}; the policy qualifies iffLCB≥Y\\text\{LCB\}\\geq Y\. The step\-by\-step computation is given in Appendix[B\.1](https://arxiv.org/html/2609.02095#A2.SS1)\. Table[3](https://arxiv.org/html/2609.02095#S5.T3)reports qualification atY=0\.76Y=0\.76\.

The table separates two ways a point\-estimate optimum fails to become a deployment\. GLM\-5\.2 requires the lowest review burden in the table \(10\.9%10\.9\\%\) and its point estimate clears the target \(R^=0\.762\\widehat\{R\}=0\.762\), yet it does not qualify\. Because it accepts89%89\\%of cases, almost all of its reliability rests on the measured component, giving it the widest confidence interval in the table \(R^−LCB=0\.037\\widehat\{R\}\-\\mathrm\{LCB\}=0\.037\), while its point estimate clearsYYby only0\.0020\.002\. The same low review burden also leaves the review assumption without leverage: raisingaha\_\{h\}moves only the10\.9%10\.9\\%of mass that is reviewed, so noah∈\[0,1\]a\_\{h\}\\in\[0,1\]rescues it\. Making GLM\-5\.2 qualify would requireδ≥0\.075\\delta\\geq 0\.075, raising its review burden to32\.0%32\.0\\%—above Opus 5 and GPT\-5\.5\. At the other extreme, Gemini\-3\.1\-Pro—whose stated confidence takes only five distinct values—can reach no development\-feasible threshold below full review, so it qualifies only by routing100%100\\%of cases to a human, attaining exactlyaha\_\{h\}without deploying\. Together these cases make the central point: optimizing a policy on fixed data is easy; making a deployment claim that survives on new data is a much stronger bar\.

Because Review% at a single target is a step function of each system’s reachable threshold grid, systems with coarse signals overshoot—R^−Y\\widehat\{R\}\-Yranges from\+0\.002\+0\.002\(GLM\-5\.2\) to\+0\.140\+0\.140\(Gemini\-3\.1\-Pro\)\. The margin is what closes the gap between a point estimate and a qualified deployment: without it only46%46\\%of policies qualify across the frontier sweep, against95%95\\%with it\. Cross\-system review burden is therefore best compared on the full frontier of Figure[3](https://arxiv.org/html/2609.02095#S5.F3), not at a single target\.

Table 3:Held\-out deployment qualification atY=0\.76Y=0\.76\.One row per system,Nqual=375N\_\{\\mathrm\{qual\}\}=375, review assumptionah=0\.90a\_\{h\}=0\.90\. Thresholds are selected on the development half atY\+0\.05Y\+0\.05and frozen before qualification; the margin absorbs selection bias, without which only 46% of policies qualify across the frontier sweep, against 95% with it; at this target, 15 of 16\. Following Equation[34](https://arxiv.org/html/2609.02095#S3.E34), confidence is attached only to the observed component \(exact Clopper–Pearson on accepted cases\), none to the assumedaha\_\{h\}, so every status is conditional on that assumption\.ahmina\_\{h\}^\{\\min\}\(Equation[35](https://arxiv.org/html/2609.02095#S3.E35)\) is the weakest review\-success rate under which the policy still qualifies; — marks policies qualifying under noah∈\[0,1\]a\_\{h\}\\in\[0,1\]\. A system reviewing 100% of cases attains exactlyaha\_\{h\}and qualifies*without deploying*, so systems are compared on Review%\.†Qualifies only by routing every case to review, attaining exactlyaha\_\{h\}; not a deployment\.
### 5\.6Similar Autonomous Performance, Different Deployment Value

The central READY hypothesis is that autonomous benchmark performance alone does not determine deployment value\. To test it, we compare two systems with nearly identical full\-autonomy accuracy but different routing quality, with the pair chosen onSdevS\_\{\\mathrm\{dev\}\}and frozen before we read the held\-out results\.

A capability leaderboard would rank GPT\-5\.4 \(a0=72\.8a\_\{0\}=72\.8\) and Sonnet 5 \(a0=72\.5a\_\{0\}=72\.5\) as essentially tied—0\.30\.3points apart—yet their stated\-confidence signals differ in quality, with Sonnet 5 the stronger router \(AUROC0\.6810\.681vs\.0\.6010\.601; AURC0\.2030\.203vs\.0\.2220\.222\)\. OnSqualS\_\{\\mathrm\{qual\}\}atY=0\.76Y=0\.76andah=0\.90a\_\{h\}=0\.90, both qualify, but at markedly different oversight cost: Sonnet 5 meets the target while routing only29\.6%29\.6\\%of cases to human review \(R^=0\.856\\widehat\{R\}=0\.856, LCB0\.8260\.826\), whereas GPT\-5\.4 requires39\.2%39\.2\\%\(R^=0\.828\\widehat\{R\}=0\.828, LCB0\.7970\.797\)—a third more human work for the same reliability\. The gap reflects routing quality, not accuracy: because Sonnet 5’s confidence more sharply separates its own successes from its failures, a less aggressive threshold already clears the reliability target and leaves more work safely autonomous\.

Better routing does not translate into less review monotonically at a single target, and we report a case that shows why rather than suppress it\. Qwen\-3\.7\-Plus and MiniMax\-M3 have identical development accuracy \(a0=66\.4a\_\{0\}=66\.4\) and Qwen has the better signal on both AURC and AUROC, yet Qwen requires61\.9%61\.9\\%review atY=0\.76Y=0\.76against MiniMax’s33\.9%33\.9\\%\. The cause is overshoot: Qwen’s nearest reachable threshold deliversR^=0\.874\\widehat\{R\}=0\.874, paying for1111points of reliability the target did not require, so single\-target review burden is a noisy summary, and the frontier of Figure[3](https://arxiv.org/html/2609.02095#S5.F3)is the fairer cross\-system comparison\.

Two systems can thus post near\-identical benchmark scores yet support different operating policies, because one more effectively identifies the cases on which autonomous execution will fail\. The difference surfaces not as another accuracy number but as the human oversight needed to reach the same reliability—the capability\-versus\-deployability distinction that motivates READY\.

### 5\.7Scope and Additional Analyses

This analysis deliberately isolates terminal accept\-or\-escalate routing so the relationship among routing quality, reliability, and review burden stays transparent; it is not a replacement for the full CliniCARE evaluation\. A richer deployment could enlarge the action space—for example, requesting additional evidence before adjudication—but doing so makes the oversight action depend on the workflow’s own state and can change the subsequent trajectory, moving into the trajectory\-dependent setting of Section[3\.7](https://arxiv.org/html/2609.02095#S3.SS7)rather than the signal\-based policy studied here\.

Appendix[B](https://arxiv.org/html/2609.02095#A2)reports the supporting analyses: sensitivity of the frontier to the assumed review successaha\_\{h\}, robustness under the process\-aware success definition \(Equation[42](https://arxiv.org/html/2609.02095#S5.E42)\), and complete per\-system results\. Taken together, the case study demonstrates the deployment\-level information READY adds: the same task evidence used by a conventional benchmark is translated into a statistically qualified relationship among autonomous performance, routing quality, human oversight, and operating reliability\.

## 6Related Work

Work / familyPrimary questionExecutable task

Interactive

Reliability

Operating cost

Human oversight

Statistical qual\.

READY \(ours\)Under what conditions and at what cost can an organization reliably deploy an agent on a specified workflow?✓✓✓✓✓✓\(§[6\.1](https://arxiv.org/html/2609.02095#S6.SS1)\) Capability and work\-product benchmarksHumanity’s Last Exam\[[13](https://arxiv.org/html/2609.02095#bib.bib19)\]Does the model know the answer to closed\-ended, frontier\-difficulty questions?×\\times×\\times×\\times×\\times×\\times×\\timesAgents’ Last Exam\[[1](https://arxiv.org/html/2609.02095#bib.bib26)\]Can a generalist computer\-use agent complete authentic, long\-horizon professional work?✓×\\times×\\times×\\times×\\times×\\timesGDPval\[[2](https://arxiv.org/html/2609.02095#bib.bib27)\]Is the model\-generated work product as good as an expert’s?✓×\\times×\\times∼\\sim×\\times×\\timesτ2\\tau^\{2\}\-bench\[[14](https://arxiv.org/html/2609.02095#bib.bib72)\]Can an agent coordinate with a user and tools to complete a task reliably across trials?✓✓✓×\\times×\\times×\\timesCRMArena\-Pro\[[15](https://arxiv.org/html/2609.02095#bib.bib64)\]Can an agent perform professional CRM tasks across business functions and interaction types?✓∼\\sim×\\times×\\times×\\times×\\timesAI Agents That Matter\[[8](https://arxiv.org/html/2609.02095#bib.bib123)\]Do agent benchmarks reward accuracy while ignoring cost?✓×\\times×\\times✓×\\times∼\\sim\(§[6\.2](https://arxiv.org/html/2609.02095#S6.SS2)\) Process, evidence, and trajectory evaluationProcess / trajectory eval\.\[[16](https://arxiv.org/html/2609.02095#bib.bib12),[17](https://arxiv.org/html/2609.02095#bib.bib13)\]Are the intermediate steps, tool use, and evidence correct, not just the final answer?✓×\\times×\\times×\\times×\\times×\\times\(§[6\.3](https://arxiv.org/html/2609.02095#S6.SS3)\) Selective prediction, abstention, and learning to deferLearning to defer\[[18](https://arxiv.org/html/2609.02095#bib.bib22),[19](https://arxiv.org/html/2609.02095#bib.bib23)\]Should the model predict this case or defer it to a human?×\\times×\\times×\\times∼\\sim✓×\\times\(§[6\.4](https://arxiv.org/html/2609.02095#S6.SS4)\) Confidence, uncertainty, and calibrationConfidence / calibration\[[20](https://arxiv.org/html/2609.02095#bib.bib30),[21](https://arxiv.org/html/2609.02095#bib.bib32)\]How likely is this output correct, and is that confidence calibrated?×\\times×\\times∼\\sim×\\times×\\times×\\times\(§[6\.5](https://arxiv.org/html/2609.02095#S6.SS5)\) Cost\-aware routing, cascades, and deferral economicsCost\-aware routing & cascades\[[6](https://arxiv.org/html/2609.02095#bib.bib77),[7](https://arxiv.org/html/2609.02095#bib.bib4)\]Can we hit a quality bar at lower cost by routing between models?∼\\sim×\\times∼\\sim✓×\\times×\\times\(§[6\.6](https://arxiv.org/html/2609.02095#S6.SS6)\) Human oversight, scalable oversight, and assuranceHuman oversight & assurance\[[22](https://arxiv.org/html/2609.02095#bib.bib91),[23](https://arxiv.org/html/2609.02095#bib.bib96)\]Does human review of AI improve outcomes and provide deployment assurance?×\\times∼\\sim∼\\sim×\\times✓∼\\simTable 4:READY as the integration point across the six related\-work families of Section[6](https://arxiv.org/html/2609.02095#S6)\.Each row group corresponds to one subsection \(§[6\.1](https://arxiv.org/html/2609.02095#S6.SS1)–§[6\.6](https://arxiv.org/html/2609.02095#S6.SS6)\), with a representative work or method, along six axes: whether it runs an agent on*executable*multi\-step work; whether it involves an*interactive*user or counterparty; whether it measures*reliability*against a target; whether it accounts for operating*cost*; whether it models*human oversight*; and whether it produces a*statistically qualified*deployment claim on held\-out data\. ✓ = addressed,∼\\sim= partial or implicit,×\\times= not addressed\. READY is the only approach that integrates all six into a single deployment\-qualification claim\.READY is not another benchmark of professional capability\. It adds a deployment\-qualification layer on top of such benchmarks: given evidence of how well an agent performs a workflow, it asks under what oversight policy, at what cost, and with what statistical support the agent can be deployed at a target reliability\. This section situates READY against six bodies of work: capability benchmarks for knowledge and professional agents \(Section[6\.1](https://arxiv.org/html/2609.02095#S6.SS1)\); outcome, process, and evidence\-based evaluation \(Section[6\.2](https://arxiv.org/html/2609.02095#S6.SS2)\); selective prediction, abstention, and learning to defer \(Section[6\.3](https://arxiv.org/html/2609.02095#S6.SS3)\); confidence, uncertainty, and calibration as routing signals \(Section[6\.4](https://arxiv.org/html/2609.02095#S6.SS4)\); cost\-aware routing, cascades, and deferral economics \(Section[6\.5](https://arxiv.org/html/2609.02095#S6.SS5)\); and human oversight, scalable oversight, and deployment assurance \(Section[6\.6](https://arxiv.org/html/2609.02095#S6.SS6)\); and, finally, the general agent\-evaluation frameworks READY builds on rather than competes with \(Section[6\.7](https://arxiv.org/html/2609.02095#S6.SS7)\)\. READY draws on all six but integrates them into a common deployment\-qualification framework that evaluates the reliability, oversight burden, and operating cost of the deployed human–AI system\.

Table[4](https://arxiv.org/html/2609.02095#S6.T4)organizes this section\. Each row group corresponds to one of the families surveyed below, and the columns are the deployment dimensions READY integrates: executable and interactive task evidence, measured reliability, operating cost, human oversight, and statistical qualification\. Reading across the READY row, every dimension is addressed; reading across any other row, the adjacent literatures each supply some dimensions but none combine them into a deployment\-qualification claim\. READY is the integration point across all six\.

### 6\.1From Knowledge Exams to Professional\-Work Agents

Evaluation of frontier models has progressed from static knowledge exams to executable professional work\. Closed\-ended exams such as Humanity’s Last Exam\[[13](https://arxiv.org/html/2609.02095#bib.bib19)\]measure expert knowledge at the frontier of human difficulty, but they score isolated answers not an agent’s ability to carry out a task or to be relied upon in operation\. Agentic benchmarks instead evaluate whether a system can*do*the work\. Agents’ Last Exam\[[1](https://arxiv.org/html/2609.02095#bib.bib26)\]targets authentic, long\-horizon computer\-use workflows; GDPval\[[2](https://arxiv.org/html/2609.02095#bib.bib27)\]compares model deliverables against expert work products and estimates the associated time or cost savings; xbench\[[3](https://arxiv.org/html/2609.02095#bib.bib28)\]tracks profession\-aligned productivity and technology–market fit; and recent work formalizes how to design and report benchmarks for knowledge work\[[24](https://arxiv.org/html/2609.02095#bib.bib29)\]\.

A large surrounding literature exercises the enterprise and tool\-use skills these workflows depend on\. General\-assistant and web benchmarks include GAIA\[[25](https://arxiv.org/html/2609.02095#bib.bib62)\], WebArena and VisualWebArena\[[26](https://arxiv.org/html/2609.02095#bib.bib67),[27](https://arxiv.org/html/2609.02095#bib.bib69)\], and OSWorld\[[28](https://arxiv.org/html/2609.02095#bib.bib70)\]; enterprise task suites include WorkArena\[[29](https://arxiv.org/html/2609.02095#bib.bib66)\], CRMArena and CRMArena\-Pro\[[30](https://arxiv.org/html/2609.02095#bib.bib63),[15](https://arxiv.org/html/2609.02095#bib.bib64)\], and Terminal\-Bench\[[31](https://arxiv.org/html/2609.02095#bib.bib76)\]; conversational tool\-use with a simulated user is studied byτ\\tau\-bench andτ2\\tau^\{2\}\-bench\[[5](https://arxiv.org/html/2609.02095#bib.bib71),[14](https://arxiv.org/html/2609.02095#bib.bib72)\]; and retrieval\- and browsing\-heavy work is covered by BrowseComp\[[32](https://arxiv.org/html/2609.02095#bib.bib60)\]and deep\-research benchmarks\[[33](https://arxiv.org/html/2609.02095#bib.bib58),[34](https://arxiv.org/html/2609.02095#bib.bib59)\]\. Recent surveys further catalog this rapidly growing space\[[35](https://arxiv.org/html/2609.02095#bib.bib124),[36](https://arxiv.org/html/2609.02095#bib.bib125)\], and enterprise\-grade benchmarks now span software engineering\[[37](https://arxiv.org/html/2609.02095#bib.bib128)\], simulated companies\[[38](https://arxiv.org/html/2609.02095#bib.bib127)\], workplace tasks\[[39](https://arxiv.org/html/2609.02095#bib.bib130),[40](https://arxiv.org/html/2609.02095#bib.bib131)\], and enterprise data analytics\[[41](https://arxiv.org/html/2609.02095#bib.bib129)\], building on general agent benchmarks such as AgentBench\[[42](https://arxiv.org/html/2609.02095#bib.bib126)\]\. A parallel line evaluates the safety and trustworthiness of agent behavior beyond task success\[[43](https://arxiv.org/html/2609.02095#bib.bib73),[44](https://arxiv.org/html/2609.02095#bib.bib65),[45](https://arxiv.org/html/2609.02095#bib.bib74)\]\.

However, these evaluations primarily characterize whether the work gets done and how well, although recent work has begun to expose additional operational dimensions\.[Kapoor et al\. \[8\]](https://arxiv.org/html/2609.02095#bib.bib123)show that accuracy\-only agent benchmarks can favor needlessly costly systems and advocate jointly reporting cost and accuracy on held\-out data, whileτ\\tau\-bench’s*passk*measures whether an agent succeeds acrosskkindependent attempts and thereby captures consistency beyond single\-attempt capability\[[5](https://arxiv.org/html/2609.02095#bib.bib71)\]\. READY builds on these richer forms of task evidence but addresses a distinct deployment question: which oversight policy meets a prespecified reliability target at minimum operating cost, and does held\-out evidence statistically support that operating point? Thus, READY consumes rather than replaces capability benchmarks, adding a reliability–oversight–cost qualification layer for deployment\. The first row group of Table[4](https://arxiv.org/html/2609.02095#S6.T4)places these capability and work\-product benchmarks against READY’s deployment dimensions\.

### 6\.2Outcome, Process, and Evidence\-Based Evaluation

Recent work increasingly recognizes that endpoint task success alone may provide an incomplete account of agent behavior\. Process\- and trajectory\-oriented evaluations therefore examine intermediate execution behavior in addition to final outcomes, including whether agents follow appropriate procedures, use tools correctly, and satisfy task\-specific intermediate requirements\[[16](https://arxiv.org/html/2609.02095#bib.bib12),[17](https://arxiv.org/html/2609.02095#bib.bib13)\]\. Related work on factuality and grounding evaluates whether generated outputs are supported by available evidence, while recent studies further show that conclusions about hallucination or grounding can depend substantially on the evaluation protocol itself\[[46](https://arxiv.org/html/2609.02095#bib.bib14),[47](https://arxiv.org/html/2609.02095#bib.bib15)\]\. Other approaches use consistency or model\-internal signals to detect unsupported generation\[[48](https://arxiv.org/html/2609.02095#bib.bib54),[49](https://arxiv.org/html/2609.02095#bib.bib16),[50](https://arxiv.org/html/2609.02095#bib.bib55)\], while work on reasoning faithfulness demonstrates that plausible model explanations need not faithfully reflect the processes that produced an answer\[[51](https://arxiv.org/html/2609.02095#bib.bib56),[52](https://arxiv.org/html/2609.02095#bib.bib57)\]\. More broadly, recent reviews of agent evaluation argue that deployment\-oriented assessment must extend beyond aggregate task completion to consider execution quality, evidence, safety, and other workflow\-specific requirements\[[53](https://arxiv.org/html/2609.02095#bib.bib17)\]\.

READY builds on these ideas but addresses a distinct deployment\-qualification problem\. It does not define a universal truthfulness or grounding metric\. These properties enter through the workflow evaluatorgwg\_\{w\}when they matter for success, for example, a workflow may require evidence grounding, provenance, required tool use, or appropriate abstention alongside outcome correctness, and the success predicateϕw\\phi\_\{w\}specifies which measurements an execution must satisfy\. Crucially, any process requirement used for qualification must be supported by observable execution evidence, such as retrieved sources, tool calls, or state changes, not unrecorded model reasoning\. READY treats outcome and process evaluation as workflow\-specific inputs to a common reliability\-and\-oversight methodology, not as universal properties of an agent\.

### 6\.3Selective Prediction, Abstention, and Learning to Defer

READY’s terminal accept\-or\-escalate setting is an instance of selective prediction, in which a system answers on a subset of cases and abstains on the remainder, trading coverage against error among accepted cases\. This risk–coverage formulation has a long history: the optimal reject rule dates to[Chow \[54\]](https://arxiv.org/html/2609.02095#bib.bib99), selective classification was formalized in later work\[[55](https://arxiv.org/html/2609.02095#bib.bib20)\], and confidence\-based selection was extended to deep networks\[[56](https://arxiv.org/html/2609.02095#bib.bib21)\]\. The reject\-option and abstention settings are surveyed by[Hendrickx et al\. \[57\]](https://arxiv.org/html/2609.02095#bib.bib100)and, for language models, by[Wen et al\. \[58\]](https://arxiv.org/html/2609.02095#bib.bib102), with optimal\-strategy characterizations in[Franc et al\. \[59\]](https://arxiv.org/html/2609.02095#bib.bib101)\. A closely related literature studies*learning to defer*, in which a model chooses between predicting and passing a case to a downstream decision\-maker\[[18](https://arxiv.org/html/2609.02095#bib.bib22),[19](https://arxiv.org/html/2609.02095#bib.bib23)\]\. Subsequent work develops calibrated deferral\[[60](https://arxiv.org/html/2609.02095#bib.bib103)\], exact deferral algorithms\[[61](https://arxiv.org/html/2609.02095#bib.bib104)\], human\-complementary predictors\[[62](https://arxiv.org/html/2609.02095#bib.bib105)\], and coverage\-constrained multi\-expert cooperation\[[63](https://arxiv.org/html/2609.02095#bib.bib106)\]\. Distribution\-free guarantees for selective decisions and thresholds are surveyed in[Angelopoulos and Bates \[64\]](https://arxiv.org/html/2609.02095#bib.bib107),[Campos et al\. \[65\]](https://arxiv.org/html/2609.02095#bib.bib108)\.

Recent work extends these ideas to language models\. AbstentionBench\[[66](https://arxiv.org/html/2609.02095#bib.bib61)\]evaluates whether models appropriately abstain on unanswerable questions; conformal methods construct coverage guarantees for LLM predictions, including settings without logit access\[[67](https://arxiv.org/html/2609.02095#bib.bib51),[68](https://arxiv.org/html/2609.02095#bib.bib52)\]; and uncertainty\-aware planners request human assistance when uncertainty is high\[[69](https://arxiv.org/html/2609.02095#bib.bib53)\]\. These approaches inherit the same basic tradeoff between automated coverage and error among accepted cases\. Accordingly, the risk–coverage curve and its area \(AURC\), which we use in Section[3\.4](https://arxiv.org/html/2609.02095#S3.SS4)to summarize routing quality, come directly from the selective\-prediction literature\[[55](https://arxiv.org/html/2609.02095#bib.bib20),[56](https://arxiv.org/html/2609.02095#bib.bib21)\]\. Likewise, the human\-review pathway evaluated by READY is closely connected to the classical learning\-to\-defer formulation, in which responsibility is allocated between the model and a downstream decision\-maker\[[18](https://arxiv.org/html/2609.02095#bib.bib22),[19](https://arxiv.org/html/2609.02095#bib.bib23)\]\.

READY builds on selective autonomy but changes what is optimized\. An abstention or deferral rate alone does not determine whether a deployment is acceptable, because it says nothing about what happens to deferred cases\. READY instead models the outcome of the declared human\-review path and evaluates the reliability of the human–AI system,

Rm,w​\(π\)=𝔼⁡\[uπ\],R\_\{m,w\}\(\\pi\)=\\mathbb\{E\}\[u^\{\\pi\}\],\(46\)together with its expected operating costCm,w​\(π\)C\_\{m,w\}\(\\pi\), and then selects the oversight policy that meets a reliability target at minimum cost\. This formulation is related to work on human\-complementary prediction and coverage\-constrained cooperation\[[62](https://arxiv.org/html/2609.02095#bib.bib105),[63](https://arxiv.org/html/2609.02095#bib.bib106)\], but READY’s objective is deployment qualification under a workflow\-level reliability requirement rather than optimization of abstention or coverage alone\.

This has therefore led to three distinctions\. First, human review is part of the evaluated operating policy, not merely an abstention\. Second, READY separates policy selection from held\-out statistical qualification, so the reported operating point is supported by independent evidence\. Third, READY distinguishes*task\-level abstention*—a valid workflow outcome, such as a correct indeterminate verdict in CliniCARE\-Bench—from*deployment\-level escalation*, the separate decision to route a completed result to human review; Appendix[B](https://arxiv.org/html/2609.02095#A2)develops this distinction\. Thus, selective prediction and learning\-to\-defer methods provide the conceptual foundation for selective autonomy, while READY uses that foundation to qualify an explicit human–AI deployment operating point in terms of reliability, oversight, and cost\.

### 6\.4Confidence, Uncertainty, and Calibration as Routing Signals

Terminal oversight requires an observable signalsis\_\{i\}that ranks completed executions by their likelihood of autonomous success\. Many candidate signals exist\. Models can self\-assess correctness through elicited probabilities such as*P\(True\)*and*P\(IK\)*\[[20](https://arxiv.org/html/2609.02095#bib.bib30)\]or token\-level self\-evaluation\[[70](https://arxiv.org/html/2609.02095#bib.bib31)\]\. Verbalized confidence\[[71](https://arxiv.org/html/2609.02095#bib.bib34),[72](https://arxiv.org/html/2609.02095#bib.bib35),[73](https://arxiv.org/html/2609.02095#bib.bib1),[74](https://arxiv.org/html/2609.02095#bib.bib2)\], relative confidence preferences\[[75](https://arxiv.org/html/2609.02095#bib.bib36)\], semantic uncertainty\[[21](https://arxiv.org/html/2609.02095#bib.bib32),[73](https://arxiv.org/html/2609.02095#bib.bib1)\], confidence\-weighted self\-consistency\[[76](https://arxiv.org/html/2609.02095#bib.bib33)\], input\-clarification ensembles\[[77](https://arxiv.org/html/2609.02095#bib.bib37)\], functional uncertainty\[[78](https://arxiv.org/html/2609.02095#bib.bib38)\], and logit\-based methods\[[79](https://arxiv.org/html/2609.02095#bib.bib39)\]offer further ways to estimate uncertainty\.

READY does not prescribe a signal; it evaluates the routing behavior induced by whatever signal the deployment policy can observe\. For threshold routing, the score need not be a calibrated probability—it need only order cases so that more selective acceptance preferentially removes higher\-risk executions \(Section[3\.4](https://arxiv.org/html/2609.02095#S3.SS4)\)\. Calibration\[[80](https://arxiv.org/html/2609.02095#bib.bib40),[81](https://arxiv.org/html/2609.02095#bib.bib44),[82](https://arxiv.org/html/2609.02095#bib.bib45)\], its behavior in language models\[[83](https://arxiv.org/html/2609.02095#bib.bib41),[84](https://arxiv.org/html/2609.02095#bib.bib42),[85](https://arxiv.org/html/2609.02095#bib.bib43),[72](https://arxiv.org/html/2609.02095#bib.bib35)\], and training\-time methods for improving it\[[86](https://arxiv.org/html/2609.02095#bib.bib46),[87](https://arxiv.org/html/2609.02095#bib.bib47),[88](https://arxiv.org/html/2609.02095#bib.bib48),[89](https://arxiv.org/html/2609.02095#bib.bib49),[90](https://arxiv.org/html/2609.02095#bib.bib50)\]remain relevant when the numerical value of the signal is interpreted probabilistically, but calibration is not a prerequisite for qualification\. Because signals that require repeated sampling or extra inference improve discrimination at a cost, READY folds that cost into the operating\-cost quantityCm,w​\(π\)C\_\{m,w\}\(\\pi\)instead of treating it as external to the comparison\.

Surveys of confidence estimation and uncertainty quantification for language models organize the space of candidate signals\[[91](https://arxiv.org/html/2609.02095#bib.bib109),[92](https://arxiv.org/html/2609.02095#bib.bib110)\], as do hallucination surveys\[[93](https://arxiv.org/html/2609.02095#bib.bib111),[94](https://arxiv.org/html/2609.02095#bib.bib112)\]\. Representative signals include semantic entropy\[[95](https://arxiv.org/html/2609.02095#bib.bib113)\], black\-box confidence elicitation\[[96](https://arxiv.org/html/2609.02095#bib.bib114)\], internal\-state detectors\[[97](https://arxiv.org/html/2609.02095#bib.bib116)\], and meaning\-aware scores\[[98](https://arxiv.org/html/2609.02095#bib.bib117),[99](https://arxiv.org/html/2609.02095#bib.bib118)\], with large\-scale comparisons of which signals are actually discriminative\[[100](https://arxiv.org/html/2609.02095#bib.bib115)\]\. Selective generation\[[101](https://arxiv.org/html/2609.02095#bib.bib119)\]and conformal generation\[[102](https://arxiv.org/html/2609.02095#bib.bib120)\]apply such signals to the accept\-or\-defer decision directly, and when a signal is interpreted probabilistically its calibration must be measured carefully\[[103](https://arxiv.org/html/2609.02095#bib.bib121),[104](https://arxiv.org/html/2609.02095#bib.bib122)\]\.

### 6\.5Cost\-Aware Routing, Cascades, and Deferral Economics

READY selects the oversight policy that meets a reliability target at minimum operating cost, which connects it to a large literature on cost\-aware inference for large language models\. Model*cascades*invoke a cheap model first and escalate to a more expensive one when a confidence signal is low\[[105](https://arxiv.org/html/2609.02095#bib.bib6),[106](https://arxiv.org/html/2609.02095#bib.bib5),[107](https://arxiv.org/html/2609.02095#bib.bib3)\]\. FrugalGPT learns such a cascade to hit a target accuracy at reduced cost\[[6](https://arxiv.org/html/2609.02095#bib.bib77)\], AutoMix escalates under noisy self\-verification via a meta\-router\[[108](https://arxiv.org/html/2609.02095#bib.bib80)\], mixture\-of\-thought cascades defer only hard reasoning cases\[[109](https://arxiv.org/html/2609.02095#bib.bib82)\], and EcoAssistant extends the pattern to code\-executing agents\[[110](https://arxiv.org/html/2609.02095#bib.bib81)\]\. A parallel line*routes*each query to the cheapest adequate model: RouteLLM learns a router from preference data\[[7](https://arxiv.org/html/2609.02095#bib.bib4)\], Hybrid LLM routes by predicted difficulty\[[111](https://arxiv.org/html/2609.02095#bib.bib78)\], and Zooter routes by predicted expertise\[[112](https://arxiv.org/html/2609.02095#bib.bib83)\], with RouterBench providing a standard cost–quality evaluation\[[113](https://arxiv.org/html/2609.02095#bib.bib79)\]\. Recent surveys formalize routing and cascading as a shared performance–cost optimization problem\[[114](https://arxiv.org/html/2609.02095#bib.bib84)\], a pattern that descends from classical test\-time cost\-sensitive cascades in detection\[[115](https://arxiv.org/html/2609.02095#bib.bib85)\]\.

READY shares the broader goal of reducing operating cost subject to an acceptable performance requirement, but differs in three deployment\-relevant respects\. First, the deferral target is a*human reviewer*, not a stronger model, so the escalation cost is human\-review effort and the reliability of the deployed system depends on the review path, not on a larger model’s accuracy\. Second, READY replaces an average cost–quality operating point with a*statistically qualified*reliability floor established on held\-out data, not a heuristic threshold or an average win rate\. Third, the quality bar is a workflow\-specific success predicate that may include process and policy requirements, not final\-answer correctness alone\. Cascade and routing methods are complementary: their confidence\-based escalation logic can supply the routing signalsis\_\{i\}, while READY supplies the qualification procedure that turns it into a deployment guarantee\.

### 6\.6Human Oversight, Scalable Oversight, and Deployment Assurance

READY evaluates a human–AI system under an oversight policy, connecting most directly to the work on human oversight of AI and assurance for deployment\. The notion that supervision is a costly, budgeted resource originates in the scalable\-oversight literature\[[116](https://arxiv.org/html/2609.02095#bib.bib86)\]; subsequent work measures whether a human assisted by an unreliable model outperforms either alone\[[117](https://arxiv.org/html/2609.02095#bib.bib87)\], studies oversight when the supervised system is stronger than its overseer\[[118](https://arxiv.org/html/2609.02095#bib.bib88)\], and proposes debate and amplification as mechanisms\[[119](https://arxiv.org/html/2609.02095#bib.bib89),[120](https://arxiv.org/html/2609.02095#bib.bib90),[121](https://arxiv.org/html/2609.02095#bib.bib11)\]\. More recent work emphasizes that effective oversight requires meaningful human agency rather than nominal human presence\[[122](https://arxiv.org/html/2609.02095#bib.bib10),[123](https://arxiv.org/html/2609.02095#bib.bib9)\], while empirical studies show that human–AI teaming does not reliably outperform the stronger individual component and depends on factors such as interaction design and reviewer expertise\[[124](https://arxiv.org/html/2609.02095#bib.bib7),[125](https://arxiv.org/html/2609.02095#bib.bib8)\]\. Particularly, a meta\-analysis of 106 studies finds human–AI combinations on average underperform the better of human or AI alone\[[22](https://arxiv.org/html/2609.02095#bib.bib91)\], complementary performance requires more than model explanations\[[126](https://arxiv.org/html/2609.02095#bib.bib92)\], and overreliance is a central failure mode of human review\[[127](https://arxiv.org/html/2609.02095#bib.bib93),[128](https://arxiv.org/html/2609.02095#bib.bib94)\]\. Mandated human oversight in particular can provide false assurance when reviewers cannot perform the checking function\[[129](https://arxiv.org/html/2609.02095#bib.bib95)\]\. Finally, assurance and auditing frameworks—model cards\[[23](https://arxiv.org/html/2609.02095#bib.bib96)\], internal algorithmic auditing\[[130](https://arxiv.org/html/2609.02095#bib.bib97)\], and socio\-technical safety evaluation\[[131](https://arxiv.org/html/2609.02095#bib.bib98)\]—establish reporting evaluated performance under intended deployment conditions\.

READY responds to these findings directly\. Rather than assuming human review improves outcomes, it treats the review path’s success rate as a measured quantity or an explicit deployment assumption, and qualifies the deployed system on held\-out evidence\. READY consequently complements model cards and audit frameworks by reporting a statistically supported deployment profile–the reliability, oversight burden, and operating cost of a specific human–AI configuration– rather than certifying the agent in isolation

### 6\.7General Agent\-Evaluation Frameworks

A final, infrastructure\-level body of work provides the execution and harness frameworks on which agent evaluations are built, as distinct from the six methodological literatures above\. Inspect AI\[[9](https://arxiv.org/html/2609.02095#bib.bib24)\]standardizes task specification, agent and tool integration, sandboxed execution, scoring, and evaluation logging; Harbor\[[132](https://arxiv.org/html/2609.02095#bib.bib25)\]packages agent evaluations as containerized tasks with verification scripts; and AgentBeats\[[133](https://arxiv.org/html/2609.02095#bib.bib18)\]provides a platform for running and comparing agent evaluations under a common protocol\. These frameworks answer*how*an evaluation is executed and recorded\. READY is deliberately not a competing harness: its reference implementation runs on Inspect and can consume Harbor\-packaged tasks through theinspect\-harboradapter\. READY adds the orthogonal layer of*what deployment operating point*the recorded evidence supports, selecting an oversight policy and statistically qualifying a reliability–oversight–cost profile on held\-out data\. In the terms of Table[4](https://arxiv.org/html/2609.02095#S6.T4), these frameworks supply the executable\-task substrate but none of the reliability, cost, oversight, or statistical\-qualification dimensions; READY is the qualification layer that sits above whichever framework produced the evidence\.

## 7Scope, Limitations, and Future Work

READY is intended as a framework for evaluating and statistically qualifying AI\-agent deployment under explicit workflow, oversight, reliability, and cost assumptions\. It does not attempt to provide a universal certification of an agent or to capture every organizational consideration involved in production deployment\.

#### Empirical scope\.

The empirical evaluation in this paper focuses on the trajectory\-invariant, terminal accept\-or\-escalate setting using CliniCARE\-Bench\. This setting provides a transparent test of the central READY methodology: routing agent outputs between autonomous acceptance and human review to meet a target reliability at minimum oversight cost\. Section[3\.7](https://arxiv.org/html/2609.02095#S3.SS7)extends the formulation to trajectory\-dependent policies in which review, clarification, correction, or other intervention changes subsequent execution\. Evaluating such policies requires policy\-in\-the\-loop execution or simulation; large\-scale empirical validation of these settings is left to future work\.

#### Human\-oversight assumptions\.

READY distinguishes measured evaluation evidence from assumptions about the deployment environment\. In particular, when the declared human\-review path is not executed directly, its success rate, latency, and cost must be supplied as deployment assumptions\. The deployment profile is conditional on those values\. Future evaluations should increasingly measure reviewer effectiveness, review time, intervention type, and recovery cost directly instead of treating them as fixed parameters\.

#### Population and distribution validity\.

A qualification claim applies to the workflow population represented by the evaluation and to the versioned configuration under which the evidence was collected\. It does not guarantee the same reliability under arbitrary distribution shift, changes in workflow composition, environment changes, or materially different agent behavior\. Continuous post\-deployment monitoring is therefore important for detecting performance degradation and determining when requalification is needed, but designing such monitoring and requalification triggers is beyond the scope of this work\.

#### Risk and deployment context\.

The primary READY objective represents workflow success through a workflow\-specific binary event and optimizes expected operating cost subject to a reliability target\. For workflows in which failures differ substantially in consequence, the framework permits additional risk constraints, including tail\-risk measures\.READY does not replace enterprise security, privacy, legal, regulatory, or organizational\-governance processes\. These considerations may define workflow constraints or evaluation criteria, but they remain outside the core deployment\-qualification methodology\.

Future work should extend READY along three directions: empirical evaluation of trajectory\-dependent and multi\-step oversight policies, direct measurement of human\-review effectiveness and economics, and broader validation across enterprise workflows and domains\. The open interface described in Section[4](https://arxiv.org/html/2609.02095#S4)supports these extensions without changing the underlying qualification objective\.

## 8Conclusion

Professional\-work benchmarks measure how well an AI agent can perform a task autonomously\. Enterprise deployment requires a different determination: whether the agent can be deployed at a required level of reliability, with an acceptable amount of human oversight and operating cost\.

*Reliable Enterprise Agent Deployment*\(READY\) formulates this as a deployment\-qualification problem\. For a given workflow, READY evaluates agent execution, selects an oversight policy that satisfies a target reliability at minimum operating cost, and statistically qualifies that operating point on held\-out evidence\. A deployment profile is not an intrinsic score of the agent\. It is a conditional statement about how that agent can be deployed under a particular workflow, oversight model, and set of operating assumptions\.

READY separates workflow\-specific definitions of successful work from a common qualification methodology and provides an open runtime testbed for executing and extending such evaluations\. Trajectory\-invariant policies can be evaluated efficiently from saved execution evidence, while trajectory\-dependent policies use the same optimization objective but require policy\-in\-the\-loop execution or simulation\. CliniCARE\-Bench provides an end\-to\-end instantiation of this framework and illustrates how autonomous task performance, routing quality, human oversight, and deployment reliability can be analyzed together\.

As agent capabilities improve, the relevant question is not simply whether benchmark scores rise, but how much human oversight remains necessary to meet a specified reliability requirement\. READY provides a framework for measuring that transition and for comparing systems by the cost of reliable deployment, not autonomous performance alone\.

## Author Contributions

Y\. Xue conceived the project, defined the vision and direction, led the manuscript effort, and supervised the work\. V\. Chatrath led benchmark execution, including the Contributor network and experiments, and served as primary manuscript lead\. B\. Zhu and J\. Fan served as technical and manuscript leads and were primary developers\. G\. Pu was a primary developer\. S\. D\. Tiwari, S\. Dan, and R\. Young developed the benchmark pipeline\. A\. Shanker contributed the mathematical formulation\. Y\. Li, Y\. Yao, and M\. Yang contributed to manuscript iteration\. D\. Y\. Zhang, Y\. He, Y\. Liu, C\. Wang, and Z\. Yin served as research advisors\. All authors contributed to the writing, reviewed and approved the final manuscript\.

## References

- \[1\]Y\. Sun, X\. Han, W\. Zhang, Y\. Pang, T\. Wang, Y\. Cao, Y\. Huang, C\. Duroiu,et al\.\(2026\)Agents’ Last Exam\.External Links:2606\.05405,[Link](https://arxiv.org/abs/2606.05405)Cited by:[§1](https://arxiv.org/html/2609.02095#S1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.02095#S1.p1.1),[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p1.1),[Table 4](https://arxiv.org/html/2609.02095#S6.T4.3.5.1.1.1)\.
- \[2\]T\. Patwardhan, R\. Dias, E\. Proehl, G\. Kim, M\. Wang, O\. Watkins, S\. P\. Fishman, M\. Aljubeh, P\. Thacker, L\. Fauconnet, N\. S\. Kim, P\. Chao, S\. Miserendino, G\. Chabot, D\. Li, M\. Sharman, A\. Barr, A\. Glaese, and J\. Tworek\(2025\)GDPval: evaluating AI model performance on real\-world economically valuable tasks\.External Links:2510\.04374,[Link](https://arxiv.org/abs/2510.04374)Cited by:[§1](https://arxiv.org/html/2609.02095#S1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.02095#S1.p1.1),[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p1.1),[Table 4](https://arxiv.org/html/2609.02095#S6.T4.3.6.1.1.1)\.
- \[3\]K\. Chen, Y\. Ren, Y\. Liu, X\. Hu, H\. Tian, T\. Xie,et al\.\(2025\)Xbench: tracking agents productivity scaling with profession\-aligned real\-world evaluations\.External Links:2506\.13651,[Link](https://arxiv.org/abs/2506.13651)Cited by:[§1](https://arxiv.org/html/2609.02095#S1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.02095#S1.p1.1),[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p1.1)\.
- \[4\]V\. Chatrath, B\. Zhu, G\. Pu, J\. Fan, A\. Shanker, V\. Ursekar, A\. Sharma, J\. Qin, K\. Han, S\. D\. Tiwari, S\. Dan, V\. Kalmath, Y\. Li, D\. Y\. Zhang, C\. Wang, Z\. Doctor, Z\. Yin, N\. H\. Shah, and Y\. Xue\(2026\)CliniCARE\-Bench: clinical calibrated audit of medical reasoning in EHR\.arXiv preprint arXiv:2608\.07796\.External Links:2608\.07796,[Link](https://arxiv.org/abs/2608.07796)Cited by:[item 3](https://arxiv.org/html/2609.02095#S1.I1.i3.p1.1),[§1](https://arxiv.org/html/2609.02095#S1.p5.1),[§2\.3](https://arxiv.org/html/2609.02095#S2.SS3.p1.1)\.
- \[5\]S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhan\(2024\)τ\\tau\-bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.External Links:2406\.12045,[Link](https://arxiv.org/abs/2406.12045)Cited by:[§1](https://arxiv.org/html/2609.02095#S1.SS0.SSS0.Px1.p1.1),[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1),[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p3.1)\.
- \[6\]L\. Chen, M\. Zaharia, and J\. Zou\(2023\)FrugalGPT: how to use large language models while reducing cost and improving performance\.External Links:2305\.05176Cited by:[§1](https://arxiv.org/html/2609.02095#S1.SS0.SSS0.Px1.p1.1),[§6\.5](https://arxiv.org/html/2609.02095#S6.SS5.p1.1),[Table 4](https://arxiv.org/html/2609.02095#S6.T4.3.17.1.1.1)\.
- \[7\]I\. Ong, A\. Almahairi, V\. Wu, W\. Chiang, T\. Wu, J\. E\. Gonzalez, M\. Kadous, and I\. Stoica\(2025\)Routellm: learning to route llms from preference data\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 34433–34448\.Cited by:[§1](https://arxiv.org/html/2609.02095#S1.SS0.SSS0.Px1.p1.1),[§6\.5](https://arxiv.org/html/2609.02095#S6.SS5.p1.1),[Table 4](https://arxiv.org/html/2609.02095#S6.T4.3.17.1.1.1)\.
- \[8\]S\. Kapoor, B\. Stroebl, Z\. S\. Siegel, N\. Nadgir, and A\. Narayanan\(2024\)AI agents that matter\.arXiv preprint arXiv:2407\.01502\.External Links:2407\.01502Cited by:[§1](https://arxiv.org/html/2609.02095#S1.SS0.SSS0.Px1.p1.1),[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p3.1),[Table 4](https://arxiv.org/html/2609.02095#S6.T4.3.9.1.1.1)\.
- \[9\]UK AI Security Institute\(2024\)Inspect ai: framework for large language model evaluations\.Note:Released May 2024External Links:[Link](https://github.com/UKGovernmentBEIS/inspect_ai)Cited by:[§1](https://arxiv.org/html/2609.02095#S1.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2609.02095#S4.p2.1),[§6\.7](https://arxiv.org/html/2609.02095#S6.SS7.p1.1)\.
- \[10\]Kidney Disease: Improving Global Outcomes \(KDIGO\) Acute Kidney Injury Work Group\(2012\)KDIGO clinical practice guideline for acute kidney injury\.Kidney International Supplements\.Cited by:[§2\.3](https://arxiv.org/html/2609.02095#S2.SS3.p5.1)\.
- \[11\]S\. Amaran, N\. V\. Sahinidis, B\. Sharda, and S\. J\. Bury\(2016\)Simulation optimization: a review of algorithms and applications\.Annals of Operations Research240\(1\),pp\. 351–380\.External Links:[Document](https://dx.doi.org/10.1007/s10479-015-2019-x)Cited by:[§3\.7](https://arxiv.org/html/2609.02095#S3.SS7.p10.1)\.
- \[12\]C\. Audet and W\. Hare\(2017\)Derivative\-free and blackbox optimization\.Springer\.External Links:[Document](https://dx.doi.org/10.1007/978-3-319-68913-5)Cited by:[§3\.7](https://arxiv.org/html/2609.02095#S3.SS7.p10.1)\.
- \[13\]L\. Phan, A\. Gatti, Z\. Han, N\. Li, J\. Hu, H\. Zhang, S\. Shi, M\. Choi, A\. Agrawal, A\. Chopra, A\. Khoja, R\. Kim, R\. Ren, J\. Hausenloy, O\. Zhang, M\. Mazeika, S\. Yue, A\. Wang, D\. Hendrycks,et al\.\(2025\)Humanity’s Last Exam\.External Links:2501\.14249,[Link](https://arxiv.org/abs/2501.14249)Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p1.1),[Table 4](https://arxiv.org/html/2609.02095#S6.T4.3.4.1.1.1)\.
- \[14\]V\. Barres, H\. Dong, S\. Ray, X\. Si, and K\. Narasimhan\(2025\)τ2\\tau^\{2\}\-bench: evaluating conversational agents in a dual\-control environment\.External Links:2506\.07982,[Link](https://arxiv.org/abs/2506.07982)Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1),[Table 4](https://arxiv.org/html/2609.02095#S6.T4.3.7.1.1.1)\.
- \[15\]K\. Huang, A\. Prabhakar, O\. Thorat, D\. Agarwal, P\. K\. Choubey, Y\. Mao, S\. Savarese, C\. Xiong, and C\. Wu\(2026\)CRMArena\-Pro: holistic assessment of LLM agents across diverse business scenarios and interactions\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=EPlpe3Fx1x)Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1),[Table 4](https://arxiv.org/html/2609.02095#S6.T4.3.8.1.1.1)\.
- \[16\]H\. Sun, S\. Zhang, L\. Niu, L\. Ren, H\. Xu, H\. Fu, F\. Zhao, C\. Yuan, and X\. Wang\(2025\)Collab\-overcooked: benchmarking and evaluating large language models as collaborative agents\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 4922–4951\.Cited by:[§6\.2](https://arxiv.org/html/2609.02095#S6.SS2.p1.1),[Table 4](https://arxiv.org/html/2609.02095#S6.T4.3.11.1.1.1)\.
- \[17\]H\. Chen, M\. Xiong, Y\. Lu, W\. Han, A\. Deng, Y\. He, J\. Wu, Y\. Li, Y\. Liu, and B\. Hooi\(2026\)Mlr\-bench: evaluating ai agents on open\-ended machine learning research\.Advances in Neural Information Processing Systems38\.Cited by:[§6\.2](https://arxiv.org/html/2609.02095#S6.SS2.p1.1),[Table 4](https://arxiv.org/html/2609.02095#S6.T4.3.11.1.1.1)\.
- \[18\]D\. Madras, T\. Pitassi, and R\. Zemel\(2018\)Predict responsibly: improving fairness and accuracy by learning to defer\.InAdvances in Neural Information Processing Systems 31 \(NeurIPS 2018\),External Links:[Link](https://proceedings.neurips.cc/paper/2018/hash/09d37c08f7b129e96277388757530c72-Abstract.html)Cited by:[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p1.1),[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p2.1),[Table 4](https://arxiv.org/html/2609.02095#S6.T4.3.13.1.1.1)\.
- \[19\]H\. Mozannar and D\. Sontag\(2020\)Consistent estimators for learning to defer to an expert\.InProceedings of the 37th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.119,pp\. 7076–7087\.External Links:[Link](https://proceedings.mlr.press/v119/mozannar20b.html)Cited by:[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p1.1),[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p2.1),[Table 4](https://arxiv.org/html/2609.02095#S6.T4.3.13.1.1.1)\.
- \[20\]S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Hatfield\-Dodds, N\. DasSarma, E\. Tran\-Johnson, S\. Johnston, S\. El\-Showk, A\. Jones, N\. Elhage, T\. Hume, A\. Chen, Y\. Bai, S\. Bowman, S\. Fort, D\. Ganguli, D\. Hernandez, J\. Jacobson, J\. Kernion, S\. Kravec, L\. Lovitt, K\. Ndousse, C\. Olsson, S\. Ringer, D\. Amodei, T\. Brown, J\. Clark, N\. Joseph, B\. Mann, S\. McCandlish, C\. Olah, and J\. Kaplan\(2022\)Language models \(mostly\) know what they know\.External Links:2207\.05221,[Link](https://arxiv.org/abs/2207.05221)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p1.1),[Table 4](https://arxiv.org/html/2609.02095#S6.T4.3.15.1.1.1)\.
- \[21\]L\. Kuhn, Y\. Gal, and S\. Farquhar\(2023\)Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation\.InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023,External Links:[Link](https://openreview.net/forum?id=VD-AYtP0dve)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p1.1),[Table 4](https://arxiv.org/html/2609.02095#S6.T4.3.15.1.1.1)\.
- \[22\]M\. Vaccaro, A\. Almaatouq, and T\. W\. Malone\(2024\)When combinations of humans and ai are useful: a systematic review and meta\-analysis\.Nature Human Behaviour8,pp\. 2293–2303\.External Links:[Document](https://dx.doi.org/10.1038/s41562-024-02024-1)Cited by:[§6\.6](https://arxiv.org/html/2609.02095#S6.SS6.p1.1),[Table 4](https://arxiv.org/html/2609.02095#S6.T4.3.19.1.1.1)\.
- \[23\]M\. Mitchell, S\. Wu, A\. Zaldivar, P\. Barnes, L\. Vasserman, B\. Hutchinson, E\. Spitzer, I\. D\. Raji, and T\. Gebru\(2019\)Model cards for model reporting\.InProceedings of the Conference on Fairness, Accountability, and Transparency \(FAT\*\),pp\. 220–229\.External Links:[Document](https://dx.doi.org/10.1145/3287560.3287596),1810\.03993Cited by:[§6\.6](https://arxiv.org/html/2609.02095#S6.SS6.p1.1),[Table 4](https://arxiv.org/html/2609.02095#S6.T4.3.19.1.1.1)\.
- \[24\]Y\. Hua, H\. Na, C\. Ayubcha, and L\. Lian\(2026\)Design and report benchmarks for knowledge work\.External Links:2605\.23262,[Link](https://arxiv.org/abs/2605.23262)Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p1.1)\.
- \[25\]G\. Mialon, C\. Fourrier, T\. Wolf, Y\. LeCun, and T\. Scialom\(2024\)GAIA: a benchmark for general AI assistants\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=fibxvahvs3)Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[26\]S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. Neubig\(2024\)WebArena: a realistic web environment for building autonomous agents\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=oKn9c6ytLx)Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[27\]J\. Y\. Koh, R\. Lo, L\. Jang, V\. Duvvur, M\. C\. Lim, P\. Huang, G\. Neubig, S\. Zhou, R\. Salakhutdinov, and D\. Fried\(2024\)VisualWebArena: evaluating multimodal agents on realistic visual web tasks\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 881–905\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.50),[Link](https://aclanthology.org/2024.acl-long.50/)Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[28\]T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng, D\. Shin, F\. Lei, Y\. Liu, Y\. Xu, S\. Zhou, S\. Savarese, C\. Xiong, V\. Zhong, and T\. Yu\(2024\)OSWorld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.InAdvances in Neural Information Processing Systems 37 \(NeurIPS 2024\), Datasets and Benchmarks Track,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper_files/paper/2024/hash/5d413e48f84dc61244b6be550f1cd8f5-Abstract-Datasets_and_Benchmarks_Track.html)Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[29\]A\. Drouin, M\. Gasse, M\. Caccia, I\. H\. Laradji, M\. Del Verme, T\. Marty, D\. Vazquez, N\. Chapados, and A\. Lacoste\(2024\)WorkArena: how capable are web agents at solving common knowledge work tasks?\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 11642–11662\.External Links:[Link](https://proceedings.mlr.press/v235/drouin24a.html)Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[30\]K\. Huang, A\. Prabhakar, S\. Dhawan, Y\. Mao, H\. Wang, S\. Savarese, C\. Xiong, P\. Laban, and C\. Wu\(2025\)CRMArena: understanding the capacity of LLM agents to perform professional CRM tasks in realistic environments\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\), NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 3830–3850\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.194),[Link](https://aclanthology.org/2025.naacl-long.194/)Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[31\]M\. A\. Merrill, A\. G\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi, J\. Y\. Shin, T\. Walshe, E\. K\. Buchanan,et al\.\(2026\)Terminal\-Bench: benchmarking agents on hard, realistic tasks in command line interfaces\.arXiv preprint arXiv:2601\.11868\.External Links:2601\.11868Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[32\]J\. Wei, Z\. Sun, S\. Papay, S\. McKinney, J\. Han, I\. Fulford, H\. W\. Chung, A\. T\. Passos, W\. Fedus, and A\. Glaese\(2025\)BrowseComp: a simple yet challenging benchmark for browsing agents\.External Links:2504\.12516,[Link](https://arxiv.org/abs/2504.12516)Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[33\]M\. Du, B\. Xu, C\. Zhu, X\. Wang, and Z\. Mao\(2025\)DeepResearch Bench: a comprehensive benchmark for deep research agents\.External Links:2506\.11763,[Link](https://arxiv.org/abs/2506.11763)Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[34\]A\. Abaskohi, T\. Chen, M\. Muñoz\-Mármol, C\. Fox, A\. V\. Ramesh, É\. Marcotte,et al\.\(2025\)DRBench: a realistic benchmark for enterprise deep research\.External Links:2510\.00172,[Link](https://arxiv.org/abs/2510.00172)Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[35\]A\. Yehudai, L\. Eden, A\. Li, G\. Uziel, Y\. Zhao, R\. Bar\-Haim, A\. Cohan, and M\. Shmueli\-Scheuer\(2025\)Survey on evaluation of llm\-based agents\.InFindings of the Association for Computational Linguistics: ACL 2025,External Links:2503\.16416Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[36\]M\. Mohammadi, Y\. Li, J\. Lo, and W\. Yip\(2025\)Evaluation and benchmarking of llm agents: a survey\.arXiv preprint arXiv:2507\.21504\.External Links:2507\.21504Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[37\]C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan\(2024\)SWE\-bench: can language models resolve real\-world github issues?\.InInternational Conference on Learning Representations \(ICLR\),External Links:2310\.06770Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[38\]F\. F\. Xu, Y\. Song, B\. Li, Y\. Tang, K\. Jain, M\. Bao, Z\. Z\. Wang, X\. Zhou, Z\. Guo, M\. Cao, M\. Yang, H\. Y\. Lu, A\. Martin, Z\. Su, L\. Maben, R\. Mehta, W\. Chi, L\. Jang, Y\. Xie, S\. Zhou, and G\. Neubig\(2024\)TheAgentCompany: benchmarking llm agents on consequential real world tasks\.arXiv preprint arXiv:2412\.14161\.External Links:2412\.14161Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[39\]O\. Styles, S\. Miller, P\. Cerda\-Mardini, T\. Guha, V\. Sanchez, and B\. Vidgen\(2024\)WorkBench: a benchmark dataset for agents in a realistic workplace setting\.arXiv preprint arXiv:2405\.00823\.External Links:2405\.00823Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[40\]H\. Trivedi, T\. Khot, M\. Hartmann, R\. Manku, V\. Dong, E\. Li, S\. Gupta, A\. Sabharwal, and N\. Balasubramanian\(2024\)AppWorld: a controllable world of apps and people for benchmarking interactive coding agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:2407\.18901Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[41\]F\. Lei, J\. Chen, Y\. Ye, R\. Cao, D\. Shin, H\. Su, Z\. Suo, H\. Gao, W\. Hu, P\. Yin, V\. Zhong, C\. Xiong, R\. Sun, Q\. Liu, S\. Wang, and T\. Yu\(2025\)Spider 2\.0: evaluating language models on real\-world enterprise text\-to\-sql workflows\.InInternational Conference on Learning Representations \(ICLR\),External Links:2411\.07763Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[42\]X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang, S\. Zhang, X\. Deng, A\. Zeng, Z\. Du, C\. Zhang, S\. Shen, T\. Zhang, Y\. Su, H\. Sun, M\. Huang, Y\. Dong, and J\. Tang\(2024\)AgentBench: evaluating llms as agents\.InInternational Conference on Learning Representations \(ICLR\),External Links:2308\.03688Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[43\]M\. Andriushchenko, A\. Souly, M\. Dziemian, D\. Duenas, M\. Lin, J\. Wang, D\. Hendrycks, A\. Zou, J\. Z\. Kolter, M\. Fredrikson, Y\. Gal, and X\. Davies\(2025\)AgentHarm: a benchmark for measuring harmfulness of LLM agents\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=AC5n7xHuR1)Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[44\]I\. Levy, B\. Wiesel, S\. Marreed, A\. Oved, A\. Yaeli, N\. Mashkif, and S\. Shlomov\(2026\)ST\-WebAgentBench: a benchmark for evaluating safety and trustworthiness in web agents\.InThe Fourteenth International Conference on Learning Representations, ICLR 2026,External Links:[Link](https://openreview.net/forum?id=MuCDzH0ctf)Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[45\]Z\. Zhang, S\. Cui, Y\. Lu, J\. Zhou, J\. Yang, H\. Wang, and M\. Huang\(2024\)Agent\-SafetyBench: evaluating the safety of LLM agents\.External Links:2412\.14470,[Link](https://arxiv.org/abs/2412.14470)Cited by:[§6\.1](https://arxiv.org/html/2609.02095#S6.SS1.p2.1)\.
- \[46\]G\. Geigle, R\. Timofte, and G\. Glavaš\(2024\)Does object grounding really reduce hallucination of large vision\-language models?\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 2728–2742\.Cited by:[§6\.2](https://arxiv.org/html/2609.02095#S6.SS2.p1.1)\.
- \[47\]D\. Janiak, J\. Binkowski, A\. Sawczyn, B\. Gabrys, R\. Shwartz\-Ziv, and T\. J\. Kajdanowicz\(2025\)The illusion of progress: re\-evaluating hallucination detection in llms\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 34728–34745\.Cited by:[§6\.2](https://arxiv.org/html/2609.02095#S6.SS2.p1.1)\.
- \[48\]P\. Manakul, A\. Liusie, and M\. J\. F\. Gales\(2023\)SelfCheckGPT: zero\-resource black\-box hallucination detection for generative large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 9004–9017\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.557),[Link](https://aclanthology.org/2023.emnlp-main.557/)Cited by:[§6\.2](https://arxiv.org/html/2609.02095#S6.SS2.p1.1)\.
- \[49\]J\. Binkowski, D\. Janiak, A\. Sawczyn, B\. Gabrys, and T\. J\. Kajdanowicz\(2025\)Hallucination detection in llms using spectral features of attention maps\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 24354–24385\.Cited by:[§6\.2](https://arxiv.org/html/2609.02095#S6.SS2.p1.1)\.
- \[50\]H\. Orgad, M\. Toker, Z\. Gekhman, R\. Reichart, I\. Szpektor, H\. Kotek, and Y\. Belinkov\(2025\)LLMs know more than they show: on the intrinsic representation of LLM hallucinations\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=KRnsX5Em3W)Cited by:[§6\.2](https://arxiv.org/html/2609.02095#S6.SS2.p1.1)\.
- \[51\]T\. Lanham, A\. Chen, A\. Radhakrishnan, B\. Steiner, C\. Denison, D\. Hernandez, D\. Li, E\. Durmus, E\. Hubinger, J\. Kernion, K\. Lukošiūtė, K\. Nguyen, N\. Cheng, N\. Joseph, N\. Schiefer, O\. Rausch, R\. Larson, S\. McCandlish, S\. Kundu, S\. Kadavath, S\. Yang, T\. Henighan, T\. Maxwell, T\. Telleen\-Lawton, T\. Hume, Z\. Hatfield\-Dodds, J\. Kaplan, J\. Brauner, S\. R\. Bowman, and E\. Perez\(2023\)Measuring faithfulness in chain\-of\-thought reasoning\.External Links:2307\.13702,[Link](https://arxiv.org/abs/2307.13702)Cited by:[§6\.2](https://arxiv.org/html/2609.02095#S6.SS2.p1.1)\.
- \[52\]M\. Turpin, J\. Michael, E\. Perez, and S\. R\. Bowman\(2023\)Language models don’t always say what they think: unfaithful explanations in chain\-of\-thought prompting\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS 2023\),A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper_files/paper/2023/hash/ed3fea9033a80fea1376299fa7863f4a-Abstract-Conference.html)Cited by:[§6\.2](https://arxiv.org/html/2609.02095#S6.SS2.p1.1)\.
- \[53\]T\. Kehkashan, M\. Abdullah, A\. S\. Al\-Shamayleh, N\. Ivković, N\. A\. Ismail, S\. S\. S\. Ahmad, A\. Rehman, and A\. Akhunzada\(2026\)From benchmarks to deployment: a comprehensive review of agentic ai evaluation\.Artificial intelligence review\.Cited by:[§6\.2](https://arxiv.org/html/2609.02095#S6.SS2.p1.1)\.
- \[54\]C\. K\. Chow\(1970\)On optimum recognition error and reject tradeoff\.IEEE Transactions on Information Theory16\(1\),pp\. 41–46\.External Links:[Document](https://dx.doi.org/10.1109/TIT.1970.1054406)Cited by:[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p1.1)\.
- \[55\]R\. El\-Yaniv and Y\. Wiener\(2010\)On the foundations of noise\-free selective classification\.Journal of Machine Learning Research11,pp\. 1605–1641\.External Links:[Link](https://jmlr.org/papers/v11/el-yaniv10a.html)Cited by:[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p1.1),[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p2.1)\.
- \[56\]Y\. Geifman and R\. El\-Yaniv\(2017\)Selective classification for deep neural networks\.InAdvances in Neural Information Processing Systems 30 \(NeurIPS 2017\),External Links:1705\.08500,[Link](https://arxiv.org/abs/1705.08500)Cited by:[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p1.1),[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p2.1)\.
- \[57\]K\. Hendrickx, L\. Perini, D\. Van der Plas, W\. Meert, and J\. Davis\(2024\)Machine learning with a reject option: a survey\.Machine Learning113,pp\. 3073–3110\.External Links:2107\.11277Cited by:[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p1.1)\.
- \[58\]B\. Wen, J\. Yao, S\. Feng, C\. Xu, Y\. Tsvetkov, B\. Howe, and L\. L\. Wang\(2025\)Know your limits: a survey of abstention in large language models\.Transactions of the Association for Computational Linguistics \(TACL\)13,pp\. 529–556\.External Links:2407\.18418Cited by:[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p1.1)\.
- \[59\]V\. Franc, D\. Průša, and V\. Voráček\(2023\)Optimal strategies for reject option classifiers\.Journal of Machine Learning Research24\(11\),pp\. 1–49\.External Links:2101\.12523Cited by:[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p1.1)\.
- \[60\]R\. Verma and E\. Nalisnick\(2022\)Calibrated learning to defer with one\-vs\-all classifiers\.InInternational Conference on Machine Learning \(ICML\),External Links:2202\.03673Cited by:[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p1.1)\.
- \[61\]H\. Mozannar, H\. Lang, D\. Wei, P\. Sattigeri, S\. Das, and D\. Sontag\(2023\)Who should predict? exact algorithms for learning to defer to humans\.InInternational Conference on Artificial Intelligence and Statistics \(AISTATS\),External Links:2301\.06197Cited by:[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p1.1)\.
- \[62\]M\. Charusaie, H\. Mozannar, D\. Sontag, and S\. Samadi\(2022\)Sample efficient learning of predictors that complement humans\.InInternational Conference on Machine Learning \(ICML\),External Links:2207\.09584Cited by:[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p1.1),[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p3.2)\.
- \[63\]Z\. Zhang, C\. Nguyen, K\. Wells, T\. Do, D\. Rosewarne, and G\. Carneiro\(2024\)Coverage\-constrained human\-ai cooperation with multiple experts\.arXiv preprint arXiv:2411\.11976\.External Links:2411\.11976Cited by:[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p1.1),[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p3.2)\.
- \[64\]A\. N\. Angelopoulos and S\. Bates\(2021\)A gentle introduction to conformal prediction and distribution\-free uncertainty quantification\.arXiv preprint arXiv:2107\.07511\.External Links:2107\.07511Cited by:[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p1.1)\.
- \[65\]M\. M\. Campos, A\. Farinhas, C\. Zerva, M\. A\. T\. Figueiredo, and A\. F\. T\. Martins\(2024\)Conformal prediction for natural language processing: a survey\.Transactions of the Association for Computational Linguistics \(TACL\)12,pp\. 1497–1516\.External Links:2405\.01976Cited by:[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p1.1)\.
- \[66\]P\. Kirichenko, M\. Ibrahim, K\. Chaudhuri, and S\. J\. Bell\(2025\)AbstentionBench: reasoning LLMs fail on unanswerable questions\.InAdvances in Neural Information Processing Systems 38 \(NeurIPS 2025\), Datasets and Benchmarks Track,D\. Belgrave, C\. Zhang, L\. N\. Montoya, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, N\. Chen, I\. V\. M\. Ruíz, and A\. Loaiza\-Bonilla \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper_files/paper/2025/hash/fb122bfc3f0127a94ded048b5b03496f-Abstract-Datasets_and_Benchmarks_Track.html)Cited by:[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p2.1)\.
- \[67\]B\. Kumar, C\. Lu, G\. Gupta, A\. Palepu, D\. Bellamy, R\. Raskar, and A\. Beam\(2023\)Conformal prediction with large language models for multi\-choice question answering\.External Links:2305\.18404,[Link](https://arxiv.org/abs/2305.18404)Cited by:[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p2.1)\.
- \[68\]J\. Su, J\. Luo, H\. Wang, and L\. Cheng\(2024\)API is enough: conformal prediction for large language models without logit\-access\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 979–995\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.54),[Link](https://aclanthology.org/2024.findings-emnlp.54/)Cited by:[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p2.1)\.
- \[69\]A\. Z\. Ren, A\. Dixit, A\. Bodrova, S\. Singh, S\. Tu, N\. Brown, P\. Xu, L\. Takayama, F\. Xia, J\. Varley, Z\. Xu, D\. Sadigh, A\. Zeng, and A\. Majumdar\(2023\)Robots that ask for help: uncertainty alignment for large language model planners\.InProceedings of the 7th Conference on Robot Learning \(CoRL\),J\. Tan, M\. Toussaint, and K\. Darvish \(Eds\.\),Proceedings of Machine Learning Research, Vol\.229,pp\. 661–682\.External Links:[Link](https://proceedings.mlr.press/v229/ren23a.html)Cited by:[§6\.3](https://arxiv.org/html/2609.02095#S6.SS3.p2.1)\.
- \[70\]J\. Ren, Y\. Zhao, T\. Vu, P\. J\. Liu, and B\. Lakshminarayanan\(2023\)Self\-evaluation improves selective generation in large language models\.InProceedings on “I Can’t Believe It’s Not Better: Failure Modes in the Age of Foundation Models” at NeurIPS 2023 Workshops,J\. Antorán, A\. Blaas, K\. Buchanan, F\. Feng, V\. Fortuin, S\. Ghalebikesabi, A\. Kriegler, I\. Mason, D\. Rohde, F\. J\. R\. Ruiz, T\. Uelwer, Y\. Xie, and R\. Yang \(Eds\.\),Proceedings of Machine Learning Research, Vol\.239,pp\. 49–64\.Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p1.1)\.
- \[71\]S\. Lin, J\. Hilton, and O\. Evans\(2022\)Teaching models to express their uncertainty in words\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=8s8K2UZGTZ)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p1.1)\.
- \[72\]K\. Tian, E\. Mitchell, A\. Zhou, A\. Sharma, R\. Rafailov, H\. Yao, C\. Finn, and C\. D\. Manning\(2023\)Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 5433–5442\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.330),[Link](https://aclanthology.org/2023.emnlp-main.330/)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p1.1),[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p2.1)\.
- \[73\]Z\. Ji, L\. Yu, Y\. Koishekenov, Y\. Bang, A\. Hartshorn, A\. Schelten, C\. Zhang, P\. Fung, and N\. Cancedda\(2025\)Calibrating verbal uncertainty as a linear feature to reduce hallucinations\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 3769–3793\.Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p1.1)\.
- \[74\]G\. K\. Liu, G\. Yona, A\. Caciularu, I\. Szpektor, T\. G\. Rudner, and A\. Cohan\(2025\)Metafaith: faithful natural language uncertainty expression in llms\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 29600–29644\.Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p1.1)\.
- \[75\]V\. Shrivastava, A\. Kumar, and P\. Liang\(2025\)Language models prefer what they know: relative confidence estimation via confidence preferences\.External Links:2502\.01126,[Link](https://arxiv.org/abs/2502.01126)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p1.1)\.
- \[76\]A\. Taubenfeld, T\. Sheffer, E\. Ofek, A\. Feder, A\. Goldstein, Z\. Gekhman, and G\. Yona\(2025\)Confidence improves self\-consistency in LLMs\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 20090–20111\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1030),[Link](https://aclanthology.org/2025.findings-acl.1030/)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p1.1)\.
- \[77\]B\. Hou, Y\. Liu, K\. Qian, J\. Andreas, S\. Chang, and Y\. Zhang\(2024\)Decomposing uncertainty for large language models through input clarification ensembling\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 19023–19042\.Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p1.1)\.
- \[78\]R\. Niu, D\. Wu, R\. Yu, and Y\. Ma\(2024\)Functional\-level uncertainty quantification for calibrated fine\-tuning on llms\.External Links:2410\.06431,[Link](https://arxiv.org/abs/2410.06431)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p1.1)\.
- \[79\]M\. D\. Ma, Y\. Ding, Z\. Huang, J\. Gao, Y\. Sun, and W\. Wang\(2025\)Inferring from logits: exploring best practices for decoding\-free generative candidate selection\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),pp\. 33125–33144\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1589),[Link](https://aclanthology.org/2025.acl-long.1589/)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p1.1)\.
- \[80\]C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger\(2017\)On calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning \(ICML\),D\. Precup and Y\. W\. Teh \(Eds\.\),Proceedings of Machine Learning Research, Vol\.70,pp\. 1321–1330\.External Links:[Link](http://proceedings.mlr.press/v70/guo17a.html)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p2.1)\.
- \[81\]J\. Xie, A\. S\. Chen, Y\. Lee, E\. Mitchell, and C\. Finn\(2024\)Calibrating language models with adaptive temperature scaling\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 18128–18138\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1007),[Link](https://aclanthology.org/2024.emnlp-main.1007/)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p2.1)\.
- \[82\]Y\. Li, S\. Wang, L\. Huang, and L\. Liu\(2025\)Graph\-based confidence calibration for large language models\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=BDPvuD5FTg)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p2.1)\.
- \[83\]M\. Braverman, X\. Chen, S\. M\. Kakade, K\. Narasimhan, C\. Zhang, and Y\. Zhang\(2020\)Calibration, entropy rates, and memory in language models\.InProceedings of the 37th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.119,pp\. 1089–1099\.External Links:[Link](http://proceedings.mlr.press/v119/braverman20a.html)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p2.1)\.
- \[84\]Z\. Jiang, J\. Araki, H\. Ding, and G\. Neubig\(2021\)How can we know when language models know? on the calibration of language models for question answering\.Transactions of the Association for Computational Linguistics9,pp\. 962–977\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00407),[Link](https://aclanthology.org/2021.tacl-1.57/)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p2.1)\.
- \[85\]Y\. Chen, L\. Yuan, G\. Cui, Z\. Liu, and H\. Ji\(2023\)A close look into the calibration of pre\-trained language models\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 1343–1367\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.75),[Link](https://aclanthology.org/2023.acl-long.75/)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p2.1)\.
- \[86\]S\. Kapoor, N\. Gruver, M\. Roberts, A\. Pal, S\. Dooley, M\. Goldblum, and A\. G\. Wilson\(2024\)Calibration\-tuning: teaching large language models to know what they don’t know\.InProceedings of the 1st Workshop on Uncertainty\-Aware NLP \(UncertaiNLP 2024\),R\. Vázquez, H\. Celikkanat, D\. Ulmer, J\. Tiedemann, S\. Swayamdipta, W\. Aziz, B\. Plank, J\. Baan, and M\. de Marneffe \(Eds\.\),St Julians, Malta,pp\. 1–14\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.uncertainlp-1.1),[Link](https://aclanthology.org/2024.uncertainlp-1.1/)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p2.1)\.
- \[87\]E\. Stengel\-Eskin, P\. Hase, and M\. Bansal\(2024\)LACIE: listener\-aware finetuning for calibration in large language models\.InAdvances in Neural Information Processing Systems 37 \(NeurIPS 2024\),A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 43080–43106\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/4b8eaf3bcdc105423a972ed90eb07217-Abstract-Conference.html)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p2.1)\.
- \[88\]Y\. Zhao, M\. Khalman, R\. Joshi, S\. Narayan, M\. Saleh, and P\. J\. Liu\(2023\)Calibrating sequence likelihood improves conditional language generation\.InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023,External Links:[Link](https://openreview.net/forum?id=0qSOodKmJaN)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p2.1)\.
- \[89\]Y\. Zhao, R\. Joshi, T\. Liu, M\. Khalman, M\. Saleh, and P\. J\. Liu\(2023\)SLiC\-HF: sequence likelihood calibration with human feedback\.External Links:2305\.10425,[Link](https://arxiv.org/abs/2305.10425)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p2.1)\.
- \[90\]Z\. Shi, S\. Land, A\. Locatelli, M\. Geist, and M\. Bartolo\(2024\)Understanding likelihood over\-optimisation in direct alignment algorithms\.External Links:2410\.11677,[Link](https://arxiv.org/abs/2410.11677)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p2.1)\.
- \[91\]J\. Geng, F\. Cai, Y\. Wang, H\. Koeppl, P\. Nakov, and I\. Gurevych\(2024\)A survey of confidence estimation and calibration in large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\),External Links:2311\.08298Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p3.1)\.
- \[92\]O\. Shorinwa, Z\. Mei, J\. Lidard, A\. Z\. Ren, and A\. Majumdar\(2024\)A survey on uncertainty quantification of large language models: taxonomy, open research challenges, and future directions\.arXiv preprint arXiv:2412\.05563\.External Links:2412\.05563Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p3.1)\.
- \[93\]Z\. Ji, N\. Lee, R\. Frieske, T\. Yu, D\. Su, Y\. Xu, E\. Ishii, Y\. Bang, D\. Chen, W\. Dai, H\. S\. Chan, A\. Madotto, and P\. Fung\(2023\)Survey of hallucination in natural language generation\.ACM Computing Surveys55\(12\),pp\. 1–38\.External Links:[Document](https://dx.doi.org/10.1145/3571730),2202\.03629Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p3.1)\.
- \[94\]L\. Huang, W\. Yu, W\. Ma, W\. Zhong, Z\. Feng, H\. Wang, Q\. Chen, W\. Peng, X\. Feng, B\. Qin, and T\. Liu\(2023\)A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions\.arXiv preprint arXiv:2311\.05232\.External Links:2311\.05232Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p3.1)\.
- \[95\]S\. Farquhar, J\. Kossen, L\. Kuhn, and Y\. Gal\(2024\)Detecting hallucinations in large language models using semantic entropy\.Nature630\(8017\),pp\. 625–630\.External Links:[Document](https://dx.doi.org/10.1038/s41586-024-07421-0)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p3.1)\.
- \[96\]M\. Xiong, Z\. Hu, X\. Lu, Y\. Li, J\. Fu, J\. He, and B\. Hooi\(2024\)Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms\.InInternational Conference on Learning Representations \(ICLR\),External Links:2306\.13063Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p3.1)\.
- \[97\]C\. Chen, K\. Liu, Z\. Chen, Y\. Gu, Y\. Wu, M\. Tao, Z\. Fu, and J\. Ye\(2024\)INSIDE: llms’ internal states retain the power of hallucination detection\.InInternational Conference on Learning Representations \(ICLR\),External Links:2402\.03744Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p3.1)\.
- \[98\]J\. Duan, H\. Cheng, S\. Wang, A\. Zavalny, C\. Wang, R\. Xu, B\. Kailkhura, and K\. Xu\(2024\)Shifting attention to relevance: towards the predictive uncertainty quantification of free\-form large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:2307\.01379Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p3.1)\.
- \[99\]Y\. F\. Bakman, D\. N\. Yaldiz, B\. Buyukates, C\. Tao, D\. Dimitriadis, and S\. Avestimehr\(2024\)MARS: meaning\-aware response scoring for uncertainty estimation in generative llms\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:2402\.11756Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p3.1)\.
- \[100\]R\. Vashurin, E\. Fadeeva, A\. Vazhentsev, L\. Rvanova, A\. Tsvigun, D\. Vasilev, R\. Xing, A\. B\. Sadallah, K\. Grishchenkov, S\. Petrakov, A\. Panchenko, T\. Baldwin, P\. Nakov, M\. Panov, and A\. Shelmanov\(2025\)Benchmarking uncertainty quantification methods for large language models with lm\-polygraph\.Transactions of the Association for Computational Linguistics \(TACL\)13,pp\. 220–248\.External Links:2406\.15627Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p3.1)\.
- \[101\]J\. Ren, J\. Luo, Y\. Zhao, K\. Krishna, M\. Saleh, B\. Lakshminarayanan, and P\. J\. Liu\(2023\)Out\-of\-distribution detection and selective generation for conditional language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:2209\.15558Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p3.1)\.
- \[102\]V\. Quach, A\. Fisch, T\. Schuster, A\. Yala, J\. H\. Sohn, T\. S\. Jaakkola, and R\. Barzilay\(2024\)Conformal language modeling\.InInternational Conference on Learning Representations \(ICLR\),External Links:2306\.10193Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p3.1)\.
- \[103\]M\. P\. Naeini, G\. F\. Cooper, and M\. Hauskrecht\(2015\)Obtaining well calibrated probabilities using bayesian binning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.29,pp\. 2901–2907\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v29i1.9602)Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p3.1)\.
- \[104\]J\. Nixon, M\. W\. Dusenberry, G\. Jerfel, T\. Nguyen, J\. Liu, L\. Zhang, and D\. Tran\(2019\)Measuring calibration in deep learning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\) Workshops,External Links:1904\.01685Cited by:[§6\.4](https://arxiv.org/html/2609.02095#S6.SS4.p3.1)\.
- \[105\]A\. Farinhas, N\. M\. Guerreiro, S\. Agrawal, R\. Rei, and A\. F\. Martins\(2025\)Translate smart, not hard: cascaded translation systems with quality\-aware deferral\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 26730–26744\.Cited by:[§6\.5](https://arxiv.org/html/2609.02095#S6.SS5.p1.1)\.
- \[106\]J\. Dekoninck, M\. Baader, and M\. Vechev\(2025\)A unified approach to routing and cascading for LLMs\.InProceedings of the 42nd International Conference on Machine Learning,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267,pp\. 12987–13010\.External Links:[Link](https://proceedings.mlr.press/v267/dekoninck25a.html)Cited by:[§6\.5](https://arxiv.org/html/2609.02095#S6.SS5.p1.1)\.
- \[107\]Y\. Shen, Y\. Liu, Z\. Huang, R\. Yin, X\. Zheng, and X\. Huang\(2025\)Sater: a self\-aware and token\-efficient approach to routing and cascading\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 10526–10540\.Cited by:[§6\.5](https://arxiv.org/html/2609.02095#S6.SS5.p1.1)\.
- \[108\]P\. Aggarwal, A\. Madaan, A\. Anand, S\. P\. Potharaju, S\. Mishra, P\. Zhou, A\. Gupta, D\. Rajagopal, K\. Kappaganthu, Y\. Yang, S\. Upadhyay, M\. Faruqui, and Mausam\(2024\)AutoMix: automatically mixing language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2310\.12963Cited by:[§6\.5](https://arxiv.org/html/2609.02095#S6.SS5.p1.1)\.
- \[109\]M\. Yue, J\. Zhao, M\. Zhang, L\. Du, and Z\. Yao\(2024\)Large language model cascades with mixture of thoughts representations for cost\-efficient reasoning\.InInternational Conference on Learning Representations \(ICLR\),External Links:2310\.03094Cited by:[§6\.5](https://arxiv.org/html/2609.02095#S6.SS5.p1.1)\.
- \[110\]J\. Zhang, R\. Krishna, A\. H\. Awadallah, and C\. Wang\(2023\)EcoAssistant: using llm assistant more affordably and accurately\.External Links:2310\.03046Cited by:[§6\.5](https://arxiv.org/html/2609.02095#S6.SS5.p1.1)\.
- \[111\]D\. Ding, A\. Mallick, C\. Wang, R\. Sim, S\. Mukherjee, V\. Rühle, L\. V\.S\. Lakshmanan, and A\. H\. Awadallah\(2024\)Hybrid llm: cost\-efficient and quality\-aware query routing\.InInternational Conference on Learning Representations \(ICLR\),External Links:2404\.14618Cited by:[§6\.5](https://arxiv.org/html/2609.02095#S6.SS5.p1.1)\.
- \[112\]K\. Lu, H\. Yuan, R\. Lin, J\. Lin, Z\. Yuan, C\. Zhou, and J\. Zhou\(2024\)Routing to the expert: efficient reward\-guided ensemble of large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\-HLT\),External Links:2311\.08692Cited by:[§6\.5](https://arxiv.org/html/2609.02095#S6.SS5.p1.1)\.
- \[113\]Q\. J\. Hu, J\. Bieker, X\. Li, N\. Jiang, B\. Keigwin, G\. Ranganath, K\. Keutzer, and S\. K\. Upadhyay\(2024\)RouterBench: a benchmark for multi\-llm routing system\.External Links:2403\.12031Cited by:[§6\.5](https://arxiv.org/html/2609.02095#S6.SS5.p1.1)\.
- \[114\]C\. Varangot\-Reille, C\. Bouvard, A\. Gourru, M\. Ciancone, M\. Schaeffer, and F\. Jacquenet\(2025\)Doing more with less: a survey on routing strategies for resource optimisation in large language model\-based systems\.Journal of Artificial Intelligence Research \(JAIR\)\.External Links:[Document](https://dx.doi.org/10.1613/jair.1.19801),2502\.00409Cited by:[§6\.5](https://arxiv.org/html/2609.02095#S6.SS5.p1.1)\.
- \[115\]P\. Viola and M\. Jones\(2001\)Rapid object detection using a boosted cascade of simple features\.InProceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.1,pp\. I–511–I–518\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2001.990517)Cited by:[§6\.5](https://arxiv.org/html/2609.02095#S6.SS5.p1.1)\.
- \[116\]D\. Amodei, C\. Olah, J\. Steinhardt, P\. Christiano, J\. Schulman, and D\. Mané\(2016\)Concrete problems in ai safety\.arXiv preprint arXiv:1606\.06565\.External Links:1606\.06565Cited by:[§6\.6](https://arxiv.org/html/2609.02095#S6.SS6.p1.1)\.
- \[117\]S\. R\. Bowman, J\. Hyun, E\. Perez, E\. Chen, C\. Pettit, S\. Heiner, K\. Lukošiūtė, A\. Askell, A\. Jones, A\. Chen,et al\.\(2022\)Measuring progress on scalable oversight for large language models\.arXiv preprint arXiv:2211\.03540\.External Links:2211\.03540Cited by:[§6\.6](https://arxiv.org/html/2609.02095#S6.SS6.p1.1)\.
- \[118\]C\. Burns, P\. Izmailov, J\. H\. Kirchner, B\. Baker, L\. Gao, L\. Aschenbrenner, Y\. Chen, A\. Ecoffet, M\. Joglekar, J\. Leike, I\. Sutskever, and J\. Wu\(2024\)Weak\-to\-strong generalization: eliciting strong capabilities with weak supervision\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),External Links:2312\.09390Cited by:[§6\.6](https://arxiv.org/html/2609.02095#S6.SS6.p1.1)\.
- \[119\]G\. Irving, P\. Christiano, and D\. Amodei\(2018\)AI safety via debate\.arXiv preprint arXiv:1805\.00899\.External Links:1805\.00899Cited by:[§6\.6](https://arxiv.org/html/2609.02095#S6.SS6.p1.1)\.
- \[120\]P\. Christiano, B\. Shlegeris, and D\. Amodei\(2018\)Supervising strong learners by amplifying weak experts\.arXiv preprint arXiv:1810\.08575\.External Links:1810\.08575Cited by:[§6\.6](https://arxiv.org/html/2609.02095#S6.SS6.p1.1)\.
- \[121\]H\. Lang, F\. Huang, and Y\. Li\(2025\)Debate helps weak\-to\-strong generalization\.InProceedings of the Thirty\-Ninth AAAI Conference on Artificial Intelligence and Thirty\-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence,pp\. 27410–27418\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v39i26.34952)Cited by:[§6\.6](https://arxiv.org/html/2609.02095#S6.SS6.p1.1)\.
- \[122\]L\. Zhu, Q\. Lu, M\. Ding, S\. U\. Lee, and C\. Wang\(2026\)Designing meaningful human oversight in ai\.AI and Ethics6\(3\),pp\. 286\.Cited by:[§6\.6](https://arxiv.org/html/2609.02095#S6.SS6.p1.1)\.
- \[123\]A\. Tsamados, L\. Floridi, and M\. Taddeo\(2025\)Human control of ai systems: from supervision to teaming\.AI and Ethics5\(2\),pp\. 1535–1548\.Cited by:[§6\.6](https://arxiv.org/html/2609.02095#S6.SS6.p1.1)\.
- \[124\]J\. Tang, C\. Chen, J\. Li, Z\. Zhang, B\. Guo, I\. Khalilov, S\. A\. Gebreegziabher, B\. Yao, D\. Wang, Y\. Ye,et al\.\(2026\)Dark patterns meet gui agents: llm agent susceptibility to manipulative interfaces and the role of human oversight\.InProceedings of the 2026 CHI Conference on Human Factors in Computing Systems,pp\. 1–26\.Cited by:[§6\.6](https://arxiv.org/html/2609.02095#S6.SS6.p1.1)\.
- \[125\]P\. Liu, J\. Zhang, S\. Chen, and S\. Chen\(2025\)Human\-ai teaming in healthcare: 1\+ 1\> 2?\.npj Artificial Intelligence1\(1\),pp\. 47\.Cited by:[§6\.6](https://arxiv.org/html/2609.02095#S6.SS6.p1.1)\.
- \[126\]G\. Bansal, T\. Wu, J\. Zhou, R\. Fok, B\. Nushi, E\. Kamar, M\. T\. Ribeiro, and D\. S\. Weld\(2021\)Does the whole exceed its parts? the effect of ai explanations on complementary team performance\.InProceedings of the 2021 CHI Conference on Human Factors in Computing Systems \(CHI\),External Links:2006\.14779Cited by:[§6\.6](https://arxiv.org/html/2609.02095#S6.SS6.p1.1)\.
- \[127\]Z\. Buçinca, M\. B\. Malaya, and K\. Z\. Gajos\(2021\)To trust or to think: cognitive forcing functions can reduce overreliance on ai in ai\-assisted decision\-making\.Proceedings of the ACM on Human\-Computer Interaction5\(CSCW1\),pp\. 1–21\.External Links:[Document](https://dx.doi.org/10.1145/3449287),2102\.09692Cited by:[§6\.6](https://arxiv.org/html/2609.02095#S6.SS6.p1.1)\.
- \[128\]V\. Lai, C\. Chen, Q\. V\. Liao, A\. Smith\-Renner, and C\. Tan\(2021\)Towards a science of human\-ai decision making: a survey of empirical studies\.arXiv preprint arXiv:2112\.11471\.External Links:2112\.11471Cited by:[§6\.6](https://arxiv.org/html/2609.02095#S6.SS6.p1.1)\.
- \[129\]B\. Green\(2022\)The flaws of policies requiring human oversight of government algorithms\.Computer Law & Security Review45,pp\. 105681\.External Links:[Document](https://dx.doi.org/10.1016/j.clsr.2022.105681)Cited by:[§6\.6](https://arxiv.org/html/2609.02095#S6.SS6.p1.1)\.
- \[130\]I\. D\. Raji, A\. Smart, R\. N\. White, M\. Mitchell, T\. Gebru, B\. Hutchinson, J\. Smith\-Loud, D\. Theron, and P\. Barnes\(2020\)Closing the ai accountability gap: defining an end\-to\-end framework for internal algorithmic auditing\.InProceedings of the 2020 Conference on Fairness, Accountability, and Transparency \(FAT\*\),pp\. 33–44\.External Links:[Document](https://dx.doi.org/10.1145/3351095.3372873),2001\.00973Cited by:[§6\.6](https://arxiv.org/html/2609.02095#S6.SS6.p1.1)\.
- \[131\]L\. Weidinger, M\. Rauh, N\. Marchal, A\. Manzini, L\. A\. Hendricks, J\. Mateos\-Garcia, S\. Bergman, J\. Kay, C\. Griffin, B\. Bariach, I\. Gabriel, V\. Rieser, and W\. Isaac\(2023\)Sociotechnical safety evaluation of generative ai systems\.arXiv preprint arXiv:2310\.11986\.External Links:2310\.11986Cited by:[§6\.6](https://arxiv.org/html/2609.02095#S6.SS6.p1.1)\.
- \[132\]Harbor Framework Team\(2026\)Harbor: a framework for evaluating and optimizing agents and models in container environments\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.20953922),[Link](https://doi.org/10.5281/zenodo.20953922)Cited by:[§6\.7](https://arxiv.org/html/2609.02095#S6.SS7.p1.1)\.
- \[133\]X\. Liu, J\. Tu, Y\. Chen, S\. Xie, S\. Ren, T\. Shi, G\. Gantar, E\. Sandoval, D\. Lee, D\. Miao,et al\.\(2026\)AgentBeats: agentifying agent assessment for openness, standardization, and reproducibility\.arXiv preprint arXiv:2606\.13608\.Cited by:[§6\.7](https://arxiv.org/html/2609.02095#S6.SS7.p1.1)\.

## Appendix AFormal definitions

### A\.1Threshold Routing and Monotonicity of the Routing Signal

Consider the trajectory\-invariant terminal\-oversight setting\. For task instanceii, autonomous execution produces an observable routing signalsi∈ℝs\_\{i\}\\in\\mathbb\{R\}and an evaluation outcomeui∈\{0,1\}u\_\{i\}\\in\\\{0,1\\\}, whereui=1u\_\{i\}=1denotes successful autonomous execution\. The outcomeuiu\_\{i\}is available to the evaluator during policy development and qualification, but is generally not known when the routing decision is made\.

A threshold policyπτ\\pi\_\{\\tau\}accepts the agent output autonomously whensi≥τs\_\{i\}\\geq\\tauand routes it to human review otherwise:

πτ​\(si\)=\{accept,si≥τ,review,si<τ\.\\pi\_\{\\tau\}\(s\_\{i\}\)=\\begin\{cases\}\\mathrm\{accept\},&s\_\{i\}\\geq\\tau,\\\\ \\mathrm\{review\},&s\_\{i\}<\\tau\.\\end\{cases\}\(47\)
For thresholdτ\\tau, define the autonomous coverage

c⁡\(τ\)=Pr⁡\(s≥τ\),c\(\\tau\)=\\Pr\(s\\geq\\tau\),\(48\)
and the success probability among autonomously accepted cases,

a⁡\(τ\)=Pr⁡\(u=1∣s≥τ\)\.a\(\\tau\)=\\Pr\(u=1\\mid s\\geq\\tau\)\.\(49\)
We callssa*monotone routing signal*if increasing the threshold does not decrease the success probability of the retained autonomous cases:

τ1<τ2⟹a⁡\(τ1\)≤a⁡\(τ2\),\\tau\_\{1\}<\\tau\_\{2\}\\quad\\Longrightarrow\\quad a\(\\tau\_\{1\}\)\\leq a\(\\tau\_\{2\}\),\(50\)
for thresholds with nonzero autonomous coverage\. Equivalently, increasingly selective autonomous acceptance should preferentially remove higher\-risk cases\.

A stronger sufficient condition is pointwise monotonicity,

s1<s2⟹Pr⁡\(u=1∣s=s1\)≤Pr⁡\(u=1∣s=s2\),s\_\{1\}<s\_\{2\}\\quad\\Longrightarrow\\quad\\Pr\(u=1\\mid s=s\_\{1\}\)\\leq\\Pr\(u=1\\mid s=s\_\{2\}\),\(51\)
but READY does not require this stronger condition\. In finite evaluation samples, local departures from monotonicity may occur; what matters operationally is whether the signal induces a sufficiently ordered risk–coverage relationship to support reliable selective routing\.

Importantly, monotonicity is distinct from calibration\. A routing signal may provide a useful ordering of cases even whensis\_\{i\}is not numerically equal to the probability of successful execution\. Calibration is required only when the numerical value of the signal is interpreted probabilistically; threshold\-based routing principally requires discrimination and ordering\.

### A\.2Tail\-Risk Estimation for Risk\-Sensitive Qualification

The main text treats workflow\-specific risk as an optional extension to the common READY reliability constraint\. This appendix gives one concrete implementation using conditional value\-at\-risk \(CVaR\)\. The construction is not required for workflows in which binary deployment success adequately captures the consequences of failure\.

Let

Liπ=Lw​\(Ziπ\)≥0L\_\{i\}^\{\\pi\}=L\_\{w\}\(Z\_\{i\}^\{\\pi\}\)\\geq 0\(52\)
denote the workflow\-specific loss incurred on qualification instanceiiunder a fixed oversight policyπ\\pi\. The loss function is defined as part of the workflow specification and may encode, for example, the severity of a failed action, downstream recovery cost, or another consequence not represented adequately by the binary success indicatoruiπu\_\{i\}^\{\\pi\}\.

For confidence levelα∈\(0,1\)\\alpha\\in\(0,1\), the value\-at\-risk is

VaRα⁡\(Lπ\)=inf\{ℓ:Pr⁡\(Lπ≤ℓ\)≥α\}\.\\operatorname\{VaR\}\_\{\\alpha\}\(L^\{\\pi\}\)=\\inf\\left\\\{\\ell:\\Pr\(L^\{\\pi\}\\leq\\ell\)\\geq\\alpha\\right\\\}\.\(53\)
CVaR measures the expected loss in the upper tail beyond this quantile\. For estimation, we use the equivalent optimization representation

CVaRα⁡\(Lπ\)=minη∈ℝ⁡\[η\+11−α​𝔼​\[\(Lπ−η\)\+\]\],\\operatorname\{CVaR\}\_\{\\alpha\}\(L^\{\\pi\}\)=\\min\_\{\\eta\\in\\mathbb\{R\}\}\\left\[\\eta\+\\frac\{1\}\{1\-\\alpha\}\\mathbb\{E\}\\left\[\(L^\{\\pi\}\-\\eta\)\_\{\+\}\\right\]\\right\],\(54\)
where\(x\)\+=max⁡\(x,0\)\(x\)\_\{\+\}=\\max\(x,0\)\. This form remains well defined for discrete empirical loss distributions and avoids requiring a separate estimator of the tail quantile\.

#### Held\-out estimator\.

After policy selection, the policyπ^\\widehat\{\\pi\}and the risk specification\(Lw,α,Bw\)\(L\_\{w\},\\alpha,B\_\{w\}\)are frozen before evaluation on the qualification set\. Given losses

L1π^,…,LNπ^,L\_\{1\}^\{\\widehat\{\\pi\}\},\\ldots,L\_\{N\}^\{\\widehat\{\\pi\}\},\(55\)
the empirical CVaR estimator is

CVaR^α=minη∈ℝ⁡\[η\+1\(1−α\)​N​∑i=1N\(Liπ^−η\)\+\]\.\\widehat\{\\operatorname\{CVaR\}\}\_\{\\alpha\}=\\min\_\{\\eta\\in\\mathbb\{R\}\}\\left\[\\eta\+\\frac\{1\}\{\(1\-\\alpha\)N\}\\sum\_\{i=1\}^\{N\}\\left\(L\_\{i\}^\{\\widehat\{\\pi\}\}\-\\eta\\right\)\_\{\+\}\\right\]\.\(56\)
For trajectory\-invariant policies, the losses may be computed by replaying the frozen policy over saved qualification trajectories\. For trajectory\-dependent policies, the losses must be obtained from qualification executions or simulations performed underπ^\\widehat\{\\pi\}itself\.

#### Statistical qualification\.

A point estimate of CVaR is not sufficient to establish a deployment\-risk constraint\. READY therefore constructs a one\-sided upper confidence bound

CVaRUCB1−δ⁡\(π^\)\\operatorname\{CVaR\}^\{\\,1\-\\delta\}\_\{\\mathrm\{UCB\}\}\\left\(\\widehat\{\\pi\}\\right\)\(57\)
at confidence level1−δ1\-\\deltaand qualifies the policy with respect to the tail\-risk constraint only if

CVaRUCB1−δ⁡\(π^\)≤Bw\.\\operatorname\{CVaR\}^\{\\,1\-\\delta\}\_\{\\mathrm\{UCB\}\}\\left\(\\widehat\{\\pi\}\\right\)\\leq B\_\{w\}\.\(58\)
In the reference procedure, this upper bound is obtained by bootstrap resampling the held\-out qualification set\. For each bootstrap replicateb=1,…,Bb=1,\\ldots,B, READY resamples qualification instances, recomputesCVaR^α\(b\)\\widehat\{\\operatorname\{CVaR\}\}\_\{\\alpha\}^\{\(b\)\}, and uses the appropriate upper percentile of the resulting bootstrap distribution as the one\-sided confidence bound\.

The bootstrap unit must match the unit of statistical generalization\. If multiple stochastic runs are collected for the same task instance, all runs from that instance are resampled together\. Likewise, if observations are clustered by patient, customer, account, or another natural unit, resampling is performed at that cluster level rather than treating individual observations as independent\.

#### Uncertain downstream outcomes\.

IfLiπL\_\{i\}^\{\\pi\}depends on an outcome that is not directly observed in the evaluation, for example, the success or cost of a hypothetical human\-review path, the corresponding uncertainty must be propagated into the risk estimate\. READY may do this by jointly sampling the uncertain review parameters and the qualification instances in each bootstrap replicate\. A risk quantity computed by substituting a single assumed review\-success rate or recovery cost should therefore be reported as conditional on that assumption rather than as an empirically qualified tail\-risk estimate\.

#### Pre\-specification and tail support\.

The loss functionLwL\_\{w\}, tail levelα\\alpha, risk toleranceBwB\_\{w\}, and confidence level1−δ1\-\\deltamust be specified before examining the held\-out qualification results\. They must not be tuned to make a candidate policy pass qualification\.

Tail\-risk estimates can be statistically weak when the qualification set contains few observations in the relevant tail\. For example, anα=0\.99\\alpha=0\.99CVaR estimate is informed primarily by roughly the worst one percent of the sampled outcomes\. READY therefore reports the effective number of observations contributing to the estimated tail together with the point estimate and confidence bound\. If the available qualification sample does not support a sufficiently precise upper bound, the appropriate result is*insufficient evidence for risk qualification*, rather than a relaxed risk threshold\.

## Appendix BSupplementary Analyses for CliniCARE\-Bench

This appendix supplements the four\-way deployment\-qualification analysis of Section[5](https://arxiv.org/html/2609.02095#S5)with the held\-out lower\-bound computation, sensitivity checks, and complete per\-system results\. All policies follow the global\-routing definition and margin\-adjusted threshold selection of Section[5\.2](https://arxiv.org/html/2609.02095#S5.SS2); all thresholds are selected on the development partition and frozen before evaluation on the held\-out qualification partition\.

### B\.1Reproducing the Held\-Out Lower Bound

The95%95\\%LCB column of Table[3](https://arxiv.org/html/2609.02095#S5.T3)instantiates Equation[34](https://arxiv.org/html/2609.02095#S3.E34), attaching confidence only to the observed accepted component\. Fix the thresholdτ^\\widehat\{\\tau\}selected onSdevS\_\{\\mathrm\{dev\}\}atY\+δY\+\\delta\(δ=0\.05\\delta=0\.05; Equation[45](https://arxiv.org/html/2609.02095#S5.E45)\) and letNqual=375N\_\{\\mathrm\{qual\}\}=375\.

1. 1\.Route\.For each qualification case setdi=𝟏\{si≥τ^\}d\_\{i\}=\\mathbf\{1\}\\\{s\_\{i\}\\geq\\widehat\{\\tau\}\\\}\(Equation[26](https://arxiv.org/html/2609.02095#S3.E26)\)\. Letnacc=∑idin\_\{\\mathrm\{acc\}\}=\\sum\_\{i\}d\_\{i\}be the number of accepted cases andkacc=∑idi​uik\_\{\\mathrm\{acc\}\}=\\sum\_\{i\}d\_\{i\}\\,u\_\{i\}the number of those that are autonomously successful, so the reviewed fraction is\(Nqual−nacc\)/Nqual\(N\_\{\\mathrm\{qual\}\}\-n\_\{\\mathrm\{acc\}\}\)/N\_\{\\mathrm\{qual\}\}andp^acc=kacc/nacc\\widehat\{p\}\_\{\\mathrm\{acc\}\}=k\_\{\\mathrm\{acc\}\}/n\_\{\\mathrm\{acc\}\}\.
2. 2\.Point estimate\.Combine the observed accepted rate with the assumed review success via Equation[32](https://arxiv.org/html/2609.02095#S3.E32):R^=naccNqual​p^acc\+Nqual−naccNqual​ah\\widehat\{R\}=\\tfrac\{n\_\{\\mathrm\{acc\}\}\}\{N\_\{\\mathrm\{qual\}\}\}\\widehat\{p\}\_\{\\mathrm\{acc\}\}\+\\tfrac\{N\_\{\\mathrm\{qual\}\}\-n\_\{\\mathrm\{acc\}\}\}\{N\_\{\\mathrm\{qual\}\}\}\\,a\_\{h\}\.
3. 3\.Lower bound\.Compute the one\-sided95%95\\%exact Clopper–Pearson lower boundp¯acc\\underline\{p\}\_\{\\mathrm\{acc\}\}for the accepted\-case success probability from\(kacc,nacc\)\(k\_\{\\mathrm\{acc\}\},n\_\{\\mathrm\{acc\}\}\), i\.e\. theα=0\.05\\alpha=0\.05quantile of theBeta⁡\(kacc,nacc−kacc\+1\)\\mathrm\{Beta\}\\\!\\left\(k\_\{\\mathrm\{acc\}\},\\,n\_\{\\mathrm\{acc\}\}\-k\_\{\\mathrm\{acc\}\}\+1\\right\)distribution \(withp¯acc=0\\underline\{p\}\_\{\\mathrm\{acc\}\}=0ifkacc=0k\_\{\\mathrm\{acc\}\}=0\)\. No confidence is attached to the assumed term, so 95% LCB=naccNqual​p¯acc\+Nqual−naccNqual​ah\.\\text\{95\\% LCB\}=\\frac\{n\_\{\\mathrm\{acc\}\}\}\{N\_\{\\mathrm\{qual\}\}\}\\,\\underline\{p\}\_\{\\mathrm\{acc\}\}\+\\frac\{N\_\{\\mathrm\{qual\}\}\-n\_\{\\mathrm\{acc\}\}\}\{N\_\{\\mathrm\{qual\}\}\}\\,a\_\{h\}\.
4. 4\.Qualify\.The policy qualifies iff95% LCB≥Y\\text\{95\\% LCB\}\\geq Y\.
5. 5\.Break\-even\.ahmina\_\{h\}^\{\\min\}\(Equation[35](https://arxiv.org/html/2609.02095#S3.E35)\) is the smallestah∈\[0,1\]a\_\{h\}\\in\[0,1\]for which the LCB expression reachesYY; because the LCB is nondecreasing inaha\_\{h\}this is a one\-dimensional solve, and — marks policies for which noah∈\[0,1\]a\_\{h\}\\in\[0,1\]suffices\.

### B\.2Sensitivity to the Human\-Review Assumption

The main analysis treats human\-review successaha\_\{h\}as a declared deployment assumption \(Equation[43](https://arxiv.org/html/2609.02095#S5.E43)\)\. Here we recompute the reliability–oversight frontier and the qualification outcomes of Section[5\.4](https://arxiv.org/html/2609.02095#S5.SS4)under alternative values ofaha\_\{h\}, reporting how the required review burden and qualified/not\-qualified status change with the assumed quality of the human review path\. We repeat the entire dev\-select/freeze/qualify procedure atah∈\{0\.80,0\.85,0\.90,0\.95,1\.00\}a\_\{h\}\\in\\\{0\.80,0\.85,0\.90,0\.95,1\.00\\\}over the same3131\-target sweep, giving31×16=49631\\times 16=496policies per assumption\. Nothing about the execution evidence changes; only the declared assumption does\.

Table[5](https://arxiv.org/html/2609.02095#A2.T5)reports the per\-system consequences at the representative targetY=0\.76Y=0\.76\. The aggregate picture over the full sweep is that the review assumption, not agent capability, governs how much of the reliability range is reachable at all:

Weakening the assumed review path fromah=0\.90a\_\{h\}=0\.90to0\.800\.80raises mean review burden by2323percentage points and caps the highest reliability any deploying policy attains at0\.7750\.775; at that assumption311311of496496policies route every case to review, qualifying only by not deploying\. The last column is censored from above atah≥0\.90a\_\{h\}\\geq 0\.90because the sweep ends atY=0\.85Y=0\.85, so those entries are lower bounds rather than ceilings\.

Two features of the table deserve comment because they are properties of the qualification procedure rather than of the systems\. First, the number of qualifying policies is*not*monotone inaha\_\{h\}\(472472atah=0\.85a\_\{h\}=0\.85against469469atah=0\.90a\_\{h\}=0\.90\)\. A more generous review assumption relaxes the development\-set constraint, which lets the optimization select a lower threshold and accept more cases; the resulting accepted set is larger but less reliable, and can fail the lower\-confidence\-bound test on the held\-out partition that the more conservative policy passed\. GPT\-5\.6\-Luna atY=0\.76Y=0\.76is the clean instance: it qualifies atah=0\.80a\_\{h\}=0\.80and atah=0\.90a\_\{h\}=0\.90, but not atah=0\.85a\_\{h\}=0\.85\. This is the expected behaviour of selecting on one partition and testing on another, and it is a reason to read the frontier rather than any single cell\. Second, review burden is weakly decreasing inaha\_\{h\}for every system, but in steps, because each system can only move between the thresholds its stated confidence actually reaches\. Gemini\-3\.1\-Pro is unchanged at100%100\\%review for everyah≤0\.95a\_\{h\}\\leq 0\.95and only drops to33\.6%33\.6\\%atah=1\.00a\_\{h\}=1\.00, which reflects the granularity limit of Section[5\.3](https://arxiv.org/html/2609.02095#S5.SS3), not a response to the assumption\.

Table 5:Held\-out qualification atY=0\.76Y=0\.76under alternative human\-review assumptions\.Nqual=375N\_\{\\mathrm\{qual\}\}=375; thresholds reselected onSdevS\_\{\\mathrm\{dev\}\}atY\+0\.05Y\+0\.05for eachaha\_\{h\}and frozen\. Rev% is the fraction of qualification cases routed to human review; Q marks whether the95%95\\%LCB reachesYY\. A system at Rev%=100=100attains exactlyaha\_\{h\}and qualifies without deploying wheneverah≥Ya\_\{h\}\\geq Y\.
### B\.3Process\-Aware Success Definition

The main analysis scores an autonomous execution as successful when the four\-way verdict matches the reference adjudication \(Equation[41](https://arxiv.org/html/2609.02095#S5.E41)\)\. Here we repeat the principal comparisons using the process\-aware variantui,dfu\_\{i,\\mathrm\{df\}\}\(Equation[42](https://arxiv.org/html/2609.02095#S5.E42)\), which additionally requires the absence of a disqualifying process defect, to test sensitivity of the routing\-quality and frontier conclusions to the deployment\-success definition\. CliniCARE records the defect as a single per\-run indicator, and it is common:21\.9%21\.9\\%of the12,00012\{,\}000runs carry one, ranging from12\.7%12\.7\\%\(GPT\-5\.5\) to30\.0%30\.0\\%\(Kimi\-K2\.7\-Code\)\. Because a defect can co\-occur with a correct verdict, substitutingui,dfu\_\{i,\\mathrm\{df\}\}foruiu\_\{i\}lowers autonomous success by4\.54\.5to13\.113\.1percentage points depending on the system\.

The conclusions of Section[5\.3](https://arxiv.org/html/2609.02095#S5.SS3)survive the substitution; the frontier of Section[5\.4](https://arxiv.org/html/2609.02095#S5.SS4)does not\. Routing quality is measuring the same underlying property under either predicate: the Spearman correlation between AUROC underuuand underudfu\_\{\\mathrm\{df\}\}across the1616systems is0\.870\.87, and per\-system AUROC moves by at most0\.0320\.032\(Table[6](https://arxiv.org/html/2609.02095#A2.T6)\)\. Qwen\-3\.7\-Plus remains among the best\-ranking signals and Gemini\-3\.1\-Pro the worst\. The deployment consequences, by contrast, shift substantially\. Mean review burden across the sweep rises from53\.0%53\.0\\%to69\.8%69\.8\\%, and atY=0\.76Y=0\.76both Gemini Flash systems move from roughly38%38\\%review to100%100\\%: under a process\-aware success definition they have no reachable operating point that deploys at all\. This strengthens rather than weakens the comparison of Section[5\.6](https://arxiv.org/html/2609.02095#S5.SS6), since the pair contrasted there collapses to two undeployable systems once process defects count against success\.

Two further observations\. The count of qualifying policies*rises*underudfu\_\{\\mathrm\{df\}\}\(488488of496496against469469\), for the same selection reason discussed in Appendix[B\.2](https://arxiv.org/html/2609.02095#A2.SS2): a harder success predicate forces the development\-set optimization to far more conservative thresholds, and those policies clear the held\-out bound more comfortably\. GLM\-5\.2 is the visible case, moving from not qualifying at10\.9%10\.9\\%review underuuto qualifying at44\.8%44\.8\\%review underudfu\_\{\\mathrm\{df\}\}\. A system’s qualified status is therefore not a monotone function of how demanding the success predicate is, and reporting review burden alongside it remains necessary\.

Table 6:Routing quality and held\-out qualification under the process\-aware success definition\.a0a\_\{0\}and AUROC are onSdevS\_\{\\mathrm\{dev\}\}\(n=375n=375\); Review%,R^\\widehat\{R\}and the LCB are onSqualS\_\{\\mathrm\{qual\}\}atY=0\.76Y=0\.76,ah=0\.90a\_\{h\}=0\.90, with thresholds reselected atY\+0\.05Y\+0\.05under each predicate\. AUROC is computed from the threshold\-based risk–coverage curve, so values are directly comparable across the two columns\.
### B\.4Complete Per\-System Results

This section reports the complete versions of the main\-text summaries: per\-system risk–coverage and qualification results for all evaluated systems, and routing\-score distributions with autonomous success broken down by predicted and reference verdict class\.

#### Qualification across targets\.

Table[7](https://arxiv.org/html/2609.02095#A2.T7)extends the single\-target view of Table[3](https://arxiv.org/html/2609.02095#S5.T3)to three targets spanning the range in which most systems have reachable deploying operating points\. The step\-function behaviour noted in Section[5\.5](https://arxiv.org/html/2609.02095#S5.SS5)is visible throughout: review burden is flat across adjacent targets for systems whose confidence takes few distinct values, then jumps\. Opus 5 moves0\.8→22\.4→37\.90\.8\\to 22\.4\\to 37\.9percent review across the three targets while Gemini\-3\.1\-Pro is pinned at100%100\\%for the upper two\.

#### Routing behaviour by verdict class\.

Table[8](https://arxiv.org/html/2609.02095#A2.T8)breaks the qualification partition down by predicted verdict\. Pooled across systems, autonomous success is strongly class\-dependent —0\.8560\.856forYes,0\.6570\.657forNo,0\.5440\.544forIndeterminate: Lack of Dataand0\.3360\.336forIndeterminate: Medically Ambiguous— while mean stated confidence varies far less \(0\.8940\.894,0\.8910\.891,0\.8390\.839,0\.7970\.797\)\. Systems are therefore somewhat but not proportionately less confident on the verdicts they get wrong more often\.

The result that matters for the policy design of Section[5\.2](https://arxiv.org/html/2609.02095#S5.SS2)is that the routing signal retains ranking power*within*every predicted class: pooled within\-class AUROC is0\.6390\.639forYes,0\.6110\.611forNo,0\.6140\.614forILDand0\.6680\.668forIMA\. The signal is thus not a proxy for the predicted class\. A class\-agnostic threshold rule is not discarding information that a class\-conditional rule would recover, which is the empirical basis for keeping class semantics insideϕw\\phi\_\{w\}and out of the escalation policy\.

Table 7:Held\-out qualification across three reliability targets\.Nqual=375N\_\{\\mathrm\{qual\}\}=375,ah=0\.90a\_\{h\}=0\.90, thresholds selected onSdevS\_\{\\mathrm\{dev\}\}atY\+0\.05Y\+0\.05and frozen\. Rev% is the fraction of qualification cases routed to human review, LCB the one\-sided95%95\\%bound of Appendix[B\.1](https://arxiv.org/html/2609.02095#A2.SS1), and Q whether the policy qualifies\. TheY=0\.76Y=0\.76block reproduces Table[3](https://arxiv.org/html/2609.02095#S5.T3); the break\-even assumptionahmina\_\{h\}^\{\\min\}is omitted here for width and is reported atY=0\.76Y=0\.76in that table\. Every policy that fails to qualify does so under noah∈\[0,1\]a\_\{h\}\\in\[0,1\]\.YesNoILDIMASystem%Succ\.%Succ\.%Succ\.%Succ\.\|\{si\}\|\|\\\{s\_\{i\}\\\}\|Claude CodeOpus 542\.486\.838\.471\.516\.058\.33\.258\.325Sonnet 543\.785\.436\.573\.016\.063\.33\.728\.626CodexGPT\-5\.6\-Sol38\.491\.739\.267\.318\.754\.33\.742\.919GPT\-5\.6\-Luna34\.186\.741\.161\.018\.440\.66\.425\.021GPT\-5\.541\.190\.942\.168\.413\.961\.52\.963\.625GPT\-5\.435\.788\.842\.165\.817\.947\.84\.331\.232GPT\-5\.4\-mini39\.583\.842\.760\.012\.345\.75\.619\.033Gemini CLIGemini\-3\.6\-Flash42\.986\.339\.767\.113\.366\.04\.040\.05Gemini\-3\.5\-Flash44\.384\.940\.566\.411\.763\.63\.538\.55Gemini\-3\.1\-Pro39\.787\.237\.171\.221\.356\.21\.971\.44opencodeDeepSeek\-V4\-Pro38\.786\.241\.662\.214\.754\.54\.833\.315DeepSeek\-V4\-Flash49\.181\.035\.770\.111\.756\.83\.530\.810GLM\-5\.245\.980\.839\.268\.011\.257\.13\.735\.723Qwen\-3\.7\-Plus35\.287\.942\.458\.514\.939\.37\.535\.718MiniMax\-M340\.079\.341\.962\.49\.956\.88\.322\.619Kimi\-K2\.7\-Code39\.784\.643\.561\.39\.155\.97\.724\.114Table 8:Predicted\-class mix and within\-class autonomous success on the qualification partition\.% is the share of the375375cases receiving each predicted verdict; Succ\. is four\-way verdict accuracy within that share\.ILDandIMAabbreviate the two indeterminate classes of Equation[40](https://arxiv.org/html/2609.02095#S5.E40)\.\|\{si\}\|\|\\\{s\_\{i\}\\\}\|is the number of distinct stated\-confidence values the system emits on this partition, the granularity limit discussed in Section[5\.3](https://arxiv.org/html/2609.02095#S5.SS3)\.

### B\.5Cost\-of\-Conflation Control: Always\-Review\-Indeterminate

The main\-text oversight policy acts only on the routing signal and is agnostic to the four adjudication classes \(Section[5\.2](https://arxiv.org/html/2609.02095#S5.SS2)\)\. As a negative control, we contrast it with a policy that instead escalates on the*predicted class*: every indeterminate adjudication is routed to human review regardless of confidence,

πIR\(si,y^i\)=\{review,y^i∈\{Indeterminate:LackofData,Indeterminate:MedicallyAmbiguous\},accept,y^i∈\{Yes,No\}∧si≥τ,review,otherwise\.\\pi\_\{\\mathrm\{IR\}\}\(s\_\{i\},\\widehat\{y\}\_\{i\}\)=\\begin\{cases\}\\mathrm\{review\},&\\widehat\{y\}\_\{i\}\\in\\\{\\mathrm\{Indeterminate\{:\}\\ Lack\\ of\\ Data\},\\mathrm\{Indeterminate\{:\}\\ Medically\\ Ambiguous\}\\\},\\\\\[3\.0pt\] \\mathrm\{accept\},&\\widehat\{y\}\_\{i\}\\in\\\{\\mathrm\{Yes\},\\mathrm\{No\}\\\}\\ \\land\\ s\_\{i\}\\geq\\tau,\\\\\[3\.0pt\] \\mathrm\{review\},&\\text\{otherwise\}\.\\end\{cases\}\(59\)
This baseline operationalizes the conflation READY is designed to avoid: it treats a task\-level abstention, a possibly\-correct indeterminate verdict, an*outcome*scored byϕw\\phi\_\{w\}, as if it were a deployment\-level escalation, a*decision*that should depend on the routing signal\. We select and freezeτ\\tauforπIR\\pi\_\{\\mathrm\{IR\}\}exactly as for the signal\-only policy, and report the comparison in Table[9](https://arxiv.org/html/2609.02095#A2.T9)\.

#### The control is not more expensive on this benchmark\.

We had expected the conflation to show up as wasted oversight, and it does not\. Averaged over the496496policies in the sweep,πIR\\pi\_\{\\mathrm\{IR\}\}routes48\.9%48\.9\\%of cases to review against53\.0%53\.0\\%for the signal\-only policy, and it requires strictly more review in only206206of496496cells\. The single\-target comparison could be dismissed as threshold\-grid overshoot, so the right\-hand columns of Table[9](https://arxiv.org/html/2609.02095#A2.T9)instead report, for each policy, the minimum review burden at which it*achieves*a given reliability on the qualification partition, which removes the grid effect\. The control is still not dominated: it needs more review on1010of1616systems atR^=0\.80\\widehat\{R\}=0\.80and66of1515atR^=0\.85\\widehat\{R\}=0\.85, with mean differences of−2\.2\-2\.2and−5\.2\-5\.2percentage points\.

The reason is visible in Table[8](https://arxiv.org/html/2609.02095#A2.T8)\. Indeterminate adjudications really are much less often correct than definite ones on this benchmark \(0\.4930\.493against0\.7570\.757pooled on the qualification partition\), so predicted class carries genuine information about autonomous failure\. Escalating on it is not waste, and we report that rather than presenting a control the data does not support\.

#### What the control does cost\.

The objection toπIR\\pi\_\{\\mathrm\{IR\}\}is structural rather than budgetary\. Because the class constraint is fixed, the policy exposes one hard floor on review burden per system — no threshold can accept an indeterminate case — so it does not trace a frontier over the reliability range the way Equation[44](https://arxiv.org/html/2609.02095#S5.E44)does\. It also inherits the benchmark’s class\-success profile as an assumption: it is competitive here only because indeterminate verdicts happen to be unreliable in this cohort, a property of these labels rather than of the deployment, and one that no part of the policy measures or would detect if it changed\. The signal\-based policy makes the same trade explicit and tunable, and Appendix[B\.4](https://arxiv.org/html/2609.02095#A2.SS4)shows the routing signal remains informative*within*each class, so the two are complementary sources of information rather than substitutes\. A deployment free to use both would do better than either; the claim defended here is only that the escalation decision should be driven by a measured routing signal rather than inferred from an outcome class\.

Table 9:Signal\-based routing against the always\-review\-indeterminate control,ah=0\.90a\_\{h\}=0\.90,Nqual=375N\_\{\\mathrm\{qual\}\}=375\. Left block: both policies selected onSdevS\_\{\\mathrm\{dev\}\}atY\+0\.05Y\+0\.05and frozen, evaluated atY=0\.76Y=0\.76\. Right block: the minimum review burden at which each policy attains the stated reliability among its own reachable operating points onSqualS\_\{\\mathrm\{qual\}\}, which removes the threshold\-grid overshoot that confounds the single\-target comparison; — marks a reliability no operating point of that policy reaches\.