EnterpriseVal：量化生成式AI在企业中的效能、可靠性和价值

# 1Introduction
Source: [https://arxiv.org/html/2609.21841](https://arxiv.org/html/2609.21841)
marginparsep has been altered\. topmargin has been altered\. marginparwidth has been altered\. marginparpush has been altered\.

The page layout violates the ICML style\.

Please do not change the page layout, or include packages like geometry, savetrees, or fullpage, which change it for you\.

We’re not able to reliably undo arbitrary changes to the style\. Please remove the offending package\(s\), or layout\-changing commands and try again\.

EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise

Abbas Raza Ali\*1Muhammad Ajmal Siddiqui2Moona Zahid3

††footnotetext:1Citigrounp, Inc, London, United Kingdom2Ernst & Young LLP, London, United Kingdom3NVIDIA Corporation, London, United Kingdom\. Correspondence to: Abbas Raza Ali <abbas\.raza\.ali@gmail\.com\>\.
Proceedings of the10t​h\\mathit\{10\}^\{th\}MLSys Conference, Bellevue, WA, USA, 2027\. Copyright 2027 by the author\(s\)\.###### Abstract

Frontier language models now produce professional deliverables that expert graders judge to match human work on a substantial share of economically valuable tasks, yet most enterprise GenAI initiatives fail to show a measurable business effect and a large fraction of agentic projects are expected to be cancelled\. We argue that this is substantially a*measurement*problem: public benchmarks answer “what can the model do?”, whereas a deployment decision requires “is this workflow fit, reliable, safe and worth scaling \- here, on our data, under our controls?”\. We presentEnterpriseVal, a use\-case\-level evaluation system that closes this gap\. It comprises \(i\) a formal specification of the use case and of the frozen socio\-technical configuration under test, model, prompts, retrieval, tools, guardrails and human oversight, with an autonomy level and consequence tier that jointly set the required evaluation intensity; \(ii\) a metric catalogue spanning fidelity, utility, efficiency, reliability, assurance and oversight; \(iii\) a grading protocol that scales blinded expert judgement with calibrated LLM\-as\-judge scoring through prediction\-powered inference; \(iv\) a two\-tier threshold gate, stated as an executable algorithm, that maps metric vectors with confidence bounds toReject/Conditional/Scaledecisions; and \(v\) a value\-and\-risk model in which the reviewer catch rate is a measured parameter\. We report a pilot across three workflows in a global bank\. In credit\-memo drafting, human\-graded citation precision reached 88% and hallucination rate 1\.6% for the best model against gates of 70% and 5%; in procedure transformation, analyst refinement effort fell from an estimated 27\.4 to 2\.9 hours per document\. We separate established results, documented pilot evidence, the proposed system and open hypotheses, and specify the experiments required for full validation\.

## 1Introduction

The central challenge for enterprise AI is no longer whether foundation models can perform valuable professional work, but whether that capability can be translated into reliable, measurable, and economically sustainable enterprise outcomes\. Three developments make this question increasingly urgent\. First, model capability on realistic professional work has risen steeply: on GDPval, 1,320 expert\-authored tasks across 44 occupations, the best frontier models in late 2025 produced deliverables rated as good as or better than experienced professionals in just under half of blinded pairwise comparisons, and performance more than tripled between GPT\-4o and GPT\-5[Patwardhan et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib1)\. Second, realised enterprise value lags far behind: an MIT study of roughly 300 deployments found about 95% of pilots with no measurable P&L impact[Challapally et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib2), and McKinsey’s 2025 survey found 88% of organisations using AI in at least one function but only 39% attributing any EBIT impact to it[Singla et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib3)\. Third, agentic systems are arriving before the evaluation discipline needed to operate them: Gartner projects that over 40% of agentic AI projects will be cancelled by 2027 for unclear value or inadequate risk controls[Gartner \(2025\)](https://arxiv.org/html/2609.21841#bib.bib4), and a survey of 306 practitioners running agents in production finds reliability to be the dominant barrier, with most restricting agents to a few steps before human intervention[Pan et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib5)\.

We contend that much of the distance between the first fact and the second is a measurement gap with two parts \(Figure[1](https://arxiv.org/html/2609.21841#S1.F1)\)\. Capability benchmarks are built to be portable across organisations, and that portability is exactly what prevents them from answering the questions that decide a deployment: whether a credit analyst may rely on a generated repayment analysis depends on the firm’s documents, credit policy, citation conventions and reviewer capacity\. Conversely, the metrics that internal teams do compute rarely reach a governance committee in a decision\-grade form: thresholds are set after the fact, sample sizes go unreported, LLM\-graded scores are presented without calibration, and the link from metric to money is asserted rather than modelled\.

![Refer to caption](https://arxiv.org/html/2609.21841v1/figs/fig1_evaluation_gap.png)Figure 1:Two gaps between public benchmarks and enterprise decisions\.Capability benchmarks measure what a model can do on portable tasks\. Enterprises must also establish transfer to their own workflows and ground truth \(Gap 1\) and convert quality metrics into a value\-and\-risk case a governance body can act on \(Gap 2\)\.EnterpriseVal \(eVal\)treats evaluation as a systems problem: a repeatable pipeline that takes a frozen workflow configuration and a firm\-specific task set and emits an auditable gate decision, an efficacy index and a value estimate with stated uncertainty\. It sits between model benchmarks such as GDPval[Patwardhan et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib1), HELM[Liang et al\. \(2023\)](https://arxiv.org/html/2609.21841#bib.bib13)and the agent suitesτ\\tau\-bench[Yao et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib9);[Barres et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib10), SWE\-bench[Jimenez et al\. \(2024\)](https://arxiv.org/html/2609.21841#bib.bib15)and HAL[Kapoor et al\. \(2025a\)](https://arxiv.org/html/2609.21841#bib.bib11), and the model\-risk\-management regimes of regulated industries[Bank of England Prudential Regulation Authority \(2023\)](https://arxiv.org/html/2609.21841#bib.bib36)\. It borrows GDPval’s blinded expert comparison, the reliability and cost metrics of the agent\-evaluation literature[Yao et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib9);[Kapoor et al\. \(2025b\)](https://arxiv.org/html/2609.21841#bib.bib12);[Rabanser et al\. \(2026\)](https://arxiv.org/html/2609.21841#bib.bib6), and the statistical hygiene of survey methodology, and adds decision rules and a value model that a risk committee can act on\.

#### Contributions\.

1. 1\.A formal evaluation object, use case𝒰\\mathcal\{U\}and frozen configurationσ\\sigmaincluding human oversight, with an*autonomy*×\\times*consequence*evaluation\-intensity matrix \(§[3](https://arxiv.org/html/2609.21841#S3)\)\.
2. 2\.A six\-family metric catalogue with operational definitions, extending pilot fidelity metrics with reliability \(pass^kk\), assurance and oversight metrics \(§[4](https://arxiv.org/html/2609.21841#S4)\)\.
3. 3\.A grading and statistical protocol: blinded experts with agreement targets, calibrated LLM\-as\-judge with prediction\-powered correction, sample\-size rules, configuration freeze and harness parity \(§[5](https://arxiv.org/html/2609.21841#S5)\)\.
4. 4\.A two\-tier gate and Efficacy Index, given as algorithms, and a value\-and\-risk model with the reviewer catch rate as an explicit parameter \(§[6](https://arxiv.org/html/2609.21841#S6)\)\.
5. 5\.Documented pilot evidence from three workflows in a global bank, reported with limitations, and the experiments needed to complete validation \(§[7](https://arxiv.org/html/2609.21841#S7)–§[9](https://arxiv.org/html/2609.21841#S9)\)\.

Throughout we label claims as\[Established\]\(cited prior work\),\[Documented\]\(our pilot\),\[Proposed\]\(our design\) or\[Hypothesis\]\.

## 2Background and Related Work

\[Established\]

#### From capability to economically valuable tasks\.

Knowledge tests such as MMLU[Hendrycks et al\. \(2021\)](https://arxiv.org/html/2609.21841#bib.bib14)saturated quickly and suffer from contamination and weak construct validity[Eriksson et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib21);[Raji et al\. \(2021\)](https://arxiv.org/html/2609.21841#bib.bib22)\. HELM broadened coverage but stayed model\-centric[Liang et al\. \(2023\)](https://arxiv.org/html/2609.21841#bib.bib13)\. GDPval shifted to expert\-authored, expert\-graded professional deliverables and reported that reasoning effort and task context each yield measurable gains and that models complete tasks roughly100×100\\timesfaster and cheaper than experts[Patwardhan et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib1)\. GDPval is organised by occupation and designed to be portable;eValis organised by use case and designed to be firm\-specific\. They are complementary: GDPval shortlists model families,eValdecides deployments\.

#### Agent evaluation and the reliability gap\.

τ\\tau\-bench introduced pass^kk, success on allkkattempts, and showed agents that succeed 60% of the time once may succeed only 25% of the time on all of eight tries[Yao et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib9)\. Across 24 months of frontier releases, reliability improved far less than accuracy and plateaued similarly across vendors[Rabanser et al\. \(2026\)](https://arxiv.org/html/2609.21841#bib.bib6)\. Accuracy without cost yields needlessly expensive agents[Kapoor et al\. \(2025b\)](https://arxiv.org/html/2609.21841#bib.bib12); HAL standardises cost logging[Kapoor et al\. \(2025a\)](https://arxiv.org/html/2609.21841#bib.bib11), and the CLEAR framework adds latency, assurance and reliability, finding that expert judgement of production success tracks a multidimensional score far better than accuracy[Mehta \(2025\)](https://arxiv.org/html/2609.21841#bib.bib8)\. KAMI’s 5\.5\-billion\-token study found public leaderboard rank a poor predictor of performance on routine enterprise tasks, with small prompt and tool\-message changes moving accuracy and cost by large factors[Roig \(2025\)](https://arxiv.org/html/2609.21841#bib.bib7); log\-level analysis is increasingly regarded as necessary[Kirgis et al\. \(2026\)](https://arxiv.org/html/2609.21841#bib.bib20)\. Enterprise sandboxes[Vishwakarma et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib16);[Wang et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib17)remain generic across firms\. A systematic review of 84 agentic\-AI evaluations finds capability measured extensively while workflow integration, trust and operation over time are rarely measured[Meimandi et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib18), and predictive validity, whether in\-sample rankings transfer, has been proposed as the standard for agent leaderboards[Patel et al\. \(2026\)](https://arxiv.org/html/2609.21841#bib.bib19)\.

#### Grounded generation\.

Enterprise drafting is retrieval\-augmented generation[Lewis et al\. \(2020\)](https://arxiv.org/html/2609.21841#bib.bib23), and its principal failure is unfaithful content[Huang et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib24);[Ji et al\. \(2023\)](https://arxiv.org/html/2609.21841#bib.bib25)\. FActScore checks atomic facts against a source[Min et al\. \(2023\)](https://arxiv.org/html/2609.21841#bib.bib26); RAGAS provides reference\-free faithfulness and relevance[Es et al\. \(2024\)](https://arxiv.org/html/2609.21841#bib.bib27)\. LLM\-as\-judge methods correlate well with human preference[Zheng et al\. \(2023\)](https://arxiv.org/html/2609.21841#bib.bib28);[Liu et al\. \(2023\)](https://arxiv.org/html/2609.21841#bib.bib29)but exhibit position, verbosity and self\-preference biases[Gu et al\. \(2024\)](https://arxiv.org/html/2609.21841#bib.bib30), and aligning judges with human criteria is itself iterative[Shankar et al\. \(2024\)](https://arxiv.org/html/2609.21841#bib.bib31)\.

#### Productivity and oversight\.

Field studies report 14% more issues resolved per hour in customer support[Brynjolfsson et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib33), 40% faster professional writing[Noy and Zhang \(2023\)](https://arxiv.org/html/2609.21841#bib.bib34), and 12% more tasks for consultants alongside*lower*accuracy outside model competence \- the “jagged frontier”[Dell’Acqua et al\. \(2023\)](https://arxiv.org/html/2609.21841#bib.bib35)\. Time saved must therefore be measured net of review, and review performance itself must be measured\.

#### Governance\.

In banking, quantitative methods fall under model risk management \(SR 11\-7, SS1/23\)[Bank of England Prudential Regulation Authority \(2023\)](https://arxiv.org/html/2609.21841#bib.bib36); the NIST AI RMF and its GenAI Profile call for measurement, thresholds and documentation[National Institute of Standards and Technology \(2023\)](https://arxiv.org/html/2609.21841#bib.bib37);[National Institute of Standards and Technology \(2024\)](https://arxiv.org/html/2609.21841#bib.bib38); the EU AI Act imposes risk\-management, data\-governance, human\-oversight, accuracy and logging duties on high\-risk systems[European Parliament and Council of the European Union \(2024\)](https://arxiv.org/html/2609.21841#bib.bib39), with stand\-alone high\-risk obligations deferred to December 2027 by the 2026 Digital Omnibus[European Parliament and Council of the European Union \(2026\)](https://arxiv.org/html/2609.21841#bib.bib40); ISO/IEC 42001 and 42005 provide management\-system and impact\-assessment standards[International Organization for Standardization \(2023\)](https://arxiv.org/html/2609.21841#bib.bib41);[International Organization for Standardization \(2025\)](https://arxiv.org/html/2609.21841#bib.bib42)\. No prior framework combines firm\-specific ground truth, decision thresholds with uncertainty, a value model and a governance mapping \(Table[1](https://arxiv.org/html/2609.21841#S2.T1)\)\.

Table 1:Positioning against representative frameworks \(● full, ◐ partial, ○ absent\)\.

## 3The EnterpriseVal Framework

\[Proposed\]

### 3\.1Design principles

Five principles, each traceable to a failure in §[2](https://arxiv.org/html/2609.21841#S2):*evaluate the workflow, not the model*;*ground truth is expert\-authored and firm\-specific*;*every score carries its uncertainty and its grader*;*thresholds are decisions set before results are seen*; and*value is modelled, not asserted*\. A sixth is a systems requirement:*the configuration is frozen and human oversight is measured*\- comparisons are valid only under harness parity, and “human in the loop” is a component whose detection performance is measured, not an assumption that converts an imperfect system into a safe one\.

### 3\.2Formal setting

A use case is a tuple

𝒰=⟨𝒯,𝒮,𝒢,ℛ,a,c⟩,ti=\(xi,Si,gi\)∈𝒯,\\mathcal\{U\}=\\langle\\mathcal\{T\},\\mathcal\{S\},\\mathcal\{G\},\\mathcal\{R\},a,c\\rangle,\\qquad t\_\{i\}=\(x\_\{i\},S\_\{i\},g\_\{i\}\)\\in\\mathcal\{T\},\(1\)where𝒯\\mathcal\{T\}is a set ofnntasks sampled from the workflow population, each an instructionxix\_\{i\}, permitted sourcesSiS\_\{i\}and an expert gold artefactgig\_\{i\};𝒮\\mathcal\{S\}the corpus and access policy;𝒢\\mathcal\{G\}the gold\-set specification including the*key elements*𝒦⁡\(gi\)\\mathcal\{K\}\(g\_\{i\}\)a competent artefact must contain;ℛ\\mathcal\{R\}the rubric;a∈\{A0,…,A4\}a\\in\\\{\\mathrm\{A0\},\\dots,\\mathrm\{A4\}\\\}the autonomy level andc∈\{C1,…,C4\}c\\in\\\{\\mathrm\{C1\},\\dots,\\mathrm\{C4\}\\\}the consequence tier \(§[3\.4](https://arxiv.org/html/2609.21841#S3.SS4)\)\. The candidate system is itself a tuple, frozen for the run:

σ=⟨M,P,Φ,U,Γ,H⟩,\(yi,τi\)=σ⁡\(xi,Si\),\\sigma=\\langle M,P,\\Phi,U,\\Gamma,H\\rangle,\\qquad\(y\_\{i\},\\tau\_\{i\}\)=\\sigma\(x\_\{i\},S\_\{i\}\),\(2\)withMMthe model\(s\) and sampling settings,PPprompts and policies,Φ\\Phiretrieval, memory and context management,UUtools and permissions,Γ\\Gammaguardrails, verification and logging, andHHhuman roles, checkpoints and escalation rules;τi\\tau\_\{i\}is the trajectory of tool calls and intermediate states\. MakingHHpart ofσ\\sigmais deliberate: two deployments of one model with different review designs are different systems\. Any change toσ\\sigmais a re\-evaluation trigger\. The evaluation returns a metric vector𝐦⁡\(σ,𝒰\)\\mathbf\{m\}\(\\sigma,\\mathcal\{U\}\)with confidence intervals and a gate decision\.

### 3\.3Architecture

Figure[2](https://arxiv.org/html/2609.21841#S3.F2)shows four layers and a governance spine\. Layer 1 constructs𝒰\\mathcal\{U\}; Layer 2 computes metrics \(§[4](https://arxiv.org/html/2609.21841#S4)\); Layer 3 produces graded judgements with uncertainty \(§[5](https://arxiv.org/html/2609.21841#S5)\); Layer 4 applies thresholds and the value model \(§[6](https://arxiv.org/html/2609.21841#S6)\)\. The spine maps each artefact to MRM, NIST, EU AI Act and ISO 42001 requirements so the evaluation doubles as the model\-risk evidence file \(Appendix C \(supplementary\)\)\.

![Refer to caption](https://arxiv.org/html/2609.21841v1/figs/fig2_architecture.png)Figure 2:eValarchitecture\.Four evaluation layers feed a governance decision; the governance spine consumes artefacts from every layer\.
### 3\.4Autonomy, consequence and evaluation intensity

Autonomy describes what a system*does*; consequence describes what is at stake when it is wrong\. We define five autonomy levels, adapted from[Morris et al\. \(2024\)](https://arxiv.org/html/2609.21841#bib.bib43): A0 assistive Q&A; A1 grounded drafting with citations and human edits; A2 tool\-using workflow \(retrieval, extraction, rule checks\); A3 multi\-step agent with human checkpoints; A4 bounded autonomy acting on systems with post\-hoc review\. Each level inherits all lower\-level metrics and adds families \(Table[2](https://arxiv.org/html/2609.21841#S4.T2)\)\. Orthogonally, consequence tiers run from C1 \(internal, reversible\) through C2 \(informs judgement\) and C3 \(alters records or decisions\) to C4 \(regulatory, customer or financial harm\), following MRM materiality logic[Bank of England Prudential Regulation Authority \(2023\)](https://arxiv.org/html/2609.21841#bib.bib36)and ISO/IEC 42005[International Organization for Standardization \(2025\)](https://arxiv.org/html/2609.21841#bib.bib42)\. The two set the*evaluation intensity*

E⁡\(a,c\)=⌈a\+c2⌉∈\{1,…,4\},E\(a,c\)=\\left\\lceil\\tfrac\{a\+c\}\{2\}\\right\\rceil\\in\\\{1,\\dots,4\\\},\(3\)witha∈\{0\.\.4\}a\\in\\\{0\.\.4\\\},c∈\{1\.\.4\}c\\in\\\{1\.\.4\\\}\(Figure[3](https://arxiv.org/html/2609.21841#S3.F3)\)\. E1 requires fidelity grading on≥\\geq100 units; E2 adds inter\-rater agreement, CIs with gate decisions on conservative bounds andK≥2K\\geq 2repeats; E3 adds pass^kkwithK≥4K\\geq 4, a cost\-per\-successful\-task gate and trajectory logging; E4 adds an adversarial suite, independent validation and production monitoring\. A low\-autonomy, high\-consequence drafting assistant is thus not under\-evaluated because it “only drafts”\.

![Refer to caption](https://arxiv.org/html/2609.21841v1/figs/fig9_intensity_matrix.png)Figure 3:Evaluation\-intensity matrix\(Eq\.[3](https://arxiv.org/html/2609.21841#S3.E3)\)\. Markers place the pilot use cases: B credit memos \(A1,C3\), A control assessment \(A2,C3\), C procedure transformation \(A1,C2\)\.

## 4Metric Layer

\[Proposed; individual metrics instantiate established constructs[Min et al\. \(2023\)](https://arxiv.org/html/2609.21841#bib.bib26);[Es et al\. \(2024\)](https://arxiv.org/html/2609.21841#bib.bib27);[Yao et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib9)\]

Metrics are grouped into six families answering distinct decision questions \-*can I trust it*\(fidelity\),*is it useful*\(utility\),*does it save effort*\(efficiency\),*will it keep working*\(reliability\),*does it stay within the rules*\(assurance\) and*does the human catch what it misses*\(oversight\)\. Units are those a reviewer inspects: sentences, cited chunks, key elements, tasks, trajectories \(Table[2](https://arxiv.org/html/2609.21841#S4.T2)\)\.

#### Fidelity\.

For outputyysegmented into sentences, the hallucination rate is the share of assertive sentences unsupported by the permitted sources,

HR⁡\(y\)=1\|y\|​∑s∈y𝟏​\[assert⁡\(s\)∧¬supp⁡\(s∣S\)\],\\mathrm\{HR\}\(y\)=\\tfrac\{1\}\{\|y\|\}\\textstyle\\sum\_\{s\\in y\}\\mathbf\{1\}\[\\,\\mathrm\{assert\}\(s\)\\wedge\\neg\\,\\mathrm\{supp\}\(s\\mid S\)\\,\],\(4\)a*faithfulness*rather than world\-factuality definition[Huang et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib24): a true statement from a non\-permitted source counts, because the workflow requires traceability\. The sentence is the operational unit; the reference definition is at the level of atomic claims[Min et al\. \(2023\)](https://arxiv.org/html/2609.21841#bib.bib26), and the two are reconciled on a calibration subset because one sentence may carry several claims\. Citation precision is the share of cited chunks actually used by the citing paragraph,CP=\|\{c:used⁡\(c,para⁡\(c\)\)\}\|/\|𝒞⁡\(y\)\|\\mathrm\{CP\}=\|\\\{c:\\mathrm\{used\}\(c,\\mathrm\{para\}\(c\)\)\\\}\|/\|\\mathcal\{C\}\(y\)\|; the pilot used a*lenient*criterion \(any part reflected\), and we recommend also reporting a strict variant and citation recall\.

#### Utility\.

Key\-element captureKEC=\|𝒦⁡\(g\)∩𝒦⁡\(y\)\|/\|𝒦⁡\(g\)\|\\mathrm\{KEC\}=\|\\mathcal\{K\}\(g\)\\cap\\mathcal\{K\}\(y\)\|/\|\\mathcal\{K\}\(g\)\|is a recall measure against expert expectation; irrelevance rateIR=1\|y\|∑s𝟏\[s∉rel\(x\)\]\\mathrm\{IR\}=\\tfrac\{1\}\{\|y\|\}\\sum\_\{s\}\\mathbf\{1\}\[s\\notin\\mathrm\{rel\}\(x\)\]is a precision\-like measure; reporting both guards against a system that captures everything by writing everything\.

#### Efficiency\.

Net of review:Δeff=1−t¯AI/t¯manual\\Delta\_\{\\mathrm\{eff\}\}=1\-\\bar\{t\}\_\{\\mathrm\{AI\}\}/\\bar\{t\}\_\{\\mathrm\{manual\}\}witht¯AI=t¯gen\+t¯review\\bar\{t\}\_\{\\mathrm\{AI\}\}=\\bar\{t\}\_\{\\mathrm\{gen\}\}\+\\bar\{t\}\_\{\\mathrm\{review\}\}\.*Estimated*baselines \(practitioner recall\) are adequate for prioritisation;*observed*baselines from timed matched tasks[Noy and Zhang \(2023\)](https://arxiv.org/html/2609.21841#bib.bib34);[Dell’Acqua et al\. \(2023\)](https://arxiv.org/html/2609.21841#bib.bib35)are required for a value claim\. Cost per*successful*task is recorded alongside[Kapoor et al\. \(2025b\)](https://arxiv.org/html/2609.21841#bib.bib12)\.

#### Reliability\.

WithKKattempts per task andcic\_\{i\}successes,

passk^=1n​∑i=1n\(cik\)/\(Kk\),\\widehat\{\\mathrm\{pass\}^\{k\}\}=\\tfrac\{1\}\{n\}\\textstyle\\sum\_\{i=1\}^\{n\}\\binom\{c\_\{i\}\}\{k\}\\big/\\binom\{K\}\{k\},\(5\)the unbiased estimator ofPr⁡\[all​k​attempts succeed\]\\Pr\[\\text\{all \}k\\text\{ attempts succeed\}\][Yao et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib9)\. We report pass^1, 2 and 4; the decay is the reliability signature that has stagnated across frontier models[Rabanser et al\. \(2026\)](https://arxiv.org/html/2609.21841#bib.bib6)\.

#### Assurance\.

Precision and recall of a*guidance verification*step against mandatory policy items \(recall weighted more, since a missed requirement costs more than a false alarm\); for A3\+ systems, policy\-violation rate, escalation appropriateness and incidents under an adversarial suite of injected instructions in tool outputs[Debenedetti et al\. \(2024\)](https://arxiv.org/html/2609.21841#bib.bib48), following the OWASP agentic taxonomy[OWASP GenAI Security Project \(2025\)](https://arxiv.org/html/2609.21841#bib.bib49); every tool identity, permission and response is logged so the attack surface tested is the one deployed[Model Context Protocol \(2025\)](https://arxiv.org/html/2609.21841#bib.bib50)\.

#### Oversight\.

BecauseH∈σH\\in\\sigma, oversight performance is measured: the*reviewer catch rate*ρcatch\\rho\_\{\\mathrm\{catch\}\}\(seeded defects detected / seeded\), the*false acceptance rate*\(defective outputs accepted / defective\), the*unplanned intervention rate*\(runs needing intervention outside checkpoints / runs\) and review burden \(reviewer minutes per completed task\)\.

Table 2:Metric catalogue\. Grading: H human, J calibrated judge, A automatic\. Level: lowest autonomy level at which the metric is gated\.

## 5Grading and Statistical Protocol

\[Proposed protocol composed of established methods[Patwardhan et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib1);[Zheng et al\. \(2023\)](https://arxiv.org/html/2609.21841#bib.bib28);[Cohen \(1960\)](https://arxiv.org/html/2609.21841#bib.bib44);[Angelopoulos et al\. \(2023\)](https://arxiv.org/html/2609.21841#bib.bib46);[Brown et al\. \(2001\)](https://arxiv.org/html/2609.21841#bib.bib47)\]

#### Blinded expert grading\.

Graders are practitioners from the workflow, see model and human artefacts in randomised order without provenance, and score each unit againstℛ\\mathcal\{R\}; each task has≥2\\geq 2graders and a third adjudicates\. Agreement is Cohen’sκ=\(po−pe\)/\(1−pe\)\\kappa=\(p\_\{o\}\-p\_\{e\}\)/\(1\-p\_\{e\}\)[Cohen \(1960\)](https://arxiv.org/html/2609.21841#bib.bib44)or Krippendorff’sα\\alpha[Krippendorff \(2004\)](https://arxiv.org/html/2609.21841#bib.bib45); a metric enters a gate decision only ifκ≥0\.6\\kappa\\geq 0\.6, otherwise the rubric, not the system, is revised\. A system’s self\-assessment is never a gated metric: it violates blinding and inherits judge self\-preference[Gu et al\. \(2024\)](https://arxiv.org/html/2609.21841#bib.bib30)\.

#### Calibrated judge with prediction\-powered correction\.

Expert time is the binding constraint\. A judgeffscores allNNoutputs; humanshhscore a random subset ofn≪Nn\\ll N; the reported estimate is the prediction\-powered estimator[Angelopoulos et al\. \(2023\)](https://arxiv.org/html/2609.21841#bib.bib46)

θ^PPI=1N​∑i=1Nf⁡\(yi\)−1n​∑i=1n\(f⁡\(yi\)−h⁡\(yi\)\),\\hat\{\\theta\}^\{\\mathrm\{PPI\}\}=\\tfrac\{1\}\{N\}\\textstyle\\sum\_\{i=1\}^\{N\}f\(y\_\{i\}\)\\;\-\\;\\tfrac\{1\}\{n\}\\textstyle\\sum\_\{i=1\}^\{n\}\\big\(f\(y\_\{i\}\)\-h\(y\_\{i\}\)\\big\),\(6\)unbiased for the human\-graded quantity whatever the judge’s accuracy, with variance shrinking as the judge improves\. A use case can thus evaluate hundreds of outputs while paying for tens of expert gradings without laundering judge bias into the gate\. Judge prompts, model versions and calibration sets are versioned inℛ\\mathcal\{R\}; trajectory scoring follows Agent\-as\-a\-Judge[Zhuge et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib32)and is confined to what logs can support[Kirgis et al\. \(2026\)](https://arxiv.org/html/2609.21841#bib.bib20)\.

#### Sample size and intervals\.

To resolve the gapδ\\deltabetween inner and outer thresholds for a proportion nearp^\\hat\{p\},

n≥z1−α/22​p^​\(1−p^\)/δ2;n\\ \\geq\\ z\_\{1\-\\alpha/2\}^\{2\}\\,\\hat\{p\}\(1\-\\hat\{p\}\)/\\delta^\{2\};\(7\)for the credit\-memo gap of 5 points andp^≈0\.85\\hat\{p\}\\approx 0\.85,n≈196n\\approx 196units per metric\. Units cluster within documents, so cluster\-robust or bootstrap intervals over documents are reported; proportions near 0 or 1 use Wilson or Agresti–Coull intervals[Brown et al\. \(2001\)](https://arxiv.org/html/2609.21841#bib.bib47)\. Gates are taken on the conservative bound\. Tasks are runK≥4K\\geq 4times at deployment settings for A2\+ \(≥2\\geq 2for A1\); paired comparisons use McNemar or Wilcoxon with Holm correction\.

#### Freeze, parity and stress sets\.

σ\\sigmais recorded and held fixed; tuning uses a development split disjoint from the graded set\. When two models are compared, everything exceptMMis identical\. A small*stress set*probes missing or conflicting sources, “not determinable” questions, tool errors and, for A3\+, injected instructions; its results are reported separately and gated for assurance only\. Algorithm[1](https://arxiv.org/html/2609.21841#alg1)states the procedure\.

Algorithm 1eValevaluation of a use case0:

𝒰=⟨𝒯,𝒮,𝒢,ℛ,a,c⟩\\mathcal\{U\}=\\langle\\mathcal\{T\},\\mathcal\{S\},\\mathcal\{G\},\\mathcal\{R\},a,c\\rangle; candidates

Σ=\{σk\}\\Sigma=\\\{\\sigma\_\{k\}\\\}; charter

\(θout,θin,𝐰,νmax\)\(\\theta^\{\\mathrm\{out\}\},\\theta^\{\\mathrm\{in\}\},\\mathbf\{w\},\\nu\_\{\\max\}\);

E=E⁡\(a,c\)E=E\(a,c\)
0:per\-system gate,

EI\\mathrm\{EI\}, value estimate, evidence file

1:Freeze

𝒯,𝒢,ℛ\\mathcal\{T\},\\mathcal\{G\},\\mathcal\{R\}and each

σk\\sigma\_\{k\}; record versions

2:

n←n\\leftarrowEq\.[7](https://arxiv.org/html/2609.21841#S5.E7); draw

𝒯\\mathcal\{T\}stratified; build stress set

𝒯s\\mathcal\{T\}\_\{s\}
3:foreach

σk\\sigma\_\{k\}do

4:for

ti∈𝒯∪𝒯st\_\{i\}\\in\\mathcal\{T\}\\cup\\mathcal\{T\}\_\{s\},

r=1\.\.K⁡\(E\)r=1\.\.K\(E\)do

5:

\(yi​r,τi​r\)←σk​\(xi,Si\)\(y\_\{ir\},\\tau\_\{ir\}\)\\leftarrow\\sigma\_\{k\}\(x\_\{i\},S\_\{i\}\); log

τi​r\\tau\_\{ir\}, cost, latency

6:endfor

7:Judge

ffscores all units;

≥2\\geq 2blinded humans grade a random subset of size

nhn\_\{h\}
8:Compute

κ\\kappa;if

κj<0\.6\\kappa\_\{j\}<0\.6thenrevise

ℛj\\mathcal\{R\}\_\{j\}, regrade

9:

∀j\\forall j:

mj←m\_\{j\}\\leftarrowPPI \(Eq\.[6](https://arxiv.org/html/2609.21841#S5.E6)\) or human mean if

nh≥nn\_\{h\}\\geq n;

CIj←\\mathrm\{CI\}\_\{j\}\\leftarrowWilson / cluster bootstrap

10:pass^

kk\(Eq\.[5](https://arxiv.org/html/2609.21841#S4.E5)\), dispersion; assurance metrics on

𝒯s\\mathcal\{T\}\_\{s\}
11:

\(Gatek,EIk,bk\)←\(\\mathrm\{Gate\}\_\{k\},\\mathrm\{EI\}\_\{k\},b\_\{k\}\)\\leftarrowAlgorithm[2](https://arxiv.org/html/2609.21841#alg2)

12:

Vk←V\_\{k\}\\leftarrowEq\.[10](https://arxiv.org/html/2609.21841#S6.E10)with

N,ch,ℓerrN,c\_\{h\},\\ell\_\{\\mathrm\{err\}\}from charter,

ρcatch\\rho\_\{\\mathrm\{catch\}\}from oversight study

13:endfor

14:Emit scorecards; rank passing systems by EI with sensitivity band; write evidence file; register re\-evaluation triggers

## 6Decision Layer

\[Proposed\]

#### Two\-tier gating\.

Each gated metric carries two thresholds agreed*before*results are unblinded by the business owner, model owner and second\-line risk function\. The*outer*thresholdθout\\theta^\{\\mathrm\{out\}\}is the minimum viable level, anchored to the harm of failure and to current compensating controls; the*inner*thresholdθin\\theta^\{\\mathrm\{in\}\}is the target at which the use case may scale without additional controls\. Between them lies a*conditional*zone: deploy with enhanced review, sampling or scope restriction and a dated commitment to reachθin\\theta^\{\\mathrm\{in\}\}\(Figure[4](https://arxiv.org/html/2609.21841#S6.F4)\)\. Metrics are normalised relative to their thresholds,

νj=clip⁡\(σj​\(mj−θjout\)σj​\(θjin−θjout\),−1,νmax\),\\nu\_\{j\}=\\mathrm\{clip\}\\\!\\left\(\\frac\{\\sigma\_\{j\}\(m\_\{j\}\-\\theta^\{\\mathrm\{out\}\}\_\{j\}\)\}\{\\sigma\_\{j\}\(\\theta^\{\\mathrm\{in\}\}\_\{j\}\-\\theta^\{\\mathrm\{out\}\}\_\{j\}\)\},\\,\-1,\\,\\nu\_\{\\max\}\\right\),\(8\)withσj=±1\\sigma\_\{j\}=\\pm 1for higher/lower\-is\-better, soνj=0\\nu\_\{j\}=0atθout\\theta^\{\\mathrm\{out\}\},11atθin\\theta^\{\\mathrm\{in\}\}, negative on failure and capped atνmax\\nu\_\{\\max\}\(3 in the pilot\) so one exceptional metric cannot mask a weak one\. WithCIj−\\mathrm\{CI\}^\{\-\}\_\{j\}the conservative bound,

Gate\(𝒰\)=\{Scale∀j:CIj−⪰θjinConditional∀j:CIj−⪰θjout,∃j:CIj−≺θjinRejectotherwise\.\\mathrm\{Gate\}\(\\mathcal\{U\}\)=\\begin\{cases\}\\textsc\{Scale\}&\\forall j:\\ \\mathrm\{CI\}^\{\-\}\_\{j\}\\succeq\\theta^\{\\mathrm\{in\}\}\_\{j\}\\\\ \\textsc\{Conditional\}&\\forall j:\\ \\mathrm\{CI\}^\{\-\}\_\{j\}\\succeq\\theta^\{\\mathrm\{out\}\}\_\{j\},\\ \\exists j:\\ \\mathrm\{CI\}^\{\-\}\_\{j\}\\prec\\theta^\{\\mathrm\{in\}\}\_\{j\}\\\\ \\textsc\{Reject\}&\\text\{otherwise\.\}\\end\{cases\}\(9\)The rule is conjunctive by design: compensation across metrics is appropriate for ranking models, not for deciding whether a bank may rely on a system\.

![Refer to caption](https://arxiv.org/html/2609.21841v1/figs/fig4_thresholds.png)Figure 4:Two\-tier gating\.Decisions use the conservative CI bound; an interval straddling both thresholds calls for more samples, not a verdict\.
#### Efficacy Index\.

For ranking candidates or versions,EI=∑jwj​νj\\mathrm\{EI\}=\\sum\_\{j\}w\_\{j\}\\nu\_\{j\}with charter\-fixed weights encoding the risk profile, always reported with a weight\-sensitivity band and never as a substitute for the gate\. A variant penalising weak dimensions,EIG=exp⁡\(∑jwj​ln⁡\(1\+νj\)\)−1\\mathrm\{EI\}\_\{G\}=\\exp\(\\sum\_\{j\}w\_\{j\}\\ln\(1\+\\nu\_\{j\}\)\)\-1forνj\>−1\\nu\_\{j\}\>\-1, equals zero when every metric sits atθout\\theta^\{\\mathrm\{out\}\}and is pulled toward the weakest metric; divergence between EI andEIG\\mathrm\{EI\}\_\{G\}is a cheap diagnostic of an unbalanced profile\. Algorithm[2](https://arxiv.org/html/2609.21841#alg2)gives the computation\.

Algorithm 2Two\-tier gate and Efficacy Index0:

mj,CIj−,σj,θjout,θjin,wj,νmaxm\_\{j\},\\ \\mathrm\{CI\}^\{\-\}\_\{j\},\\ \\sigma\_\{j\},\\ \\theta^\{\\mathrm\{out\}\}\_\{j\},\\ \\theta^\{\\mathrm\{in\}\}\_\{j\},\\ w\_\{j\},\\ \\nu\_\{\\max\}
0:Gate, EI,

EIG\\mathrm\{EI\}\_\{G\}, binding metric

bb, flags

1:Gate

←\\leftarrowScale

2:for

j=1\.\.Jj=1\.\.Jdo

3:

νj←\\nu\_\{j\}\\leftarrowEq\.[8](https://arxiv.org/html/2609.21841#S6.E8)

4:if

CIj−\\mathrm\{CI\}^\{\-\}\_\{j\}fails

θjout\\theta^\{\\mathrm\{out\}\}\_\{j\}then

5:Gate

←\\leftarrowReject

6:elseif

CIj−\\mathrm\{CI\}^\{\-\}\_\{j\}fails

θjin\\theta^\{\\mathrm\{in\}\}\_\{j\}andGate

≠\\neqRejectthen

7:Gate

←\\leftarrowConditional

8:endif

9:if

CIj\\mathrm\{CI\}\_\{j\}straddles both thresholdsthen

10:flag

jjunder\-sampled \(Eq\.[7](https://arxiv.org/html/2609.21841#S5.E7)\)

11:endif

12:endfor

13:

b←arg⁡minj⁡νjb\\leftarrow\\arg\\min\_\{j\}\\nu\_\{j\};

EI←∑jwj​νj\\mathrm\{EI\}\\leftarrow\\sum\_\{j\}w\_\{j\}\\nu\_\{j\};

EIG←exp⁡\(∑jwj​ln⁡\(1\+νj\)\)−1\\mathrm\{EI\}\_\{G\}\\leftarrow\\exp\(\\sum\_\{j\}w\_\{j\}\\ln\(1\+\\nu\_\{j\}\)\)\-1
14:Recompute EI under each alternative

𝐰′\\mathbf\{w\}^\{\\prime\}in the charter

⇒\\Rightarrowsensitivity band

15:returnGate, EI,

EIG\\mathrm\{EI\}\_\{G\},

bb, flags

#### Value\-and\-risk model\.

For a task performedNNtimes a year at loaded hourly costchc\_\{h\}, with inference costcinferc\_\{\\mathrm\{infer\}\}per task and expected lossℓerr\\ell\_\{\\mathrm\{err\}\}if a residual error reaches a decision,

Vannual\\displaystyle V\_\{\\mathrm\{annual\}\}=N​ch​Δ​t¯\+Vquality\+Vavoided−N​cinfer−Cintegr\\displaystyle=Nc\_\{h\}\\Delta\\bar\{t\}\+V\_\{\\mathrm\{quality\}\}\+V\_\{\\mathrm\{avoided\}\}\-Nc\_\{\\mathrm\{infer\}\}\-C\_\{\\mathrm\{integr\}\}−N​ch​t¯review−Crework−N​πerr​ℓerr,\\displaystyle\\quad\-Nc\_\{h\}\\bar\{t\}\_\{\\mathrm\{review\}\}\-C\_\{\\mathrm\{rework\}\}\-N\\pi\_\{\\mathrm\{err\}\}\\ell\_\{\\mathrm\{err\}\},\(10\)πerr\\displaystyle\\pi\_\{\\mathrm\{err\}\}≈HR⋅\(1−ρcatch\),\\displaystyle\\approx\\mathrm\{HR\}\\cdot\(1\-\\rho\_\{\\mathrm\{catch\}\}\),\(11\)whereΔ​t¯=t¯manual−t¯review\\Delta\\bar\{t\}=\\bar\{t\}\_\{\\mathrm\{manual\}\}\-\\bar\{t\}\_\{\\mathrm\{review\}\},VqualityV\_\{\\mathrm\{quality\}\}andVavoidedV\_\{\\mathrm\{avoided\}\}capture outcome improvements and losses avoided, andCintegrC\_\{\\mathrm\{integr\}\},CreworkC\_\{\\mathrm\{rework\}\}the amortised integration and redo costs; each term is entered as a range andVannualV\_\{\\mathrm\{annual\}\}reported as an interval\. Review appears as a cost in Eq\.[10](https://arxiv.org/html/2609.21841#S6.E10)and as the control on expected loss throughρcatch\\rho\_\{\\mathrm\{catch\}\}in Eq\.[11](https://arxiv.org/html/2609.21841#S6.E11), so a proposal to “save time” by shortening review is automatically charged with the risk it creates\. Estimatingρcatch\\rho\_\{\\mathrm\{catch\}\}, how often a reviewer reading a fluent draft detects an unsupported number, is the least\-studied quantity in the literature; the jagged\-frontier result[Dell’Acqua et al\. \(2023\)](https://arxiv.org/html/2609.21841#bib.bib35)suggests it may be well below one and fall as trust rises \(§[9](https://arxiv.org/html/2609.21841#S9)\)\.

## 7Pilot in a Global Bank

\[Documented; sample sizes, CIs and agreement were not recorded in the pilot and are flagged\]

The framework was piloted by the innovation function of a global systemically important bank across three workflows spanning finance controls, wholesale credit and operations knowledge management \(Table[3](https://arxiv.org/html/2609.21841#S7.T3)\)\. Institution, internal platform names and identifiers are anonymised; public model names are retained\. Appendix A \(supplementary\) gives workflow, prompt and rubric detail\.

Table 3:Pilot portfolio\.### 7\.1Use Case A: control design effectiveness \(A2\)

Issue quality assurance reviews remediation of high\-severity control issues; the assessed step classifies each control in a process–risk–control matrix as compliant, partially compliant or non\-compliant against a control standard and rule library\. The A2 workflow parses control attributes \(who, what, when, where, how\), retrieves the standard, emits a per\-control judgement table and a structured export\. Thresholds and the gold set were fixed before unblinding; results are pending and none are claimed\. The charter stage’s most useful output was discovering that the original design asked the model to score its own hallucination and coherence; undereValself\-assessment is informational only\.

### 7\.2Use Case B: credit\-memo drafting \(A1\)

Analysts upload filings, research and spreads; the platform drafts one of 13 memo sections with inline citations; the analyst edits; a guidance\-verification step checks the draft against credit\-policy items\. Table[4](https://arxiv.org/html/2609.21841#S7.T4)and Figure[5](https://arxiv.org/html/2609.21841#S7.F5)report human\-graded results against pre\-agreed thresholds\. Both models clear every outer threshold\. Gemini 2\.5 has higher fidelity \(88% CP, 1\.6% HR vs\. 76%, 3\.2%\), capture and guidance precision; Llama 4 has higher guidance recall \(95% vs\. 80%\)\.

Table 4:Use Case B human\-graded results\.↓\\downarrowlower is better\. Gate column applies Eq\.[9](https://arxiv.org/html/2609.21841#S6.E9)to point estimates for illustration; CIs to be added\.†Recorded thresholds are inverted \(inner should be stricter\); retained for fidelity to the pilot and flagged for the next charter\.

![Refer to caption](https://arxiv.org/html/2609.21841v1/figs/fig5_credit_memo_results.png)Figure 5:Use Case B against two\-tier thresholds\.Dashed/dotted lines mark outer/inner thresholds\.#### Interpretation\.

Applied mechanically to point estimates, Eq\.[9](https://arxiv.org/html/2609.21841#S6.E9)places Llama 4 inScaleon all metrics and Gemini 2\.5 inConditionalbecause its recall of 80% does not strictly exceed 80%\. This is exactly what the framework should expose rather than resolve by fiat: without an interval the distinction is meaningless and the correct action is to grade more guidance items \(Figure[4](https://arxiv.org/html/2609.21841#S6.F4)\)\. The normalised view \(Figure[6](https://arxiv.org/html/2609.21841#S7.F6)\) makes a second point: with equal weights andνmax=3\\nu\_\{\\max\}=3, EI is 2\.45 for Gemini and 2\.59 for Llama; with fidelity\-weighted weights \(0\.30 HR, 0\.25 CP, 0\.15 each remaining\) the order reverses to 2\.62 vs\. 2\.37\. Which system to prefer depends on whether the credit function values fewer unsupported statements or fewer missed guidance items, a judgement the charter fixes in advance and the index makes explicit rather than hiding in an average\. The pilot did not document harness parity between the two models, so differences cannot be attributed to the models alone, and no timed efficiency comparison was run, so no efficiency result is claimed here\.

![Refer to caption](https://arxiv.org/html/2609.21841v1/figs/fig8_radar.png)Figure 6:Threshold\-normalised scoresνj\\nu\_\{j\}\(Use Case B\)\.Origin=θout=\\theta^\{\\mathrm\{out\}\}, unit circle=θin=\\theta^\{\\mathrm\{in\}\}, cap 3\. Equal\-weight and fidelity\-weighted EI rank the systems differently\.

### 7\.3Use Case C: procedure transformation \(A1, batch\)

Ten frequently used procedures were rewritten by two pipeline versions; owners estimated the effort to refine each output to publishable standard and the effort to transform it manually \(Figure[7](https://arxiv.org/html/2609.21841#S7.F7)\)\. Mean estimated manual effort was 27\.4 h/document; refinement after v1 was 10\.6 h \(−61%\-61\\%\) and after v2 2\.9 h over the seven documents with v2 outputs \(−89%\-89\\%against the ten\-document manual mean;−88%\-88\\%like\-for\-like, since the v2 subset’s manual mean is 25\.0 h\)\. The v1→\\tov2 improvement is consistent within documents, the more credible signal\. Three caveats: the baseline is estimated and clusters at three values \(17, 25, 33 h\), indicating coarse banding subject to recall bias; the v2 mean is on a subset; and fidelity was not graded, so a faster procedure that silently drops a regulatory step would carry negative value under Eq\.[10](https://arxiv.org/html/2609.21841#S6.E10)\.

![Refer to caption](https://arxiv.org/html/2609.21841v1/figs/fig6_procedure_hours.png)Figure 7:Use Case C estimated effort by document and pipeline version\.
### 7\.4Cross\-case lessons and required additions

Charters force clarity: writing thresholds first exposed an inverted pair, a self\-grading step and an unstated baseline\. Fidelity and utility trade off across models, and the trade\-off is a business judgement to be fixed in weights before results\. Efficiency claims are only as good as the baseline\. To bring the evidence to the framework’s own E2/E3 standard we will: reportnn, cluster\-robust 95% CIs andκ\\kappafor every metric and re\-issue gates on conservative bounds; runK≥4K\\geq 4repeats and report pass^\{1,2,4\}\\\{1,2,4\\\}; complete Use Case A; replace estimated with observed efficiency in a within\-subject timed design[Noy and Zhang \(2023\)](https://arxiv.org/html/2609.21841#bib.bib34);[Dell’Acqua et al\. \(2023\)](https://arxiv.org/html/2609.21841#bib.bib35)with four baselines \(unaided, assisted, model\-only, agentic with matched tooling\); add KEC grading to Use Case C; calibrate a judge and report PPI estimates; re\-run B under documented parity with a stress set; and populate Eq\.[10](https://arxiv.org/html/2609.21841#S6.E10)withρcatch\\rho\_\{\\mathrm\{catch\}\}elicited by seeding defects into review samples\.

## 8Discussion

#### What the pilot shows and does not show\.

\[Documented\]The framework is*operable*: three business functions in a large regulated institution defined tasks, gold artefacts, metrics and thresholds and, in two cases, produced results a governance forum could read\. Frontier models wrapped in retrieval and citation scaffolding reached 1\.6–3\.2% hallucination and 96–99% key\-element capture on genuine wholesale\-credit documents under human grading, consistent with GDPval[Patwardhan et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib1); and standardisation effort fell by an order of magnitude\. It does*not*yet show statistically qualified gates, repeated\-run reliability, observed efficiency or realised value\.

#### Extending to agentic systems\.

\[Proposed\]As workflows are promoted to A3, a credit agent that gathers filings, computes ratios, drafts sections and reconciles them, four things change\. The*unit of grading*becomes the trajectory as well as the artefact: plan validity, retrieval and tool\-call correctness, memory reads and writes, checkpoints and final state are logged and graded \(Appendix B \(supplementary\)\), because an agent that reaches a correct memo by an incorrect path is a production liability[Kirgis et al\. \(2026\)](https://arxiv.org/html/2609.21841#bib.bib20)\.*Reliability becomes gated*, and the vendor\-wide plateau[Rabanser et al\. \(2026\)](https://arxiv.org/html/2609.21841#bib.bib6)implies the outer threshold on pass^4 will often bind\.*Cost is gated*per successful task[Kapoor et al\. \(2025a\)](https://arxiv.org/html/2609.21841#bib.bib11);[Kapoor et al\. \(2025b\)](https://arxiv.org/html/2609.21841#bib.bib12), since reasoning modes can multiply token cost by an order of magnitude for modest gains[Roig \(2025\)](https://arxiv.org/html/2609.21841#bib.bib7)\.*Assurance expands*to policy violations, escalation and injection incidents[National Institute of Standards and Technology \(2024\)](https://arxiv.org/html/2609.21841#bib.bib38);[Debenedetti et al\. \(2024\)](https://arxiv.org/html/2609.21841#bib.bib48)\. Multi\-agent pipelines[Guo et al\. \(2024\)](https://arxiv.org/html/2609.21841#bib.bib52)additionally require each agent to be evaluated as its own use case, with end\-to\-end pass^kkcompared against the product of component values to detect interaction effects\.

#### Threats to validity\.

\[Proposed analysis\]*Construct*: sentence\-level HR and lenient CP bias in known directions; key\-element lists encode one expert view\.*Internal*: graders were not always blind; no agreement statistics; harness parity undocumented; estimated baselines\.*External*: one institution, one model generation, small task sets \- results are about the workflows, not the models, which is why a model change is a re\-evaluation trigger\.*Statistical*: withoutnnand CIs, Table[4](https://arxiv.org/html/2609.21841#S7.T4)’s gates illustrate the rule rather than establish findings\.

## 9Limitations and Hypotheses

The framework presumes a gold artefact can be authored, which holds for generation and classification workflows and less so for open\-ended advisory tasks\. Thresholds and weights are firm\-specific by design, so EI is not comparable across firms; sector reference thresholds are a desirable outcome the framework enables but cannot supply\. The conjunctive gate is conservative and its rejection rate grows with the number of gated metrics, a multiplicity effect the charter manages by gating only metrics mapped to a material harm\. The agentic extension is specified but not yet exercised\.

\[Hypothesis\]We state testable conjectures\.H1ρcatch\\rho\_\{\\mathrm\{catch\}\}for fluent, cited drafts is below 0\.7 and declines as measured HR falls \(seeded\-defect blinded review study\)\.H2Thresholds agreed independently by peer institutions fall in a narrow band, enabling sector references \(multi\-institution charter on a synthetic task set\)\.H3A calibrated judge reachesκ≥0\.6\\kappa\\geq 0\.6on HR and IR but not on KEC or value\-add\.H4For A3 agents on document workflows, pass^4 atθout\\theta^\{\\mathrm\{out\}\}binds more often than any fidelity metric\.H5Use cases passing the inner gate show measurable capacity or P&L effects within two quarters at a rate well above the 5% reported for pilots generally[Challapally et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib2)\(prospective registration; the predictive\-validity standard of[Patel et al\. \(2026\)](https://arxiv.org/html/2609.21841#bib.bib19)\)\.H6Conjunctive gating produces fewer approvals later reversed than any weighted average of the same metrics, at the cost of moreConditionaloutcomes \(retrospective portfolio classification\)\.H7pass^kkdeclines with human\-equivalent task duration[Kwa et al\. \(2025\)](https://arxiv.org/html/2609.21841#bib.bib51)unless checkpoint density scales with it\. Ablations over the components ofσ\\sigma\(retrieval, scaffold, tools, model, autonomy, review design\) are a further priority, given KAMI’s finding that tool\-message changes alone move accuracy and cost by large factors[Roig \(2025\)](https://arxiv.org/html/2609.21841#bib.bib7)\. We plan an open reference implementation of Algorithms[1](https://arxiv.org/html/2609.21841#alg1)–[2](https://arxiv.org/html/2609.21841#alg2)with a schema for Eqs\.[1](https://arxiv.org/html/2609.21841#S3.E1)–[2](https://arxiv.org/html/2609.21841#S3.E2)and a synthetic public task set\.

## 10Conclusion

Enterprise GenAI has an evaluation problem before it has a capability problem\. Public benchmarks show what frontier models can do; they cannot tell a bank whether a specific workflow is fit, reliable, safe and worth scaling on its own documents and under its own controls, and the internal evaluations that try rarely meet the standard a risk committee or a reviewer requires\.eValis an evaluation system for closing that gap: a formal, frozen evaluation object; a metric catalogue spanning fidelity, utility, efficiency, reliability, assurance and oversight; a grading protocol that scales expert judgement with calibrated automation without laundering its bias; a conjunctive two\-tier gate on conservative bounds; and a value model that puts review effort and residual risk on the same ledger as time saved\. A pilot in a global bank shows the system to be operable and yields encouraging fidelity and efficiency results, while making plain the evidence still required\. We offer the framework, pilot and hypotheses in the expectation that enterprise AI evaluation will mature as model benchmarking has \- through shared definitions, pre\-registered thresholds and honest reporting of uncertainty\.

## References

- A\. N\. Angelopoulos, S\. Bates, C\. Fannjiang, and et al\.Prediction\-powered inference\.Science382\(6671\),pp\. 669–674\.Cited by:[§5](https://arxiv.org/html/2609.21841#S5.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2609.21841#S5.p1.1.1)\.
- Bank of England Prudential Regulation Authority \(2023\)Bank of England Prudential Regulation AuthorityModel risk management principles for banks\.Technical reportSupervisory Statement SS1/23\.Cited by:[§1](https://arxiv.org/html/2609.21841#S1.p3.1),[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px5.p1.1),[§3\.4](https://arxiv.org/html/2609.21841#S3.SS4.p1.1)\.
- Barreset al\.\(2025\)V\. Barres, H\. Dong, S\. Ray, and et al\.τ2\\tau^\{2\}\-bench: evaluating conversational agents in a dual\-control environment\.arXiv preprint arXiv:2506\.07982\.Cited by:[§1](https://arxiv.org/html/2609.21841#S1.p3.1)\.
- Brownet al\.\(2001\)L\. D\. Brown, T\. T\. Cai, and A\. DasGuptaInterval estimation for a binomial proportion\.Statistical Science16\(2\),pp\. 101–133\.Cited by:[§5](https://arxiv.org/html/2609.21841#S5.SS0.SSS0.Px3.p1.2),[§5](https://arxiv.org/html/2609.21841#S5.p1.1.1)\.
- Brynjolfssonet al\.\(2025\)E\. Brynjolfsson, D\. Li, and L\. RaymondGenerative AI at work\.The Quarterly Journal of Economics140\(2\),pp\. 889–942\.Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px4.p1.1)\.
- Challapallyet al\.\(2025\)A\. Challapally, C\. Pease, R\. Raskar, and et al\.The GenAI divide: state of AI in business 2025\.Technical reportMIT NANDA\.Cited by:[§1](https://arxiv.org/html/2609.21841#S1.p1.1),[§9](https://arxiv.org/html/2609.21841#S9.p2.1)\.
- Cohen \(1960\)J\. CohenA coefficient of agreement for nominal scales\.Educational and Psychological Measurement20\(1\),pp\. 37–46\.Cited by:[§5](https://arxiv.org/html/2609.21841#S5.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.21841#S5.p1.1.1)\.
- Debenedettiet al\.\(2024\)E\. Debenedetti, J\. Zhang, and et al\.AgentDojo: a dynamic environment to evaluate prompt injection attacks and defenses for LLM agents\.InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,Vol\.37\.Cited by:[§4](https://arxiv.org/html/2609.21841#S4.SS0.SSS0.Px5.p1.1),[§8](https://arxiv.org/html/2609.21841#S8.SS0.SSS0.Px2.p1.1)\.
- Dell’Acquaet al\.\(2023\)F\. Dell’Acqua, E\. M\. III, E\. R\. Mollick, and et al\.Navigating the jagged technological frontier: field experimental evidence of the effects of AI on knowledge worker productivity and quality\.Technical reportTechnical ReportWorking Paper 24\-013,Harvard Business School\.Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px4.p1.1),[§4](https://arxiv.org/html/2609.21841#S4.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2609.21841#S6.SS0.SSS0.Px3.p1.2),[§7\.4](https://arxiv.org/html/2609.21841#S7.SS4.p1.1)\.
- Erikssonet al\.\(2025\)M\. Eriksson, E\. Purificato, A\. Noroozian, and et al\.Can we trust AI benchmarks? an interdisciplinary review of current issues in AI evaluation\.arXiv preprint arXiv:2502\.06559\.Note:A version also appears in Proceedings of AAAI/ACM AIES 2025Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px1.p1.1)\.
- Eset al\.\(2024\)S\. Es, J\. James, L\. Espinosa\-Anke, and et al\.RAGAS: automated evaluation of retrieval augmented generation\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(EACL\): System Demonstrations,Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2609.21841#S4.p1.1.1)\.
- European Parliament and Council of the European Union \(2024\)European Parliament and Council of the European UnionRegulation \(EU\) 2024/1689 laying down harmonised rules on artificial intelligence \(artificial intelligence act\)\.Note:Official Journal of the European Union, L 2024/1689, 12 July 2024Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px5.p1.1)\.
- European Parliament and Council of the European Union \(2026\)European Parliament and Council of the European UnionRegulation \(EU\) 2026/1744 of 8 july 2026 amending regulations \(EU\) 2024/1689, \(EU\) 2018/1139 and \(EU\) 2023/1230 as regards the simplification of the implementation of harmonised rules on artificial intelligence \(digital omnibus on AI\)\.Note:Official Journal of the European Union, L 2026/1744, 24 July 2026Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px5.p1.1)\.
- Gartner \(2025\)GartnerGartner predicts over 40% of agentic AI projects will be canceled by end of 2027\.Note:Press release, 25 June 2025Cited by:[§1](https://arxiv.org/html/2609.21841#S1.p1.1)\.
- Guet al\.\(2024\)J\. Gu, X\. Jiang, Z\. Shi, and et al\.A survey on LLM\-as\-a\-judge\.arXiv preprint arXiv:2411\.15594\.Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2609.21841#S5.SS0.SSS0.Px1.p1.1)\.
- Guoet al\.\(2024\)T\. Guo, X\. Chen, Y\. Wang, and et al\.Large language model based multi\-agents: a survey of progress and challenges\.InProceedings of the 33rd International Joint Conference on Artificial Intelligence \(IJCAI\),Cited by:[§8](https://arxiv.org/html/2609.21841#S8.SS0.SSS0.Px2.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Basart, and et al\.Measuring massive multitask language understanding\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px1.p1.1)\.
- Huanget al\.\(2025\)L\. Huang, W\. Yu, W\. Ma, and et al\.A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions\.ACM Transactions on Information Systems43\(2\)\.Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2609.21841#S4.SS0.SSS0.Px1.p1.2)\.
- International Organization for Standardization \(2023\)International Organization for StandardizationISO/IEC 42001:2023 information technology — artificial intelligence — management system\.Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px5.p1.1)\.
- International Organization for Standardization \(2025\)International Organization for StandardizationISO/IEC 42005:2025 information technology — artificial intelligence — AI system impact assessment\.Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px5.p1.1),[§3\.4](https://arxiv.org/html/2609.21841#S3.SS4.p1.1)\.
- Jiet al\.\(2023\)Z\. Ji, N\. Lee, R\. Frieske, and et al\.Survey of hallucination in natural language generation\.ACM Computing Surveys55\(12\)\.Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px3.p1.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, and et al\.SWE\-bench: can language models resolve real\-world GitHub issues?\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2609.21841#S1.p3.1)\.
- Kapooret al\.\(2025a\)S\. Kapoor, B\. Stroebl, P\. Kirgis, and et al\.Holistic agent leaderboard: the missing infrastructure for AI agent evaluation\.arXiv preprint arXiv:2510\.11977\.Note:A version also appears at ICLR 2026Cited by:[§1](https://arxiv.org/html/2609.21841#S1.p3.1),[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px2.p1.1),[§8](https://arxiv.org/html/2609.21841#S8.SS0.SSS0.Px2.p1.1)\.
- Kapooret al\.\(2025b\)S\. Kapoor, B\. Stroebl, Z\. S\. Siegel, and et al\.AI agents that matter\.Transactions on Machine Learning Research\.Note:Preprint arXiv:2407\.01502, 2024Cited by:[§1](https://arxiv.org/html/2609.21841#S1.p3.1),[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2609.21841#S4.SS0.SSS0.Px3.p1.1),[§8](https://arxiv.org/html/2609.21841#S8.SS0.SSS0.Px2.p1.1)\.
- Kirgiset al\.\(2026\)P\. Kirgis, S\. Kapoor, S\. Rabanser, and et al\.Log analysis is necessary for credible evaluation of AI agents\.arXiv preprint arXiv:2605\.08545\.Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2609.21841#S5.SS0.SSS0.Px2.p1.2),[§8](https://arxiv.org/html/2609.21841#S8.SS0.SSS0.Px2.p1.1)\.
- Krippendorff \(2004\)K\. KrippendorffContent analysis: an introduction to its methodology\.2nd edition,Sage,Thousand Oaks, CA\.Cited by:[§5](https://arxiv.org/html/2609.21841#S5.SS0.SSS0.Px1.p1.1)\.
- Kwaet al\.\(2025\)T\. Kwa, B\. West, J\. Becker, and et al\.Measuring AI ability to complete long software tasks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:Model Evaluation & Threat Research \(METR\)\. Preprint arXiv:2503\.14499Cited by:[§9](https://arxiv.org/html/2609.21841#S9.p2.1)\.
- Lewiset al\.\(2020\)P\. Lewis, E\. Perez, A\. Piktus, and et al\.Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems,Vol\.33\.Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px3.p1.1)\.
- Lianget al\.\(2023\)P\. Liang, R\. Bommasani, T\. Lee, and et al\.Holistic evaluation of language models\.Transactions on Machine Learning Research\.Cited by:[§1](https://arxiv.org/html/2609.21841#S1.p3.1),[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2023\)Y\. Liu, D\. Iter, Y\. Xu, and et al\.G\-Eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px3.p1.1)\.
- Mehta \(2025\)S\. MehtaBeyond accuracy: a multi\-dimensional framework for evaluating enterprise agentic AI systems\.arXiv preprint arXiv:2511\.14136\.Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px2.p1.1)\.
- Meimandiet al\.\(2025\)K\. J\. Meimandi, G\. Aránguiz\-Dias, G\. R\. Kim, and et al\.The measurement imbalance in agentic AI evaluation undermines industry productivity claims\.arXiv preprint arXiv:2506\.02064\.Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px2.p1.1)\.
- Minet al\.\(2023\)S\. Min, K\. Krishna, X\. Lyu, and et al\.FActScore: fine\-grained atomic evaluation of factual precision in long form text generation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2609.21841#S4.SS0.SSS0.Px1.p1.2),[§4](https://arxiv.org/html/2609.21841#S4.p1.1.1)\.
- Model Context Protocol \(2025\)Model Context ProtocolModel context protocol specification, revision 2025\-06\-18\.Note:[https://modelcontextprotocol\.io](https://modelcontextprotocol.io/)Cited by:[§4](https://arxiv.org/html/2609.21841#S4.SS0.SSS0.Px5.p1.1)\.
- Morriset al\.\(2024\)M\. R\. Morris, J\. Sohl\-Dickstein, N\. Fiedel, and et al\.Levels of AGI for operationalizing progress on the path to AGI\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Cited by:[§3\.4](https://arxiv.org/html/2609.21841#S3.SS4.p1.1)\.
- National Institute of Standards and Technology \(2023\)National Institute of Standards and TechnologyArtificial intelligence risk management framework \(AI RMF 1\.0\)\.Technical reportNIST AI 100\-1\.Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px5.p1.1)\.
- National Institute of Standards and Technology \(2024\)National Institute of Standards and TechnologyArtificial intelligence risk management framework: generative artificial intelligence profile\.Technical reportNIST AI 600\-1\.Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px5.p1.1),[§8](https://arxiv.org/html/2609.21841#S8.SS0.SSS0.Px2.p1.1)\.
- Noy and Zhang \(2023\)S\. Noy and W\. ZhangExperimental evidence on the productivity effects of generative artificial intelligence\.Science381\(6654\),pp\. 187–192\.Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px4.p1.1),[§4](https://arxiv.org/html/2609.21841#S4.SS0.SSS0.Px3.p1.1),[§7\.4](https://arxiv.org/html/2609.21841#S7.SS4.p1.1)\.
- OWASP GenAI Security Project \(2025\)OWASP GenAI Security ProjectOWASP top 10 for agentic applications\.Note:Released 9 December 2025 \(2026 edition\)Cited by:[§4](https://arxiv.org/html/2609.21841#S4.SS0.SSS0.Px5.p1.1)\.
- Panet al\.\(2025\)M\. Z\. Pan, N\. Arabzadeh, R\. Cogo, and et al\.Measuring agents in production\.arXiv preprint arXiv:2512\.04123\.Cited by:[§1](https://arxiv.org/html/2609.21841#S1.p1.1)\.
- Patelet al\.\(2026\)D\. C\. Patel, K\. E\. Maghraoui, S\. Lin, and et al\.Beyond static leaderboards: predictive validity for the evaluation of LLM agents\.arXiv preprint arXiv:2606\.19704\.Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px2.p1.1),[§9](https://arxiv.org/html/2609.21841#S9.p2.1)\.
- Patwardhanet al\.\(2025\)T\. Patwardhan, R\. Dias, E\. Proehl, and et al\.GDPval: evaluating AI model performance on real\-world economically valuable tasks\.arXiv preprint arXiv:2510\.04374\.Note:OpenAI\. A version also appears at ICLR 2026Cited by:[§1](https://arxiv.org/html/2609.21841#S1.p1.1),[§1](https://arxiv.org/html/2609.21841#S1.p3.1),[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.21841#S5.p1.1.1),[§8](https://arxiv.org/html/2609.21841#S8.SS0.SSS0.Px1.p1.1)\.
- Rabanseret al\.\(2026\)S\. Rabanser, S\. Kapoor, P\. Kirgis, and et al\.Towards a science of AI agent reliability\.arXiv preprint arXiv:2602\.16666\.Cited by:[§1](https://arxiv.org/html/2609.21841#S1.p3.1),[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2609.21841#S4.SS0.SSS0.Px4.p1.2),[§8](https://arxiv.org/html/2609.21841#S8.SS0.SSS0.Px2.p1.1)\.
- Rajiet al\.\(2021\)I\. D\. Raji, E\. M\. Bender, A\. Paullada, and et al\.AI and the everything in the whole wide world benchmark\.InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px1.p1.1)\.
- Roig \(2025\)J\. V\. RoigTowards a standard, enterprise\-relevant agentic AI benchmark: lessons from 5\.5 billion tokens’ worth of agentic AI evaluations\.arXiv preprint arXiv:2511\.08042\.Note:Kamiwaza Agentic Merit Index \(KAMI\) v0\.1Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px2.p1.1),[§8](https://arxiv.org/html/2609.21841#S8.SS0.SSS0.Px2.p1.1),[§9](https://arxiv.org/html/2609.21841#S9.p2.1)\.
- Shankaret al\.\(2024\)S\. Shankar, J\. D\. Zamfirescu\-Pereira, B\. Hartmann, and et al\.Who validates the validators? aligning LLM\-assisted evaluation of LLM outputs with human preferences\.InProceedings of the ACM Symposium on User Interface Software and Technology \(UIST\),Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px3.p1.1)\.
- Singlaet al\.\(2025\)A\. Singla, A\. Sukharevsky, L\. Yee, and et al\.The state of AI in 2025: agents, innovation, and transformation\.Technical reportQuantumBlack, AI by McKinsey\.Note:McKinsey Global Survey on the state of AICited by:[§1](https://arxiv.org/html/2609.21841#S1.p1.1)\.
- Vishwakarmaet al\.\(2025\)H\. Vishwakarma, A\. Agarwal, O\. Patil, and et al\.Can LLMs help you at work? a sandbox for evaluating LLM agents in enterprise environments\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Suzhou, China\.Note:Preprint arXiv:2510\.27287Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2025\)L\. Wang, D\. Yi, D\. Jose, and et al\.Enterprise large language model evaluation benchmark\.arXiv preprint arXiv:2506\.20274\.Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px2.p1.1)\.
- Yaoet al\.\(2025\)S\. Yao, N\. Shinn, P\. Razavi, and et al\.τ\\tau\-bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2609.21841#S1.p3.1),[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2609.21841#S4.SS0.SSS0.Px4.p1.2),[§4](https://arxiv.org/html/2609.21841#S4.p1.1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, and et al\.Judging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,Vol\.36\.Cited by:[§2](https://arxiv.org/html/2609.21841#S2.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2609.21841#S5.p1.1.1)\.
- Zhugeet al\.\(2025\)M\. Zhuge, C\. Zhao, D\. R\. Ashley, and et al\.Agent\-as\-a\-judge: evaluate agents with agents\.InProceedings of the 42nd International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.267,pp\. 80569–80611\.Note:Preprint arXiv:2410\.10934, 2024Cited by:[§5](https://arxiv.org/html/2609.21841#S5.SS0.SSS0.Px2.p1.2)\.

Supplementary Appendix

## Appendix APilot Detail

### A\.1Evaluation lifecycle

Figure[A1](https://arxiv.org/html/2609.21841#A1.F1)shows the seven\-stage lifecycle that Algorithm 1 of the main paper formalises; stages 5 \(grade\) and 6 \(gate\) are where internal evaluations are typically weakest\.

![Refer to caption](https://arxiv.org/html/2609.21841v1/figs/fig3_lifecycle.png)Figure A1:Evaluation lifecycle with re\-evaluation feedback loop\.
### A\.2Autonomy ladder

Figure[A2](https://arxiv.org/html/2609.21841#A1.F2)expands the five autonomy levels of §3\.4 and the metric families each adds; Table[A1](https://arxiv.org/html/2609.21841#A1.T1)lists what changes qualitatively when a workflow becomes agentic\.

![Refer to caption](https://arxiv.org/html/2609.21841v1/figs/fig7_autonomy_ladder.png)Figure A2:Autonomy levels A0–A4 and the metric families they require\.Table A1:What changes when a workflow becomes agentic, and what must therefore be measured\.
### A\.3Use Case A: workflow phases

Table A2:Use Case A: phases, AI contribution and applicable metrics\.The system specification includes two guardrails that are part of the evaluation object: no analysis begins before the input file is present and the analyst explicitly instructs the run, and outputs are exported to a structured deliverable rather than free text\. Thresholds for KEC, IR, HR, guidance precision/recall and classification pass^4 were left as “to be determined” at charter until the control owner fixed them before unblinding; grading of Gemini 2\.5 Flash and Pro outputs was in progress at submission\.

### A\.4Use Case B: generalised drafting instruction

The following is the drafting instruction for the*source of repayment*memo section, generalised for publication\. It illustrates why prompts are part of the evaluation object: the prohibition in item 4 is a policy rule whose violation is an assurance failure, and the accounting\-period requirement in item 1 defines key elements the gold specification must enumerate\.

> *Role\.*You are a corporate banking analyst preparing the source\-of\-repayment section of a credit memo for a corporate borrower seeking a revolving credit facility\. *Objective\.*Identify and document the primary, secondary and other sources of repayment with specific numerical values and explicit accounting periods\. 1. 1\.State the accounting periods used \(e\.g\., LTM ended \[date\], FY\[YYYY\], three\-year average\)\. 2. 2\.Align repayment sources with the debt structure, distinguishing short\- and long\-term tenors\. 3. 3\.Use consistent currency units and cite source materials explicitly \(financial statements, base\-case model, appraisals\)\. 4. 4\.Neverpresent revolving credit facilities, undrawn lines or short\-term borrowings as a source of repayment\. 5. 5\.*Primary source:*free cash flow \(CFO less capex\) relative to scheduled amortisation\. 6. 6\.*Secondary source:*refinancing capacity, collateral realisation \(asset values, LTV, advance rates\) or pending transactions\. 7. 7\.*Other sources:*realistic alternatives aligned to debt tenor\.

### A\.5Use Case C: per\-document effort

Table A3:Estimated effort per procedure \(hours\)\. n/a: no v2 output assessed\.DocTitle \(abbrev\.\)Manualv1v2101889Security word rules2532102123Do\-not\-release to any caller3310n/a129826Inactive and dormant accounts3310n/a130293Verifications and overrides25202130724Travel suppression33104130763Vulnerable adult incident report3316n/a131377Regulation DD25101\.5131411Regulation E33248131733Consumer fairness policy1731131926Branch investment procedures1732Mean \(n=10n\{=\}10\)27\.410\.6–Mean \(v2 subset,n=7n\{=\}7\)25\.010\.42\.9
### A\.6Grading rubric templates

Table A4:Sentence\-level fidelity and relevance rubric \(Use Cases A, B\)\.FieldInstruction to graderValuesAssertive?Does the sentence make a checkable factual claim? Headings, questions and explicit recommendations are not assertive\.Y / NSupported?For each claim, can you locate support in the permitted sources? Support from your own knowledge does*not*count\.Fully / Partially / NotRelevant?Does the sentence contribute to the instruction for this section?Y / NCitation used?Is the cited chunk’s content actually used \(strict\) or at least reflected \(lenient\)?Strict / Lenient / NotKey elementsTick each gold key element present and correct; note incorrect presentations\.checklistValue\-addCorrect, relevant content absent from the human baseline\.0 / 1 / 2*Guidance\-verification rubric*Flag correct?For each flagged guidance item, does the draft in fact breach or omit it?TP / FPMissed item?For each mandatory item not flagged, does the draft breach or omit it?FN / correctSeverityBusiness\-assigned severity of the item\.Low / Med / High

## Appendix BTrajectory\-Level Evaluation Record

Table A5:Trajectory\-level record for A3–A4 systems; each layer is logged per run and graded\.
## Appendix CGovernance Mapping

Table A6:Mapping of EnterpriseVal artefacts to governance requirements\.
## Appendix DThreshold\-Setting Protocol

Thresholds are set in a facilitated session before any results are unblinded, with the business owner, model owner, second\-line risk function and evaluation lead present\.

1. 1\.Enumerate harms\.For each candidate metric, describe the concrete harm of failure\. Metrics with no material harm are informational and not gated\.
2. 2\.Anchorθout\\theta^\{\\mathrm\{out\}\}to harm and compensating controls\.At what level does residual risk, given current review design, become unacceptable?
3. 3\.Anchorθin\\theta^\{\\mathrm\{in\}\}to scale\.At what level would the risk function accept lighter compensating controls? The gap should be at least the expected sampling uncertainty at the plannednn\(Eq\. 7\)\.
4. 4\.Fix weights andνmax\\nu\_\{\\max\}, with rationale and at least one alternative weighting for sensitivity reporting\.
5. 5\.Sign and version the charter\.Any post\-unblinding change is a recorded deviation\.

## Appendix EValue Model Decomposition

![Refer to caption](https://arxiv.org/html/2609.21841v1/figs/fig10_value_waterfall.png)Figure A3:Schematic decomposition of Eq\. 11 \(main paper\)\. Bar heights are illustrative, not pilot data\.
## Appendix FScorecard Template

Table A7:One scorecard per use case per cycle; every field is mandatory\.Use case ID; business owner; model owner; second\-line reviewerBusiness outcome and value hypothesisTask specification𝒯,𝒢,ℛ\\mathcal\{T\},\\mathcal\{G\},\\mathcal\{R\}\(versions\)Autonomy levelaa, consequence tiercc, intensityE⁡\(a,c\)E\(a,c\)Configurationσ=⟨M,P,Φ,U,Γ,H⟩\\sigma=\\langle M,P,\\Phi,U,\\Gamma,H\\ranglewith versionsBaseline workflow and measurement mode \(estimated / observed\)Gated metrics: estimate, CI,θout\\theta^\{\\mathrm\{out\}\},θin\\theta^\{\\mathrm\{in\}\},νj\\nu\_\{j\}, grading mode,κ/α\\kappa/\\alphaReliability:KK, pass^1/2/4, dispersionAssurance and stress\-set resultsOversight:ρcatch\\rho\_\{\\mathrm\{catch\}\}, FAR, intervention rate, review burdenSample sizes \(units, documents, tasks\); under\-sampled flagsGate decision and binding metricbbEI andEIG\\mathrm\{EI\}\_\{G\}with weight\-sensitivity bandValue\-model inputs \(NN,chc\_\{h\},ℓerr\\ell\_\{\\mathrm\{err\}\},ρcatch\\rho\_\{\\mathrm\{catch\}\}\) andVannualV\_\{\\mathrm\{annual\}\}intervalObserved failure modes; conditions of use; compensating controlsRe\-evaluation triggers and next review date