A Capability Framework for Evaluating Agentic AI Platforms in Regulated Enterprises
Summary
A paper reporting an industrial bakeoff evaluating ten agentic AI platforms across nine capability dimensions for regulated financial enterprises, revealing a halo effect in demonstration-based scoring and proposing a gate-then-score evaluation protocol.
View Cached Full Text
Cached at: 11/20/26, 09:13 AM
# 1Introduction
Source: [https://arxiv.org/html/2611.24039](https://arxiv.org/html/2611.24039)
marginparsep has been altered\. topmargin has been altered\. marginparwidth has been altered\. marginparpush has been altered\.
The page layout violates the MLSys style\.
Please do not change the page layout, or include packages like geometry, savetrees, or fullpage, which change it for you\.
We’re not able to reliably undo arbitrary changes to the style\. Please remove the offending package\(s\), or layout\-changing commands and try again\.
A Capability Framework for Evaluating Agentic AI Platforms in Regulated Enterprises
Abbas Raza Ali\*1Muhammad Ajmal Siddiqui2Moona Zahid3
††footnotetext:1Citigrounp Inc\., London, United Kingdom2Ernst & Young LLP, London, United Kingdom3NVIDIA Corporation, London, United Kingdom\. Correspondence to: Abbas Raza Ali <abbas\.raza\.ali@gmail\.com\>\.
Proceedings of the10th\\mathit\{10\}^\{th\}MLSys Conference, 2027\. Copyright 2027 by the author\(s\)\.###### Abstract
Agentic AI platforms\-systems in which LLM agents plan, invoke tools, retain memory and coordinate to change the state of enterprise systems\-are being procured faster than they can be measured\. Public agent benchmarks score*models*in fixed harnesses; enterprises buy integrated*platforms*whose orchestration, memory, governance and tooling layers dominate outcomes, and they buy them largely on the strength of vendor demonstrations\. We report an industrial evaluation programme \(“bakeoff”\) run by the innovation laboratory of a global bank in which ten platforms\-spanning open\-source frameworks, proprietary products and hyperscaler ecosystems\-were assessed by a cross\-functional panel on nine capability dimensions across three financial\-services workloads: counterparty credit risk, retrieval\-augmented generation and process automation\. Scores separate the field into three tiers that are stable under alternative weightings, and surface workload\-specific strengths that aggregates hide\. A secondary analysis, however, finds Cronbach’sα=0\.98\\alpha=0\.98across nominally distinct dimensions and rank correlations≥0\.93\\geq 0\.93with the total for eight of nine: a halo effect indicating that demonstration\-based scoring measures presentation as much as capability\. From these lessons we propose a*gate\-then\-score*evaluation protocol with four layers \(capability, execution reliability, governance gates, business value\), trace\-level reliability metrics estimated over repeated runs, an autonomy\-readiness scale and a reproducibility protocol\. We state precisely what the present evidence supports, and what a controlled harness must add to turn enterprise platform evaluation from structured judgement into measurement\.
## 1Introduction
Within three years, LLM agents have progressed from prompting patterns that interleave reasoning and action\([Yao et al\., 2023b](https://arxiv.org/html/2611.24039#bib.bib59);[Wei et al\., 2022](https://arxiv.org/html/2611.24039#bib.bib52)\)to systems that resolve real software issues\([Jimenez et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib22)\), operate desktop environments\([Xie et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib57)\)and coordinate as teams of specialised roles\([Wu et al\., 2023](https://arxiv.org/html/2611.24039#bib.bib55);[Hong et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib20);[Fourney et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib16)\)\. Vendors have responded with a rapidly expanding catalogue of “agent platforms”, and interoperability protocols\-the Model Context Protocol \(MCP\)\([Anthropic, 2024b](https://arxiv.org/html/2611.24039#bib.bib5)\)and Agent2Agent \(A2A\)\([Google, 2025](https://arxiv.org/html/2611.24039#bib.bib18)\)\-promise a common substrate for connecting agents to tools and to each other\. For a global bank the question is no longer whether such systems are interesting but which of them can be trusted to formulate and execute sequences of*state\-changing*actions inside a regulated operating environment\.
That question is hard for three reasons\. First, the benchmark literature measures models, not platforms: AgentBench\([Liu et al\., 2024b](https://arxiv.org/html/2611.24039#bib.bib30)\), GAIA\([Mialon et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib33)\), WebArena\([Zhou et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib62)\)andτ\\tau\-bench\([Yao et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib61)\)score an LLM’s behaviour in a fixed harness, whereas an enterprise buys a product whose orchestration, memory, governance and tooling layers it does not control\. Second, vendor claims outrun evidence:[Kapoor et al\. \(2024\)](https://arxiv.org/html/2611.24039#bib.bib24)document that many agent evaluations are irreproducible, ignore cost and overfit leaderboards\. Third, regulated institutions must satisfy model\-risk, auditability and accountability obligations\([Board of Governors of the Federal Reserve System and Office of the Comptroller of the Currency, 2011](https://arxiv.org/html/2611.24039#bib.bib15);[National Institute of Standards and Technology, 2023](https://arxiv.org/html/2611.24039#bib.bib35);[European Parliament and Council, 2024](https://arxiv.org/html/2611.24039#bib.bib14)\)that no public benchmark addresses\. Buyers therefore fall back on vendor demonstrations\-with the attendant risk that presentation quality is mistaken for engineering quality\.
This paper reports on a programme designed to confront that gap: an “Agentic AI Bakeoff” run by the innovation laboratory of a global financial institution \(hereafter “the Lab”\)\. We treat it both as an industrial case study and as an opportunity to formalise a reusable evaluation methodology, and we are explicit about which parts of the paper are\[established\],\[documented\],\[observed\],\[proposed\]or\[hypothesis\]\.
#### Research questions\.
RQ1Which system\-level capabilities, beyond model quality, determine whether an agentic platform is deployable in a regulated enterprise?RQ2How can they be measured in a manner that is scenario\-based, traceable and reproducible rather than demonstration\-based?RQ3What patterns emerge when a multi\-vendor bakeoff is scored against such a rubric across risk\-analysis, retrieval and automation workloads?RQ4What additional evidence is required to move from an industrial evaluation to a controlled benchmark study?
#### Contributions\.
\(i\) A conceptual grounding that positions agentic AI relative to RPA and single agents, with a working definition, a sequential\-decision formalism and a reference architecture mapped to the literature \(§[2](https://arxiv.org/html/2611.24039#S2)\)\. \(ii\) A nine\-dimension capability taxonomy, a multi\-stage protocol and a scoring model with reliability diagnostics \(§[4](https://arxiv.org/html/2611.24039#S4)\)\. \(iii\) Empirical results from ten platforms on three financial\-services scenarios, with a psychometric analysis of the scores and an explicit threats\-to\-validity assessment \(§[6](https://arxiv.org/html/2611.24039#S6)–[7](https://arxiv.org/html/2611.24039#S7)\)\. \(iv\) A*gate\-then\-score*layered evaluation model with trace\-level metrics and a pseudocode protocol \(§[8](https://arxiv.org/html/2611.24039#S8)\), an autonomy\-readiness scale and a business\-value identity \(§[9](https://arxiv.org/html/2611.24039#S9)\), and a research agenda specifying the controlled experiments still required\. Platform profiles, an agentic document\-extraction case study, scoring anchors and a reproducibility protocol are in the supplementary appendix \(separate PDF\)\.
## 2Background and Formalisation
### 2\.1The automation continuum
*RPA*,*AI agent*and*agentic AI*are routinely conflated\. They occupy distinct positions on a continuum defined by how much of the “what” and the “how” of a task is decided at run time rather than fixed at design time \(Fig\.[1](https://arxiv.org/html/2611.24039#S2.F1)\)\. RPA bots mimic human interaction with application UIs to execute rule\-based work\([van der Aalst et al\., 2018](https://arxiv.org/html/2611.24039#bib.bib1)\); they are deterministic and non\-invasive but brittle to UI change\. A classical agent perceives its environment and acts toward goals\([Wooldridge and Jennings, 1995](https://arxiv.org/html/2611.24039#bib.bib54);[Russell and Norvig, 2020](https://arxiv.org/html/2611.24039#bib.bib41)\)within one bounded task\. We use*agentic AI*for systems\-often multi\-agent\-that dynamically plan, adapt and collaborate toward open\-ended goals, discovering and invoking tools and maintaining memory across interactions\([Wang et al\., 2024b](https://arxiv.org/html/2611.24039#bib.bib50);[Xi et al\., 2023](https://arxiv.org/html/2611.24039#bib.bib56)\)\. A further practical distinction: an*agentic workflow*fixes control flow at design time and lets the model fill adaptive steps, whereas an*agentic system*lets the model direct control flow\([Anthropic, 2024a](https://arxiv.org/html/2611.24039#bib.bib4)\)\. Most enterprise deployments we observed were workflows with agentic steps; the evaluation problem is partly a design\-selection problem in which the target is the*minimum*autonomy that reliably achieves the outcome\.
Figure 1:The automation continuum\. Rightward movement increases run\-time autonomy, the scope of delegated decisions and, consequently, the governance burden\.
### 2\.2Working definition
The Lab’s parent institution defines agentic AI as systems that*achieve goals on behalf of a user or another system by autonomously formulating and executing a sequence of actions that invoke other systems and change their state*\. Two clauses carry the weight: \(a\) autonomous formulation and execution of an action sequence, and \(b\) state\-changing invocation of other systems\. Clause \(b\) separates agentic AI from conversational assistants and raises its risk profile: errors propagate into records, workflows and downstream decisions\([Chan et al\., 2023](https://arxiv.org/html/2611.24039#bib.bib9);[Shavit et al\., 2023](https://arxiv.org/html/2611.24039#bib.bib44)\)\.
### 2\.3A formal characterisation
In the vocabulary of sequential decision\-making\([Kaelbling et al\., 1998](https://arxiv.org/html/2611.24039#bib.bib23);[Russell and Norvig, 2020](https://arxiv.org/html/2611.24039#bib.bib41)\), an agentic system operates in an environment with latent states∈𝒮s\\in\\mathcal\{S\}, receives observationso∈𝒪o\\in\\mathcal\{O\}and selects actionsa∈𝒜=𝒜tool∪𝒜msg∪𝒜texta\\in\\mathcal\{A\}=\\mathcal\{A\}\_\{\\text\{tool\}\}\\cup\\mathcal\{A\}\_\{\\text\{msg\}\}\\cup\\mathcal\{A\}\_\{\\text\{text\}\}\(tool invocations, messages to agents or humans, text outputs\)\. A single agent is a policy with explicit memory statemtm\_\{t\}:
at=π\(ot,mt,θ\),mt\+1=μ\(mt,ot,at\),a\_\{t\}=\\pi\(o\_\{t\},m\_\{t\};\\theta\),\\qquad m\_\{t\+1\}=\\mu\(m\_\{t\},o\_\{t\},a\_\{t\}\),\(1\)whereμ\\muis a memory\-management function \(in LLM agents, context management plus external stores\([Packer et al\., 2023](https://arxiv.org/html/2611.24039#bib.bib37);[Sumers et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib48)\)\) andθ\\thetathe underlying model\. Tool actions induce transitionsst\+1∼T\(⋅∣st,at\)s\_\{t\+1\}\\sim T\(\\cdot\\mid s\_\{t\},a\_\{t\}\), formalising clause \(b\)\. Planning selectsa1…ana\_\{1\}\\ldots a\_\{n\}to maximise expected goal attainment; reflection is an inner loop in which the agent evaluates its trajectory and revises behaviour through verbal feedback rather than gradient updates\([Shinn et al\., 2023](https://arxiv.org/html/2611.24039#bib.bib45)\)\. A multi\-agent system is a set\{πi\}\\\{\\pi^\{i\}\\\}with communication topologyΓ\\Gammaand an orchestration function routing sub\-goals; most multi\-agent failures arise in specification and coordination rather than in individual reasoning\([Cemri et al\., 2025](https://arxiv.org/html/2611.24039#bib.bib8)\)\. The formalism is standard; its role here is to state which component each evaluation dimension probes\.
### 2\.4Reference architecture
Figure[2](https://arxiv.org/html/2611.24039#S2.F2)abstracts the platforms we evaluated, consistent with the cognitive\-architecture framing of[Sumers et al\. \(2024\)](https://arxiv.org/html/2611.24039#bib.bib48)and recent pattern surveys\([Masterman et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib31);[Anthropic, 2024a](https://arxiv.org/html/2611.24039#bib.bib4)\)\. Each component maps to an active research thread: reasoning\([Wei et al\., 2022](https://arxiv.org/html/2611.24039#bib.bib52);[Yao et al\., 2023a](https://arxiv.org/html/2611.24039#bib.bib60);[Yao et al\., 2023b](https://arxiv.org/html/2611.24039#bib.bib59)\); planning\([Wang et al\., 2024a](https://arxiv.org/html/2611.24039#bib.bib51)\); tool use\([Schick et al\., 2023](https://arxiv.org/html/2611.24039#bib.bib43);[Patil et al\., 2023](https://arxiv.org/html/2611.24039#bib.bib39);[Qin et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib40);[Yan et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib58)\); memory, motivated by finite and unevenly used context\([Liu et al\., 2024a](https://arxiv.org/html/2611.24039#bib.bib29);[Packer et al\., 2023](https://arxiv.org/html/2611.24039#bib.bib37);[Park et al\., 2023](https://arxiv.org/html/2611.24039#bib.bib38)\); reflection, whose unaided form is unreliable without external feedback\([Shinn et al\., 2023](https://arxiv.org/html/2611.24039#bib.bib45);[Huang et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib21)\); multi\-agent orchestration\([Wu et al\., 2023](https://arxiv.org/html/2611.24039#bib.bib55);[Hong et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib20);[Li et al\., 2023](https://arxiv.org/html/2611.24039#bib.bib28);[Fourney et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib16)\); and a governance plane demanded by risk frameworks\([National Institute of Standards and Technology, 2023](https://arxiv.org/html/2611.24039#bib.bib35);[National Institute of Standards and Technology, 2024](https://arxiv.org/html/2611.24039#bib.bib36);[European Parliament and Council, 2024](https://arxiv.org/html/2611.24039#bib.bib14)\)and by the literature on agentic harms, including indirect prompt injection through tool outputs\([Greshake et al\., 2023](https://arxiv.org/html/2611.24039#bib.bib19);[Debenedetti et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib11)\)\.
Figure 2:Reference architecture for an enterprise agentic platform\. The dashed boundary is the governance and control plane that regulated deployments must wrap around every component\.
## 3Related Work
Agent evaluation has developed along two tracks\.*Model\-level*benchmarks\-AgentBench, GAIA, WebArena, SWE\-bench, OSWorld\-fix a harness and vary the model\.*Enterprise\-workload*benchmarks are more recent: WorkArena evaluates knowledge\-work tasks on a live ServiceNow instance\([Drouin et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib12)\);τ\\tau\-bench adds multi\-turn user interaction under policy constraints and the passkreliability metric\([Yao et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib61)\); AgentArch compares agent*architectures*on enterprise tasks and reports at most≈\\approx35% success on its harder task\([Bogavelli et al\., 2025](https://arxiv.org/html/2611.24039#bib.bib3)\); EnterpriseOps\-Gym tests multi\-step operations with irreversible actions\([ServiceNow and Artificial Analysis, 2026](https://arxiv.org/html/2611.24039#bib.bib13)\); DRBench targets enterprise deep research\([Abaskohi et al\., 2026](https://arxiv.org/html/2611.24039#bib.bib2)\); and a multi\-dimensional evaluation framework for enterprise agents has been proposed\([Mehta, 2025](https://arxiv.org/html/2611.24039#bib.bib7)\)\. AgentDojo adds adversarial robustness, including banking tasks\([Debenedetti et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib11)\)\. Measurement\-quality critiques\([Kapoor et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib24);[Stroebl et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib47)\)and the long\-horizon analysis of[Kwa et al\. \(2025\)](https://arxiv.org/html/2611.24039#bib.bib26)motivate our reliability layer\. This paper is complementary: it is a*buyer\-side, platform\-level*evaluation of commercial products as delivered, inside a regulated institution, with the organisational and governance attributes that public benchmarks abstract away\. Its contribution is the protocol and its diagnostics rather than a new task suite\.
## 4Evaluation Framework
Figure[3](https://arxiv.org/html/2611.24039#S4.F3)summarises the protocol\. Solid stages were executed in the bakeoff; the dashed stage is the extension proposed in §[8](https://arxiv.org/html/2611.24039#S8)\.
Figure 3:Multi\-stage evaluation protocol\. Each stage narrows the candidate pool while increasing evidential depth and per\-vendor cost\. Solid: executed; dashed: proposed extension\.### 4\.1Stage 1: organisational screening
A landscape scan and desk assessment reduced a long list to ten candidates\. Each was profiled on eight non\-technical attributes\-company age, funding stage and amount, key investors, client base, SDKs, deployment options \(cloud/on\-premise/hybrid\) and open\-source status\-as proxies for vendor viability and integration risk in a multi\-year regulated commitment\.\[proposed\]
### 4\.2Stage 2: the nine\-dimension capability taxonomy
Table[1](https://arxiv.org/html/2611.24039#S4.T1)defines nine dimensions, the guiding question and the architectural component each probes\. The mapping lets the rubric be read as more than a checklist: a low reflection score is a statement about the critic loop of Eq\. \([1](https://arxiv.org/html/2611.24039#S2.E1)\), not merely a missing feature\. Two choices merit comment\. D7 treats protocol interoperability as first\-class, because a bank with dozens of agent initiatives cannot afford bespoke integration per platform\. D8 and D9 are where vendor evidence is weakest and the literature most insistent\([Kapoor et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib24);[Stroebl et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib47)\); we recommend requiring artefacts \(traces, evaluation reports\) rather than verbal assurance\.
Table 1:Nine\-dimension capability taxonomy\[proposed\], mapped to the reference architecture and the literature each dimension probes\.
### 4\.3Stage 3: scenario design
Three scenarios span bespoke, regulation\-bound analysis to generic, high\-volume automation\.Counterparty credit risk \(CCR\), formulated with business and technical SMEs, combines structured exposure data, unstructured documentation and regulatory logic\([Basel Committee on Banking Supervision, 2011](https://arxiv.org/html/2611.24039#bib.bib6)\); because vendors are not risk practitioners, the objective was explicitly*adaptability*to a specialised domain, not output correctness \(a limitation we return to\)\.RAG\([Lewis et al\., 2020](https://arxiv.org/html/2611.24039#bib.bib27);[Gao et al\., 2023](https://arxiv.org/html/2611.24039#bib.bib17)\)underpins most enterprise agent workloads and exposes differences in retrieval, grounding and verifiability\.Process automationexercises the defining property of agentic AI: goal attainment from minimal instruction within guardrails, with multi\-step orchestration and exception handling\. CCR stresses D3, D4, D9; RAG stresses D2, D3, D8; automation stresses D1, D2, D4, D5, D7\.
### 4\.4Scoring model and reliability diagnostics
Each raterrrin a cross\-functional panelℛ\\mathcal\{R\}\(technologists, architects, developers, SMEs, business stakeholders\) assignedxv,d,u,r∈\{1,…,10\}x\_\{v,d,u,r\}\\in\\\{1,\\dots,10\\\}to vendorvvon dimensionddfor scenariouu\. Scores were aggregated by unweighted means,
sv,d=1\|ℛv\|∑r1\|𝒰\|∑uxv,d,u,r,Sv=∑d=19wdsv,d,s\_\{v,d\}=\\frac\{1\}\{\|\\mathcal\{R\}\_\{v\}\|\}\\sum\_\{r\}\\frac\{1\}\{\|\\mathcal\{U\}\|\}\\sum\_\{u\}x\_\{v,d,u,r\},\\qquad S\_\{v\}=\\sum\_\{d=1\}^\{9\}w\_\{d\}\\,s\_\{v,d\},\(2\)withwd=1w\_\{d\}=1soSv∈\[9,90\]S\_\{v\}\\in\[9,90\]\. Equal weights are defensible for landscape education; adoption decisions should elicit workload\-specific weights, e\.g\. by pairwise comparison\([Saaty, 1980](https://arxiv.org/html/2611.24039#bib.bib42)\)\(§[6\.4](https://arxiv.org/html/2611.24039#S6.SS4)\)\. Because ratings are subjective, the protocol should report inter\-rater agreement\-ICC\(2,kk\)\([Shrout and Fleiss, 1979](https://arxiv.org/html/2611.24039#bib.bib46)\)or Krippendorff’sα\\alpha\([Krippendorff, 2004](https://arxiv.org/html/2611.24039#bib.bib25)\)\-and internal consistency across dimensions\([Cronbach, 1951](https://arxiv.org/html/2611.24039#bib.bib10)\)\. Very high consistency across conceptually distinct dimensions is diagnostic of a*halo effect*\([Thorndike, 1920](https://arxiv.org/html/2611.24039#bib.bib49)\); we test for it in §[6\.3](https://arxiv.org/html/2611.24039#S6.SS3)\. Per\-rater scores were not retained, so inter\-rater reliability could not be computed\-a required improvement\.
## 5Vendor Landscape
Ten platforms participated \(Table[2](https://arxiv.org/html/2611.24039#S5.T2)\); one invited open\-source framework \(LangChain/LangGraph\) declined and can be assessed independently in future work\. A salient axis is the spectrum of developer involvement: pro\-code SDKs for control and bespoke integration versus low/no\-code builders for “citizen developers”\. Vendors that span the spectrum*with lineage*between modes\-prototype in a visual studio, complete in the SDK\-preserve both accessibility and auditability of the final artefact, which we regard as the most enterprise\-friendly pattern\.
Table 2:Evaluated platforms\[documented\]; vendor material current at evaluation\. Microsoft was scored but not profiled in the original programme; its entry reflects public documentation\.
## 6Results
### 6\.1Aggregate scores and a three\-tier structure
Table[3](https://arxiv.org/html/2611.24039#S6.T3)and Fig\.[4](https://arxiv.org/html/2611.24039#S6.F4)report mean panel scores\. They derive exclusively from live demonstrations and accompanying presentations under assisted conditions \(§[7](https://arxiv.org/html/2611.24039#S7)\); they reflect\[observed\]performance in the Lab’s context, not market reputation\. Totals cluster into three tiers separated by gaps far larger than within\-tier differences \(Fig\.[5](https://arxiv.org/html/2611.24039#S6.F5)a\):Tier 1\(61\.7–69\.2\) H2O\.ai, Google, Microsoft, Writer, Letta;Tier 2\(41\.4–43\.0\) Glean, CrewAI, UiPath;Tier 3\(32\.0–37\.0\) Portia AI, Squid AI\. The 18\.7\-point gap between Tiers 1 and 2 dwarfs the 7\.5\-point spread within Tier 1\. The numerical ranking coincided with the consensus reached in panel deliberation\-reassuring, but not independent, since the same people produced both\.
Table 3:Mean panel scores \(1–10\) by platform and dimension, ordered by total \(max 90\)\. Dimension codes as in Table[1](https://arxiv.org/html/2611.24039#S4.T1)\.Figure 4:Heat map of mean panel scores; horizontal rules mark the three tiers\.Figure 5:\(a\) Aggregate scores with tier colouring\. \(b\) Cross\-vendor standard deviation per dimension \(discriminative power\);μ\\muis the cross\-vendor mean\.
### 6\.2Dimension\-level patterns
Four patterns are visible \(Table[3](https://arxiv.org/html/2611.24039#S6.T3), Fig\.[5](https://arxiv.org/html/2611.24039#S6.F5)b; radar profiles in Appendix A \(supplementary PDF\)\)\.Multi\-agent \(D5\) and protocol support \(D7\) favour hyperscalers\.Google recorded the highest cell in the matrix \(8\.86 on D5\) and Microsoft the highest D7 \(8\.17\), consistent with both firms’ stewardship of A2A and MCP investment\.Memory \(D3\) favours the research\-derived platform\.Letta, built on MemGPT, achieved the top memory score despite a mid\-Tier\-1 total, showing that the taxonomy surfaces specialist strength that aggregates obscure\.Ease of use \(D6\) is the least discriminating and most independent dimension\(SD 1\.28;ρ=0\.66\\rho=0\.66with total versus≥0\.93\\geq 0\.93for every other dimension\)\. Microsoft’s D6 \(5\.00\) was the lowest in Tier 1 and UiPath’s \(6\.40\) the highest in Tier 2\-evidence that raters could and did decouple usability from capability\.Reflection \(D1\) and evaluation/explainability \(D9\) have the lowest means and widest ranges\.These are the dimensions on which the literature finds systems weakest\([Huang et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib21);[Kapoor et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib24)\)and regulators are most demanding; they should carry heavier weights in adoption decisions\.
### 6\.3Internal consistency and the halo hypothesis
Treating the nine dimensions as items and ten platforms as cases, Cronbach’sα=0\.98\\alpha=0\.98and eight of nine dimensions correlate with the total atρ≥0\.93\\rho\\geq 0\.93\. Genuinely distinct capabilities\-memory management and protocol support, say\-should not co\-vary this tightly across products of such different heritage\. The parsimonious explanation is a strong common factor driven by overall demonstration impression: a halo effect\([Thorndike, 1920](https://arxiv.org/html/2611.24039#bib.bib49)\)\. This does not invalidate the tiering, which deliberation corroborated, but it implies the rubric as administered measured roughly one\-and\-a\-half constructs rather than nine\. Two protocol changes follow: require dimension\-specific evidence artefacts, and score dimensions in separate sessions or by specialist sub\-panels to break the shared impression\.
### 6\.4Weighting sensitivity
Totals were recomputed under a*governance\-first*profile \(D9×\\times2, D7×\\times1\.5, D6×\\times0\.5\) and a*developer\-first*profile \(D6×\\times2, D2×\\times1\.5, D9×\\times0\.5\), each normalised to sum to nine\. Tier membership was unchanged under both\. Within\-tier order shifted modestly \(developer\-first: Writer 63\.7 overtakes Microsoft 61\.1; UiPath 43\.5 overtakes CrewAI and Glean\)\. Tiers, not ranks, are the robust finding\.
### 6\.5Scenario\-level disaggregation
For the top three platforms \(supplementary Appendix B, Table A1\), H2O\.ai’s profile is flat across scenarios with a peak on D9 in CCR \(8\.75\); Google’s D5 is uniformly high \(8\.67–9\.00\) while its D6 is uniformly capped at 6\.00\-a systematic rather than scenario\-specific concern; Microsoft’s scores are almost invariant across scenarios, which is itself informative: raters saw little scenario\-specific adaptation\.
## 7Threats to Validity
Table[4](https://arxiv.org/html/2611.24039#S7.T4)states the threats precisely rather than claiming more than the evidence supports\. Two deserve emphasis\.*Assisted conditions*: the Lab helped vendors prepare, so the object measured is demonstrated capability under assisted conditions, not out\-of\-the\-box performance\.*Unequal panels*: score granularity implies panels of roughly 3–10 raters per vendor \(e\.g\.7\.44=67/97\.44=67/9,4\.33=13/34\.33=13/3; an inference from decimal patterns\), inflating variance for small panels\.
Table 4:Threats to validity and mitigations\.
## 8A Gate\-then\-Score Layered Model
The nine dimensions describe*capability*\. Deployment additionally requires evidence about*execution reliability*,*risk and governance*and*business value*, and these are not interchangeable: no capability score should compensate for a failed authorisation control\. We propose the layered model of Fig\.[6](https://arxiv.org/html/2611.24039#S8.F6): Layer 3 is evaluated first as non\-compensatory gates, Layers 1–2 are scored within the admissible set, Layer 4 is validated in pilot\.\[proposed\]
Figure 6:Layered evaluation model\. Gates are pass/fail and precede scoring; business value is measured in pilot, not demonstration\.#### Decision rule\.
Let𝒢=\{G1,…,GK\}\\mathcal\{G\}=\\\{G\_\{1\},\\dots,G\_\{K\}\\\}be binary gates \(authorisation correctness, audit\-trace completeness, injection resistance, data residency, human\-approval policy, minimum reliability\),xv,d=\(sv,d−1\)/9∈\[0,1\]x\_\{v,d\}=\(s\_\{v,d\}\-1\)/9\\in\[0,1\]normalised scores, andwρw\_\{\\rho\}a weight vector for risk tierρ\\rhowith∑dwρ,d=1\\sum\_\{d\}w\_\{\\rho,d\}=1\. Then
EAS\(v,ρ\)\\displaystyle\\mathrm\{EAS\}\(v,\\rho\)=∑dwρ,dxv,d,\\displaystyle=\\textstyle\\sum\_\{d\}w\_\{\\rho,d\}\\,x\_\{v,d\},\(3\)v⋆\\displaystyle v^\{\\star\}=argmaxv:Gk\(v\)=1∀kEAS\(v,ρ\)\\displaystyle=\\operatorname\*\{arg\\,max\}\_\{v:\\,G\_\{k\}\(v\)=1\\ \\forall k\}\\mathrm\{EAS\}\(v,\\rho\)s\.t\.C\(v\)≤Cmax,L\(v\)≤Lmax,\\displaystyle\\text\{s\.t\.\}\\ C\(v\)\\leq C\_\{\\max\},\\ L\(v\)\\leq L\_\{\\max\},withC,LC,Lcost and latency constraints\. Eq\. \([3](https://arxiv.org/html/2611.24039#S8.E3)\) replaces an unconstrained ranking with a constrained optimisation; a platform failing any gate is inadmissible regardless of EAS\. Applied retrospectively, the bakeoff evaluated Layer 1 only\.
#### Trace\-level reliability \(Layer 2\)\.
Trajectories are stochastic, so Layer\-2 metrics must be estimated fromKKrepeated runs with uncertainty\. For taskτ\\tauwith outcomesyk∈\{0,1\}y\_\{k\}\\in\\\{0,1\\\}and per\-run costckc\_\{k\},
p^τ=1K∑kyk,passk≈p^τk,CoPτ=∑kck∑kyk,\\hat\{p\}\_\{\\tau\}=\\tfrac\{1\}\{K\}\\textstyle\\sum\_\{k\}y\_\{k\},\\quad\\mathrm\{pass\}^\{k\}\\approx\\hat\{p\}\_\{\\tau\}^\{\\,k\},\\quad\\mathrm\{CoP\}\_\{\\tau\}=\\frac\{\\sum\_\{k\}c\_\{k\}\}\{\\sum\_\{k\}y\_\{k\}\},\(4\)with a Wilson 95% interval onp^τ\\hat\{p\}\_\{\\tau\}\([Wilson, 1927](https://arxiv.org/html/2611.24039#bib.bib53)\), preferred for smallKK; passkmeasures consistency rather than best\-case ability\([Yao et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib61)\), and cost\-of\-pass prices a success inclusive of failed attempts\([Kapoor et al\., 2024](https://arxiv.org/html/2611.24039#bib.bib24)\)\. Table[5](https://arxiv.org/html/2611.24039#S8.T5)defines the metric set; every metric is computed from the execution trace, so trace retention is a precondition\. Tools must be tagged*before*evaluation with permission scope, statefulness, reversibility and side\-effect severity\-a read\-only search and a payment API are not equivalent actions\([Shavit et al\., 2023](https://arxiv.org/html/2611.24039#bib.bib44);[National Institute of Standards and Technology, 2024](https://arxiv.org/html/2611.24039#bib.bib36)\)\.
Table 5:Trace\-level metrics for execution reliability and governance gates\[proposed\]\.Algorithm 1Gate\-then\-score enterprise evaluation protocol0:platforms
VV; scenarios
UUwith success predicates; gates
𝒢\\mathcal\{G\}; tier
ρ\\rho; weights
wρw\_\{\\rho\}; runs
KK
0:admissible set
AAranked by EAS with Layer\-2 CIs
1:for
v∈Vv\\in Vdo\{Stages 1–2\}
2:record organisational attributes; collect per\-dimension evidence artefacts
3:
s\[v,d\]←s\[v,d\]\\leftarrowpanel median of anchored ratings for each
dd; report IRR
4:endfor
5:for
v∈V,u∈Uv\\in V,\\ u\\in Udo\{Stages 3, 5: controlled harness\}
6:pin model, prompts, tool schemas, data; tag tools \(scope, reversibility, severity\)
7:for
k=1\.\.Kk=1\.\.Kdo
8:run
vvon
uu; store trace
t\[v,u,k\]t\[v,u,k\];
yk←predicate\(final state\)y\_\{k\}\\leftarrow\\mathrm\{predicate\}\(\\text\{final state\}\)
9:endfor
10:compute
p^\\hat\{p\}, passk, CoP, recovery, unsafe\-action rate from traces \{Eq\. \([4](https://arxiv.org/html/2611.24039#S8.E4)\)\}
11:run adversarial variants of
uu\(indirect injection, authorisation probes\)
12:endfor
13:
A←\{v:Gk\(v\)=1∀k\}A\\leftarrow\\\{v:G\_\{k\}\(v\)=1\\ \\forall k\\\}\{non\-compensatory gates\}
14:for
v∈Av\\in Ado
15:
EAS\[v\]←∑dwρ,d\(s\[v,d\]−1\)/9\\mathrm\{EAS\}\[v\]\\leftarrow\\sum\_\{d\}w\_\{\\rho,d\}\(s\[v,d\]\-1\)/9; attach Layer\-2 CIs
16:endfor
17:shortlist
←\\leftarrowtop of
AAs\.t\.
C\(v\)≤CmaxC\(v\)\\leq C\_\{\\max\},
L\(v\)≤LmaxL\(v\)\\leq L\_\{\\max\}
18:pilot shortlist; measure Layer\-4 terms \(Eq\. \([5](https://arxiv.org/html/2611.24039#S9.E5)\)\); repeat lines 5–10 on version change
Algorithm[1](https://arxiv.org/html/2611.24039#alg1)states the protocol\. Lines 1\-4 correspond to what the bakeoff executed; lines 5–16 are the proposed extension\. The asymmetry is deliberate: gating before scoring means a platform can be excluded on a single control failure, which is correct for state\-changing systems in a regulated setting\.
## 9Discussion
### 9\.1Connecting capability to business value
Layer 4 asks whether a workflowjjis worth deploying at all\. We propose the accounting identity
NVj=Tj\+Ej\+Rj−Cjcompute−Cjtools−Oj−𝔼\[ℓj\],\\mathrm\{NV\}\_\{j\}=T\_\{j\}\+E\_\{j\}\+R\_\{j\}\-C^\{\\text\{compute\}\}\_\{j\}\-C^\{\\text\{tools\}\}\_\{j\}\-O\_\{j\}\-\\mathbb\{E\}\[\\ell\_\{j\}\],\(5\)whereTTis labour time saved at loaded cost,EEavoided error cost,RRincremental revenue,CCcompute and tooling cost,OOoversight cost \(approvals, exceptions, audit\) and𝔼\[ℓ\]\\mathbb\{E\}\[\\ell\]the risk\-adjusted expected loss from erroneous state changes\. In financial services the last two terms often dominate: an automation that saves analyst time but raises the probability of a material misstatement has negative net value\. All terms are measured from pilot data; none can be read from a demonstration\.\[hypothesis\]
### 9\.2Lessons for practitioners
Three changes raise the evidential standard of enterprise bakeoffs without prohibitive cost: \(1\) move from vendor\-run to buyer\-run demonstrations on controlled infrastructure with pinned task definitions; \(2\) require dimension\-specific artefacts\-trace logs, evaluation dashboards, memory dumps\-so raters score evidence, not narration; \(3\) retain per\-rater data and report reliability statistics alongside means\. With these, the taxonomy becomes a repeatable instrument rather than a structured opinion survey\.
## 10Conclusion
Enterprises are adopting a class of technology whose defining property\-autonomous, state\-changing action\-is what regulated institutions must control most tightly\. We formalised a bakeoff into a reusable protocol: an organisational screen, a nine\-dimension taxonomy grounded in the literature, a scenario portfolio from bespoke analysis to autonomous execution, and a scoring model with reliability diagnostics\. Applied to ten platforms it produced a robust three\-tier structure in which hyperscaler ecosystems and research\-derived platforms led, while exposing\-throughα=0\.98\\alpha=0\.98\-the halo risk inherent in demonstration\-based scoring\. The specialist strengths it surfaced \(memory at Letta, protocol depth at Microsoft, evaluation tooling at H2O\.ai, action grounding at UiPath\) are a reminder that “best platform” is workload\-relative\. We then extended the rubric into a gate\-then\-score layered model with trace\-level metrics and a reproducibility protocol so the next iteration can measure rather than rate\. The most consequential finding is procedural: current enterprise evaluation practice measures presentation more than capability, and the remedy\-buyer\-controlled tasks, evidence artefacts and measured reliability\-is within reach\.
## References
- A\. Abaskohi, T\. Chen, M\. Muñoz\-Mármol, and et al\.DRBench: a realistic benchmark for enterprise deep research\.InInternational Conference on Learning Representations,Cited by:[§3](https://arxiv.org/html/2611.24039#S3.p1.1)\.
- Anthropic \(2024a\)AnthropicBuilding effective agents\.Note:[https://www\.anthropic\.com/research/building\-effective\-agents](https://www.anthropic.com/research/building-effective-agents)Cited by:[§2\.1](https://arxiv.org/html/2611.24039#S2.SS1.p1.1),[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1),[Table 1](https://arxiv.org/html/2611.24039#S4.T1.4.7.4.1.1)\.
- Anthropic \(2024b\)AnthropicIntroducing the Model Context Protocol\.Note:[https://www\.anthropic\.com/news/model\-context\-protocol](https://www.anthropic.com/news/model-context-protocol)Cited by:[§1](https://arxiv.org/html/2611.24039#S1.p1.1),[Table 1](https://arxiv.org/html/2611.24039#S4.T1.4.8.4.1.1)\.
- Basel Committee on Banking Supervision \(2011\)Basel Committee on Banking SupervisionBasel III: a global regulatory framework for more resilient banks and banking systems\.Technical reportBank for International Settlements\.Cited by:[§4\.3](https://arxiv.org/html/2611.24039#S4.SS3.p1.1)\.
- Board of Governors of the Federal Reserve System and Office of the Comptroller of the Currency \(2011\)Board of Governors of the Federal Reserve System and Office of the Comptroller of the CurrencySupervisory guidance on model risk management\.Technical reportTechnical ReportSR 11\-7 / OCC 2011\-12,Federal Reserve\.Cited by:[§1](https://arxiv.org/html/2611.24039#S1.p2.1)\.
- Bogavelliet al\.\(2025\)T\. Bogavelli, R\. Sharma, and H\. SubramaniAgentArch: a comprehensive benchmark to evaluate agent architectures in enterprise\.arXiv preprint arXiv:2509\.10769\.Cited by:[§3](https://arxiv.org/html/2611.24039#S3.p1.1)\.
- Cemriet al\.\(2025\)M\. Cemri, M\. Z\. Pan, S\. Yang, and et al\.Why do multi\-agent LLM systems fail?\.arXiv preprint arXiv:2503\.13657\.Cited by:[§2\.3](https://arxiv.org/html/2611.24039#S2.SS3.p1.2),[Table 1](https://arxiv.org/html/2611.24039#S4.T1.4.6.4.1.1)\.
- Chanet al\.\(2023\)A\. Chan, R\. Salganik, A\. Markelius, and et al\.Harms from increasingly agentic algorithmic systems\.InProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency,pp\. 651–666\.External Links:[Document](https://dx.doi.org/10.1145/3593013.3594033)Cited by:[§2\.2](https://arxiv.org/html/2611.24039#S2.SS2.p1.1)\.
- CrewAI \(2026\)CrewAICrewAI documentation: agents, crews, flows, memory and enterprise\.Note:[https://docs\.crewai\.com/](https://docs.crewai.com/)Accessed October 2026Cited by:[B\.7 CrewAI\.](https://arxiv.org/html/2611.24039#Ax1.SSx2.SSS0.Px7.p1.1)\.
- Cronbach \(1951\)L\. J\. CronbachCoefficient alpha and the internal structure of tests\.Psychometrika16\(3\),pp\. 297–334\.Cited by:[§4\.4](https://arxiv.org/html/2611.24039#S4.SS4.p1.2)\.
- Debenedettiet al\.\(2024\)E\. Debenedetti, J\. Zhang, M\. Balunović, and et al\.AgentDojo: a dynamic environment to evaluate prompt injection attacks and defenses for LLM agents\.InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,Vol\.37\.Cited by:[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1),[§3](https://arxiv.org/html/2611.24039#S3.p1.1),[Table 4](https://arxiv.org/html/2611.24039#S7.T4.2.12.4.1.1),[Table 5](https://arxiv.org/html/2611.24039#S8.T5.4.8.2.1.1)\.
- Drouinet al\.\(2024\)A\. Drouin, M\. Gasse, M\. Caccia, and et al\.WorkArena: how capable are web agents at solving common knowledge work tasks?\.InProceedings of the 41st International Conference on Machine Learning,Cited by:[§3](https://arxiv.org/html/2611.24039#S3.p1.1)\.
- European Parliament and Council \(2024\)European Parliament and CouncilRegulation \(EU\) 2024/1689 laying down harmonised rules on artificial intelligence \(Artificial Intelligence Act\)\.Note:Official Journal of the European UnionCited by:[§1](https://arxiv.org/html/2611.24039#S1.p2.1),[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1)\.
- Fourneyet al\.\(2024\)A\. Fourney, G\. Bansal, H\. Mozannar, and et al\.Magentic\-One: a generalist multi\-agent system for solving complex tasks\.arXiv preprint arXiv:2411\.04468\.Cited by:[§1](https://arxiv.org/html/2611.24039#S1.p1.1),[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1)\.
- Gaoet al\.\(2023\)Y\. Gao, Y\. Xiong, X\. Gao, and et al\.Retrieval\-augmented generation for large language models: a survey\.arXiv preprint arXiv:2312\.10997\.Cited by:[§4\.3](https://arxiv.org/html/2611.24039#S4.SS3.p1.1)\.
- Glean \(2025\)GleanGlean’s open agent platform\.Note:[https://glean\.com/blog/open\-agent\-platform](https://glean.com/blog/open-agent-platform)Accessed October 2026Cited by:[B\.6 Glean\.](https://arxiv.org/html/2611.24039#Ax1.SSx2.SSS0.Px6.p1.1)\.
- Glean \(2026\)GleanConnect remote MCP servers to Glean; Glean MCP server security, data flow and permissions\.Note:[https://docs\.glean\.com/administration/platform/mcp/security](https://docs.glean.com/administration/platform/mcp/security)Accessed October 2026Cited by:[B\.6 Glean\.](https://arxiv.org/html/2611.24039#Ax1.SSx2.SSS0.Px6.p1.1)\.
- Google Cloud \(2026a\)Google CloudOverview of agent development kit\.Note:[https://docs\.cloud\.google\.com/agent\-builder/agent\-development\-kit/overview](https://docs.cloud.google.com/agent-builder/agent-development-kit/overview)Accessed October 2026Cited by:[B\.2 Google \(Vertex AI Agent Builder\)\.](https://arxiv.org/html/2611.24039#Ax1.SSx2.SSS0.Px2.p1.1)\.
- Google Cloud \(2026b\)Google CloudVertex AI agent builder overview\.Note:[https://docs\.cloud\.google\.com/agent\-builder/overview](https://docs.cloud.google.com/agent-builder/overview)Accessed October 2026Cited by:[B\.2 Google \(Vertex AI Agent Builder\)\.](https://arxiv.org/html/2611.24039#Ax1.SSx2.SSS0.Px2.p1.1)\.
- Google \(2025\)GoogleAnnouncing the Agent2Agent protocol \(A2A\)\.Note:Google for Developers BlogCited by:[§1](https://arxiv.org/html/2611.24039#S1.p1.1),[Table 1](https://arxiv.org/html/2611.24039#S4.T1.4.8.4.1.1)\.
- Greshakeet al\.\(2023\)K\. Greshake, S\. Abdelnabi, S\. Mishra, and et al\.Not what you’ve signed up for: compromising real\-world LLM\-integrated applications with indirect prompt injection\.InProceedings of the 16th ACM Workshop on Artificial Intelligence and Security,pp\. 79–90\.Cited by:[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1),[Table 4](https://arxiv.org/html/2611.24039#S7.T4.2.12.4.1.1)\.
- H2O\.ai \(2024\)H2O\.aiH2O\.ai unveils agentic AI that converges generative and predictive AI with purpose\-built SLMs\.Note:Press release,[https://h2o\.ai/platform/enterprise\-h2ogpte/](https://h2o.ai/platform/enterprise-h2ogpte/)Accessed October 2026Cited by:[B\.1 H2O\.ai \(h2oGPTe\)\.](https://arxiv.org/html/2611.24039#Ax1.SSx2.SSS0.Px1.p1.1)\.
- H2O\.ai \(2026\)H2O\.aiEnterprise h2oGPTe documentation\.Note:[https://docs\.h2o\.ai/enterprise\-h2ogpte/](https://docs.h2o.ai/enterprise-h2ogpte/)Accessed October 2026Cited by:[B\.1 H2O\.ai \(h2oGPTe\)\.](https://arxiv.org/html/2611.24039#Ax1.SSx2.SSS0.Px1.p1.1)\.
- Honget al\.\(2024\)S\. Hong, M\. Zhuge, J\. Chen, and et al\.MetaGPT: meta programming for a multi\-agent collaborative framework\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2611.24039#S1.p1.1),[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1)\.
- Huanget al\.\(2024\)J\. Huang, X\. Chen, S\. Mishra, and et al\.Large language models cannot self\-correct reasoning yet\.InInternational Conference on Learning Representations,Cited by:[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1),[Table 1](https://arxiv.org/html/2611.24039#S4.T1.4.2.4.1.1),[§6\.2](https://arxiv.org/html/2611.24039#S6.SS2.p1.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, and et al\.SWE\-bench: can language models resolve real\-world GitHub issues?\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2611.24039#S1.p1.1)\.
- Kaelblinget al\.\(1998\)L\. P\. Kaelbling, M\. L\. Littman, and A\. R\. CassandraPlanning and acting in partially observable stochastic domains\.Artificial Intelligence101\(1–2\),pp\. 99–134\.Cited by:[§2\.3](https://arxiv.org/html/2611.24039#S2.SS3.p1.1)\.
- Kapooret al\.\(2024\)S\. Kapoor, B\. Stroebl, Z\. S\. Siegel, and et al\.AI agents that matter\.arXiv preprint arXiv:2407\.01502\.Cited by:[§1](https://arxiv.org/html/2611.24039#S1.p2.1),[§3](https://arxiv.org/html/2611.24039#S3.p1.1),[§4\.2](https://arxiv.org/html/2611.24039#S4.SS2.p1.1),[Table 1](https://arxiv.org/html/2611.24039#S4.T1.4.9.4.1.1),[§6\.2](https://arxiv.org/html/2611.24039#S6.SS2.p1.1),[Table 4](https://arxiv.org/html/2611.24039#S7.T4.2.11.4.1.1),[§8](https://arxiv.org/html/2611.24039#S8.SS0.SSS0.Px2.p1.2)\.
- Krippendorff \(2004\)K\. KrippendorffContent analysis: an introduction to its methodology\.2 edition,Sage\.Cited by:[§4\.4](https://arxiv.org/html/2611.24039#S4.SS4.p1.2)\.
- Kwaet al\.\(2025\)T\. Kwa, B\. West, J\. Becker, A\. Deng, and et al\.Measuring AI ability to complete long tasks\.arXiv preprint arXiv:2503\.14499\.Cited by:[§3](https://arxiv.org/html/2611.24039#S3.p1.1)\.
- Letta \(2026\)LettaLetta documentation: stateful agents, memory blocks, sleep\-time agents and the agent development environment\.Note:[https://docs\.letta\.com/](https://docs.letta.com/)Accessed October 2026Cited by:[B\.5 Letta\.](https://arxiv.org/html/2611.24039#Ax1.SSx2.SSS0.Px5.p1.1)\.
- Lewiset al\.\(2020\)P\. Lewis, E\. Perez, A\. Piktus, and et al\.Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems,Vol\.33\.Cited by:[§4\.3](https://arxiv.org/html/2611.24039#S4.SS3.p1.1)\.
- Liet al\.\(2023\)G\. Li, H\. A\. A\. K\. Hammoud, H\. Itani, and et al\.CAMEL: communicative agents for “mind” exploration of large language model society\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1)\.
- Liuet al\.\(2024a\)N\. F\. Liu, K\. Lin, J\. Hewitt, and et al\.Lost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.Cited by:[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1),[Table 1](https://arxiv.org/html/2611.24039#S4.T1.4.4.4.1.1)\.
- Liuet al\.\(2024b\)X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, and et al\.AgentBench: evaluating LLMs as agents\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2611.24039#S1.p2.1)\.
- Mastermanet al\.\(2024\)T\. Masterman, S\. Besen, M\. Sawtell, and A\. ChaoThe landscape of emerging AI agent architectures for reasoning, planning, and tool calling: a survey\.arXiv preprint arXiv:2404\.11584\.Cited by:[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1)\.
- Mathewet al\.\(2021\)M\. Mathew, D\. Karatzas, and C\. V\. JawaharDocVQA: a dataset for VQA on document images\.InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,pp\. 2200–2209\.Cited by:[Table 2](https://arxiv.org/html/2611.24039#Ax1.T2.2.2.4.1.1)\.
- Mehta \(2025\)S\. MehtaBeyond accuracy: a multi\-dimensional framework for evaluating enterprise agentic AI systems\.arXiv preprint arXiv:2511\.14136\.Cited by:[§3](https://arxiv.org/html/2611.24039#S3.p1.1)\.
- Mialonet al\.\(2024\)G\. Mialon, C\. Fourrier, and et al\.GAIA: a benchmark for general AI assistants\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2611.24039#S1.p2.1)\.
- Microsoft \(2026a\)MicrosoftFoundry agent service documentation\.Note:[https://learn\.microsoft\.com/azure/ai\-foundry/agents/](https://learn.microsoft.com/azure/ai-foundry/agents/)Accessed October 2026Cited by:[B\.3 Microsoft \(Foundry Agent Service, Agent Framework\)\.](https://arxiv.org/html/2611.24039#Ax1.SSx2.SSS0.Px3.p1.1)\.
- Microsoft \(2026b\)MicrosoftMicrosoft agent framework overview\.Note:[https://learn\.microsoft\.com/agent\-framework/overview/](https://learn.microsoft.com/agent-framework/overview/)Accessed October 2026Cited by:[B\.3 Microsoft \(Foundry Agent Service, Agent Framework\)\.](https://arxiv.org/html/2611.24039#Ax1.SSx2.SSS0.Px3.p1.1)\.
- Mitchellet al\.\(2019\)M\. Mitchell, S\. Wu, A\. Zaldivar, and et al\.Model cards for model reporting\.InProceedings of the ACM Conference on Fairness, Accountability, and Transparency,pp\. 220–229\.Cited by:[Table 1](https://arxiv.org/html/2611.24039#S4.T1.4.10.4.1.1)\.
- National Institute of Standards and Technology \(2023\)National Institute of Standards and TechnologyAI risk management framework \(AI RMF 1\.0\)\.Technical reportTechnical ReportNIST AI 100\-1,National Institute of Standards and Technology\.Cited by:[§1](https://arxiv.org/html/2611.24039#S1.p2.1),[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1),[Table 1](https://arxiv.org/html/2611.24039#S4.T1.4.10.4.1.1)\.
- National Institute of Standards and Technology \(2024\)National Institute of Standards and TechnologyArtificial intelligence risk management framework: generative artificial intelligence profile\.Technical reportTechnical ReportNIST AI 600\-1,National Institute of Standards and Technology\.Cited by:[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1),[§8](https://arxiv.org/html/2611.24039#S8.SS0.SSS0.Px2.p1.2)\.
- Packeret al\.\(2023\)C\. Packer, S\. Wooders, K\. Lin, V\. Fang, and et al\.MemGPT: towards LLMs as operating systems\.arXiv preprint arXiv:2310\.08560\.Cited by:[B\.5 Letta\.](https://arxiv.org/html/2611.24039#Ax1.SSx2.SSS0.Px5.p1.1),[§2\.3](https://arxiv.org/html/2611.24039#S2.SS3.p1.2),[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1),[Table 1](https://arxiv.org/html/2611.24039#S4.T1.4.4.4.1.1)\.
- Parket al\.\(2023\)J\. S\. Park, J\. C\. O’Brien, C\. J\. Cai, and et al\.Generative agents: interactive simulacra of human behavior\.InProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology,Cited by:[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1)\.
- Patilet al\.\(2023\)S\. G\. Patil, T\. Zhang, X\. Wang, and J\. E\. GonzalezGorilla: large language model connected with massive APIs\.arXiv preprint arXiv:2305\.15334\.Cited by:[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1)\.
- Portia Labs \(2026a\)Portia LabsIntroducing steel thread: evals for agentic workflows\.Note:[https://docs\.portialabs\.ai/steel\-thread\-intro](https://docs.portialabs.ai/steel-thread-intro)Accessed October 2026Cited by:[B\.9 Portia AI\.](https://arxiv.org/html/2611.24039#Ax1.SSx2.SSS0.Px9.p1.1)\.
- Portia Labs \(2026b\)Portia LabsPortia SDK documentation: execution hooks and plan runs\.Note:[https://docs\.portialabs\.ai/execution\-hooks](https://docs.portialabs.ai/execution-hooks)Accessed October 2026Cited by:[B\.9 Portia AI\.](https://arxiv.org/html/2611.24039#Ax1.SSx2.SSS0.Px9.p1.1)\.
- Qinet al\.\(2024\)Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, and et al\.ToolLLM: facilitating large language models to master 16000\+ real\-world APIs\.InInternational Conference on Learning Representations,Cited by:[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1)\.
- Russell and Norvig \(2020\)S\. Russell and P\. NorvigArtificial intelligence: a modern approach\.4 edition,Pearson\.Cited by:[§2\.1](https://arxiv.org/html/2611.24039#S2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2611.24039#S2.SS3.p1.1)\.
- Saaty \(1980\)T\. L\. SaatyThe analytic hierarchy process\.McGraw\-Hill\.Cited by:[§4\.4](https://arxiv.org/html/2611.24039#S4.SS4.p1.2)\.
- Schicket al\.\(2023\)T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, and et al\.Toolformer: language models can teach themselves to use tools\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1),[Table 1](https://arxiv.org/html/2611.24039#S4.T1.4.3.4.1.1)\.
- ServiceNow and Artificial Analysis \(2026\)ServiceNow and Artificial AnalysisEnterpriseOps\-Gym: an agentic benchmark for enterprise operations\.Note:[https://artificialanalysis\.ai/articles/announcing\-enterpriseops\-gym\-aa](https://artificialanalysis.ai/articles/announcing-enterpriseops-gym-aa)Cited by:[§3](https://arxiv.org/html/2611.24039#S3.p1.1)\.
- Shavitet al\.\(2023\)Y\. Shavit, S\. Agarwal, M\. Brundage, and et al\.Practices for governing agentic AI systems\.Technical reportOpenAI\.Cited by:[§2\.2](https://arxiv.org/html/2611.24039#S2.SS2.p1.1),[§8](https://arxiv.org/html/2611.24039#S8.SS0.SSS0.Px2.p1.2)\.
- Shinnet al\.\(2023\)N\. Shinn, F\. Cassano, A\. Gopinath, and et al\.Reflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§2\.3](https://arxiv.org/html/2611.24039#S2.SS3.p1.2),[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1),[Table 1](https://arxiv.org/html/2611.24039#S4.T1.4.2.4.1.1)\.
- Shrout and Fleiss \(1979\)P\. E\. Shrout and J\. L\. FleissIntraclass correlations: uses in assessing rater reliability\.Psychological Bulletin86\(2\),pp\. 420–428\.Cited by:[§4\.4](https://arxiv.org/html/2611.24039#S4.SS4.p1.2)\.
- Squid AI \(2026\)Squid AIPrivate AI agents and automations\.Note:[https://squid\.cloud/use\-cases/ai\-agents](https://squid.cloud/use-cases/ai-agents)Accessed October 2026Cited by:[B\.10 Squid AI\.](https://arxiv.org/html/2611.24039#Ax1.SSx2.SSS0.Px10.p1.1)\.
- Stroeblet al\.\(2024\)B\. Stroebl, S\. Kapoor, and A\. NarayananInference scaling fLaws: the limits of LLM resampling with imperfect verifiers\.arXiv preprint arXiv:2411\.17501\.Cited by:[§3](https://arxiv.org/html/2611.24039#S3.p1.1),[§4\.2](https://arxiv.org/html/2611.24039#S4.SS2.p1.1)\.
- Sumerset al\.\(2024\)T\. R\. Sumers, S\. Yao, K\. Narasimhan, and T\. L\. GriffithsCognitive architectures for language agents\.Transactions on Machine Learning Research\.Cited by:[§2\.3](https://arxiv.org/html/2611.24039#S2.SS3.p1.2),[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1)\.
- Thorndike \(1920\)E\. L\. ThorndikeA constant error in psychological ratings\.Journal of Applied Psychology4\(1\),pp\. 25–29\.Cited by:[§4\.4](https://arxiv.org/html/2611.24039#S4.SS4.p1.2),[§6\.3](https://arxiv.org/html/2611.24039#S6.SS3.p1.1)\.
- UiPath \(2026a\)UiPathAgent builder: build agents low\-code in UiPath studio\.Note:[https://www\.uipath\.com/platform/agentic\-automation/agentic\-ai/agent\-builder](https://www.uipath.com/platform/agentic-automation/agentic-ai/agent-builder)Accessed October 2026Cited by:[B\.8 UiPath\.](https://arxiv.org/html/2611.24039#Ax1.SSx2.SSS0.Px8.p1.1)\.
- UiPath \(2026b\)UiPathMaestro user guide: orchestrate an AI agent\.Note:[https://docs\.uipath\.com/maestro/automation\-cloud/latest/user\-guide/orchestrate\-an\-ai\-agent](https://docs.uipath.com/maestro/automation-cloud/latest/user-guide/orchestrate-an-ai-agent)Accessed October 2026Cited by:[B\.8 UiPath\.](https://arxiv.org/html/2611.24039#Ax1.SSx2.SSS0.Px8.p1.1)\.
- van der Aalstet al\.\(2018\)W\. M\. P\. van der Aalst, M\. Bichler, and et al\.Robotic process automation\.Business & Information Systems Engineering60\(4\),pp\. 269–272\.Cited by:[§2\.1](https://arxiv.org/html/2611.24039#S2.SS1.p1.1)\.
- Wanget al\.\(2024a\)G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, and et al\.Voyager: an open\-ended embodied agent with large language models\.Transactions on Machine Learning Research\.Cited by:[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1)\.
- Wanget al\.\(2024b\)L\. Wang, C\. Ma, X\. Feng, Z\. Zhang, and et al\.A survey on large language model based autonomous agents\.Frontiers of Computer Science18\(6\),pp\. 186345\.Cited by:[§2\.1](https://arxiv.org/html/2611.24039#S2.SS1.p1.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, and et al\.Chain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§1](https://arxiv.org/html/2611.24039#S1.p1.1),[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1)\.
- Wilson \(1927\)E\. B\. WilsonProbable inference, the law of succession, and statistical inference\.Journal of the American Statistical Association22\(158\),pp\. 209–212\.Cited by:[§8](https://arxiv.org/html/2611.24039#S8.SS0.SSS0.Px2.p1.2)\.
- Wooldridge and Jennings \(1995\)M\. Wooldridge and N\. R\. JenningsIntelligent agents: theory and practice\.The Knowledge Engineering Review10\(2\),pp\. 115–152\.Cited by:[§2\.1](https://arxiv.org/html/2611.24039#S2.SS1.p1.1)\.
- Writer \(2026\)WriterWriter AI studio documentation\.Note:[https://dev\.writer\.com/home/introduction](https://dev.writer.com/home/introduction)Accessed October 2026Cited by:[B\.4 Writer \(AI Studio\)\.](https://arxiv.org/html/2611.24039#Ax1.SSx2.SSS0.Px4.p1.1)\.
- Wuet al\.\(2023\)Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, and et al\.AutoGen: enabling next\-gen LLM applications via multi\-agent conversation\.arXiv preprint arXiv:2308\.08155\.Cited by:[§1](https://arxiv.org/html/2611.24039#S1.p1.1),[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1),[Table 1](https://arxiv.org/html/2611.24039#S4.T1.4.6.4.1.1)\.
- Xiet al\.\(2023\)Z\. Xi, W\. Chen, X\. Guo, W\. He, and et al\.The rise and potential of large language model based agents: a survey\.arXiv preprint arXiv:2309\.07864\.Cited by:[§2\.1](https://arxiv.org/html/2611.24039#S2.SS1.p1.1)\.
- Xieet al\.\(2024\)T\. Xie, D\. Zhang, J\. Chen, X\. Li, and et al\.OSWorld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[§1](https://arxiv.org/html/2611.24039#S1.p1.1)\.
- Yanet al\.\(2024\)F\. Yan, H\. Mao, C\. C\. Ji, and et al\.Berkeley function calling leaderboard\.Note:[https://gorilla\.cs\.berkeley\.edu/leaderboard\.html](https://gorilla.cs.berkeley.edu/leaderboard.html)Cited by:[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1),[Table 1](https://arxiv.org/html/2611.24039#S4.T1.4.3.4.1.1)\.
- Yaoet al\.\(2024\)S\. Yao, N\. Shinn, P\. Razavi, and et al\.τ\\tau\-Bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv preprint arXiv:2406\.12045\.Cited by:[§1](https://arxiv.org/html/2611.24039#S1.p2.1),[§3](https://arxiv.org/html/2611.24039#S3.p1.1),[§8](https://arxiv.org/html/2611.24039#S8.SS0.SSS0.Px2.p1.2)\.
- Yaoet al\.\(2023a\)S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, and et al\.Tree of thoughts: deliberate problem solving with large language models\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1),[Table 1](https://arxiv.org/html/2611.24039#S4.T1.4.5.4.1.1)\.
- Yaoet al\.\(2023b\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, and et al\.ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2611.24039#S1.p1.1),[§2\.4](https://arxiv.org/html/2611.24039#S2.SS4.p1.1),[Table 1](https://arxiv.org/html/2611.24039#S4.T1.4.5.4.1.1)\.
- Zhouet al\.\(2024\)S\. Zhou, F\. Xu, H\. Zhu, and et al\.WebArena: a realistic web environment for building autonomous agents\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2611.24039#S1.p2.1)\.
## Supplementary Appendix
### A\. Radar Profiles of Tier\-1 Platforms
Figure 7:Capability profiles of the five Tier\-1 platforms \(axis range 4–9\.5 to emphasise differences\)\.
### B\. Platform Profiles
Each profile separates what is\[documented\]in vendor documentation from what was\[observed\]in the bakeoff demonstration, and closes with the panel scores\. Vendor\-reported figures that we could not verify are marked\. Platforms evolve monthly; where documentation now differs from what was demonstrated, we say so\.
#### B\.1 H2O\.ai \(h2oGPTe\)\.
\[documented\]Multi\-agent platform \(general, data\-science and vertical agents\) that calls H2O’s predictive models \(Driverless AI\) from generative workflows; deploys air\-gapped, on\-premise, VPC or managed cloud; guardrails and PII controls; third\-party LLM support\([H2O\.ai, 2024](https://arxiv.org/html/2611.24039#bib.bib63);[H2O\.ai, 2026](https://arxiv.org/html/2611.24039#bib.bib64)\)\. Vendor\-reported 79\.7% on GAIA \(June 2025; not independently verified\)\.\[observed\]Low\-code builder emitting editable Python; web research, database access, code execution, self\-critique; citation\-based RAG verification\. Highest total \(69\.2\) and highest D9 \(8\.44\)\.
#### B\.2 Google \(Vertex AI Agent Builder\)\.
\[documented\]Open\-source, model\-agnostic Agent Development Kit \(ADK\) for multi\-agent systems; Agent Designer \(no\-code\) with hand\-off to ADK; managed Agent Engine runtime with Sessions, Memory Bank, Code Execution, evaluation and tracing; also hosts LangChain, LangGraph, AG2 and LlamaIndex agents; A2A template \(preview\); RAG Engine and Vertex AI Search grounding; agent identity via IAM and Agent Engine threat detection\([Google Cloud, 2026b](https://arxiv.org/html/2611.24039#bib.bib65);[Google Cloud, 2026a](https://arxiv.org/html/2611.24039#bib.bib66)\)\.\[observed\]Strongest multi\-agent demonstration in the field\. Highest D5 \(8\.86\); D6 capped at 6\.00 in every scenario, reflecting the breadth of the cloud surface\.
#### B\.3 Microsoft \(Foundry Agent Service, Agent Framework\)\.
\[documented\]Foundry Agent Service: managed runtime with Python/C\#/TypeScript SDKs, MCP tool integration, built\-in tools \(Fabric, SharePoint, Azure AI Search\), vector store and conversation state\. Microsoft Agent Framework \(open source\) is the declared successor to Semantic Kernel and AutoGen: session state, middleware, telemetry, graph workflows \(sequential, concurrent, hand\-off, group chat, Magentic\), and MCP, A2A and OpenAPI interoperability\([Microsoft, 2026b](https://arxiv.org/html/2611.24039#bib.bib67);[Microsoft, 2026a](https://arxiv.org/html/2611.24039#bib.bib68)\)\.\[observed\]Scored but not profiled in the original programme; the toolset demonstrated was fragmented \(SK \+ AutoGen\), which the unified framework has since addressed\. Highest D7 \(8\.17\); lowest Tier\-1 D6 \(5\.00\)\.
#### B\.4 Writer \(AI Studio\)\.
\[documented\]Full\-stack platform: no\-code Agent Builder, Writer Framework \(code\) and API; proprietary Palmyra models \(X5, Fin, Med, Vision\) plus external models via Bedrock; Knowledge Graph RAG with inline citations and connectors; guardrails \(content safety, PII, compliance\); platform and agent observability; MCP server; LangChain and AWS Strands integrations\([Writer, 2026](https://arxiv.org/html/2611.24039#bib.bib69)\)\.\[observed\]Four\-level capability model \(assistive, knowledge, action, multi\-agent\); “Action AI” agents executing in sandboxed Linux environments \(vendor description\)\. Balanced profile; strong D6 \(7\.57\)\.
#### B\.5 Letta\.
\[documented\]Open\-source server and cloud from the MemGPT authors\([Packer et al\., 2023](https://arxiv.org/html/2611.24039#bib.bib37)\): self\-editing memory split into in\-context memory blocks and out\-of\-context archival memory; blocks shareable across agents; sleep\-time agents that rewrite memory in the background; Agent Development Environment \(ADE\) exposing memory, context window and tool calls; MCP tool servers; Agent File \(\.af\) for portable agent state; Letta Evals; REST/Python/TypeScript SDKs; self\-host or cloud\([Letta, 2026](https://arxiv.org/html/2611.24039#bib.bib70)\)\.\[observed\]Agents migrating between model providers without losing state\. Top D3 \(7\.86\); weaker D1 \(6\.43\) and D9 \(6\.29\)\.
#### B\.6 Glean\.
\[documented\]Agent layer over permission\-aware enterprise search: Agent Builder with a Plan \+ Execute step, event and schedule triggers, model\-agnostic; Glean acts as both MCP host \(admin\-gated remote servers\) and managed MCP server exposing search, chat and agents as tools; Glean Protect enforces source permissions on every action; LangChain, LangGraph and OpenAI Agents SDK integrations\([Glean, 2026](https://arxiv.org/html/2611.24039#bib.bib71);[Glean, 2025](https://arxiv.org/html/2611.24039#bib.bib72)\)\.\[observed\]Uniform mid\-range scores \(4\.00–5\.33\); its strength, permission\-aware knowledge access, sits largely outside the rubric’s emphasis on autonomous execution\.
#### B\.7 CrewAI\.
\[documented\]Open\-source Python framework: Agents \(role, goal, tools\), Tasks, Crews with sequential or hierarchical process \(manager agent delegates and validates\); Flows for event\-driven, stateful, resumable workflows with human\-in\-the\-loop; unified Memory with hierarchical scopes; knowledge, guardrails and observability\. Enterprise \(AMP\) adds a console, triggers \(Gmail, Slack, Salesforce, HubSpot\), RBAC and environment management\([CrewAI, 2026](https://arxiv.org/html/2611.24039#bib.bib73)\)\.\[observed\]Demonstrated memory and reflection depth was limited; the unified memory system post\-dates the bakeoff\. Tier 2 \(41\.6\) despite strong community standing: a clear instance of the presentation confound\.
#### B\.8 UiPath\.
\[documented\]Agent Builder: low\-code canvas in Studio with Autopilot natural\-language authoring, context grounding, evaluations and an Agent Score; Maestro orchestrates agents, RPA robots and people in one process, with agent decisions routing execution; tools include connectors, RPA/API workflows, document extraction and MCP servers; AI Trust Layer provides model governance, audit logs, RBAC and PII filtering; agents, Maestro and MCP servers also available on\-premise in Automation Suite\([UiPath, 2026a](https://arxiv.org/html/2611.24039#bib.bib74);[UiPath, 2026b](https://arxiv.org/html/2611.24039#bib.bib75)\)\.\[observed\]Distinctive grounding in action against legacy systems without APIs\. Strongest Tier\-2 D6 \(6\.40\); weakest D4 and D5 \(4\.00\), consistent with a workflow\-centric heritage\.
#### B\.9 Portia AI\.
\[documented\]Open\-source Python SDK for “predictable, stateful, authenticated” agents: a planner emits an explicit Plan; a PlanRun carries all execution state; Clarifications pause a run for human input, approval or authentication; execution hooks before/after each step and tool call; tool registries from MCP servers, local tools or Portia cloud tools; Portia Cloud dashboard logs runs; Steel Thread provides offline and online evals \(latency, LLM\-judge, tool\-call correctness\)\([Portia Labs, 2026b](https://arxiv.org/html/2611.24039#bib.bib76);[Portia Labs, 2026a](https://arxiv.org/html/2611.24039#bib.bib77)\)\.\[observed\]Structured agent\-to\-human hand\-overs\. Lowest D1 and D9 \(2\.33\) despite an auditability thesis; most plausibly demonstration scope, since the documented hooks and evals target exactly these dimensions\.
#### B\.10 Squid AI\.
\[documented\]Serverless backend\-as\-a\-service with built\-in vector store, embeddings and “universal RAG” over structured and unstructured data; connects to any database or API in cloud or on\-premise; knowledge, system and data agents that inherit source permissions; client and backend SDKs and AI functions for action\-taking\([Squid AI, 2026](https://arxiv.org/html/2611.24039#bib.bib78)\)\.\[observed\]Lead agent orchestrating expert agents; low\-code studio; RAG over tables and images\. Lowest scores on every dimension \(32\.0 total\)\.
### C\. Scenario\-Level Scores
Table[1](https://arxiv.org/html/2611.24039#Ax1.T1)and Figure[8](https://arxiv.org/html/2611.24039#Ax1.F8)disaggregate the three highest\-scoring platforms by scenario\. H2O\.ai is flat across scenarios with a peak on D9 in CCR; Google’s D5 is uniformly high while its D6 is capped at 6\.00 in every scenario; Microsoft’s profile is almost invariant, indicating little scenario\-specific adaptation in demonstration\.
Table 3:Behavioural anchors for the nine dimensions \(intermediate scores interpolate\)\.Table 1:Scenario\-level mean scores for the three highest\-scoring platforms \(CCR: counterparty credit risk; PA: process automation\)\.Figure 8:Scenario\-level scores for the three highest\-scoring platforms\.Table 2:Supervised ML extraction versus agentic document extraction\.#### C\.1 Vendor observation \(Landing AI\)\.
Vision APIs extract structured data from PDFs and images without templates or training, returning hierarchical JSON with page and coordinate references; vendor\-reported median processing time reduction from 135 s to 8 s\[documented, not verified\]\. A qualitative comparison with direct LLM extraction \(Gemini\) suggested the LLM excels at contextual reasoning while a specialised extractor embedded in an agentic pipeline offers more control over cost, checkpoints and vendor independence\-a design\[hypothesis\], not a measured result\.
#### C\.2 Publication\-grade ADE metrics\.
A labelled corpus stratified by type, layout complexity and modality; field\-level precision/recall/F1; table\- and layout\-level accuracy; bounding\-box or citation correctness; abstention quality \(precision of the review flag\); human\-review rate; latency and cost per page; robustness under distribution shift; all with confidence intervals\.
### D\. Proposed Scoring Anchors
The original programme used an unanchored 1–10 scale\. To reduce rater drift and make scores comparable across panels, Table[3](https://arxiv.org/html/2611.24039#Ax1.T3)proposes behavioural anchors at four points on each dimension; intermediate scores interpolate\. Anchors are illustrative and should be refined per organisation and workload before use\.Similar Articles
One Capability or Many? Testing the Economic Validity of Frontier AI Evaluation
This study tests whether economic benchmarks in AI leaderboards measure distinct capabilities or just a general trend, finding they add incremental information but are largely time-driven.
Open-source procurement rubric for agentic AI vendors, I scored 5 of them and want feedback on the methodology
The author created an open-source rubric tool to evaluate agentic AI vendor documentation on tool-call correctness, loop termination, and multi-step state coherence, scored five vendors (Anthropic, OpenAI, LangGraph, Sierra, Salesforce), and requests feedback on methodology and potential bias toward public documentation depth.
SWE-WebDevBench: Evaluating Coding Agent Application Platforms as Virtual Software Agencies
This paper introduces SWE-WebDevBench, a comprehensive 68-metric framework for evaluating AI-powered application development platforms as virtual software agencies. The study highlights critical gaps in current platforms regarding specification understanding, backend reliability, production readiness, and security.
The CASE Framework: A Multi-Disciplinary Control Architecture for Governing Enterprise Agentic AI
This paper introduces the CASE framework, a multi-disciplinary control architecture for governing enterprise agentic AI, integrating control theory, complex adaptive systems theory, supervisory cybernetics, and engineering operations. It presents empirical studies showing an 'Emergence Gap' in current governance practices and proposes a maturity model aligned with regulatory requirements like the EU AI Act.
Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills
The paper presents ACES, a framework for continuous evaluation of AI agent skills through live trials, measuring Skill Lift to quantify added value, and demonstrating its effectiveness on enterprise repositories compared to scan-only gates.