ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents

# ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents
Source: [https://arxiv.org/html/2606.31174](https://arxiv.org/html/2606.31174)
Kaiwen Xiong1, Haonian Ji1, Shi Qiu1, Zeyu Zheng2, Cihang Xie3, Xinyu Ye1, Huaxiu Yao111footnotemark:1 1UNC\-Chapel Hill2University of California, Berkeley3University of California, Santa Cruz

###### Abstract

Production large language\-model \(LLM\) agents are increasingly deployed not as lone problem\-solvers but as*managers*: a main model creates specialized subagents, delegates work, and orchestrates their parallel, asynchronous returns through dynamic workflows\. Whether one model can actually run such a team is largely unmeasured: existing benchmarks score a policy’s own task\-solving or a fixed multi\-agent system’s emergent behavior, but none isolate the management ability of the single LLM acting as leader\. We introduceClawArena\-Team, a benchmark of 41 multi\-turn, multimodal, multi\-directory scenarios spanning 258 evaluation rounds and 72 staged updates that measures this management ability\. The main agent is deliberately constrained: it natively perceives only text and directly accesses only part of the workspace\. It commands a*fixed*, locally served subagent pool, so score differences reflect management skill, not raw capability\. All scoring is execution\-based with no LLM judge: an overall score—the Subagent\-Management Score \(Sms\)—multiplies task correctness by a least\-privilege and modality\-routing factor\. Across twelve proprietary, community\-hosted, and self\-hosted models, experiments show that the management bottleneck is*privilege granting*rather than perception \(no model exceeds50%50\\%workspace\-permission precision\); that cost and management quality are*decoupled*\(API cost spans over100×100\\timeswhile the overall score spans under4×4\\times, with the cheapest open models on the Pareto frontier\); and that most leaderboard scores*cluster*within a9\.99\.9\-point band while orchestration behaviors diverge by more than an order of magnitude\. Code and data will be released\.

![Refer to caption](https://arxiv.org/html/2606.31174v1/figures/fig_hero_gen_v09.png)Figure 1:Overview ofClawArena\-Team\. A text\-only main agent \(the “conductor”\) creates, empowers, and schedules a*fixed*, local pool ofllm/vlm/omnisubagents for multi\-turn tasks over an evolving workspace, under execution\-based scoring \(no LLM judge\)\.## 1Introduction

Production LM agents are increasingly deployed not as lone problem\-solvers but as*managers*: a single main model creates specialized subagents, delegates work to them, and orchestrates their parallel and asynchronous returns through dynamic, programmable workflows\(Anthropic,[2026a](https://arxiv.org/html/2606.31174#bib.bib21);[c](https://arxiv.org/html/2606.31174#bib.bib22)\)\. Yet whether one model can actually run such a team is largely unmeasured\. Management is a structured skill with three coupled requirements\. The manager must route each task piece to the right specialist by modality and capability, so an image\-grounded question reaches a vision worker rather than being attempted on the text it cannot see \(*modality routing*\)\. The manager must grant each subagent only the tools and workspace paths it needs, so over\-granting does not expand the blast radius of a misbehaving worker or inflate context \(*least\-privilege empowerment*\)\. The manager must schedule subagents concurrently, in the background, or as continued sessions, and integrate their returns into a correct deliverable rather than merely relay them \(*dynamic orchestration*\)\. A manager that routes correctly but over\-grants, or grants tightly but cannot schedule concurrently, still produces unsafe, expensive, or wrong outcomes\.

Existing benchmarks test fragments of this setting but not the full management capability\. Single\-agent benchmarks score a policy’s own reasoning, tool use, and policy compliance\(Jimenezet al\.,[2024](https://arxiv.org/html/2606.31174#bib.bib1); Mialonet al\.,[2023](https://arxiv.org/html/2606.31174#bib.bib2); Liuet al\.,[2024](https://arxiv.org/html/2606.31174#bib.bib3); Yaoet al\.,[2024](https://arxiv.org/html/2606.31174#bib.bib4); Xieet al\.,[2024](https://arxiv.org/html/2606.31174#bib.bib5); Levyet al\.,[2024](https://arxiv.org/html/2606.31174#bib.bib6)\)\. Multi\-agent frameworks supply orchestration mechanisms but are validated as systems with predefined roles or peer dialogue rather than as benchmarks of a single manager\(Wuet al\.,[2024](https://arxiv.org/html/2606.31174#bib.bib10); Liet al\.,[2023](https://arxiv.org/html/2606.31174#bib.bib11); Honget al\.,[2024](https://arxiv.org/html/2606.31174#bib.bib12)\)\. Permission and least\-privilege tooling formalizes over\-granting as an external enforcement layer that the \(often single\) tool\-calling agent is not trusted to perform itself, rather than as a measured ability of the agent\(Shiet al\.,[2025](https://arxiv.org/html/2606.31174#bib.bib24); Zhuet al\.,[2025a](https://arxiv.org/html/2606.31174#bib.bib25)\)\. Taken together, existing benchmarks either fix the team and score its emergent behavior\(Zhuet al\.,[2025b](https://arxiv.org/html/2606.31174#bib.bib17)\)or score a manager over a pre\-given workflow set\(Masterset al\.,[2025](https://arxiv.org/html/2606.31174#bib.bib16)\), leaving open the question of whether a single LM can create, empower, and orchestrate its own team from scratch\.

We introduceClawArena\-Team\(WorkflowOrchestration variant in ClawArena series\), a benchmark that isolates the subagent\-management ability of a single LM acting as the main agent \(Figure[1](https://arxiv.org/html/2606.31174#S0.F1)\)\.ClawArena\-Teamframes the setting as a principal–agent problem \(§[3\.2](https://arxiv.org/html/2606.31174#S3.SS2)\): the main agent \(principal\) must accomplish multi\-turn tasks it cannot complete alone by creating, empowering, scheduling, and integrating subagents \(agents\)\. Three design choices make the score reflect management rather than raw capability: the main agent natively perceives only text and reaches part of the workspace only through subagents, so delegation is mandatory; the subagent pool is held*fixed*and locally served, so every manager commands the same workers and only the manager varies across runs; and all scoring is execution\-based with no LLM judge\. The benchmark comprises 41 multimodal, multi\-directory scenarios spanning 258 evaluation rounds with 72 staged updates that change subsequent answers, with a composite Subagent\-Management Score \(Sms\) that multiplies task correctness by a least\-privilege and modality\-routing management factor\.

In summary, our primary contribution isClawArena\-Team, a controlled\-comparison benchmark framework that isolates the subagent\-management ability of a single LM main agent through a fixed locally\-served subagent pool, fine\-grained per\-subagent management metrics, and execution\-based scoring with no LLM judge\. On the 41\-scenario benchmark, evaluations across twelve proprietary, community\-hosted, and self\-hosted main\-agent models show that the management bottleneck is privilege granting rather than perception \(no model exceeds50%50\\%workspace\-permission precision\), that cost and management quality are decoupled \(API cost spans over100×100\\timeswhileSmsspans under4×4\\times, with the cheapest open models on the Pareto frontier\), and that leaderboard scores cluster within a9\.99\.9\-point band while orchestration behaviors diverge by more than an order of magnitude \(subagent forbidden\-access rates differ by roughly12×12\\timesamong capable models\)\.

## 2Related Work

#### Agent benchmarks\.

Single\-agent benchmarks such as SWE\-bench\(Jimenezet al\.,[2024](https://arxiv.org/html/2606.31174#bib.bib1)\), GAIA\(Mialonet al\.,[2023](https://arxiv.org/html/2606.31174#bib.bib2)\), AgentBench\(Liuet al\.,[2024](https://arxiv.org/html/2606.31174#bib.bib3)\),τ\\tau\-bench\(Yaoet al\.,[2024](https://arxiv.org/html/2606.31174#bib.bib4)\), OSWorld\(Xieet al\.,[2024](https://arxiv.org/html/2606.31174#bib.bib5)\), AgentBoard\(Maet al\.,[2024](https://arxiv.org/html/2606.31174#bib.bib7)\), ToolLLM\(Qinet al\.,[2023](https://arxiv.org/html/2606.31174#bib.bib8)\), and WebArena\(Zhouet al\.,[2024](https://arxiv.org/html/2606.31174#bib.bib9)\)score a single policy’s reasoning, tool use, and multi\-turn interaction; ST\-WebAgentBench\(Levyet al\.,[2024](https://arxiv.org/html/2606.31174#bib.bib6)\)adds safety and policy compliance, but the unit of evaluation remains a single agent\. Multi\-agent frameworks such as AutoGen\(Wuet al\.,[2024](https://arxiv.org/html/2606.31174#bib.bib10)\), CAMEL\(Liet al\.,[2023](https://arxiv.org/html/2606.31174#bib.bib11)\), MetaGPT\(Honget al\.,[2024](https://arxiv.org/html/2606.31174#bib.bib12)\), OpenAI Swarm\(OpenAI,[2024](https://arxiv.org/html/2606.31174#bib.bib13)\), AgentOrchestra\(Zhanget al\.,[2025](https://arxiv.org/html/2606.31174#bib.bib14)\), and DynTaskMAS\(Yuet al\.,[2025a](https://arxiv.org/html/2606.31174#bib.bib15)\)supply orchestration mechanisms but are validated as systems with predefined roles or peer dialogue rather than runtime single\-manager control\. Neither group introduces a main\-agent/subagent relationship that lets a single LM be scored on its management of a team\.

#### Agent\-as\-manager and the closest precedents\.

The Manager Agent challenge\(Masterset al\.,[2025](https://arxiv.org/html/2606.31174#bib.bib16)\)is the only work that genuinely evaluates a manager orchestrating a team at runtime \(the MA\-Gym simulator, a GPT\-5 manager, graph\-editing actions\)\. It differs fromClawArena\-Teamin three ways: its workflows are*given*task\-dependency graphs that the manager edits rather than creates from scratch; it dispatches to a*pre\-existing*pool of AI and simulated\-human workers rather than creating subagents at runtime; and its five metrics \(preference, constraint, goal, stakeholder, runtime\) cover multi\-objective optimization but not least\-privilege empowerment, modality\-based expert selection, or asynchronous and background scheduling\. MultiAgentBench\(Zhuet al\.,[2025b](https://arxiv.org/html/2606.31174#bib.bib17)\)evaluates the emergent collaboration and competition of a multi\-agent*system*under fixed, predefined topologies, not the delegation ability of a single manager\. Collaborative Gym\(Shaoet al\.,[2026](https://arxiv.org/html/2606.31174#bib.bib18)\)and TheAgentCompany\(Xuet al\.,[2026](https://arxiv.org/html/2606.31174#bib.bib19)\)target human–agent collaboration and realistic workplace tasks, and analyses of multi\-agent failure modes\(Cemriet al\.,[2026](https://arxiv.org/html/2606.31174#bib.bib20)\)motivate better orchestration evaluation\. We positionClawArena\-Teamagainst these by holding the team fixed and varying only the manager, so that score differences isolate management skill rather than the underlying capability of the workers\.

#### Management primitives in isolation\.

Two of the management dimensionsClawArena\-Teamscores have been studied outside the management context\. Progent\(Shiet al\.,[2025](https://arxiv.org/html/2606.31174#bib.bib24)\), MiniScope\(Zhuet al\.,[2025a](https://arxiv.org/html/2606.31174#bib.bib25)\), and the mandatory\-access\-control framework ofJiet al\.\([2026b](https://arxiv.org/html/2606.31174#bib.bib26)\)formalize over\-privileging and multi\-agent confused\-deputy risks, but treat least privilege as an external enforcement layer that the \(often single\) tool\-calling agent is not trusted to perform itself, rather than as a measured ability of the agent\. LM routers such as RouterBench\(Huet al\.,[2024](https://arxiv.org/html/2606.31174#bib.bib29)\)and RouteLLM\(Onget al\.,[2025](https://arxiv.org/html/2606.31174#bib.bib30)\)route queries by capability or cost tier rather than by modality; MEXA\(Yuet al\.,[2025b](https://arxiv.org/html/2606.31174#bib.bib28)\)selects modality\-task experts only as an internal step scored by end\-task accuracy, and MLLM\-Tool\(Wanget al\.,[2025](https://arxiv.org/html/2606.31174#bib.bib27)\)scores modality\-aware selection at the granularity of an API rather than a modality\-specialist subagent\. None scores per\-decision routing to expert subagents or self\-granted least privilege as a first\-class, execution\-checked management dimension, and none brings these dimensions together under a single manager, which is the gapClawArena\-Teamfills \(Table[1](https://arxiv.org/html/2606.31174#S2.T1)\)\.

Table 1:ClawArena\-Teamagainst representative agent benchmarks along the six management dimensions it isolates \(plus execution\-based scoring\)\.✓: measured/first\-class;◐: partial or indirect \(e\.g\. a fixed pool, a given workflow, an external enforcement layer, or modality handled only as an internal step\);✗: absent\. No prior benchmark scores a*single*LM as a manager that creates subagents at runtime, routes by modality, grants least privilege, schedules asynchronously, and authors dynamic workflows under execution\-based scoring;ClawArena\-Teamis the first to combine all of them\.

## 3TheClawArena\-TeamBenchmark

### 3\.1Overview

ClawArena\-Teamis a controlled\-comparison benchmark for subagent management: every manager commands the same fixed, locally\-served subagent pool, so score differences reflect management quality rather than worker quality\. The pool serves three model keys via local vLLM, withllmandvlmbacked bygemma\-4\-31b\-itandomnibygemma\-4\-e4b\-it\. The benchmark comprises 41 scenarios across law, medicine, engineering, business, and science, totaling 258 multi\-turn evaluation rounds\. Each scenario is a workspace task where the main agent completes a series of user tasks, and between some rounds the workspace receives*staged updates*\(new or replaced files\) that change subsequent answers\. Workspaces are large \(170\.5 MiB; 28\.9 M tokens,71\.9%71\.9\\%workspace content and27\.9%27\.9\\%updates\) and heterogeneous \(text, code, office documents, images, audio, video\) across at least eight top\-level directories, some of which are decoys\. What “managing” such a scenario actually requires of the main agentMMis formalized next asMM’s capability surface\.

### 3\.2Capability surface

We formalize subagent management as a principal–agent problem under information and capability asymmetry: a main agentMM\(the principal\) faces a multi\-turn task whose information and modalities exceed what it can directly consume, and must accomplish it by managing a set of subagents\{ai\}\\\{a\_\{i\}\\\}\(the agents\) instantiated from the same pool\.MM’s capability surface decomposes into six concrete operations:

- •Creation\.MMcreates a subagent with a system prompt, a model key, a tool subset, and a workspace path whitelist \(a subset of its own\)\.
- •Modality routing\.BecauseMMnatively perceives only text, images and video must be routed to avlmsubagent and audio to anomni\.
- •Least\-privilege empowerment\.Tool and path grants should match what the subagent actually needs; over\-granting is wasteful and unsafe\.
- •Scheduling\.Subagents can run in the foreground or background, as new sessions or continued \(resumed\) sessions, and in parallel; background tasks notifyMMon completion\.
- •Dynamic workflows\.A programmatic workflow tool letsMMauthor multi\-subagent orchestration \(parallel and pipeline stages\) at runtime over the same pool\.
- •Integration\.MMmust fuse subagent returns into a correct deliverable rather than relay them; only the integrated answer is scored, so a faithful relay of correct subagent reports can still fail\.

How every scenario forces these operations to be exercised, rather than routed around, is the subject of the next subsection\.

### 3\.3Tasks, staged updates, and design criteria

A scenario provides \(i\) a workspace, \(ii\) a sequence of user questions, and \(iii\) ground\-truth checks\. Of the 258 rounds, 44 \(17\.1%17\.1\\%\) are preceded by staged updates \(72 update groups, 255 files\); later rounds depend on earlier subagents’ outputs, requiring sustained management of an evolving task rather than a one\-shot solution\. Every scenario satisfies ten hard constraints \(Table[2](https://arxiv.org/html/2606.31174#S3.T2)\) that jointly force genuine management; each constraint targets a distinct management failure mode that a single LM could otherwise route around \(skipping delegation, ignoring an update\)\.

Table 2:The ten hard design constraints \(C1–C10\)\. Every scenario satisfies all ten; each targets a distinct management failure mode that a single LM could otherwise route around\.Several of these constraints, C5 \(modality grounding\) and C9 \(modality decoys\) in particular, require workspace anchors that text cannot reproduce\. Images include tables, forms, diagrams, and medical regions of interest; audio includes single\- and multi\-speaker recordings, with a voice\-profile matrix to defeat single\-timbre shortcuts; video covers algorithm visualizations, screen recordings, and data animations\. The workspaces also mix many text formats \(CSV, YAML, JSON, SQL, logs, email, calendars\) with decoy directories that may be directly accessible toMMyet are irrelevant, testing whetherMMwithholds them from subagents \(C4\)\. Hand\-assembling 41 scenarios that each satisfy all ten constraints is not viable; we synthesize them programmatically, as described next\.

### 3\.4Construction

We synthesizeClawArena\-Teamend\-to\-end\. Workspace corpora, multimodal assets, ground\-truth values, and execution checks all come from the same version\-controlled scripts, so corpus and answer are emitted together and the benchmark is regenerable from source \(Appendix[G](https://arxiv.org/html/2606.31174#A7)\)\. Two ideas organize the methodology\.*Workflow\-driven parallel authoring*decomposes a batch into per\-scenario contracts that a controller fans out to isolated subagent\-authors and reassembles behind objective gates\. An*end\-to\-end test flywheel*adds a real baseline run of the subagent\-management harness as the final acceptance gate; the run is scored by the rule formalized in §[3\.5](https://arxiv.org/html/2606.31174#S3.SS5)\. A scenario is finished only after this baseline run leaves every failure classified as*true\-difficulty*\(modality decoy taken, fabricated identifier, over\-grant\) rather than*false\-kill*\(brittle regex, order\-locked field\)\. The construction pipeline is itself an instance of subagent management: a single controller that decomposes work, selects model tiers, confines parallel authors under least\-privilege contracts, and integrates their deliverables behind gates\.ClawArena\-Teamis in effect authored by the capability it measures\.

### 3\.5Scoring

That scoring rule is execution\-based, following a “verify the result, not the method” principle: each round ships a shell command whose exit code \(with optional output matching\) determines pass/fail\. No LLM judge is used, which avoids both rejecting legitimate alternative solutions and judge drift\. We report five per\-run components and one composite, all in\[0,1\]\[0,1\]:

- •TCR\(Task Completion Rate\): mean pass rate over user questions\.
- •TPP\(Tool\-Permission Precision\): per subagent, the fraction of granted tool types that are actually used\.
- •ROC\(Read\-Only Compliance\): per subagent,0if a read\-only subagent was granted a mutating tool, else11\.
- •WPP\(Workspace\-Permission Precision\): per subagent, accessed files divided by granted files\.
- •MCA\(Modality\-Choice Accuracy\): per subagent, whether avlmactually reads image/video and anomnireads audio \(llmscores11\)\.

The composite Subagent\-Management Score is

Sms=TCR×TPP\+ROC\+WPP\+MCA4\.\\textsc\{Sms\}\\;=\\;\\mathrm\{TCR\}\\times\\frac\{\\mathrm\{TPP\}\+\\mathrm\{ROC\}\+\\mathrm\{WPP\}\+\\mathrm\{MCA\}\}\{4\}\.\(1\)Because the management factor lies in\[0,1\]\[0,1\],Sms≤\\leqTCR always: management can only discount task correctness, never inflate it\. A model that produces correct work while managing sloppily is penalized, and a model that manages perfectly but fails the task scores zero, encoding that good management without correct deliverables is worthless\.

ModelTCRTPPROCWPPMCASmsCost \($\)RoundsProprietaryclaude\-fable\-574\.476\.498\.849\.297\.960\.092\.8192/258gemini\-3\.5\-flash69\.869\.895\.945\.796\.753\.823\.7180/258gpt\-5\.563\.679\.998\.747\.595\.251\.043\.3164/258gpt\-5\.463\.277\.398\.840\.897\.649\.719\.5163/258gemini\-3\.1\-pro65\.576\.497\.336\.692\.949\.620\.5169/258claude\-sonnet\-4\-661\.677\.399\.443\.295\.248\.539\.7159/258Open\-weightglm\-5\.266\.367\.495\.444\.496\.750\.422\.9171/258kimi\-k2\.664\.376\.892\.041\.993\.949\.028\.7166/258deepseek\-v4\-pro58\.973\.998\.546\.396\.546\.41\.7152/258qwen3\.6\-27b60\.972\.595\.740\.796\.246\.415\.0157/258gemma\-4\-31b56\.679\.097\.737\.995\.543\.93\.5146/258glm\-4\.7\-flash34\.522\.485\.811\.757\.215\.30\.889/258Table 3:ClawArena\-Teamleaderboard \(%\), twelve models×\\times41 scenarios, grouped into proprietary and open\-weight models and sorted bySmswithin each group\. Within each group,boldmarks the best andunderlinethe runner\-up per column \(Sms, its five components, and Rounds; Cost is left unmarked as lower is not strictly better\)\. TCR: task completion; TPP/ROC/WPP/MCA: management components \(§[3\.5](https://arxiv.org/html/2606.31174#S3.SS5)\); Cost \($\): per\-run main\-agent API cost, computed at provider or OpenRouter list rates \(methodology in Appendix[F](https://arxiv.org/html/2606.31174#A6)\); Rounds: rounds passed out of 258\.fable\-5is evaluated as shipped with its refusal→\\rightarrowopus\-4\-8fallback \(Appendix[D](https://arxiv.org/html/2606.31174#A4)\)\.![Refer to caption](https://arxiv.org/html/2606.31174v1/x1.png)Figure 2:Management performance \(Sms\) vs\. main\-agent API cost \(log scale\)\. The cheapest open models lie on the Pareto frontier; the top\-scoring flagship \(fable\-5\) is also the costliest, while several mid\-to\-high\-cost models \(e\.g\.gpt\-5\.5,sonnet\-4\-6,kimi\-k2\.6\) are dominated\.

## 4Experiments

To isolate the main agent’s own management ability, we hold the subagent pool and the execution\-based scoring fixed and vary only the manager model\. We evaluate twelve main\-agent models on the full 41\-scenarioClawArena\-Teambenchmark in a single evaluation run, then analyze it through four lenses\. \(1\) Is the benchmark discriminative across models, and does it leave room above current flagships? \(2\) Within the six\-operation capability surface \(§[3\.2](https://arxiv.org/html/2606.31174#S3.SS2)\), where does the management bottleneck actually lie? \(3\) Does main\-agent API cost track management quality, or do cheap models reach the same management score? \(4\) When leaderboard scores cluster, do underlying orchestration behaviors cluster too, or do they diverge? The leaderboard \(§[4\.2](https://arxiv.org/html/2606.31174#S4.SS2)\) answers \(1\); the three findings that follow within §[4\.2](https://arxiv.org/html/2606.31174#S4.SS2)answer \(2\), \(3\), and \(4\) in turn\.

### 4\.1Setup

Main\-agent models\.We evaluate twelve main\-agent models over all 41 scenarios\. Proprietary models include Anthropicclaude\-fable\-5andclaude\-sonnet\-4\-6,gemini\-3\.5\-flashandgemini\-3\.1\-pro, and twogpt\-5versions accessed throughcodex\. Community\-hosted open\-weight models includekimi\-k2\.6anddeepseek\-v4\-provia OpenRouter, andglm\-5\.2via the official z\.ai API\. Self\-hosted open models served through local vLLM includeqwen3\.6\-27b,gemma\-4\-31b, andglm\-4\.7\-flash\.

Subagent pool and scoring\.Every run uses the same fixed subagent pool \(§[3](https://arxiv.org/html/2606.31174#S3)\) and the same execution\-based scoring \(§[3\.5](https://arxiv.org/html/2606.31174#S3.SS5)\)\. Per\-turn generation is capped to bound runaway output, and the main\-agent context budget is advisory rather than hard \(Appendix[D](https://arxiv.org/html/2606.31174#A4)\)\. API cost is computed from token usage at provider or OpenRouter list prices \(Appendix[F](https://arxiv.org/html/2606.31174#A6)\); self\-hosted models are priced at their OpenRouter reference rates for cross\-model comparability\.claude\-fable\-5is evaluated as shipped, with a vendor\-recommendedrefusal→\\rightarrowclaude\-opus\-4\-8fallback\(Anthropic,[2026b](https://arxiv.org/html/2606.31174#bib.bib23)\); refusal\-handling details are in Appendix[D](https://arxiv.org/html/2606.31174#A4)\.

Evaluation protocol\.Each scenario is a sequence of multi\-turn rounds, and a round is one complete interaction in which the manager may issue anywhere from a single subagent call to several dozen; a complete run over the benchmark \(41 scenarios, 258 rounds, 44 of them preceded by staged updates\) is therefore substantially costlier than a single\-inference benchmark\. All twelve models are evaluated in a single run under identical scenarios, subagent pool, and protocol, so that any difference in the leaderboard is attributable to the manager model alone\.

![Refer to caption](https://arxiv.org/html/2606.31174v1/x2.png)Figure 3:Permission violations, three bars per model \(log scale\): main\-agent forbidden count \(MAF\), subagent forbidden count \(SAFt\), and per\-subagent average \(SAFa\)\. SAFa diverges by roughly12×12\\timesacross the capable models despite clusteredSms\.
### 4\.2Main results

Table[3](https://arxiv.org/html/2606.31174#S3.T3)reports the leaderboard, grouped into proprietary and open\-weight models and sorted bySmswithin each group\. The new flagshipclaude\-fable\-5leads onSms\(60\.0%60\.0\\%\) and TCR \(74\.4%74\.4\\%\), withgemini\-3\.5\-flashsecond onSms\(53\.8%53\.8\\%\)\. The newly releasedglm\-5\.2is the strongest open\-weight manager \(50\.4%50\.4\\%, fourth overall\), overtakingkimi\-k2\.6\(49\.0%49\.0\\%\) to top the open\-weight group, behind only three proprietary models\. The twelve\-modelSmsrange is44\.744\.7points, but it is concentrated in two gaps: a6\.26\.2\-point jump from the runner\-up to the flagship at the top, and a28\.628\.6\-point cliff from the bottom of the cluster \(gemma\-4\-31b,43\.9%43\.9\\%\) down toglm\-4\.7\-flash\(15\.3%15\.3\\%\)\. Between these two endpoints, ten models sit in a tight9\.99\.9\-point band \(43\.943\.9–53\.8%53\.8\\%\)\. Only the flagship clearsSms=54%=54\\%: every other manager loses roughly half its score to imperfect task correctness and privilege management, and even the leader leaves40%40\\%on the table\.

Together, Table[3](https://arxiv.org/html/2606.31174#S3.T3), Figures[2](https://arxiv.org/html/2606.31174#S3.F2)–[4](https://arxiv.org/html/2606.31174#S4.F4), and Table[4](https://arxiv.org/html/2606.31174#S4.T4)answer three questions about manager behavior\.*First*, the management bottleneck is privilege granting rather than perception: the two “easy” axes are nearly saturated but the two privilege\-precision axes are not\.*Second*, cost and management quality are decoupled: cost spans over100×100\\timeswhileSmsspans under4×4\\times, and several high\-cost models are dominated by mid\-cost ones\.*Third*, leaderboard scores cluster while behaviors diverge: the middle ten models sit in a9\.99\.9\-pointSmsband but their orchestration behaviors differ by more than an order of magnitude\. We unpack each in turn\.

#### Finding 1: the management bottleneck is privilege granting\.

The two “easy” management axes are nearly saturated: read\-only compliance \(ROC\) is at least92%92\\%and modality\-choice accuracy \(MCA\) is at least92%92\\%for all eleven capable models, because both have clear objectives: not granting a mutating tool to a read\-only worker, and routing images to the vision model\. The discriminating axes are the two privilege\-precision metrics\. Tool\-permission precision \(TPP\) sits around7070–80%80\\%and workspace\-permission precision \(WPP\)*never*reaches50%50\\%for any model \(Table[3](https://arxiv.org/html/2606.31174#S3.T3)\): subagents are routinely granted roughly twice the files they touch and more tools than they use\. Management is therefore far from solved; the failure concentrates in least\-privilege empowerment—both a safety concern \(over\-broad grants enlarge a misbehaving subagent’s blast radius\) and a cost concern \(irrelevant paths inflate context\)\.

#### Finding 2: cost and management quality are decoupled\.

Figure[2](https://arxiv.org/html/2606.31174#S3.F2)plotsSmsagainst main\-agent API cost \(Cost in Table[3](https://arxiv.org/html/2606.31174#S3.T3)\)\. Cost spans over100×100\\times\($0\.8 to $93 per run\) whileSmsspans under4×4\\times\. The new leaderclaude\-fable\-5is also the most expensive \(≈\\approx$93 per run\): it buys the top score, but at roughly4×4\\timesthe cost ofgemini\-3\.5\-flash\($23\.7\) for only\+6\+6Smspoints\. Below the top, the two decouple sharply\.deepseek\-v4\-proreachesSms=46\.4%=46\.4\\%at just $1\.7 thanks to a steep cache\-read discount, and the costlygpt\-5\.5\($43\.3\),sonnet\-4\-6\($39\.7\), andkimi\-k2\.6\($28\.7\) are each*dominated*bygemini\-3\.5\-flash, whereas the open\-weightglm\-5\.2\($22\.9\) lands*on*the frontier, delivering the top open\-weightSmsat a quarter of the flagship’s cost\. Cheap open models sit on the frontier; outside the lone flagship, spending more does not buy better management\.

#### Finding 3: scores converge while behaviors diverge\.

Below thefable\-5flagship, ten models cluster within a9\.99\.9\-pointSmsband \(43\.943\.9–53\.8%53\.8\\%, withglm\-4\.7\-flashfar below\), but their orchestration behaviors differ by more than an order of magnitude\. The subagent forbidden\-access rate \(mean per subagent\) ranges from0\.480\.48\(gpt\-5\.5\) to5\.785\.78\(deepseek\-v4\-pro\) among capable models and11\.4411\.44forglm\-4\.7\-flash\(Figure[3](https://arxiv.org/html/2606.31174#S4.F3)\)\. Managers overwhelmingly default to thellmkey and under\-usevlmandomni\(Figure[4](https://arxiv.org/html/2606.31174#S4.F4)\): when they route by modality they do it correctly \(hence high MCA\), but create few specialist subagents\. Dynamic\-workflow use ranges from88to112112invocations, and background scheduling and session continuation are unevenly adopted \(Table[4](https://arxiv.org/html/2606.31174#S4.T4)\); most models lean on new foreground runs\. The weakest model \(glm\-4\.7\-flash\) both creates the fewest specialists and almost never uses workflows or background scheduling, consistent with its collapse in Table[3](https://arxiv.org/html/2606.31174#S3.T3)\. A single leaderboard number conceals these differences—precisely whyClawArena\-Teamreports fine\-grained management metrics, not task success alone\.

![Refer to caption](https://arxiv.org/html/2606.31174v1/x3.png)Figure 4:Modality routing: subagent model\-key distribution \(llm/vlm/omni\); bar height is the total number of subagents created\.Table 4:Orchestration style: invocation counts over \{new, continued\}×\\times\{foreground \(fg\), background \(bg\)\} plus dynamic workflows \(wf\)\. Continuation, background, and workflow adoption vary widely;glm\-4\.7\-flashbarely uses any non\-default mode\.

### 4\.3Error analysis

Scenario\-level scores \(Appendix[J](https://arxiv.org/html/2606.31174#A10), Figure[7](https://arxiv.org/html/2606.31174#A10.F7)\) show that difficulty is uneven and model\-specific: no scenario is solved by all models, and the full\-pass rate \(all rounds in a scenario correct\) is near zero, indicating that the hard part is sustained management across an evolving, multimodal workspace rather than any single round\. We ground these patterns in three case studies \(Appendix[L](https://arxiv.org/html/2606.31174#A12)\), to which per\-scenario transcript evidence is deferred; three recurring failure modes emerge\.

Managers take text shortcuts on modality\-decoy rounds\.Managers collapse a multimodal round into a text shortcut by trusting a plausible\-but\-wrong text transcript next to the real asset, rather than delegating to a vision or audio specialist\. The C9 decoys are designed for this, and Appendix[L](https://arxiv.org/html/2606.31174#A12)\(smart\-home anomaly\) shows a concrete instance where the wrong humidity reading is taken from a flagged transcript\.

Stale beliefs persist across staged updates\.Managers fail to propagate a staged update into a later round, either because they advance to the next question before rewriting an earlier deliverable or because they treat the update as new context without re\-checking the prior answer\. The product\-launch warroom case in Appendix[L](https://arxiv.org/html/2606.31174#A12)captures this pattern: a pending correction is acknowledged in conversation but never reaches the deliverable\.

Over\-granting does not buy correctness\.Managers over\-grant workspace paths and tools, but the loose scoping does not improve TCR; subagents simply receive paths they never read and tools they never call\. This is the structural signature of Finding 1 in case\-level form\.

## 5Conclusion

We introducedClawArena\-Team, a benchmark that isolates a single LM’s subagent\-management ability by fixing the worker pool and scoring execution outcomes alongside fine\-grained per\-subagent management metrics\. Across twelve models and4141scenarios, the management bottleneck is privilege granting rather than perception \(workspace\-permission precision never reaches50%50\\%\), and cost and management quality are decoupled \(API cost spans over100×100\\timeswhileSmsspans under4×4\\times, with several mid\-cost models dominating higher\-priced ones\)\. Behind a clustered9\.99\.9\-pointSmsband, the middle ten models’ orchestration behaviors \(forbidden accesses, workflow adoption, background scheduling\) diverge by more than an order of magnitude\. A natural extension scales to broader model coverage and stronger subagent pools, tracking management ability as worker capability improves\.

## Ethical Considerations

ClawArena\-Teamuses synthetic scenarios and assets; any resemblance to real entities is incidental\. The benchmark surfaces a safety\-relevant behavior—over\-broad privilege grants by manager agents—which we report to encourage least\-privilege design rather than to enable misuse\. No human\-subjects data is used\.

## References

- Anthropic \(2026a\)Claude managed agents: get to production 10x faster\.Note:[https://claude\.com/blog/claude\-managed\-agents](https://claude.com/blog/claude-managed-agents)Accessed 2026Cited by:[§1](https://arxiv.org/html/2606.31174#S1.p1.1)\.
- Anthropic \(2026b\)Introducing claude fable 5 and claude mythos 5\.Note:[https://platform\.claude\.com/docs/en/about\-claude/models/introducing\-claude\-fable\-5\-and\-claude\-mythos\-5](https://platform.claude.com/docs/en/about-claude/models/introducing-claude-fable-5-and-claude-mythos-5)Refusals, fallback, and billing on Claude Fable 5\. Accessed 2026Cited by:[Appendix D](https://arxiv.org/html/2606.31174#A4.SS0.SSS0.Px1.p1.3),[§4\.1](https://arxiv.org/html/2606.31174#S4.SS1.p2.1)\.
- Anthropic \(2026c\)Introducing dynamic workflows in claude code\.Note:[https://claude\.com/blog/introducing\-dynamic\-workflows\-in\-claude\-code](https://claude.com/blog/introducing-dynamic-workflows-in-claude-code)Accessed 2026Cited by:[§1](https://arxiv.org/html/2606.31174#S1.p1.1)\.
- M\. Cemri, M\. Z\. Pan, S\. Yang, L\. A\. Agrawal, B\. Chopra, R\. Tiwari, K\. Keutzer, A\. Parameswaran, D\. Klein, K\. Ramchandran, M\. Zaharia, J\. E\. Gonzalez, and I\. Stoica \(2026\)Why do multi\-agent LLM systems fail?\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=fAjbYBmonr)Cited by:[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Hong, M\. Zhuge, J\. Chen, X\. Zheng, Y\. Cheng, J\. Wang, C\. Zhang, Z\. Wang, S\. K\. S\. Yau, Z\. Lin, L\. Zhou, C\. Ran, L\. Xiao, C\. Wu, and J\. Schmidhuber \(2024\)MetaGPT: meta programming for a multi\-agent collaborative framework\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VtmBAGCN7o)Cited by:[§1](https://arxiv.org/html/2606.31174#S1.p2.1),[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px1.p1.1)\.
- Q\. J\. Hu, J\. Bieker, X\. Li, N\. Jiang, B\. Keigwin, G\. Ranganath, K\. Keutzer, and S\. K\. Upadhyay \(2024\)RouterBench: a benchmark for multi\-llm routing system\.External Links:2403\.12031,[Link](https://arxiv.org/abs/2403.12031)Cited by:[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px3.p1.1),[Table 1](https://arxiv.org/html/2606.31174#S2.T1.1.1.13.12.1)\.
- H\. Ji, K\. Xiong, S\. Han, P\. Xia, S\. Qiu, Y\. Zhou, J\. Liu, J\. Li, B\. Li, Z\. Zheng, C\. Xie, and H\. Yao \(2026a\)ClawArena: benchmarking ai agents in evolving information environments\.External Links:2604\.04202,[Link](https://arxiv.org/abs/2604.04202)Cited by:[Table 1](https://arxiv.org/html/2606.31174#S2.T1.1.1.14.13.1)\.
- Z\. Ji, D\. Wu, W\. Jiang, P\. Ma, Z\. Li, Y\. Gao, S\. Wang, and Y\. Li \(2026b\)Taming various privilege escalation in llm\-based agent systems: a mandatory access control framework\.External Links:2601\.11893,[Link](https://arxiv.org/abs/2601.11893)Cited by:[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px3.p1.1)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. Narasimhan \(2024\)SWE\-bench: can language models resolve real\-world github issues?\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VTF8yNQM66)Cited by:[§1](https://arxiv.org/html/2606.31174#S1.p2.1),[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2606.31174#S2.T1.1.1.5.4.1)\.
- I\. Levy, B\. Wiesel, S\. Marreed, A\. Oved, A\. Yaeli, N\. Mashkif, and S\. Shlomov \(2024\)ST\-webagentbench: a benchmark for evaluating safety and trustworthiness in web agents\.External Links:2410\.06703,[Link](https://arxiv.org/abs/2410.06703)Cited by:[§1](https://arxiv.org/html/2606.31174#S1.p2.1),[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2606.31174#S2.T1.1.1.7.6.1)\.
- G\. Li, H\. A\. A\. K\. Hammoud, H\. Itani, D\. Khizbullin, and B\. Ghanem \(2023\)CAMEL: communicative agents for "mind" exploration of large language model society\.InThirty\-seventh Conference on Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2606.31174#S1.p2.1),[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang, S\. Zhang, X\. Deng, A\. Zeng, Z\. Du, C\. Zhang, S\. Shen, T\. Zhang, Y\. Su, H\. Sun, M\. Huang, Y\. Dong, and J\. Tang \(2024\)AgentBench: evaluating LLMs as agents\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2308\.03688Cited by:[§1](https://arxiv.org/html/2606.31174#S1.p2.1),[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Ma, J\. Zhang, Z\. Zhu, C\. Yang, Y\. Yang, Y\. Jin, Z\. Lan, L\. Kong, and J\. He \(2024\)AgentBoard: an analytical evaluation board of multi\-turn llm agents\.External Links:2401\.13178Cited by:[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Masters, A\. Vellanki, J\. Shangguan, B\. Kultys, J\. Gilmore, A\. Moore, and S\. V\. Albrecht \(2025\)Orchestrating human\-ai teams: the manager agent as a unifying research challenge\.External Links:2510\.02557,[Link](https://arxiv.org/abs/2510.02557)Cited by:[§1](https://arxiv.org/html/2606.31174#S1.p2.1),[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2606.31174#S2.T1.1.1.10.9.1)\.
- G\. Mialon, C\. Fourrier, C\. Swift, T\. Wolf, Y\. LeCun, and T\. Scialom \(2023\)GAIA: a benchmark for general ai assistants\.External Links:2311\.12983,[Link](https://arxiv.org/abs/2311.12983)Cited by:[§1](https://arxiv.org/html/2606.31174#S1.p2.1),[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2606.31174#S2.T1.1.1.6.5.1)\.
- I\. Ong, A\. Almahairi, V\. Wu, W\. Chiang, T\. Wu, J\. E\. Gonzalez, M\. W\. Kadous, and I\. Stoica \(2025\)RouteLLM: learning to route LLMs from preference data\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=8sSqNntaMr)Cited by:[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px3.p1.1)\.
- OpenAI \(2024\)Swarm: educational framework exploring ergonomic, lightweight multi\-agent orchestration\.Note:[https://github\.com/openai/swarm](https://github.com/openai/swarm)Experimental; superseded by the OpenAI Agents SDK \(2025\)Cited by:[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian, S\. Zhao, R\. Tian, R\. Xie, J\. Zhou, M\. Gerstein, D\. Li, Z\. Liu, and M\. Sun \(2023\)ToolLLM: facilitating large language models to master 16000\+ real\-world apis\.External Links:2307\.16789Cited by:[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Shao, V\. Samuel, Y\. Jiang, J\. Yang, and D\. Yang \(2026\)Collaborative gym: a framework for enabling and evaluating human\-agent collaboration\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=GDYueXtKXT)Cited by:[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px2.p1.1)\.
- T\. Shi, J\. He, Z\. Wang, H\. Li, L\. Wu, W\. Guo, and D\. Song \(2025\)Progent: securing ai agents with privilege control\.External Links:2504\.11703,[Link](https://arxiv.org/abs/2504.11703)Cited by:[§1](https://arxiv.org/html/2606.31174#S1.p2.1),[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px3.p1.1),[Table 1](https://arxiv.org/html/2606.31174#S2.T1.1.1.12.11.1)\.
- C\. Wang, W\. Luo, S\. Dong, X\. Xuan, Z\. Li, L\. Ma, and S\. Gao \(2025\)MLLM\-tool: a multimodal large language model for tool agent learning\.External Links:2401\.10727,[Link](https://arxiv.org/abs/2401.10727)Cited by:[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px3.p1.1)\.
- Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu, A\. H\. Awadallah, R\. W\. White, D\. Burger, and C\. Wang \(2024\)AutoGen: enabling next\-gen LLM applications via multi\-agent conversation\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=BAakY1hNKS)Cited by:[§1](https://arxiv.org/html/2606.31174#S1.p2.1),[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px1.p1.1)\.
- T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng, D\. Shin, F\. Lei, Y\. Liu, Y\. Xu, S\. Zhou, S\. Savarese, C\. Xiong, V\. Zhong, and T\. Yu \(2024\)OSWorld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.External Links:2404\.07972Cited by:[§1](https://arxiv.org/html/2606.31174#S1.p2.1),[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px1.p1.1)\.
- F\. F\. Xu, Y\. Song, B\. Li, Y\. Tang, K\. Jain, M\. Bao, Z\. Z\. Wang, X\. Zhou, Z\. Guo, M\. Cao, M\. Yang, H\. Y\. Lu, A\. Martin, Z\. Su, L\. M\. Maben, R\. Mehta, W\. Chi, L\. K\. Jang, Y\. Xie, S\. Zhou, and G\. Neubig \(2026\)TheAgentCompany: benchmarking LLM agents on consequential real world tasks\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=LZnKNApvhG)Cited by:[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhan \(2024\)τ\\tau\-Bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.External Links:2406\.12045,[Link](https://arxiv.org/abs/2406.12045)Cited by:[§1](https://arxiv.org/html/2606.31174#S1.p2.1),[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2606.31174#S2.T1.1.1.1.1)\.
- J\. Yu, Y\. Ding, and H\. Sato \(2025a\)DynTaskMAS: a dynamic task graph\-driven framework for asynchronous and parallel llm\-based multi\-agent systems\.Proceedings of the International Conference on Automated Planning and Scheduling35\(1\),pp\. 288–296\.External Links:ISSN 2334\-0835,[Link](http://dx.doi.org/10.1609/icaps.v35i1.36130),[Document](https://dx.doi.org/10.1609/icaps.v35i1.36130)Cited by:[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Yu, Y\. Zhang, Z\. Wang, J\. Yoon, and M\. Bansal \(2025b\)MEXA: towards general multimodal reasoning with dynamic multi\-expert aggregation\.External Links:2506\.17113,[Link](https://arxiv.org/abs/2506.17113)Cited by:[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px3.p1.1),[Table 1](https://arxiv.org/html/2606.31174#S2.T1.1.1.13.12.1)\.
- W\. Zhang, L\. Zeng, Y\. Xiao, Y\. Li, C\. Cui, Y\. Zhao, R\. Hu, Y\. Liu, Y\. Zhou, and B\. An \(2025\)AgentOrchestra: orchestrating multi\-agent intelligence with the tool\-environment\-agent\(tea\) protocol\.External Links:2506\.12508,[Link](https://arxiv.org/abs/2506.12508)Cited by:[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. Neubig \(2024\)WebArena: a realistic web environment for building autonomous agents\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=oKn9c6ytLx)Cited by:[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Zhu, K\. Tseng, G\. Vernik, X\. Huang, S\. G\. Patil, V\. Fang, and R\. A\. Popa \(2025a\)MiniScope: a least privilege framework for authorizing tool calling agents\.External Links:2512\.11147,[Link](https://arxiv.org/abs/2512.11147)Cited by:[§1](https://arxiv.org/html/2606.31174#S1.p2.1),[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px3.p1.1),[Table 1](https://arxiv.org/html/2606.31174#S2.T1.1.1.12.11.1)\.
- K\. Zhu, H\. Du, Z\. Hong, X\. Yang, S\. Guo, Z\. Wang, Z\. Wang, C\. Qian, X\. Tang, H\. Ji, and J\. You \(2025b\)MultiAgentBench: evaluating the collaboration and competition of llm agents\.Note:Published at ACL 2025 \(Main Conference\)External Links:2503\.01935,[Link](https://arxiv.org/abs/2503.01935)Cited by:[§1](https://arxiv.org/html/2606.31174#S1.p2.1),[§2](https://arxiv.org/html/2606.31174#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2606.31174#S2.T1.1.1.9.8.1)\.

## Appendix ALimitations

Worker\-quality coupling\.The fixed subagent pool is what makes management the only variable across runs \(§[3](https://arxiv.org/html/2606.31174#S3)\), but it ties our findings to a particular worker capability\. A stronger pool could let a sloppier manager succeed by absorbing the slack; a weaker pool could force even careful managers to over\-grant or over\-route\. Both are open questionsClawArena\-Teamdoes not currently answer\.

Execution\-based scoring blind spot\.All checks gate on machine\-verifiable artifacts\. This deliberately rejects LLM\-judge drift but also under\-rewards correct reasoning that produces an unverifiable deliverable, such as a thoughtful conditional answer that the shell command then deems wrong because it does not match an expected literal form\.

Sample size and implementation notes\.The current study evaluates twelve main\-agent models on 41 scenarios; broader coverage is ongoing and will refresh all tables\.claude\-fable\-5is evaluated as shipped with the vendor\-recommended refusal→\\rightarrowopus\-4\-8fallback \(§[4](https://arxiv.org/html/2606.31174#S4)\), so its score is a composite rather than the base model in isolation\. Self\-hosted\-model API costs are imputed from OpenRouter list rates and are approximate\.

## Appendix BData Sample Overview

Each scenario directory contains the workspace, the question sequence, staged updates, and execution checks\. All 258 rounds use execution checks \(100%100\\%\), each declaring an expected exit code and a timeout \(mean82\.982\.9s\) and referencing scenario scripts and the workspace\. Updates split intonew\(61\.1%61\.1\\%\) andreplace\(38\.9%38\.9\\%\) groups, averaging3\.543\.54files per group\.

## Appendix CFramework Details

The main agent is exposed CapitalCase tools \(Read/Write/Edit/Bash/Grep/Glob\) plus subagent\-management tools \(CreateSubagent/RunSubagent/ListSubagents/InspectSubagent\) and aWorkflowtool\. System\-level signals \(environment, token thresholds\) are injected via<system\-reminder\>blocks and background\-task completions via<task\-notification\>blocks\.Readdegrades out\-of\-modality files to a textual placeholder and suggests delegation, so a text\-only main agent must route multimodal content to avlm/omnisubagent\.

#### Subagent control surface\.

CreateSubagentfixes a subagent’s system prompt, model key \(llm/vlm/omni\), tool subset, and path whitelist \(a subset of the main agent’s own paths\);RunSubagentexecutes it in the foreground or background and may either start a new session or continue an existing one \(session reuse\)\. TheWorkflowtool accepts a compact JavaScript\-style DSL withagent\(\),parallel\(\), andpipeline\(\)primitives that drive multiple subagents over the*same*pool, enabling programmatic parallel and staged orchestration\.Readapplies soft/hard size thresholds \(notice then truncation\) to bound context; all path access isrealpath\-checked and symlink escapes are rejected and counted as forbidden accesses \(the MAF/SAFt statistics\)\.

## Appendix DImplementation Details

The subagent pool is served locally with vLLM and held identical across all evaluated main agents \(Table[5](https://arxiv.org/html/2606.31174#A4.T5)\)\. Per\-turn generation is capped at24,00024\{,\}000tokens to bound runaway generation\. The200200k main\-agent context value is an advisory upper bound \(not a hard cutoff in the evaluated runs\); effective context depends on each model, whereas subagent budgets are uniform across runs by construction\. Main agents are reached through provider\-native or OpenAI\-compatible APIs \(Gemini, OpenRouter, a ChatMock bridge forcodex, and a local server for self\-hosted models\)\.

Table 5:Fixed subagent pool \(controlled variable\)\. Every main agent commands this same team; only the main agent varies across runs\.#### claude\-fable\-5refusals and fallback\.

claude\-fable\-5ships with safety classifiers that can decline a request, returningstop\_reason: refusalas a successful response; the vendor\-recommended pattern is to retry the refused turn on another Claude model\(Anthropic,[2026b](https://arxiv.org/html/2606.31174#bib.bib23)\)\. We evaluatefable\-5as shipped, with an automatic refusal→\\rightarrowclaude\-opus\-4\-8fallback;124124such fallbacks fired over the run, concentrated in a single security and PII\-heavy scenario \(security\_pcap\_triage\)\. Because that scenario also exposed a scenario\-levelasynciodeadlock unrelated tofable\(it likewise stalls underopus\-4\-8but completes undersonnet\-4\-6\), itsfable\-5entry was recovered by runningopus\-4\-8directly with a per\-round timeout, yielding only1/61/6rounds and draggingfable\-5’s score down on that single scenario\.fable\-5’s reported cost \(Appendix[F](https://arxiv.org/html/2606.31174#A6)\) prices all main\-agent tokens atfable\-5list rates and is therefore an upper estimate: refused turns are unbilled and fallback credit refunds the prompt\-cache cost of switching\.

## Appendix EScoring Details

The management factor weights TPP, ROC, WPP, and MCA equally\. ROC and MCA are near\-saturated because they encode clear\-cut decisions; TPP and WPP carry most of the discriminative signal\. We deliberately use no LLM judge: checks gate on exact numeric/string/regex matches or recomputation, and method constraints are applied only when a question explicitly requests a behavior \(an instruction\-following gate\)\.

#### Correctness variants we omit from the headline\.

Two alternative correctness summaries were computed during development but excluded from the mainSmsfor parsimony\. A per\-scenario averaging of TCR \(mean of per\-scenario completion rates\) tracks the headline TCR almost identically, so the two convey the same information; we report only the headline\. A full\-pass rate \(the fraction of scenarios in which every round is correct\) is near zero for every evaluated model, which makes it useful as a difficulty signal but uninformative as a per\-model headline; we record it in the data release but exclude it fromSms\.

## Appendix FPricing Methodology

Main\-agent cost isMIT⋅pcw\+MCR⋅pcr\+MOT⋅pout\\mathrm\{MIT\}\\cdot p\_\{\\mathrm\{cw\}\}\+\\mathrm\{MCR\}\\cdot p\_\{\\mathrm\{cr\}\}\+\\mathrm\{MOT\}\\cdot p\_\{\\mathrm\{out\}\}, where MIT/MCR/MOT are input/cache\-read/output token totals andpcwp\_\{\\mathrm\{cw\}\}uses the listed cache\-write price when one is published—1\.25×1\.25\\timesinput forclaudemodels \(fable\-5,opus\-4\-8,sonnet\-4\-6\) and the smaller dedicated rates for the twogeminimodels—and the input price otherwise\. Subagents always run on the fixed local pool and are not priced\. Prices are provider or OpenRouter list rates \(Table[6](https://arxiv.org/html/2606.31174#A6.T6)\); because the underlying token counts are post\-hoc estimates \(Appendix[I](https://arxiv.org/html/2606.31174#A9)\), the resulting costs are likewise estimates\.claude\-fable\-5is priced entirely at its own list rates, so its cost is an upper estimate \(refused turns are unbilled and fallback credit refunds the cache cost of switching toopus\-4\-8; §[4](https://arxiv.org/html/2606.31174#S4)\)\.

Table 6:Provider / OpenRouter list prices \(USD / 1M tokens\) and computed main\-agent cost per full run\. A cache\-write price of “—” means no separate rate is published, so input tokens are priced at the input rate;claudemodels use a1\.25×1\.25\\times\-input cache\-write rate\.†Default provider lacks prompt caching; cache\-read priced at the input rate\.‡opus\-4\-8has no standalone run; its cost repricesfable\-5’s token totals atopus\-4\-8rates \(half offable\-5\), shown because it isfable\-5’s refusal fallback \(§[4](https://arxiv.org/html/2606.31174#S4)\)\.
## Appendix GBenchmark Construction: Workflow\-Driven Synthesis under a Test Flywheel

ClawArena\-Teamis built rather than collected: every scenario—workspace corpora, multimodal assets, ground truth, and execution checks—is procedurally synthesized\. Two ideas organize the methodology\. First, scenarios are produced by*workflow\-driven parallel authoring*: a controller decomposes a batch into per\-scenario contracts and fans the work out to isolated subagent\-authors\. Second, no scenario is considered finished until it survives an*end\-to\-end test flywheel*in which a real subagent\-management run is the final acceptance gate, and its failures are fed back into the data\. There is a deliberate reflexivity here: the construction pipeline is itself an instance of subagent management—a single controller that decomposes work, selects model tiers \(empowerment\), confines parallel authors under least\-privilege contracts, and integrates their deliverables behind objective gates\.ClawArena\-Teamis, in effect, authored by the very capability it measures\.

### G\.1Why full synthesis

Full synthesis is chosen over scraping for five reasons\.Controllable truth:when content is generated, the ground truth is known by construction, so the answer and the corpus can be emitted by the same script\.License cleanliness:no third\-party copyrighted assets enter the release\.Reproducibility:generation scripts are version\-controlled, so a rerun yields byte\-identical or near\-identical artifacts\.Conflict control:adversarial designs such as an audio recording that contradicts its transcript, or an image whose caption misleads, require the divergence point to be placed with precision—only possible when both sides are authored\.Distribution control:because the corpus is authored rather than found, its composition can be steered to what the benchmark must measure—modality mix, file\-format diversity, domain coverage, difficulty\-vector incidence, and the capability tags each round exercises—instead of inheriting whatever a scrape happens to contain; the resulting distribution is characterized in Appendix[H](https://arxiv.org/html/2606.31174#A8)\. A consistent resource priority governs asset production: locally served models first \(free at inference\), then deterministic local code and tools \(Pillow/matplotlib/graphviz/ffmpeg/office libraries\), and paid APIs only as a last resort for photoreal textures that code cannot render\.

### G\.2The authoring substrate

Synthesis is organized as a reproducible authoring layer kept separate from the validated release: generation code is version\-controlled while bulky intermediate renders are not, so the pipeline is reproducible rather than hidden\. A shared helper library centralizes asset synthesis across modalities behind a uniform interface—token accounting against the same tokenizer the evaluation harness uses; text\-to\-speech audio; slideshow and procedurally animated video; office and binary container formats \(spreadsheets, documents, PDFs, encrypted archives, columnar and embedded databases\); and a family of image renderers \(charts, engineering diagrams, annotated medical and remote\-sensing imagery, synthetic screenshots\)—paired with post\-hoc validators that reject undersized or malformed assets\. Each scenario is a self\-contained unit that pairs a frozen design contract, deterministic synthesis code, the check sources, and a reverse\-verification harness\. Its build entry point is idempotent—it resets and regenerates the entire scenario—and ends by self\-checking the corpus against two budget floors derived from the design criteria: the inaccessible\-only corpus that the main agent must delegate to read, and the staged\-update payload\.

### G\.3Authoring a single scenario

A single scenario follows a fixed seven\-step recipe: \(1\) read the spec \(the C1–C10 criteria and the format and pitfall checklists\); \(2\) scaffold the idempotent build script; \(3\) lay down the bulk text corpus to satisfy the context\-pressure floor; \(4\) render multimodal assets \(image→\\rightarrowaudio→\\rightarrowvideo, each double\-checked at generation\); \(5\) write the scenario manifest, the question sequence, and the per\-round checks, attaching each staged update to the*first*round that must observe its effect; \(6\) simulate the golden answer in a temporary copy and confirm every check passes, then write a plausible\-but\-wrong answer and confirm a check fails; \(7\) strip any ground\-truth residue and pass the whole\-set validator\.

Two engineering principles keep scenarios robust\.Prompt–check interlock:the more a question is written in natural intent \(no field names, no tool names, no subagent recipe\), the more its check must match*by shape*\(an integer≥45\\geq 45, an ISO date, a non\-empty string containing a stem\) rather than by key name; a recipe\-style prompt paired with a shape\-blind check is the dominant source of spurious failure\.Three\-layer checks:every check verifies structure \(file exists, skeleton present, JSON parses\), then fields \(non\-empty, correctly typed\), then truth \(exact numeric match, regex/substring, logical consistency\)—never whole\-segment semantic comparison, and never an LLM judge\. To make these difficulties composable, the methodology maintains a reusable difficulty\-vector library \(Table[7](https://arxiv.org/html/2606.31174#A7.T7)\); new scenarios combine de\-recipe with content traps plus at least one structural and one detail vector\.

Table 7:Reusable difficulty vectors combined when authoring or hardening a scenario\.The seventh step is enforced by a per\-scenario reverse\-verification harness: it materializes the golden deliverables in a temporary workspace and asserts every round passes, then injects targeted perturbations and asserts the corresponding round fails\. In one incident\-analysis scenario, for instance, the perturbations replace the correct failed\-component identifier \(its round must then fail\) and substitute a transcript’s decoy numeric value for the recording’s true value \(a modality trap; that round must then fail\)\. This guards against checks silently degrading into always\-pass\.

### G\.4Workflow\-driven parallel authoring

Each batch of scenarios is produced by a three\-phase procedure\.Phase A \(lock the skeleton\)is controller\-only: the controller internalizes the design handbook and the prior batch’s post\-mortem, writes a batch plan and an authoring guide, hand\-builds one proof\-of\-concept scenario through all gates as the gold reference, and freezes every scenario’s anchors in a short per\-scenario contract\.Phase B \(parallel fan\-out\)spawns one subagent\-author per remaining scenario, bounded at a small concurrency \(beyond it, shared\-file write contention and progress\-tracking cost come to dominate\), with file\-system isolation for slow asset jobs\.Phase C \(convergence\)returns to the controller, who assembles the shared manifests once, runs whole\-set validation, and starts the flywheel\.

The chief risk of parallel authoring is*ground\-truth drift*—independent authors silently choosing conflicting anchors\. Four mechanisms prevent it: the per\-scenario contract as a binding specification \(anchor table, workspace layout, per\-round question and check anchors, token\-budget split\); the proof\-of\-concept referenced as a worked example; a distilled authoring guide so authors need not re\-derive the handbook; and an explicit no\-touch list of shared files whose violation triggers an immediate revert rather than author self\-correction\. Model tiers are assigned by task: cheaper models for mechanical padding and I/O\-bound jobs, stronger models for check\-interlock logic and anchor design\. This division—decompose, contract, empower by tier, confine by least privilege, integrate behind gates—is exactly the management surface the benchmark scores, the reflexivity noted above\. Crucially, the procedure is*portable*: nothing in it is specific toClawArena\-Team’s domains or asset types, so the same controller–contract–gate workflow transfers to authoring other execution\-checked agentic benchmarks\. The orchestration workflow, not any individual scenario, is the reusable methodological artifact we expect to outlast the particular set it produced\.

### G\.5The end\-to\-end test flywheel

Acceptance proceeds through four static gates and one dynamic gate \(Table[8](https://arxiv.org/html/2606.31174#A7.T8)\)\. The static gates establish that the data is*well\-formed*; they are necessary but not sufficient\. The dynamic gate—a real baseline run of the full subagent\-management harness against the fixed local pool—establishes that the data is*actually solvable*and that checks are neither over\-strict nor over\-loose\. Without a baseline run, one cannot tell whether a scenario is the right difficulty, whether a check is too tight, or whether the ground truth can be recovered at all\.

Table 8:Four static gates plus the dynamic baseline gate\. G1–G4 verify form; D verifies that the scenario is solvable and the checks are calibrated\.Every baseline failure is then assigned to exactly one of two classes\. A*true\-difficulty*failure—an omni pool missing a key spoken figure, a model taking the modality decoy, a fabricated identifier, a directory over\-granted to a subagent—is the value of the benchmark and is never loosened\. A*false\-kill*—an equivalent answer rejected by a brittle regex, an order\-independent field locked to an order, a tolerance gap—is repaired by loosening the check on*both*the deployed and source copies, after which the perturbation guard must still fail\. The flywheel \(Figure[5](https://arxiv.org/html/2606.31174#A7.F5)\) then repeats until every remaining failure is true\-difficulty\. The governing health criterion is counter\-intuitive: a low completion rate with100%100\\%true\-difficulty failures is far healthier than a high rate propped up by loosened false\-kills\.

Lock skeletonplan/guide/briefParallel fan\-out≤6\\leq 6authorsStatic four gatesbuild/verify/check/statsBaseline runfixed local poolFail triagetrue\-diff vs\. false\-killData iterationharden/fix/re\-anchorSynthesis–Verification FlywheelFigure 5:The end\-to\-end test flywheel\. Static gates verify form; the baseline run verifies solvability; triage routes each failure back into the data until only true\-difficulty failures remain\.
### G\.6Iterative refinement and reproducibility

The released set is not a single\-pass artifact but the fixed point of repeated flywheel turns\. Across successive authoring iterations the difficulty was ratcheted deliberately: each iteration introduced harder structural and adversarial vectors and was accepted only once its baseline run left every failure classified as true difficulty rather than a check artifact\. A final systematic audit re\-triaged the checks across the whole set for the over\-loose, over\-strict, and parsing\-miss patterns the flywheel had surfaced, paired with a multimodal re\-rendering pass and a full re\-check against the fixed local pool\. Because every asset, check, and budget floor is scripted and self\-verifying, the entire benchmark is regenerable from source, and its construction history lives in version control rather than in any released artifact\.

## Appendix HDataset Composition and Statistics

All statistics in this section describe the full 41\-scenario set \(clawarena\-team\); per\-wave subsets are not reported\. Quantities are computed with a singleqwen3tokenizer for cross\-item comparability\. At a glance: 41 scenarios, 258 rounds \(44 preceded by staged updates,17\.1%17\.1\\%\), 72 update groups over 255 files,170\.5170\.5MiB of workspace content, and28\.928\.9M tokens\.

### H\.1Token and modality composition

Workspace content dominates token volume while staged updates contribute over a quarter \(Table[9](https://arxiv.org/html/2606.31174#A8.T9)\)\. By modality \(Table[10](https://arxiv.org/html/2606.31174#A8.T10)\), text accounts for most files and tokens, but audio and image dominate raw*bytes*—confirming a substantial non\-text payload that a text\-only main agent cannot consume directly\.

Table 9:Token composition of the benchmark\.Table 10:Workspace composition by modality \(document=\.pdf/\.docx/\.xlsx;other= data/code/archives\)\.
### H\.2File types

The benchmark deliberately mixes many formats so that managers must route diverse content \(Table[11](https://arxiv.org/html/2606.31174#A8.T11)\): Markdown and logs carry most text tokens, uncompressed\.wavaudio dominates bytes, and office/data formats \(\.docx,\.pdf,\.parquet,\.csv,\.ndjson,\.eml,\.html\) appear throughout\.

Table 11:Top file extensions by byte share\.
### H\.3Multimodal coverage

Audio spans 29 files totaling3,090\.63\{,\}090\.6s \(≈51\.5\\approx 51\.5minutes\); video spans 19 files totaling9,1069\{,\}106frames; images number 188\. The largest files are uncompressed\.wavinterviews and statements \(up to4\.74\.7MiB\)\. Multimodal payload appears in nearly every scenario—e\.g\.hospital\_safety\_event\_reviewalone contains 37 images—so modality routing is exercised broadly rather than in a few special cases\.

### H\.4Staged updates

Update groups are61\.1%61\.1\\%newand38\.9%38\.9\\%replace, averaging3\.543\.54files per group \(max 15; 255 files total\)\. Replaced files force the manager to supersede earlier beliefs; new files extend the workspace mid\-task\.

### H\.5Capability\-tag taxonomy

Every round is annotated with controlled capability tags from a 51\-tag vocabulary organized into eight sections \(Table[12](https://arxiv.org/html/2606.31174#A8.T12)\);98\.4%98\.4\\%of rounds carry≥1\\geq 1tag, averaging4\.484\.48tags per round, and all eight sections reach100%100\\%section coverage\. Table[13](https://arxiv.org/html/2606.31174#A8.T13)lists the most frequent tags and Figure[6](https://arxiv.org/html/2606.31174#A8.F6)the full distribution\. The taxonomy spans the entire management capability surface—delegation/permission, modality routing, update handling, adversarial traps, office/data formats, code execution, and information synthesis—evidencing that the benchmark systematically exercises subagent management rather than a single skill\.

Table 12:Capability\-tag taxonomy: eight sections covering the management surface \(51 controlled tags; all sections100%100\\%used\)\.Table 13:Most frequent capability tags\. Rd%: share of the 258 rounds; Scn%: share of the 41 scenarios\.![Refer to caption](https://arxiv.org/html/2606.31174v1/x4.png)Figure 6:Capability\-tag coverage across rounds\.
### H\.6Largest scenarios

By tokens:trading\_tz\_incident\(5\.095\.09M\),satellite\_change\_detection\(1\.501\.50M\),security\_incident\_triage\(1\.331\.33M\),security\_pcap\_triage\(1\.141\.14M\),oss\_supply\_chain\_audit\(1\.121\.12M\)\. By workspace size:candidate\_background\_check\(9\.49\.4MiB\),smart\_home\_anomaly\_triage\(9\.39\.3MiB\),trading\_tz\_incident\(9\.19\.1MiB\)\. The most rounds occur incodebase\_migration\_review\(10\) andml\_rl\_policy\_review\(8\)\.

## Appendix IToken Usage

#### Estimation caveat\.

All token counts in this paper are computed*post hoc*by applying a single, unified tokenizer and chat template uniformly across all models, rather than read from each provider’s billing\. They therefore approximate—and may deviate slightly from—each provider’s own tokenization and true token consumption\. All token totals below, and the costs derived from them \(Appendix[F](https://arxiv.org/html/2606.31174#A6)\), should be read as estimates for cross\-model comparison rather than exact bills\.

Tables[14](https://arxiv.org/html/2606.31174#A9.T14)and[15](https://arxiv.org/html/2606.31174#A9.T15)report main\- and subagent\-side token usage\. Main context peak \(MCM\) relative to the200200k advisory value \(MCU%\) shows that some models approach or exceed it\. Subagent totals \(SIT/SOT/SCH\) confirm that the bulk of token volume is borne by the fixed local pool, which is why only the main agent is priced\.

Table 14:Main\-agent token usage\. MCM: context peak; MCU%: vs\. 200k advisory; MIT: input total; MCR: cache\-read total\. Output totals \(MOT\) omitted for space\.Table 15:Subagent token usage \(fixed local pool, not priced\)\. SIT/SOT: input/output totals; SCH: cache\-hit total\.

## Appendix JPer\-Scenario Breakdown

Figure[7](https://arxiv.org/html/2606.31174#A10.F7)showsSmsfor every scenario×\\timesmodel, with scenarios ordered by mean difficulty\. The figure is regenerated as the model set grows\.

![Refer to caption](https://arxiv.org/html/2606.31174v1/x5.png)Figure 7:Per\-scenarioSms\(%\): 12 models \(rows\)×\\times41 scenarios \(columns\), scenarios sorted by mean score \(left = easiest\)\. The separated strip on top shows, for each scenario, the meanSmsover the eleven capable models \(excludingglm\-4\.7\-flash, the capability\-cliff outlier\), giving a per\-scenario difficulty readout undistorted by the weakest model\.
## Appendix KTool\-Grant and Bash\-Mode Statistics

We additionally log, per model, the distribution of granted tool types \(TGC\) and the foreground/background split ofBashcalls \(BSH\) together with structured\-output usage \(SOC\); these support the behavioral analysis in §[4](https://arxiv.org/html/2606.31174#S4)and are omitted here for space\.

## Appendix LCase Studies

The twelve cases below, presented as Figures[8](https://arxiv.org/html/2606.31174#A12.F8)–[10](https://arxiv.org/html/2606.31174#A12.F10), are drawn directly from the recorded run transcripts andmetadata\.jsonfiles; all numbers and quoted strings are verbatim from those artifacts\. They span single\-model deep\-dives, head\-to\-head model comparisons, successes, and characteristic failure modes, concretely grounding the three findings and the management capability surface\.

![Refer to caption](https://arxiv.org/html/2606.31174v1/x6.png)Figure 8:Case studies 1–4: over\-granting under the strongest manager, correct modality routing, the modality\-decoy trap, and cost decoupled from quality\. All values are verbatim from run transcripts andmetadata\.json\.![Refer to caption](https://arxiv.org/html/2606.31174v1/x7.png)Figure 9:Case studies 5–8: identical scores with divergent forbidden\-access, authoring a dynamic workflow \(and a fan\-out that crashes\), logical parallelism without true background scheduling, and true background plus workflow parallelism\. All values are verbatim from run transcripts andmetadata\.json\.![Refer to caption](https://arxiv.org/html/2606.31174v1/x8.png)Figure 10:Case studies 9–12: a pending correction that never propagates, a capability cliff the manager cannot operate, the management ceiling \(parallel fan\-out, adaptive retry, least privilege\), and what good management looks like\. All values are verbatim from run transcripts andmetadata\.json\.