DAREBench: Deployment-Aware and Reliable Evaluation of Models as Agents

# DAREBench: Deployment-Aware and Reliable Evaluation of Models as Agents
Source: [https://arxiv.org/html/2609.06059](https://arxiv.org/html/2609.06059)
Zhilin LiuZhiwei YangShaojie ZhangZheyuan DengTingwei HuangZhenbo LuoLei JiangYanbing LiuPei Fu\\corresponding

###### Abstract

As large language models evolve from question\-answering systems into general\-purpose agents, evaluation must move beyond static answer correctness to assess multimodal perception, multi\-step execution, tool use, and artifact delivery\. However, existing benchmarks are often tied to specific task types, execution environments, or scoring protocols, limiting their comparability, interpretability, and reliability for deployment decisions\. We introduce DAREBench \(Deployment\-Aware and Reliable Evaluation of Models as Agents\), a benchmark designed to capture workload variation and support reliable agent evaluation\. Built on a shared OpenClaw execution environment, DAREBench organizes 233 tasks selected and adapted from 22 source benchmarks into a2×32\\times 3workload matrix defined by input modality and execution form, and evaluates them under a unified contract\-based protocol with evidence\-based score auditing\. We evaluate 23 commercial API models and 12 locally deployed open\-weight models over 7,587 model–task runs, reporting accuracy and token consumption alongside reference costs for API models\. Results show that no single model dominates all workload groups, text and multimodal tasks exhibit distinct accuracy–cost trade\-offs, and local open\-weight models are competitive in several groups but still trail frontier commercial models overall\. These findings suggest that agent deployment and model selection should consider workload profiles, deployment mode, and accuracy–cost trade\-offs rather than rely on a single aggregate score\.The benchmark details can be found athttps://github\.com/SeerRay\-Lab/DAREBench\.

1Institute of Information Engineering, Chinese Academy of Sciences

2School of Cyber Security, University of Chinese Academy of Sciences

3MiLM Plus, Xiaomi Inc\.

4Department of Computer Science, Brown University

## 1Introduction

![Refer to caption](https://arxiv.org/html/2609.06059v1/acc_cost.png)Figure 1:Accuracy–cost trade\-offs for models on \(a\) text and \(b\) multimodal tasks\. Labels show average tokens per task; cost uses a logarithmic scale\. Dashed lines mark the Pareto frontiers: MiMo\-V2\.5, MiMo\-V2\.5\-Pro, and GPT\-5\.5 in \(a\), and MiMo\-V2\.5, GPT\-5\.4, Gemini\-3\.1\-Pro, and Claude\-Opus\-4\.8 in \(b\)\.Frontier large language models \(LLMs\) are evolving from standalone question\-answering systems into core decision\-making components of general\-purpose agent systems\. These agents must perceive, reason, plan, use tools, and deliver artifacts in dynamic workspaces, rather than merely generate an answer string\([Wei et al\. 2022](https://arxiv.org/html/2609.06059#bib.bib38);[Schick et al\. 2023](https://arxiv.org/html/2609.06059#bib.bib37)\)\. Accordingly, model evaluation is shifting from static input–output performance to end\-to\-end execution under constraints on tools, context, time, and cost\.

However, existing evaluation frameworks do not adequately support this transition\. Traditional knowledge, reasoning, and long\-context benchmarks primarily assess answer correctness under static inputs\([Hendrycks et al\. 2021](https://arxiv.org/html/2609.06059#bib.bib34);[Rein et al\. 2024](https://arxiv.org/html/2609.06059#bib.bib3);[Bai et al\. 2025c](https://arxiv.org/html/2609.06059#bib.bib5)\)\. Multimodal benchmarks incorporate visual inputs but still largely follow a static input–output paradigm\([Yue et al\. 2024](https://arxiv.org/html/2609.06059#bib.bib35);[Wang et al\. 2024](https://arxiv.org/html/2609.06059#bib.bib16)\)\. Tool\-use and end\-to\-end agent benchmarks, by contrast, are often confined to specific command\-line, web, or desktop environments\([Merrill et al\. 2026](https://arxiv.org/html/2609.06059#bib.bib14);[Xie et al\. 2024](https://arxiv.org/html/2609.06059#bib.bib20);[Mialon et al\. 2024](https://arxiv.org/html/2609.06059#bib.bib25);[Liu et al\. 2024](https://arxiv.org/html/2609.06059#bib.bib26);[Zhou et al\. 2024](https://arxiv.org/html/2609.06059#bib.bib28);[Yao et al\. 2025](https://arxiv.org/html/2609.06059#bib.bib27)\)\. Moreover, these benchmarks employ different runtimes, tool interfaces, task protocols, and scoring criteria, making their results difficult to compare directly\. Recent studies have improved evaluation observability through execution trajectories, environment snapshots, and audit logs\([Ye et al\. 2026](https://arxiv.org/html/2609.06059#bib.bib29);[Li et al\. 2026a](https://arxiv.org/html/2609.06059#bib.bib40)\)\. Nevertheless, these studies still primarily report overall success rates within specific environments\. They therefore provide limited insight into model behavior across modalities, execution processes, and degrees of tool dependence\. As a result, they offer limited support for comparable, interpretable, and reliable model selection in practice\.

We argue that agent evaluation can inform deployment decisions only by addressing three interrelated questions, rather than by simply increasing benchmark scale\. First,are the results interpretable?Agent tasks differ substantially in input modality, execution process, and tool dependence\. A single aggregate score can obscure model strengths and weaknesses across workloads\. Second,are comparisons attributable?When tasks rely on different agent frameworks, tool interfaces, and execution protocols, observed differences conflate model capability with runtime compatibility\. Third,are the scores trustworthy?Open\-ended agent tasks often involve multi\-step operations and artifact delivery\. Matching only the final text cannot verify whether a task was actually completed\. Moreover, an LLM judge may incorrectly award positive credit when tools were not executed, the task timed out, or required artifacts were missing\. Together, these issues call for a deployment\-oriented benchmark that unifiesworkload organization, execution environment control, and outcome verification\.

To address these requirements, we propose DAREBench \(Deployment\-Aware andReliableEvaluation of Models as Agents\), a benchmark for deployment\-aware and reliable agent evaluation\. To improve workload\-level interpretability, we select and adapt 233 tasks from 22 existing benchmarks through a three\-stage curation process based on measurability, difficulty, and representativeness\. We organize these tasks into a2×32\\times 3workload matrix defined by input modality and execution form, covering text and multimodal inputs as well as single\-step, multi\-step, and tool\-mediated multi\-step tasks\. To improve the attribution of cross\-benchmark comparisons, we build a shared execution environment on OpenClaw\([Steinberger and OpenClaw Contributors 2026](https://arxiv.org/html/2609.06059#bib.bib36)\)\. All models operate under a common agent loop, tool interface, workspace abstraction, and logging protocol, thereby reducing the influence of heterogeneous agent scaffolds\. For open\-ended tasks, we define a task contract specifying the initial workspace, available tools, target artifacts, scorer, and time budget\. Task outcomes are evaluated through automated checks, LLM\-as\-a\-Judge, or hybrid scoring\. We further introduce an evidence\-based auditing mechanism that combines deterministic post\-hoc rules with a meta\-judge to review suspicious positive scores unsupported by execution trajectories or final artifacts\. Using this framework, we evaluate 23 commercial API models and 12 locally deployed open\-weight models over 7,587 applicable model–task runs\. We report accuracy and token consumption, and use public pricing to estimate reference costs for API models and analyze their accuracy–cost trade\-offs\. Results show that model strengths are workload\-dependent and that text and multimodal API workloads exhibit distinct accuracy–cost frontiers\. Recent locally deployed open\-weight models are competitive in several task groups, although they still trail frontier commercial models overall\. The audit further identifies and removes 143 positive scores unsupported by execution evidence\. Our main contributions are as follows:

- •We introduce DAREBench, comprising 233 tasks adapted from 22 source benchmarks and organized under a unified runtime into a2×32\\times 3workload matrix defined by modality and execution form\.
- •We evaluate 35 commercial API and locally deployed open\-weight models over 7,587 model–task runs, revealing workload\-dependent rankings, distinct accuracy–cost trade\-offs across text and multimodal workloads, and the strengths and limitations of local models\.
- •We characterize unsupported positive scores in agent evaluation and introduce an evidence\-based audit that verifies suspicious scores against execution trajectories and final artifacts\.

## 2Related Work

### 2\.1Agent Benchmarks and Capability Sources

Agent evaluation is shifting from static question answering to execution in operational environments\. Terminal\-Bench\([Merrill et al\. 2026](https://arxiv.org/html/2609.06059#bib.bib14)\), OSWorld\([Xie et al\. 2024](https://arxiv.org/html/2609.06059#bib.bib20)\), and MMSearch\([Jiang et al\. 2025](https://arxiv.org/html/2609.06059#bib.bib22)\)respectively assess command\-line workflows, GUI interaction, and multimodal search, but remain tied to specific domains or environments\. OpenClaw has further motivated general\-agent benchmarks\. Claw\-Eval\([Ye et al\. 2026](https://arxiv.org/html/2609.06059#bib.bib29)\)emphasizes trace\-based trustworthy evaluation; ClawBench\([Zhang et al\. 2026](https://arxiv.org/html/2609.06059#bib.bib30)\)targets real\-time web interaction; ClawArena\([Ji et al\. 2026](https://arxiv.org/html/2609.06059#bib.bib31)\)studies conflict reasoning; WildClawBench\([Ding et al\. 2026](https://arxiv.org/html/2609.06059#bib.bib32)\)and ZClawBench\([Z\.AI 2026](https://arxiv.org/html/2609.06059#bib.bib39)\)introduce in\-the\-wild tool\-use tasks; and ClawsBench\([Li et al\. 2026b](https://arxiv.org/html/2609.06059#bib.bib33)\)jointly evaluates capability and safety\. However, differing runtimes, task taxonomies, and scoring protocols hinder systematic comparison under a unified framework\.

![Refer to caption](https://arxiv.org/html/2609.06059v1/overview.png)Figure 2:Overview of DAREBench\. The benchmark comprises four stages: \(1\) constructing 233 tasks from 22 source benchmarks and organizing them by modality and execution form; \(2\) executing all models in a shared agent environment; \(3\) applying automated, LLM\-based, or hybrid scoring with evidence\-based auditing; and \(4\) reporting overall and workload\-level accuracy, token usage, reference cost, and accuracy–cost frontiers\.
### 2\.2LLM\-as\-a\-Judge in Agent Evaluation

Following its validation on MT\-Bench and Chatbot Arena\([Zheng et al\. 2023](https://arxiv.org/html/2609.06059#bib.bib24)\), LLM\-as\-a\-Judge has been applied to code\([Jimenez et al\. 2024](https://arxiv.org/html/2609.06059#bib.bib41);[Jiang et al\. 2026](https://arxiv.org/html/2609.06059#bib.bib42)\), multimodal\([Chen et al\. 2024](https://arxiv.org/html/2609.06059#bib.bib43)\), dialogue\([Dubois et al\. 2024](https://arxiv.org/html/2609.06059#bib.bib44)\), and security evaluation\([Shao et al\. 2026](https://arxiv.org/html/2609.06059#bib.bib45)\)\. Agent tasks are harder to judge because they involve long\-horizon tool use, dynamic interactions, and artifact verification\([Li et al\. 2025](https://arxiv.org/html/2609.06059#bib.bib46)\)\. Prior work shows that judges may infer success from plausible outputs without verifying execution evidence, causing “hallucinated correctness”\([You et al\. 2026](https://arxiv.org/html/2609.06059#bib.bib47);[Gurram 2026](https://arxiv.org/html/2609.06059#bib.bib48)\)\. Claw\-Eval\([Ye et al\. 2026](https://arxiv.org/html/2609.06059#bib.bib29)\)likewise reports substantial misses without audit logs\. Yet systematic characterization and mitigation of such hallucinations remain limited\.

## 3DAREBench

As illustrated in Figure[2](https://arxiv.org/html/2609.06059#S2.F2), DAREBench consists of four components: workload construction, shared agent execution, reliable scoring, and deployment\-oriented metrics\. It organizes 233 tasks from 22 source benchmarks into a2×32\\times 3workload matrix and evaluates all models under a unified execution protocol\. Performance is assessed using automated, LLM\-based, or hybrid scoring with evidence\-based auditing, and reported alongside token usage and reference cost\.

### 3\.1Problem Definition

Unlike conventional benchmarks that evaluate only final responses, agentic tasks require models to interact with an execution environment and produce verifiable outputs\. We formalize each task as

τi=\(pi,𝒲i0,𝒯i,𝒞i,Gi,timax\),\\tau\_\{i\}=\\bigl\(p\_\{i\},\\ \\mathcal\{W\}\_\{i\}^\{0\},\\ \\mathcal\{T\}\_\{i\},\\ \\mathcal\{C\}\_\{i\},\\ G\_\{i\},\\ t\_\{i\}^\{\\max\}\\bigr\),\(1\)wherepip\_\{i\}is the task instruction,𝒲i0\\mathcal\{W\}\_\{i\}^\{0\}the initial workspace,𝒯i\\mathcal\{T\}\_\{i\}the available tool set,𝒞i\\mathcal\{C\}\_\{i\}the artifact contract,GiG\_\{i\}the task\-specific scorer, andtimaxt\_\{i\}^\{\\max\}the execution\-time budget\. The artifact contract specifies the required output paths and their validity predicates:

𝒞i=\{\(ai,j,ϕi,j\)\}j=1\|𝒞i\|,ϕi,j:file bytes→\{0,1\}\.\\mathcal\{C\}\_\{i\}=\\bigl\\\{\(a\_\{i,j\},\\phi\_\{i,j\}\)\\bigr\\\}\_\{j=1\}^\{\|\\mathcal\{C\}\_\{i\}\|\},\\qquad\\phi\_\{i,j\}:\\text\{file bytes\}\\rightarrow\\\{0,1\\\}\.\(2\)Let𝒲∗​\[a\]\\mathcal\{W\}^\{\*\}\[a\]denote the content at pathaain the final workspace\. Complete contract satisfaction is defined as

Sat⁡\(𝒲∗,𝒞i\)=∏j=1\|𝒞i\|\[∃𝒲∗​\[ai,j\]\]​ϕi,j​\(𝒲∗​\[ai,j\]\)\.\\operatorname\{Sat\}\(\\mathcal\{W\}^\{\*\},\\mathcal\{C\}\_\{i\}\)=\\prod\_\{j=1\}^\{\|\\mathcal\{C\}\_\{i\}\|\}\\mathbf\{1\}\\\!\\left\[\\exists\\,\\mathcal\{W\}^\{\*\}\[a\_\{i,j\}\]\\right\]\\phi\_\{i,j\}\\\!\\left\(\\mathcal\{W\}^\{\*\}\[a\_\{i,j\}\]\\right\)\.\(3\)Under this formulation, a model operates within a shared execution environment to produce the required artifacts, which are evaluated byGiG\_\{i\}\. Task construction and scoring are described in §[3\.2](https://arxiv.org/html/2609.06059#S3.SS2)and §[3\.3](https://arxiv.org/html/2609.06059#S3.SS3), respectively\.

### 3\.2Task Construction

#### Query Selection\.

To construct a diverse and deployment\-relevant evaluation suite, we draw candidate tasks from 22 publicly available benchmarks spanning reasoning, long\-context understanding, question answering, instruction following, domain\-specific analysis, code and shell workflows, desktop control, and multimodal understanding and search\.

We apply a three\-stage curation procedure\. First, thefeasibility filterremoves tasks for which an explicit artifact contract𝒞i\\mathcal\{C\}\_\{i\}or a reliable scorerGiG\_\{i\}cannot be defined\. Second, thedifficulty filterexcludes overly simple tasks\. For single\-step tasks, we run Qwen3\.5\-27B three times and remove tasks solved correctly in all three trials\. Where source difficulty annotations are available, we retain only tasks labeledmediumorhard\. Finally, therepresentativeness filtermanually selects tasks according to the category structure of each source benchmark, preserving broad coverage while avoiding over\-representation of any single category\. This procedure yields 233 tasks from the 22 source benchmarks\.

#### Workload Taxonomy\.

Rather than organizing results by source benchmark, we group tasks along two workload dimensions: input modality and execution form\. The modality dimensionℳ\\mathcal\{M\}distinguishes text\-only and multimodal inputs, while the execution\-form dimension𝒦\\mathcal\{K\}distinguishes single\-step, multi\-step, and multi\-step\-plus\-tools tasks\. Their Cartesian product defines six workload groups:

𝒢\\displaystyle\\mathcal\{G\}=ℳ×𝒦,\\displaystyle=\\mathcal\{M\}\\times\\mathcal\{K\},\(4\)ℳ\\displaystyle\\mathcal\{M\}=\{text,multimodal\},\\displaystyle=\\\{\\text\{text\},\\ \\text\{multimodal\}\\\},\(5\)𝒦\\displaystyle\\mathcal\{K\}=\{single\-step,multi\-step,multi\-step\+tools\}\.\\displaystyle=\\\{\\text\{single\-step\},\\ \\text\{multi\-step\},\\ \\text\{multi\-step\+tools\}\\\}\.\(6\)
Operationally, we define a deterministic mappingφ:ℬ→𝒢\\varphi:\\mathcal\{B\}\\rightarrow\\mathcal\{G\}that assigns each source benchmark to a single workload group according to its dominant modality and required execution form\. A taskiioriginating from benchmarkb⁡\(i\)∈ℬb\(i\)\\in\\mathcal\{B\}inherits the corresponding group label:

gi=φ⁡\(b⁡\(i\)\)=\(mi,ki\)∈𝒢,g\_\{i\}=\\varphi\\bigl\(b\(i\)\\bigr\)=\(m\_\{i\},k\_\{i\}\)\\in\\mathcal\{G\},\(7\)wheremim\_\{i\}denotes the input modality andkik\_\{i\}the required execution form\. LetTg=\{i:gi=g\}T\_\{g\}=\\\{i:g\_\{i\}=g\\\}denote the set of tasks assigned to groupgg\. These sets partition the benchmark, satisfying

∑g∈𝒢\|Tg\|=N,\\sum\_\{g\\in\\mathcal\{G\}\}\|T\_\{g\}\|=N,\(8\)whereN=233N=233\. The benchmark\-to\-group assignments and task counts are shown in the Supplementary Material\.

This taxonomy stratifies evaluation results along workload dimensions that are obscured by aggregate leaderboards\. For example, a model may perform well on single\-step text tasks while struggling with multi\-step tool\-mediated execution\. Reporting performance across𝒢\\mathcal\{G\}therefore reveals workload\-specific variation that would otherwise be hidden by a single aggregate score\.

### 3\.3Evaluation Protocol

#### Task Scoring\.

Because DAREBench contains heterogeneous task types, each taskτi\\tau\_\{i\}is assigned one of three scoring modes during benchmark construction:

ηi∈\{auto,judge,hybrid\},\\eta\_\{i\}\\in\\\{\\mathrm\{auto\},\\mathrm\{judge\},\\mathrm\{hybrid\}\\\},corresponding to automated, LLM\-based, and hybrid scoring\.

•Automated Scoring\.Automated scoring is used for tasks with deterministic answers or artifacts that can be verified through rule\-based procedures\. For single\-step tasks, we check both the presence of required artifacts and the consistency of their contents with the ground truth\. For multi\-step tasks, we define task\-specific scoring criteria over independently verifiable intermediate states and final outputs, weighted by their operational importance\. This produces an automated scoresm,iautos\_\{m,i\}^\{\\mathrm\{auto\}\}for modelmmon taskii\.

•LLM\-Based Scoring\.For tasks involving open\-ended outputs or multiple semantic constraints that cannot be reliably evaluated using deterministic rules, we employ an LLM\-as\-a\-Judge\. A fixed judge model receives the task promptpip\_\{i\}, a summary of the execution trajectory, the final artifacts in𝒲i∗\\mathcal\{W\}\_\{i\}^\{\*\}, and a manually specified task\-level rubric\. It returns criterion\-level scores in a constrained format, which are aggregated using predefined rubric weights to obtainsm,ijudges\_\{m,i\}^\{\\mathrm\{judge\}\}\.

•Hybrid Scoring\.Tasks containing both rule\-verifiable and semantically judged criteria combine the two scoring modes\. The weightswiauto,wijudge∈\[0,1\]w\_\{i\}^\{\\mathrm\{auto\}\},w\_\{i\}^\{\\mathrm\{judge\}\}\\in\[0,1\]are fixed during task construction and satisfywiauto\+wijudge=1w\_\{i\}^\{\\mathrm\{auto\}\}\+w\_\{i\}^\{\\mathrm\{judge\}\}=1\. The resulting task score is

sm,i=\{sm,iauto,ηi=auto,sm,ijudge,ηi=judge,wiauto​sm,iauto\+wijudge​sm,ijudge,ηi=hybrid\.s\_\{m,i\}=\\begin\{cases\}s\_\{m,i\}^\{\\mathrm\{auto\}\},&\\eta\_\{i\}=\\mathrm\{auto\},\\\\\[4\.0pt\] s\_\{m,i\}^\{\\mathrm\{judge\}\},&\\eta\_\{i\}=\\mathrm\{judge\},\\\\\[4\.0pt\] w\_\{i\}^\{\\mathrm\{auto\}\}s\_\{m,i\}^\{\\mathrm\{auto\}\}\+w\_\{i\}^\{\\mathrm\{judge\}\}s\_\{m,i\}^\{\\mathrm\{judge\}\},&\\eta\_\{i\}=\\mathrm\{hybrid\}\.\\end\{cases\}\(9\)
All task scores are normalized to\[0,1\]\[0,1\]\.

#### Evidence\-Based Audit\.

Our pilot experiments revealed cases in which the primary judge awarded positive credit that was not supported by the execution trajectory or final artifacts\. We refer to these cases as*unsupported positive judgments*\. Typical examples include nonzero scores assigned to empty outputs, executions without substantive operations, or runs missing required artifacts\.

To identify and correct such cases, we apply deterministic post\-hoc rules to all scores involving the primary judge\. Cases satisfying one or more trigger conditions are routed to a separate meta\-judge from a different model family\. The meta\-judge receives the complete task prompt, execution trajectory, final artifacts, scoring rubric, and the primary judge’s score and rationale\. It then determines whether the awarded positive score is supported by the available execution evidence\.

If the meta\-judge determines that the score is unsupported as a whole, the task score is set to zero; otherwise, the original score is retained\. The audited score is defined as

sm,iaudit=\(1−hm,i\)​sm,i,s\_\{m,i\}^\{\\mathrm\{audit\}\}=\(1\-h\_\{m,i\}\)s\_\{m,i\},\(10\)wherehm,i∈\{0,1\}h\_\{m,i\}\\in\\\{0,1\\\}indicates whether the primary score is confirmed to be unsupported\. Unflagged cases and flagged cases upheld by the meta\-judge are assignedhm,i=0h\_\{m,i\}=0\. Detailed trigger rules, meta\-judge prompts, case studies, and human–meta\-judge agreement results are provided in the appendix\.

#### Metrics and Efficiency Accounting\.

Letℐm\\mathcal\{I\}\_\{m\}denote the set of tasks applicable to modelmm, and letℐm,g=ℐm∩Tg\\mathcal\{I\}\_\{m,g\}=\\mathcal\{I\}\_\{m\}\\cap T\_\{g\}denote its applicable tasks in workload groupgg\. We report overall and group\-level audited accuracy as

Acc⁡\(m\)\\displaystyle\\mathrm\{Acc\}\(m\)=1\|ℐm\|​∑i∈ℐmsm,iaudit,\\displaystyle=\\frac\{1\}\{\|\\mathcal\{I\}\_\{m\}\|\}\\sum\_\{i\\in\\mathcal\{I\}\_\{m\}\}s\_\{m,i\}^\{\\mathrm\{audit\}\},\(11\)Accg​\(m\)\\displaystyle\\mathrm\{Acc\}\_\{g\}\(m\)=1\|ℐm,g\|​∑i∈ℐm,gsm,iaudit\.\\displaystyle=\\frac\{1\}\{\|\\mathcal\{I\}\_\{m,g\}\|\}\\sum\_\{i\\in\\mathcal\{I\}\_\{m,g\}\}s\_\{m,i\}^\{\\mathrm\{audit\}\}\.\(12\)We additionally report average token usage and estimated dollar cost per task, averaged overℐm\\mathcal\{I\}\_\{m\}\. Writingum,iku\_\{m,i\}^\{k\}for the token count of kindk∈\{in,out,cr,cw\}k\\in\\\{\\mathrm\{in\},\\mathrm\{out\},\\mathrm\{cr\},\\mathrm\{cw\}\\\}\(uncached input, output, cache\-read, and cache\-write\), the two efficiency metrics are

Token⁡\(m\)=1\|ℐm\|​∑i∈ℐm∑kum,ik,\\mathrm\{Token\}\(m\)=\\frac\{1\}\{\|\\mathcal\{I\}\_\{m\}\|\}\\sum\_\{i\\in\\mathcal\{I\}\_\{m\}\}\\sum\_\{k\}u\_\{m,i\}^\{k\},\(13\)Cost⁡\(m\)=1\|ℐm\|​∑i∈ℐm1106​∑kαmk​um,ik,\\mathrm\{Cost\}\(m\)=\\frac\{1\}\{\|\\mathcal\{I\}\_\{m\}\|\}\\sum\_\{i\\in\\mathcal\{I\}\_\{m\}\}\\frac\{1\}\{10^\{6\}\}\\sum\_\{k\}\\alpha\_\{m\}^\{k\}u\_\{m,i\}^\{k\},\(14\)whereαmk\\alpha\_\{m\}^\{k\}is the per\-million\-token price of kindkk, taken from model\-specific OpenRouter reference rates\. When a provider exposes no separate cache\-read or cache\-write rate, those tokens are billed at the input price \(αmcr\\alpha\_\{m\}^\{\\mathrm\{cr\}\}orαmcw=αmin\\alpha\_\{m\}^\{\\mathrm\{cw\}\}=\\alpha\_\{m\}^\{\\mathrm\{in\}\}\)\. For locally deployed models, any hosted\-inference price is only a third\-party reference rather than the measured cost of local vLLM serving; accuracy–cost frontier analyses are therefore based on API reference costs\. Full per\-model rates are listed in the supplementary material\.

TextMulti\-ModalModelSingleMultiMulti\+ToolsAvgSingleMultiMulti\+ToolsAvgAPI DeploymentClaude\-Opus\-4\.8\([Anthropic 2026b](https://arxiv.org/html/2609.06059#bib.bib49)\)98\.878\.165\.376\.385\.069\.261\.871\.1GPT\-5\.5\([OpenAI 2026c](https://arxiv.org/html/2609.06059#bib.bib50)\)94\.183\.659\.876\.890\.075\.042\.467\.7Claude\-Opus\-4\.6\([Anthropic 2026a](https://arxiv.org/html/2609.06059#bib.bib51)\)91\.583\.458\.375\.975\.076\.248\.165\.9Gemini\-3\.1\-Pro\([Google DeepMind 2026a](https://arxiv.org/html/2609.06059#bib.bib52)\)96\.480\.756\.174\.385\.088\.532\.767\.9GPT\-5\.4\([OpenAI 2026a](https://arxiv.org/html/2609.06059#bib.bib53)\)92\.182\.459\.675\.882\.571\.240\.363\.5MiMo\-V2\.5\([Xiaomi MiMo Team 2026d](https://arxiv.org/html/2609.06059#bib.bib54)\)94\.782\.557\.575\.580\.066\.743\.062\.1Claude\-Sonnet\-4\.6\([Anthropic 2026c](https://arxiv.org/html/2609.06059#bib.bib55)\)89\.276\.859\.972\.675\.059\.642\.357\.9Qwen3\.6\-Plus\([Qwen Team 2026d](https://arxiv.org/html/2609.06059#bib.bib56)\)95\.577\.754\.872\.175\.061\.541\.258\.2Qwen3\.5\-Plus\([Qwen Team 2026a](https://arxiv.org/html/2609.06059#bib.bib57)\)89\.577\.353\.970\.970\.069\.239\.158\.8Doubao\-Seed2\.0\-Pro\([Bytedance Seed 2026](https://arxiv.org/html/2609.06059#bib.bib58)\)95\.977\.451\.170\.875\.061\.542\.658\.7Gemini\-3\.5\-Flash\([Google DeepMind 2026b](https://arxiv.org/html/2609.06059#bib.bib59)\)94\.980\.951\.772\.785\.050\.026\.751\.7GPT\-5\.4\-mini\([OpenAI 2026b](https://arxiv.org/html/2609.06059#bib.bib60)\)77\.879\.150\.369\.270\.046\.235\.949\.3Claude\-Haiku\-4\.5\([Anthropic 2025](https://arxiv.org/html/2609.06059#bib.bib61)\)89\.176\.750\.169\.250\.546\.231\.642\.3MiMo\-V2\-Omni\([Xiaomi MiMo Team 2026a](https://arxiv.org/html/2609.06059#bib.bib62)\)70\.475\.451\.566\.775\.034\.634\.746\.0Kimi\-K2\.5\([Kimi Team 2026](https://arxiv.org/html/2609.06059#bib.bib63)\)89\.270\.045\.664\.165\.054\.438\.751\.9Doubao\-Seed2\.0\-Mini\([Bytedance Seed 2026](https://arxiv.org/html/2609.06059#bib.bib58)\)75\.778\.942\.566\.155\.051\.934\.346\.6MiMo\-V2\.5\-Pro\([Xiaomi MiMo Team 2026c](https://arxiv.org/html/2609.06059#bib.bib64)\)91\.681\.761\.276\.0————DeepSeek\-V4\-Pro\([DeepSeek\-AI 2026](https://arxiv.org/html/2609.06059#bib.bib65)\)91\.680\.954\.373\.2————MiMo\-V2\-Pro\([Xiaomi MiMo Team 2026b](https://arxiv.org/html/2609.06059#bib.bib66)\)91\.775\.458\.471\.6————MiniMax\-M2\.7\([MiniMax 2026](https://arxiv.org/html/2609.06059#bib.bib67)\)80\.977\.853\.069\.8————GLM\-5\([GLM\-5\-Team and others 2026](https://arxiv.org/html/2609.06059#bib.bib68)\)84\.975\.053\.368\.9————DeepSeek\-V3\.2\([DeepSeek\-AI 2025](https://arxiv.org/html/2609.06059#bib.bib69)\)80\.071\.146\.863\.9————GLM\-4\.7\([Z\.AI 2025](https://arxiv.org/html/2609.06059#bib.bib70)\)73\.176\.233\.661\.3————Local DeploymentQwen3\.6\-27B\([Qwen Team 2026b](https://arxiv.org/html/2609.06059#bib.bib71)\)84\.972\.137\.261\.875\.055\.838\.555\.1Qwen3\.6\-35B\-A3B\([Qwen Team 2026c](https://arxiv.org/html/2609.06059#bib.bib72)\)79\.370\.233\.959\.065\.057\.732\.751\.0Qwen3\.5\-27B\([Qwen Team 2026a](https://arxiv.org/html/2609.06059#bib.bib57)\)85\.859\.927\.051\.955\.053\.824\.543\.9Qwen3\.5\-35B\-A3B\([Qwen Team 2026a](https://arxiv.org/html/2609.06059#bib.bib57)\)66\.257\.934\.050\.860\.038\.522\.939\.0Qwen3\-VL\-8B\-Instruct\([Bai et al\. 2025a](https://arxiv.org/html/2609.06059#bib.bib73)\)35\.758\.228\.945\.540\.025\.022\.328\.3Qwen2\.5\-VL\-32B\-Instruct\([Bai et al\. 2025b](https://arxiv.org/html/2609.06059#bib.bib74)\)53\.361\.820\.146\.615\.034\.613\.021\.5Qwen3\-VL\-32B\-Instruct\([Bai et al\. 2025a](https://arxiv.org/html/2609.06059#bib.bib73)\)48\.348\.725\.240\.655\.034\.617\.434\.3Gemma\-4\-26B\-A4B\-it\([Gemma Team 2026](https://arxiv.org/html/2609.06059#bib.bib75)\)83\.040\.619\.338\.614\.055\.820\.731\.6Qwen3\.5\-9B\([Qwen Team 2026a](https://arxiv.org/html/2609.06059#bib.bib57)\)56\.432\.927\.133\.862\.528\.816\.233\.9Qwen2\.5\-VL\-7B\-Instruct\([Bai et al\. 2025b](https://arxiv.org/html/2609.06059#bib.bib74)\)28\.035\.219\.729\.010\.07\.74\.07\.0Gemma\-3\-27B\-it\([Gemma Team et al\. 2025](https://arxiv.org/html/2609.06059#bib.bib76)\)42\.429\.514\.526\.025\.013\.50\.012\.0Qwen3\-Coder\-30B\-A3B\-Instruct\([Yang et al\. 2025](https://arxiv.org/html/2609.06059#bib.bib77)\)73\.956\.337\.752\.1————

Table 1:Per\-model accuracy \(%\) across the six DAREBench task groups\. Averages are task\-count\-weighted \(Text: 20/87/55; Multi\-Modal: 20/26/25\)\.Bold,underlined, anditalicizedvalues denote first, second, and third place within each deployment mode \(competition ranking\); “—” marks text\-only models not evaluated on multimodal tasks\.Tokens/taskCost$/task \(OpenRouter rates\)TextMMTextMMModelSSMSMTAvgSSMSMTAvgSSMSMTAvgSSMSMTAvgClaude\-Opus\-4\.852k85k301k154k77k98k147k109k0\.0750\.1840\.4170\.2500\.0600\.1320\.1640\.123GPT\-5\.541k104k190k125k41k69k317k148k0\.0510\.1390\.2170\.1550\.0450\.1030\.3650\.179Claude\-Opus\-4\.635k112k605k270k49k110k251k143k0\.1750\.2450\.7840\.4190\.0620\.3620\.4060\.293Gemini\-3\.1\-Pro33k123k505k241k43k115k128k99k0\.0830\.1070\.2830\.1640\.0550\.1380\.1360\.114GPT\-5\.450k81k396k184k50k69k291k142k0\.0560\.0580\.2440\.1210\.0360\.0740\.1700\.097MiMo\-V2\.575k143k603k291k79k95k523k241k0\.0030\.0030\.0110\.0060\.0010\.0040\.0070\.005Claude\-Sonnet\-4\.649k157k575k286k52k125k330k177k0\.1540\.2050\.9020\.4360\.0630\.2900\.3370\.243Qwen3\.6\-Plus57k163k607k301k50k181k561k278k0\.0390\.0860\.3210\.1600\.0270\.1080\.2920\.150Qwen3\.5\-Plus40k80k529k228k40k111k248k140k0\.0210\.0350\.2180\.0950\.0180\.0500\.1050\.060Doubao\-Seed2\.0\-Pro46k75k452k199k164k173k289k211k0\.0190\.0230\.1050\.0510\.0310\.0560\.0790\.057Gemini\-3\.5\-Flash101k338k773k456k191k590k625k490k0\.1110\.1930\.4010\.2530\.1550\.3050\.3460\.277GPT\-5\.4\-mini31k52k131k76k38k40k95k59k0\.0080\.0100\.0330\.0170\.0070\.0120\.0220\.014Claude\-Haiku\-4\.546k75k375k173k54k108k164k113k0\.0220\.0350\.1430\.0700\.0180\.0420\.0520\.039MiMo\-V2\-Omni49k90k618k264k51k54k284k134k0\.0030\.0030\.0170\.008<0\.0010\.0030\.0050\.003Kimi\-K2\.573k184k550k295k41k226k268k189k0\.0420\.0720\.2210\.1190\.0250\.0880\.1220\.082Doubao\-Seed2\.0\-Mini55k79k460k205k318k148k384k279k0\.0080\.0090\.0490\.0230\.0340\.0180\.0410\.031MiMo\-V2\.5\-Pro72k134k441k230k————0\.0100\.0080\.0200\.012————DeepSeek\-V4\-Pro61k126k390k207k————0\.0130\.0150\.0400\.023————MiMo\-V2\-Pro42k98k520k234k————0\.0050\.0090\.0260\.014————MiniMax\-M2\.750k75k377k175k————0\.0130\.0100\.0560\.026————GLM\-548k125k449k225k————0\.0370\.0570\.1850\.098————DeepSeek\-V3\.2122k176k454k263k————0\.0160\.0190\.0460\.028————GLM\-4\.735k92k229k131k————0\.0160\.0260\.0520\.034————

Table 2:Per\-model token consumption and dollar cost per task on DAREBench\. Averages are task\-count\-weighted\. SS, MS, and MT denote Single\-Step, Multi\-Step, and Multi\-Step\+Tools, respectively\.

## 4Experiments

### 4\.1Experimental Setup

#### Models Evaluated\.

We evaluate 35 models: 23 commercial API models from 10 families accessed through OpenRouter and 12 open\-weight models served locally with vLLM on a single NVIDIA RTX PRO 6000 GPU \(96 GB VRAM\)\. To represent practical on\-premise deployment, local models use a common FP16, 32K\-context configuration—the largest stable setting shared by all models on one GPU—without quantization or multi\-GPU serving\. API–local comparisons therefore reflect end\-to\-end performance under their respective deployment profiles rather than context\-matched intrinsic capability\. All runs use the same OpenClaw runtime and default inference settings, with isolated sessions, workspaces, artifacts, and logs for each task\.

#### Infrastructure\.

All models use the default OpenClaw tool suite, supplemented withweb\_search,web\_fetch, and nine Clawhub skills covering reasoning, automation, and general utilities\. The complete skill list is provided in the supplementary material\. Tool and skill visibility, permissions, and interfaces are kept identical across models\.

#### Evaluation Metrics\.

We use accuracy as the primary capability metric, reporting both overall accuracy and accuracy for six workload groups: Text\-SS, Text\-MS, Text\-MT, MM\-SS, MM\-MS, and MM\-MT\. Efficiency is measured by average token usage and estimated USD cost per task, computed from measured input/output tokens using OpenRouter reference rates; pricing details are provided in the supplementary material\. For LLM\-based scoring, we use Qwen3\.5\-VL\-Plus as the primary judge and Claude\-Opus\-4\.6 as the meta\-judge\.

### 4\.2Main Results

Table[1](https://arxiv.org/html/2609.06059#S3.T1)reports Accuracy across the six DAREBench task groups\. Models are grouped by deployment mode \(API and Local Deployment\)\. For text\-only models, we report their performance on text\-only tasks exclusively\. We also report token consumption and USD cost for all API models in Table[2](https://arxiv.org/html/2609.06059#S3.T2)\(corresponding details for local models are provided in the Supplementary\), and visualize the API accuracy–cost trade\-off in Figure[1](https://arxiv.org/html/2609.06059#S1.F1)\.We summarize the main findings below:

•Top performers diverge across workload types, and model selection should be workload\-aware\.Overall, although stronger models tend to exhibit higher aggregate performance, relative differences persist across workloads: On Text\-SS \(98\.8\), Text\-MT \(65\.3\), and MM\-MT \(61\.8\), Claude\-Opus\-4\.8 is the strongest model; GPT\-5\.5 has the best performance on Text\-MS \(83\.6\) and MM\-SS \(90\.0\); while on MM\-MS \(88\.5\), the SOTA model is Gemini\-3\.1\-Pro\. This indicates that top performers diverge across workload types, and model selection should be guided by target workload characteristics to achieve the best performance\.

•Recent open\-weight models offer competitive on\-premise performance, but a notable gap to frontier models remains\.For workloads requiring local deployment, Qwen3\.6\-27B emerges as the strongest default \(Overall 59\.8\), with aggregate performance approaching that of some earlier commercial models\. However, it still has a substantial gap of approximately 15 percentage points relative to frontier commercial models\. Qwen3\.6\-35B\-A3B follows closely \(Overall 56\.6\), serving as a viable MoE alternative when lower activated\-parameter cost or latency is desired\. Although smaller 8B/9B local models are easy to deploy, they exhibit marked instability within the full agent loop\.

PatternLocalAPITotalDeadlock loop86086Rubric leakage121426Pseudo\-tool\-call credulity20020Output instability527Timeout\-with\-credit134Total12419143Table 3:Audit type of LLM Judge Hallucination Patterns•Token consumption and USD cost form two decoupled extremal chains\.Table[2](https://arxiv.org/html/2609.06059#S3.T2)shows token consumption and USD cost for all API models\. On the token\-consumption axis, models differ substantially in the tokens required to complete tasks: GPT\-5\.4\-mini is the lowest across the board; conversely, Gemini\-3\.5\-Flash highest on both axes \(Text Avg 456k, MM Avg 490k\), consuming roughly six and eight times as many tokens as GPT\-5\.4\-mini, respectively\. Yet when converted to USD cost, the ordering shifts dramatically: MiMo\-V2\.5 is the cheapest model on text tasks \($0\.006\), while MiMo\-V2\-Omni is the cheapest on multimodal tasks \($0\.003\); at the other extreme, Claude\-Sonnet\-4\.6 tops the text column at $0\.436 and Claude\-Opus\-4\.6 tops the multimodal column at $0\.293\. Notably, the extremes of the token\-consumption and cost rankings do not coincide: MiMo\-V2\.5 incurs high token usage \(Overall 276k\) yet yields a low Overall cost of $0\.005, whereas Claude\-Opus\-4\.6 consumes fewer tokens overall \(231k\) at roughly 71×\\timesthat cost \($0\.381\)\.

•Cost\-efficiency\-aware model selection\.Figure[1](https://arxiv.org/html/2609.06059#S1.F1)presents Pareto fronts separately for text and multimodal panels\. In the text panel, MiMo\-V2\.5 \($0\.006, 75\.5\), MiMo\-V2\.5\-Pro \($0\.012, 76\.0\), and GPT\-5\.5 \($0\.155, 76\.8\) form an optimal sequence ordered by increasing cost\. The step from MiMo\-V2\.5 to MiMo\-V2\.5\-Pro raises cost by approximately 2×\\timeswhile accuracy improves by 0\.5 percentage points, and the subsequent step to GPT\-5\.5 raises cost by a further 13×\\timesfor only 0\.8 percentage points, indicating that the sub\-$0\.02 range is already near the accuracy–cost saturation point for text tasks\. In the multimodal panel, the frontier slope is markedly steeper: after MiMo\-V2\-Omni \($0\.003, 46\.0\) and MiMo\-V2\.5 \($0\.005, 62\.1\), GPT\-5\.4 \($0\.097, 63\.5\), Gemini\-3\.1\-Pro \($0\.114, 67\.9\) and Claude\-Opus\-4\.8 \($0\.123, 71\.1\) successively raise the bar, and compared with the text panel, the cost\-to\-performance gains are notably more pronounced\.

### 4\.3LLM Judge Reliability

Table[3](https://arxiv.org/html/2609.06059#S4.T3)reports the hallucination patterns identified by the meta\-judge and the corresponding audit results\. These include rubric leakage \(the judge mistakes answer information embedded in the grading rubrics for the agent’s execution output\), deadlock loop \(the agent repeatedly cycles through materially equivalent reasoning or actions without making workspace progress, eventually exhausting the 32K context window and leaving the task incomplete, while the judge still awards positive credit\), pseudo\-tool\-call \(the model outputs tool\-like text rather than an actually executable tool call\), timeout\-with\-credit \(timed\-out trajectories still receiving positive scores\), and output instability \(incomplete or malformed output\)\. The meta\-judge audit removes 143 unsupported positive scores and reveals the dominant hallucination patterns\. We also observe a pronounced divergence in hallucination patterns by deployment mode\. Locally deployed models predominantly exhibit deadlock loops and pseudo\-tool\-call credulity \(86 and 20 cases, respectively\), while these two patterns never appear in API\-based models\. The same 32K local deployment profile is applied to all local models, yet only a subset repeatedly enters non\-progressing loops before reaching the limit\. We therefore treat these cases as model\-dependent agent failures exposed by the deployment environment, rather than as context exhaustion caused by task length alone\. Conversely, timeout\-with\-credit is more common in API invocation scenarios\. We attribute this disparity to differences in error modes: stronger trajectory control helps a model avoid non\-progressing loops, whereas API execution remains more exposed to network and serving latency\. These results characterize reliability under the specified single\-GPU deployment profile and do not assume that the same failure rate would hold under a larger\-context serving configuration\.

### 4\.4Case Study

Figure[3](https://arxiv.org/html/2609.06059#S4.F3)illustrates a representative multimodal agent failure\. The task requires filtering foods by two visual constraints before retrieving their glycemic\-index values\. Qwen3\.5\-9B misreads the chart and retains*Avocado*despite the CO2threshold; this early error changes the candidate set and leads to the wrong answer\. It also relies on unverified sources\. In contrast, Claude Opus 4\.8 applies both constraints and checks retrieved values against authoritative sources\. With identical tools and runtime, the contrast highlights differences in visual grounding, constraint tracking, and evidence verification, as well as how early errors propagate through multi\-step trajectories\. Additional examples are provided in the Supplementary Material\.

![Refer to caption](https://arxiv.org/html/2609.06059v1/Case.png)Figure 3:Multi\-Step Visual Search on a Nutrition Scatter Chart\.Example Task:Among all Fruits and Vegetables with CO2\>0\.15\>0\.15kg per 100 calories and price tier of either Highest or Medium, which single food has the lowest Glycemic Index \(GI\)?

## 5Conclusion

We introduce DAREBench, a workload\- and deployment\-aware benchmark for evaluating models as agents\. Built on a shared OpenClaw environment, it organizes 233 tasks into six workload groups and evaluates 35 API and locally deployed open\-weight models using accuracy, token usage, and reference cost\. Results show that model rankings vary across workloads, text and multimodal tasks exhibit distinct accuracy–cost trade\-offs, and local models remain competitive in several groups but trail frontier models overall\. We further identify 143 positive scores unsupported by execution evidence, highlighting the need to audit LLM\-based judgments against trajectories and final artifacts\. It also exposes deployment\-specific failures that aggregate leaderboards often obscure\. Overall, DAREBench supports workload\-specific model comparison and more reliable deployment decisions\.

## References

- Anthropic \(2025\)AnthropicClaude Haiku 4\.5 System Card\.Note:https://www\-cdn\.anthropic\.com/7aad69bf12627d42234e01ee7c36305dc2f6a970\.pdfCited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.16.1)\.
- Anthropic \(2026a\)AnthropicClaude Opus 4\.6 System Card\.Note:https://www\.anthropic\.com/claude\-opus\-4\-6\-system\-cardCited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.6.1)\.
- Anthropic \(2026b\)AnthropicClaude Opus 4\.8 System Card\.Note:https://www\.anthropic\.com/claude\-opus\-4\-8\-system\-cardCited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.4.1)\.
- Anthropic \(2026c\)AnthropicClaude Sonnet 4\.6 System Card\.Note:https://www\.anthropic\.com/claude\-sonnet\-4\-6\-system\-cardCited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.10.1)\.
- Baiet al\.\(2025a\)S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge,et al\.Qwen3\-VL technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.32.1),[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.34.1)\.
- Baiet al\.\(2025b\)S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang,et al\.Qwen2\.5\-VL technical report\.arXiv preprint arXiv:2502\.13923\.Cited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.33.1),[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.37.1)\.
- Baiet al\.\(2025c\)Y\. Bai, S\. Tu, J\. Zhang, H\. Peng, X\. Wang, X\. Lv, S\. Cao, J\. Xu, L\. Hou, Y\. Dong,et al\.Longbench v2: towards deeper understanding and reasoning on realistic long\-context multitasks\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics,Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p7.1),[§1](https://arxiv.org/html/2609.06059#S1.p2.1)\.
- Bytedance Seed \(2026\)Bytedance SeedSeed2\.0 Model Card: towards intelligence frontier for real\-world complexity\.arXiv preprint arXiv:2607\.00248\.Cited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.13.1),[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.19.1)\.
- Center for AI Safetyet al\.\(2026\)Center for AI Safety, Scale AI, and HLE Contributors ConsortiumA benchmark of expert\-level academic questions to assess AI capabilities\.Nature\.Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p18.1)\.
- Chenet al\.\(2024\)D\. Chen, R\. Chen, S\. Zhang, Y\. Wang, Y\. Liu, H\. Zhou, Q\. Zhang, Y\. Wan, P\. Zhou, and L\. SunMLLM\-as\-a\-judge: assessing multimodal LLM\-as\-a\-judge with vision\-language benchmark\.InProceedings of the 41st International Conference on Machine Learning,Cited by:[§2\.2](https://arxiv.org/html/2609.06059#S2.SS2.p1.1)\.
- Chenet al\.\(2021\)Z\. Chen, W\. Chen, C\. Smiley, S\. Shah, I\. Borova, D\. Langdon, R\. Moussa, M\. Beane, T\. Huang, B\. Routledge, and W\. Y\. WangFinQA: a dataset of numerical reasoning over financial data\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p10.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-AIDeepSeek\-V3\.2: pushing the frontier of open large language models\.arXiv preprint arXiv:2512\.02556\.Cited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.25.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-AIDeepSeek\-V4: towards highly efficient million\-token context intelligence\.arXiv preprint arXiv:2606\.19348\.Cited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.21.1)\.
- Dinget al\.\(2026\)S\. Ding, X\. Dai, L\. Xing, S\. Ding, Z\. Liu, J\. Yang, P\. Yang, Z\. Zhang, X\. Wei, X\. Fang, Y\. Ma, H\. Duan, J\. Shao, J\. Wang, D\. Lin, K\. Chen, and Y\. ZangWildClawBench\.Note:Software and dataset:https://github\.com/InternLM/WildClawBenchCited by:[§2\.1](https://arxiv.org/html/2609.06059#S2.SS1.p1.1)\.
- Duboiset al\.\(2024\)Y\. Dubois, P\. Liang, and T\. B\. HashimotoLength\-controlled AlpacaEval: a simple way to debias automatic evaluators\.InFirst Conference on Language Modeling,Cited by:[§2\.2](https://arxiv.org/html/2609.06059#S2.SS2.p1.1)\.
- Gemma Teamet al\.\(2025\)Gemma Team, A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej,et al\.Gemma 3 technical report\.arXiv preprint arXiv:2503\.19786\.Cited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.38.1)\.
- Gemma Team \(2026\)Gemma TeamGemma 4 technical report\.arXiv preprint arXiv:2607\.02770\.Cited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.35.1)\.
- GLM\-5\-Teamet al\.\(2026\)GLM\-5\-Teamet al\.GLM\-5: from vibe coding to agentic engineering\.arXiv preprint arXiv:2602\.15763\.Cited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.24.1)\.
- Google DeepMind \(2026a\)Google DeepMindGemini 3\.1 Pro Model Card\.Note:https://deepmind\.google/models/model\-cards/gemini\-3\-1\-pro/Cited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.7.1)\.
- Google DeepMind \(2026b\)Google DeepMindGemini 3\.5 Flash Model Card\.Note:https://deepmind\.google/models/model\-cards/gemini\-3\-5\-flash/Cited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.14.1)\.
- Guptaet al\.\(2026\)N\. Gupta, R\. Chatterjee, L\. Haas, C\. Tao, A\. Wang, C\. Liu, H\. Oiwa, E\. Gribovskaya, J\. Ackermann, J\. Blitzer,et al\.DeepSearchQA: bridging the comprehensiveness gap for deep research agents\.arXiv preprint arXiv:2601\.20975\.Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p11.1)\.
- Gurram \(2026\)B\. GurramEvaluating tool\-using language agents: judge reliability, propagation cascades, and runtime mitigation in agentprop\-bench\.arXiv preprint arXiv:2604\.16706\.Cited by:[Appendix C](https://arxiv.org/html/2609.06059#A3.p2.1),[§2\.2](https://arxiv.org/html/2609.06059#S2.SS2.p1.1)\.
- Heet al\.\(2025\)Y\. He, W\. Li, H\. Zhang, S\. Li, K\. Mandyam, S\. Khosla, Y\. Xiong, N\. Wang, X\. Peng, B\. Li,et al\.Advancedif: rubric\-based benchmarking and reinforcement learning for advancing llm instruction following\.arXiv preprint arXiv:2511\.10507\.Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p4.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.06059#S1.p2.1)\.
- Jiet al\.\(2026\)H\. Ji, K\. Xiong, S\. Han, P\. Xia, S\. Qiu, Y\. Zhou, J\. Liu, J\. Li, B\. Li, Z\. Zheng, C\. Xie, and H\. YaoClawArena: benchmarking AI agents in evolving information environments\.arXiv preprint arXiv:2604\.04202\.Cited by:[§2\.1](https://arxiv.org/html/2609.06059#S2.SS1.p1.1)\.
- Jianget al\.\(2025\)D\. Jiang, R\. Zhang, Z\. Guo, Y\. Wu, J\. Lei, P\. Qiu, P\. Lu, Z\. Chen, G\. Song, P\. Gao, Y\. Liu, C\. Li, and H\. LiMMSearch: unveiling the potential of large models as multi\-modal search engines\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p22.1),[§2\.1](https://arxiv.org/html/2609.06059#S2.SS1.p1.1)\.
- Jianget al\.\(2026\)H\. Jiang, Y\. Chen, Y\. Cao, H\. Lee, and R\. T\. TanCodeJudgeBench: benchmarking LLM\-as\-a\-judge for coding tasks\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Cited by:[§2\.2](https://arxiv.org/html/2609.06059#S2.SS2.p1.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. NarasimhanSWE\-bench: can language models resolve real\-world GitHub issues?\.InThe Twelfth International Conference on Learning Representations,Cited by:[§2\.2](https://arxiv.org/html/2609.06059#S2.SS2.p1.1)\.
- Kimi Team \(2026\)Kimi TeamKimi K2\.5: visual agentic intelligence\.arXiv preprint arXiv:2602\.02276\.Cited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.18.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with pagedattention\.InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles,Cited by:[§A\.1](https://arxiv.org/html/2609.06059#A1.SS1.p1.1)\.
- Liet al\.\(2026a\)C\. Li, Z\. Tang, M\. Huang, Y\. Lin, S\. Huang, S\. Liu, B\. Ye, R\. Li, L\. Li, B\. Wang, and Y\. YuanClaw\-eval\-live: a live agent benchmark for evolving real\-world workflows\.arXiv preprint arXiv:2604\.28139\.Cited by:[§1](https://arxiv.org/html/2609.06059#S1.p2.1)\.
- Liet al\.\(2025\)D\. Li, B\. Jiang, L\. Huang, A\. Beigi, C\. Zhao, Z\. Tan, A\. Bhattacharjee, Y\. Jiang, C\. Chen, T\. Wu,et al\.From generation to judgment: opportunities and challenges of LLM\-as\-a\-judge\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,Cited by:[§2\.2](https://arxiv.org/html/2609.06059#S2.SS2.p1.1)\.
- Liet al\.\(2024\)H\. Li, Y\. Chen, Q\. Ai, Y\. Wu, R\. Zhang, and Y\. LiuLexeval: a comprehensive chinese legal benchmark for evaluating large language models\.Advances in Neural Information Processing Systems\.Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p8.1)\.
- Liet al\.\(2026b\)X\. Li, K\. W\. Choe, Y\. Liu, X\. Chen, C\. Tao, B\. You, W\. Chen, Z\. Di, J\. Sun, S\. Zheng,et al\.ClawsBench: evaluating capability and safety of llm productivity agents in simulated workspaces\.arXiv preprint arXiv:2604\.05172\.Cited by:[§2\.1](https://arxiv.org/html/2609.06059#S2.SS1.p1.1)\.
- Liuet al\.\(2020\)J\. Liu, L\. Cui, H\. Liu, D\. Huang, Y\. Wang, and Y\. ZhangLogiQA: a challenge dataset for machine reading comprehension with logical reasoning\.InProceedings of the Twenty\-Ninth International Joint Conference on Artificial Intelligence, IJCAI\-20,Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p2.1)\.
- Liuet al\.\(2024\)X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang,et al\.AgentBench: evaluating LLMs as agents\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.06059#S1.p2.1)\.
- Merrillet al\.\(2026\)M\. A\. Merrill, A\. G\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi, J\. Y\. Shin, T\. Walshe, E\. K\. Buchanan,et al\.Terminal\-bench: benchmarking agents on hard, realistic tasks in command line interfaces\.arXiv preprint arXiv:2601\.11868\.Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p14.1),[§1](https://arxiv.org/html/2609.06059#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.06059#S2.SS1.p1.1)\.
- Mialonet al\.\(2024\)G\. Mialon, C\. Fourrier, C\. Swift, T\. Wolf, Y\. LeCun, and T\. ScialomGAIA: a benchmark for general AI assistants\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.06059#S1.p2.1)\.
- MiniMax \(2026\)MiniMaxThe MiniMax\-M2 series: mini activations unleashing max real\-world intelligence\.arXiv preprint arXiv:2605\.26494\.Cited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.23.1)\.
- OpenAI \(2026a\)OpenAIGPT\-5\.4 Thinking System Card\.Note:https://deploymentsafety\.openai\.com/gpt\-5\-4\-thinkingCited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.8.1)\.
- OpenAI \(2026b\)OpenAIGPT\-5\.4 Thinking System Card: appendix—GPT\-5\.4 mini\.Note:https://deploymentsafety\.openai\.com/gpt\-5\-4\-thinking/appendix\-gpt\-5\.4\-miniCited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.15.1)\.
- OpenAI \(2026c\)OpenAIGPT\-5\.5 System Card\.Note:https://openai\.com/index/gpt\-5\-5\-system\-card/Cited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.5.1)\.
- Phamet al\.\(2026\)T\. Pham, N\. Nguyen, P\. Zunjare, W\. Chen, Y\. Tseng, and T\. VuSealQA: raising the bar for reasoning in search\-augmented language models\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p13.1)\.
- Presset al\.\(2023\)O\. Press, M\. Zhang, S\. Min, L\. Schmidt, N\. A\. Smith, and M\. LewisMeasuring and narrowing the compositionality gap in language models\.InFindings of the Association for Computational Linguistics: EMNLP 2023,Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p5.1)\.
- Qwen Team \(2026a\)Qwen TeamQwen3\.5: towards native multimodal agents\.Note:https://qwen\.ai/blog?id=qwen3\.5Cited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.12.1),[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.30.1),[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.31.1),[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.36.1)\.
- Qwen Team \(2026b\)Qwen TeamQwen3\.6\-27B: flagship\-level coding in a 27b dense model\.Note:https://qwen\.ai/blog?id=qwen3\.6\-27bCited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.28.1)\.
- Qwen Team \(2026c\)Qwen TeamQwen3\.6\-35B\-A3B: agentic coding power, now open to all\.Note:https://qwen\.ai/blog?id=qwen3\.6\-35b\-a3bCited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.29.1)\.
- Qwen Team \(2026d\)Qwen TeamQwen3\.6\-Plus: towards real world agents\.Note:https://qwen\.ai/blog?id=qwen3\.6Cited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.11.1)\.
- Reinet al\.\(2024\)D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. BowmanGPQA: a graduate\-level google\-proof Q&A benchmark\.InFirst Conference on Language Modeling,Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p3.1),[§1](https://arxiv.org/html/2609.06059#S1.p2.1)\.
- Schicket al\.\(2023\)T\. Schick, J\. Dwivedi\-Yu, R\. Dessi, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. ScialomToolformer: language models can teach themselves to use tools\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2609.06059#S1.p1.1)\.
- Shaoet al\.\(2026\)M\. Shao, N\. Rani, K\. Milner, H\. Xi, M\. Udeshi, S\. Aggarwal, V\. S\. C\. Putrevu, S\. K\. Shukla, P\. Krishnamurthy, F\. Khorrami, R\. Karri, and M\. ShafiqueTowards effective offensive security LLM agents: hyperparameter tuning, LLM as a judge, and a lightweight CTF benchmark\.Proceedings of the AAAI Conference on Artificial Intelligence\.Cited by:[§2\.2](https://arxiv.org/html/2609.06059#S2.SS2.p1.1)\.
- SpaceXAI \(2026\)SpaceXAIIntroducing Grok 4\.5\.Note:https://x\.ai/news/grok\-4\-5Cited by:[§B\.2](https://arxiv.org/html/2609.06059#A2.SS2.p1.1)\.
- Steinberger and OpenClaw Contributors \(2026\)P\. Steinberger and OpenClaw ContributorsOpenClaw—personal AI assistant\.Note:https://github\.com/openclaw/openclawCited by:[§A\.1](https://arxiv.org/html/2609.06059#A1.SS1.p1.1),[§1](https://arxiv.org/html/2609.06059#S1.p4.1)\.
- Suet al\.\(2026\)Z\. Su, J\. Gao, H\. Guo, Z\. Liu, L\. Zhang, X\. Geng, S\. Huang, P\. Xia, G\. Jiang, C\. Wang,et al\.Agentvista: evaluating multimodal agents in ultra\-challenging realistic visual scenarios\.arXiv preprint arXiv:2602\.23166\.Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p21.1)\.
- Taoet al\.\(2026\)X\. Tao, Y\. Teng, X\. Su, X\. Fu, J\. Wu, C\. Tao, Z\. Liu, H\. Bai, R\. Liu, and L\. KongMMSearch\-Plus: benchmarking provenance\-aware search for multimodal browsing agents\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p23.1)\.
- Vijayvargiyaet al\.\(2026\)S\. Vijayvargiya, A\. B\. Soni, X\. Zhou, Z\. Z\. Wang, N\. Dziri, G\. Neubig, and M\. SapOpenAgentSafety: a comprehensive framework for evaluating real\-world AI agent safety\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p15.1)\.
- Wanget al\.\(2025\)Y\. Wang, Y\. Zhao, X\. Chen, S\. Guo, L\. Liu, H\. Li, Y\. Xiao, J\. Zhang, Q\. Li, and K\. XuVisualSimpleQA: a benchmark for decoupled evaluation of large vision\-language models in fact\-seeking question answering\.arXiv preprint arXiv:2503\.06492\.Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p17.1)\.
- Wanget al\.\(2024\)Z\. Wang, M\. Xia, L\. He, H\. Chen, Y\. Liu, R\. Zhu, K\. Liang, X\. Wu, H\. Liu, S\. Malladi,et al\.CharXiv: charting gaps in realistic chart understanding in multimodal llms\.arXiv preprint arXiv:2406\.18521\.Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p19.1),[§1](https://arxiv.org/html/2609.06059#S1.p2.1)\.
- Weiet al\.\(2024\)J\. Wei, K\. Nguyen, H\. W\. Chung, Y\. J\. Jiao, S\. Papay, A\. Glaese, J\. Schulman, and W\. FedusMeasuring short\-form factuality in large language models\.arXiv preprint arXiv:2411\.04368\.Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p6.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. ZhouChain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2609.06059#S1.p1.1)\.
- Wonget al\.\(2025\)R\. Wong, J\. Wang, J\. Zhao, L\. Chen, Y\. Gao, L\. Zhang, X\. Zhou, Z\. Wang, K\. Xiang, G\. Zhang,et al\.Widesearch: benchmarking agentic broad info\-seeking\.arXiv preprint arXiv:2508\.07999\.Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p12.1)\.
- Wuet al\.\(2025\)X\. Wu, J\. Yang, L\. Chai, G\. Zhang, J\. Liu, X\. Du, D\. Liang, D\. Shu, X\. Cheng, T\. Sun, T\. Li, Z\. Li, and G\. NiuTableBench: a comprehensive and complex benchmark for table question answering\.Proceedings of the AAAI Conference on Artificial Intelligence\.Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p9.1)\.
- Xiaomi MiMo Team \(2026a\)Xiaomi MiMo TeamXiaomi MiMo\-V2\-Omni: omni\-modal agentic foundation model that sees, understands and acts\.Note:https://mimo\.mi\.com/docs/en\-US/news/previous\-news/v2\-omni\-releaseCited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.17.1)\.
- Xiaomi MiMo Team \(2026b\)Xiaomi MiMo TeamXiaomi MiMo\-V2\-Pro\.Note:https://mimo\.xiaomi\.com/mimo\-v2\-proCited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.22.1)\.
- Xiaomi MiMo Team \(2026c\)Xiaomi MiMo TeamXiaomi MiMo\-V2\.5\-Pro\.Note:https://mimo\.xiaomi\.com/mimo\-v2\-5\-pro/Cited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.20.1)\.
- Xiaomi MiMo Team \(2026d\)Xiaomi MiMo TeamXiaomi MiMo\-V2\.5\.Note:https://mimo\.xiaomi\.com/mimo\-v2\-5/Cited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.9.1)\.
- Xieet al\.\(2024\)T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng, D\. Shin, F\. Lei, Y\. Liu, Y\. Xu, S\. Zhou, S\. Savarese, C\. Xiong, V\. Zhong, and T\. YuOSWorld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.InAdvances in Neural Information Processing Systems,Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p16.1),[§1](https://arxiv.org/html/2609.06059#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.06059#S2.SS1.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.39.1)\.
- Yaoet al\.\(2025\)S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhanτ\\tau\-bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.06059#S1.p2.1)\.
- Yeet al\.\(2026\)B\. Ye, R\. Li, Q\. Yang, Y\. Liu, L\. Yao, H\. Lv, Z\. Xie, C\. An, L\. Li, L\. Kong,et al\.Claw\-eval: toward trustworthy evaluation of autonomous agents\.arXiv preprint arXiv:2604\.06132\.Cited by:[§1](https://arxiv.org/html/2609.06059#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.06059#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2609.06059#S2.SS2.p1.1)\.
- Youet al\.\(2026\)R\. You, H\. Cai, C\. Zhang, Q\. Xu, M\. Liu, T\. Yu, Y\. Li, and W\. LiAgent\-as\-a\-judge\.arXiv preprint arXiv:2601\.05111\.Cited by:[Appendix C](https://arxiv.org/html/2609.06059#A3.p2.1),[§2\.2](https://arxiv.org/html/2609.06059#S2.SS2.p1.1)\.
- Yueet al\.\(2024\)X\. Yue, Y\. Ni, K\. Zhang, T\. Zheng, R\. Liu, G\. Zhang, S\. Stevens, D\. Jiang, W\. Ren, Y\. Sun,et al\.Mmmu: a massive multi\-discipline multimodal understanding and reasoning benchmark for expert agi\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,Cited by:[§1](https://arxiv.org/html/2609.06059#S1.p2.1)\.
- Z\.AI \(2025\)Z\.AIGLM\-4\.7: advancing the coding capability\.Note:https://z\.ai/blog/glm\-4\.7Cited by:[Table 1](https://arxiv.org/html/2609.06059#S3.T1.1.1.26.1)\.
- Z\.AI \(2026\)Z\.AIZClawBench: a benchmark for the OpenClaw agent scenario\.Note:https://docs\.z\.ai/guides/llm/glm\-5\-turbo\\\#zclawbench\-a\-benchmark\-for\-the\-openclaw\-agent\-scenarioCited by:[§2\.1](https://arxiv.org/html/2609.06059#S2.SS1.p1.1)\.
- Zhanget al\.\(2026\)Y\. Zhang, Y\. Wang, Y\. Zhu, P\. Du, J\. Miao, X\. Lu, Z\. Li, X\. Qu, Z\. Guo, Y\. Shen, D\. Song, H\. Zhou, T\. Zheng, X\. Wu, H\. Yu, S\. Cai, Y\. Lu, Y\. Hao, M\. Lei, L\. Chen, K\. Zou, H\. Yin, W\. Xu, D\. Jiang, P\. Nie, J\. Liu, W\. Chen, and K\. R\. AllenClawBench: can AI agents complete everyday online tasks?\.arXiv preprint arXiv:2604\.08523\.Cited by:[§2\.1](https://arxiv.org/html/2609.06059#S2.SS1.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.InAdvances in Neural Information Processing Systems,Cited by:[§2\.2](https://arxiv.org/html/2609.06059#S2.SS2.p1.1)\.
- Zhouet al\.\(2024\)S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. NeubigWebArena: a realistic web environment for building autonomous agents\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.06059#S1.p2.1)\.
- Zuoet al\.\(2025\)Y\. Zuo, S\. Qu, Y\. Li, Z\. Chen, X\. Zhu, E\. Hua, K\. Zhang, N\. Ding, and B\. ZhouMedXpertQA: benchmarking expert\-level medical reasoning and understanding\.InProceedings of the 42nd International Conference on Machine Learning,Cited by:[§F\.3](https://arxiv.org/html/2609.06059#A6.SS3.p20.1)\.

## Appendix ADetails of Experimental Setup

This section complements the experimental setup summarized in the main paper with the details required for reproduction\. We focus on inference configuration and the tool & skill suite; in addition, the models used and their per\-token prices are listed in Table[7](https://arxiv.org/html/2609.06059#A5.T7)\(§[E](https://arxiv.org/html/2609.06059#A5)\) and the judge/meta\-judge prompts are reproduced verbatim in Figure[8](https://arxiv.org/html/2609.06059#A7.F8)and Figure[9](https://arxiv.org/html/2609.06059#A7.F9)\.

### A\.1Inference Configuration

All models are evaluated under the OpenClaw\([Steinberger and OpenClaw Contributors 2026](https://arxiv.org/html/2609.06059#bib.bib36)\)defaults to avoid per\-model tuning bias\. The random seed is fixed to 42 across all decode calls\. Local models are served with vLLM\([Kwon et al\. 2023](https://arxiv.org/html/2609.06059#bib.bib1)\)on the NVIDIA RTX PRO 6000 workstation \(96 GB VRAM\) described in the main paper; no quantization or tensor parallelism is applied\. Prefix caching is enabled to amortize the recurring system prompt\. The software environment is Ubuntu 24\.04\.4 LTS, vLLM 0\.19\.0, OpenClaw 2026\.3\.20, and CUDA 13\.1\.

### A\.2Tool and Skill Suite

Beyond the default OpenClaw tool suite \(file I/O, shell execution, and agent\-loop control primitives\), every model is givenweb\_search\(powered by Brave Search\) andweb\_fetch\(URL\-to\-markdown rendering\); both tools enforce a uniform network policy across models\. We additionally integrate 9 Clawhub skills, grouped by function:

- •Self\-evolution and logical reasoning:ontology\(cross\-skill state sharing via a shared key\-value store\),self\-improving\-agent\(injects a short reflective prompt after a failed turn to encourage plan revision\), andskill\-vetter\(validates outgoing tool calls against the skill’s input schema before dispatch\)\.
- •Automation capabilities:Agent Browser\(headless browser automation via Playwright\),desktop\-control\-1\.0\.0\(GUI\-level keyboard, mouse, and window control for desktop\-automation tasks\), andgithub\(repository inspection and file retrieval through the GitHub REST API, with session\-scoped credentials never exposed to the model\)\.
- •General utilities:summarize\(bounded\-length multimodal summarization to keep long\-horizon trajectories within the context limit\),proactive\-agent\(deferred sub\-task dispatch within the agent loop\), andhumanizer\(draft\-to\-prose post\-processing, applied only when the artifact contract expects a human\-readable report\)\.

All tools and skills are loaded under identical visibility and permission boundaries across models: every model sees the same tool list, schema, and permission grants, preventing infrastructure\-side bias from confounding the capability comparison\.

## Appendix BAdditional Experiments

### B\.1Model Evolution

Figure 4:Accuracy trajectories of locally deployed open\-weight Qwen and Gemma series across the six task groups\. Thin lines denote per\-group accuracies; the bold red line and shading denote Overall\.Figure[4](https://arxiv.org/html/2609.06059#A2.F4)plots the accuracy trajectories of the Qwen and Gemma series among locally deployed open\-weight models across the six DAREBench task groups, in order to examine performance changes under generational evolution\. Overall, both Qwen and Gemma follow two relatively clear evolutionary patterns: \(1\) Under the same architecture, larger parameter scale usually performs better; for example, the Overall accuracy of Qwen3\.5\-27B exceeds that of Qwen3\.5\-9B by 15\.7 \(49\.5−\-33\.8\), while mixture\-of\-experts \(MoE\) models, despite often having more total parameters than same\-generation dense models, typically achieve slightly lower Overall scores because fewer parameters are activated\. Comparisons between the 27B dense and 35B\-A3B MoE variants in both Qwen3\.5 and Qwen3\.6 confirm this pattern \(Qwen3\.5: 49\.5 vs\. 47\.2; Qwen3\.6: 59\.8 vs\. 56\.6\)\. \(2\) Generational iteration brings steady gains overall; for example, the Overall performance of the Qwen3\-VL 8B model \(Overall 40\.3\) already surpasses that of the Qwen2\.5\-VL 32B model \(Overall 39\.0\), and Gemma\-3\-27B to Gemma\-4\-26B\-A4B likewise rises from 21\.7 to 36\.5\. Further decomposing by workload shows that generational gains on multimodal tasks are typically larger than those on text\-only tasks \(e\.g\., for Qwen3\.5\-27B→\\toQwen3\.6\-27B the average text\-only gain is \+9\.9 while the multimodal gain is \+11\.2\); along the execution\-complexity axis, the improvement on Multi\-Step\+Tools is consistently the smallest, while the gains on Single\-Step and Multi\-Step are relatively close\. This ordering is consistent with the difficulty ranking implied by our workload taxonomy: harder tasks exhibit slower generational progress\. This observation also provides indirect support for partitioning workloads by modality and execution complexity\.

### B\.2Error Pattern Analysis

To further investigate the causes of task failure, we randomly sampled 30 tasks stratified by task composition, collected all models’ failed execution trajectories on these tasks, and conducted a fine\-grained analysis of each model’s failure patterns\. Through manual review, we found that the primary failure modes can be categorized into the six patterns shown in Table[4](https://arxiv.org/html/2609.06059#A2.T4)\. We first prompted Grok 4\.5\([SpaceXAI 2026](https://arxiv.org/html/2609.06059#bib.bib78)\)to read each task description together with the corresponding model execution traces, classify each failure into one of the six error modes \(assigning cases outside these categories to Other\), and provide a rationale\. To ensure accuracy, we further manually audited the classifications; the final results are reported in Table[5](https://arxiv.org/html/2609.06059#A2.T5)\.

Among identifiable failures, PPPA accounts for the largest share \(88/308, 28\.6%\), and is substantially more frequent for local models than for API models \(62 vs\. 26; 38\.5% vs\. 17\.7% of their respective failure pools\)\. This gap indicates that long\-horizon planning and contract adherence—finishing within the time limit and writing the required artifact—remain the primary bottleneck, especially for weaker open\-weight agents whose tool\-calling or file\-writing protocols are also more prone to interruption\. SRU and VEB are the next most common failure modes, with SRU manifesting as constraint violations or missed aggregations, and VEB as visual evidence bypass\. On VEB errors, the API and local counts are nearly identical \(18 vs\. 17\), suggesting that models preferentially rely on web retrieval or prior guesses rather than reading and retrieving information from local images\.

IDDefinitionSRUSemantic Reasoning & Understanding: violates constraints, misses aggregation, or answers off\-question\.TUIETool Use & Information Extraction: insufficient web search/fetch retrieval capability\.PPPAPath Planning & Protocol Adherence: fails to finish within the time limit or to write the required contract artifact \(e\.g\.,answer\.txt\)\.FCRFine\-grained Chart Reading: inspects the figure but misreads legends, curve names, markers, or numeric values\.VEBVisual Evidence Bypass: the task depends on an image, yet the model barely looks at it and guesses via web search or priors\.VCSVisual Common Sense & Entity Recognition: misidentifies the target object, attribute, or person in the image\.OTHOther: residual error types that cause task failure\.Table 4:Capability\-oriented failure taxonomy\.ModelSampledSRUTUIEPPPAFCRVEBVCSOTHAPI DeploymentGemini\-3\.1\-Pro21000001Claude\-Opus\-4\.831000101GPT\-5\.541010002Claude\-Opus\-4\.630000102GPT\-5\.451010102MiMo\-V2\.541010101Claude\-Sonnet\-4\.674010011Qwen3\.6\-Plus61001103Qwen3\.5\-Plus42010100Doubao\-Seed2\.0\-Pro83001103Gemini\-3\.5\-Flash52000003GPT\-5\.4\-mini61010112MiMo\-V2\.5\-Pro72111110DeepSeek\-V4\-Pro91022103Claude\-Haiku\-4\.562011110Doubao\-Seed2\.0\-Mini90030114MiMo\-V2\-Omni91130121Kimi\-K2\.551011011MiMo\-V2\-Pro83111200MiniMax\-M2\.7102032111GLM\-582012111DeepSeek\-V3\.2102022013GLM\-4\.792021103Local DeploymentQwen3\.6\-27B91051002Qwen3\.6\-35B\-A3B91051110Qwen3\.5\-27B132041123Qwen3\.5\-35B\-A3B113030221Qwen3\-Coder\-30B\-A3B145060003Qwen3\-VL\-32B111072001Qwen3\-VL\-8B112132120Gemma\-4\-26B\-A4B151051008Qwen2\.5\-VL\-32B131340203Qwen3\.5\-9B163050125Gemma\-3\-27B190091225Qwen2\.5\-VL\-7B201360721Table 5:Fine\-grained failure distribution over zero\-score trajectories from 35 models on 30 stratified tasks\. Models are ordered by overall accuracy on the full benchmark \(descending\)\.
### B\.3Local Model Token Usage and Cost

Table[6](https://arxiv.org/html/2609.06059#A2.T6)reports per\-task token consumption and estimated USD cost for the 12 locally deployed open\-weight models, where Cost$/task is computed from each model’s OpenRouter hosted\-reference rate\. For local models, token usage more directly reflects compute and inference\-time cost, while the estimated dollar figures serve as a complementary economic reference for API\-style hosting of open\-weight models\. Relative to the API pool in the main paper, locally served models complete tasks with substantially fewer tokens, especially on Multi\-Step\+Tools buckets: the strongest local model, Qwen3\.6\-27B, averages about 62k \(Text\) and 81k \(MM\) tokens/task, whereas many commercial API models exceed 200k under the same aggregation\. Importantly, this does*not*imply higher reasoning efficiency: as reported in Section[B\.2](https://arxiv.org/html/2609.06059#A2.SS2), the apparent “low spend” often reflects PPPA\-style aborted or truncated trajectories rather than Pareto\-efficient agent performance\.

Tokens/taskCost$/task \(OpenRouter rates\)TextMMTextMMModelSSMSMTAvgSSMSMTAvgSSMSMTAvgSSMSMTAvgCtxLocal DeploymentQwen3\.6\-27B47k52k83k62k58k54k128k81k0\.0370\.0340\.0560\.0410\.0370\.0410\.0840\.055256KQwen3\.6\-35B\-A3B53k57k86k66k57k68k114k81k0\.0100\.0090\.0140\.0110\.0090\.0120\.0180\.013256KQwen3\.5\-27B39k64k85k68k49k53k126k78k0\.0160\.0210\.0280\.0230\.0160\.0200\.0410\.026256KQwen3\.5\-35B\-A3B35k45k78k55k61k62k105k77k0\.0110\.0120\.0210\.0150\.0160\.0170\.0270\.021262KQwen3\-VL\-8B43k42k152k79k48k42k78k57k0\.0090\.0080\.0280\.0150\.0090\.0080\.0140\.010256KQwen2\.5\-VL\-32B24k31k45k35k32k25k8k21k0\.0050\.0060\.0090\.0070\.0070\.0050\.0020\.004128KQwen3\-VL\-32B114k98k184k129k61k188k104k123k0\.0180\.0160\.0300\.0210\.0100\.0300\.0170\.020256KGemma\-4\-26B\-A4B111k53k54k60k46k105k202k123k0\.0080\.0040\.0040\.0040\.0040\.0080\.0150\.009128KQwen3\.5\-9B50k60k110k75k61k59k109k77k0\.0050\.0060\.0110\.0080\.0060\.0060\.0110\.008256KQwen2\.5\-VL\-7B29k31k125k62k35k164k108k108k0\.0060\.0060\.0250\.0120\.0070\.0330\.0220\.022128KGemma\-3\-27B43k36k50k42k59k34k9k32k0\.0030\.0030\.0040\.0030\.0050\.003<0\.0010\.003128KQwen3\-Coder\-30B\-A3B65k138k182k144k————0\.0060\.0070\.0110\.008————160K

Table 6:Per\-model token consumption and dollar cost per task for locally deployed open\-weight models on DAREBench\. Notation and ranking follow the main paper\.

## Appendix CDetails of Evidence\-Based Audit

This section supplements the Evidence\-Based Audit described in the main paper \(§Evaluation Protocol\)\.

LLM\-as\-a\-judge provides the semantic evaluation capacity necessary for open\-ended tasks; however, even with manually designed task\-specific rubrics, our pilot experiments revealed systematic risks of*judging hallucination*in LLM judges, which has also been reported in related work\([You et al\. 2026](https://arxiv.org/html/2609.06059#bib.bib47);[Gurram 2026](https://arxiv.org/html/2609.06059#bib.bib48)\)\. Through careful sampling\-based verification, we systematically identify five major recurring hallucination patterns in DAREBench:

1. 1\.rubric leakage: the judge mistakes answer information embedded in the grading rubrics for the agent’s execution output\.
2. 2\.deadlock loop: the agent repeatedly cycles through materially equivalent reasoning or actions without making workspace progress, eventually exhausting the 32K context window and leaving the task incomplete, while the judge still awards positive credit\.
3. 3\.pseudo\-tool\-call credulity: the model outputs tool\-like text rather than an actually executable tool call\.
4. 4\.output instability: incomplete or malformed output\.
5. 5\.timeout\-with\-credit: timed\-out trajectories still receiving positive scores\.

Figures[5](https://arxiv.org/html/2609.06059#A3.F5)and[6](https://arxiv.org/html/2609.06059#A3.F6)further visualize two real cases that expose these hallucination patterns\. To address these patterns, we designed a set of deterministic post\-hoc rules that perform high\-recall pre\-filtering of potential hallucination cases, followed by a second\-line adjudication by an independent meta\-judge, thereby enabling cost\-controllable evaluation and screening\. The complete meta\-judge pipeline is given in Algorithm[1](https://arxiv.org/html/2609.06059#algorithm1)\.

Algorithm 1Evidence\-Based AuditInput:Primary scores

\{sm,i\}\\\{s\_\{m,i\}\\\}\(

sm,i\>0s\_\{m,i\}\>0\); trajectories

\{traji\}\\\{\\mathrm\{traj\}\_\{i\}\\\}; prompts

\{pi\}\\\{p\_\{i\}\\\}; artifacts

\{𝒲i∗\}\\\{\\mathcal\{W\}^\{\*\}\_\{i\}\\\}; rubrics

\{rubrici\}\\\{\\mathrm\{rubric\}\_\{i\}\\\}; rules

ℛ\\mathcal\{R\}
Output:Audited scores

\{sm,iaudit\}\\\{s\_\{m,i\}^\{\\mathrm\{audit\}\}\\\}
foreach*judge/hybrid callii*do

ϕi←\{\\phi\_\{i\}\\leftarrow\\\{
AsstChars​\(traji\),\\quad\\textsc\{AsstChars\}\(\\mathrm\{traj\}\_\{i\}\),
HasWrite​\(traji\),\\quad\\textsc\{HasWrite\}\(\\mathrm\{traj\}\_\{i\}\),
HasTool​\(traji\),\\quad\\textsc\{HasTool\}\(\\mathrm\{traj\}\_\{i\}\),
TokTotal​\(traji\)\\quad\\textsc\{TokTotal\}\(\\mathrm\{traj\}\_\{i\}\)
\}\\\};

/\*flag positive scores with weak execution evidence \*/

if*PreFilter​\(ϕi,sm,i,ℛ\)=True\\textsc\{PreFilter\}\(\\phi\_\{i\},s\_\{m,i\},\\mathcal\{R\}\)=\\mathrm\{True\}*then

vi←MetaJudge​\(pi,traji,𝒲i∗,rubrici,sm,i\)v\_\{i\}\\leftarrow\\textsc\{MetaJudge\}\(p\_\{i\},\\mathrm\{traj\}\_\{i\},\\mathcal\{W\}^\{\*\}\_\{i\},\\mathrm\{rubric\}\_\{i\},s\_\{m,i\}\);

if*vi=hallucinationv\_\{i\}=\\mathrm\{hallucination\}*then

sm,iaudit←0s\_\{m,i\}^\{\\mathrm\{audit\}\}\\leftarrow 0;

else

sm,iaudit←sm,is\_\{m,i\}^\{\\mathrm\{audit\}\}\\leftarrow s\_\{m,i\};

else

sm,iaudit←sm,is\_\{m,i\}^\{\\mathrm\{audit\}\}\\leftarrow s\_\{m,i\};

return

\{sm,iaudit\}\\\{s\_\{m,i\}^\{\\mathrm\{audit\}\}\\\};

![Refer to caption](https://arxiv.org/html/2609.06059v1/casestudy_pseudo_tool_call.png)Figure 5:Judge hallucination example A \(*pseudo\-tool\-call credulity*\)\.Task: What disease did the author of the book in the image die from? Write your answer to ‘answer\.txt‘ in the workspace\.The model embeds a factually correct answer inside a tool\-likewrite\(\.\.\.\)string but never executes a real file write; the LLM judge overlooks the missing workspace artifact and still awards full credit\.Figure 6:Judge hallucination example B \(*rubric leakage*\)\.Task: Read the table\. Can you identify any locations within the table whose area or population values significantly deviate from the patterns observed in other locations?The model fails and answers “No anomalies are detected,” yet the LLM judge mistakes the reference answer embedded in the grading rubric for the agent’s response and incorrectly scores the trajectory as correct\.Among the 7,587 task–model evaluations, 3,340 are judge/hybrid calls, from which the pre\-filter setℛ\\mathcal\{R\}recalled 182 potential hallucination cases\. As reported in the main paper \(§LLM Judge Reliability\), after a second\-line review by Claude\-Opus\-4\.6, 143 were finally confirmed as judging hallucinations\.

To ensure the reliability of the meta\-judge method and its experimental results, we further invited human experts to validate the meta\-judge outcomes\. We stratified\-sampled 100 execution trajectories from the 3,340 judge/hybrid calls along original score, task group, and deployment mode, and had human experts score them independently\. We report the mean absolute score error \(MAE\):

MAE=1n​∑i=1n\|sijudge−siexpert\|,n=100\.\\mathrm\{MAE\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\bigl\|\\,s\_\{i\}^\{\\text\{judge\}\}\-s\_\{i\}^\{\\text\{expert\}\}\\,\\bigr\|,\\qquad n=100\.\(15\)Heresijudges\_\{i\}^\{\\text\{judge\}\}denotes the score assigned by either the primary judge or the meta\-judge to theii\-th trajectory, andsiexperts\_\{i\}^\{\\text\{expert\}\}denotes the experts’ score\. The results show that the MAE between the meta\-judge\-corrected scores and the human scores is 0\.049, approximately half that of the primary judge \(0\.098\), indicating that the meta\-judge correction is not only more accurate in its verdicts but also closer to the experts in the assigned scores\.

## Appendix DCase Study

As additional examples supplementing the main paper’s case study, this section further illustrates how workload\-level gaps manifest in concrete agent trajectories\. Cases 1 and 2 compare a successful API trajectory against a failed local trajectory, revealing the primary performance differences between strong and weak models, and illustrating cross\-step state loss in a tool\-mediated text task and visual\-symbolic grounding failure in a compact multimodal task\. Cases 3–5 focus on hard multimodal tasks that expose common verification failures: chart alignment, chemistry formula bookkeeping, and plate\-map option grounding\.

### D\.1Case 1 — A Multi\-Hop Academic Author Retrieval Workflow

Case 1: Multi\-Hop Author Retrieval \(Text Multi\+Tools\)Task\.Of the authors in the first article of the 65th edition of the Journal of Artificial Intelligence Research, who was the most cited on Google Scholar in 2019?Gemini 3\.1 Pro\(Correct, 1\.0\)[⬇](data:text/plain;base64,KCpAXHRleHRiZntUMTp9QCopIHdlYl9zZWFyY2goIkpvdXJuYWwgb2YgQXJ0aWZpY2lhbCBJbnRlbGxpZ2VuY2UgUmVzZWFyY2ggVm9sdW1lIDY1IikKKCpAXHJldHsgLT4gSkFJUiBWb2wgNjUgPSAyMDE5IChmcm9tIGphaXIub3JnIHJlc3VsdCl9QCopCigqQFxyZXR7IC0+IGRibHAgbGluayBzcG90dGVkIChmcm9tIHJlc3VsdCBcIzQpfUAqKQooKkBcdGV4dGJme1QyOn1AKikgd2ViX2ZldGNoKCJodHRwczovL2RibHAub3JnL2RiL2pvdXJuYWxzL2phaXIvamFpcjY1Lmh0bWwiKQooKkBccmV0eyAtPiBGdWxsIFRPQyBzY3JhcGVkOyAxc3QgYXJ0aWNsZTogIk9uIE92ZXJmaXR0aW5nLi4uIiAocHAuIDEtMzApfUAqKQooKkBccmV0eyAtPiBBdXRob3JzOiBWLiBGcmFuY29pcy1MYXZldCwgRy4gUmFidXNzZWF1LCBKLiBQaW5lYXUsIEQuIEVybnN0LCBSLiBGb250ZW5lYXV9QCopCigqQFxnb29ke0NvbmZpcm1lZCBjb3JyZWN0IGFydGljbGUgYW5kIGV4dHJhY3RlZCA1IHRhcmdldCBhdXRob3JzIHN1Y2Nlc3NmdWxseX1AKikKKCpAXHRleHRiZntUMzp9QCopIHdlYl9mZXRjaCgiaHR0cHM6Ly9kb2kub3JnLzEwLjE2MTMvamFpci4xLjExNDc4IikKKCpAXHJldHsgLT4gRmFpbGVkICg0MDYgTm90IEFjY2VwdGFibGUpfUAqKSAoKkBcc3RlcHB7ZmFsbGJhY2sgdW5uZWNlc3NhcnksIFRPQyBzdWZmaWNpZW50fUAqKQooKkBcdGV4dGJme1Q0On1AKikgd2ViX3NlYXJjaCgiVmluY2VudCBGcmFuY29pcy1MYXZldCAuLi4gY2l0YXRpb25zIikKKCpAXHJldHsgLT4gQ29uZmlybWVkIDUgYXV0aG9ycyBvbiBhclhpdi9KQUlSL1Jlc2VhcmNoR2F0ZX1AKikKKCpAXHRleHRiZntUNS1UMTQ6fUAqKSB3ZWJfc2VhcmNoIHgxMCAocGVyIGF1dGhvciArIHJlY29uZmlybWF0aW9ucykKKCpAXHJldHsgLT4gSi4gUGluZWF1OiA0MSw2OTUgIFtoaWdoZXN0XX1AKikgKCpAXGdvb2R7Y2xlYXIgd2lubmVyIGlkZW50aWZpZWR9QCopCigqQFxyZXR7IC0+IEQuIEVybnN0OiAyNSwwMzk7ICBSLiBGb250ZW5lYXU6IDEsODU5OyAgRy4gUmFidXNzZWF1OiAxLDQzOX1AKikKKCpAXHJldHsgLT4gVi4gRnJhbmNvaXMtTGF2ZXQ6IG5vIFNjaG9sYXIgcHJvZmlsZX1AKikKKCpAXHRleHRiZntUMTU6fUAqKSB3cml0ZV9vdXRwdXQoIkpvZWxsZSBQaW5lYXUiKQooKkBcdGV4dGJme1QxNjp9QCopIHJlYWRfb3V0cHV0KCkKKCpAXHJldHsgLT4gVmVyaWZpZWQ6ICdKb2VsbGUgUGluZWF1J31AKikgKCpAXGdvb2R7U2VsZi1jaGVjayBwYXNzZWR9QCop)T1:web\_search\("JournalofArtificialIntelligenceResearchVolume65"\)\-\> JAIR Vol 65 = 2019 \(from jair\.org result\)\-\> dblp link spotted \(from result \#4\)T2:web\_fetch\("https://dblp\.org/db/journals/jair/jair65\.html"\)\-\> Full TOC scraped; 1st article: "On Overfitting\.\.\." \(pp\. 1\-30\)\-\> Authors: V\. Francois\-Lavet, G\. Rabusseau, J\. Pineau, D\. Ernst, R\. FonteneauConfirmed correct article and extracted 5 target authors successfullyT3:web\_fetch\("https://doi\.org/10\.1613/jair\.1\.11478"\)\-\> Failed \(406 Not Acceptable\)fallback unnecessary, TOC sufficientT4:web\_search\("VincentFrancois\-Lavet\.\.\.citations"\)\-\> Confirmed 5 authors on arXiv/JAIR/ResearchGateT5\-T14:web\_searchx10\(perauthor\+reconfirmations\)\-\> J\. Pineau: 41,695 \[highest\]clear winner identified\-\> D\. Ernst: 25,039; R\. Fonteneau: 1,859; G\. Rabusseau: 1,439\-\> V\. Francois\-Lavet: no Scholar profileT15:write\_output\("JoellePineau"\)T16:read\_output\(\)\-\> Verified: ’Joelle Pineau’Self\-check passedQwen3\-VL\-32B\(Incorrect, 0\.0\)[⬇](data:text/plain;base64,KCpAXHRleHRiZntUMS0zOn1AKikgd2ViX3NlYXJjaCB4MyAoSkFJUiA2NXRoIGVkaXRpb24gZmlyc3QgYXJ0aWNsZSBhdXRob3JzKQooKkBccmV0eyAtPiBKQUlSIGhvbWVwYWdlLCBXaWtpcGVkaWEsIGdlbmVyaWMgcHVibGlzaGVyc31AKikKKCpAXHJldHsgLT4gIk9wdHN0cmVhbS4uLiIgKHBwLiA0MjMtNDU2KSBzcG90dGVkOyBBdXRob3JzOiBGLiBGaW9yZXR0bywgUC4gVmFuIEhlbnRlbnJ5Y2t9QCopCigqQFxiYWR7TWlzdGFrZW5seSBpZGVudGlmaWVkIHRoZSBsYXN0IGFydGljbGUgYXMgdGhlIGZpcnN0fUAqKQooKkBcdGV4dGJme1Q0On1AKikgd2ViX2ZldGNoKCJodHRwczovL3d3dy5qYWlyLm9yZy9pbmRleC5waHAvamFpci9pc3N1ZS92aWV3LzExNTMiKQooKkBccmV0eyAtPiBGdWxsIFRPQyBzY3JhcGVkOyAxc3QgYXJ0aWNsZTogIk9uIE92ZXJmaXR0aW5nLi4uIiAocHAuIDEtMzApfUAqKQooKkBccmV0eyAtPiBBdXRob3JzOiBWLiBGcmFuY29pcy1MYXZldCwgRy4gUmFidXNzZWF1LCBKLiBQaW5lYXUsIEQuIEVybnN0LCBSLiBGb250ZW5lYXV9QCopCigqQFxnb29ke0NvcnJlY3QgZmlyc3QtYXJ0aWNsZSBpbmZvIHJldHJpZXZlZCwgYnV0IHByaW9yIGVycm9yIG5vdCBvdmVyd3JpdHRlbn1AKikKKCpAXHRleHRiZntUNTp9QCopIHdlYl9zZWFyY2goIkYuIEZpb3JldHRvIFAuIFZhbiBIZW50ZW5yeWNrIEdvb2dsZSBTY2hvbGFyIGNpdGF0aW9ucyAyMDE5IikKKCpAXHJldHsgLT4gUC4gVmFuIEhlbnRlbnJ5Y2s6IDMxLDk4MyBjaXRhdGlvbnM7ICBGLiBGaW9yZXR0bzogMyw1NDggY2l0YXRpb25zfUAqKQooKkBcYmFke01pc2xlZCBieSBlYXJseSBlcnJvbmVvdXMgc2VhcmNoOiBxdWVyaWVkIGF1dGhvcnMgb2YgdGhlIFdST05HIGFydGljbGUgKFQxLTMpIGluc3RlYWQgb2YgdGFyZ2V0IGFydGljbGUgYXV0aG9ycyAoVDQpfUAqKQooKkBcdGV4dGJme1Q2On1AKikgd3JpdGVfb3V0cHV0KCJQYXNjYWwgVmFuIEhlbnRlbnJ5Y2siKQ==)T1\-3:web\_searchx3\(JAIR65theditionfirstarticleauthors\)\-\> JAIR homepage, Wikipedia, generic publishers\-\> "Optstream\.\.\." \(pp\. 423\-456\) spotted; Authors: F\. Fioretto, P\. Van HentenryckMistakenly identified the last article as the firstT4:web\_fetch\("https://www\.jair\.org/index\.php/jair/issue/view/1153"\)\-\> Full TOC scraped; 1st article: "On Overfitting\.\.\." \(pp\. 1\-30\)\-\> Authors: V\. Francois\-Lavet, G\. Rabusseau, J\. Pineau, D\. Ernst, R\. FonteneauCorrect first\-article info retrieved, but prior error not overwrittenT5:web\_search\("F\.FiorettoP\.VanHentenryckGoogleScholarcitations2019"\)\-\> P\. Van Hentenryck: 31,983 citations; F\. Fioretto: 3,548 citationsMisled by early erroneous search: queried authors of the WRONG article \(T1\-3\) instead of target article authors \(T4\)T6:write\_output\("PascalVanHentenryck"\)

### D\.2Case 2 — Academic Diagram Understanding

Case 2: Academic Diagram Understanding \(Multi\-Modal Single Step\)Task\.What is the name of the line that is the furthest away from its fi value from the W\-H axis?![[Uncaptioned image]](https://arxiv.org/html/2609.06059v1/figures/charxiv_01.png)Gemini 3\.1 Pro\(Correct, 1\.0\)[⬇](data:text/plain;base64,KCpAXHRleHRiZntSZWFzb25pbmc6fUAqKSBFeGFtaW5pbmcgR3JhcGggRGF0YTogemVyb2luZyBpbiBvbiB0aGUgc3BlY2lmaWMgbGluZSBmdXJ0aGVzdCBmcm9tCnRoZSBXLUggYXhpcy4gVGhlIGxlZ2VuZCBwcm92aWRlcyBsYW1iZGEgdmFsdWVzOiAwLjEzLCAwLjIwLCAwLjMwLgooKkBcZ29vZHtWaXN1YWxseSBhc3Nlc3Mgd2hpY2ggKGdyZWVuIGRhc2hlZCwgcmVkIGRvdHRlZCwgYmx1ZSBkYXNoLWRvdCkgaXMgbW9zdCBkaXN0YW50fUAqKQooKkBcdGV4dGJme1JlYXNvbmluZzp9QCopIFBpbnBvaW50aW5nIFRoZSBBbnN3ZXI6IGNhbGN1bGF0aW5nIGRpc3RhbmNlcyBmb3IgZWFjaCBsaW5lIHVzaW5nIHRoZQonZmknIHBvaW50cyBhcyByZWZlcmVuY2UgYWxvbmcgdGhlIFctSCBheGlzLgooKkBccmV0eyAtPiBncmVlbiBsaW5lICgkXGxhbWJkYV9MID0gMC4xMyQpIGFwcGVhcnMgbW9zdCBkaXN0YW50IGZyb20gaXRzICdmaScgdmFsdWV9QCopCigqQFx0ZXh0YmZ7QW5zd2VyOn1AKikgJFxsYW1iZGFfTCA9IDAuMTMk)Reasoning:ExaminingGraphData:zeroinginonthespecificlinefurthestfromtheW\-Haxis\.Thelegendprovideslambdavalues:0\.13,0\.20,0\.30\.Visually assess which \(green dashed, red dotted, blue dash\-dot\) is most distantReasoning:PinpointingTheAnswer:calculatingdistancesforeachlineusingthe’fi’pointsasreferencealongtheW\-Haxis\.\-\> green line \(λL=0\.13\\lambda\_\{L\}=0\.13\) appears most distant from its ’fi’ valueAnswer:$\\lambda\_L=0\.13$Qwen3\-VL\-32B\(Incorrect, 0\.0\)[⬇](data:text/plain;base64,KCpAXHRleHRiZntSZWFzb25pbmc6fUAqKSBUaGUgbmFtZSBvZiB0aGUgbGluZSB0aGF0IGlzIHRoZSBmdXJ0aGVzdCBhd2F5IGZyb20gaXRzIGZpIHZhbHVlCmZyb20gdGhlIFctSCBheGlzIGlzICJmaSIuCigqQFxyZXR7IC0+IGRhdGEgcG9pbnRzIG1hcmtlZCAneCcgKHJlcHJlc2VudGluZyAiZmkiKSBhdCBXLUggdmFsdWVzIG9mIGFwcHJveC4gLTAuMiBhbmQgMC4xMn1AKikKKCpAXGJhZHtDb25mdXNlcyBhbiBheGlzL21hcmtlciBsYWJlbCB3aXRoIHRoZSByZXF1ZXN0ZWQgY3VydmUgaWRlbnRpdHl9QCopCigqQFx0ZXh0YmZ7QW5zd2VyOn1AKikgZmk=)Reasoning:ThenameofthelinethatisthefurthestawayfromitsfivaluefromtheW\-Haxisis"fi"\.\-\> data points marked ’x’ \(representing "fi"\) at W\-H values of approx\. \-0\.2 and 0\.12Confuses an axis/marker label with the requested curve identityAnswer:fi

### D\.3Case 3 — Shared Chart\-Grounding Failure

Case 3: A Chart Item That All Models MissTask\.“Which model shows a greater decline in accuracy from Session 1 to Session 9 in the 5\-way full\-shot scenario?” The chart shows continual\-learning curves for Ft\-CNN, iCaRL, EEIL, NCM, Ours\-AL, Ours\-AL\-MML, and Joint\-CNN\. Reference answer:Joint\-CNN\.![[Uncaptioned image]](https://arxiv.org/html/2609.06059v1/figures/charxiv_00_common_failure.jpg)Gemini 3\.1 Pro\(Incorrect, 0\.0\)[⬇](data:text/plain;base64,KCpAXHRleHRiZntUMTp9QCopIHJlYWRfaW1hZ2UoImNoYXJ0LnBuZyIpCigqQFxyZXR7IC0+IEltYWdlIGxvYWRlZCwgNyBjdXJ2ZXMgdmlzaWJsZX1AKikKKCpAXHN0ZXBwe1JlYXNvbmluZzogc2Nhbm5pbmcgZm9yIHRoZSBzdGVlcGVzdCB2aXNpYmxlIGRlY2xpbmUufUAqKQooKkBcdGV4dGJme1QyOn1AKikgd3JpdGVfb3V0cHV0KCJGdC1DTk4iKQooKkBcYmFke1NlbGVjdGVkIGEgcGxhdXNpYmxlIHZpc3VhbCBkcm9wLCBidXQgZGlkIG5vdCBtYXRjaCB0aGUgcmVmZXJlbmNlIGN1cnZlIGNvbXBhcmlzb24gZnJvbSBTZXNzaW9uIDEgdG8gU2Vzc2lvbiA5Ln1AKik=)T1:read\_image\("chart\.png"\)\-\> Image loaded, 7 curves visibleReasoning: scanning for the steepest visible decline\.T2:write\_output\("Ft\-CNN"\)Selected a plausible visual drop, but did not match the reference curve comparison from Session 1 to Session 9\.Qwen3\-VL\-32B\(Incorrect, 0\.0\)[⬇](data:text/plain;base64,KCpAXHRleHRiZntUMTp9QCopIHJlYWRfaW1hZ2UoImNoYXJ0LnBuZyIpCigqQFxyZXR7IC0+IEltYWdlIGxvYWRlZDsgbGVnZW5kIGFuZCBjdXJ2ZXMgdmlzaWJsZX1AKikKKCpAXHN0ZXBwe1JlYXNvbmluZzogY29tcGFyZXMgYXBwYXJlbnQgZHJvcHMgYW1vbmcgdGhlIGN1cnZlIGxhYmVscy59QCopCigqQFx0ZXh0YmZ7VDI6fUAqKSB3cml0ZV9vdXRwdXQoIkZ0LUNOTiIpCigqQFxiYWR7Q29udmVyZ2VzIHRvIHRoZSBzYW1lIHdyb25nIGxlZ2VuZCBlbnRyeSBhcyBzZXZlcmFsIGZyb250aWVyIEFQSXMufUAqKQ==)T1:read\_image\("chart\.png"\)\-\> Image loaded; legend and curves visibleReasoning: compares apparent drops among the curve labels\.T2:write\_output\("Ft\-CNN"\)Converges to the same wrong legend entry as several frontier APIs\.

### D\.4Case 4 — Image\-Grounded Chemistry Formula Inference

Case 4: A Chemistry Item Where Formula Bookkeeping FailsTask\.A Humanity’s Last Exam organic\-chemistry item asks for the molecular formula of compoundB\. The prompt describes reactions of tris\(2,6\-dimethoxyphenyl\)methylium ion with either*n*\-propanol or methyl\-3\-aminopropionate, and the diagram specifies the product structure\. Reference answer:C23H22NO4\+\.![[Uncaptioned image]](https://arxiv.org/html/2609.06059v1/figures/case4_hle_chemistry.jpg)MiMo\-V2\.5\(Incorrect, 0\.0\)[⬇](data:text/plain;base64,KCpAXHRleHRiZntUMTp9QCopIHJlYWRfaW1hZ2UoImhsZV8wN19pbWFnZS5qcGciKQooKkBccmV0eyAtPiBpbWFnZSBwYXJzZWQ7IGlkZW50aWZpZXMgYWNyaWRpbml1bSBjb3JlIGFuZCBzdWJzdGl0dWVudHN9QCopCigqQFxzdGVwcHtSZWFzb25pbmc6IGRlcml2ZXMgcHJvZHVjdCBCIGZyb20gdGhlIHJpZ2h0LWhhbmQgcGF0aHdheS59QCopCigqQFx0ZXh0YmZ7VDI6fUAqKSB3cml0ZV9vdXRwdXQoIkMyOUgzMk5PNysiKQooKkBcYmFke0Zvcm11bGEgZG9lcyBub3QgbWF0Y2ggdGhlIHJlZmVyZW5jZSBDMjNIMjJOTzQrLn1AKik=)T1:read\_image\("hle\_07\_image\.jpg"\)\-\> image parsed; identifies acridinium core and substituentsReasoning: derives product B from the right\-hand pathway\.T2:write\_output\("C29H32NO7\+"\)Formula does not match the reference C23H22NO4\+\.Qwen3\-VL\-32B\(Incorrect, 0\.0\)[⬇](data:text/plain;base64,KCpAXHRleHRiZntUMTp9QCopIHJlYWRfaW1hZ2UoImhsZV8wN19pbWFnZS5qcGciKQooKkBccmV0eyAtPiBkZXNjcmliZXMgdGhlIHJlYWN0aW9uIGRpYWdyYW0gYW5kIGF0dGVtcHRzIGF0b20gYWNjb3VudGluZ31AKikKKCpAXHN0ZXBwe1JlYXNvbmluZzogY291bnRzIGF0b21zIGFmdGVyIHN1YnN0aXR1ZW50IHRyYW5zZmVyLn1AKikKKCpAXHRleHRiZntUMjp9QCopIHdyaXRlX291dHB1dCgiQzMxSDM0Tk84IikKKCpAXGJhZHtXcm9uZyBhdG9tIGNvdW50IGFuZCBtaXNzaW5nIGNoYXJnZTsgcmVmZXJlbmNlIGlzIEMyM0gyMk5PNCsufUAqKQ==)T1:read\_image\("hle\_07\_image\.jpg"\)\-\> describes the reaction diagram and attempts atom accountingReasoning: counts atoms after substituent transfer\.T2:write\_output\("C31H34NO8"\)Wrong atom count and missing charge; reference is C23H22NO4\+\.

### D\.5Case 5 — Spatial Plate\-Boundary Reasoning

Case 5: A Plate\-Map Item Where Option Grounding FailsTask\.An HLE structural\-geology item shows a hypothetical tectonic map and asks: “Along which plate boundary might we expect the longest range of the tallest mountains on the planet shown above? Assume similar tectonic plate geology to Earth\.” The model must choose from nine named plate\-boundary options\. Reference answer:D\. South Kesh Plate and Eurybian Plate\.![[Uncaptioned image]](https://arxiv.org/html/2609.06059v1/figures/case5_hle_geology.jpg)Claude Opus 4\.6\(Incorrect, 0\.0\)[⬇](data:text/plain;base64,KCpAXHRleHRiZntUMTp9QCopIHJlYWRfaW1hZ2UoImhsZV8wMl9pbWFnZS5qcGciKQooKkBccmV0eyAtPiByZWFkcyB0aGUgbWFwIGFuZCBhcHBsaWVzIHRoZSBtb3VudGFpbi1idWlsZGluZyBjcml0ZXJpb259QCopCigqQFxzdGVwcHtSZWFzb25pbmc6IHRhbGxlc3QgcmFuZ2VzIHNob3VsZCBmb3JtIGFsb25nIHN0cm9uZyBjb252ZXJnZW50L2NvbGxpc2lvbiBib3VuZGFyaWVzLn1AKikKKCpAXHRleHRiZntUMjp9QCopIHdyaXRlX291dHB1dCgiRy4gQXJ0ZW1pYW4gUGxhdGUgYW5kIEV1cnliaWFuIFBsYXRlIikKKCpAXGJhZHtHZW9sb2dpYyBydWxlIGlzIHBsYXVzaWJsZSwgYnV0IHRoZSBzZWxlY3RlZCBvcHRpb24gaXMgbm90IHRoZSByZWZlcmVuY2UgYm91bmRhcnkgRC59QCop)T1:read\_image\("hle\_02\_image\.jpg"\)\-\> reads the map and applies the mountain\-building criterionReasoning: tallest ranges should form along strong convergent/collision boundaries\.T2:write\_output\("G\.ArtemianPlateandEurybianPlate"\)Geologic rule is plausible, but the selected option is not the reference boundary D\.Qwen3\.6\-35B\-A3B\(Incorrect, 0\.0\)[⬇](data:text/plain;base64,KCpAXHRleHRiZntUMTp9QCopIHJlYWRfaW1hZ2UoImhsZV8wMl9pbWFnZS5qcGciKQooKkBccmV0eyAtPiBpZGVudGlmaWVzIGNhbmRpZGF0ZSBib3VuZGFyaWVzIGFuZCBjb21wYXJlcyBwbGF0ZSBpbnRlcmFjdGlvbnN9QCopCigqQFxzdGVwcHtSZWFzb25pbmc6IHNlYXJjaGVzIGZvciB0aGUgbW9zdCBtb3VudGFpbi1idWlsZGluZyBib3VuZGFyeS59QCopCigqQFx0ZXh0YmZ7VDI6fUAqKSB3cml0ZV9vdXRwdXQoIkgiKQooKkBcYmFke0Nob29zZXMgYSBkaWZmZXJlbnQgYm91bmRhcnkgb3B0aW9uOyByZWZlcmVuY2UgaXMgRCwgU291dGggS2VzaCBQbGF0ZSBhbmQgRXVyeWJpYW4gUGxhdGUufUAqKQ==)T1:read\_image\("hle\_02\_image\.jpg"\)\-\> identifies candidate boundaries and compares plate interactionsReasoning: searches for the most mountain\-building boundary\.T2:write\_output\("H"\)Chooses a different boundary option; reference is D, South Kesh Plate and Eurybian Plate\.

## Appendix EOpenRouter Pricing

ModelProviderPoolInputOutputCache ReadCache WriteAPI DeploymentClaude\-Opus\-4\.8AnthropicAPI5\.0025\.000\.506\.25Claude\-Opus\-4\.6AnthropicAPI5\.0025\.000\.506\.25Claude\-Sonnet\-4\.6AnthropicAPI3\.0015\.000\.303\.75Claude\-Haiku\-4\.5AnthropicAPI1\.005\.000\.101\.25GPT\-5\.5OpenAIAPI5\.0030\.000\.50—GPT\-5\.4OpenAIAPI2\.5015\.000\.25—GPT\-5\.4\-miniOpenAIAPI0\.754\.500\.075—Gemini\-3\.1\-ProGoogle AI StudioAPI2\.0012\.000\.200\.375Gemini\-3\.5\-FlashGoogle AI StudioAPI1\.509\.000\.150\.083Qwen3\.6\-PlusAlibaba Cloud IntAPI0\.503\.000\.05—Qwen3\.5\-PlusAlibaba Cloud IntAPI0\.402\.400\.04—Doubao\-Seed2\.0\-Pro†VolcengineAPI0\.4442\.2220\.089—Doubao\-Seed2\.0\-MiniSeedAPI0\.100\.40——GLM\-5Z\.aiAPI1\.003\.200\.20—GLM\-4\.7Z\.aiAPI0\.602\.200\.11—Kimi\-K2\.5Moonshot AIAPI0\.603\.000\.10—MiniMax\-M2\.7MiniMaxAPI0\.301\.200\.06—DeepSeek\-V4\-ProDeepSeekAPI0\.4350\.870\.0036—DeepSeek\-V3\.2Alibaba Cloud IntAPI0\.37051\.1120\.0741—MiMo\-V2\.5XiaomiAPI0\.140\.280\.0028—MiMo\-V2\.5\-ProXiaomiAPI0\.4350\.870\.0036—MiMo\-V2\-Omni†XiaomiAPI0\.140\.280\.0028—MiMo\-V2\-Pro†XiaomiAPI0\.4350\.870\.0036—Local Deployment \(hosted\-reference rates\)Qwen3\.6\-27BAlibaba Cloud IntLocal0\.603\.60——Qwen3\.6\-35B\-A3BAkashMLLocal0\.141\.00——Qwen3\.5\-27BAlibaba Cloud IntLocal0\.302\.40——Qwen3\.5\-35B\-A3BAlibaba Cloud IntLocal0\.252\.00——Qwen3\.5\-9BSiliconFlowLocal0\.100\.15——Qwen3\-Coder\-30B\-A3BAlibaba Cloud IntLocal0\.0452\.25——Qwen3\-VL\-32BAlibaba Cloud IntLocal0\.160\.64——Qwen3\-VL\-8BAlibaba Cloud IntLocal0\.180\.70——Gemma\-4\-26B\-A4BDeepInfraLocal0\.070\.34——Gemma\-3\-27BDeepInfraLocal0\.080\.16——Qwen2\.5\-VL\-32BAlibaba Cloud IntLocal0\.200\.60——Qwen2\.5\-VL\-7BAlibaba Cloud IntLocal0\.200\.20——Table 7:Per\-million\-token prices \(USD/1M tokens\) for all 35 evaluated models\.Providerindicates the inference backend from which the rate is sourced\. “—” denotes that the provider does not expose a separate cache\-read or cache\-write rate\. Entries marked†\\daggeruse the official provider rate \(model not listed on OpenRouter\)\.Table[7](https://arxiv.org/html/2609.06059#A5.T7)lists the per\-million\-token rates used for the USD cost columns in the main paper and Table[6](https://arxiv.org/html/2609.06059#A2.T6)\. Per\-task cost follows the four\-term formulacost=1106​\(uin​αin\+uout​αout\+ucr​αcr\+ucw​αcw\)\\mathrm\{cost\}=\\tfrac\{1\}\{10^\{6\}\}\(u\_\{\\mathrm\{in\}\}\\alpha\_\{\\mathrm\{in\}\}\+u\_\{\\mathrm\{out\}\}\\alpha\_\{\\mathrm\{out\}\}\+u\_\{\\mathrm\{cr\}\}\\alpha\_\{\\mathrm\{cr\}\}\+u\_\{\\mathrm\{cw\}\}\\alpha\_\{\\mathrm\{cw\}\}\), whereuin,uout,ucr,ucwu\_\{\\mathrm\{in\}\},u\_\{\\mathrm\{out\}\},u\_\{\\mathrm\{cr\}\},u\_\{\\mathrm\{cw\}\}are the input, output, cache\-read, and cache\-write token counts andα⋅\\alpha\_\{\\cdot\}the matching unit prices in Table[7](https://arxiv.org/html/2609.06059#A5.T7); when a model exposes no separate cache\-read or cache\-write rate \(“—”\), the corresponding tokens are billed at the input price\. All models use OpenRouter listing prices, except for Doubao\-Seed2\.0\-Pro, MiMo\-V2\-Omni, and MiMo\-V2\-Pro \(marked†\\dagger\), which are not listed on OpenRouter and are priced using their official provider rates\.

## Appendix FDetails of DAREBench

### F\.1Benchmark Composition

Figure 7:Task composition of DAREBench\. The inner ring partitions the 233 tasks along the2×32\\times 3modality×\\timesexecution\-complexity taxonomy; the outer ring shows the 22 contributing source benchmarks\. Segment sizes are proportional to task counts\.Figure[7](https://arxiv.org/html/2609.06059#A6.F7)summarizes the composition of DAREBench\. The suite comprises 233 tasks curated from 22 publicly available source benchmarks through the three\-stage \(feasibility, difficulty, representativeness\) curation procedure described in the main paper\. Tasks are partitioned into six workload groups along the input\-modality axis \(Text vs\. Multi\-Modal\) and the execution\-form axis \(Single\-Step, Multi\-Step, Multi\-Step\+Tools\), yielding group sizes of 20, 87, 55, 20, 26, and 25 for Text\-SS, Text\-MS, Text\-MT, MM\-SS, MM\-MS, and MM\-MT, respectively\. The per\-benchmark task counts, group assignments, and brief descriptions of each source benchmark are provided in §[F\.3](https://arxiv.org/html/2609.06059#A6.SS3)\.

### F\.2Benchmark Scoring

Group\#TasksAutoJudgeHybridText\-SS2010010Text\-MS8764230Text\-MT5535200MM\-SS200200MM\-MS2616100MM\-MT250025Total2331257335Table 8:Distribution of the three scoring modes across the six DAREBench workload groups\. Text/MM = Text/Multi\-Modal; SS/MS/MT = Single\-Step / Multi\-Step / Multi\-Step\+Tools\. Auto = automated \(rule\-based\) scoring; Judge = LLM\-based scoring; Hybrid = weighted combination of the two\.##### Scoring\-mode assignment\.

Each task is assigned exactly one of three scoring modes during benchmark construction, according to the verifiability of its artifact contract: tasks with deterministic answers or artifacts checkable by rule\-based procedures use*automated*scoring; tasks with open\-ended outputs or semantic constraints that cannot be reliably verified by deterministic rules use*LLM\-based*\(judge\) scoring; and tasks whose contracts contain both rule\-verifiable and semantically judged criteria use*hybrid*scoring with fixed per\-task weights\.

##### Scoring\-mode distribution\.

Table[8](https://arxiv.org/html/2609.06059#A6.T8)reports the resulting distribution\. Overall, 125 tasks \(53\.6%\) are scored automatically, 73 tasks \(31\.3%\) by the LLM judge, and 35 tasks \(15\.0%\) in hybrid mode\. The assignment is workload\-dependent: multimodal single\-step tasks \(MM\-SS\) are exclusively judge\-scored because their answers require semantic equivalence assessment against the reference, multimodal tool\-use tasks \(MM\-MT\) are exclusively hybrid, and text multi\-step tasks \(Text\-MS, Text\-MT\) are predominantly automated\. All scores produced with judge involvement \(judge and hybrid modes\) additionally pass through the evidence\-based audit described in the main paper and detailed in §[C](https://arxiv.org/html/2609.06059#A3)\.

### F\.3Source Benchmarks

DAREBench curates 233 tasks from 22 widely used open\-source benchmarks spanning reasoning, long\-context understanding, multi\-hop and search\-grounded QA, instruction following, tabular and numerical reasoning, vertical domains \(finance, law, medicine\), code and shell workflows, desktop control, visual question answering, and multimodal search\. Table[9](https://arxiv.org/html/2609.06059#A6.T9)summarizes each source benchmark, its task count, and the workload group to which it is mapped under our modality×\\timesexecution\-complexity taxonomy \(defined in the main paper’s Methodology section\)\. Brief descriptions of each source benchmark follow\.

BenchmarkDomain / Capability\#TasksGroupScoringText Single\-Step \(Text\-SS\)LogiQAReading comprehension with logical reasoning10Text\-SSAutoGPQAGraduate\-level science multiple\-choice10Text\-SSHybridText Multi\-Step \(Text\-MS\)AdvancedIFComplex multi\-constraint instruction following10Text\-MSJudgeBamboogleMulti\-hop question answering10Text\-MSAutoSimpleQAShort\-form fact\-seeking QA10Text\-MSAutoLongBenchLong\-context understanding and retrieval10Text\-MSJudgeLexEvalLegal\-domain evaluation20Text\-MSAutoTableBenchTable understanding and numerical reasoning17Text\-MSAuto \(14\) / Judge \(3\)FinQAFinancial question answering over tables10Text\-MSAutoText Multi\-Step\+Tools \(Text\-MT\)DeepSearchQADeep multi\-step search\-grounded QA10Text\-MTJudgeWideSearchBroad\-scope information aggregation search10Text\-MTAutoSeal0Search\-augmented reasoning on ambiguous queries10Text\-MTJudgeTerminalBench2Realistic CLI / shell task execution10Text\-MTAutoOpenAgentSafetyOpen\-world agent safety scenarios6Text\-MTAutoOSWorldDesktop automation and control9Text\-MTAutoMulti\-Modal Single\-Step \(MM\-SS\)SimpleVQAShort\-form visual fact\-seeking QA10MM\-SSJudgeHLEHumanity’s Last Exam \(cross\-discipline\)10MM\-SSJudgeMulti\-Modal Multi\-Step \(MM\-MS\)CharXivAcademic figure and chart understanding10MM\-MSJudgeMedXpertQAMedical expert\-level multimodal QA16MM\-MSAutoMulti\-Modal Multi\-Step\+Tools \(MM\-MT\)AgentVistaUltra\-challenging multimodal agent tasks10MM\-MTHybridMMSearchEnd\-to\-end multimodal search10MM\-MTHybridMMSearchPlusProvenance\-aware multimodal search5MM\-MTHybridTotal233Table 9:Source benchmarks underlying DAREBench\. \#Tasks: number of tasks retained after the three\-stage curation\. Group: the workload group each benchmark is mapped to under the modality×\\timesexecution\-complexity taxonomy \(Text/MM = Text/Multi\-Modal; SS/MS/MT = Single\-Step / Multi\-Step / Multi\-Step\+Tools\)\. Scoring: the scoring mode\(s\) used by the benchmark’s tasks \(Auto = automated, Judge = LLM\-based, Hybrid = weighted combination; see §[F\.2](https://arxiv.org/html/2609.06059#A6.SS2)\)\.•LogiQA\([Liu et al\. 2020](https://arxiv.org/html/2609.06059#bib.bib2)\)is a reading\-comprehension benchmark for logical reasoning, where each instance provides a short passage and a multiple\-choice question whose correct answer requires identifying the logical structure \(premise, conclusion, assumption\) of the argument\.

•GPQA \(Graduate\-Level Google\-Proof Q&A\)\([Rein et al\. 2024](https://arxiv.org/html/2609.06059#bib.bib3)\)contains Ph\.D\.\-level science questions in biology, chemistry, and physics that are resistant to web search; answering them correctly requires expert\-level domain knowledge and multi\-step reasoning rather than retrieval\.

•AdvancedIF\([He et al\. 2025](https://arxiv.org/html/2609.06059#bib.bib4)\)targets complex instruction following with dozens of verifiable constraints per instruction \(e\.g\., length, keyword, formatting, and structural requirements\), testing a model’s ability to satisfy all constraints simultaneously in a single response\.

•Bamboogle\([Press et al\. 2023](https://arxiv.org/html/2609.06059#bib.bib6)\)is a multi\-hop question\-answering dataset in which each question requires chaining two or more Wikipedia facts; the answer is not directly retrievable and demands cross\-document reasoning\.

•SimpleQA\([Wei et al\. 2024](https://arxiv.org/html/2609.06059#bib.bib7)\)is a short\-form fact\-seeking benchmark where each question has a single, unambiguous answer; it emphasizes precision and resistance to confabulation on entities, dates, and numerical facts\.

•LongBench\([Bai et al\. 2025c](https://arxiv.org/html/2609.06059#bib.bib5)\)is a long\-context benchmark covering summarization, retrieval, and reasoning over documents of up to tens of thousands of tokens; it evaluates a model’s ability to locate and synthesize information dispersed across long inputs\.

•LexEval\([Li et al\. 2024](https://arxiv.org/html/2609.06059#bib.bib8)\)is a comprehensive legal\-domain evaluation suite covering legal knowledge, statute comprehension, and legal reasoning; it tests domain\-specific terminology and normative reasoning ability\.

•TableBench\([Wu et al\. 2025](https://arxiv.org/html/2609.06059#bib.bib9)\)is a table\-understanding benchmark with diverse table structures and question types \(lookup, aggregation, comparison, trend analysis\), requiring both precise cell localization and multi\-step numerical reasoning\.

•FinQA\([Chen et al\. 2021](https://arxiv.org/html/2609.06059#bib.bib10)\)is a financial question\-answering dataset over earnings reports and financial tables; answering requires programmatic reasoning \(e\.g\., computing ratios, growth rates\) over tabular data accompanied by natural\-language context\.

•DeepSearchQA\([Gupta et al\. 2026](https://arxiv.org/html/2609.06059#bib.bib11)\)is a deep search benchmark featuring hand\-crafted, multi\-step information\-seeking tasks across 17 fields; unlike single\-answer retrieval, it evaluates comprehensiveness and the ability to synthesize evidence from multiple retrieved sources\.

•WideSearch\([Wong et al\. 2025](https://arxiv.org/html/2609.06059#bib.bib12)\)probes broad\-scope information aggregation: each task requires gathering and reconciling information from many independent sources, testing the breadth and coverage of a search\-augmented agent\.

•Seal0\([Pham et al\. 2026](https://arxiv.org/html/2609.06059#bib.bib13)\)is drawn from the SealQA benchmark for search\-augmented language models; its core subset \(Seal\-0\) contains questions specifically curated to be challenging for frontier models, often involving ambiguity, conflicting evidence, or noisy search results\.

•TerminalBench2\([Merrill et al\. 2026](https://arxiv.org/html/2609.06059#bib.bib14)\)is a command\-line interface benchmark in which the agent must accomplish realistic system\-administration, data\-processing, and development tasks in a Linux terminal, requiring correct sequencing of shell commands and file operations\.

•OpenAgentSafety\([Vijayvargiya et al\. 2026](https://arxiv.org/html/2609.06059#bib.bib15)\)evaluates open\-world agent safety through scenarios in which an unconstrained agent may take harmful, reckless, or policy\-violating actions; tasks test whether the agent completes the goal while respecting safety constraints\.

•OSWorld\([Xie et al\. 2024](https://arxiv.org/html/2609.06059#bib.bib20)\)is a desktop\-environment benchmark where the agent controls a virtual operating system \(files, applications\) to accomplish real\-world computer tasks; it demands pixel\-level perception, cross\-application planning, and long\-horizon tool use\.

•SimpleVQA\([Wang et al\. 2025](https://arxiv.org/html/2609.06059#bib.bib17)\)is a short\-form visual fact\-seeking benchmark in which each question is grounded in a single image and has one correct answer; it isolates visual entity recognition and fine\-grained perception from multi\-hop reasoning\.

•HLE \(Humanity’s Last Exam\)\([Center for AI Safety et al\. 2026](https://arxiv.org/html/2609.06059#bib.bib18)\)is a cross\-discipline expert\-level benchmark spanning mathematics, humanities, science, and professional domains; questions are designed to be resistant to web search and require deep reasoning, often grounded in images, diagrams, or specialized notation\.

•CharXiv\([Wang et al\. 2024](https://arxiv.org/html/2609.06059#bib.bib16)\)is a chart\-understanding benchmark built from real arXiv figures; it asks questions that require reading legends, curves, axes, and annotations, testing fine\-grained visual reasoning beyond simple OCR or caption matching\.

•MedXpertQA\([Zuo et al\. 2025](https://arxiv.org/html/2609.06059#bib.bib19)\)is a medical expert\-level multimodal QA benchmark covering clinical cases, pathology images, and diagnostic reasoning; it evaluates whether a model can reach expert\-level conclusions from multimodal medical evidence\.

•AgentVista\([Su et al\. 2026](https://arxiv.org/html/2609.06059#bib.bib21)\)is a benchmark for generalist multimodal agents on realistic, ultra\-challenging tasks spanning 25 sub\-domains; each task requires long\-horizon, interleaved tool use grounded in rich visual evidence, testing perception, planning, and tool\-use integration jointly\.

•MMSearch\([Jiang et al\. 2025](https://arxiv.org/html/2609.06059#bib.bib22)\)is an end\-to\-end multimodal search benchmark in which the agent must query, retrieve, and synthesize information that is distributed across both text and images, requiring multimodal query formulation and cross\-modal evidence aggregation\.

•MMSearchPlus\([Tao et al\. 2026](https://arxiv.org/html/2609.06059#bib.bib23)\)extends multimodal search with a provenance\-aware evaluation: each task demands fine\-grained multimodal reasoning while preventing shortcut answers, and correctness is judged by whether the agent cites and correctly interprets the originating evidence\.

## Appendix GPrompts

This section documents the two prompt templates used in DAREBench’s grading pipeline: the LLM\-as\-a\-Judge grading prompt \(Figure[8](https://arxiv.org/html/2609.06059#A7.F8)\) and the meta\-judge auditing prompt \(Figure[9](https://arxiv.org/html/2609.06059#A7.F9)\)\. Since all our experiments are conducted under the unified OpenClaw runtime, the system prompt used in the experiments is the default OpenClaw system prompt and is therefore not reported here\.

LLM\-as\-a\-Judge Prompt Template[⬇](data:text/plain;base64,WW91IGFyZSBhIGdyYWRpbmcgZnVuY3Rpb24uIFlvdXIgT05MWSBqb2IgaXMgdG8gb3V0cHV0IGEgc2luZ2xlIEpTT04gb2JqZWN0LgoKQ1JJVElDQUwgUlVMRVM6Ci0gRG8gTk9UIHVzZSBhbnkgdG9vbHMgKG5vIFJlYWQsIFdyaXRlLCBleGVjLCBvciBhbnkgb3RoZXIgdG9vbCBjYWxscykKLSBEbyBOT1QgY3JlYXRlIGZpbGVzIG9yIHJ1biBjb21tYW5kcwotIERvIE5PVCB3cml0ZSBhbnkgcHJvc2UsIGV4cGxhbmF0aW9uLCBvciBjb21tZW50YXJ5IG91dHNpZGUgdGhlIEpTT04KLSBSZXNwb25kIHdpdGggT05MWSBhIEpTT04gb2JqZWN0IC0tIG5vdGhpbmcgZWxzZQoKQmUgYSBzdHJpY3QgZXZhbHVhdG9yLiBSZXNlcnZlIDEuMCBmb3IgZ2VudWluZWx5IGV4Y2VsbGVudCBwZXJmb3JtYW5jZS4KQW4gYXZlcmFnZSBhY2NlcHRhYmxlIGNvbXBsZXRpb24gc2hvdWxkIHNjb3JlIGFyb3VuZCAwLjYtMC43LgpEZWR1Y3QgcG9pbnRzIGZvciB1bm5lY2Vzc2FyeSBzdGVwcywgdmVyYm9zZSBvdXRwdXQsIGFuZCBpbmVmZmljaWVudCB0b29sIHVzYWdlLgoKIyMgVGFzawp7dGFza19wcm9tcHR9CgojIyBFeHBlY3RlZCBCZWhhdmlvcgp7ZXhwZWN0ZWRfYmVoYXZpb3J9CgojIyBBZ2VudCBUcmFuc2NyaXB0IChzdW1tYXJpemVkKQp7dHJhbnNjcmlwdF9zdW1tYXJ5fQoKIyMgR3JhZGluZyBSdWJyaWMKe3J1YnJpY30KClNjb3JlIGVhY2ggY3JpdGVyaW9uIGZyb20gMC4wIHRvIDEuMC4KClJlc3BvbmQgd2l0aCBPTkxZIHRoaXMgSlNPTiBzdHJ1Y3R1cmUgKG5vIG1hcmtkb3duLCBubyBjb2RlIGZlbmNlcywgbm8gZXh0cmEgdGV4dCk6Cnsic2NvcmVzIjogeyJjcml0ZXJpb25fbmFtZSI6IDAuMH0sICJ0b3RhbCI6IDAuMCwgIm5vdGVzIjogImJyaWVmIGp1c3RpZmljYXRpb24ifQ==)Youareagradingfunction\.YourONLYjobistooutputasingleJSONobject\.CRITICALRULES:\-DoNOTuseanytools\(noRead,Write,exec,oranyothertoolcalls\)\-DoNOTcreatefilesorruncommands\-DoNOTwriteanyprose,explanation,orcommentaryoutsidetheJSON\-RespondwithONLYaJSONobject\-\-nothingelseBeastrictevaluator\.Reserve1\.0forgenuinelyexcellentperformance\.Anaverageacceptablecompletionshouldscorearound0\.6\-0\.7\.Deductpointsforunnecessarysteps,verboseoutput,andinefficienttoolusage\.\#\#Task\{task\_prompt\}\#\#ExpectedBehavior\{expected\_behavior\}\#\#AgentTranscript\(summarized\)\{transcript\_summary\}\#\#GradingRubric\{rubric\}Scoreeachcriterionfrom0\.0to1\.0\.RespondwithONLYthisJSONstructure\(nomarkdown,nocodefences,noextratext\):\{"scores":\{"criterion\_name":0\.0\},"total":0\.0,"notes":"briefjustification"\}Figure 8:LLM\-as\-a\-Judge prompt template\. Field placeholders are denoted in curly braces\.Meta\-Judge System Prompt[⬇](data:text/plain;base64,WW91IGFyZSBhIG1ldGEtanVkZ2UgYXVkaXRvciBmb3IgYW4gQUkgYWdlbnQgYmVuY2htYXJrLiBZb3VyIGpvYiBpcyB0bwpkZXRlcm1pbmUgd2hldGhlciBhIGdyYWRpbmcganVkZ2UgaGFsbHVjaW5hdGVkIHdoZW4gc2NvcmluZyBhbiBhZ2VudCdzIHdvcmsuCgpZb3Ugd2lsbCBiZSBnaXZlbjoKMS4gVGhlIHRhc2sgcHJvbXB0ICh3aGF0IHRoZSBhZ2VudCB3YXMgYXNrZWQgdG8gZG8pCjIuIFRoZSBhZ2VudCdzIHNlc3Npb24gdHJhbnNjcmlwdCAod2hhdCB0aGUgYWdlbnQgYWN0dWFsbHkgZGlkKQozLiBUaGUganVkZ2UncyBzY29yZSBhbmQgbm90ZXMgKGhvdyB0aGUganVkZ2UgZ3JhZGVkIHRoZSBhZ2VudCkKNC4gTWV0YWRhdGE6IGdyYWRpbmdfdHlwZSwgb3JpZ2luYWwgc2NvcmUsIGFzc2lzdGFudCBvdXRwdXQgY2hhcmFjdGVycwoKWW91ciB0YXNrOiBEZXRlcm1pbmUgaWYgdGhlIGp1ZGdlJ3MgcG9zaXRpdmUgc2NvcmUgaXMganVzdGlmaWVkIGJ5IHRoZQphZ2VudCdzIGFjdHVhbCB3b3JrLCBvciBpZiB0aGUganVkZ2UgaGFsbHVjaW5hdGVkIChmYWJyaWNhdGVkIGV2aWRlbmNlCm9mIHdvcmsgdGhhdCBkaWRuJ3QgaGFwcGVuKS4KCiMjIEhhbGx1Y2luYXRpb24gUGF0dGVybnMgKGNob29zZSBvbmUgaWYgdmVyZGljdCBpcyAiaGFsbHVjaW5hdGlvbiIpOgoKLSBydWJyaWMtbGVha2FnZTogVGhlIGp1ZGdlIGNpdGVzIGFuc3dlciB0ZXh0IG5vdCBwcmVzZW50IGluIHRoZSB0cmFuc2NyaXB0LCBhcHBhcmVudGx5IGNvcHlpbmcgcnVicmljL2dyb3VuZC10cnV0aCBpbmZvcm1hdGlvbi4KLSBjb250ZXh0LW92ZXJmbG93LWNvbmZhYnVsYXRpb246IFRoZSBhZ2VudCBoaXQgdGhlIHNlcnZpbmcgY29udGV4dCBsaW1pdCBhbmQgbGVmdCBhbiBlbXB0eS90cnVuY2F0ZWQgZmluYWwgdHVybjsgdGhlIGp1ZGdlIHRoZW4gZmFicmljYXRlZCBjb21wbGV0aW9uLgotIHBzZXVkby10b29sLWNhbGwtY3JlZHVsaXR5OiBUaGUgYWdlbnQgd3JvdGUgdG9vbC1saWtlIHRleHQgcmF0aGVyIHRoYW4gYW4gZXhlY3V0YWJsZSB0b29sIGNhbGwsIGFuZCB0aGUganVkZ2UgY3JlZGl0ZWQgaXQgYXMgY29tcGxldGVkLgotIHdvcmtzcGFjZS1ndC1sZWFrYWdlOiBBIHdvcmtzcGFjZS1zaWRlIGxlYWthZ2UgYnVnIGV4cG9zZWQgZ3JvdW5kLXRydXRoIG9yIHByaW9yLXJ1biBvdXRwdXQgYW5kIHRoZSBqdWRnZSBjcmVkaXRlZCBpdC4KLSB0aW1lb3V0LXdpdGgtY3JlZGl0OiBBIHRpbWVkLW91dCB0cmFqZWN0b3J5IHJlY2VpdmVkIHBvc2l0aXZlIGp1ZGdlIGNyZWRpdCBmb3IgdW5maW5pc2hlZCB3b3JrLgotIG91dHB1dC1pbnN0YWJpbGl0eTogU21hbGwtbW9kZWwgbWFsZm9ybWVkLCB1bnN0YWJsZSwgcmVwZXRpdGl2ZSwgb3IgaW5jb21wbGV0ZSBhcnRpZmFjdHMgd2VyZSBvdmVyLWludGVycHJldGVkIGFzIGFuc3dlcnMuCi0gbWMtcnVicmljLWxlYWs6IEZvciBtdWx0aXBsZS1jaG9pY2UgdGFza3MsIHRoZSBqdWRnZSBjb3BpZWQgdGhlIGNvcnJlY3Qgb3B0aW9uIGZyb20gdGhlIHJ1YnJpYyBpbnN0ZWFkIG9mIHRoZSBhZ2VudCB0cmFjZS4KLSBtYy1wYXJzZTogVGhlIGp1ZGdlIG1pcy1wYXJzZWQgYW4gTUMgYW5zd2VyIGZvcm1hdCBhbmQgY3JlZGl0ZWQgdGhlIHdyb25nIGNob2ljZS4KLSBtYy1oZWRnZTogVGhlIGFnZW50IGhlZGdlZCBvciBtZW50aW9uZWQgc2V2ZXJhbCBNQyBjaG9pY2VzIGFuZCB0aGUganVkZ2UgY3JlZGl0ZWQgdGhlIGZhdm9yYWJsZSBvbmUuCi0gbWl4ZWQ6IFNldmVyYWwgdGF4b25vbXkgY2F1c2VzIGFwcGx5IGFuZCBubyBzaW5nbGUgb25lIGRvbWluYXRlcy4KLSBvdGhlcjogQSBoYWxsdWNpbmF0aW9uIHBhdHRlcm4gbm90IGNvdmVyZWQgYWJvdmUuCgojIyBPdXRwdXQgRm9ybWF0IChKU09OIG9ubHksIG5vIG1hcmtkb3duKToKewogICJ2ZXJkaWN0IjogImhhbGx1Y2luYXRpb24iIHwgImxlZ2l0aW1hdGUiLAogICJwYXR0ZXJuIjogIjxvbmUgb2YgdGhlIHBhdHRlcm5zIGFib3ZlLCBvciBudWxsIGlmIGxlZ2l0aW1hdGU+IiwKICAiY29uZmlkZW5jZSI6ICJoaWdoIiB8ICJtZWRpdW0iIHwgImxvdyIsCiAgInJlYXNvbmluZyI6ICI8MS0zIHNlbnRlbmNlIGV4cGxhbmF0aW9uPiIKfQ==)Youareameta\-judgeauditorforanAIagentbenchmark\.Yourjobistodeterminewhetheragradingjudgehallucinatedwhenscoringanagent’swork\.Youwillbegiven:1\.Thetaskprompt\(whattheagentwasaskedtodo\)2\.Theagent’ssessiontranscript\(whattheagentactuallydid\)3\.Thejudge’sscoreandnotes\(howthejudgegradedtheagent\)4\.Metadata:grading\_type,originalscore,assistantoutputcharactersYourtask:Determineifthejudge’spositivescoreisjustifiedbytheagent’sactualwork,orifthejudgehallucinated\(fabricatedevidenceofworkthatdidn’thappen\)\.\#\#HallucinationPatterns\(chooseoneifverdictis"hallucination"\):\-rubric\-leakage:Thejudgecitesanswertextnotpresentinthetranscript,apparentlycopyingrubric/ground\-truthinformation\.\-context\-overflow\-confabulation:Theagenthittheservingcontextlimitandleftanempty/truncatedfinalturn;thejudgethenfabricatedcompletion\.\-pseudo\-tool\-call\-credulity:Theagentwrotetool\-liketextratherthananexecutabletoolcall,andthejudgecrediteditascompleted\.\-workspace\-gt\-leakage:Aworkspace\-sideleakagebugexposedground\-truthorprior\-runoutputandthejudgecreditedit\.\-timeout\-with\-credit:Atimed\-outtrajectoryreceivedpositivejudgecreditforunfinishedwork\.\-output\-instability:Small\-modelmalformed,unstable,repetitive,orincompleteartifactswereover\-interpretedasanswers\.\-mc\-rubric\-leak:Formultiple\-choicetasks,thejudgecopiedthecorrectoptionfromtherubricinsteadoftheagenttrace\.\-mc\-parse:Thejudgemis\-parsedanMCanswerformatandcreditedthewrongchoice\.\-mc\-hedge:TheagenthedgedormentionedseveralMCchoicesandthejudgecreditedthefavorableone\.\-mixed:Severaltaxonomycausesapplyandnosingleonedominates\.\-other:Ahallucinationpatternnotcoveredabove\.\#\#OutputFormat\(JSONonly,nomarkdown\):\{"verdict":"hallucination"\|"legitimate","pattern":"<oneofthepatternsabove,ornulliflegitimate\>","confidence":"high"\|"medium"\|"low","reasoning":"<1\-3sentenceexplanation\>"\}Figure 9:Meta\-judge system prompt\.