EvoClawBench: Can Agents Learn Reusable Skills from Their Own Runs?

# EvoClawBench: Can Agents Learn Reusable Skills from Their Own Runs?
Source: [https://arxiv.org/html/2607.09711](https://arxiv.org/html/2607.09711)
Zhiyuan Peng1Xin Yin2Chenhao Ying1Zhe Cui3 Zixiang Ding3Zhenhua Liu3Jiang Wu3Yuan Luo1 1Shanghai Jiao Tong University2Zhejiang University3Hithink Research \{pzy2000, yingchenhao, yuanluo\}@sjtu\.edu\.cn xyin@zju\.edu\.cn \{cuizhe, dingzixiang, liuzhenhua, wujiang2\}@myhexin\.com

###### Abstract

Existing agent benchmarks primarily test task completion, tool use, or skill utility, but do not isolate whether a runtime can convert evidence from its own runs into reusable skills that improve fresh executions after authoring overhead\. We introduceEvoClawBench, a benchmark for this closed\-loop skill\-learning question on repeated, fixture\-backed tasks\. EvoClawBench compares direct execution without skills,PreSkillauthoring before execution, andPostSkillsummarization from first\-run evidence followed by a fresh second execution\. The suite contains 100 tasks and 502 sub\-problems across coding, data, office, security, operations, and domain\-document workflows, with support for multiple agent runtimes\. Experiments with OpenClaw and nanobot under local execution show that directBaselineperformance is strongly runtime\-dependent: OpenClaw remains below 20% across models, while nanobot ranges from 56\.45% to 96\.13%\. Self\-authored skills have mixed effects\. nanobotGPT\-5\.4stays above 96% in all modes andMiniMax\-M2\.7improves from 90\.97% to 94\.50% underPostSkill, but nanobotDeepSeek\-V4\-Prodrops from 77\.77% to 4\.80% withPreSkilland 0\.99% withPostSkill\. OpenClaw shows similarly non\-monotonic behavior, with some skill runs near baseline and others collapsing\. These results indicate that learning reusable skills from an agent’s own runs is selective and cost\-sensitive, rather than an automatic benefit of adding skill authoring to an agent loop\.

EvoClawBench: Can Agents Learn Reusable Skills from Their Own Runs?

Table 1:Comparison of EvoClawBench with existing benchmarks\. “Skill Cond\.” indicates whether the benchmark includes agent skills\. “Det\. Verifier” indicates whether deterministic \(non\-LLM\) verification is included\. “Closed Loop” indicates whether the same runtime creates and reuses skills in fresh runs\.## 1Introduction

Large language model agents are increasingly used to turn foundation models into interactive workers\. Recent evaluations place agents in repositories, terminals, browsers, graphical interfaces, tool APIs, and executable environments\(Fourneyet al\.,[2024](https://arxiv.org/html/2607.09711#bib.bib26); Wanget al\.,[2025](https://arxiv.org/html/2607.09711#bib.bib24); Xuet al\.,[2025](https://arxiv.org/html/2607.09711#bib.bib25); Agasheet al\.,[2025](https://arxiv.org/html/2607.09711#bib.bib27); Researchet al\.,[2026](https://arxiv.org/html/2607.09711#bib.bib23)\)\. As a result, the benchmark ecosystem has expanded rapidly\. SWE\-Bench and SWE\-agent\-style evaluations emphasize repository\-level software engineering\(Jimenezet al\.,[2024](https://arxiv.org/html/2607.09711#bib.bib1); Yanget al\.,[2024](https://arxiv.org/html/2607.09711#bib.bib5)\); Terminal\-Bench, AppWorld, WebArena, and OSWorld measure command\-line, app\-world, browser, and desktop interaction\(Zhouet al\.,[2024](https://arxiv.org/html/2607.09711#bib.bib10); Trivediet al\.,[2024](https://arxiv.org/html/2607.09711#bib.bib6); Xieet al\.,[2024](https://arxiv.org/html/2607.09711#bib.bib13); Merrillet al\.,[2026](https://arxiv.org/html/2607.09711#bib.bib3)\); and SkillsBench evaluates skills as first\-class artifacts across diverse tasks\(Liet al\.,[2026](https://arxiv.org/html/2607.09711#bib.bib2)\)\. These benchmarks have made agent capability more measurable, but they leave a closed\-loop learning question under\-specified\. They primarily test task completion, tool use, environment interaction, or the utility of curated and self\-generated skills\. They do not isolate the process in which the same agent runtime creates a skill, summarizes evidence from its own completed run, reuses that skill in a fresh execution, and pays the full token, cost, and time overhead of doing so\. This leaves a focused evaluation question:*Can LLM agents learn reusable skills from their own runs?*

To answer this question, we introduce EvoClawBench, a benchmark designed around this question\. EvoClawBench evaluates an agent on repeated, structured tasks that contain multiple related sub\-problems\. This design creates an opportunity for the agent to recognize shared patterns and, in the own\-run condition, convert first\-run evidence into reusable skills\. The benchmark focuses on reusable skills because they are becoming a practical way to specialize LLM agents for recurring tool\-use tasks\. SkillsBench reports 84,192 skills collected within 136 days\(Liet al\.,[2026](https://arxiv.org/html/2607.09711#bib.bib2)\), suggesting a shift from prompt engineering by individual users toward a marketplace of reusable procedural memory\. Alongside stronger foundation models and richer harnesses, a complementary line of work treats procedural artifacts as external skills or learned procedures that can be derived, refined, reused, or moved into compact task\-family state from agent experience\(Yanget al\.,[2026b](https://arxiv.org/html/2607.09711#bib.bib28); Niet al\.,[2026](https://arxiv.org/html/2607.09711#bib.bib29); Wanget al\.,[2026](https://arxiv.org/html/2607.09711#bib.bib30); Yanget al\.,[2026a](https://arxiv.org/html/2607.09711#bib.bib31); Xieet al\.,[2026](https://arxiv.org/html/2607.09711#bib.bib32)\)\. In this paper, we use*skills*for lightweight, inspectable procedural documents with optional scripts, templates, and references, because such artifacts can be installed, edited, and injected into an agent’s context without retraining the model\.

For each task, we compare three strategies\. InBaseline, the agent solves the task directly and is forbidden from creating or editing skills\. InPreSkill, the same model and runtime first create one or more task\-specific skills, then a fresh execution workspace solves the task using only those generated skills\. InPostSkill, the agent first solves the task without skills, receives a compact first\-run context containing the prompt, grading summary, output previews, and transcript summary, then writes reusable skills that are copied into a fresh second execution\. This protocol isolates two related but distinct forms of self\-authored skill construction\.PreSkillis a pre\-execution control that measures whether an agent can infer a useful reusable procedure before solving a task\.PostSkillmost directly tests the title question: whether an agent can distill reusable procedural memory from its own completed run\. Both are compared againstBaselineunder execution\-only and end\-to\-end metrics, so a skill workflow must improve not only task score but also justify its additional token, cost, and time overhead\.

![Refer to caption](https://arxiv.org/html/2607.09711v1/x1.png)Figure 1:Overview for EvoClawBench\. The task\-suite panel summarizes the 100 non\-sanity tasks in the official suite;[Figure˜2](https://arxiv.org/html/2607.09711#S3.F2)breaks down their families and fixture formats\. The pipeline routes tasks through a runtime\-agnostic agent interface, comparesBaseline,PreSkill, andPostSkillin fresh workspaces, and reports grader\-backed quality, resource, skill, and mutation\-integrity metrics\.Our paper makes the following contributions:

- •Benchmark\.We build EvoClawBench, a benchmark for evaluating whether LLM agents can learn reusable skills from controlled self\-authored skill workflows and their own first\-run evidence\. It covers 100 tasks with repeated sub\-problems, fixtures, automated or hybrid graders, and runtime\-adapter support\.
- •Three\-mode evaluation protocol\.We formalizeBaseline,PreSkill, andPostSkillconditions, separating execution\-only performance from end\-to\-end workflow cost and detecting skill mutation violations during reuse\.
- •Empirical findings\.We provide a current cross\-runtime result table showing that runtime scaffolding strongly affects absolute scores, and that self\-authored skills do not produce monotonic gains\. Focused subset checks make the cost tradeoff, skill\-content controls, and hybrid\-judge robustness explicit\.
- •

## 2Related Work

#### Agent and software engineering benchmarks\.

Repository\-level benchmarks evaluate whether agents can modify realistic codebases and satisfy executable checks\. SWE\-Bench evaluates real GitHub issue resolution\(Jimenezet al\.,[2024](https://arxiv.org/html/2607.09711#bib.bib1)\), while SWE\-agent studies agent\-computer interfaces for automated software engineering\(Yanget al\.,[2024](https://arxiv.org/html/2607.09711#bib.bib5)\)\. Terminal\-Bench focuses on realistic command\-line tasks in isolated environments\(Merrillet al\.,[2026](https://arxiv.org/html/2607.09711#bib.bib3)\)\. MLE\-bench extends this line to end\-to\-end machine\-learning engineering on Kaggle competitions\(Chanet al\.,[2025](https://arxiv.org/html/2607.09711#bib.bib15)\)\. AgentBench and GAIA evaluate broader agentic reasoning, tool use, and multi\-step problem solving across heterogeneous environments or questions\(Liuet al\.,[2024](https://arxiv.org/html/2607.09711#bib.bib8); Mialonet al\.,[2024](https://arxiv.org/html/2607.09711#bib.bib9)\)\. These benchmarks are useful for measuring task completion, but they do not isolate whether an agent can write reusable skills at runtime and then benefit from them\.

#### Web, GUI, and tool\-use benchmarks\.

Several benchmarks focus on agents that act through browsers, operating systems, applications, or APIs\. WebArena builds realistic web environments for long\-horizon browser tasks\(Zhouet al\.,[2024](https://arxiv.org/html/2607.09711#bib.bib10)\), and Mind2Web provides open\-ended web tasks collected from real websites\(Denget al\.,[2023](https://arxiv.org/html/2607.09711#bib.bib11)\)\. WorkArena\+\+ targets compositional knowledge\-work workflows in ServiceNow\(Boisvertet al\.,[2024](https://arxiv.org/html/2607.09711#bib.bib12)\), while OSWorld evaluates multimodal agents on real desktop\-computer tasks\(Xieet al\.,[2024](https://arxiv.org/html/2607.09711#bib.bib13)\)\. ToolLLM and ToolBench evaluate API\-use capability at large scale\(Qinet al\.,[2023](https://arxiv.org/html/2607.09711#bib.bib14)\)\. AppWorld andτ\\tau\-Bench broaden the scope to controllable app worlds and tool\-agent\-user interaction settings\(Trivediet al\.,[2024](https://arxiv.org/html/2607.09711#bib.bib6); Yaoet al\.,[2024](https://arxiv.org/html/2607.09711#bib.bib7)\)\. These environments stress planning and tool execution, but their evaluation conditions do not center on agent\-authored artifacts that are reused in fresh runs\.

#### Skill and procedural memory evaluation\.

Skills can be viewed as external procedural memory for agents, related to work on agent skills, reflection, experience accumulation, procedural memory, and open\-ended skill acquisition\(Zhanget al\.,[2025](https://arxiv.org/html/2607.09711#bib.bib4); Yaoet al\.,[2023](https://arxiv.org/html/2607.09711#bib.bib16); Shinnet al\.,[2023](https://arxiv.org/html/2607.09711#bib.bib17); Zhaoet al\.,[2024](https://arxiv.org/html/2607.09711#bib.bib18); Fanget al\.,[2026](https://arxiv.org/html/2607.09711#bib.bib21); Wanget al\.,[2023](https://arxiv.org/html/2607.09711#bib.bib22); Xu and Yan,[2026](https://arxiv.org/html/2607.09711#bib.bib19); Wu and Zhang,[2026](https://arxiv.org/html/2607.09711#bib.bib20)\)\. SkillsBench is closest to our setting because it evaluates skills as first\-class artifacts, including curated and self\-generated skill conditions\(Liet al\.,[2026](https://arxiv.org/html/2607.09711#bib.bib2)\)\. Unlike SkillsBench, which evaluates the utility of curated Skills and a pre\-solution self\-generated Skills condition, our benchmark separates pre\-execution skill authoring from post\-run skill learning\. In particular,PreSkillcontrols for skills authored before task execution, whilePostSkilldirectly tests whether an agent can convert evidence from its own prior run into a reusable skill for later execution\. Taken together, these lines of work motivate task and tool\-use evaluation, whereas EvoClawBench makes the skill lifecycle itself the evaluated object: authoring, own\-run distillation, reuse, mutation checks, and end\-to\-end cost\.[Table˜1](https://arxiv.org/html/2607.09711#S0.T1)summarizes the benchmark position at a high level\. EvoClawBench is designed to expose whether agent\-authored and own\-run\-derived skills improve task success after accounting for the full cost of creating and reusing skills\.

## 3EvoClawBench Construction

EvoClawBench is constructed to test whether an agent can recognize repeated structure across sub\-problems and encode that structure as a reusable skill\. The benchmark implementation is organized around task definitions, workspace fixtures, runtime adapters, graders, and metric aggregation\.

### 3\.1Task Suite

Tasks are specified as markdown files with YAML frontmatter, a natural\-language prompt, expected behavior, sub\-problem descriptions, grading criteria, and automated or hybrid grading logic\. Each task typically contains five to ten structurally related sub\-problems, creating repeated situations in which an agent can benefit from recording a general procedure rather than solving each instance independently\. The repository also includes a sanity task used by the loader, but we exclude it from official suite counts\. After this exclusion, the suite contains 100 tasks and 502 sub\-problems\. It covers recurring workflows such as data transformation, log analysis, API scaffolding, test generation, configuration migration, security review, document extraction, database operations, Excel analytics, web extraction, document generation, data pipelines, email and invoice processing, shell automation, CI generation, dependency audit, environment configuration, and metrics anomaly detection\. The full suite extends these surfaces into harder repository\-local families, including finance, legal, healthcare, procurement, HR, support, DevOps/SRE, privacy, localization, commerce, facilities, public\-sector audit, CRM, and travel/office coordination\. Many hard\-mode fixtures are synthetic repository\-local cases with competing evidence packets, stale revisions, and decoy records\. This design reduces privacy risk while preserving repeated, artifact\-producing workflows\.[Figure˜2](https://arxiv.org/html/2607.09711#S3.F2)summarizes official\-suite coverage from task metadata rather than model results: the suite combines seed workflows, generated hard\-mode families, and repository\-local fixtures in multiple file formats\.

![Refer to caption](https://arxiv.org/html/2607.09711v1/x2.png)Figure 2:Official\-suite task and fixture distributions in EvoClawBench\. Panel \(a\) counts the 100 non\-sanity tasks by seed or generated family\. Panel \(b\) counts repository\-local workspace fixture files by grouped file format\. These metadata counts describe benchmark coverage, not model performance\.
### 3\.2Workspace and Fixtures

Each task run is executed in an isolated workspace\. The benchmark copies input assets from its fixture directory into that workspace, and the agent must write grader\-visible outputs to the expected paths\. This design avoids relying on conversation\-only answers and instead requires concrete artifacts, including JSON files, scripts, reports, Dockerfiles, CI files, and structured extraction outputs\. Task definitions instruct agents not to modify input fixtures, while graders inspect output artifacts rather than trusting the agent’s final natural\-language answer\.

The benchmark supports both local subprocess execution and Docker\-based execution\. For the reported experiments, we use local subprocess execution through the OpenClaw and nanobot runtimes, set\-\-environment local, run 32 workers, and execute one benchmark run per task and mode\. Because these runs do not use the Docker sandbox, no Docker resource limit is applied\. The reported table should therefore be read as local runtime\-adapter results rather than as a claim about all supported sandbox configurations\.

### 3\.3Skill Authoring and Reuse

EvoClawBench represents reusable procedures with the standardSKILL\.mdformat adopted by the supported runtimes\. During skill\-authoring phases, the workspace is seeded with askill\-creatorbundle that instructs the agent to create a well\-formed skill\. The benchmark then scans generated skills from the authoring workspace and copies them into fresh execution workspaces for reuse\. To prevent a reuse run from becoming another authoring run, EvoClawBench hashes all non\-seeded skill files before and after execution\. If a reuse execution adds, deletes, or edits a skill, the benchmark records askill\_mutation\_violation\. This guardrail fixes the experimental condition:PreSkillandPostSkillexecution may consult generated skills, but may not revise them while solving the task\.

### 3\.4Runtimes

EvoClawBench currently supports OpenClaw and nanobot\. OpenClaw is invoked throughopenclaw agent \-\-message, while nanobot is invoked throughnanobot agent \-\-workspace \.\.\. \-\-config \.\.\. \-\-message \.\.\.\. The runtime abstraction keeps task prompts, workspaces, grading, and metrics shared across claws, allowing the benchmark to compare the same model under different agent scaffolds\. Additional runtimes can be added by implementing the same execution interface and usage extraction logic\.

## 4Evaluation Protocol

![Refer to caption](https://arxiv.org/html/2607.09711v1/x3.png)Figure 3:Structure of a reusable skill artifact in EvoClawBench\. Generated skills use aSKILL\.mddocument with optional supporting scripts or references, are injected into the agent context during reuse, and are protected by before/after mutation checks during execution\.### 4\.1Three Execution Strategies

For a tasktt, modelmm, and runtimerr, EvoClawBench evaluates three strategies\.

#### Baseline\.

The agent receives the task prompt and solves it directly\. The prompt prefix forbids creating, editing, or deleting skills\. This condition measures the agent’s direct task\-solving ability\.

#### PreSkill\.

The agent first enters a skill\-authoring phase\. It reads the skill\-creator bundle and writes one or more task\-specific skills underskills/<skill\-name\>/SKILL\.md\. The final task outputs are not graded during this phase\. The benchmark then creates a fresh workspace, copies the generated skills into it, and runs the task execution phase with skill mutation disabled\.

#### PostSkill\.

The agent first solves the task in a no\-skill execution phase\. The benchmark writes a compact file containing the task prompt, grading details, output summaries, and transcript summaries\. The same runtime and model then summarize reusable skills from that first\-run evidence\. A fresh second workspace executes the same task using the summarized skills\.

### 4\.2Metrics

LetSb​\(t\)S\_\{b\}\(t\),Sp​\(t\)S\_\{p\}\(t\), andSq​\(t\)S\_\{q\}\(t\)denote the mean task scores forBaseline,PreSkill, andPostSkill, respectively\. The primary execution\-only ratios against baseline are:

Rp​\(t\)=Sp​\(t\)Sb​\(t\),Rq​\(t\)=Sq​\(t\)Sb​\(t\)\.R\_\{p\}\(t\)=\\frac\{S\_\{p\}\(t\)\}\{S\_\{b\}\(t\)\},\\qquad R\_\{q\}\(t\)=\\frac\{S\_\{q\}\(t\)\}\{S\_\{b\}\(t\)\}\.\(1\)When the baseline score is zero, positive candidate scores are reported as infinite improvement and zero\-to\-zero comparisons are reported as1\.01\.0\.

EvoClawBench reports two metric scopes\.Execution\-onlycompares only the task execution phases: direct baseline execution, preskill reuse execution, and postskill second execution\.End\-to\-endincludes all workflow cost: skill generation forPreSkill, first execution and skill summary forPostSkill, and direct execution forBaseline\.

For each scope, the benchmark reports token, cost, and time usage\. Given baseline usageUbU\_\{b\}and candidate usageUxU\_\{x\}, efficiency gain is computed as:

Ex=UbUx\.E\_\{x\}=\\frac\{U\_\{b\}\}\{U\_\{x\}\}\.\(2\)Values above1\.01\.0therefore indicate that the skill workflow uses fewer resources than the baseline\. The benchmark also reports created\-skill counts, heuristic skill quality, and skill mutation violations\. Execution\-only metrics test whether a generated skill improves the final task attempt, whereas end\-to\-end metrics test whether the full workflow justifies its authoring and summarization overhead\. This distinction matters because a skill may preserve task score yet remain unattractive if it substantially increases token use, cost, or wall time\.

## 5Results

### 5\.1Experimental Setup

We evaluate five models on the hardened benchmark suite under both OpenClaw and nanobot using local subprocess execution\. All runs useuv run scripts/benchmark\.py \-\-mode all \-\-workers 32 \-\-environment local, with\-\-runtime openclawor\-\-runtime nanobot\. Each task is run once per mode\. Unless otherwise stated, results report the execution\-only mean over the benchmark loader’s 101 tasks, which include the 100 official tasks plustask\_00\_sanity\. The numerical results in[Table˜2](https://arxiv.org/html/2607.09711#S5.T2)are computed from the repository\-local benchmark outputs\.

Table 2:Repository\-local EvoClawBench results from current OpenClaw and nanobot three\-mode outputs\. Scores are execution\-only mean task scores in percent over the 101\-task loader suite; Skills listsPreSkill/PostSkillcreated\-skill counts\.
### 5\.2Evaluation Findings

#### Finding 1: Runtime scaffolding strongly changes the execution regime\.

Under OpenClaw, directBaselineperformance remains below 20%, ranging from 18\.63 forGPT\-5\.4to 19\.99 forDeepSeek\-V4\-Pro\. Under nanobot, directBaselinescores are substantially higher, ranging from 56\.45 forQwen3\.6\-Plusto 96\.13 forGPT\-5\.4\. This contrast shows why EvoClawBench reports runtime as part of the experimental condition rather than treating the model identifier alone as sufficient\. It should not be read as a leaderboard claim that one runtime is inherently superior, because the runtime also determines prompt wrapping, workspace setup, tool invocation, and usage extraction\. The hardened tasks are designed to resist shallow copying of exposed answer fields or schemas: agents must reconcile multi\-record evidence, conflicts, revisions, and grader\-visible output constraints\. As a result, a generated skill must improve the specific model\-runtime execution setting rather than merely preserve performance on an already saturated benchmark\.

#### Finding 2: Skill timing is model\-dependent rather than monotonic\.

GPT\-5\.4illustrates the runtime dependence sharply: OpenClaw falls from 18\.63 underBaselineto 1\.14 underPostSkill, while nanobot stays above 96% in all three modes\.Qwen3\.6\-Plusimproves under nanobotPreSkillbut regresses under nanobotPostSkill, whereas its OpenClaw row remains near 19% forBaselineandPostSkill\.GPT\-5\.4 miniimproves under OpenClaw skill workflows and under nanobotPostSkill, but its nanobotPreSkillscore is slightly below its nanobotBaseline\. These rows do not support a blanket claim thatPreSkillconsistently outperformsPostSkill, or that either skill workflow reliably improves over direct execution across runtimes\.

#### Finding 3: Skill workflows still add substantial overhead\.

Even when execution\-only quality is similar, skill workflows include authoring or summarization phases\. ForQwen3\.6\-Plus, end\-to\-end token\-efficiency ratios are 0\.38 forPreSkilland 0\.30 forPostSkill, meaning the full workflows use substantially more tokens than directBaselineexecution\. ForMiniMax\-M2\.7, the corresponding ratios are 0\.21 and 0\.26; forGPT\-5\.4 mini, they are 0\.32 and 0\.28\. ForGPT\-5\.4,PreSkillhas an end\-to\-end token\-efficiency ratio of 0\.36; forDeepSeek\-V4\-Pro, the corresponding ratio is 0\.40\. The current evidence therefore suggests that runtime skill creation must be selective: skill creation needs to clear both a quality bar and an amortized\-cost bar\.

#### Finding 4: Skill counts are not sufficient to predict gains\.

The OpenClaw rows create between 21 and 26PreSkillskills, but theirPreSkillscores range from 15\.06 to 19\.42\. The nanobot rows create many morePreSkillskills, yet the score range remains wide\.PostSkillskill counts are also not enough to explain outcomes: rows with manyPostSkillskills can land below, near, or above their ownBaselinescores\. This suggests that future evaluations should inspect generated skill content and reuse behavior, not only the number of produced skill directories\. Most observed regressions are therefore better explained by the quality and fit of generated skills than by the raw number of skill directories\.[Figure˜4](https://arxiv.org/html/2607.09711#S5.F4)illustrates why this inspection matters forPostSkill\. The summary phase receives compact first\-run evidence rather than hidden state, but that evidence can still make a generated skill too specific to one execution context\. When reused in a fresh workspace, such a skill may preserve an incidental assumption, route attention to the wrong fixture pattern, or skip checks that the second execution still requires\. The qualitative case therefore complements the aggregate skill\-count result: the question is not only whether a skill exists, but whether the procedure it encodes remains valid under reuse\.

![Refer to caption](https://arxiv.org/html/2607.09711v1/x4.png)Figure 4:Qualitative failure mode for experience\-derived skills\. InPostSkill, first\-run evidence can help produce reusable procedural memory, but it can also encode over\-specific assumptions that interfere with the second execution context\.

### 5\.3Subset Cost, Amortization, and Robustness Checks

To address cost and judge\-sensitivity concerns directly, we ran focused subset experiments on the 12 hybrid tasks using OpenClaw andGPT\-5\.4 mini\.[Table˜5](https://arxiv.org/html/2607.09711#A3.T5)shows the end\-to\-end resource tradeoff\. On this subset, both skill workflows slightly reduce score while increasing tokens, cost, and wall\-clock time\. The negative score\-per\-extra\-token values therefore make the cost\-quality tradeoff explicit rather than hiding it behind execution\-only scores\. The subset is not a replacement for the full\-suite table; it is a targeted audit of the hybrid tasks where judge cost and scoring robustness are most salient\.

[Table˜6](https://arxiv.org/html/2607.09711#A3.T6)estimates how many repeated reuses would be needed before the authoring or summary overhead is amortized\. For tokens and provider cost, neitherPreSkillnorPostSkillbreaks even on this subset because the reuse executions are not cheaper than direct execution\. Wall\-clock time can break even after repeated reuse, but only after 14PreSkillexecutions or 9PostSkillexecutions under the observed local run times\. Thus the subset supports a cost\-sensitive interpretation: skills may be worth keeping only when they are reused many times and preserve or improve quality\.

We also ran a 4\-task ablation subset \(task\_02,task\_07,task\_15, andtask\_21\) to separate skill content from reuse scaffolding\.[Table˜7](https://arxiv.org/html/2607.09711#A3.T7)shows that normalPreSkillreuse improves the mean score by 4\.05 points, but an empty\-skill scaffold improves by 19\.84 points, while an irrelevant skill improves by only 1\.52 points\. This pattern is not evidence that empty skills are generally useful; rather, it shows that the reuse prompt, context reset, and run\-to\-run variance can explain some gains that might otherwise be attributed to skill content\. ThePostSkillcomparison is also cautious: normalPostSkillreuse is slightly above baseline, but discarding the generated skill after the summary phase falls below baseline\. Because these ablations are single subset runs rather than repeated\-seed estimates, we use them as diagnostic controls for attribution, not as a new headline result\.

Finally, we regraded the 36 task\-mode outputs from the 12 hybrid\-task subset with alternate judges\. Because hybrid scores combine deterministic checks with an LLM judgment, the regrade holds the automated sub\-scores fixed and replaces only the LLM\-judge component\. One alternate judge shows high agreement with the original scores: Pearson correlation is 0\.9970, mean absolute score delta is 0\.0141, 35 of 36 pairs are within 0\.05, and all 36 pairs are within 0\.10\. The largest disagreement istask\_18\_dep\_auditinPreSkill\(\+0\.0880\)\. As a second check,GPT\-5\.4 minialso preserves the overall trend, though less tightly: Pearson correlation is 0\.9512, mean absolute score delta is 0\.0609, and 30 of 36 pairs are within 0\.10\. The largestGPT\-5\.4 minidisagreement istask\_11\_web\_extractioninBaseline\(\-0\.3972\)\. These subset audits reduce the concern that the reported hybrid\-task pattern is an artifact of a single judge, while the remaining disagreements still motivate auditing high\-stakes leaderboard claims\. They do not remove judge dependence as a limitation; instead, they show how future benchmark releases should report both original and alternate\-judge agreement when hybrid grading affects a result\.

## 6Conclusion

EvoClawBench evaluates the full lifecycle of procedural memory: skill creation, own\-run skill distillation, reuse, and library maintenance\. The current results suggest that agents are not yet reliable at this lifecycle: neither the pre\-executionPreSkillcontrol nor the own\-runPostSkillcondition is uniformly dominant, and[Figure˜4](https://arxiv.org/html/2607.09711#S5.F4)shows how own\-run summaries can overfit a second execution context\. Future skill systems should use selective creation policies and validation before reuse, while benchmark reports should include provenance, mutation checks, and end\-to\-end resources\.

## Limitations

#### Transfer scope\.

The v1 protocol evaluatesPostSkillby rerunning the same task with the same fixtures, which measures within\-task skill distillation but does not yet test transfer to unseen related tasks\. This means the reported results should not be interpreted as evidence that generated skills generalize across new tasks, users, or domains\.

#### Experimental coverage\.

The task suite covers many practical office, data, coding, DevOps, security, and domain\-document workflows, but it is still smaller than large public agent benchmarks\. The current reported table covers two runtimes under one local execution configuration with one run per task and mode\. Repeated seeds, additional models, confidence intervals, Docker resource limits, and more runtime replications remain future work\.

#### Grading dependence\.

Hybrid grading tasks may depend on the selected judge model, although automated checks are used whenever feasible\. Future result files should store the judge model explicitly because absolute scores may change under a different judge even if the task prompts and artifacts are unchanged\.

#### Runtime scope\.

The current implementation and reported experiments cover OpenClaw and nanobot\. Broader runtime coverage is still needed before claiming conclusions across a wider set of agent runtimes\.

## Ethical Considerations

EvoClawBench is intended as a research benchmark for evaluating agent skill authoring, own\-run skill distillation, and reuse, not as a deployment recipe for autonomous skill libraries\. The benchmark uses repository\-local task fixtures and synthetic examples whenever sensitive domains are represented; it does not involve human subjects, crowdworkers, or newly collected personal data\. Several tasks simulate workflows involving privacy, security, legal, healthcare, finance, or public\-sector records, but these are benchmark scenarios for controlled evaluation rather than operational recommendations\. The main risk is dual use: better skill creation from an agent’s own runs could make agents more effective at legitimate repeated work, but could also preserve unsafe procedures, overfit to private context, or automate harmful operational tasks if deployed without review\. The benchmark therefore records skill mutation violations and separates execution\-only quality from end\-to\-end cost, but it does not solve policy questions about which generated skills should be installed, shared, or trusted\. Systems using self\-authored skills should include human review, privacy checks, provenance tracking, and mechanisms for disabling or deleting unsafe skills\.

## Artifacts and Data Statement

The benchmark artifact used for this manuscript is the repository\-localevoclawbench/package\. It contains task definitions undertasks/, input fixtures underassets/, grading and metric code underscripts/, and reported OpenClaw and nanobot result exports underresults/\. The project metadata inpyproject\.tomldeclares the package license as MIT\. For anonymous review, the artifact is available at[https://anonymous\.4open\.science/r/EvoClawBench\-9380/](https://anonymous.4open.science/r/EvoClawBench-9380/)\. The reported task fixtures are benchmark cases rather than newly collected human\-subject data, and the manuscript does not rely on private external datasets\.

## References

- Agent s2: a compositional generalist\-specialist framework for computer use agents\.External Links:2504\.00906,[Link](https://arxiv.org/abs/2504.00906)Cited by:[§1](https://arxiv.org/html/2607.09711#S1.p1.1)\.
- L\. Boisvert, M\. Thakkar, M\. Gasse, M\. Caccia, T\. L\. De Chezelles, Q\. Cappart, N\. Chapados, A\. Lacoste, and A\. Drouin \(2024\)Workarena\+\+: towards compositional planning and reasoning\-based common knowledge work tasks\.Advances in Neural Information Processing Systems37,pp\. 5996–6051\.Cited by:[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px2.p1.1)\.
- J\. S\. Chan, N\. Chowdhury, O\. Jaffe, J\. Aung, D\. Sherburn, E\. Mays, G\. Starace, K\. Liu, L\. Maksin, T\. Patwardhan, L\. Weng, and A\. Mądry \(2025\)MLE\-bench: evaluating machine learning agents on machine learning engineering\.External Links:2410\.07095,[Link](https://arxiv.org/abs/2410.07095)Cited by:[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Deng, Y\. Gu, B\. Zheng, S\. Chen, S\. Stevens, B\. Wang, H\. Sun, and Y\. Su \(2023\)Mind2web: towards a generalist agent for the web\.Advances in Neural Information Processing Systems36,pp\. 28091–28114\.Cited by:[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px2.p1.1)\.
- R\. Fang, Y\. Liang, X\. Wang, J\. Wu, S\. Qiao, P\. Xie, F\. Huang, H\. Chen, and N\. Zhang \(2026\)Memp: exploring agent procedural memory\.External Links:2508\.06433,[Link](https://arxiv.org/abs/2508.06433)Cited by:[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Fourney, G\. Bansal, H\. Mozannar, C\. Tan, E\. Salinas, Erkang, Zhu, F\. Niedtner, G\. Proebsting, G\. Bassman, J\. Gerrits, J\. Alber, P\. Chang, R\. Loynd, R\. West, V\. Dibia, A\. Awadallah, E\. Kamar, R\. Hosn, and S\. Amershi \(2024\)Magentic\-one: a generalist multi\-agent system for solving complex tasks\.External Links:2411\.04468,[Link](https://arxiv.org/abs/2411.04468)Cited by:[§1](https://arxiv.org/html/2607.09711#S1.p1.1)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan \(2024\)SWE\-bench: can language models resolve real\-world github issues?\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 54107–54157\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/edac78c3e300629acfe6cbe9ca88fb84-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2607.09711#S1.p1.1),[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Li, W\. Chen, Y\. Liu, S\. Zheng, X\. Chen, Y\. He, Y\. Li, B\. You, H\. Shen, J\. Sun, S\. Wang, B\. Li, Q\. Zeng, D\. Wang, X\. Zhao, Y\. Wang, R\. B\. Chaim, Z\. Di, Y\. Gao, J\. He, Y\. He, L\. Jing, L\. Kong, X\. Lan, J\. Li, S\. Li, Y\. Li, Y\. Lin, X\. Liu, X\. Liu, H\. Lyu, Z\. Ma, B\. Wang, R\. Wang, T\. Wang, W\. Ye, Y\. Zhang, H\. Xing, Y\. Xue, S\. Dillmann, and H\. Lee \(2026\)SkillsBench: benchmarking how well agent skills work across diverse tasks\.External Links:2602\.12670,[Link](https://arxiv.org/abs/2602.12670)Cited by:[§1](https://arxiv.org/html/2607.09711#S1.p1.1),[§1](https://arxiv.org/html/2607.09711#S1.p2.1),[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px3.p1.1)\.
- X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang,et al\.\(2024\)Agentbench: evaluating llms as agents\.2024,pp\. 52989–53046\.Cited by:[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px1.p1.1)\.
- M\. A\. Merrill, A\. G\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi, J\. Y\. Shin, T\. Walshe, E\. K\. Buchanan, J\. Shen, G\. Ye, H\. Lin, J\. Poulos, M\. Wang, M\. Nezhurina, J\. Jitsev, D\. Lu, O\. M\. Mastromichalakis, Z\. Xu, Z\. Chen, Y\. Liu, R\. Zhang, L\. L\. Chen, A\. Kashyap, J\. Uslu, J\. Li, J\. Wu, M\. Yan, S\. Bian, V\. Sharma, K\. Sun, S\. Dillmann, A\. Anand, A\. Lanpouthakoun, B\. Koopah, C\. Hu, E\. Guha, G\. H\. S\. Dreiman, J\. Zhu, K\. Krauth, L\. Zhong, N\. Muennighoff, R\. Amanfu, S\. Tan, S\. Pimpalgaonkar, T\. Aggarwal, X\. Lin, X\. Lan, X\. Zhao, Y\. Liang, Y\. Wang, Z\. Wang, C\. Zhou, D\. Heineman, H\. Liu, H\. Trivedi, J\. Yang, J\. Lin, M\. Shetty, M\. Yang, N\. Omi, N\. Raoof, S\. Li, T\. Y\. Zhuo, W\. Lin, Y\. Dai, Y\. Wang, W\. Chai, S\. Zhou, D\. Wahdany, Z\. She, J\. Hu, Z\. Dong, Y\. Zhu, S\. Cui, A\. Saiyed, A\. Kolbeinsson, J\. Hu, C\. M\. Rytting, R\. Marten, Y\. Wang, A\. Dimakis, A\. Konwinski, and L\. Schmidt \(2026\)Terminal\-bench: benchmarking agents on hard, realistic tasks in command line interfaces\.External Links:2601\.11868,[Link](https://arxiv.org/abs/2601.11868)Cited by:[§1](https://arxiv.org/html/2607.09711#S1.p1.1),[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px1.p1.1)\.
- G\. Mialon, C\. Fourrier, T\. Wolf, Y\. LeCun, and T\. Scialom \(2024\)Gaia: a benchmark for general ai assistants\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 9025–9049\.Cited by:[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Ni, Y\. Liu, X\. Liu, Y\. Sun, M\. Zhou, P\. Cheng, D\. Wang, E\. Zhao, X\. Jiang, and G\. Jiang \(2026\)Trace2Skill: distill trajectory\-local lessons into transferable agent skills\.External Links:2603\.25158,[Link](https://arxiv.org/abs/2603.25158)Cited by:[§1](https://arxiv.org/html/2607.09711#S1.p2.1)\.
- Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian, S\. Zhao, L\. Hong, R\. Tian, R\. Xie, J\. Zhou, M\. Gerstein, D\. Li, Z\. Liu, and M\. Sun \(2023\)ToolLLM: facilitating large language models to master 16000\+ real\-world apis\.External Links:2307\.16789,[Link](https://arxiv.org/abs/2307.16789)Cited by:[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Research, A\. Chan, A\. Shalaby, A\. Wettig, A\. Sanger, A\. Zhai, A\. Ajay, A\. Nair, C\. Snell, C\. Lu,et al\.\(2026\)Composer 2 technical report\.arXiv preprint arXiv:2603\.24477\.Cited by:[§1](https://arxiv.org/html/2607.09711#S1.p1.1)\.
- N\. Shinn, F\. Cassano, E\. Berman, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.External Links:2303\.11366,[Link](https://arxiv.org/abs/2303.11366)Cited by:[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px3.p1.1)\.
- H\. Trivedi, T\. Khot, M\. Hartmann, R\. Manku, V\. Dong, E\. Li, S\. Gupta, A\. Sabharwal, and N\. Balasubramanian \(2024\)Appworld: a controllable world of apps and people for benchmarking interactive coding agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 16022–16076\.Cited by:[§1](https://arxiv.org/html/2607.09711#S1.p1.1),[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Wang, Z\. Yu, X\. Xie, W\. Yao, R\. Fang, S\. Qiao, K\. Cao, G\. Zheng, X\. Qi, P\. Zhang, and S\. Deng \(2026\)SkillX: automatically constructing skill knowledge bases for agents\.External Links:2604\.04804,[Link](https://arxiv.org/abs/2604.04804)Cited by:[§1](https://arxiv.org/html/2607.09711#S1.p2.1)\.
- G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2023\)Voyager: an open\-ended embodied agent with large language models\.External Links:2305\.16291,[Link](https://arxiv.org/abs/2305.16291)Cited by:[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px3.p1.1)\.
- X\. Wang, B\. Li, Y\. Song, F\. F\. Xu, X\. Tang, M\. Zhuge, J\. Pan, Y\. Song, B\. Li, J\. Singh, H\. Tran, F\. Li, R\. Ma, M\. Zheng, B\. Qian, D\. Shao, N\. Muennighoff, Y\. Zhang, B\. Hui, J\. Lin, R\. Brennan, H\. Peng, H\. Ji, and G\. Neubig \(2025\)OpenHands: an open platform for AI software developers as generalist agents\.InInternational Conference on Learning Representations,Y\. Yue, A\. Garg, N\. Peng, F\. Sha, and R\. Yu \(Eds\.\),Vol\.2025,pp\. 65882–65919\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/a4b6ad6b48850c0c331d1259fc66a69c-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2607.09711#S1.p1.1)\.
- Y\. Wu and Y\. Zhang \(2026\)Agent skills from the perspective of procedural memory: a survey\.Authorea Preprints\.Cited by:[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px3.p1.1)\.
- H\. Xie, X\. Wang, Y\. Wang, P\. Zhao, and F\. Ju \(2026\)From history to state: constant\-context skill learning for llm agents\.External Links:2605\.05413,[Link](https://arxiv.org/abs/2605.05413)Cited by:[§1](https://arxiv.org/html/2607.09711#S1.p2.1)\.
- T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng, D\. Shin, F\. Lei, Y\. Liu, Y\. Xu, S\. Zhou, S\. Savarese, C\. Xiong, V\. Zhong, and T\. Yu \(2024\)OSWorld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.External Links:2404\.07972,[Link](https://arxiv.org/abs/2404.07972)Cited by:[§1](https://arxiv.org/html/2607.09711#S1.p1.1),[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px2.p1.1)\.
- F\. \(\. Xu, Y\. Song, B\. Li, Y\. Tang, K\. Jain, M\. Bao, Z\. Wang, X\. Zhou, Z\. Guo, M\. Cao, M\. Yang, H\. Y\. Lu, A\. Martin, Z\. Su, L\. Maben, R\. Mehta, W\. Chi, L\. Jang, Y\. Xie, S\. Zhou, and G\. Neubig \(2025\)TheAgentCompany: benchmarking LLM agents on consequential real world tasks\.Vol\.38,Curran Associates, Inc\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/0d744742f6fac4d1134c019b7cef3c8a-Paper-Datasets_and_Benchmarks_Track.pdf)Cited by:[§1](https://arxiv.org/html/2607.09711#S1.p1.1)\.
- R\. Xu and Y\. Yan \(2026\)Agent skills for large language models: architecture, acquisition, security, and the path forward\.External Links:2602\.12430,[Link](https://arxiv.org/abs/2602.12430)Cited by:[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press \(2024\)Swe\-agent: agent\-computer interfaces enable automated software engineering\.Advances in Neural Information Processing Systems37,pp\. 50528–50652\.Cited by:[§1](https://arxiv.org/html/2607.09711#S1.p1.1),[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Yang, J\. Piao, X\. Xia, X\. Lan, J\. Chen, Y\. Gong, and Y\. Li \(2026a\)SkillMaster: toward autonomous skill mastery in llm agents\.External Links:2605\.08693,[Link](https://arxiv.org/abs/2605.08693)Cited by:[§1](https://arxiv.org/html/2607.09711#S1.p2.1)\.
- Y\. Yang, J\. Li, Q\. Pan, B\. Zhan, Y\. Cai, L\. Du, J\. Zhou, K\. Chen, Q\. Chen, X\. Li, B\. Zhang, and L\. He \(2026b\)AutoSkill: experience\-driven lifelong learning via skill self\-evolution\.External Links:2603\.01145,[Link](https://arxiv.org/abs/2603.01145)Cited by:[§1](https://arxiv.org/html/2607.09711#S1.p2.1)\.
- S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhan \(2024\)τ\\tau\-Bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.Cited by:[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.External Links:2210\.03629,[Link](https://arxiv.org/abs/2210.03629)Cited by:[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px3.p1.1)\.
- B\. Zhang, K\. Lazuka, and M\. Murag \(2025\)Equipping agents for the real world with agent skills\.Cited by:[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Zhao, D\. Huang, Q\. Xu, M\. Lin, Y\. Liu, and G\. Huang \(2024\)ExpeL: llm agents are experiential learners\.External Links:ISBN 978\-1\-57735\-887\-9,[Link](https://doi.org/10.1609/aaai.v38i17.29936),[Document](https://dx.doi.org/10.1609/aaai.v38i17.29936)Cited by:[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried,et al\.\(2024\)Webarena: a realistic web environment for building autonomous agents\.2024,pp\. 15585–15606\.Cited by:[§1](https://arxiv.org/html/2607.09711#S1.p1.1),[§2](https://arxiv.org/html/2607.09711#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix AMode Prompt Prefixes

EvoClawBench controls each execution condition by prepending a mode\-specific prefix to the task prompt loaded from the task markdown file\. The prefix text below reproduces the benchmark runtime helper wording; trailing blank lines are omitted and long lines are wrapped by LaTeX for typesetting\. The assembled prompt is always the selected prefix followed by the task prompt; task prompts themselves remain stored inevoclawbench/tasks/task\_\*\.md\.

Baseline execution prefixBASELINE MODE:Complete the task directly\. You must NOT create, edit, or delete any skills or SKILL\.md files\. Solve each sub\-problem independently from scratch, and focus on producing the required grader\-visible outputs\.

Preskill authoring prefixPRESKILL AUTHOR MODE:Before solving the task, create a task\-specific reusable skill\. Read ‘skills/skill\-creator/SKILL\.md‘ and write one or more new skills under ‘skills/<skill\-name\>/SKILL\.md‘ that would help an agent solve this task later\. Do not produce the final task outputs in ‘outputs/‘; this phase is only for skill authoring\. Do not replace or delete ‘skills/skill\-creator‘\.

Postskill summary prefixPOSTSKILL SUMMARY MODE:Summarize a reusable task\-specific skill from a completed first run\. Read ‘\.evoclawbench/first\_run\_context\.json‘ for the first run prompt, grading details, workspace output summary, and transcript summary\. Create one or more skills under ‘skills/<skill\-name\>/SKILL\.md‘ so a later agent can solve the same task more accurately and efficiently\. Do not redo the task and do not write final task outputs in ‘outputs/‘\. Do not replace or delete ‘skills/skill\-creator‘\.

Skill reuse execution prefixSKILL REUSE EXECUTION MODE:Complete the task using the existing skills in ‘skills/‘ when helpful\. You must NOT create, edit, or delete any skills or SKILL\.md files during this execution phase\. Focus on producing the required grader\-visible outputs and validate them before finishing\.

Table 3:Prompt assembly by benchmark phase\.

## Appendix BPostSkill First\-Run Context

InPostSkill, the first execution is graded before any summary skill is written\. The benchmark then writes\.evoclawbench/first\_run\_context\.jsoninto the first workspace and copies that file into the summary workspace\. This prevents the summary phase from relying on hidden state while still giving it compact evidence from the completed attempt\.

Table 4:First\-run context fields serialized for the summary phase\.
The summary phase receives the same task prompt after thePostSkillsummary prefix, but it is instructed not to redo the task and not to write final task outputs underoutputs/\. Skills produced in the summary workspace are copied into a fresh second execution workspace, where mutation checks are applied before and after execution\.

## Appendix CSubset Audit Details

The following tables provide the detailed subset evidence summarized in[Section˜5\.3](https://arxiv.org/html/2607.09711#S5.SS3)\.

Table 5:End\-to\-end cost on the 12\-task hybrid subset\. Scores are percentages\. “Delta/1M tok\.” is score\-point change per one million extra tokens relative toBaseline\.
Table 6:Amortized break\-even reuse count on the same subset\. “Never” means the reuse execution is not cheaper than directBaselineexecution for that resource\.
Table 7:Skill\-content ablation on a 4\-task representative subset\. Scores and deltas are percentages; resources are end\-to\-end for each variant\.

## Appendix DOfficial Task Inventory

The loader currently finds 101 task files\. The official benchmark size reported in the paper excludestask\_00\_sanity, leaving 100 tasks and 502 sub\-problems\. The table uses ASCII\-safe names because the review build usespdflatexwithout CJK font configuration\.

Table 8:Official task inventory, excludingtask\_00\_sanity\.TaskNameFamily/categoryGradeSub\.FilesExt\.task\_01Batch Data TransformSeed / data transformationautomated66csv, json, tsv, xml, yamltask\_02Log AnalysisSeed / data analysishybrid55logtask\_03API Integration ScaffoldSeed / code generationautomated55jsontask\_04Test GenerationSeed / testingautomated66pytask\_05Config Migration v1 to v2Seed / data transformationautomated55jsontask\_06Security Code ReviewSeed / code analysishybrid55go, js, pytask\_07Document Data ExtractionSeed / data extractionhybrid55txttask\_08Database Schema OperationsSeed / databaseautomated55sqltask\_09Excel Analytics Report GenerationSeed / data transformationautomated55csvtask\_10Git Repository History AnalysisSeed / data analysisautomated55shtask\_11Web Page Structured Data ExtractionSeed / data extractionhybrid55htmltask\_12Word Document GenerationSeed / code generationautomated55jsontask\_13Multi\-Dataset Statistical Analysis PipelineSeed / data analysishybrid515csvtask\_14Enterprise Email Thread AnalysisSeed / office automationhybrid55jsontask\_15Shell Automation Script GenerationSeed / automationhybrid55yamltask\_16Dockerfile and CI Pipeline GenerationSeed / devopshybrid55jsontask\_17Enterprise Invoice Expense ProcessingSeed / office dailyhybrid55csv, json, tsv, txt, xmltask\_18Dependency Security AuditSeed / securityhybrid55json, mod, none, txt, xmltask\_19Meeting Notes Structured ExtractionSeed / office dailyhybrid55txttask\_20Multi\-Environment Config GenerationSeed / devops automationautomated55jsontask\_21System Metrics Anomaly DetectionSeed / data analysishybrid55csvtask\_22Finance Ledger ReconciliationFinanceautomated55csvtask\_23Subscription Revenue AuditFinanceautomated55csvtask\_24Bank KYC Packet ReviewFinanceautomated55csvtask\_25Loan Document ChecklistFinanceautomated55csvtask\_26Tax Form ConsistencyFinanceautomated55csvtask\_27Contract Clause ExtractionLegalautomated55txttask\_28Discovery Document TaggingLegalautomated55txttask\_29Policy Exception MappingLegalautomated55txttask\_30Nda Obligation ReviewLegalautomated55txttask\_31Vendor Contract RiskLegalautomated55txttask\_32Appointment Referral TriageHealthcareautomated55jsontask\_33Pharmacy Inventory ReorderHealthcareautomated55jsontask\_34Clinical Trial EligibilityHealthcareautomated55jsontask\_35Insurance Prior Auth ReviewHealthcareautomated55jsontask\_36Lab Result Followup QueueHealthcareautomated55jsontask\_37Procurement Bid ScoringProcurement/Logisticsautomated55yamltask\_38Purchase Order Three Way MatchProcurement/Logisticsautomated55yamltask\_39Shipping Exception ResolutionProcurement/Logisticsautomated55yamltask\_40Warehouse Picklist ValidationProcurement/Logisticsautomated55yamltask\_41Supplier Risk PacketProcurement/Logisticsautomated55yamltask\_42Employee Onboarding ChecklistHR/Educationautomated55jsontask\_43Interview Feedback CalibrationHR/Educationautomated55jsontask\_44Admissions Packet ScreeningHR/Educationautomated55jsontask\_45Assignment Rubric GradingHR/Educationautomated55jsontask\_46Training Completion AuditHR/Educationautomated55jsontask\_47Support Ticket EscalationSupport/Productautomated55jsontask\_48Kb Gap AnalysisSupport/Productautomated55jsontask\_49App Review Theme MiningSupport/Productautomated55jsontask\_50Bug Report DeduplicationSupport/Productautomated55jsontask\_51Call Center Quality ReviewSupport/Productautomated55jsontask\_52Kubernetes Policy ReviewDevOps/SREautomated55yamltask\_53Terraform Plan DriftDevOps/SREautomated55yamltask\_54SLO Burn Rate AnalysisDevOps/SREautomated55yamltask\_55CI Pipeline HardeningDevOps/SREautomated55yamltask\_56Backup Restore DrillDevOps/SREautomated55yamltask\_57Security Alert CorrelationSecurity/Privacyautomated55jsontask\_58Phishing Report TriageSecurity/Privacyautomated55jsontask\_59Vulnerability Exception ReviewSecurity/Privacyautomated55jsontask\_60PII Redaction ReleaseSecurity/Privacyautomated55jsontask\_61DSR Request RoutingSecurity/Privacyautomated55jsontask\_62SQL Schema Migration ReviewData/Analyticsautomated55csvtask\_63Messy Csv NormalizationData/Analyticsautomated55csvtask\_64Dashboard Metric ReconciliationData/Analyticsautomated55csvtask\_65Data Quality Rule AuthoringData/Analyticsautomated55csvtask\_66Experiment Results AnalysisData/Analyticsautomated55csvtask\_67Research Claim EvidenceResearch/Mediaautomated55htmltask\_68Newsroom Fact CheckResearch/Mediaautomated55htmltask\_69Literature Table ExtractionResearch/Mediaautomated55htmltask\_70Public Web Directory ExtractionResearch/Mediaautomated55htmltask\_71Citation DeduplicationResearch/Mediaautomated55htmltask\_72Localization Placeholder QaLocalization/Releaseautomated55jsontask\_73Glossary Compliance ReviewLocalization/Releaseautomated55jsontask\_74Release Notes CurationLocalization/Releaseautomated55jsontask\_75Changelog Impact MatrixLocalization/Releaseautomated55jsontask\_76Feature Flag Cleanup PlanLocalization/Releaseautomated55jsontask\_77Ecommerce Catalog NormalizationCommerce/Foodautomated55csvtask\_78Retail Returns Root CauseCommerce/Foodautomated55csvtask\_79Restaurant Inspection SummaryCommerce/Foodautomated55csvtask\_80Food Delivery Refund ReviewCommerce/Foodautomated55csvtask\_81Marketplace Listing PolicyCommerce/Foodautomated55csvtask\_82Facilities Maintenance PrioritizationFacilities/IoTautomated55jsontask\_83Fleet Maintenance ScheduleFacilities/IoTautomated55jsontask\_84Iot Sensor Anomaly ReviewFacilities/IoTautomated55jsontask\_85Smart Home Support DiagnosisFacilities/IoTautomated55jsontask\_86Robotics Workcell Event ReportFacilities/IoTautomated55jsontask\_87Civic Service Request RoutingPublic Sector/Auditautomated55jsontask\_88Public Records RedactionPublic Sector/Auditautomated55jsontask\_89Grant Application CompletenessPublic Sector/Auditautomated55jsontask\_90Audit Evidence CollectionPublic Sector/Auditautomated55jsontask\_91Risk Register RollupPublic Sector/Auditautomated55jsontask\_92CRM Pipeline HygieneCRM/Executiveautomated55csvtask\_93Sales Forecast VarianceCRM/Executiveautomated55csvtask\_94Marketing Campaign QaCRM/Executiveautomated55csvtask\_95Board Packet PreparationCRM/Executiveautomated55csvtask\_96Executive Action Item TrackingCRM/Executiveautomated55csvtask\_97Calendar Conflict ResolutionTravel/Officeautomated55jsontask\_98Travel Itinerary ExceptionTravel/Officeautomated55jsontask\_99Meeting Notes Action ItemsTravel/Officeautomated55jsontask\_100Inbox Rules ClassificationTravel/Officeautomated55json
## Appendix EPer\-Task Prompt and Output Surfaces

The following table expands the inventory with the prompt focus and grader\-visible output surface for every official task\. It is generated from the task markdown files and is intended to make the 100\-task composition auditable without requiring reviewers to open each fixture directory\.

Table 9:Per\-task prompt and output surfaces\.TaskNamePrompt focusGrader\-visible output surfaceCheckstask\_01Batch Data TransformYou have 6 data files in artifact that each contain user records in different formats\.outputs/users\_01\.json, outputs/users\_02\.json, outputs/users\_03\.json …automated; 6 criteriatask\_02Log AnalysisYou have 5 log files in artifact from different microservices\.outputs/api\_gateway\_report\.json, outputs/auth\_service\_report\.json, outputs/notification\_service\_report\.json …hybrid; 6 criteriatask\_03API Integration ScaffoldYou have 5 API specification files in artifact\.outputs/analytics\_client\.py, outputs/inventory\_client\.py, outputs/notifications\_client\.py …automated; 6 criteriatask\_04Test GenerationYou have 6 Python utility modules in artifact\.outputs/test\_data\_utils\.py, outputs/test\_date\_utils\.py, outputs/test\_file\_utils\.py …automated; 6 criteriatask\_05Config Migration v1 to v2You have 5 application configuration files in artifact that use a legacy flat key format \(v1\)\.outputs/gateway\_v2\.json, outputs/mailer\_v2\.json, outputs/scheduler\_v2\.json …automated; 7 criteriatask\_06Security Code ReviewYou have 5 source code files in artifact that contain intentional security vulnerabilities\.outputs/admin\_panel\_review\.json, outputs/api\_handler\_review\.json, outputs/file\_manager\_review\.json …hybrid; 7 criteriatask\_07Document Data ExtractionYou have 5 text documents in artifact representing different document types\.outputs/contract\_001\.json, outputs/expense\_report\_001\.json, outputs/invoice\_XX\.json …hybrid; 8 criteriatask\_08Database Schema OperationsYou have 5 SQL schema files in artifact representing existing PostgreSQL tables\.outputs/migrate\_orders\.sql, outputs/migrate\_products\.sql, outputs/migrate\_reviews\.sql …automated; 7 criteriatask\_09Excel Analytics Report GenerationYou are given 5 regional sales CSV files in artifact\.outputs/report\_central\.xlsx, outputs/report\_east\.xlsx, outputs/report\_north\.xlsx …automated; 7 criteriatask\_10Git Repository History AnalysisYou are given 5 bash setup scripts in artifact\.task\-specific grader\-visible files under outputs/automated; 6 criteriatask\_11Web Page Structured Data ExtractionYou are given 5 HTML files in artifact that simulate web pages\.task\-specific grader\-visible files under outputs/hybrid; 7 criteriatask\_12Word Document GenerationYou are given 5 JSON data files in artifact\.task\-specific grader\-visible files under outputs/automated; 6 criteriatask\_13Multi\-Dataset Statistical Analysis PipelineYou are given 5 groups of related CSV files in artifact\.task\-specific grader\-visible files under outputs/hybrid; 6 criteriatask\_14Enterprise Email Thread AnalysisComplete enterprise email thread analysis and write grader\-visible outputs\.outputs/thread\_XX\_report\.jsonhybrid; 7 criteriatask\_15Shell Automation Script GenerationYou have 5 automation task specification files in artifact\.outputs/task\_XX\_db\_dump\.sh, outputs/task\_XX\_dir\_monitor\.sh, outputs/task\_XX\_file\_backup\.sh …hybrid; 9 criteriatask\_16Dockerfile and CI Pipeline GenerationYou have 5 application specification files in artifact\.outputs/app\_XX/Dockerfilehybrid; 8 criteriatask\_17Enterprise Invoice Expense ProcessingComplete enterprise invoice expense processing and write grader\-visible outputs\.outputs/invoice\_XX\_parsed\.jsonhybrid; 8 criteriatask\_18Dependency Security AuditYou have 5 dependency manifest files in artifact from different application stacks\.outputs/gemfile\_audit\.json, outputs/go\_audit\.json, outputs/package\_audit\.json …hybrid; 8 criteriatask\_19Meeting Notes Structured ExtractionComplete meeting notes structured extraction and write grader\-visible outputs\.outputs/meeting\_XX\_minutes\.jsonhybrid; 8 criteriatask\_20Multi\-Environment Config GenerationYou have 5 application specification files in artifact\.outputs/data\-exporter/config\.dev\.yml, outputs/ml\-inference/config\.dev\.yml, outputs/notification\-service/config\.dev\.yml …automated; 8 criteriatask\_21System Metrics Anomaly DetectionYou have 5 system metrics CSV files in artifact\.outputs/db\_query\_perf\_report\.json, outputs/disk\_io\_report\.json, outputs/network\_traffic\_report\.json …hybrid; 8 criteriatask\_22Finance Ledger ReconciliationSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_23Subscription Revenue AuditSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_24Bank KYC Packet ReviewSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_25Loan Document ChecklistSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_26Tax Form ConsistencySelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_27Contract Clause ExtractionSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_28Discovery Document TaggingSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_29Policy Exception MappingSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_30Nda Obligation ReviewSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_31Vendor Contract RiskSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_32Appointment Referral TriageSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_33Pharmacy Inventory ReorderSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_34Clinical Trial EligibilitySelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_35Insurance Prior Auth ReviewSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_36Lab Result Followup QueueSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_37Procurement Bid ScoringSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_38Purchase Order Three Way MatchSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_39Shipping Exception ResolutionSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_40Warehouse Picklist ValidationSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_41Supplier Risk PacketSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_42Employee Onboarding ChecklistSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_43Interview Feedback CalibrationSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_44Admissions Packet ScreeningSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_45Assignment Rubric GradingSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_46Training Completion AuditSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_47Support Ticket EscalationSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_48Kb Gap AnalysisSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_49App Review Theme MiningSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_50Bug Report DeduplicationSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_51Call Center Quality ReviewSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_52Kubernetes Policy ReviewSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_53Terraform Plan DriftSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_54SLO Burn Rate AnalysisSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_55CI Pipeline HardeningSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_56Backup Restore DrillSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_57Security Alert CorrelationSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_58Phishing Report TriageSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_59Vulnerability Exception ReviewSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_60PII Redaction ReleaseSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_61DSR Request RoutingSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_62SQL Schema Migration ReviewSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_63Messy Csv NormalizationSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_64Dashboard Metric ReconciliationSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_65Data Quality Rule AuthoringSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_66Experiment Results AnalysisSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_67Research Claim EvidenceSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_68Newsroom Fact CheckSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_69Literature Table ExtractionSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_70Public Web Directory ExtractionSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_71Citation DeduplicationSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_72Localization Placeholder QaSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_73Glossary Compliance ReviewSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_74Release Notes CurationSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_75Changelog Impact MatrixSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_76Feature Flag Cleanup PlanSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_77Ecommerce Catalog NormalizationSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_78Retail Returns Root CauseSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_79Restaurant Inspection SummarySelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_80Food Delivery Refund ReviewSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_81Marketplace Listing PolicySelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_82Facilities Maintenance PrioritizationSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_83Fleet Maintenance ScheduleSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_84Iot Sensor Anomaly ReviewSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_85Smart Home Support DiagnosisSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_86Robotics Workcell Event ReportSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_87Civic Service Request RoutingSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_88Public Records RedactionSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_89Grant Application CompletenessSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_90Audit Evidence CollectionSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_91Risk Register RollupSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_92CRM Pipeline HygieneSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_93Sales Forecast VarianceSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_94Marketing Campaign QaSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_95Board Packet PreparationSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_96Executive Action Item TrackingSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_97Calendar Conflict ResolutionSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_98Travel Itinerary ExceptionSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_99Meeting Notes Action ItemsSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteriatask\_100Inbox Rules ClassificationSelect the authoritative evidence packet, discard decoys, derive strict JSON case reports\.outputs/case\_XX\_report\.jsonautomated; 8 criteria
## Appendix FPer\-Task Grading Surfaces

This table records how each official task is checked\. It separates the grading type from the concrete surface that the grader inspects, which is useful because hybrid tasks still require file artifacts and automated tasks may include rich schema or content checks\.

Table 10:Per\-task grading surfaces\.TaskNameGradeCheck surfaceRepresentative criteriatask\_01Batch Data Transformautomatedautomated codeEach of the 6 output files exists in artifact; Each output file contains valid JSON; Each record conforms to the target schema \(all 6 fields present\)task\_02Log Analysishybridautomated code, LLM rubric, weighted blendEach of the 5 output report files exists; Each report contains valid JSON; Each report has all required top\-level fieldstask\_03API Integration Scaffoldautomatedautomated codeEach of the 5 output \.py files exists; Each file contains valid Python \(parseable by ast\.parse\); Each file contains credential\-handling codetask\_04Test Generationautomatedautomated codeEach of the 6 test files exists in artifact; Each file contains valid Python syntax; Each file imports pytesttask\_05Config Migration v1 to v2automatedautomated codeAll 5 output files exist in outputs/; All output files are valid JSON; All output files use nested structure \(no flat dot\-notation keys\)task\_06Security Code Reviewhybridautomated code, LLM rubric, weighted blendAll 5 output files exist in outputs/; All output files are valid JSON; Each report has a "vulnerabilities" array with at least 3 entriestask\_07Document Data Extractionhybridautomated code, LLM rubric, weighted blendAll 5 output files exist in outputs/; All output files are valid JSON; Invoice has line\_items array and grand\_totaltask\_08Database Schema Operationsautomatedautomated codeAll 5 output files exist in outputs/; All files contain valid SQL statements; Each migration contains the required ALTER TABLE statementstask\_09Excel Analytics Report Generationautomatedautomated codeOutput \.xlsx files exist for all 5 regions; Each workbook has exactly 3 sheets named correctly; Raw Data sheet contains the expected number of rows \(20 data rows \+ header\)task\_10Git Repository History Analysisautomatedautomated codeOutput JSON files exist for all 5 repositories; artifact matches expected value exactly; All expected contributors appear in the contributors listtask\_11Web Page Structured Data Extractionhybridautomated code, LLM rubric, weighted blendAll 5 output JSON files exist; Each file contains a list with at least 5 items; Product catalog: prices are numeric, product IDs match sourcetask\_12Word Document Generationautomatedautomated codeAll 5 \.docx files exist in outputs/; Files are valid Word documents \(openable by python\-docx\); Each document contains the correct title as a Heading 1task\_13Multi\-Dataset Statistical Analysis Pipelinehybridautomated code, LLM rubric, weighted blendAll 5 analysis JSON files exist; artifact / artifact / artifact / artifact match expected counts; Numeric statistics are present and of correct typetask\_14Enterprise Email Thread Analysishybridautomated code, LLM rubric, weighted blendAll 5 output files exist; each report is valid JSON; each report contains all required top\-level fieldstask\_15Shell Automation Script Generationhybridautomated code, LLM rubric, weighted blendAll 10 output files exist \(5 scripts \+ 5 READMEs\); Each artifact file starts with artifact or artifact; Each artifact file contains artifact or artifacttask\_16Dockerfile and CI Pipeline Generationhybridautomated code, LLM rubric, weighted blendAll 10 output files exist \(5 Dockerfiles \+ 5 workflow YAMLs\); Each Dockerfile starts with artifact; Multi\-stage Dockerfiles \(apps 01–04\) contain at least 2 artifact instructionstask\_17Enterprise Invoice Expense Processinghybridautomated code, LLM rubric, weighted blendAll 6 output files exist \(5 parsed files plus 1 summary\); each parsed file is valid JSON; parsed files contain all required top\-level fieldstask\_18Dependency Security Audithybridautomated code, LLM rubric, weighted blendAll 5 output files exist; Each report contains valid JSON; Each report has all required top\-level fieldstask\_19Meeting Notes Structured Extractionhybridautomated code, LLM rubric, weighted blendAll 5 output files exist; each file is valid JSON; each file contains all required top\-level fieldstask\_20Multi\-Environment Config Generationautomatedautomated codeAll 15 output YAML files exist; Each YAML file is parseable \(valid YAML\); artifact has artifacttask\_21System Metrics Anomaly Detectionhybridautomated code, LLM rubric, weighted blendAll 5 output report files exist; Each report contains valid JSON; Each report has all required top\-level fieldstask\_22Finance Ledger Reconciliationautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_23Subscription Revenue Auditautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_24Bank KYC Packet Reviewautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_25Loan Document Checklistautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_26Tax Form Consistencyautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_27Contract Clause Extractionautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_28Discovery Document Taggingautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_29Policy Exception Mappingautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_30Nda Obligation Reviewautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_31Vendor Contract Riskautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_32Appointment Referral Triageautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_33Pharmacy Inventory Reorderautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_34Clinical Trial Eligibilityautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_35Insurance Prior Auth Reviewautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_36Lab Result Followup Queueautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_37Procurement Bid Scoringautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_38Purchase Order Three Way Matchautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_39Shipping Exception Resolutionautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_40Warehouse Picklist Validationautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_41Supplier Risk Packetautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_42Employee Onboarding Checklistautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_43Interview Feedback Calibrationautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_44Admissions Packet Screeningautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_45Assignment Rubric Gradingautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_46Training Completion Auditautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_47Support Ticket Escalationautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_48Kb Gap Analysisautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_49App Review Theme Miningautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_50Bug Report Deduplicationautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_51Call Center Quality Reviewautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_52Kubernetes Policy Reviewautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_53Terraform Plan Driftautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_54SLO Burn Rate Analysisautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_55CI Pipeline Hardeningautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_56Backup Restore Drillautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_57Security Alert Correlationautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_58Phishing Report Triageautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_59Vulnerability Exception Reviewautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_60PII Redaction Releaseautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_61DSR Request Routingautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_62SQL Schema Migration Reviewautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_63Messy Csv Normalizationautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_64Dashboard Metric Reconciliationautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_65Data Quality Rule Authoringautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_66Experiment Results Analysisautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_67Research Claim Evidenceautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_68Newsroom Fact Checkautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_69Literature Table Extractionautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_70Public Web Directory Extractionautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_71Citation Deduplicationautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_72Localization Placeholder Qaautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_73Glossary Compliance Reviewautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_74Release Notes Curationautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_75Changelog Impact Matrixautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_76Feature Flag Cleanup Planautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_77Ecommerce Catalog Normalizationautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_78Retail Returns Root Causeautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_79Restaurant Inspection Summaryautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_80Food Delivery Refund Reviewautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_81Marketplace Listing Policyautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_82Facilities Maintenance Prioritizationautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_83Fleet Maintenance Scheduleautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_84Iot Sensor Anomaly Reviewautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_85Smart Home Support Diagnosisautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_86Robotics Workcell Event Reportautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_87Civic Service Request Routingautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_88Public Records Redactionautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_89Grant Application Completenessautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_90Audit Evidence Collectionautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_91Risk Register Rollupautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_92CRM Pipeline Hygieneautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_93Sales Forecast Varianceautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_94Marketing Campaign Qaautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_95Board Packet Preparationautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_96Executive Action Item Trackingautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_97Calendar Conflict Resolutionautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_98Travel Itinerary Exceptionautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_99Meeting Notes Action Itemsautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.task\_100Inbox Rules Classificationautomatedautomated codeAll five artifact files exist\.; Each report is valid JSON and contains every required field\.; The report ignores draft, superseded, invalid\-checksum, and decoy packet records\.
## Appendix GTask Families and Fixture Mechanics

The current suite combines the original heterogeneous seed tasks with generated hard\-mode families\. The generated families share a common evidence\-packet protocol, but each family changes the business action, output schema, and grader\-visible decision surface\. This is intended to make label diversity insufficient: an agent must handle repeated mechanics while still mapping them to distinct artifacts\.

Table 11:Task\-family summary\.FamilyTasksSub\-problemsFixture and output mechanicsOriginal seed tasks21107Early heterogeneous tasks covering code, logs, documents, office files, shell automation, CI, dependencies, and metrics; these use task\-specific fixtures and graders rather than the shared hard\-mode packet protocol\. Tasks: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21\.CRM/Executive525Pipeline hygiene, forecast variance, campaign QA, board packets, and executive action tracking with repeated business\-report outputs\. Tasks: 92, 93, 94, 95, 96\.Commerce/Food525Delivery refunds, inspections, marketplace policies, maintenance\-like retail cases, and supportable operational decisions over generated local fixtures\. Tasks: 77, 78, 79, 80, 81\.Data/Analytics525Normalization, reconciliation, rule\-authoring, experiment, and extraction tasks that stress repeatable data derivation under decoys and stale revisions\. Tasks: 62, 63, 64, 65, 66\.DevOps/SRE525Policy, drift, burn\-rate, pipeline\-hardening, and restore\-drill tasks with infrastructure\-oriented evidence channels and output artifacts\. Tasks: 52, 53, 54, 55, 56\.Facilities/IoT525Facilities, fleet, sensor, smart\-home, and workcell event tasks that derive maintenance or anomaly reports from selected evidence packets\. Tasks: 82, 83, 84, 85, 86\.Finance525Ledger, revenue, loan, tax, and KYC\-style reconciliation with selected evidence packets, stale revisions, numeric deltas, list edits, and boolean compliance gates\. Tasks: 22, 23, 24, 25, 26\.HR/Education525Onboarding, interview, admissions, grading, and completion\-audit cases requiring structured checklist or rubric outputs from decoy\-rich evidence\. Tasks: 42, 43, 44, 45, 46\.Healthcare525Clinical\-administration tasks such as triage, pharmacy reorder, trial eligibility, prior authorization, and follow\-up queues with strict output schemas\. Tasks: 32, 33, 34, 35, 36\.Legal525Clause, discovery, policy\-exception, NDA, and contract\-risk review tasks that require choosing authoritative records and writing structured legal\-operation outputs\. Tasks: 27, 28, 29, 30, 31\.Localization/Release525Placeholder QA, release notes, flag cleanup, catalog normalization, and returns analysis tasks with structured release or commerce\-adjacent artifacts\. Tasks: 72, 73, 74, 75, 76\.Procurement/Logistics525Supplier, bid, purchase\-order, warehouse, and shipping cases built around conflicting packets, exception lists, and derived routing or scoring reports\. Tasks: 37, 38, 39, 40, 41\.Public Sector/Audit525Service routing, records redaction, grant completeness, evidence collection, and risk register tasks using synthetic civic/audit fixtures\. Tasks: 87, 88, 89, 90, 91\.Research/Media525Claim checking, table extraction, directory extraction, citation deduplication, and glossary compliance with evidence selection and citation\-like outputs\. Tasks: 67, 68, 69, 70, 71\.Security/Privacy525Alert, phishing, exception, release\-redaction, and request\-routing tasks that test privacy/security triage without using real sensitive data\. Tasks: 57, 58, 59, 60, 61\.Support/Product525Ticket, knowledge\-base, app\-review, bug\-deduplication, and call\-quality tasks that convert repeated records into grader\-visible JSON reports\. Tasks: 47, 48, 49, 50, 51\.Travel/Office420Calendar, itinerary, meeting\-action, and inbox\-rule cases that convert office coordination evidence into structured decisions\. Tasks: 97, 98, 99, 100\.
## Appendix HGenerated Family Design Matrix

LABEL:tab:family\-design\-matrixexpands the family summary with the repeated structure that is intended to make skill creation plausible, and the anti\-shortcut mechanism that prevents a shallow schema\-only answer from satisfying the grader\. The matrix is written at the family level rather than the task level because generated tasks within a family share the same evidence\-selection contract while changing the concrete business operation and required output fields\.

Table 12:Design matrix for generated hard\-mode task families\.FamilyRepeated structureAnti\-shortcut mechanismGrader\-visible artifactCRM/ExecutiveRepeated business\-status packets must be reconciled into executive\-ready decisions\.Decoy opportunities, stale campaign revisions, and conflicting forecast evidence make visible totals unreliable\.Structured pipeline, forecast, campaign, board, or action\-item JSON reports\.Commerce/FoodRepeated commerce cases map operational evidence to refund, inspection, marketplace, or returns decisions\.Draft and superseded packets expose plausible but wrong SKUs, refund IDs, or policy codes\.Per\-case operational JSON reports underoutputs/\.Data/AnalyticsRepeated analytics records require applying numeric, list, and text derivation channels consistently\.Schema\-only outputs fail because selected packets must be verified before computing rows, rules, or experiment winners\.Strict JSON reports with derived counts, deltas, failures, and winners\.DevOps/SRERepeated infrastructure cases ask for drift, burn\-rate, policy, pipeline, or restore conclusions\.Stale revisions and decoy resource identifiers make direct copying from fixtures unsafe\.JSON reports that encode affected resources, exception IDs, and remediation signals\.Facilities/IoTRepeated sensor, fleet, smart\-home, and workcell packets require deriving maintenance or anomaly actions\.Conflicting packet states and boolean gates separate approved evidence from noisy telemetry\.Prioritization, diagnosis, schedule, or event\-summary JSON reports\.FinanceRepeated finance cases share ledger\-style aggregation, exception lists, tax\-like numeric fields, and compliance gates\.Invalid checksums, superseded packets, and decoy ledgers prevent copying visible totals\.Reconciliation or packet\-review JSON reports\.HR/EducationRepeated people\-process cases map evidence to checklists, rubrics, screening, or calibration decisions\.Decoy records include plausible missing items or scores that must be ignored unless final and selected\.Checklist, rubric, admissions, interview, or completion\-audit JSON reports\.HealthcareRepeated clinical\-administration cases derive triage, reorder, eligibility, authorization, or follow\-up decisions\.Synthetic healthcare\-like evidence avoids real patient data while still requiring selected\-packet derivation\.Strict healthcare\-administration JSON reports\.LegalRepeated legal\-operation cases extract obligations, clauses, exceptions, or contract risks\.Draft clauses and stale revisions test whether the agent follows packet provenance instead of surface wording\.Clause, discovery, NDA, policy, or risk JSON outputs\.Localization/ReleaseRepeated release\-management cases reconcile placeholders, glossary issues, flags, changelog impacts, or catalog data\.Multiple revisions and alias/remove list actions require ordered list derivation rather than keyword matching\.Release, localization, flag, changelog, or catalog\-normalization JSON reports\.Procurement/LogisticsRepeated sourcing and logistics cases derive vendors, purchase\-order exceptions, warehouse issues, or shipment actions\.Competing packet weights and revisions expose believable but wrong vendor and exception values\.Procurement, warehouse, and shipping JSON reports\.Public Sector/AuditRepeated public\-service and audit packets require routing, redaction, evidence, grant, or risk decisions\.Synthetic public\-sector records contain missing\-evidence decoys and privacy\-like fields without real PII\.Civic\-routing, redaction, grant, evidence, and risk\-register JSON reports\.Research/MediaRepeated research and media cases derive evidence tables, citations, claims, facts, or glossary compliance\.Text candidates and list edits force deterministic selection rather than free\-form summarization\.Claim, fact\-check, extraction, citation, or compliance JSON reports\.Security/PrivacyRepeated security and privacy cases derive incidents, redactions, phishing, exceptions, or DSR routing\.Decoy indicators and boolean gates separate final selected security evidence from raw noisy records\.Alert, privacy, phishing, exception, or release\-redaction JSON reports\.Support/ProductRepeated support and product cases derive escalations, KB gaps, app\-review themes, bug clusters, or call quality\.Alias and remove actions test whether the agent can update lists rather than aggregate all visible labels\.Support/product JSON reports with routed or clustered decisions\.Travel/OfficeRepeated office\-coordination cases derive conflicts, itinerary exceptions, meeting actions, or inbox classifications\.Stale calendar or inbox\-rule revisions make the latest visible item insufficient without packet selection\.Calendar, travel, meeting, or inbox\-rule JSON reports\.
## Appendix IHard\-Mode Evidence Protocol

Generated hard\-mode tasks are written so that visible values in a fixture are not necessarily authoritative\. Each case contains a packet manifest plus evidence records\. The agent must select the approved packet with no superseding packet and a valid checksum equal to the first 16 hexadecimal characters of the hash of the benchmark salt, task id, case id, packet id, and nonce\. If more than one packet remains, it chooses the highest revision, then the highest source weight, then the lowest packet identifier\. Only final records from the selected packet may drive the output\.

> sha256\(evoclawbench\-difficulty\-hardening\-20260524\-v4\| <task\_id\>\|<case\_id\>\|<packet\_id\>\|<nonce\>\)

Table 13:Evidence\-channel derivation rules used by generated hard\-mode tasks\.
The protocol appears across JSON, YAML, CSV, text, and HTML fixtures\. JSON and YAML expose manifest and record arrays directly; CSV fixtures use section\-tagged rows; text fixtures use JSON lines; HTML fixtures embed JSON in script blocks\. The grader checks the derived files underoutputs/and does not accept a final natural\-language explanation as a substitute for those artifacts\.

## Appendix JRepresentative Output Schemas

For generated hard\-mode tasks, the prompt names the exact JSON fields that must appear in each case report\. The table lists one representative schema per generated family; seed tasks use task\-specific artifact formats such as scripts, workbooks, SQL migrations, Dockerfiles, CI files, or extraction JSON\.

Table 14:Representative output schemas for generated hard\-mode families\.FamilyRepresentative taskRequired output fieldsCRM/Executivetask\_92: CRM Pipeline Hygienestale\_opportunities, forecast\_delta, campaign\_errors, action\_items, executive\_summaryCommerce/Foodtask\_77: Ecommerce Catalog Normalizationnormalized\_skus, refund\_ids, inspection\_score, policy\_violations, recommended\_actionData/Analyticstask\_62: SQL Schema Migration Reviewrow\_count, quality\_failures, metric\_delta, rule\_ids, experiment\_winnerDevOps/SREtask\_52: Kubernetes Policy Reviewpolicy\_violations, required\_changes, slo\_status, backup\_passed, risk\_scoreFacilities/IoTtask\_82: Facilities Maintenance Prioritizationpriority\_assets, maintenance\_due, anomaly\_ids, diagnostic\_codes, dispatch\_requiredFinancetask\_22: Finance Ledger Reconciliationledger\_total, exception\_ids, currencies, tax\_total, balancedHR/Educationtask\_42: Employee Onboarding Checklistcompletion\_rate, missing\_items, calibrated\_scores, assigned\_track, intervention\_idsHealthcaretask\_32: Appointment Referral Triageeligible\_ids, routing\_queue, followup\_ids, stockout\_ids, urgent\_countLegaltask\_27: Contract Clause Extractionclause\_labels, missing\_fields, high\_risk\_terms, parties, effective\_dateLocalization/Releasetask\_72: Localization Placeholder Qaplaceholder\_errors, glossary\_violations, release\_sections, flag\_actions, publish\_readyProcurement/Logisticstask\_37: Procurement Bid Scoringselected\_vendor, match\_exceptions, late\_shipments, risk\_suppliers, savings\_estimatePublic Sector/Audittask\_87: Civic Service Request Routingrouting\_queue, redactions\_required, missing\_evidence, risk\_summary, compliantResearch/Mediatask\_67: Research Claim Evidencesupported\_claims, unsupported\_claims, source\_count, duplicate\_citations, confidenceSecurity/Privacytask\_57: Security Alert Correlationincident\_ids, redacted\_fields, raw\_pii\_present, approved\_exceptions, severity\_countsSupport/Producttask\_47: Support Ticket Escalationpriority\_queue, duplicate\_groups, themes, sla\_breaches, reply\_templateTravel/Officetask\_97: Calendar Conflict Resolutionconflicts, exceptions, action\_items, rule\_labels, resolved
## Appendix KSuite Distributions

Table 16:Sub\-problem counts\.
Table 17:Fixture extensions\.

## Appendix LSkill Artifact Structure

Generated skill directories are discovered underskills/\. The seededskills/skill\-creatorbundle is excluded from created\-skill metrics\. A generated skill must includeSKILL\.md; optional scripts and references are counted when present\. Before reuse execution, non\-seeded skill files are hashed\. The same files are hashed after execution, and any add, delete, or content change is recorded as a mutation violation for that phase\.

![[Uncaptioned image]](https://arxiv.org/html/2607.09711v1/x5.png)

Figure 5:Three\-mode evaluation protocol\.Baselinedirectly executes a task without skill creation;PreSkillfirst creates task\-specific skills and then reuses them in a fresh workspace;PostSkillsummarizes reusable skills from first\-run evidence before a second execution\. Skill reuse phases are checked for mutation violations\.

## Appendix MResult JSON Schema

The aggregate output JSON containsbaseline\_results,preskill\_results,postskill\_results, andmetrics\. The table below summarizes the fields used for benchmark reporting\.

Table 18:Result JSON fields used for benchmark reporting\.

## Appendix NResult Validity Checklist

Before adding a result row, record the model, runtime, execution mode, environment, worker count, judge model, loaded task count, official task count, mean scores for all modes, created\-skill counts, and mutation checks\. Runs should be reported consistently acrossBaseline,PreSkill, andPostSkillso that score, cost, and skill\-reuse comparisons refer to the same benchmark protocol\.