SkillCorpus: Consolidating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents

# SkillCorpus: Consolidating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents
Source: [https://arxiv.org/html/2607.15557](https://arxiv.org/html/2607.15557)
Yanze Wang1,2,3\\equalcontrib, Pengfei Yao1,2\\equalcontrib, Tianyi Sun1,2,3, Chuanrui Hu1,2111Project leader\., Yan Xiao1,2, Yunyun Han1,2, Jun Sun3\\corresponding, Yafeng Deng1,2\\corresponding

###### Abstract

Agent skills,SKILL\.mdfiles that package reusable procedural knowledge for an LLM agent, are a popular mechanism for extending agent capabilities\. Public repositories now host them in large and growing numbers, yet these artifacts are fragmented, redundant, and uneven in quality, and their value in practice is unclear\. A core question remains open, namely how to consolidate this open\-sourceSKILL\.mdecosystem into a single usable corpus, and what bounds its benefit on real\-world agent tasks\. We presentSkillCorpus, a framework that aggregates, curates, matches, and evaluates the open skill ecosystem at scale\. It filters∼\\sim821,000821\{,\}000crawled skills through a multi\-stage pipeline into96,40196\{,\}401skills organised by a 16\-class taxonomy and three quality facets \(utility, robustness, safety\), and pairs them with a fine\-tuned retrieval\-and\-selection stack that matches task\-relevant skills\. We evaluate end\-to\-end across three benchmarks \(SkillsBench, GDPVal, QwenClawBench\), two harnesses, and two open backbones with a frontier robustness check\. Integrating SkillCorpus yields consistent gains across all three benchmarks, largest on SkillsBench \(\+7\.5\+7\.5pp\)\. An operational analysis traces the gains to a coverage boundary and a harness boundary\. SkillCorpus is, to our knowledge, the first end\-to\-end account of when a curated, retrieval\-served community corpus improves real agent tasks, and where it does not\. The dataset, models, and code will be released upon acceptance\.

## 1Introduction

Large language models increasingly act as autonomous agents that orchestrate long\-horizon executable tasks rather than answer isolated queries\(Yaoet al\.[2023](https://arxiv.org/html/2607.15557#bib.bib1); Shinnet al\.[2023](https://arxiv.org/html/2607.15557#bib.bib2); Parket al\.[2023](https://arxiv.org/html/2607.15557#bib.bib3)\)\. Yet on composite real\-world workloads, even the strongest frontier systems remain measurably bounded\(Merrillet al\.[2026](https://arxiv.org/html/2607.15557#bib.bib26); OpenAI[2025b](https://arxiv.org/html/2607.15557#bib.bib30)\)\. This gap is difficult to close by raw model scaling alone\. A growing consensus instead treats agent capability as a layered system, in which a frozen foundation model is paired with externalised skills and an agent harness that loads and executes them\(Anthropic[2025a](https://arxiv.org/html/2607.15557#bib.bib16)\)\. A skill packages reusable procedural knowledge invoked at inference time\. Unlike opaque, slow\-to\-update model weights or transient in\-context examples, it is interpretable and modular\. The same externalise\-and\-retrieve pattern is already well\-established for tool use, where agents retrieve and invoke calls over large API inventories\(Qinet al\.[2024](https://arxiv.org/html/2607.15557#bib.bib7); Patilet al\.[2023](https://arxiv.org/html/2607.15557#bib.bib8)\)and the recent Model Context Protocol \(MCP\) further standardises access to external tools and resources\(Houet al\.[2025](https://arxiv.org/html/2607.15557#bib.bib9); Linet al\.[2025](https://arxiv.org/html/2607.15557#bib.bib10)\)\. Skills extend this pattern from external tool calls to procedural knowledge itself\.

Industry\-grade harnesses now ingest skill files routinely\(Anthropic[2025b](https://arxiv.org/html/2607.15557#bib.bib32); OpenAI[2025a](https://arxiv.org/html/2607.15557#bib.bib33)\), and a community ecosystem has formed around theSKILL\.mdspecification\(Anthropic[2025a](https://arxiv.org/html/2607.15557#bib.bib16)\), now spanning thousands of public repositories\. Whether this abundance actually improves agent performance on real tasks, and what it takes to make it do so, remains unsettled\. Curated skills paired one\-to\-one with tasks raise pass rates on average but with sharply heterogeneous per\-domain effects\(Liet al\.[2026b](https://arxiv.org/html/2607.15557#bib.bib18)\), while uncurated community libraries can fail to improve over a no\-skill baseline\(Liet al\.[2025](https://arxiv.org/html/2607.15557#bib.bib19); Liuet al\.[2026b](https://arxiv.org/html/2607.15557#bib.bib20)\)\. Two gaps stand in the way\.First, we know of no released corpus that is at once broadly aggregated, strictly quality\-gated, and cleared for open redistribution\. Existing artifacts draw from only one or two source channels\(Liet al\.[2025](https://arxiv.org/html/2607.15557#bib.bib19); Liuet al\.[2026b](https://arxiv.org/html/2607.15557#bib.bib20)\), and those that filter for quality show no evidence of broad coverage at scale\(Lianget al\.[2026](https://arxiv.org/html/2607.15557#bib.bib17)\)\.Second, the downstream value of a community corpus on real agent tasks is largely uncharacterised\. Prior evaluations pair tasks with hand\-picked oracle skills rather than a deployed corpus\(Liet al\.[2026b](https://arxiv.org/html/2607.15557#bib.bib18)\), run in simulated environments\(Lianget al\.[2026](https://arxiv.org/html/2607.15557#bib.bib17)\), or measure in\-the\-wild usage through a single harness over limited sources\(Liuet al\.[2026b](https://arxiv.org/html/2607.15557#bib.bib20)\)\.

To close both gaps, we presentSkillCorpus, a single end\-to\-end framework that consolidates the openSKILL\.mdecosystem into a curated, deployable corpus and evaluates it on real\-world agent tasks — making it possible to measure when community skills help real agents, and where they do not\.

Our contributions are threefold:

- •Framework\.We unify aggregation, curation, matching, and real\-world evaluation of the openSKILL\.mdecosystem, turning a fragmented, redundant, and uneven pool of community skills into a curated foundation for reuse at scale\.
- •Open resource\.We release the resulting96,40196\{,\}401\-skill corpus \(curated from∼\\sim821,000821\{,\}000raw files\), its fine\-tuned retrieval\-and\-selection stack, and the full curation, training, and evaluation code, all licence\-audited and100%100\\%OSI\-permissive — a fully open, ecosystem\-scale resource for skill\-augmented agent research\.
- •Findings\.Evaluating across three real\-world agent benchmarks, independent harnesses, and open\-to\-frontier backbones \(§[4](https://arxiv.org/html/2607.15557#S4)\), we find that integrating SkillCorpus consistently improves agent performance, with gains modulated by corpus coverage and the harness, characterising the real utility and limits of community skills\.

## 2Related Work

### 2\.1Skill acquisition and curation

Work on the skill layer divides by when skills enter the system\. One strand acquires them at run time, while the agent is solving tasks\. Voyager grows a skill library through environment exploration\(Wanget al\.[2024](https://arxiv.org/html/2607.15557#bib.bib4)\), Toolformer trains the model to invoke tools self\-supervisedly\(Schicket al\.[2023](https://arxiv.org/html/2607.15557#bib.bib6)\), ToolLLM and Gorilla scale API calling to large tool inventories\(Qinet al\.[2024](https://arxiv.org/html/2607.15557#bib.bib7); Patilet al\.[2023](https://arxiv.org/html/2607.15557#bib.bib8)\), and recent agent\-evolution work lets agents redesign their own scaffolding\(Zhouet al\.[2026](https://arxiv.org/html/2607.15557#bib.bib14); Linet al\.[2026](https://arxiv.org/html/2607.15557#bib.bib15)\)\. All of these build on agent harness scaffolds\(Yaoet al\.[2023](https://arxiv.org/html/2607.15557#bib.bib1); Shinnet al\.[2023](https://arxiv.org/html/2607.15557#bib.bib2)\)\. The other strand, which this paper occupies, prepares skills before the agent ever runs\. We curate a community\-contributed corpus at ingest time, so that runtime systems have a trustworthy library to draw from\. This ingest\-time curation mirrors data\-quality work in language\-model training, where deduplication\(Leeet al\.[2022](https://arxiv.org/html/2607.15557#bib.bib36)\)and quality\-based data selection\(Zhouet al\.[2023](https://arxiv.org/html/2607.15557#bib.bib37)\)improve downstream models more than raw scale\.

Released community skill corpora vary in emphasis\.Lianget al\.\([2026](https://arxiv.org/html/2607.15557#bib.bib17)\), concurrent with our work, releases a 200K\-skill ontology rated along five quality rubrics and linked by a skill\-relation graph\. It measures benefit on simulated environments, with skills synthesised from each benchmark’s own expert trajectories rather than on real tasks served by a deployed corpus\.Liet al\.\([2025](https://arxiv.org/html/2607.15557#bib.bib19)\)pairs a 36K uncurated library with a four\-stage retrieval pipeline and treats quality only implicitly, through artifact\-density signals\.Linget al\.\([2026](https://arxiv.org/html/2607.15557#bib.bib21)\)provides a data\-driven characterisation of a smaller curated set\. None of these releases reports a source\-licence audit, so their redistribution rights are left unresolved\. Nor does prior work tie these per\-skill scores to their downstream operational role\. We introduce a compact three\-facet framework chosen for measurable facet separation rather than rubric coverage, and characterise its operational signal directly \(§[4\.6](https://arxiv.org/html/2607.15557#S4.SS6)\)\.

### 2\.2Skill retrieval and evaluation

Skill retrieval architectures range from lexical and dense recall to learned skill routers\(Zhenget al\.[2026](https://arxiv.org/html/2607.15557#bib.bib24)\), supported by dedicated retrieval benchmarks\(Choet al\.[2026](https://arxiv.org/html/2607.15557#bib.bib23)\), corpus\-size scaling analyses\(Chenet al\.[2026](https://arxiv.org/html/2607.15557#bib.bib22)\), and dependency\-aware composition\(Liuet al\.[2026a](https://arxiv.org/html/2607.15557#bib.bib25)\)\. A complementary line of work emphasises skill orchestration and context\-injection over retrieval per se\.Liet al\.\([2026a](https://arxiv.org/html/2607.15557#bib.bib28)\)couples capability\-tree retrieval with DAG\-based orchestration at∼\\sim200200K scale, andMenget al\.\([2026](https://arxiv.org/html/2607.15557#bib.bib29)\)reports\+11\.7%\+11\.7\\%on SkillsBench via a skill graph plus compact compilation\. That gain, however, is measured against the prior state\-of\-the\-art injection method rather than a no\-skill or community\-corpus baseline, so it is not directly comparable to our end\-to\-endΔ\\Delta\. The retrieval strand reports ranking metrics on its own curated pools rather than task outcomes\. Our stack is fine\-tuned on the deduplicated corpus it serves and adds an LLM selector that reads full skill bodies before injection, with both its standalone retrieval performance and its end\-to\-end effect on task pass rates measured directly\.

On evaluation,Liet al\.\([2026b](https://arxiv.org/html/2607.15557#bib.bib18)\)characterises oracle\-skill efficacy, andLiuet al\.\([2026b](https://arxiv.org/html/2607.15557#bib.bib20)\)reports diminishing returns on uncurated libraries at scale\. Neither evaluates a single community\-curated corpus under a deployable retrieval\-and\-selection stack across multiple real\-world benchmarks\. §[4](https://arxiv.org/html/2607.15557#S4)fills this gap with a cross\-benchmark, cross\-harness, multi\-backbone measurement of end\-to\-end task gains\.

## 3SkillCorpus

![Refer to caption](https://arxiv.org/html/2607.15557v1/figures/fig_pipeline_v10.png)Figure 1:The SkillCorpus framework\. Building the corpus \(top\), we aggregate communitySKILL\.mdfiles from public source channels and curate them through a six\-stage funnel into a96,40196\{,\}401\-skill corpus organised by a 16\-class taxonomy and three quality facets\. Using the corpus \(bottom\), a fine\-tuned retrieval\-and\-selection stack matches skills to an incoming task and an agent runs the selected skill end\-to\-end\. The worked example traces one real task from request to result, and we evaluate this pipeline across three real\-world benchmarks, two harnesses, and open\-to\-frontier backbones\.SkillCorpus is a single end\-to\-end framework that turns the openSKILL\.mdecosystem into a curated, deployable corpus and measures its effect on real\-world agent tasks\. As shown in Figure 1, it runs in four stages: aggregation and curation of the ecosystem into a corpus \(§3\.1–§3\.3\), matching of task\-relevant skills via a fine\-tuned retrieval\-and\-selection stack \(§3\.4\), and end\-to\-end evaluation on real\-world agent tasks \(§[4](https://arxiv.org/html/2607.15557#S4)\)\.

### 3\.1Multi\-source aggregation and the curation pipeline

The corpus’s value is its coverage, spanning the whole public ecosystem rather than a single channel\. The crawl is driven by a single machine\-readable source registry of6262sources, detailed in Appendix[A\.6](https://arxiv.org/html/2607.15557#A1.SS6)\. The registry spans five ingestion mechanisms, from directly cloned repositories and awesome\-list scrapes to marketplace index APIs, sitemap crawls, and JSON catalogs, and every mechanism feeds one shared discovery entry point\. Our crawl gathers∼\\sim821,000821\{,\}000rawSKILL\.mdfiles\. After source selection and repository\-level de\-duplication,25,15925\{,\}159GitHub repositories feed the six\-stage pipeline, which curates them into the released active set\.

Figure[1](https://arxiv.org/html/2607.15557#S3.F1)visualises the pipeline\. Each skill follows theSKILL\.mdconvention ofAnthropic \([2025a](https://arxiv.org/html/2607.15557#bib.bib16)\)— a Markdown body with optional executable resources; Stages 1–2 apply structural filters on this contract \(parse and length/form\)\. Stage 3 deduplicates\. Stage 4 runs the LLM judge that emits the 3\-facet quality score and 19 safety/quality flags\. Stage 5 applies the safety hard\-gate plus Open Source Initiative \(OSI\)\-permissive licence filter \(Table[5](https://arxiv.org/html/2607.15557#A1.T5)\)\. Stage 6 attaches source\-prior shrinkage, the 1024\-dim retrieval embedding, and index entries\. Parse and deduplication dominate the funnel, while the remaining filters contribute a few percent\.

#### Two\-tier deduplication \(stage 3\)\.

Deduplication is the funnel’s steepest reduction \(283,844→101,111283\{,\}844\\to 101\{,\}111,−64%\-64\\%\), in two tiers\. An exact tier collapses169,465169\{,\}465content\- and name\-fingerprint duplicates,59\.7%59\.7\\%of the stage input, which reflects how heavily the ecosystem replicates the same artifacts across repositories\. A semantic tier embeds each survivor \(1024\-dim\) and flags pairs above cosine0\.900\.90, and pairs above0\.9950\.995merge automatically\. The66,75166\{,\}751borderline pairs are adjudicated by an LLM judge, which confirms25\.4%25\.4\\%of pairs and collapses13,26813\{,\}268near\-duplicate skills\. The resulting low\-overlap distribution also lets the retrieval stack train on the corpus itself\.

### 3\.2Three\-facet quality framework

We decompose skill quality into three facets, each capturing a distinct concern, rather than collapsing them into a single composite or a flat multi\-dimensional list\. These facets serve curation and retrieval ranking, with the safety facet additionally serving the release gate, as characterised in §[4\.6](https://arxiv.org/html/2607.15557#S4.SS6)\.

#### Three facets\.

The utility facet scores the description alone, asking whether the task class is well\-scoped with clear triggers and applicability conditions\. The robustness facet scores the body content and its consistency with the description’s promise\. A body that is internally coherent but silently delivers something narrower than promised is penalised here\. The safety facet captures harm and risk along categories such as prompt injection, command injection, credential leakage, and unsafe execution\. By design, the utility and robustness facets are scored independently\. Description\-level issues only affect utility, and body\-level issues only affect robustness\.

#### Flag taxonomy\.

The 19\-flag vocabulary is partitioned across the three facets \(22utility\-bound,66robustness\-bound,1111safety\-bound\), with the full list in Appendix[A\.1](https://arxiv.org/html/2607.15557#A1.SS1)\. Each flag, when fired, constrains its assigned facet’s score\. Five flags act as hard gates:prompt\_injection,cmd\_injection,unsafe\_exec,auth\_bypass, andcsam\_risk\. Firing any one of these forces the quality score to zero and excludes the skill from the releasedactiveset\. The remaining fourteen flags act as soft signals\. When such a flag fires, the LLM judge caps the corresponding facet score\. A firedplaceholderflag, for example, holds robustness at≤4\\leq 4\. No additional numeric deduction is applied, so the same observation is not double\-counted\.

#### Facet separation\.

The three facets are not redundant\. The safety facet is markedly more independent from the two craftsmanship facets \(r​\(u,s\)=0\.15r\(u,s\)\{=\}0\.15,r​\(r,s\)=0\.30r\(r,s\)\{=\}0\.30\) than those are from each other \(r​\(u,r\)=0\.59r\(u,r\)\{=\}0\.59\), matching the design, sinceuuandrrboth track artifact craftsmanship whilesstargets an orthogonal harm axis on which a well\-crafted skill can still be unsafe\. We read these contrasts qualitatively and do not claim construct\-validity orthogonality\. Rigorous validation would require independent human gold annotation\(Lianget al\.[2026](https://arxiv.org/html/2607.15557#bib.bib17)\)\. The off\-diagonal\-mass analysis, the per\-pair figure, and the score\-bucket distribution are given in Appendix[A\.2](https://arxiv.org/html/2607.15557#A1.SS2)\.

#### Score formula\.

Following the LLM\-as\-judge paradigm\(Zhenget al\.[2023](https://arxiv.org/html/2607.15557#bib.bib39)\), the judge outputs three integer subscoresu,r,su,r,son\{0,…,10\}\\\{0,\\ldots,10\\\}, normalised to\[0,1\]\[0,1\]\. These combine into𝚌𝚘𝚗𝚝𝚎𝚗𝚝​\_​𝚚=0\.50​u\+0\.35​r\+0\.15​s\\mathtt\{content\\\_q\}=0\.50\\,u\+0\.35\\,r\+0\.15\\,s, attenuated by a factor0\.5\+0\.5​\(s−0\.3\)/0\.40\.5\+0\.5\\,\(s\{\-\}0\.3\)/0\.4when the safety subscore is marginal \(s∈\[0\.3,0\.7\]s\\in\[0\.3,0\.7\]\)\. The final per\-skill score, or composite quality score, shrinks content quality toward a per\-source prior \(Appendix[A\.3](https://arxiv.org/html/2607.15557#A1.SS3)\):

𝚜𝚌𝚘𝚛𝚎=0\.85​𝚌𝚘𝚗𝚝𝚎𝚗𝚝​\_​𝚚\+0\.15​𝚙𝚛𝚒𝚘𝚛src\+b,\\mathtt\{score\}=0\.85\\,\\mathtt\{content\\\_q\}\+0\.15\\,\\mathtt\{prior\}\_\{\\mathrm\{src\}\}\+b,wherebbis a small structural bonus \(\+0\.05\+0\.05for skills shippingscripts/,\+0\.02\+0\.02forreferences/\), and the result is clipped to\[0,1\]\[0,1\]\.

### 3\.3Classification, filtering, and release

#### Taxonomy\.

We adopt a single\-label, 16\-class taxonomy designed to cover the observed distribution of community skills while remaining decision\-tree classifiable fromname,description, and a short body excerpt\. The classifier prompt enforces six conflict\-resolution rules, the most consequential being theAI\-MLrule that a skill is labelledAI\-MLonly when its primary deliverable is an AI system \(persona, agent, or model\), not merely when it uses an LLM internally\. The full rule set is released with the classifier prompt\.

#### Distribution\.

The class distribution on the active release set is summarised in Figure[2](https://arxiv.org/html/2607.15557#S3.F2)\. The corpus is software\- and data\-leaning\.Dev,Data,Writing, andDevOps\-Infratogether account for over half of active skills\. The catch\-allOtherclass is small \(2\.01%2\.01\\%\), indicating that the taxonomy covers the observed distribution well\. Every active skill carries exactly one of the sixteen classes\.

![Refer to caption](https://arxiv.org/html/2607.15557v1/x1.png)Figure 2:16\-class distribution on the active release set \(N=96,401N\{=\}96\{,\}401, shares in %, sorted clockwise\)\.
#### Safety and licence filters \(stage 5\)\.

The two filters run in sequence on the∼\\sim101101K skills surviving stages 1–4\. A safety hard\-gate removes915915skills triggering any of three conditions: a pre\-judgeblocked\.malwareregex match \(Appendix[A\.4](https://arxiv.org/html/2607.15557#A1.SS4)\), firing of any of the five LLM\-judged hard\-gate flags defined in §[3\.2](https://arxiv.org/html/2607.15557#S3.SS2), or an LLM safety subscore below33\. A licence filter then enriches each source repository’sspdx\_idvia the GitHub API and admits only skills carrying an OSI\-approved permissive licence \(Table[4](https://arxiv.org/html/2607.15557#A1.T4)\), excluding a further3,7953\{,\}795skills whose source repositories lack a usable permissive licence \(per\-reason counts in Table[5](https://arxiv.org/html/2607.15557#A1.T5)\)\. Every skill in the resulting96,40196\{,\}401\-skill active set is therefore commercially redistributable under its source repository’s declared licence\.

#### Release\.

The corpus is released as a single artifact comprising the metadata SQLite database, the per\-stage ingest report of raw counts at each pipeline stage, the precomputed 3\-facet quality cache, the 1024\-dim retrieval embeddings, and the retrieval\-and\-selection stack used in our experiments and released as engineering tooling\. All curation source code is published under a permissive licence at the project repository\.

### 3\.4Retrieval and selection stack

The corpus is released together with a fine\-tuned retrieval\-and\-selection stack so the artifact ships ready to use rather than requiring a retrieval stack to be built\. We use “deployable” in this packaging sense, not as a production\-validation claim\. The stack follows the standard retrieve\-then\-rerank pattern of retrieval\-augmented generation\(Lewiset al\.[2020](https://arxiv.org/html/2607.15557#bib.bib38)\), with three operational stages: candidate recall, rerank, and final selection by an LLM gate\.

#### Recall and rerank\.

Candidate recall uses an embedding model fine\-tuned from Qwen3\-Emb\-0\.6B\. A scoring model fine\-tuned from Qwen3\-Rank\-0\.6B then reranks the top candidates\. Both models are trained on the strictly deduplicated active set\. The low\-overlap data distribution is intended to give a cleaner contrastive learning signal by preventing functionally identical near\-duplicates from being sampled as hard negatives, which could otherwise produce false\-negative collisions during contrastive fine\-tuning\. We adopt this as a training\-design rationale rather than an isolated ablation\. Recall and rerank operate over a3,0003\{,\}000\-character retrieval field distilled from each skill body, avoiding length\-induced truncation noise\.

#### LLM selector \(gate\)\.

Reranked top\-kkcandidates can still include topically irrelevant items, and brief summaries are insufficient for an agent to judge tool fit at execution time\. We therefore insert an LLM selector that, given the task query and the full skill body of each reranked candidate, returns0–22precise tools for injection into the agent prompt\. This trades one additional LLM call for the reliability of full\-body inspection while keeping the injected context within strict per\-task budgets\.

#### Query rewriter \(optional\)\.

An optional LLM\-based query rewriter normalises domain\-specific jargon in the task description to vocabulary that better matches the corpus’s skill descriptions, and is permitted to short\-circuit \(no skills required\) when the task is clearly outside the corpus’s scope\.

We evaluate this stack both as part of the end\-to\-end pipeline, where it drives the main\-grid pass\-rate gains, and as a standalone retrieval system on Hit@1 / Recall@10 against published baselines\.

## 4Experiments and Results

### 4\.1Setup

Table 1:Main results: absolute performance per cell and benchmark, no\-skill baseline→\\toSkillCorpus condition\. SkillsBench reports pass@1 averaged over three runs \(%,n=87n\{=\}87tasks, the44that do not build scored as failures\)\. GDPVal reports the mean LLM\-judge reward \(3\-rep\) over all220220tasks\. QwenClawBench reports the mean hybrid score \(automated checks and an LLM judge\), 3\-rep mean\. Rewards are on\[0,1\]\[0,1\], shown×100\\times 100\. Per\-cell gains are visualised in Figure[3](https://arxiv.org/html/2607.15557#S4.F3)and computed on unrounded values\. The bottom row gives the pooledΔ\\Deltaacross the four cells, with a task\-clustered standard error overn=87/220/100n\{=\}87/220/100tasks and two\-sidedz=Δ/SEz\{=\}\\Delta/\\mathrm\{SE\}\.#### Benchmarks, harnesses, and backbones\.

We evaluate on three third\-party agent benchmarks, chosen because frontier LLMs do not saturate them and their domain coverage is complementary\. SkillsBench\(Liet al\.[2026b](https://arxiv.org/html/2607.15557#bib.bib18)\)contributes8787deterministic\-verifier tasks across88domains and is the closest direct analog to our setting\. GDPVal\(OpenAI[2025b](https://arxiv.org/html/2607.15557#bib.bib30)\)contributes220220real\-world economic tasks that span legal, financial, design, and writing work\. QwenClawBench\(Qwen Team, Alibaba Group[2026](https://arxiv.org/html/2607.15557#bib.bib31)\)adds100100agent tasks across88domains drawn from a real\-user task distribution\. In total the evaluation covers407407tasks spanning2626domain labels \(Appendix[B\.2](https://arxiv.org/html/2607.15557#A2.SS2), Figure[7](https://arxiv.org/html/2607.15557#A2.F7)\)\. We use two open\-sourceSKILL\.md\-conformant agent harnesses, OpenClaw\(Steinberger[2025](https://arxiv.org/html/2607.15557#bib.bib34)\)and Raven\(Raven Contributors[2026](https://arxiv.org/html/2607.15557#bib.bib35)\), developed independently, paired with two open backbones that span the lightweight\-to\-large range, Qwen3\.5\-27B \(Q\-27B\) and Qwen3\.5\-397B\-A17B\-GPTQ\-Int4 \(Q\-397B\), yielding four \(harness, backbone\) cells\. A frontier\-model robustness check on Claude Opus 4\.7 verifies that the effect is not specific to the open backbones\.

#### Configurations, inference, and metrics\.

Each cell is evaluated under a no\-skill baseline and under SkillCorpus paired with our retrieval\-and\-selection stack \(§[3\.4](https://arxiv.org/html/2607.15557#S3.SS4)\), whose standalone retrieval performance is reported in §[4\.4](https://arxiv.org/html/2607.15557#S4.SS4)\. The per\-cellΔ\\Deltatherefore measures the end\-to\-end pipeline effect\. All runs use each backbone’s API\-default inference configuration\. The Qwen models run greedy decoding with their default context and tool\-use template, and Opus 4\.7 runs at its default thinking effort\. Both conditions in a cell share the same setting, so the reported within\-cellΔ\\Deltais unaffected by inference effort and remains valid even though our vendor\-default absolute baselines are not directly comparable to published leaderboard numbers obtained under other configurations\. For SkillsBench we report the binary pass rate over all8787tasks, scoring the44that do not build in our environment as failures in both conditions\. For GDPVal and QwenClawBench we report the mean continuous reward, both on\[0,1\]\[0,1\]withΔ\\Deltain percentage points\. GDPVal is scored by a GPT\-4o judge, from a different model family than our Qwen and Opus backbones, and QwenClawBench by its official grading configuration combining automated checks with an LLM judge\(Qwen Team, Alibaba Group[2026](https://arxiv.org/html/2607.15557#bib.bib31)\)\. The main grid holds2424configurations, the four cells evaluated on three benchmarks under both conditions, each averaged over three runs\. With the two\-condition single\-run frontier check, this comes to7474end\-to\-end runs\. Appendix[C\.1](https://arxiv.org/html/2607.15557#A3.SS1)reports the ingest and evaluation compute\.

### 4\.2Main results

Table[1](https://arxiv.org/html/2607.15557#S4.T1)reports the absolute performance of the no\-skill baseline and the SkillCorpus condition \(the corpus with our retrieval\-and\-selection stack\) for each of the four \(harness, backbone\) cells on each benchmark\. The per\-cell gains \(Δ\\Delta\) and their per\-cell means are visualised in Figure[3](https://arxiv.org/html/2607.15557#S4.F3)\.

![Refer to caption](https://arxiv.org/html/2607.15557v1/x2.png)Figure 3:Per\-cell, per\-benchmarkΔ\\Delta\(pp\) computed from the absolute values in Table[1](https://arxiv.org/html/2607.15557#S4.T1)\. Green = positive, red = negative\.†Mean over the three benchmarks\.On SkillsBench, which has a published oracle baseline\(Liet al\.[2026b](https://arxiv.org/html/2607.15557#bib.bib18)\), all four open\-backbone cells show positiveΔ\\Deltaaveraging\+7\.5\+7\.5pp over three runs, from\+4\.2±2\.7\+4\.2\{\\pm\}2\.7on the weakest cell to\+13\.4±3\.1\+13\.4\{\\pm\}3\.1on the strongest \(Raven×\\timesQ\-397B\), with an aligned\+8\.0\+8\.0pp on the Claude Opus 4\.7 frontier check\. In one run the strongest cell rescues1919baseline\-failing tasks while breaking22\(exact McNemar test,p<0\.001p\{<\}0\.001\)\. No cell shows a net\-negative mean, so at the cell\-mean level the corpus behaves as a no\-harm attachment whose upside varies by cell, even though individual tasks can still regress \(§[4\.6](https://arxiv.org/html/2607.15557#S4.SS6.SSS0.Px3)\)\.

#### Consistent positive effect across benchmarks\.

We quantify the effect at the benchmark level by averaging each task’s paired delta across the four cells and computing a task\-clustered standard error,SE=std/n\\mathrm\{SE\}=\\mathrm\{std\}/\\sqrt\{n\}over thenntasks, so that repeated measurements of the same task across cells are not treated as independent observations\. This is the more conservative of two reasonable clusterings\. Pooling the cell–task deltas as independent observations would roughly halve the standard errors, and the positive effect holds under either choice\. All three benchmarks show a consistent positive effect, each exceeding twice its standard error \(bottom row of Table[1](https://arxiv.org/html/2607.15557#S4.T1)\)\. Effect size broadly tracks the headroom a benchmark leaves\. SkillsBench starts near a10%10\\%no\-skill baseline and gains the most, and within it the low\-baseline Industrial & Physical category gains most \(\+45\.5\+45\.5pp\), whereas GDPVal and QwenClawBench start at7070–85%85\\%and gain less, where ceiling effects and judge noise both bound the realisable gain\. Per\-task repetition noise is large on the high\-baseline benchmarks, with per\-task reward standard deviations of2020–2727pp on GDPVal, so most individual cells are not separately distinguishable from zero\. We do not observe the negative tail thatLiet al\.\([2026b](https://arxiv.org/html/2607.15557#bib.bib18)\)reports for hand\-attached skills, nor the failure to improve over a no\-skill baseline reported for uncurated community libraries\(Liet al\.[2025](https://arxiv.org/html/2607.15557#bib.bib19); Liuet al\.[2026b](https://arxiv.org/html/2607.15557#bib.bib20)\), a difference consistent with the release gates that remove the low\-quality mass driving them\. Raven outgains OpenClaw on SkillsBench and QwenClawBench across both backbones\. On GDPVal neither harness leads, as the gains there are small and individually not significant\. The harness thus shapes how much of the gain is realised \(§[4\.6](https://arxiv.org/html/2607.15557#S4.SS6)\)\.

### 4\.3Frontier robustness check

The four main\-grid cells span the open capability range \(Q\-27B, Q\-397B\)\. To verify that the pipeline’s effect is not specific to this range, we add a single\-cell check on Claude Opus 4\.7 on SkillsBench \(OpenClaw harness\) under the same two\-condition protocol\. Opus 4\.7 reaches a39\.1%39\.1\\%no\-skill baseline on the8787\-task set, and adding SkillCorpus shifts pass@1 to47\.1%47\.1\\%, aΔ\\Deltaof\+8\.0\+8\.0pp\. Although the Opus baseline is several times the open\-backbone baselines, its gain is of the same order as the open\-backbone cross\-cell mean, indicating that stronger frontier models have not made an external curated skill pipeline redundant on this benchmark\. We treat this as a robustness check, and a broader frontier grid is left for follow\-up work\.

### 4\.4Standalone retrieval

End\-to\-endΔ\\Deltaentangles corpus quality with retrieval quality\. We additionally evaluate the recall\-and\-rerank stack as a standalone retrieval system on the SkillRouter Easy and Hard tiers\(Zhenget al\.[2026](https://arxiv.org/html/2607.15557#bib.bib24)\)and on our SkillCorpus pool, a 75\-task core SkillsBench set\. Figure[4](https://arxiv.org/html/2607.15557#S4.F4)reports Hit@1 and Recall@10 across the three pools, with full setup and per\-method results in Appendix[B\.1](https://arxiv.org/html/2607.15557#A2.SS1)\.

![Refer to caption](https://arxiv.org/html/2607.15557v1/x3.png)Figure 4:Standalone retrieval performance ofBase,SkillRouter, and our fine\-tuned stack on the three candidate pools \(Hit@1 / Recall@10, exact numbers in Table[7](https://arxiv.org/html/2607.15557#A2.T7)\)\.Despite being trained on a different pool, our stack matchesSkillRouteron the Easy tier and exceeds it on the Hard tier, staying within1\.71\.7pp on Recall@10\. On SkillCorpus it outperformsSkillRouterby\+10\.7\+10\.7pp Hit@1 and\+2\.7\+2\.7pp Recall@10, a margin that partly reflects the matched train and evaluation distribution and the ground\-truth skills injected into this pool \(Appendix[B\.1](https://arxiv.org/html/2607.15557#A2.SS1)\)\.

### 4\.5Ablation: corpus and retrieval

To gauge how much of the gain comes from corpus curation versus the fine\-tuned retrieval stack, we lesion each in turn on the strongest cell \(Raven×\\timesQ\-397B, SkillsBench\)\. Replacing our retrieval stack with off\-the\-shelf Qwen3 embedding and reranking models lowers pass@1 from22\.6%22\.6\\%to13\.8%13\.8\\%, and replacing the filtered corpus with the∼\\sim821,000821\{,\}000\-file raw crawl lowers it to14\.9%14\.9\\%\(Table[2](https://arxiv.org/html/2607.15557#S4.T2)\)\. Both remain well above the9\.2%9\.2\\%no\-skill baseline yet well below the full pipeline, so curation and retrieval each contribute and neither alone accounts for the full gain, though the end\-to\-endΔ\\Deltaalso folds in the LLM rewriter/selector layer that this ablation does not isolate\. These are single\-run numbers with inter\-arm gaps of only a few tasks, so we read them as directional and leave a repeated ablation to future work\.

Table 2:Single\-run ablation on the strongest cell \(Raven×\\timesQ\-397B, SkillsBench, pass@1 in%\\%over8787tasks\)\. The full pipeline pairs the filtered corpus with our fine\-tuned stack, and each lesion swaps exactly one component\. Replacing either the corpus \(filtered→\\toraw\) or the retrieval stack \(ours→\\tooff\-the\-shelf\) drops the single\-run gain to roughly14%14\\%\.
### 4\.6Discussion: where the pipeline helps, where it does not

The per\-skill quality score and 16\-class taxonomy play supporting roles rather than driving the gains directly\. The taxonomy steers selection benchmark\-adaptively, its post\-gate class distribution concentrating on each benchmark’s task\-relevant classes \(Appendix[B\.3](https://arxiv.org/html/2607.15557#A2.SS3)\)\. The composite quality score serves deduplication tie\-breaks and retrieval\-time ranking rather than per\-task prediction, and the active set is gated by safety and licence, not by a score threshold, with only a weak safety\-facet signal \(Appendix[A\.2](https://arxiv.org/html/2607.15557#A1.SS2)\)\.

#### When the pipeline helps\.

Across cells, the per\-cellΔ\\Deltais largest when the benchmark contains procedural sub\-tasks beyond the model’s pretraining coverage and the corpus covers the task’s domain\. SkillsBench and the QwenClawBenchDevOps\-Infra/Testing/Metasubset \(Figure[7](https://arxiv.org/html/2607.15557#A2.F7)\) satisfy both and show the largest cell\-level gains\.

#### Why the harness matters\.

Both harnesses receive identical pre\-computed skill selections, and traces show that both ingest them, reading and referencing the injected content\. The difference is what happens afterwards\. Raven realises a far larger gain than OpenClaw on SkillsBench \(\+13\.4\+13\.4versus\+5\.7\+5\.7pp on Q\-397B\), although both receive the same skills\. The gain also grows with backbone capability, though entangled with the harness\. Raven’s\+6\.5\+6\.5at Q\-27B rises to\+13\.4\+13\.4at Q\-397B, and the frontier Opus check adds\+8\.0\+8\.0pp on OpenClaw, so how much of a skill is realised depends on both the harness and the backbone\. What follows is a qualitative case analysis of the two Q\-397B cells, not an established mechanism\. On the tasks that Raven passes and OpenClaw fails, trace inspection suggests OpenClaw consistently stops after the reasoning phase, writing scripts it never executes and ending without running the verifier, whereas Raven completes an execute–verify–fix loop\. This is consistent with skill content converting into verifier passes mainly through such a loop, so that skill utility appears to be a joint property of the corpus and the harness, in line with the harness\-utilisation differencesLiet al\.\([2026b](https://arxiv.org/html/2607.15557#bib.bib18)\)reports across commercial harnesses\. On this reading, OpenClaw’s weaker gains on SkillsBench are a harness property rather than a corpus one\. The same execution gap carries no penalty on GDPVal, whose LLM\-judged writing and document tasks reward output quality rather than a verifier\-passing loop, and there the harness gap closes\.

#### Coverage shapes the gain\.

The gains track how well the corpus covers each task\. As a proxy we take the relevance of the best\-retrieved skill \(its top reranker score\), which reflects both whether the corpus holds a matching skill and whether retrieval surfaces it\. Binning tasks by this retrieval\-match score, meanΔ\\Deltaclimbs from\+2\.2\+2\.2pp to\+25\.1\+25\.1pp from the lowest to the highest bin \(Figure[5](https://arxiv.org/html/2607.15557#S4.F5), pooled over the four cells\)\. Per task the score correlates withΔ\\Deltaat Pearsonr=0\.31r\{=\}0\.31–0\.400\.40across all four cells \(all\|t\|\>2\.9\|t\|\{\>\}2\.9,n=83n\{=\}83\), and since it is essentially uncorrelated with the no\-skill baseline the association survives controlling for task difficulty \(partialr=0\.34r\{=\}0\.34–0\.400\.40; Appendix[B\.2](https://arxiv.org/html/2607.15557#A2.SS2)\)\. At the 8\-category granularity the gains stay non\-negative throughout, with two thin\-coverage categories on the Raven×\\timesQ\-397B cell atΔ≈0\\Delta\{\\approx\}0\(Mathematics,N=4N\{=\}4; Finance & Economics,N=9N\{=\}9\)\. Where the corpus lacks skills for a category, retrieval cannot manufacture them, so the gain floors at zero rather than turning negative\. Closing these gaps is a supply problem rather than a retrieval one, calling for generating or self\-evolving the skills the crawl does not yet contain rather than better matching over the current corpus\. The rare per\-task breaks \(two on this cell\) stem from procedure–task mismatch rather than missing coverage, and Appendix[B\.4](https://arxiv.org/html/2607.15557#A2.SS4)dissects one \(exceltable\-in\-ppt\)\.

![Refer to caption](https://arxiv.org/html/2607.15557v1/x4.png)Figure 5:Mean per\-task SkillsBench gain by retrieval\-match bin \(top reranker score of the best\-matched skill, a proxy for corpus coverage\), pooled over the four cells, with±\\pmSE error bars\. The score is uncorrelated with the no\-skill baseline, so the bins are not a headroom artefact\. The per\-task correlation and raw scatter are in Appendix[B\.2](https://arxiv.org/html/2607.15557#A2.SS2)\.

## 5Conclusion

SkillCorpus curates∼\\sim821,000821\{,\}000crawled community skill files into a corpus of96,40196\{,\}401skills, scored along utility, robustness, and safety and matched to tasks by a fine\-tuned retrieval\-and\-selection stack\. Across our cross\-benchmark, cross\-harness, multi\-backbone evaluation \(§[4](https://arxiv.org/html/2607.15557#S4)\), integrating the corpus yields consistent positive gains on all three benchmarks\. They are large on SkillsBench and modest on GDPVal and QwenClawBench, strongest on the Raven harness, with per\-domain effects that remain heterogeneous\. Where coverage is thin, growing the corpus by skill generation is the natural next step\. The 16\-class taxonomy steers benchmark\-aware retrieval\. The composite quality score serves curation and ranking rather than per\-task prediction, and its safety facet is the one component that both gates unsafe skills and carries a suggestive per\-task signal\. We release SkillCorpus as an open, reusable framework for community skill ecosystems; to our knowledge, it is the first end\-to\-end account of when a curated community corpus improves real agent tasks, and where it does not\.

## Limitations

#### Breadth\-over\-depth evaluation design\.

The main grid prioritises breadth across twelve\(harness,backbone,benchmark\)\(\\text\{harness\},\\text\{backbone\},\\text\{benchmark\}\)combinations plus a frontier robustness check\. Depth is uneven by design\. All three benchmarks are run with three repeated evaluations per cell, and we report task\-clustered standard errors over the per\-task paired deltas \(§[4\.2](https://arxiv.org/html/2607.15557#S4.SS2)\)\. On the high\-baseline LLM\-judged benchmarks \(GDPVal, QwenClawBench\) per\-task judge noise exceeds the effect size, so the positive effect is established at the pooled benchmark level rather than cell by cell\. SkillsBench additionally admits paired per\-task statistics \(exact McNemar on a representative run\) and the logistic safety\-facet regression \(Appendix[A\.2](https://arxiv.org/html/2607.15557#A1.SS2)\)\. A larger number of repetitions and seeds per cell would tighten the per\-cell estimates that the pooled analysis currently leans on\.

#### Text\-based skill scoring\.

The 3\-facet quality framework and 19\-flag taxonomy are evaluated by an LLM judge against theSKILL\.mdtext content rather than by sandboxed execution\. This is a deliberate scale trade\-off\. Judge\-time text scoring is feasible at the∼\\sim821,000821\{,\}000\-file crawl scale, whereas runtime sandboxing would require per\-skill compatible execution environments not generally available for community\-contributed code\. Safety hard\-gates and licence filtering at Stage 5 catch the most severe content\-level risks at ingest time\. Complementary runtime verification at retrieval\-time or agent\-execution\-time is a principled extension for future work\. A second consequence of the scale trade\-off is that the quality, safety, and classification labels driving the release decision come from a single LLM judge and are not validated against human gold annotations or cross\-checked against an alternative judge model\. Judge bias therefore propagates into the released active set\. The facet\-separation analysis is an internal\-consistency check, not a construct\-validity study, and human\-annotation and judge\-robustness studies are left to future work\.

#### Benchmark and harness coverage\.

The evaluation spans three publicly available benchmarks \(SkillsBench, GDPVal, QwenClawBench\) chosen for non\-saturation on frontier LLMs and for complementary task\-type coverage \(deterministic\-verifier software tasks, real\-world knowledge work, and a real\-user\-distribution agent benchmark\), and twoSKILL\.md\-conformant agent harnesses \(OpenClaw, Raven\) covering the most widely adopted skill\-loading convention in the public ecosystem\. Per\-benchmark and per\-harness heterogeneity within this coverage is reported transparently in the main results \(§[4\.6](https://arxiv.org/html/2607.15557#S4.SS6.SSS0.Px3)\) rather than relegated to this section\. Extending the evaluation to additional task types \(e\.g\., long\-horizon embodied tasks, non\-English workflows\) and to harnesses with substantially different tool\-use interfaces or planning architectures would broaden external validity\. The corpus is also English\-dominant, both in its source repositories and in our evaluation, which bounds claims about non\-English skills\. The corpus\-side findings \(Stage 5 active–vs–filtered separation, class\-adaptive retrieval, and the suggestive safety\-facet signal\) are expected to transfer modulo such adapter work, while per\-cellΔ\\Deltamagnitudes would re\-anchor against each new harness’s own no\-skill baseline\.

#### Snapshot semantics\.

The released active set is a snapshot of the publicSKILL\.mdecosystem at crawl time, while the community itself continues to evolve \(new repositories, licence updates, shifting class distributions\)\. We do not study temporal churn or re\-curation cycles in this work, but the pipeline \(per\-stage reject logs, deterministic Stage 5 cut, deduplication keyed on content hash and embedding cosine\) is constructed to support periodic re\-runs against future ecosystem snapshots\. The active set released with this paper should be read as a 2026\-Q2 snapshot rather than a one\-shot artifact\.

## Ethical Statement

#### Data provenance and licensing\.

SkillCorpus aggregates publicly availableSKILL\.mdfiles from GitHub and community marketplaces that link to GitHub\-hosted sources\. Each released skill carries metadata including its source URL and content hash\. We do not modify body content or claim authorship\. Every released skill carries an OSI\-approved permissive licence inferred from the source repository’s GitHubspdx\_idand enforced as a Stage\-5 ingest filter \(Table[4](https://arxiv.org/html/2607.15557#A1.T4), per\-reason exclusion counts in Table[5](https://arxiv.org/html/2607.15557#A1.T5)\)\. Downstream consumers redistributing the corpus or its derivatives are advised to perform their own due\-diligence audit ofscripts/content and transitive dependencies\.

#### Safety filtering and dual\-use\.

The Stage\-5 safety hard\-gate \(§[3\.3](https://arxiv.org/html/2607.15557#S3.SS3)\) excludes the most severe categories from the active release set, and the soft\-flag categories are recorded in each skill’s metadata for downstream per\-deployment filtering\. We acknowledge the dual\-use risk that a publicly searchable skill corpus could be queried for adversarial\-style content patterns, a residual risk that the hard\-gate and released soft\-flag metadata reduce but do not eliminate\.

#### Benchmark contamination\.

SkillCorpus is community\-contributed and pre\-dates our benchmark selection \(§[4\.1](https://arxiv.org/html/2607.15557#S4.SS1)\)\. By design the corpus aims for broad coverage of the skill ecosystem, so released skills overlap the domains of our evaluation tasks\. These are general procedural knowledge rather than task\-specific solutions, and the per\-task gate injects at most a few topically relevant skills\. We do not perform task\-specific leakage audits beyond the corpus\-level safety and licence filters described above, and consumers using verifier\-backed benchmarks should re\-audit for task\-specific contamination when their evaluation regime is sensitive to it\.

## References

- Anthropic skill creator: a rubric for agent skill quality\.Note:anthropic\.com/blog, 2025\.Cited by:[§1](https://arxiv.org/html/2607.15557#S1.p1.1),[§1](https://arxiv.org/html/2607.15557#S1.p2.1),[§3\.1](https://arxiv.org/html/2607.15557#S3.SS1.p2.1)\.
- Anthropic \(2025b\)Claude Code: an agentic coding tool\.Note:github\.com/anthropics/claude\-code, 2025\.Cited by:[§1](https://arxiv.org/html/2607.15557#S1.p2.1)\.
- C\. Chen, Q\. Yu, Y\. Gu, Z\. Huang, H\. Li, H\. Liu, S\. Liu, J\. Liu, D\. Peng, J\. Wang, Z\. Yan, F\. Meng, E\. Qin, C\. Che, and M\. Hu \(2026\)The scaling laws of skills in LLM agent systems\.arXiv preprint arXiv:2605\.16508\.External Links:[Link](https://arxiv.org/abs/2605.16508)Cited by:[§2\.2](https://arxiv.org/html/2607.15557#S2.SS2.p1.4)\.
- H\. Cho, R\. Kang, and Y\. Kim \(2026\)SkillRet: a large\-scale benchmark for skill retrieval in LLM agents\.arXiv preprint arXiv:2605\.05726\.External Links:[Link](https://arxiv.org/abs/2605.05726)Cited by:[§2\.2](https://arxiv.org/html/2607.15557#S2.SS2.p1.4)\.
- X\. Hou, Y\. Zhao, S\. Wang, and H\. Wang \(2025\)Model context protocol \(MCP\): landscape, security threats, and future research directions\.ACM Transactions on Software Engineering and Methodology\.Note:arXiv:2503\.23278External Links:[Document](https://dx.doi.org/10.1145/3796519),[Link](https://arxiv.org/abs/2503.23278)Cited by:[§1](https://arxiv.org/html/2607.15557#S1.p1.1)\.
- K\. Lee, D\. Ippolito, A\. Nystrom, C\. Zhang, D\. Eck, C\. Callison\-Burch, and N\. Carlini \(2022\)Deduplicating training data makes language models better\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(ACL\),Note:arXiv:2107\.06499Cited by:[§2\.1](https://arxiv.org/html/2607.15557#S2.SS1.p1.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela \(2020\)Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2005\.11401Cited by:[§3\.4](https://arxiv.org/html/2607.15557#S3.SS4.p1.1)\.
- F\. Li, P\. Tagkopoulos, and I\. Tagkopoulos \(2025\)SkillFlow: scalable and efficient agent skill retrieval system\.arXiv preprint arXiv:2504\.06188\.External Links:[Link](https://arxiv.org/abs/2504.06188)Cited by:[§1](https://arxiv.org/html/2607.15557#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.15557#S2.SS1.p2.1),[§4\.2](https://arxiv.org/html/2607.15557#S4.SS2.SSS0.Px1.p1.8)\.
- H\. Li, C\. Mu, J\. Chen, S\. Ren, Z\. Cui, Y\. Zhang, L\. Bai, and S\. Hu \(2026a\)Organizing, orchestrating, and benchmarking agent skills at ecosystem scale\.arXiv preprint arXiv:2603\.02176\.External Links:[Link](https://arxiv.org/abs/2603.02176)Cited by:[§2\.2](https://arxiv.org/html/2607.15557#S2.SS2.p1.4)\.
- X\. Li, W\. Chen, Y\. Liu, S\. Zheng, X\. Chen, Y\. He, Y\. Li, B\. You, H\. Shen, J\. Sun,et al\.\(2026b\)SkillsBench: benchmarking how well agent skills work across diverse tasks\.arXiv preprint arXiv:2602\.12670\.External Links:[Link](https://arxiv.org/abs/2602.12670)Cited by:[§1](https://arxiv.org/html/2607.15557#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.15557#S2.SS2.p2.1),[§4\.1](https://arxiv.org/html/2607.15557#S4.SS1.SSS0.Px1.p1.7),[§4\.2](https://arxiv.org/html/2607.15557#S4.SS2.SSS0.Px1.p1.8),[§4\.2](https://arxiv.org/html/2607.15557#S4.SS2.p2.9),[§4\.6](https://arxiv.org/html/2607.15557#S4.SS6.SSS0.Px2.p1.5)\.
- Y\. Liang, R\. Zhong, H\. Xu, C\. Jiang, Y\. Zhong, R\. Fang, J\. Gu, S\. Deng, Y\. Yao, M\. Wang, S\. Qiao, X\. Xu, T\. Wu, K\. Wang, Y\. Liu,et al\.\(2026\)SkillNet: create, evaluate, and connect AI skills\.arXiv preprint arXiv:2603\.04448\.External Links:[Link](https://arxiv.org/abs/2603.04448)Cited by:[§1](https://arxiv.org/html/2607.15557#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.15557#S2.SS1.p2.1),[§3\.2](https://arxiv.org/html/2607.15557#S3.SS2.SSS0.Px3.p1.6)\.
- M\. Lin, H\. Lu, Z\. Shi, B\. He, R\. Mao, Z\. Zhang, Z\. Wu, X\. Tang, H\. Liu, Z\. Dai, X\. Zhang, S\. Wang, B\. Dumoulin, and J\. Pei \(2026\)Position: agentic evolution is the path to evolving LLMs\.arXiv preprint arXiv:2602\.00359\.External Links:[Link](https://arxiv.org/abs/2602.00359)Cited by:[§2\.1](https://arxiv.org/html/2607.15557#S2.SS1.p1.1)\.
- Z\. Lin, B\. Ruan, J\. Liu, and W\. Zhao \(2025\)A large\-scale evolvable dataset for model context protocol ecosystem and security analysis\.arXiv preprint arXiv:2506\.23474\.External Links:[Link](https://arxiv.org/abs/2506.23474)Cited by:[§1](https://arxiv.org/html/2607.15557#S1.p1.1)\.
- G\. Ling, S\. Zhong, and R\. Huang \(2026\)Agent skills: a data\-driven analysis of Claude skills for extending large language model functionality\.arXiv preprint arXiv:2602\.08004\.External Links:[Link](https://arxiv.org/abs/2602.08004)Cited by:[§2\.1](https://arxiv.org/html/2607.15557#S2.SS1.p2.1)\.
- D\. Liu, Z\. Li, H\. Du, X\. Wu, S\. Gui, Y\. Kuang, and L\. Sun \(2026a\)Graph of skills: dependency\-aware structural retrieval for massive agent skills\.arXiv preprint arXiv:2604\.05333\.External Links:[Link](https://arxiv.org/abs/2604.05333)Cited by:[§2\.2](https://arxiv.org/html/2607.15557#S2.SS2.p1.4)\.
- Y\. Liu, J\. Ji, L\. An, T\. Jaakkola, Y\. Zhang, and S\. Chang \(2026b\)How well do agentic skills work in the wild: benchmarking LLM skill usage in realistic settings\.arXiv preprint arXiv:2604\.04323\.External Links:[Link](https://arxiv.org/abs/2604.04323)Cited by:[§1](https://arxiv.org/html/2607.15557#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.15557#S2.SS2.p2.1),[§4\.2](https://arxiv.org/html/2607.15557#S4.SS2.SSS0.Px1.p1.8)\.
- X\. Meng, S\. Wang, and Y\. Fang \(2026\)SkillRAE: agent skill\-based context compilation for retrieval\-augmented execution\.arXiv preprint arXiv:2605\.10114\.External Links:[Link](https://arxiv.org/abs/2605.10114)Cited by:[§2\.2](https://arxiv.org/html/2607.15557#S2.SS2.p1.4)\.
- M\. A\. Merrill, A\. G\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi, J\. Y\. Shin, T\. Walshe, E\. K\. Buchanan,et al\.\(2026\)Terminal\-Bench: benchmarking agents on hard, realistic tasks in command line interfaces\.arXiv preprint arXiv:2601\.11868\.External Links:[Link](https://arxiv.org/abs/2601.11868)Cited by:[§1](https://arxiv.org/html/2607.15557#S1.p1.1)\.
- OpenAI \(2025a\)Codex CLI: lightweight coding agent that runs in your terminal\.Note:github\.com/openai/codex, 2025\.Cited by:[§1](https://arxiv.org/html/2607.15557#S1.p2.1)\.
- OpenAI \(2025b\)GDPval: measuring LLM performance on real\-world economic tasks\.Note:openai\.com/research, 2025\.Cited by:[§1](https://arxiv.org/html/2607.15557#S1.p1.1),[§4\.1](https://arxiv.org/html/2607.15557#S4.SS1.SSS0.Px1.p1.7)\.
- J\. S\. Park, J\. C\. O’Brien, C\. J\. Cai, M\. R\. Morris, P\. Liang, and M\. S\. Bernstein \(2023\)Generative agents: interactive simulacra of human behavior\.InUIST,External Links:[Link](https://arxiv.org/abs/2304.03442)Cited by:[§1](https://arxiv.org/html/2607.15557#S1.p1.1)\.
- S\. G\. Patil, T\. Zhang, X\. Wang, and J\. E\. Gonzalez \(2023\)Gorilla: large language model connected with massive APIs\.arXiv preprint arXiv:2305\.15334\.External Links:[Link](https://arxiv.org/abs/2305.15334)Cited by:[§1](https://arxiv.org/html/2607.15557#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.15557#S2.SS1.p1.1)\.
- Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian,et al\.\(2024\)ToolLLM: facilitating large language models to master 16000\+ real\-world apis\.InICLR,External Links:[Link](https://arxiv.org/abs/2307.16789)Cited by:[§1](https://arxiv.org/html/2607.15557#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.15557#S2.SS1.p1.1)\.
- Qwen Team, Alibaba Group \(2026\)QwenClawBench: a real\-user\-distribution agent benchmark for OpenClaw\.Note:Open\-source benchmark, github\.com/SKYLENAGE\-AI/QwenClawBench, released April 2026\.External Links:[Link](https://github.com/SKYLENAGE-AI/QwenClawBench)Cited by:[§4\.1](https://arxiv.org/html/2607.15557#S4.SS1.SSS0.Px1.p1.7),[§4\.1](https://arxiv.org/html/2607.15557#S4.SS1.SSS0.Px2.p1.8)\.
- Raven Contributors \(2026\)Raven: an open\-source agent framework\.Note:Open\-source repository; supportsSKILL\.mdskill\-loading convention\.External Links:[Link](https://github.com/EverMind-AI/Raven)Cited by:[§4\.1](https://arxiv.org/html/2607.15557#S4.SS1.SSS0.Px1.p1.7)\.
- T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom \(2023\)Toolformer: language models can teach themselves to use tools\.InNeurIPS,External Links:[Link](https://arxiv.org/abs/2302.04761)Cited by:[§2\.1](https://arxiv.org/html/2607.15557#S2.SS1.p1.1)\.
- N\. Shinn, F\. Cassano, E\. Berman, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.NeurIPS\.External Links:[Link](https://arxiv.org/abs/2303.11366)Cited by:[§1](https://arxiv.org/html/2607.15557#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.15557#S2.SS1.p1.1)\.
- P\. Steinberger \(2025\)OpenClaw: an open\-source autonomous agent framework\.Note:Originally published Nov 2025 as Clawdbot; renamed OpenClaw 2026\. Skill\-loading viaSKILL\.mdconvention\.External Links:[Link](https://github.com/openclaw/openclaw)Cited by:[§4\.1](https://arxiv.org/html/2607.15557#S4.SS1.SSS0.Px1.p1.7)\.
- G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2024\)Voyager: an open\-ended embodied agent with large language models\.Transactions on Machine Learning Research\.External Links:[Link](https://arxiv.org/abs/2305.16291)Cited by:[§2\.1](https://arxiv.org/html/2607.15557#S2.SS1.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.ICLR\.External Links:[Link](https://arxiv.org/abs/2210.03629)Cited by:[§1](https://arxiv.org/html/2607.15557#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.15557#S2.SS1.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2306\.05685Cited by:[§3\.2](https://arxiv.org/html/2607.15557#S3.SS2.SSS0.Px4.p1.6)\.
- Y\. Zheng, Z\. Zhang, C\. Ma, Y\. Yu, J\. Zhu, Y\. Wu, T\. Xu, B\. Dong, H\. Zhu, R\. Huang, and G\. Yu \(2026\)SkillRouter: skill routing for LLM agents at scale\.arXiv preprint arXiv:2603\.22455\.External Links:[Link](https://arxiv.org/abs/2603.22455)Cited by:[§B\.1](https://arxiv.org/html/2607.15557#A2.SS1.p1.2),[§2\.2](https://arxiv.org/html/2607.15557#S2.SS2.p1.4),[§4\.4](https://arxiv.org/html/2607.15557#S4.SS4.p1.1)\.
- C\. Zhou, P\. Liu, P\. Xu, S\. Iyer, J\. Sun, Y\. Mao, X\. Ma, A\. Efrat, P\. Yu, L\. Yu, S\. Zhang, G\. Ghosh, M\. Lewis, L\. Zettlemoyer, and O\. Levy \(2023\)LIMA: less is more for alignment\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2305\.11206Cited by:[§2\.1](https://arxiv.org/html/2607.15557#S2.SS1.p1.1)\.
- H\. Zhou, S\. Guo, A\. Liu, Z\. Yu, Z\. Gong, B\. Zhao, Z\. Chen, M\. Zhang, Y\. Chen, J\. Li,et al\.\(2026\)Memento\-Skills: let agents design agents\.arXiv preprint arXiv:2603\.18743\.External Links:[Link](https://arxiv.org/abs/2603.18743)Cited by:[§2\.1](https://arxiv.org/html/2607.15557#S2.SS1.p1.1)\.

## Appendix ACorpus Construction Details

### A\.1Quality Flag Vocabulary

The 19\-flag taxonomy used by the LLM quality judge is structured by its primary semantic dimension\. Flags marked†are hard\-gate flags whose firing forces the per\-skill quality score to zero and the skill to be excluded from the released active set\.

- •Utility\-bound \(2\):marketing\_only,vague\_purpose\.
- •Robustness\-bound \(6\):placeholder,no\_steps,deprecated\_api,fabricated\_call,syntax\_error,inconsistent\_doc\_code\.
- •Safety\-bound \(11\):destructive\_no\_confirm,secret\_leak,network\_exfil,prompt\_injection†,cmd\_injection†,unsafe\_exec†,auth\_bypass†,fact\_poisoning,tos\_violation,csam\_risk†,bias\_content\.

Flag classification follows the LLM judge prompt\. The same flag may in principle constrain more than one facet \(e\.g\., aplaceholderskill primarily caps the robustness facet but may also indicate poor utility scope\), but for scoring we use the single\-facet assignment shown above\.

#### Hard\-gate vs soft\-signal rationale\.

The five hard\-gate flags \(prompt\_injection,cmd\_injection,unsafe\_exec,auth\_bypass,csam\_risk\) share two properties that distinguish them from the 14 soft signals: \(i\) a single firing is sufficient to make the skill unsafe to ship even in conjunction with downstream sandboxing or partial\-load policies, and \(ii\) remediation cannot be deferred to per\-deployment filtering without re\-vetting the body content\. Soft signals such assecret\_leakordestructive\_no\_confirm, by contrast, can be mitigated by deployment\-side wrappers \(credential rotation, confirmation prompts\) or by skipping affected skills in a particular deployment without invalidating the broader artifact\. The distinction is therefore an integrability boundary, not an absolute\-severity ranking\.

### A\.2Quality\-Score Distribution and Facet Separation

![Refer to caption](https://arxiv.org/html/2607.15557v1/x5.png)Figure 6:Three\-facet independence on the active set \(N=96,401N\{=\}96\{,\}401\)\. Bar length is the Pearsonrrfor each facet pair \(dashed line atr=0r\{=\}0marks statistical independence\)\. Italic annotations report off\-diagonal massPr⁡\(a≥8∧b≤4\)\\Pr\(a\{\\geq\}8\\wedge b\{\\leq\}4\)versus the independence baseline\.Whether the 3\-facet prompt produces genuinely separate scores is subtler than Pearsonrralone can answer, because real skills share latent craftsmanship that elevates all facets simultaneously, so some positive correlation is expected even under independent measurement\. For each facet pair we report both Pearsonrrand the off\-diagonal massPr⁡\(a≥8∧b≤4\)\\Pr\(a\{\\geq\}8\\wedge b\{\\leq\}4\)against the product\-of\-marginals independence baseline, computed on the96,40196\{,\}401\-skill active set and visualised in Figure[6](https://arxiv.org/html/2607.15557#A1.F6)\. Observed off\-diagonal mass sits below the independence baseline for all three pairs\. Among the craftsmanship\-coupled pair, the prompt still isolates a2\.85%2\.85\\%“description\-good / body\-weak” quadrant \(u≥8∧r≤4u\{\\geq\}8\\wedge r\{\\leq\}4\) that a one\-dimensional composite score cannot represent\. Because the subscores are integer0–1010values subject to soft\-flag capping, the Pearsonrris an attenuated estimate of association\.

#### Per\-task outcome relationship\.

The composite score does not predict per\-task pass rate on any benchmark \(all\|r\|<0\.10\|r\|<0\.10,p\>0\.35p\>0\.35\)\. Only the safety facet carries a suggestive per\-task signal, a pooled SkillsBench logistic givingβ^safety=\+1\.03\\hat\{\\beta\}\_\{\\mathrm\{safety\}\}\{=\}\+1\.03\(z=\+2\.29z\{=\}\+2\.29,p=0\.022p\{=\}0\.022\) that rests on a single grid and would not survive a multiple\-comparison correction, so whether it can serve as a ranking cue beyond topical relevance merits validation on further grids\.

#### Score distribution\.

On the96,40196\{,\}401active skills the composite quality score \(on the\[0,1\]\[0,1\]scale\) has mean0\.6900\.690; the bucket\-level distribution is in Table[3](https://arxiv.org/html/2607.15557#A1.T3)\.

Table 3:Quality\-bucket distribution on the active set\.

### A\.3Source Prior

Each sourcessin the corpus is assigned a prior

𝚙𝚛𝚒𝚘𝚛​\(s\)=\\displaystyle\\mathtt\{prior\}\(s\)=\\;0\.95⋅𝚝𝚛𝚊𝚌𝚔​\(s\)\\displaystyle 0\.95\\cdot\\mathtt\{track\}\(s\)\+0\.05⋅𝚕𝚒𝚌​\_​𝚛𝚊𝚝𝚎​\(s\)\\displaystyle\+0\.05\\cdot\\mathtt\{lic\\\_rate\}\(s\)\+0\.10⋅𝟙​\[s∈𝒯\],\\displaystyle\+0\.10\\cdot\\mathbb\{1\}\[s\\in\\mathcal\{T\}\],clipped to\[0\.30,1\.00\]\[0\.30,1\.00\], where:

- •𝚝𝚛𝚊𝚌𝚔​\(s\)=n​q¯s\+K​μ0n\+K\\mathtt\{track\}\(s\)=\\dfrac\{n\\bar\{q\}\_\{s\}\+K\\mu\_\{0\}\}\{n\+K\}is the Bayesian\-shrunk source quality, withnnthe number of skills fromss,q¯s∈\[0,1\]\\bar\{q\}\_\{s\}\\in\[0,1\]the source’s empirical normalised content\-quality mean \(§[3\.2](https://arxiv.org/html/2607.15557#S3.SS2)\),K=10K=10a shrinkage strength in skill\-count units, andμ0=0\.685\\mu\_\{0\}=0\.685the dataset\-wide normalised content\-quality mean\.
- •𝚕𝚒𝚌​\_​𝚛𝚊𝚝𝚎​\(s\)∈\[0,1\]\\mathtt\{lic\\\_rate\}\(s\)\\in\[0,1\]is the fraction ofss’s skills carrying an OSI\-permissive licence\.
- •𝒯\\mathcal\{T\}is a small explicit trust list \(anthropics,antigravity,karanb192\)\. The indicator contributes at most\+0\.10\+0\.10to dampen the influence of hand\-curation relative to the data\-driven track score\.

K=10K=10was chosen*a priori*from the heuristic "ten observations is the minimum credible sample"\. We did not tune it on downstream task performance\.

### A\.4False\-Positive Audit of Safety Regex Patterns

The Stage 5 safety hard\-gate applies a single pre\-judge malware regex \(blocked\.malware\)\. A set of additional regex patterns \(e\.g\., suspicious\-string heuristics\) are retained for audit logging only\. A stratified 100\-skill manual audit indicates that several audit\-only regex patterns have empirical false\-positive rates above90%90\\%, which is why they are not used as filtering gates\. Detailed false\-positive counts per regex pattern are provided in the released ingest report\.

### A\.5Licence Breakdown and Exclusion Counts

All96,40196\{,\}401released skills carry an OSI\-approved permissive licence\. Table[4](https://arxiv.org/html/2607.15557#A1.T4)reports the licence composition of the active set, dominated by MIT \(88\.3%88\.3\\%\) and Apache\-2\.0 \(10\.5%10\.5\\%\)\. Table[5](https://arxiv.org/html/2607.15557#A1.T5)itemises the3,7953\{,\}795skills excluded at the Stage\-5 licence filter by exclusion reason\. MissingLICENSEfiles and unreachable source repositories account for∼\\sim87%87\\%of exclusions, while explicitly incompatible terms \(copyleft or non\-commercial/share\-alike\) account for under9%9\\%\.

Table 4:Licence breakdown of the released active set \(N=96,401N\{=\}96\{,\}401\)\.†CC0 / BSD / MPL\-2\.0 / Unlicense / ISC / Mulan\-PSL\-2\.0 / WTFPL / 0BSD\.Table 5:Skills excluded at the Stage 5 licence filter to guarantee commercial redistributability\.
### A\.6Crawl Source Inventory

The SkillCorpus crawl \(§[3\.1](https://arxiv.org/html/2607.15557#S3.SS1)\) is configured by a single machine\-readable registry of6262sources\. Each entry declares one of five ingestion mechanisms, and all mechanisms feed a shared discovery entry point, so adding a source is a one\-line registry change\. The mechanisms are itemised below with per\-mechanism counts and representative sources\.

Table 6:Crawl sources by ingestion mechanism\. The complete registry ships with the released artifact as a machine\-readable file\.

## Appendix BRetrieval and Evaluation Details

### B\.1Standalone Retrieval Performance

We evaluate the recall\-and\-rerank stack as a standalone retrieval system on three candidate pools using the SkillRouter evaluation protocol\(Zhenget al\.[2026](https://arxiv.org/html/2607.15557#bib.bib24)\): the SkillRouter Easy tier \(standard large\-pool retrieval\), the SkillRouter Hard tier \(with adversarial distractors\), and our own SkillCorpus pool \(with the ground\-truth SkillsBench skills explicitly injected at evaluation time but excluded from training and from the downstream agent\)\. We compare against two baselines:Base\(off\-the\-shelf Qwen3\-Emb\-0\.6B and Qwen3\-Rank\-0\.6B\) andSkillRouter\(Zhenget al\.[2026](https://arxiv.org/html/2607.15557#bib.bib24)\)using its released models\. Performance is measured with Hit@1 and Recall@10 on the7575core SkillsBench tasks \(excluding1212generic\-only cases per the SkillRouter protocol\)\.

Table 7:Standalone retrieval performance \(Hit@1 and Recall@10 on the7575\-task core SkillsBench evaluation set\)\.The Easy / Hard performance shows cross\-pool generalisability, since our stack is competitive on tiers it was not trained on\. The SkillCorpus performance reflects the expected advantage of train\-time and evaluation\-time distributions matching\. The gap between Easy and SkillCorpus for bothSkillRouterandOurs\(−14\.7\-14\.7and−4\.0\-4\.0pp on Hit@1 respectively\) indicates that SkillCorpus is intrinsically harder than the Easy tier, consistent with its larger size and more diverse distractor mass\.

### B\.2Cross\-Benchmark Domain Coverage

Figure[7](https://arxiv.org/html/2607.15557#A2.F7)documents the breadth of task domains the corpus is exercised against\. It plots the407407evaluated tasks over2626domain labels in each benchmark’s native taxonomy \(88SkillsBench categories plus a residual Other bin,99GDPVal sectors, and88QwenClawBench categories\), with bar length giving the task count and bar colour the per\-domainΔ\\Deltaon the strongest cell \(Raven×\\timesQ\-397B\)\. Those per\-domainΔ\\Deltavalues are coarse and noisy atN≤25N\\,\{\\leq\}\\,25per cell, so they mark only where coverage is dense or thin and are not per\-domain effect claims, which the main text makes at the pooled level \(§[4\.2](https://arxiv.org/html/2607.15557#S4.SS2)\)\.

![Refer to caption](https://arxiv.org/html/2607.15557v1/x6.png)Figure 7:Cross\-benchmark domain coverage of the evaluation, each panel in its benchmark’s native taxonomy\.At the task level, the relevance of the best\-retrieved skill predicts the gain more directly than these coarse domain bins\. Beyond the binned summary in the main text \(Figure[5](https://arxiv.org/html/2607.15557#S4.F5)\), Figure[8](https://arxiv.org/html/2607.15557#A2.F8)plots each SkillsBench task’sΔ\\Deltaagainst the top reranker score of its best\-matched skill, a proxy that reflects both whether the corpus holds a matching skill and whether retrieval surfaces it\. The positive slope holds across all four cells \(Pearsonr=0\.31r\{=\}0\.31–0\.400\.40, all\|t\|\>2\.9\|t\|\{\>\}2\.9,n=83n\{=\}83\), the panel showing the strongest cell\. The score is essentially uncorrelated with the no\-skill baseline \(\|r\|<0\.05\|r\|<0\.05\), so partialling out task difficulty leaves the association unchanged \(partialr=0\.34r\{=\}0\.34–0\.400\.40\)\.

![Refer to caption](https://arxiv.org/html/2607.15557v1/x7.png)Figure 8:Per\-task SkillsBench gain versus corpus coverage \(top reranker score of the best\-matched skill\), Raven×\\timesQ\-397B \(n=83n\{=\}83,r=0\.40r\{=\}0\.40\)\.
### B\.3Class\-Benchmark Alignment

Table[8](https://arxiv.org/html/2607.15557#A2.T8)reports the per\-benchmark class over\-selection underlying the class\-adaptive selection result of §[4\.6](https://arxiv.org/html/2607.15557#S4.SS6)\. For each benchmark it lists the skill classes most over\-represented in the post\-gate selected pool relative to their corpus\-wide frequency, showing that each concentrates the gate on a different, task\-appropriate set of classes\. All three reject the null of the same class distribution as the corpus \(Pearsonχ2\\chi^\{2\}goodness\-of\-fit,p<10−10p<10^\{\-10\}\)\.

Table 8:Per\-benchmark 16\-class over\-selection relative to the corpus\-wide baseline\.
### B\.4Case Studies: a Rescue and a Break

We illustrate the end\-to\-end pipeline with two worked examples, one task that skills rescue and one of the rare tasks that skills break\. The rescue ismanufacturing\-codebook\-normalization, a SkillsBench industrial\-physical\-systems task that the frontier check of §[4\.3](https://arxiv.org/html/2607.15557#S4.SS3)\(Claude Opus 4\.7 on the OpenClaw harness\) fails without skills and passes with them\.

#### Task\.

Normalise free\-form defect\-reason texts from manufacturing test logs \(typos, abbreviations, mixed\-language fragments\) into a standardised codebook: segment each raw text, assign a valid code per segment subject to per\-station code validity, and emit calibrated confidence scores that distinguishUnknownfrom low\-evidence known predictions\.

#### Retrieved skills\.

The gate \(§[3\.4](https://arxiv.org/html/2607.15557#S3.SS4)\) selects two skills from the corpus: a domain skill,manufacturing\-failure\-reason\-codebook\-normalization, and a generic utility,fuzzy\-match\. The domain skill’s body specifies the procedure rather than the topic:

> “Ifentry\.stationsis notNone, the predicted code should only be considered valid when the record station matches \[…\] otherwise the code should be rejected\. \[…\] consider multiple evidence sources \[…\] \(overlap or fuzzy similarity betweenspan\_textand codebook text \[…\], station compatibility,fail\_codealignment,test\_itemalignment, and conflict cues\), and normalise the score to a stable range\[0\.0,1\.0\]\[0\.0,1\.0\]\.”

#### Without skills\.

The agent writes a plausible solution built on a hard\-coded phrase dictionary with station filtering, but no multi\-evidence scoring and no confidence calibration\. It passes1515of1616verifier tests and fails the semantic\-alignment check\. Only66\.2%66\.2\\%of its non\-Unknownpredictions carry non\-trivial lexical evidence against the codebook \(the verifier requires≥70%\\geq 70\\%\), i\.e\. it confidently emits codes the text does not support\. Binary task reward:0\.

#### With skills\.

The agent implements the skill’s procedure directly: multi\-evidence weighted scoring, station\-scope penalties, a deterministic tie\-break for near\-tied candidates, and calibrated confidence with an explicitUnknownband\. All1616tests pass, with19%19\\%less wall\-clock time and20%20\\%fewer total tokens than the failing no\-skill run\.

#### The break:exceltable\-in\-ppt\.

The complementary failure mode appears on one of the two tasks the strongest open cell \(Raven×\\timesQ\-397B\) breaks \(no\-skill passes, with\-skill fails\)\. The task is to update an exchange rate inside an Excel table embedded in a PowerPoint slide, preserving its formulas\. Without skills the agent improvises a working path through the four\-layer nesting, from the PPTX part through the OLE blob and its embedded ZIP down to the sheet XML, patches the cell, and re\-packs the blob \(8/88/8verifier tests pass\)\. With skills, the gate injects two topically relevant Office skills whose recipes all operate on top\-level files \(load\_workbook\("file\.xlsx"\),Presentation\("file\.pptx"\)\)\. The agent follows the recipes, hits an API wall \(property ’blob’ of ’\_OleFormat’ has no setter\), and never replaces the embedded object\. The verifier readsnanwhere the table should be \(6/86/8\)\. The skills are on\-topic, but their procedures do not extend to this task’s structure, and having a recipe displaced the improvisation that succeeds without one\.

#### Takeaway\.

The rescue and the break are two sides of the same property\. Skills transfer procedures, not topics\. When the injected procedure matches the task’s structure, as with the multi\-evidence scoring formulas and tie\-breaks above, it supplies exactly what the model does not reliably improvise\. When it does not, as with flat\-file recipes against an embedded object, following it can displace an improvisation that would have succeeded\. This is also one concrete reason composite quality scores cannot predict per\-task outcomes \(§[4\.6](https://arxiv.org/html/2607.15557#S4.SS6)\)\. The skills in both cases are well\-crafted, and what differs is procedure–task fit\.

## Appendix CCompute and Reproducibility

### C\.1Per\-Stage Compute

The ingest pipeline was served on an internal GPU cluster using Qwen3\.5\-397B\-A17B\-GPTQ\-Int4 as the LLM judge for quality scoring, classification, and embedding\-based dedup\. This judge is an open\-weight model, publicly available on Hugging Face and via OpenRouter, so the quality, safety, and classification labels can be re\-derived independently; the internal cluster refers only to where we ran it\. The compute reported below is the cumulative total over the crawl that produces the released active set\.

#### Ingest compute\.

The ingest pipeline issued approximately269,000269\{,\}000LLM\-judge calls in total:101,111101\{,\}111per\-skill quality \+ 3\-facet score calls on the post\-dedup survivors \(one call per row of the releasedquality\_judgmentstable\),101,111101\{,\}111single\-label 16\-class classification calls on the same set, and66,75166\{,\}751pair\-wise dedup adjudication calls on the borderline embedding\-cosine near\-duplicate pairs \(§[3\.1](https://arxiv.org/html/2607.15557#S3.SS1)\), per the released dedup report\. Calls were issued at standard worker concurrency on an internal GPU cluster running Qwen3\.5\-397B\-A17B\-GPTQ\-Int4\. The LLM\-judge stage dominates wall time, while content\-hash, length\-gate, and licence\-API stages run at file\-IO / network speed and contribute negligible compute\.

#### Evaluation compute\.

The main grid comprises2424configurations \(22conditions×\\times44cells×\\times33benchmarks\), each run three times, on the two open Q\-27B and Q\-397B backbones, served through Volcano \(and, for one cell of OpenClaw, DashScope\)\. The Opus 4\.7 frontier robustness check \(22conditions×\\times8787SkillsBench tasks=174=174task runs\) consumed approximately2424million API tokens in total, estimated from the per\-task token counters recorded in the released eval cache \(mean158158K total tokens per task for the SkillCorpus condition and122122K for the no\-skill baseline,8787tasks each\)\. The per\-task safety\-facet regression on SkillsBench \(§[4\.6](https://arxiv.org/html/2607.15557#S4.SS6)\) reuses traces from the main grid and adds negligible incremental cost\. Aggregate wall\-clock and GPU\-hour accounting per ingest pass and per evaluation cell is provided in the released compute log alongside the corpus artifact\.

### C\.2Reproducibility

The corpus artifact \(SQLite database, embeddings, quality cache, per\-stage reject reports\) and the ingest pipeline source will be released upon acceptance, linked from a public Hugging Face dataset entry and a GitHub repository hosting the pipeline code under a permissive licence\.

#### Multi\-run merge protocol\.

The released active set is the cumulative union of multiple ingest passes over the same SQLite database\. Content\-hash and name\-hash dedup operate across all passes, so ingesting a repository whose skills are content\-identical to rows already in the database produces no duplicate rows\. The released active set is the database snapshot at release time restricted to rows with𝚊𝚌𝚝𝚒𝚟𝚎=1\\mathtt\{active\}=1and𝚍𝚎𝚕𝚎𝚝𝚎𝚍=0\\mathtt\{deleted\}=0, the canonical release predicate\. Per\-pass reject counts and the chronological sequence of pipeline configuration deltas are preserved in the released ingest report\.

#### Reproducing the release predicate\.

A consumer reproducing the released set from the SQLite database applies the predicate𝚊𝚌𝚝𝚒𝚟𝚎=1∧𝚍𝚎𝚕𝚎𝚝𝚎𝚍=0\\mathtt\{active\}=1\\wedge\\mathtt\{deleted\}=0to theskillstable, which yields exactly96,40196\{,\}401rows\. All downstream tables \(quality\_judgments,source\_priors,vec\_skills\) are keyed oncontent\_hashorskill\_idand join on the released set\.