Finally, a good paper testing whether Agent Skills actually help.

Worth reading if you are maintaining a skill library in a team.

WebDev-Skills-Bench runs 31 public WebDev Skills over 50 Web-Bench projects and 1,000 ordered tasks, with a length-matched irrelevant control so prompt-length effects can be separated from Skill content.

Injecting the target Skill reduces mean Pass@2 by 1.3% to 4.2%, lowers task completion depth, and raises token cost by 72% to 394%. Gains show up in 17% to 36% of Skill-project pairs.

The controls split the damage into two failure modes. Some models are length-distracted, where an equally long irrelevant Skill reproduces most of the loss. Others are content-misled, where prompt length is neutral, and the Skill content still costs 1.1 to 1.4 points.

Other findings:

Losses concentrate on easy early tasks Skill rankings transfer weakly across models Anti-pattern rules outperform example-heavy content inside the Skills that do help

Paper: https://arxiv.org/abs/2608.23067

Track more trending AI papers in our academy: https://academy.dair.ai

Signal or Noise? A Benchmark Study of Agent Skills in Web Development

Source: https://arxiv.org/html/2608.23067 Ziyue YangThanks:These authors contributed equally.Affiliation:Baidu, NLPAffiliation:Beijing, ChinaEmail:[email protected]Ding Fan11footnotemark:1

Abstract

Agent Skills are reusable procedural modules that are increasingly injected into coding-agent sessions to encode framework conventions, anti-patterns, and reusable tools. However, because each injected Skill expands the prompt of every query, an effective Skill benchmark must determine not only whether an agent can solve a task, but whether the Skill should have been injected at all. We introduce WebDev-Skills-Bench and use it for a controlled empirical study of 31 public WebDev Skills on 50 Web-Bench projects and 1,000 ordered tasks. The benchmark compares four matched conditions, including a length-matched irrelevant control and leave-one-out component ablations. To isolate Skill effects from prompt-length artifacts, we place onlySKILL.mdin the prompt while mounting auxiliary files into the agent workspace. Across four models, target Skill injection reduces mean Pass@2 by 1.3% to 4.2%, lowers task completion depth, and increases token cost by 72% to 394%, with gains in only 17% to 36% of Skill-project pairs. Length-matched controls reveal two failure modes: some models are length-distracted (an equally long irrelevant Skill reproduces most of the loss), while others are content-misled (prompt length is neutral, yet Skill content still lowers Pass@2 by 1.1% to 1.4%). Further analysis shows losses concentrate on easy early tasks, Skill rankings transfer weakly across models, and anti-pattern rules outperform example-heavy content within helpful Skills. These findings recast a matched Skill as a hypothesis about a particular (Skill, project, model) triple rather than a portable asset, reframing injection as a per-deployment routing decision and making length-matched controls and per-model audits a minimum standard for Agent-Skill evaluation.

1Introduction

Agent Skills2;11are reusable procedural modules, typically a Markdown file with optional auxiliary scripts and references, that are already deployed inside commercial coding agents and community tool stacks. Unlike a one-off instruction, a Skill acts as a persistent behavioral prior: the same content is injected into every prompt of a session to encode framework conventions, anti-patterns, and reusable tools. Vendor and community marketplaces now distribute thousands of such Skills2;13;14;17;18, and many agent configurations attach them at session start by default. This deployment pattern turns Skill evaluation into a benchmark problem. Every injected Skill enlarges the prompt of every query, and when the injected content does not help, the agent incurs token and latency overhead without a corresponding reliability gain, and in some cases a net loss. A WebDev Skill benchmark therefore needs to answer not only whether an agent can solve a task, but whether a matched Skill should have been injected at all.

Web development is a natural focus for this benchmark: JavaScript/TypeScript and HTML/CSS account for a large share of LLM coding traffic1and dominate developer activity more broadly6;16, and WebDev-targeted Skills are widely published. Yet the central measurement question remains open: does injecting a matched Skill improve agent performance on realistic WebDev tasks, or does it mostly enlarge the prompt without changing behavior? Existing evidence is conflicting and indirect. Native WebDev benchmarks such as Web-Bench19and ArtifactsBench21measure generation capability but never vary Skill injection, and cross-domain Skill benchmarks disagree on direction: SkillsBench reports a+16.2+16.2pp gain over heterogeneous tasks11, whereas SWE-Skills-Bench finds 39 of 49 Skills yield zero pass-rate improvement on SWE-bench-style problems8. None isolates WebDev, and none separates Skill content from the prompt-length increase injection introduces.

We introduceWebDev-Skills-Benchand use it to conduct a controlled empirical study of 31 public WebDev Skills on Web-Bench’s 50 projects and 1,000 ordered tasks. The benchmark is built around four matched conditions:C0(no Skill),C1(target Skill),C2(a length-matched irrelevant Skill that separates Skill content from prompt-length effects), andC3(a leave-one-out slice ablation that attributes a Skill’s effect to its structural components: positive rules, anti-patterns, and example code). The core device is aworkspace-aware injection protocol: onlySKILL.mdenters the prompt, while auxiliary files (references/,examples/,scripts/) are mounted into the agent’s filesystem, so prompt length is determined bySKILL.mdalone and the length-matched control is well defined even for multi-file Skills.

The controlled study yields five findings. First, on all four models target injection produces a negative averageΔ\DeltaPass@2 (between−1.3-1.3and−4.2-4.2pp), lowers Task Completion Depth, and raises token cost by7272to394394%, while only1717to3636% of (Skill, project) pairs gain. Second, the C2 length-control splits this average into two mechanisms: Sonnet and Qwen are length-distracted (an equally long irrelevant Skill reproduces most of the loss), whereas GPT-5.1 and DeepSeek are content-misled (length is neutral but the content steers the model off-target). Third, the loss concentrates on easy early tasks rather than challenging ones—injection is most costly where the model already holds a strong prior. Fourth, per-pair Skill effects are nearly uncorrelated across models (cross-model Pearson|r|≤0.12|r|\leq 0.12onΔ\DeltaPass@2), which limits static cross-model recommendations. Fifth, decomposing a helpful Skill with C3 shows cheap anti-pattern rules carry its most reliable benefit, whereas example code helps weaker models but hurts the strongest, making example-heavy Skills a poor default.

We open-source the benchmark, the workspace-aware injection harness, and all per-(model, condition, pair) outputs.111https://anonymous.4open.science/r/webdev-skills-bench-1C32/Together these recast Skill injection as a per-deployment routing decision—finding the beneficial minority before paying its injection cost—so length-matched controls and per-model audits should be a minimum standard for future Agent-Skill benchmarks.

2Related Work

Agent Skills as a prompting paradigm.

Skills package procedural knowledge as prompt-readable Markdown plus optional workspace resources2. Unlike retrieval-augmented generation10or few-shot prompting3, a Skill supplies a persistent behavioral prior, since the same content is injected for every query in a session and encodes conventions, anti-patterns, and reusable tools. This makes Skills attractive for coding agents, whose failures often stem from framework-specific practice rather than syntax, but it also creates an evaluation challenge: a Skill can shift performance through its content, its length, or behavioral biases from framing the task as requiring an external module. Public Skill marketplaces2;13;14;17;18typically treat Skill quality as a property intrinsic to the Skill; we instead treat utility as a property of the (Skill, project, model) tuple.

Skill benchmarks.

Recent Skill benchmarks disagree on whether Skills help on average. SWE-Skills-Bench8reports 39 of 49 Skills with zero pass-rate improvement on SWE-bench-style tasks, while SkillsBench11reports a+16.2+16.2pp gain over heterogeneous domains. Neither isolates WebDev, and neither uses a length-matched irrelevant control to separate content from prompt-length effects; we hold the Web-Bench task harness fixed and vary only the Skill condition.

WebDev benchmarks.

A growing body of work evaluates LLMs on web development with the prompt held fixed. These span functional code generation (Web-Bench19, ArtifactsBench21, WebApp1K4, WebGen-Bench12), design- or screenshot-to-code (Design2Code15, WebSight9, WebCode2M7, Web2Code20), and autonomous web agents (WebArena22, Mind2Web5). All measure capability on a fixed prompt; isolating the marginal effect of a matched Skill, as we do, is orthogonal to these efforts.

3Benchmark Design

Task corpus and Skill suite.

WebDev-Skills-Bench is a pre-deployment Skill benchmark: it uses reproducible project tasks as a proxy for production WebDev work and measures the marginal effect of Skill injection rather than absolute model capability. We use Web-Bench19because, among recent WebDev benchmarks, it is not yet saturated by frontier models, relies on deterministic Playwright tests rather than LLM-as-judge scoring, and uses sequentially dependent tasks that expose long-horizon resilience. Web-Bench comprises 50 projects across 11 stack categories (React/Vue/Angular/Svelte front-ends, Express/Fastify back-ends, ORM/DB, CSS, Canvas/SVG/Three.js, bundlers, and DOM apps), with 20 ordered tasks per project, for 1,000 tasks in total.

All 31 Skills are third-party content drawn from prominent public repositories2;13;14;17; we authored none of them. Selection followed six principles: stack relevance, non-leakage, authoritative provenance, best-effort self-containedness, structural decomposability, and length coverage (the suite’sSKILL.mdfiles—the only text injected into the prompt—span roughly 1.2K–22K characters, so the byte-matched C2 control has comparable-length substitutes across the range). The Skill manifest and the full31×5031\times 50routing matrix are in the supplementary material.

Routing.

Indiscriminate Skill injection would mostly produce toolchain mismatches and dilute the utility signal. Two annotators (both authors, with professional WebDev experience) independently judged each of the1,5501{,}550(Skill, project) pairs ascore(the Skill’s declared frameworks, libraries, or domain directly cover the project’s primary stack) orskip(no meaningful overlap), at roughly 5 minutes per pair—about 129 hours per annotator (≈\approx258 person-hours in total). They agreed on1,4951{,}495pairs (raw agreement96.5%96.5\%, Cohen’sκ≈0.74\kappa\approx 0.74); the5555disagreements were resolved by discussion to consensus, with any unreconcilable pair assignedskipto keep routing conservative. This yields117 core pairs that cover all 50 projects.

Workspace-aware injection.

Many Skills ship auxiliary assets alongsideSKILL.md; concatenating them into the prompt would inflate length unequally and break the length-matched control. We therefore inject onlySKILL.mdand mount the auxiliary directories into the agent’s filesystem under.skills/<skill-id>/, so prompt length depends onSKILL.mdalone and the byte-matched C2 control is tractable for any Skill.

Conditions.

We hold the project workspace, execution harness, task order, decoding settings, and Playwright test suite fixed across conditions; the only intended variable is the prompt-levelSKILL.mdcontent.C0(native baseline) injects no Skill content and establishes intrinsic project difficulty, the zero-point for allΔ​Pk=Pk​(Cx)−Pk​(C​0)\Delta P_{k}=P_{k}(C_{x})-P_{k}(C0).C1(target Skill) injects the core-matched Skill’sSKILL.md; the pairwise differenceΔ​Total=C​1−C​0\Delta\mathrm{Total}=C1-C0is the gross utility.C2(length-matched irrelevant Skill) replaces the targetSKILL.mdwith askip-tier Skill of approximately equal byte length (±5%\pm 5\%), which separates the target-content effect from the effect of adding a length-matched Skill block and givesΔ​Length=C​2−C​0\Delta\mathrm{Length}=C2-C0andΔ​Content=C​1−C​2\Delta\mathrm{Content}=C1-C2.C3(leave-one-out slice ablation) removes one structural slice of the targetSKILL.mdat a time (positive rules−Rp-R_{p}, anti-patterns−Rn-R_{n}, or example code−X-X), attributingΔ​Total\Delta\mathrm{Total}to its components on a focused subset of helpful pairs (§4.5). The C0–C2 contrasts measurewhethera Skill helps; the C3 contrast attributeswhich partof it does (slice definitions and per-pair protocol in AppendixD).

Table 1:Model-level C1−-C0 effects on the 117 core pairs (N=3N{=}3per cell). Brackets give 95% paired-bootstrap CIs.ρ\rhois relative total-token overhead.Table 2:Δ\DeltaPass@2 (pp) by Web-Bench task difficulty (N=3N{=}3); brackets are 95% paired-bootstrap CIs. Bold marks a CI that excludes zero. A dash marks fewer than five pair-cells.

Models and metrics.

We evaluate a four-model panel: Claude Sonnet 4 (claude-sonnet-4-20250514), GPT-5.1, DeepSeek-V4-flash, and Qwen3-Coder-30B-A3B. The panel spans the contrasts most likely to modulate Skill utility: closed frontier (Sonnet 4, GPT-5.1) versus open-weight (DeepSeek-V4-flash, Qwen3-Coder-30B), general-purpose versus coding-specialized (Qwen3-Coder), and larger versus smaller backends. All models use greedy decoding (temperature=0) with a 64 kmaxTokensbudget, and each (model, condition, pair) cell is run forN=3N{=}3independent replicates so that within-condition variance is estimated uniformly across the panel. The workspace is reset (git clean -fdx) before every agent execution. All comparisons are paired at the (Skill, project) level. We report meanΔ\DeltaPass@1 andΔ\DeltaPass@2, meanΔ\DeltaTask Completion Depth (TCD, the longest consecutive Pass@2 prefix in the 20-task chain), and relative token overheadρ=(tokensCx−tokensC​0)/tokensC​0\rho=(\mathrm{tokens}_{C_{x}}-\mathrm{tokens}_{C0})/\mathrm{tokens}_{C0}, each with 95% paired-bootstrap intervals over the pair set (1,000 resamples).

4Results

We organize the controlled empirical results around five claims about when Skill injection helps, hurts, and can be attributed.

4.1Average Skill injection does not justify its token cost

Table1reports model-wise C1−-C0 effects on the 117 core pairs. All four models show a negative meanΔ\DeltaPass@2: Sonnet 4−4.2-4.2pp, Qwen−2.3-2.3, DeepSeek−2.0-2.0, and GPT-5.1−1.3-1.3(95% CIs in Table1). Three of the fourΔ\DeltaPass@2 intervals exclude zero; GPT-5.1’s marginally includes zero, but itsΔ\DeltaPass@1 (−1.6-1.6pp[−3.3,−0.1][-3.3,-0.1]) does not. Task Completion Depth falls in parallel on every model (−0.85-0.85/−0.47-0.47/−0.40-0.40/−0.26-0.26for Sonnet/Qwen/DeepSeek/GPT-5.1), so the negative average is not a pass-rate artifact. Token cost rises in parallel:ρ\rhois+72+72to+91+91% for three models and+394+394% for DeepSeek (amplified by early C0 failures that shrink its denominator), confirming that C1 is cost-increasing.

The negative mean coexists with a positive-gain tail on every model: even Sonnet, whose mean effect is most negative, wins on30%30\%of pairs, GPT-5.1 on35%35\%, DeepSeek on36%36\%, and Qwen on only17%17\%.

4.2The negative effect concentrates on easy tasks

Web-Bench labels each of the 20 tasks per project aseasy,moderate, orchallenging. A natural hypothesis is that Skills help most where they are most needed, that is, on challenging late-chain tasks. The evidence indicates the opposite. Table2reports meanΔ\DeltaPass@2 within each difficulty bucket, with per-pair-bucket CIs.

On every model the easy-task degradation is large enough that its CI excludes zero, from−4.0-4.0pp (GPT-5.1) to−10.7-10.7pp (Qwen). The moderate and challenging buckets are noisier—fewer pair-cells, with ceiling and floor effects—and show no consistent loss; DeepSeek even improves on its moderate tasks. Injection thus incurs its largest and most reliable losses on the tasks the model already handles correctly, so Skill utility should be reported by chain position rather than only as a project-level average.

A mechanism for this easy-task harm isretry lock-in: on early tasks single-attempt mistakes are common but inexpensive, since Web-Bench’s two-attempt budget recovers most of them when the model can vary simple structural choices (button text, class names, element nesting) between attempts. An injected Skill that fixes those choices in place, even when not technically wrong, converts recoverable first-attempt mistakes into chain-terminating failures and reduces self-repair flexibility on retry. AppendixAgives a concrete Sonnet×\timeszustand×\timesreact-experttrace.

4.3The negative average has two distinct mechanisms

Table 3:Length/content decomposition (meanΔ\DeltaPass@2 in pp,N=3N{=}3).Survivalis the share of C1 wins that remain positive after the length control.The length-matched C2 control decomposes the gross utility into a length artifactΔ​Length=C​2−C​0\Delta\mathrm{Length}=C2-C0and a content effectΔ​Content=C​1−C​2\Delta\mathrm{Content}=C1-C2. Table3shows that this decomposition partitions the panel into two groups.

Length distraction (Sonnet, Qwen).

Sonnet’s total loss (−4.2-4.2pp) is largely accounted for by the length control (Δ​Length=−3.3\Delta\mathrm{Length}=-3.3pp, CI excludes zero), with a small content term that is not significant (Δ​Content=−0.9\Delta\mathrm{Content}=-0.9pp, CI includes zero); Qwen follows the same shape (Δ​Length=−3.5\Delta\mathrm{Length}=-3.5, CI excludes zero;Δ​Content=+1.2\Delta\mathrm{Content}=+1.2). For these two models the data fit attention dilution: a Skill-sized prompt block degrades performance, while the target content itself is not reliably worse than an equally long irrelevant block.

Content misalignment (GPT-5.1, DeepSeek).

GPT-5.1 and DeepSeek show the opposite shape. Their length terms are near zero and their CIs include zero (−0.2-0.2and−0.6-0.6pp), while their content terms are negative (−1.1-1.1and−1.4-1.4pp). For these two models the prompt-length increase alone is essentially harmless, but injecting the specific content of the target Skill degrades performance. We read this as mechanism-level evidence: the same aggregate effect can arise from different causes across models, and the mitigation differs accordingly—prompt shortening versus content review.

Pair-level view.

TheSurvivalcolumn adds a complementary perspective: a majority of each model’s C1 wins remain positive after the length control—60%60\%for Sonnet,71%71\%for GPT-5.1,64%64\%for DeepSeek, and95%95\%for Qwen—so even where the average effect is length-driven, most individual wins are content-positive rather than length artifacts. The decomposition thus describes model-level tendencies rather than certifying any individual pair; the per-pair content effect is what a benchmark should expose before routing.

4.4Cross-model contradictions caution against static Skill rankings

Table 4:Cross-model Pearson correlation of per-pairΔ\DeltaPass@2 (C​1−C​0C1{-}C0) over the 117 core pairs (N=3N{=}3). All coefficients are near zero (|r|≤0.12|r|\leq 0.12; Spearman is similar,|ρs|≤0.16|\rho_{s}|\leq 0.16): a Skill’s measured utility on one model barely predicts its utility on another.The decomposition in Table3reports model-level statistics, but whether per-pair effects transfer across models is a separate, global property. Table4correlates each model’s per-pairΔ\DeltaPass@2 against every other’s over the 117 core pairs. All six coefficients are near zero (Pearson−0.08-0.08to+0.12+0.12, mean≈0.00\approx 0.00; Spearman similar), and disagreement is pervasive rather than confined to outliers:74%74\%of pairs carry at least one positive and one negative model sign, while only1%1\%gain on all four models and4%4\%lose on all four.

This decorrelation is starkest in individual pairs: onlowdb×\timesdatabase-optimizer(S14), Sonnet 4 gains+33+33pp while DeepSeek and Qwen each lose2222and GPT-5.1 is unchanged—a5555pp swing on one core-tier pair that baseline difficulty does not explain (AppendixC).

Static rankings transfer poorly across models.

The near-zero cross-model correlation cautions against two common deployment shortcuts. First, ranking Skills by marketplace stars assumes one ranking transfers across backends; here the same content is a strong gainer for Sonnet and a clear liability for DeepSeek. Second, validating a Skill on a frontier model and deploying it on a cheaper one is unsafe, since the Sonnet signal (+33+33pp on S14) gives no hint that the Skill should be withheld from DeepSeek. A practical router needs per-(Skill, project, model) traces, re-collected whenever any of the three vertices is upgraded.

4.5In helpful Skills, anti-patterns are the most cost-effective slice

Conditions C0–C2 askwhethera Skill helps; condition C3, a leave-one-out slice ablation, askswhich part of itdoes. We partition eachSKILL.mdinto three removable slices (positive rulesRpR_{p}, anti-patternsRnR_{n}, and example codeXX) and remove one at a time; a slice’scontributionisC​1−(variant)C1-(\text{variant})on Pass@2. C3 runs on the positive tail—five (Skill, project) pairs with a robustly positive cross-modelC​1−C​0C1{-}C0signal (+3.3+3.3to+6.7+6.7pp each)—so the complete Skill here raises Pass@2 by+5.1+5.1pp, opposite in sign to the panel average (§4.1); the C3 numbers describehow a helpful Skill is built, not how often Skills help. Pair selection, run protocol, statistical tests, and the selection caveat are in AppendixD.

Table5and Figure3(AppendixD) report per-slice contributions pooled over the 20 model×\timesproject cells: anti-patterns (RnR_{n}) are the only slice with a directionally reliable task-level effect (p=0.008p{=}0.008), positive rules (RpR_{p}) are neutral, and example code (XX) is null on average (−0.7-0.7pp). YetXXis by far the costliest slice: removing it saves34,48234{,}482input tokens per run on average, about22.7%22.7\%of the fullSKILL.mdbudget—so cheap proscriptive content is the most cost-effective signal.

Table 5:C3 slice contributions to Pass@2 (pp),C​1−(variant)C1-(\text{variant}); cell-level means with paired-bootstrap CIs. “All” pools 20 cells; “−-Sonnet” the 15 non-Sonnet cells. Positive = slice helps. Wilcoxon signed-rank and McNemar significance tests are reported in AppendixD.The pooled example null, however, is an artifact of cancellation. Dropping Sonnet, every slice (including examples) turns positive, and the example effect becomes the strongest and most significant of all (+4.2+4.2pp; Table5, right). Figure4(AppendixD) shows the split: examples help DeepSeek (+8.3+8.3pp) and Qwen (+3.7+3.7), are neutral for GPT-5.1 (+0.7+0.7), and clearly hurt Sonnet (−15.3-15.3). For the strongest model, in-skill examples act as a constraint that suppresses its own better priors—the retry-lock-in mechanism of §4.2. Example-heavy Skills should thus not be a default: they are expensive and model-dependent, whereas concise anti-pattern rules are most worth retaining.

5Benchmark Implications

Taken together, these results make Skill injection a deployment decision rather than a configuration default. WebDev-Skills-Bench is intended as a pre-deployment audit for estimating whether a Skill’s marginal gain survives its length cost and model-specific risks. The implications below translate this audit view into practical routing and reporting practices.

Benchmark injection as an opt-in decision.

Unconditional injection is small-and-negative on every model and adds7272to394394% to token cost, so injecting a long Skill at session start, on average, reduces reliability while raising cost. Skills should be injected only when a pair-level signal (model, project, possibly task difficulty) crosses an empirical utility threshold.

Evaluate by chain position, not only by stack.

Skill-induced degradation concentrates on easy initial tasks, where the model already produces the correct output. A reasonable heuristic is to skip the Skill on early tasks and inject one only once error rates rise—contradicting the common pattern of attaching Skills before any task begins.

Per-model curation is necessary.

Per-pair effects are near-uncorrelated across models (Table4), so marketplaces that publish a single ranking are of limited use for multi-model deployment. A practical system needs per-model evaluation traces and a ranking conditioned on the model backend; a useful marketplace listing should report model-conditioned utility, the target stack, prompt length, and whether the gain survives a length-matched control.

A length-matched control should be the minimum bar.

Without C2, we would have reported a uniform negative effect and missed that it arises from two mechanisms needing different mitigations. Future Agent-Skill benchmarks should adopt a length-matched control as a basic requirement; the workspace-aware protocol of §3makes this tractable even for multi-file Skills.

6Conclusion

We introduced WebDev-Skills-Bench, a controlled benchmark that asks not whether an agent can solve a task but whether a Skill should have been injected at all. Its contribution is methodological: a byte-matched control and a slice ablation turn a single negative average into a mechanistic account—a small positive tail, losses on easy tasks, and a length/content split across models that demands opposite mitigations. The claim is conditional: a Skill is a hypothesis about a particular (Skill, project, model) triple, not a portable asset, and capturing its value is a routing problem—finding the beneficial minority before paying its injection cost. We hope the controls that expose that minority become a default in Agent-Skill benchmarking.

Limitations

Our study has several limitations. First, the seed spread is non-trivial: across the three Sonnet replicates, aggregate C0 and C1 Pass@2 vary by4.44.4and3.63.6pp, which is comparable to the headline effect size, so individual pair-level estimates should be read with corresponding caution even though theN=3N{=}3panel makes the model-level means stable. Second, our C2 measurements use109109unique length-matched runs across the117117pairs, because some C2 prompts are shared across same-project pairs; a cluster bootstrap and full per-pair C2 de-duplication are open follow-ups. Third, the routing is intentionally conservative, since C1 is evaluated only on the117117core-tier pairs, so we cannot speak to what happens when Skills are deployed off-target. Fourth, the Skill set is drawn from high-visibility public repositories, and closed Skills from enterprise pipelines or fine-tuned skill-routers may produce different patterns. Fifth, WebDev-Skills-Bench is a pre-deployment benchmark rather than an online A/B test: Web-Bench projects approximate realistic WebDev work, but they do not include live user traffic, human developer interventions, or product-specific acceptance criteria. Finally, Web-Bench measures functional correctness via Playwright rather than visual fidelity or interactive UX, and we report token overhead rather than end-to-end latency, so Skills that primarily improve readability, accessibility, design quality, or developer review time may produce gains that our metrics do not capture.

References

• Anthropic (2025a)AnthropicAnthropic economic index: AI’s impact on software development.Technical reportAnthropic.Note:https://www.anthropic.com/research/impact-software-developmentCited by:§1.
• Anthropic (2025b)AnthropicClaude agent skills documentation.Note:https://docs.claude.com/en/docs/agents-and-tools/agent-skillsAccessed 2026-04Cited by:§1,§2,§3.
• Brownet al.(2020)T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. AmodeiLanguage models are few-shot learners.InAdvances in Neural Information Processing Systems (NeurIPS),Note:arXiv:2005.14165Cited by:§2.
• Cui (2024)Y. CuiWebApp1K: a practical code-generation benchmark for web app development.arXiv preprint arXiv:2408.00019.Cited by:§2.
• Denget al.(2023)X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. SuMind2Web: towards a generalist agent for the web.InAdvances in Neural Information Processing Systems (NeurIPS) 36,Note:arXiv:2306.06070Cited by:§2.
• GitHub (2025)GitHubOctoverse: a new developer joins GitHub every second as AI leads TypeScript to #1.Note:https://github.blog/news-insights/octoverse/octoverse-a-new-developer-joins-github-every-second-as-ai-leads-typescript-to-1/Accessed 2026-04Cited by:§1.
• Guiet al.(2024)Y. Gui, Z. Li, Y. Wan, Y. Shi, H. Zhang, Y. Su, B. Chen, D. Chen, S. Wu, X. Zhou, W. Jiang, H. Jin, and X. ZhangWebCode2M: a real-world dataset for code generation from webpage designs.arXiv preprint arXiv:2404.06369.Cited by:§2.
• Hanet al.(2026)T. Han, Y. Zhang, W. Song, C. Fang, Z. Chen, Y. Sun, and L. HuSWE-Skills-Bench: do agent skills actually help in real-world software engineering?.arXiv preprint arXiv:2603.15401.Cited by:§1,§2.
• Laurençonet al.(2024)H. Laurençon, L. Tronchon, and V. SanhUnlocking the conversion of web screenshots into HTML code with the WebSight dataset.arXiv preprint arXiv:2403.09029.Cited by:§2.
• Lewiset al.(2020)P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, S. Riedel, and D. KielaRetrieval-augmented generation for knowledge-intensive NLP tasks.InAdvances in Neural Information Processing Systems (NeurIPS) 33,Note:arXiv:2005.11401Cited by:§2.
• Liet al.(2026)X. Li, W. Chen, Y. Liu, S. Zheng, X. Chen, Y. He, Y. Li, B. You, H. Shen, J. Sun, S. Wang, Q. Zeng, D. Wang, X. Zhao, Y. Wang, R. B. Chaim, Z. Di, Y. Gao, J. He, Y. He, L. Jing, L. Kong, X. Lan, J. Li, S. Li, Y. Li, Y. Lin, X. Liu, X. Liu, H. Lyu, Z. Ma, B. Wang, R. Wang, T. Wang, W. Ye, Y. Zhang, H. Xing, Y. Xue, S. Dillmann, and H. LeeSkillsBench: benchmarking how well agent skills work across diverse tasks.arXiv preprint arXiv:2602.12670.Note:Project page:https://skillsbench.aiCited by:§1,§1,§2.
• Luet al.(2025)Z. Lu, Y. Yang, H. Ren, H. Hou, H. Xiao, K. Wang, W. Shi, A. Zhou, M. Zhan, and H. LiWebGen-Bench: evaluating LLMs on generating interactive and functional websites from scratch.arXiv preprint arXiv:2505.03733.Cited by:§2.
• Mindrally (2025)MindrallyAgent skills collection.Note:GitHub repository (MIT).https://github.com/mindrally/skillsAccessed 2026-04Cited by:§1,§2,§3.
• Osmani (2025)A. OsmaniWeb quality skills.Note:GitHub repository.https://github.com/addyosmani/web-quality-skillsAccessed 2026-04Cited by:§1,§2,§3.
• Siet al.(2024)C. Si, Y. Zhang, Z. Yang, R. Liu, and D. YangDesign2Code: how far are we from automating front-end engineering?.arXiv preprint arXiv:2403.03163.Cited by:§2.
• Stack Overflow (2025)Stack Overflow2025 developer survey results.Note:https://survey.stackoverflow.co/2025/Accessed 2026-04Cited by:§1.
• Vercel Labs (2025)Vercel LabsAgent skills.Note:GitHub repository.https://github.com/vercel-labs/agent-skillsAccessed 2026-04Cited by:§1,§2,§3.
• VoltAgent (2026)VoltAgentAwesome agent skills: a curated collection of 1000+ agent skills.Note:GitHub repository.https://github.com/VoltAgent/awesome-agent-skillsAccessed 2026-04Cited by:§1,§2.
• Xuet al.(2025)K. Xu, Y. Mao, X. Guan, and Z. FengWeb-Bench: a LLM code benchmark based on web standards and frameworks.arXiv preprint arXiv:2505.07473.Note:ByteDance ResearchCited by:§1,§2,§3.
• Yunet al.(2024)S. Yun, H. Lin, R. Thushara, M. Q. Bhat, Y. Wang, Z. Jiang, M. Deng, J. Wang, T. Tao, J. Li, H. Li, P. Nakov, T. Baldwin, Z. Liu, E. P. Xing, X. Liang, and Z. ShenWeb2Code: a large-scale webpage-to-code dataset and evaluation framework for multimodal LLMs.arXiv preprint arXiv:2406.20098.Cited by:§2.
• Zhanget al.(2025)C. Zhang, Y. Li, C. Xu, J. Liu, A. Liu, C. Zhou, K. Deng, D. Wu, G. Huang, K. Li, Q. Yi, R. Xiong, S. Hu, Y. Zhang, Y. Jiang, Z. Xu, Y. Zhang, W. Zhou, C. Zhou, and F. LianArtifactsBench: bridging the visual-interactive gap in LLM code generation evaluation.arXiv preprint arXiv:2507.04952.Note:Tencent HunyuanCited by:§1,§2.
• Zhouet al.(2024)S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. NeubigWebArena: a realistic web environment for building autonomous agents.InInternational Conference on Learning Representations (ICLR),Note:arXiv:2307.13854Cited by:§2.

Appendix AWorked Example: Retry Lock-In onzustand×\timesreact-expert

This trace illustrates the easy-task retry lock-in of §4.2.Zustandis a React state-management project, andreact-expert(S07) is the most stack-aligned Skill routed to it. On the Sonnet C0 baseline the project reaches55%55\%Pass@2 (mean over three seeds); injecting S07 lowers it to40%40\%(Δ=−15\Delta=-15pp), and the loss stems from a single seed whose chain terminates at task-4. Task-4 asks the model to create aBlogFormmodal titled “Create Blog” with an “Add Blog” button in the header. On both C0 and C1 the first attempt emits a modal whose<h2>Create Blog</h2>heading and submit<button>Create Blog</button>collide under the same Playwright locator, and both fail with the identical strict-mode trace (Figure1). The conditions diverge on the retry: the C0 attempt relabels the submit button toSubmit, so the locator resolves to a single element and the project continues to task-11, whereas the C1 attempt relabels it toCreate Blog Post, in keeping with the Skill’s prescription that button labels restate the operation; because the locator matches text as a substring,Create Blog Poststill collides with the heading, strict mode fails again, and the chain terminates at task-4.

Both conditions, 1st attempt, fail with:

strict mode violation:
getByText(’Create Blog’)
  resolved to 2 elements:
  1) <h2>Create Blog</h2>
  2) <button type="submit">
       Create Blog</button>

(a) C0 (no Skill) 2nd attempt: relabel, pass

<button type="submit">
  Submit
</button>

(b) C1 (react-expert, S07) 2nd attempt: still collides, fail

<button type="submit">
  Create Blog Post
</button>

Figure 1:Sonnet×\timeszustand×\timesreact-expert: the C0 retry relabels the colliding submit button to a structurally distinct string (a); the C1 retry, anchored on the Skill’s naming convention, keeps the operation name and still matches the heading as a substring, so strict mode fails again (b).

Appendix BContent-Driven Win:pull-loading×\timesjs-dom-web-components

The model-level decomposition in §4.3can conceal per-pair effects that exceed the panel mean by more than an order of magnitude.Pull-loadingis a vanilla-JavaScript pull-to-refresh widget routed tojs-dom-web-components(S31), a Skill on DOM lifecycle and Web Component visibility patterns. On the Sonnet C0 baseline the project stays at0%0\%Pass@2 across all three seeds, with the failure cascading from task-1, which asks the model to initialize thecontentandnoticeTxtcontainers. Sonnet emits the requested DOM, but Playwright’stoBeVisible()assertion fails because the empty wrapper has zero box dimensions; the C0 retry adds a defensivedisplay: blockstyle but never adds text content, so the second attempt fails for the same reason (Figure2a). Injecting S31 raises Pass@2 on the same pair to18%18\%, while the length-matched control returns to0%0\%, soΔ​Content=+18\Delta\mathrm{Content}=+18pp withΔ​Length=0\Delta\mathrm{Length}=0. The mechanism is visible in the C1 second-attempt output (Figure2b): the model adds text inside the wrapper together with an explicit visibility style rule, directly applying S31’s prescription that DOM nodes require both text content and explicit visibility properties to satisfy rendering assertions. Task-1 then passes and the project advances.

Task-1 Playwright assertion:expect\(noticeTxt\)\.toBeVisible\(\)

(a) C0 (no Skill), both attempts fail:

<div id="content">
  <div id="noticeTxt"></div>
</div>

Empty wrapper gives zero box dimensions, sotoBeVisible()fails.

(b) C1 (js-dom-web-components, S31), 2nd attempt: pass

<div id="content">
  <div id="noticeTxt">
    Notice text content
  </div>
</div>

/* style.css */
#noticeTxt {
  display: block;
  visibility: visible;
}

Figure 2:Sonnet×\timespull-loading×\timesjs-dom-web-components: an empty wrapper fails Playwright’s visibility check (a); S31 adds text content and an explicit visibility rule, which satisfies the assertion (b). This single pair contributes a+18+18pp content effect, far above Sonnet’s−0.9-0.9pp model-level average, which shows that the decomposition describes tendencies rather than per-pair guarantees.

Appendix CCross-Model Sign Reversal:lowdb×\timesdatabase-optimizer

The near-zero cross-model correlation of §4.4(Table4) is most vivid in individual pairs. Table6reports the single core-tier pairlowdbrouted todatabase-optimizer(S14), a mainstream back-end pair, as a concrete instance of the panel-wide decorrelation.

Table 6:Cross-model effects onlowdb×\timesdatabase-optimizer(S14), an extreme, illustrative single pair (N=3N{=}3). The same Skill content produces opposite-signed effects across the panel.The same Skill content produces opposite-signed effects: Sonnet 4 gains+33+33pp (among the largest single-pair gains we observe), DeepSeek and Qwen each lose2222, and GPT-5.1 is approximately unchanged. Baseline difficulty does not explain this: DeepSeek has the strongest C0 baseline (42%42\%) yet drops most, while Sonnet rises from33%33\%to67%67\%. The5555pp gap between Sonnet and DeepSeek/Qwen on one core-tier pair is the extreme tail of the panel-wide pattern that a single deployment ranking cannot capture.

Appendix DC3 Slice Ablation: Protocol Detail

This appendix expands the C3 leave-one-out (LOO) protocol summarized in §4.5.

Slice definitions.

EachSKILL.mdis segmented into a meta/overview header (always retained), positive rulesRpR_{p}(prescriptive “do” conventions), anti-patternsRnR_{n}(proscriptive “don’t” rules), and example codeXX(fenced code blocks demonstrating usage). Each ablation variant is the completeSKILL.mdwith exactly one slice removed (−Rp-R_{p},−Rn-R_{n},−X-X); the header is never removed so the prompt remains well-formed. A variant is marked N/A for any Skill that lacks the relevant slice, and only Skills with at least two removable slices are eligible.

Pair selection.

Decomposition requires a gross effect to attribute. Pairs were screened by two rules on the cross-model main-experiment aggregates: the Skill must contain at least two cleanly removable slices (so there is something to decompose; this excludes, e.g.,svelte/S22 (rules only) andbackend-patterns/S15 (examples only), whoseSKILL.mdreduces to a single removable slice after segmentation), and the pair must gain (C​1>C​0C1{>}C0) on at least two of the four models (cross-model consistency rejects single-model noise). The five selected pairs arejavascript-pro(S13) onviteandsvg-chart,bundler-config(S29) onwebpack,react-expert(S07) onfastify-react, anddatabase-optimizer(S14) onsequelize. On the finalN=3N{=}3aggregates,vite/S13,svg-chart/S13, andwebpack/S29 gain on three of four models andfastify-react/S07 andsequelize/S14 gain on two; all five retain a positive cross-model mean gain (+3.3+3.3to+6.7+6.7pp). The released candidate table reports these aggregates. With only five pairs we make no per-domain claims and report contributions pooled and per model. This positive-tail selection is why the gross effect here (+5.1+5.1pp) is opposite in sign to the panel average of §4.1and must not be read as evidence that Skills help on average.

Runs and statistics.

Every variant is evaluated for all four models atN=3N{=}3seeds, anchored on the existing C0 and C1 runs, giving5×3×3×4=1805\times 3\times 3\times 4=180ablation runs. The unit of inference is the cell (model×\timesproject,N=20N{=}20, seeds averaged). Per-slice contributions are tested with a Wilcoxon signed-rank test over cells with a10,00010{,}000-resample paired-bootstrap CI; the task-level McNemar test (discordant countsb,cb,cover matched task attempts, missing tasks scored as failures) corroborates direction. For the pooled anti-pattern (RnR_{n}) contribution the McNemar discordants areb=111b{=}111,c=74c{=}74(p=0.008p{=}0.008); excluding Sonnet, the example-code (XX) contribution of+4.2+4.2pp has Wilcoxonp=0.005p{=}0.005, the most significant slice effect in the panel. Figures3and4visualize the pooled per-slice contributions and the per-model split of the example-code effect. Per-pair contribution tables and the full per-(model, slice, seed) outputs are in the released artifacts.

Figure 3:Per-slice contribution to Pass@2, pooled over the 20 model×\timesproject cells (bar = mean, whisker = paired-bootstrap CI; each gray dot is one cell). Bars are green when positive and red when negative, matching the shading of Table5. Only anti-patterns (RnR_{n}) sit reliably above zero; positive rules and examples are null on average with wide per-cell spread.Figure 4:Contribution of example code (XX) to Pass@2 by model (C​1−(−X)C1-(-X), cell mean with paired-bootstrap CI). Examples help DeepSeek, Qwen, and weakly GPT-5.1 but hurt Sonnet, so the pooled effect is near zero.

Appendix EReleased Artifacts

The analysis code, the routing for all conditions (C1 core pairs, C2 length-matched pairs, C3 candidate pairs), the 31 Skills with provenance, the C3 slice definitions, and the derived per-pair and per-task CSVs (C0–C3 values, pairwiseΔ\Deltas, per-slice C3 contributions, chain-position data, per-model rankings, and seed-variance tables) are released athttps://anonymous.4open.science/r/webdev-skills-bench-1C32/. The repository includes the pipeline that reproduces the paper’s tables and figures from these derived CSVs. The raw per-task evaluation reports and the base Web-Bench task harness are omitted for size and obtained from the upstream Web-Bench project, so the release fully reproduces the reported analyses but not a byte-for-byte rerun of every agent trajectory.