Paper page - SWE-WebDevBench: Evaluating Coding Agent Application Platforms as Virtual Software Agencies

Source: https://huggingface.co/papers/2605.04637

Abstract

A comprehensive evaluation framework called SWE-WebDev Bench is presented to assess AI-powered application development platforms across multiple dimensions including requirement understanding, architectural decision-making, code quality, and production readiness.

The emergence of “vibe coding” platforms, where users describe applications in natural language andAI agentsautonomously generate full-stack software, has created a need for rigorous evaluation beyond code-level benchmarks. In order to assess them as virtualsoftware development agencieson understandingbusiness requirements, makingarchitectural decisions, writingproduction code, handlingiterative modifications, and maintainingbusiness readiness, we introduceSWE-WebDev Bench, a 68-metricevaluation frameworkspanning 25 primary and 43 diagnostic metrics across seven groups, organized along three dimensions:Interaction Mode(App Creation Request (ACR) vs. App Modification Request (AMR)),Agency Angle(Product Manager (PM), Engineering, Ops), andComplexity Tier(T4 multi-role SaaS, T5 AI-native). Our evaluation (six platforms, three domains, 18 evaluation cells) reveals four recurring shortcomings in the current generation of AI app builders: (1) Aspecification bottleneck, where platforms compress richbusiness requirementsinto oversimplified technical plans, (2) A pervasivefrontend-backend decoupling, where visually polished UIs mask absent or broken backend infrastructure, (3) A steepproduction-readiness cliff, where no platform scores above 60% on engineering quality and post-generation human effort varies substantially across platforms and (4) Widespread security and infrastructure failures, with no platform exceeding 65%Security Scoreagainst a 90% target andconcurrency handlingas low as 6%. These observations are descriptive of our sample and require larger-scale replication to establish generality. We releaseSWE-WebDev Benchas a community benchmark to enable such replication and help platform builders identify and address these gaps. Code and benchmark resources are available at: https://github.com/snowmountainAi/webdevbench and https://webdevbench.com/.

View arXiv pageView PDFProject pageGitHub8Add to collection

Get this paper in your agent:

hf papers read 2605\.04637

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.04637 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.04637 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.04637 in a Space README.md to link it from this page.

Collections including this paper1