80 Canary Tests: 6 Vibe Coding Platforms with Pretty UIs, but None with a Runnable Backend

There’s a rather gut-punching paper on arxiv recently called SWE-WebDev Bench that pits 6 major vibe coding platforms against each other: Lovable, Replit Agent3, Vercel v0-Max, Base44, Emergent E1-OPUS, and QwikBuild. One-line takeaway: no platform scored above 60% on Engineering overall. Everyone nails the frontend UI, but backend, security, and production readiness all collectively tank — not a single one can deliver a runnable system out of the box. Here’s how they tested it, what they found, and how to think about using it. First, a conflict of interest note: 2 of the paper’s authors are from QwikBuild, and the AMR modification scenario was only tested on QwikBuild — the paper discloses this upfront, which we’ll cover in detail.

How the Benchmark Works: 3 Dimensions × 68 Metrics × 6 Platforms

Testing vibe coding platforms with LeetCode-style function problems doesn’t cut it, because these platforms market themselves as “one sentence to production.” The SWE-WebDev Bench approach treats these platforms as complete “virtual software agencies” for evaluation.

The benchmark is structured across 3 dimensions. The Mode dimension splits into ACR (Create New App) and AMR (Modify Existing App) — AMR is new in this paper. The Angle dimension splits into three roles: PM, Engineering, and Ops, looking at requirement understanding, code quality, and deployment/operations respectively. The Tier dimension separates traditional SaaS (T4) from AI-native applications (T5).

The business scenarios come from 3 real domains: education (exam system like ExamEdge), field services (inspection system like FieldOps), and fintech + AI (risk assessment like VettAI). The 68 metrics are split into 7 groups (G1 spec fidelity to G7 production readiness), with 25 primary metrics and 43 diagnostic metrics. Scoring uses a 4-tier pyramid: Tier 0 fully automated (Lighthouse, k6, npm audit) accounts for 40%, Tier 1 LLM scoring 35%, Tier 2 LLM + human 15%, and Tier 3 expert panel 10%.

The core design principle is evaluating “can the platform turn vague requirements into deployable systems,” not “can it write correct functions.”

4 Common Weak Points: No Platform Escaped These

Testing all 6 platforms revealed they all fell into the same set of pitfalls.

Pitfall 1: Spec Compression. Given the same prompt, the number of clarifying questions the PM role could ask ranged from 0 to 15, with a 3.5x spread in spec inference quality scores. The worst platforms quietly compress requirements and start coding — users have no idea which details got cut.

Pitfall 2: Frontend-Backend Decoupling. Frontend engineering scores ranged 68-74% across platforms, and the UIs all look decent. But background batch processing and scheduled tasks (CBS) scored from 0 to 49.3% — a 50 percentage point spread. This means some platforms’ UIs are completely “pretty shells” with no real workflows behind the buttons.

Pitfall 3: Production Readiness Cliff. The paper quantifies this with ETF (manual fix hours) and CDI (claimed vs. actual functionality gap). ETF ranges from 12 to 60 hours, a 5x spread. Platforms with high CDI make you think the system is running — but users find the bugs before you do.

Pitfall 4: Security Failures Across the Board. Security scores ranged 38-65%, with a target of 90%. Common issues include hardcoded API keys, missing CSRF protection, and no rate limiting. Concurrent load scores (CLS) only reached 6-42%, well below the 70% target.

All 6 platforms have significant weaknesses in some dimension, but the distribution of weaknesses differs.

Canary Design: 80 Culture-Specific Requirements to Test Real Understanding vs. Assembling

The smartest design element in this paper is the canary requirements methodology.

Canaries are specific requirements embedded in prompts, like “date format must be DD/MM/YYYY,” “amounts must include the Baht symbol ฿,” or “must support Thai input.” These culture-specific, domain-embedded details will be preserved if the model truly understands the business, but silently dropped if it’s just assembling templates from training data.

The 80 canaries are split into 4 types. Original: 21 canaries explicitly stated in the initial prompt. New: 37 canaries added during the AMR modification phase. Surviving: 18 canaries that must be retained after modification. Contradiction: 4 canaries with logical conflicts, testing whether the model dares to question the user rather than blindly implementing.

The 5.5x spread in CRR (Canary Retention Rate) is the most informative metric from this methodology. The highest-scoring platform retained 97.7%, while the lowest retained only 17.7% — meaning in the worst case, 80% of a user’s hour-long requirement description gets silently deleted.

6 Platforms, 3 Paths: Architecture Choices Determine Where Weaknesses Lie

All platforms failed, but they failed differently. The paper categorizes the 6 platforms into 3 architectural paths.

Path 1: Integrated Infrastructure — QwikBuild is the representative. Frontend engineering at 68%, background batch at 49.3%. The strength is having a real complete backend infrastructure running. The weakness is still insufficient spec understanding depth.

Path 2: Ecosystem Leverage — Replit Agent3 is the representative. Frontend at 74.3%, backend at 29.7%. Leverages Replit’s own development environment ecosystem and can pull in external services, but cross-domain stability is limited. Replit’s score variance across different business domains is 13 percentage points, indicating insufficient generalization for specific domains.

Path 3: Frontend-First — Vercel v0-Max and Lovable are representatives. Frontend engineering can reach 68%, but backend batch at 0-2%. The biggest problem with these two platforms is “the pretty UI gives you confidence, making you think the system is running” — but when you get to production, you discover clicking the buttons does nothing behind the scenes.

Base44 and Emergent E1-OPUS also participated, with scores falling between these three tiers. The complete metric comparison table in the paper is worth checking against platforms you’ve used.

Conflict of Interest Must Be Addressed First

The most critical thing about this paper is the conflict of interest.

Two of the paper’s 3 authors are from QwikBuild, and QwikBuild is one of the 6 platforms being evaluated. The cross-platform testing for AMR (modification scenarios) isn’t complete yet — so far only QwikBuild has run the full AMR evaluation. The pretty numbers like CAR 100% and 0% regression rate are single-platform data.

The paper discloses this COI itself, conducts blind review for G1 metrics (reviewers don’t know which platform the code is from), and specifically lists QwikBuild’s failure points. But full double-blind wasn’t achieved. The third author, Saxena, independently designed prompts and canaries — that’s another mechanism to reduce COI impact.

How to Use This Paper: Borrow the Methodology

The canary methodology, the 3-dimension evaluation framework, and the 4 common weakness findings can be safely borrowed. For specific platform rankings, I’d suggest reproducing the evaluation yourself before drawing conclusions. The paper promises to make this a living benchmark, retested quarterly, with future versions using independent evaluators.

One-Line Takeaway

Current vibe coding platforms are still in the “Demo Stage” — UIs are good enough to show the boss, but bringing them to production requires 12-60 hours of manual fixes. SWE-WebDev Bench doesn’t provide the answer to “which one should I buy,” but rather provides the scaffold for “what metrics to evaluate when assessing any vibe coding platform.”

Who should read this: Product managers, CTOs, and independent developers selecting vibe coding platforms; entrepreneurs building AI coding products, who can see which weaknesses offer differentiation opportunities; LLM evaluation researchers, who can reference the canary methodology.

Who shouldn’t bother: Casual enthusiasts just playing with demos don’t need this level of detail — it’ll just cause unnecessary anxiety.

Full paper: arxiv.org/abs/2605.04637 | Repo: github.com/snowmountainAi/webdevbench

The paper is titled “Evaluating Coding Agent Application Platforms as Virtual Software Agencies,” authored by 3 people, 2 from the QwikBuild team, with the third author Saxena responsible for prompt and canary design.