WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models
Summary
WebCompass is a multimodal benchmark for evaluating LLMs on web coding tasks across three input modalities (text, image, video) and three task types (generation, editing, repair). It introduces an Agent-as-a-Judge paradigm that autonomously executes generated websites in a real browser to assess visual fidelity and interactivity.
View Cached Full Text
Cached at: 04/21/26, 07:20 AM
Paper page - WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models
Source: https://huggingface.co/papers/2604.18224 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
WebCompass evaluates web development capabilities through diverse input modalities and task types, using automated evaluation methods that simulate real-world coding workflows.
Large language modelsare rapidly evolving into interactive coding agents capable of end-to-end web coding, yet existing benchmarks evaluate only narrow slices of this capability, typically text-conditioned generation with static-correctness metrics, leavingvisual fidelity, interaction quality, andcodebase-level reasoninglargely unmeasured. We introduce WebCompass, amultimodal benchmarkthat provides unified lifecycle evaluation ofweb engineering capability. Recognizing that real-world web coding is an iterative cycle of generation, editing, and repair, WebCompass spans three input modalities (text, image, video) and three task types (generation, editing, repair), yielding seven task categories that mirror professional workflows. Through a multi-stage,human-in-the-loop pipeline, we curate instances covering 15 generation domains, 16 editing operation types, and 11 repair defect types, each annotated at Easy/Medium/Hard levels. For evaluation, we adopt a checklist-guidedLLM-as-a-Judgeprotocol for editing and repair, and propose a novelAgent-as-a-Judgeparadigm for generation that autonomously executes generated websites in a real browser, exploresinteractive behaviorsvia theModel Context Protocol(MCP), and iteratively synthesizes targeted test cases, closely approximating human acceptance testing. We evaluate representative closed-source and open-source models and observe that: (1) closed-source models remain substantially stronger and more balanced; (2) editing and repair exhibit distinct difficulty profiles, with repair preserving interactivity better but remaining execution-challenging; (3) aesthetics is the most persistent bottleneck, especially for open-source models; and (4) framework choice materially affects outcomes, with Vue consistently challenging while React and Vanilla/HTML perform more strongly depending on task type.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2604\.18224
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2604.18224 in a model README.md to link it from this page.
Datasets citing this paper1
#### NJU-LINK/WebCompass Viewer• Updatedabout 2 hours ago • 933 • 1.83k • 3
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2604.18224 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
OpenCompass: A Universal Evaluation Platform for Large Language Models
OpenCompass is a one-stop, scalable, high-concurrency evaluation platform for large language models, supporting diverse benchmarks and modular design to unify and standardize LLM assessment.
AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities
AgentCompass is an open-source, lightweight, and extensible evaluation infrastructure for LLM-based agents, decoupling benchmarks, harness, and environment for flexible configurations. It supports over 20 benchmarks across five capability dimensions and provides fault-tolerant runtime and trajectory analysis tools.
Compliance, Capability, and Conflict: Benchmarking Multimodal LLMs under System Messages
The article introduces VSysBench, a benchmark for evaluating multimodal large language models on constraint compliance and answer correctness under system messages, finding that system messages reduce task accuracy and compliance varies between open-weight and proprietary models.
MM-WebAgent: A Hierarchical Multimodal Web Agent for Webpage Generation
MM-WebAgent is a hierarchical agentic framework that generates coherent and visually consistent webpages by coordinating AIGC-based element generation through joint optimization of layout and multimodal content. The paper introduces a benchmark and multi-level evaluation protocol, demonstrating improvements over code-generation and agent-based baselines.
What We are Missing in Multimodal LLM Evaluation?
This paper reviews current multimodal LLM evaluation benchmarks and identifies key gaps such as temporal-spatial coherence, physical world understanding, multimodal consistency, and selective attention, arguing that existing isolated-task benchmarks fail to measure true cross-modal integration.