Paper page - Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models

Source: https://huggingface.co/papers/2610.35138 Can a coding agent turn a real user’s multimodal request into a complete, working application revision?

We’re sharingCo-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models, a benchmark built around real users’ requests to revise existing applications. Each task starts with a runnable application and asks the agent to understand the user’s intent, make the necessary changes, and satisfy every acceptance criterion.

Real requests, multimodal inputs, and connected changes.

Co-Design Bench contains273 tasks and 2,331 expert-authored acceptance criteria, curated from37,463 deduplicated queries across 26,314 real user coding sessions. It covers two complementary tracks:

• **Multimodal Editing (100 tasks):**localized revisions grounded in annotated screenshots and optional image, audio, video, or document inputs.
• **Repository Editing (173 tasks):**coordinated changes across components, routes, state, data flow, and content.

Requests can involve reproducing distinct text animations from a video, correcting a dashboard from a receipt and updating its dependent totals, or coordinating layout and interaction changes across an application. Tasks spanGrounded Understanding, UI Transformation, and Functional Implementation, often combining several capabilities within one request.

Evaluate the running application, with evidence for each requirement.

Our reference-freejudge-then-refuteprotocol assesses hidden acceptance criteria using source code and browser evidence. A separate verifier challenges positive judgments for stubs, hardcoded outputs, unreachable functionality, and requirement mismatches. Evidence audits require screenshots for visual claims and recorded browser actions for interaction claims.No gold implementation or target screenshot is required.

Partial requirement satisfaction and complete delivery tell different stories.

Across10 model endpoints under a shared agentic harness, the strongest endpoint,gpt-6-astra, passes an average of69.9% of criteria per task, yet fully satisfies only26.4% of tasks. All ten endpoints achieve lower criterion completion on repository requests spanning three or four edit types than on those spanning one or two.

Qwen3.8-27Breaches64.4% overall strict completion, ranking fourth overall, and64.5% on Repository Editing, ranking second on that track in this evaluation. It also leads the text-only Multimodal Editing subset at77.9%. Rankings vary with the editing setting; the small repository-track score differences do not establish a general model advantage.

TheME-96 subset comprises 96 Multimodal Editing tasks and 521 acceptance criteria, a distinct scope from the full 273-task paper evaluation.

Explore the real cases, demonstrations, evaluation tools, and results:

Project & demos·Paper·Code & evaluation·Dataset

We welcome researchers and builders working on coding agents, multimodal grounding, and reliable application editing to try the benchmark and share their findings!