ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders

Hugging Face Daily Papers Papers
coding-agents benchmark evaluation interactive-project-building vibe-coding user-simulation black-box-tests

Summary

Introduces ICAE-Bench, a benchmark for evaluating coding agents in interactive project-building settings, simulating fuzzy product requirements with an automated User Agent and incorporating standardized black-box tests and multi-dimensional diagnostics.

The recent emergence of vibe-coding workflows is changing what coding agents are expected to do. Instead of merely completing code under fully specified instructions, agents are increasingly expected to transform incomplete product intent into working software by combining various abilities including planning, requirement clarification, tool use, debugging, and repository-level construction. Yet existing benchmarks have not fully caught up with this shift, evaluating agents on static, fully specified tasks. In this paper, we introduce ICAE-Bench, a benchmark for evaluating coding agents under interactive project-building settings. The basic idea is to start from a fuzzy product requirement, simulating the dynamic paradigm with an automated User Agent. To make this setting both realistic and evaluable, ICAE-Bench introduces three key designs. First, to avoid the ambiguity of unconstrained fuzzy requirements, each task derives ambiguity from a precise real open-source repository with executable behavior. Second, to ensure high-quality and reproducible user simulation, ICAE-Bench grounds interaction through User Agent Data, allowing the User Agent to reveal hidden constraints without inventing new requirements or leaking implementation artifacts. Third, to evaluate open-ended repositories fairly, ICAE-Bench uses standardized black-box tests together with multi-dimensional diagnostics, including functional correctness, semantic and API similarity, structural fidelity, design quality, and interaction quality.
Original Article
View Cached Full Text

Cached at: 07/24/26, 05:06 AM

Paper page - ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders

Source: https://huggingface.co/papers/2607.21217 Authors:

,

,

,

,

,

,

,

,

,

Abstract

Therecentemergenceofvibe-codingworkflowsischangingwhatcodingagentsareexpectedtodo.Insteadofmerelycompletingcodeunderfullyspecifiedinstructions,agentsareincreasinglyexpectedtotransformincompleteproductintentintoworkingsoftwarebycombiningvariousabilitiesincludingplanning,requirementclarification,tooluse,debugging,andrepository-levelconstruction.Yetexistingbenchmarkshavenotfullycaughtupwiththisshift,evaluatingagentsonstatic,fullyspecifiedtasks.Inthispaper,weintroduceICAE-Bench,abenchmarkforevaluatingcodingagentsunderinteractiveproject-buildingsettings.Thebasicideaistostartfromafuzzyproductrequirement,simulatingthedynamicparadigmwithanautomatedUserAgent.Tomakethissettingbothrealisticandevaluable,ICAE-Benchintroducesthreekeydesigns.First,toavoidtheambiguityofunconstrainedfuzzyrequirements,eachtaskderivesambiguityfromapreciserealopen-sourcerepositorywithexecutablebehavior.Second,toensurehigh-qualityandreproducibleusersimulation,ICAE-BenchgroundsinteractionthroughUserAgentData,allowingtheUserAgenttorevealhiddenconstraintswithoutinventingnewrequirementsorleakingimplementationartifacts.Third,toevaluateopen-endedrepositoriesfairly,ICAE-Benchusesstandardizedblack-boxteststogetherwithmulti-dimensionaldiagnostics,includingfunctionalcorrectness,semanticandAPIsimilarity,structuralfidelity,designquality,andinteractionquality.

View arXiv pageView PDFGitHub3Add to collection

Get this paper in your agent:

hf papers read 2607\.21217

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.21217 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.21217 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.21217 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

Hugging Face Daily Papers

SWE-Interact is a new testbed that evaluates coding agents in realistic multi-turn, user-driven software engineering tasks, revealing that strong single-turn benchmark performance does not reliably transfer to interactive, iterative workflows where agents must discover user intent and adapt to evolving requirements.