AgentSWE: Can Coding Agents Build the Agent You Actually Want?

Hugging Face Daily Papers Papers
coding-agents benchmark agent-evaluation software-engineering agentic-ai

Summary

AgentSWE introduces a benchmark spanning agent creation, editing, and optimization, finding that no evaluated coding agent reliably delivers agents matching user requirements, with recurring failures like overfitting and hollow runtimes.

As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their users' own workflows. This delegation carries a quiet assumption, that the delivered agent will faithfully serve the requirement it was built for, rather than merely appear to. No existing benchmark tests that assumption. We introduce AgentSWE, a benchmark for agent software engineering spanning three stages of the agent development lifecycle: Creation of a complete agent from a natural-language requirement, Editing of production agent codebases, and Optimization toward a measurable target. Each of its 25 tasks is framed as a commission rather than an exam: the builder iterates against a handful of illustrative cases, while acceptance exercises the frozen submission on held-out cases designed to span the full requirement, grading behavior against requirement-derived rubrics under hard pass/fail gates or, for Optimization, against the source benchmark's own objective. Evaluating a range of coding agents across models and harnesses, we find that none comes close to delivering what a requirement asks for: the gap between what a builder demonstrates and what it delivers is the norm, not the corner case. An accompanying empirical analysis of the failure trajectories identifies recurring failure modes, among them overfitting to the visible cases, hollow runtimes behind code that looks complete, and unsupported claims of completion, each a signal by which coding agents can learn to build the agents they are asked for.
Original Article
View Cached Full Text

Cached at: 10/13/26, 06:44 AM

Paper page - AgentSWE: Can Coding Agents Build the Agent You Actually Want?

Source: https://huggingface.co/papers/2610.13361

Abstract

Astheproductionofsoftwarepassesintothehandsofsoftwareitself,codingagentsareincreasinglycommissionedtobuildagentsfittedtotheirusers’ownworkflows.Thisdelegationcarriesaquietassumption,thatthedeliveredagentwillfaithfullyservetherequirementitwasbuiltfor,ratherthanmerelyappearto.Noexistingbenchmarkteststhatassumption.WeintroduceAgentSWE,abenchmarkforagentsoftwareengineeringspanningthreestagesoftheagentdevelopmentlifecycle:Creationofacompleteagentfromanatural-languagerequirement,Editingofproductionagentcodebases,andOptimizationtowardameasurabletarget.Eachofits25tasksisframedasacommissionratherthananexam:thebuilderiteratesagainstahandfulofillustrativecases,whileacceptanceexercisesthefrozensubmissiononheld-outcasesdesignedtospanthefullrequirement,gradingbehavioragainstrequirement-derivedrubricsunderhardpass/failgatesor,forOptimization,againstthesourcebenchmark’sownobjective.Evaluatingarangeofcodingagentsacrossmodelsandharnesses,wefindthatnonecomesclosetodeliveringwhatarequirementasksfor:thegapbetweenwhatabuilderdemonstratesandwhatitdeliversisthenorm,notthecornercase.Anaccompanyingempiricalanalysisofthefailuretrajectoriesidentifiesrecurringfailuremodes,amongthemoverfittingtothevisiblecases,hollowruntimesbehindcodethatlookscomplete,andunsupportedclaimsofcompletion,eachasignalbywhichcodingagentscanlearntobuildtheagentstheyareaskedfor.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2610\.13361

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.13361 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.13361 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.13361 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Are AI coding agents hitting a wall, or are we just measuring them wrong?

Reddit r/AI_Agents

This article examines the gap between hype and reality for AI coding agents, arguing that they are effective for accelerating workflow parts but still require human oversight for architecture, debugging, and review, and questioning whether current benchmarks measure the right things.

SWE-Touch: Benchmarking Coding Agents When Users Touch the Code

Hugging Face Daily Papers

Introduces SWE-Touch, a benchmark framework that injects conflicting user edits during agent coding trajectories, showing that current coding agents significantly degrade in collaborative settings despite strong standalone benchmark performance.

Are coding agents exposing how bad our specs actually are?

Reddit r/AI_Agents

The article argues that many failures of AI coding agents stem from vague specifications, not just model weaknesses. It suggests that writing clearer, more detailed work packets may be the next essential skill for developers using coding agents.