Paper page - RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests

Source: https://huggingface.co/papers/2608.27831

Abstract

Real-world coding requests are shorter and more casual than benchmark tasks, and explicitly stating desired behavior and motivation improves LLM software engineering performance.

Coding agents are now commonly evaluated on theSWE-benchfamily of benchmarks, whose tasks are built from curated GitHub issues: long, structured, and information-rich. Real user requests, however, are typically far shorter and less structured. To characterize this gap, we define a six-categoryinformation taxonomyand four dimensions oflinguistic style, and apply them to real user prompts from SWE-chat and problem statements fromSWE-benchVerified and Pro. We find that requests carrying only a problem statement, alone or with limited additional context, account for 88% of real prompts but just 7% of benchmark problems. Furthermore, 87% of real prompts are casually written whereas 94% of benchmark problems are formal. Guided by these observations, we introduceRealSWE, 381 multi-variant task families derived fromSWE-benchVerified and Pro. Variants within each family share the same underlying task andgold patchwhile differing only in information composition andlinguistic style. Evaluating seven contemporaryLLMswithRealSWE, we find that i) realistic inputs reduceresolution ratesby 6.4 pp on average and can change model rankings. Controlled analysis further shows that ii) including Desired Behavior and Motivation significantly affects performance, whereas Environment Information and Reproduction Steps merely add tokens without measurable benefit; iii)linguistic stylehas only small, model-dependent effects. These findings provide actionable guidance for users and agents: explicitly stating the desired behavior and motivation, which most real prompts omit, substantially improves the LLM’s software engineering performance.

View arXiv pageView PDFGitHubAdd to collection

Get this paper in your agent:

hf papers read 2608\.27831

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.27831 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.27831 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.27831 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.