SWE-chat: Coding Agent Interactions From Real Users in the Wild

Hugging Face Daily Papers Papers
coding-agents real-world-dataset developer-workflows ai-assisted-coding empirical-study

Summary

SWE-chat introduces a 6,000-session dataset of real-world coding agent interactions, revealing that only 44% of agent-generated code survives in commits and highlighting inefficiencies and security issues in current AI-assisted development.

AI coding agents are being adopted at scale, yet we lack empirical evidence on how people actually use them and how much of their output is useful in practice. We present SWE-chat, the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild. The dataset currently contains 6,000 sessions, comprising more than 63,000 user prompts and 355,000 agent tool calls. SWE-chat is a living dataset; our collection pipeline automatically and continually discovers and processes sessions from public repositories. Leveraging SWE-chat, we provide an initial empirical characterization of real-world coding agent usage and failure modes. We find that coding patterns are bimodal: in 41% of sessions, agents author virtually all committed code ("vibe coding"), while in 23%, humans write all code themselves. Despite rapidly improving capabilities, coding agents remain inefficient in natural settings. Just 44% of all agent-produced code survives into user commits, and agent-written code introduces more security vulnerabilities than code authored by humans. Furthermore, users push back against agent outputs -- through corrections, failure reports, and interruptions -- in 44% of all turns. By capturing complete interaction traces with human vs. agent code authorship attribution, SWE-chat provides an empirical foundation for moving beyond curated benchmarks towards an evidence-based understanding of how AI agents perform in real developer workflows.
Original Article
View Cached Full Text

Cached at: 04/23/26, 03:35 AM

Paper page - SWE-chat: Coding Agent Interactions From Real Users in the Wild

Source: https://huggingface.co/papers/2604.20779

Abstract

SWE-chat presents a large-scale dataset of real coding agent interactions that reveals significant inefficiencies and challenges in current AI-assisted development practices.

AIcodingagentsarebeingadoptedatscale,yetwelackempiricalevidenceonhowpeopleactuallyusethemandhowmuchoftheiroutputisusefulinpractice.WepresentSWE-chat,thefirstlarge-scaledatasetofrealcodingagentsessionscollectedfromopen-sourcedevelopersinthewild.Thedatasetcurrentlycontains6,000sessions,comprisingmorethan63,000userpromptsand355,000agenttoolcalls.SWE-chatisalivingdataset;ourcollectionpipelineautomaticallyandcontinuallydiscoversandprocessessessionsfrompublicrepositories.LeveragingSWE-chat,weprovideaninitialempiricalcharacterizationofreal-worldcodingagentusageandfailuremodes.Wefindthatcodingpatternsarebimodal:in41%ofsessions,agentsauthorvirtuallyallcommittedcode(“vibecoding”),whilein23%,humanswriteallcodethemselves.Despiterapidlyimprovingcapabilities,codingagentsremaininefficientinnaturalsettings.Just44%ofallagent-producedcodesurvivesintousercommits,andagent-writtencodeintroducesmoresecurityvulnerabilitiesthancodeauthoredbyhumans.Furthermore,userspushbackagainstagentoutputs--throughcorrections,failurereports,andinterruptions--in44%ofallturns.Bycapturingcompleteinteractiontraceswithhumanvs.agentcodeauthorshipattribution,SWE-chatprovidesanempiricalfoundationformovingbeyondcuratedbenchmarkstowardsanevidence-basedunderstandingofhowAIagentsperforminrealdeveloperworkflows.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2604\.20779

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2604.20779 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2604.20779 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2604.20779 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

@Xudong07452910: This paper is a must-read for heavy users of Claude Code, Codex, or other AI Agents. It doesn't study how Agents fail on benchmarks, but a more real problem: In real development, what exactly are AI coding agents doing...

X AI KOLs Timeline

This paper analyzes 20,574 real-world coding-agent sessions to identify how AI agents misalign with developer intent, finding that constraint violations and inaccurate self-reporting are the most common failure modes, imposing trust and effort costs rather than irreversible damage.

Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents

arXiv cs.CL

Introduces Dialogue-SWE-Bench, a benchmark for evaluating coding agents' ability to resolve software engineering problems through dialogue with a user. Proposes a persona-grounded user simulator and a schema-guided agent that improves dialogue capabilities.

Agentic Code Review (15 minute read)

TLDR AI

An analysis of how AI coding agents have shifted the bottleneck from writing code to reviewing it, with data showing a 861% increase in code churn and a rise in defect rates, making code review the most leveraged skill in software engineering.