SWEeper-Bench: Can Agents Discover Bugs in Interactive Software?

Hugging Face Daily Papers Papers
coding-agents benchmark software-bugs interactive-software agentic-verification web-applications

Summary

SWEeper-Bench 引入了一个包含200个任务的基准，这些任务来自200个广泛使用的开源Web应用的真实bug修复，用于评估编码agent能否在交互式软件中自主发现依赖多步交互才能暴露的缺陷。分析显示，即使是15个前沿agent中最好的也仅通过59.0%的任务，主要瓶颈在于识别bug行为本身。

Coding agents increasingly interact with users by generating interactive software on the fly rather than replying in text, yet many bugs in such software reveal themselves only after multi-step interactions. Today, users must discover these bugs and wait for fixes, making the development loop slow and expensive. Existing benchmarks typically assume the bugs have already been identified, leaving agents' ability to find interaction-dependent failures largely unevaluated. We introduce SWEeper-Bench, 200 tasks built from real bug fixes in 200 widely used open-source web applications. Each task gives the agent only the codebase and an open-ended request such as "Find and fix issues in Focalboard's board sharing and membership." Because these bugs are hard to detect from code alone, agents must design interactive tests to uncover them. An agentic verifier then judges each repair by what a user would observe in the browser. The best of 15 frontier agents passes only 59.0% of tasks. Through stage-by-stage analysis of agent trajectories, we show that the biggest bottleneck is identifying the buggy behavior: agents often reach the relevant feature but never trigger the failure or fail to recognize it. SWEeper-Bench measures progress toward agents that close this loop themselves, delivering software that works without waiting for user feedback.
Original Article
View Cached Full Text

Cached at: 10/15/26, 07:03 AM

Paper page - SWEeper-Bench: Can Agents Discover Bugs in Interactive Software?

Source: https://huggingface.co/papers/2610.14096

Abstract

Codingagentsincreasinglyinteractwithusersbygeneratinginteractivesoftwareontheflyratherthanreplyingintext,yetmanybugsinsuchsoftwarerevealthemselvesonlyaftermulti-stepinteractions.Today,usersmustdiscoverthesebugsandwaitforfixes,makingthedevelopmentloopslowandexpensive.Existingbenchmarkstypicallyassumethebugshavealreadybeenidentified,leavingagents’abilitytofindinteraction-dependentfailureslargelyunevaluated.WeintroduceSWEeper-Bench,200tasksbuiltfromrealbugfixesin200widelyusedopen-sourcewebapplications.Eachtaskgivestheagentonlythecodebaseandanopen-endedrequestsuchas“FindandfixissuesinFocalboard’sboardsharingandmembership.“Becausethesebugsarehardtodetectfromcodealone,agentsmustdesigninteractiveteststouncoverthem.Anagenticverifierthenjudgeseachrepairbywhatauserwouldobserveinthebrowser.Thebestof15frontieragentspassesonly59.0%oftasks.Throughstage-by-stageanalysisofagenttrajectories,weshowthatthebiggestbottleneckisidentifyingthebuggybehavior:agentsoftenreachtherelevantfeaturebutnevertriggerthefailureorfailtorecognizeit.SWEeper-Benchmeasuresprogresstowardagentsthatclosethisloopthemselves,deliveringsoftwarethatworkswithoutwaitingforuserfeedback.

View arXiv pageView PDFProject pageGitHub2Add to collection

Get this paper in your agent:

hf papers read 2610\.14096

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.14096 in a model README.md to link it from this page.

Datasets citing this paper2

#### zlab-princeton/SWEeper-Bench-traces Viewer• Updatedabout 15 hours ago • 5.2k • 5.12k • 1 #### zlab-princeton/SWEeper-Bench Viewer• Updatedabout 15 hours ago • 200 • 258 • 2

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.14096 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

SWE-MISBEHAVIOR: Benchmarking and Mitigating Boundary Violations in Software Coding Agents

Hugging Face Daily Papers

该论文提出 SWE-MISBEHAVIOR，首个用于诊断软件编码智能体边界违规行为的分类体系与评测框架，包含 7.4K 个人工参与合成样本，覆盖代码生成、文件管理与系统交互三层行动空间，共 11 种失当行为。基于此，作者进一步提出轻量级自适应框架 CODE-KIT，使 smolagents 与 OpenHands 的失当行为分别减少 52% 和 34%，同时提升任务完成表现。

SWE-at-Scale: Scalable Issue Synthesis from Historical Bug Fixes

Hugging Face Daily Papers

SWE-at-Scale 是一个通过反演真实世界软件修复来自动生成可执行 bug 任务的框架，用于大规模、多样化的编码智能体评估，实验表明其生成的任务比现有合成基线更具多样性和挑战性。

Property Testing with Agent Swarms

Lobsters Hottest

作者展示了如何用 AI 编码代理（agent swarms）构建属性测试基础设施，在不消耗额外 token 的情况下自动发现真实开源项目中的正确性 bug，仅在 Codex、uv、Prometheus、Router 等项目中就发现了数十个已合并修复的缺陷。