SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?
Summary
SWE-bench Science introduces a repository-level benchmark for evaluating coding agents on scientific software repair tasks, revealing failure mechanisms and mixed effects of scientific guidance.
View Cached Full Text
Cached at: 08/21/26, 08:08 AM
Paper page - SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?
Source: https://huggingface.co/papers/2608.19799
Abstract
SWE-bench Science benchmarks coding agents on scientific software repair, revealing failure mechanisms and mixed effects of scientific guidance.
Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions. Yet existing evaluations ofcoding agentslargely emphasize aggregate task success, providing limited insight into why agents fail when repairing scientific software. We introduceSWE-bench Science, a repository-level benchmark forscientific software engineeringcomprising 119 tasks from 98 GitHub repositories across 20 scientific domains. Each task is organized into one of three paradigms: Issue-driven, Expert-exploratory, and Engineering-integration. Even the best-performing agent, Claude Code with Opus-5 (max), achieves apass@1below 50\%, highlighting the substantial challenges posed byscientific software engineering. We identify four recurringfailure mechanisms: deficits inscientific knowledgeor abstraction, misguided exploration or surface-level repair, incomplete repair coverage or system integration, and failures to generalizescientific knowledgebeyond observed cases in our analysis. We further conduct a pairedablationthat removes explicit scientific guidance while preserving the repository and executable engineering context. The results show thatscientific knowledgeis not uniformly beneficial: well-grounded information can constrain repair and improve average performance and token efficiency, whereas poorly aligned guidance can induce anchoring and does not necessarily improve exact repair success. Together,SWE-bench Scienceprovides a broad testbed for studying both the capabilities andfailure mechanismsofcoding agentsinscientific software engineering.
View arXiv pageView PDFProject pageGitHub13Add to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.19799 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.19799 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.19799 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers
Senior SWE-Bench is an open-source benchmark that evaluates AI agents on software engineering tasks requiring senior-level skills.
SWE Context Bench just proved something I think a lot of coding agent users already feel
A new benchmark paper 'SWE Context Bench' tests whether coding agents can reuse knowledge across tasks, highlighting a gap in existing benchmarks that only evaluate isolated problem-solving. The author discusses solutions like external memory and mentions tools such as langmem, mem0, supermemory, and Greplica.
SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions
SWE-Interact is a new testbed that evaluates coding agents in realistic multi-turn, user-driven software engineering tasks, revealing that strong single-turn benchmark performance does not reliably transfer to interactive, iterative workflows where agents must discover user intent and adapt to evolving requirements.
Senior SWE Bench: a new benchmark focussed on realistically underspecified feature tasks
Senior SWE-Bench is a new open-source benchmark designed to evaluate AI agents on realistic, underspecified software engineering tasks, emphasizing skills like intent alignment and code quality rather than overly detailed specifications.
SWE-Touch: Benchmarking Coding Agents When Users Touch the Code
Introduces SWE-Touch, a benchmark framework that injects conflicting user edits during agent coding trajectories, showing that current coding agents significantly degrade in collaborative settings despite strong standalone benchmark performance.