@garrytan: This is the new standard for engineering evals
Summary
Announcing DeepSWE, a new benchmark for agentic coding that reveals true differences between models, reflecting real-world developer experiences.
View Cached Full Text
Cached at: 05/26/26, 10:58 PM
This is the new standard for engineering evals https://t.co/UkEwUWybab
Serena Ge (Datacurve) (@serenaa_ge): Today we’re releasing DeepSWE, a new standard for agentic coding benchmarks.
On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.
Similar Articles
Someone did an audit on the new DeepSWE, the results aren't pretty
DeepSWE is a new benchmark for evaluating AI coding agents on real-world software engineering tasks from active open-source repositories, comprising 113 tasks across TypeScript, Go, Python, JavaScript, and Rust with isolated environments and program-based verifiers.
@SanthProject: Now this is a bench i can get behind not the rigged as fuck deepswe benchmark
SanthProject praises Cognition's new FrontierCode coding evaluation benchmark, calling it a fair alternative to the DeepSwe benchmark.
@garrytan: https://x.com/garrytan/status/2054064931515855118
Garry Tan argues that AI coding agents like Claude Code and Codex have changed software engineering by making high test coverage affordable, creating a 'complexity ratchet' that ensures code quality improves over time without sacrificing speed.
@garrytan: My simple secret to agentic coding https://forbes.com/sites/josipamajic/2026/04/12/the-yc-chief-who-codes-10000-lines-a…
Garry Tan of Y Combinator shares his 'thin harness, fat skills' agentic coding framework, while the accidental leak of Claude Code's source code reveals the complex scaffolding behind AI coding agents.
SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions
SWE-Interact is a new testbed that evaluates coding agents in realistic multi-turn, user-driven software engineering tasks, revealing that strong single-turn benchmark performance does not reliably transfer to interactive, iterative workflows where agents must discover user intent and adapt to evolving requirements.