Devin’s engineering system guarantees more output than cost, with $10M pledged per customer. The system’s effectiveness is validated using independent data. This bold claim aims to bolster confidence in their engineering productivity.

$10,000,000 on the line: how we measure Devin’s engineering output

We’re putting up to $10M per customer behind a single claim: Devin delivers more engineering output than you pay for. This is the system we built to prove it, validated on independent data we collected from engineers across our enterprise customers.

A first-of-its-kind dataset

We began by collecting a dataset of 258 sessions from 126 users across a diverse set of enterprise customers. To the best of our knowledge, this is the first human-annotated dataset of enterprise coding agent traces.

Filtering for real work

Not every session results in productive work, so we filtered the dataset down to sessions that did. We removed sessions where none of the PRs were ever merged. For sessions that produced no PRs at all, we ran a classifier to filter out the ones that didn’t result in a valuable outcome.

The estimator agent

We then built an estimator agent, which we thought of as two parts.

Context: we gave the agent the user’s messages, the PR produced (when there was one), the full agent trace, and additional codebase context from DeepWiki.

Prompting: we set aside 25 sessions as a development set. By manually triaging the agent’s runs on those sessions, we added 4 principles to our prompt to improve accuracy and err conservative. The full details are in our technical report.

How well does it work?

On our held-out evaluation set, the estimator reaches a correlation of 0.74 on a log scale and appears to be unbiased. Individual estimates are noisy, but because the errors are roughly unbiased, they wash out when aggregated across many sessions.

What’s actually driving the signal?

We also tested simpler predictors to understand how much of the signal comes from the final code change versus the full Devin session: the total lines changed (additions plus deletions) across all PRs in a session, and the estimator given only the trace of the agent’s edit tool calls. The takeaway: the signal lives outside the diff. Much of a session’s real effort is investigation, diagnosis, and reasoning, and the most valuable work sometimes produces no code at all.

The result, and the guarantee

Run across historical customer data, Devin delivers roughly 4x the engineering output you pay for, valued at a conservative blended global wage.

We’re now standing behind it with the AI Productivity Guarantee. If Devin ever delivers less value than you pay for, we fund your usage until it does, up to $10M per customer.

Read the full methodology, eval, and limitations in our technical report: https://cognition.ai/blog/ai-productivity.