@LangChain: Brand new Max Agency with @cognition President @russelljkaplan + @hwchase17. YouTube: https://youtu.be/bBUotstDLdk?si=F…

Brand new Max Agency with @cognition President @russelljkaplan + @hwchase17. YouTube: https://youtu.be/bBUotstDLdk?si=Ff-wq0tBPTx-SC8V… Apple: https://podcasts.apple.com/us/podcast/the-misaligned-incentives-behind-ai-coding-agents/id1891551672?i=1000779120914… Spotify: https://open.spotify.com/episode/7JNB5YXA6NVgUDAhy8eA3V…

TL;DR: Cognition President Russell Kaplan talks about Devon’s evolution from viral demo to enterprise deployment, sharing insights on model intelligence saturation, cost optimization, and how the new coding benchmark Frontier Code measures “mergeability.”

Starting with the Devon Launch

Russell Kaplan began his machine learning career on Tesla’s autonomous driving team. Around 2017, the bottleneck was obvious: GPUs and compute. And now when he talks to his friends at Tesla, he notes that the real bottleneck is no longer training bigger models—it’s running evaluations. Because interventions are so rare, you have to do massive amounts of driving to find problems.

This line of thinking carries over to his views on software engineering agents. Cognition’s Devon was initially released in March 2024, and the demo video went massively viral. At that time, coding agents were just on the edge of possibility—if you squinted a little, you could see that “this thing could work someday.” Devon scored around 13% on SweetBench, tripling the previous high.

Early Product-Market Fit: Migration and Refactoring

It took months to go from “prototype” to “actually useful internally.” In June 2024, Devon became Devon’s top committer, which was the first big milestone. But there was a lot of manual self-testing at that time, working hard to polish internal workflows, and it took several more months before it was truly deployed to production and creating value for customers.

Back in that 2024 era, async cloud coding agents couldn’t handle most software engineering tasks, but there were already some niche use cases. Russell believes one of the best early applications was migration and refactoring. If you essentially ran a ReAct++-style workflow across a large codebase, with a few intelligent touches sprinkled in, that could already work quite well by late 2024.

Why was this scenario particularly suited?

• For very large refactors, migrations, or ETL transformations, it’s worth investing effort in careful prompt engineering.
• You can iteratively refine the prompts, settings, and context until it’s just right, then apply it across ten thousand modules in the codebase.
• This is far better than simple find-and-replace or string manipulation, but it doesn’t yet require fully general software intelligence.

So the early product-market fit came more from larger, enterprise-oriented companies that had huge amounts of code needing these kinds of transformations.

Self-Serve Launch and the Slack Interface

In December 2024, Cognition released a version of Devon that anyone could sign up for. Russell recalls that internally there was a big debate about “what should be the focus and what should be the core.” They even brought the entire company to Utah to sprint through December and ship before the end of the year.

The decision ultimately landed on making Slack the primary interface. The entire Devon self-serve launch video—internally called “Devon inside Devon”—was made by @-mentioning Devon inside Devon. It was meant to emphasize this user experience shift: working with an agent feels more like collaborating with a teammate.

Since then, users and momentum have both increased. Cloud agents keep getting better, infrastructure has matured, models have improved, and the toolchain has gotten better. Now most of the work is done by delegating to async agents.

What Really Makes Agents Take Off?

Russell mentions that people always say models have gotten better, which is certainly important, but infrastructure maturity is just as critical.

When asked, “Which model was the first that was truly good enough?” he said they kept saying “this is the one, this is the one,” but there were indeed several step changes:

• The model series around November 2024 was a huge step function, coming from OpenAI. Devon’s self-serve launch was partly based on that.
• Anthropic’s models started getting very good, and there was another big step around July 2025.
• Now new-generation models like Fable 5 are making continuous step changes.

But the meaning of those step changes has shifted. More and more software engineering tasks are hitting intelligence saturation. Many developers’ focus has moved from “wait, I need to use the best model” to “oh my god, I’m spending so much money on coding agents—how can I be more efficient?”

Russell believes this is an underappreciated aspect of frontier intelligence’s continued progress: for certain workloads, you always want the frontier model—especially in adversarial settings between participants, like competitive trading. But for a lot of software you want to build, there’s a saturation threshold; once it’s good enough, what you care about is speed and cost. The demand he sees from developers and partner companies is exactly that: “Okay, this already works. How do I now optimize for speed and cost?” Some organizations are already seeing per-person token spending exceed labor compensation costs.

Multi-Model Strategy: Combining Fable and GP 5.5

Among the latest generation of models, Russell points to Fable as an example, noting that the biggest improvements first came in detecting and fixing cybersecurity vulnerabilities. However, the public versions of these models have many safety guardrails that can actually make it hard to get them to help.

The example they see is “cleaning up my security backlog”—which is a huge workload for partner companies. Fable-class models are very good at high precision, but they actually found that GP 5.5 and 5.5 Cyber perform better on recall. So the best current toolchain for finding and fixing security vulnerabilities has to use both: essentially using the Fable family to triage and filter, while the vulnerabilities themselves may be better discovered by the GP family.

Another category of step-change improvement is handling data from integrations like Datadog or other systems. Russell believes this is largely because the data is too large and too messy, requiring more capacity considerations and multi-step reasoning. On these very real data sources, the scale of reinforcement learning has also expanded dramatically.

Frontier Code: Measuring “Mergeability”

When Devon launched, SweetBench had only a score in the teens; now SweetBench is completely saturated. To find the next level of difficulty for evaluating models, Cognition looked at all existing benchmarks but found that none of them matched that fuzzy internal intuition—“this model feels significantly better.”

Digging deeper, they found the core gap in evaluation lies in “mergeability”: the code is technically correct, but would you actually merge it? Would you be satisfied with it? Does it raise the quality of your codebase?

“Mergeability” can take many subtle forms:

• Following the style conventions you already expect;
• Being done in a way that’s easy to modify in the future;
• Handling future modifications gracefully if other changes come along.

Cognition set out to build a new benchmark to better measure this, and also to set a new, harder bar for frontier coding capability. This benchmark was recently released, called Frontier Code. The Diamond subset scored in the teens before Fable 5; now Fable 5 has pushed it into the thirties. Russell laughed, “I don’t know how many more months this benchmark can hold up.” He expects it to be like last year—a year from now people will look back and say, “Oh, remember when it first scored in the teens on this benchmark?”

Building these benchmarks is itself getting harder. Russell says software engineering evaluation hasn’t yet reached the level of Tesla’s autonomous driving, but they might hit that threshold sooner than people expect.

How Frontier Code Was Created

During the interview, it was mentioned that many teams use tools like linksmith to create benchmarks for their own systems. Cognition’s approach takes several steps:

First, they go to the developers with the most refined taste and the highest standards, the ones with strong stylistic quality standards and code correctness standards, to help answer “would you actually merge this code?”—not just “does it pass tests and function correctly.”

The second step is deeply collaborating with and recruiting many leading open source authors. They go to the most well-known open source libraries with very high coding standards, then work closely together to encode the human intuition of “what kind of PR would I accept” into carefully designed tests.

What do these tests look like?

• Some are programmatic assertions.
• There are certainly correctness tests, which are programmatic.
• There can also be programmatic assertions on style elements: for example, if you make this change in this pull request, you must use this module, even though using the other module is equivalent right now. But over time, if they diverge, not using the correct module will introduce bugs later. This can be deterministically asserted, but it takes the human author’s judgment to make that call.

Another important thing: every researcher on the Cognition team personally contributed to and reviewed the benchmark. As Russell put it, “This isn’t something you can hand off to someone else and say, ‘Okay, the data quality part—that’s grunt work, have someone else do it.’ You have to personally be involved.”

To prevent data contamination, they haven’t released the full problem set, only some sample problems, hoping to preserve the benchmark for the community as long as possible—until it gets solved.

Source: YouTube interview video (https://www.youtube.com/watch?v=bBUotstDLdk)