@denizbirlikci: To understand why we built FrontierCode, read @METR_Evals's blog post on why "many SWE-bench-passing PRs would not be m…
Summary
Cognition announces FrontierCode, a new coding evaluation benchmark that goes beyond unit tests to measure code quality, scope, test correctness, and human reviewer approval, addressing the issue of agents writing sloppy code that passes tests but is not maintainable.
View Cached Full Text
Cached at: 06/09/26, 10:46 AM
To understand why we built FrontierCode, read @METR_Evals’s blog post on why “many SWE-bench-passing PRs would not be merged into main.” A bit old now, but the point still stands.
Agents often write more code — and more slop — than they should. But unit tests have no way to penalize those unnecessary changes; passing is passing, no matter how much junk came along with it.
FrontierCode actively tests for this with rubric types that grade on more ambiguous metrics, like:
- SCOPE — did it change more than it should have?
- Test correctness — do the agent’s own tests actually catch the bug?
- Code quality — would a human reviewer approve this diff? (LLM-judge on human rubric)
Cognition (@cognition): Introducing FrontierCode: a coding eval that raises the bar for difficulty & quality. Each task took 40+ hrs of work by leading open-source maintainers.
Models write sloppy code that works but isn’t maintainable. Our eval is first to measure: would you actually merge this code?
Similar Articles
@dabit3: FrontierCode is the first eval to measure the metric that matters most in real software engineering: would you actually…
FrontierCode is a new coding evaluation benchmark that measures code mergeability, claiming 81% fewer misclassification errors than SWE-Bench Pro. Tasks were crafted by maintainers of open-source projects like Celery, uppy, and Mattermost.
FrontierCode: a coding eval that raises the bar for difficulty & quality.
FrontierCode is a new coding evaluation benchmark designed to increase difficulty and quality standards for AI code generation.
FrontierCode
FrontierCode is a new benchmark from Cognition AI that measures AI models' ability to write high-quality, maintainable code by evaluating mergeability. Results show even top models like Claude Opus 4.8 score only 13.4% on the hardest subset, highlighting a significant gap in code quality.
@Murderlon: FrontierCode finally dropped, a coding agents benchmark for the real world. Human-verified through an extensive hardeni…
FrontierCode is a new benchmark for coding agents, human-verified with a continuous scoring model, designed to evaluate real-world performance.
@swyx: It's finally out!!! @METR_Evals found that more than half of SWEBench results is unmergeable slop. FrontierCode represe…
FrontierCode is a new coding benchmark from METR and Cognition that evaluates AI models on code maintainability and quality, revealing that many models produce unmergeable code. It includes over 1000 hours of work and shows that even top models struggle, with Opus 4.8 achieving only 13.8% on the hardest tier.