Tag
Cognition introduces FrontierCode 2.0, a coding benchmark that evaluates whether a model's pull request would actually be merged by a maintainer — testing intelligence, codebase fluency, and judgement beyond simply meeting a spec.
Cognition 发布 FrontierCode 2.0 编码基准，采用更广泛的仓库覆盖、更严格的可合并性标准和更大的任务规模，结果显示顶级模型 Claude Opus 5.5 也仅得 45.5%，表明智能体软件工程远未被解决。
GPT-6 Sol and Luna models are now available in Devin, with GPT-6 Sol matching GPT-5.6 Sol's performance at 61% lower cost and GPT-6 Luna outperforming at about a quarter of the cost.
A public service announcement advises using medium thinking effort for the Opus 5 AI model, which ranked second on the FrontierCode benchmark with better cost-efficiency than the xhigh effort setting.
Cognition updated FrontierCode 1.1 to include new discounts for GPT-5.6 Terra and GPT-5.6 Luna, placing the GPT-5.6 series on the Pareto curve of price/performance efficiency.
Kimi K3, an open source AI model, is now available in Devin Desktop and CLI, approaching frontier-level performance on FrontierCode 1.1.
FrontierCode is a new coding evaluation benchmark that measures code mergeability, claiming 81% fewer misclassification errors than SWE-Bench Pro. Tasks were crafted by maintainers of open-source projects like Celery, uppy, and Mattermost.
FrontierCode is a new coding benchmark from METR and Cognition that evaluates AI models on code maintainability and quality, revealing that many models produce unmergeable code. It includes over 1000 hours of work and shows that even top models struggle, with Opus 4.8 achieving only 13.8% on the hardest tier.
Cognition announces FrontierCode, a new coding evaluation benchmark that goes beyond unit tests to measure code quality, scope, test correctness, and human reviewer approval, addressing the issue of agents writing sloppy code that passes tests but is not maintainable.
Cognition introduces FrontierCode, a high-quality coding benchmark that goes beyond unit tests to measure code maintainability, regression safety, and quality, with 150 handcrafted tasks by open-source developers.
SanthProject praises Cognition's new FrontierCode coding evaluation benchmark, calling it a fair alternative to the DeepSwe benchmark.