FrontierCS 2: A Benchmark for Computer Science R&D

Hugging Face Daily Papers Papers
benchmark llm-agents computer-science research-evaluation open-ended-tasks ai-evaluation

Summary

FrontierCS 2 introduces a benchmark of 48 open-ended computer science R&D tasks for LLM agents, placing them inside real research repositories with hidden evaluation suites; the best agent, Claude Opus 5.5, passes only about 58% of its runs.

Can LLM agents make contributions to frontier computer science R&D? We introduce FrontierCS 2, a benchmark of 48 open-ended tasks. We take a paper and the code its authors released and delete the paper's contribution. The deleted code becomes the human reference that sets the bar. Earlier open-ended benchmarks give each agent a problem statement and grade one program after the run. FrontierCS 2 places the agent inside a research environment. The agent inherits a working repository and may submit a patch at any point. For each patch, a judge returns a development-suite score but keeps the evaluator, the held-out final suite and the reference private, and grades the final submission on that suite against the reference. The tasks span 13 domains, from machine-learning systems and high-performance computing to databases, security and robotics. Beyond these 48 tasks, FrontierCS 2 keeps a private set of more than 100 tasks, to be announced in the future. We evaluate 12 agents. The best one is Claude Opus 5.5, passing 58% of its runs. No run passes on 16 tasks. A higher score is not the same as a pass: most runs that beat the reference on the combined final score still fall short on some metric or workload. On a shared capability scale for agents and tasks, the median task is roughly as difficult as what the strongest agent can handle.
Original Article
View Cached Full Text

Cached at: 10/22/26, 07:35 PM

Paper page - FrontierCS 2: A Benchmark for Computer Science R&D

Source: https://huggingface.co/papers/2610.29932 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

FrontierCS 2 evaluates LLM agents on open-ended computer science R&D tasks using hidden evaluation suites and reveals that even the best agents pass only about half of tasks.

CanLLM agentsmake contributions to frontier computer science R&D? We introduceFrontierCS 2, a benchmark of 48 open-ended tasks. We take a paper and the code its authors released and delete the paper’s contribution. The deleted code becomes the human reference that sets the bar. Earlieropen-ended benchmarks give each agent a problem statement and grade one program after the run.FrontierCS 2places the agent inside a research environment. The agent inherits a working repository and may submit a patch at any point. For each patch, a judge returns adevelopment-suite scorebut keeps the evaluator, the held-out final suite and the reference private, and grades the final submission on that suite against the reference. The tasks span 13 domains, frommachine-learning systemsandhigh-performance computingto databases, security and robotics. Beyond these 48 tasks,FrontierCS 2keeps a private set of more than 100 tasks, to be announced in the future. We evaluate 12 agents. The best one is Claude Opus 5.5, passing 58% of its runs. No run passes on 16 tasks. A higher score is not the same as a pass: most runs that beat the reference on the combined final score still fall short on some metric or workload. On a shared capability scale for agents and tasks, the median task is roughly as difficult as what the strongest agent can handle.

View arXiv pageView PDFProject pageGitHub27Add to collection

Get this paper in your agent:

hf papers read 2610\.29932

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.29932 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.29932 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.29932 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

FrontierCode

Hacker News Top

FrontierCode is a new benchmark from Cognition AI that measures AI models' ability to write high-quality, maintainable code by evaluating mergeability. Results show even top models like Claude Opus 4.8 score only 13.4% on the hardest subset, highlighting a significant gap in code quality.

FrontierCode 2.0 (13 minute read)

TLDR AI

Cognition 发布 FrontierCode 2.0 编码基准，采用更广泛的仓库覆盖、更严格的可合并性标准和更大的任务规模，结果显示顶级模型 Claude Opus 5.5 也仅得 45.5%，表明智能体软件工程远未被解决。