Can AI agents conduct open-ended AI research? Early evidence from two case studies

Hugging Face Daily Papers Papers
ai-agents ai-research automation evaluation case-studies failure-modes shadow-evaluations

Summary

This paper introduces a novel evaluation method called shadow evaluations to test whether AI agents can conduct open-ended AI research. In two case studies, agents completed all engineering without human help but could not make substantial progress on the research questions, revealing five recurring failure modes.

Forecasts of explosive AI progress hinge on AI agents automating AI research. But evidence on whether agents can carry out open-ended AI research is thin. Current evaluations either test agents on narrow, verifiable tasks, which excludes open-ended research, or submit AI-generated papers to blind peer review, which is overstretched, stochastic, and suffers from poor review quality. We introduce a third way to measure progress towards AI R\&D automation. An agent takes on the central, open-ended research question of a high-quality unpublished paper, and the paper's original authors grade its output. We call these shadow evaluations. We ran shadow evaluations on two unpublished NeurIPS 2026 submissions, giving frontier agents six days and thousands of dollars of compute. The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions. As a result, both papers were unambiguously rejected by the authors. We identify five recurring failure modes: poor judgment about the bar for publishable research, uncreative responses to shortcomings in the research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift. A robustness check with a second model and scaffold reproduced these failures. We release the expert reviews, survey responses, agent repositories, and logs. Our results provide early evidence that today's agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle.
Original Article
View Cached Full Text

Cached at: 07/30/26, 05:46 AM

Paper page - Can AI agents conduct open-ended AI research? Early evidence from two case studies

Source: https://huggingface.co/papers/2607.27191 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

ForecastsofexplosiveAIprogresshingeonAIagentsautomatingAIresearch.Butevidenceonwhetheragentscancarryoutopen-endedAIresearchisthin.Currentevaluationseithertestagentsonnarrow,verifiabletasks,whichexcludesopen-endedresearch,orsubmitAI-generatedpaperstoblindpeerreview,whichisoverstretched,stochastic,andsuffersfrompoorreviewquality.WeintroduceathirdwaytomeasureprogresstowardsAIR\&Dautomation.Anagenttakesonthecentral,open-endedresearchquestionofahigh-qualityunpublishedpaper,andthepaper’soriginalauthorsgradeitsoutput.Wecalltheseshadowevaluations.WeranshadowevaluationsontwounpublishedNeurIPS2026submissions,givingfrontieragentssixdaysandthousandsofdollarsofcompute.Theagentscompletedalloftheengineeringwithouthumanhelp,yetcouldnotmakesubstantialprogresstowardsansweringtheresearchquestions.Asaresult,bothpaperswereunambiguouslyrejectedbytheauthors.Weidentifyfiverecurringfailuremodes:poorjudgmentaboutthebarforpublishableresearch,uncreativeresponsestoshortcomingsintheresearchdesign,ineffectivebacktrackingfromdeadends,poorresourceawareness,andinstructiondrift.Arobustnesscheckwithasecondmodelandscaffoldreproducedthesefailures.Wereleasetheexpertreviews,surveyresponses,agentrepositories,andlogs.Ourresultsprovideearlyevidencethattoday’sagentscandotheengineeringofAIresearch,butstrugglewithcriticalpartsoftheresearchlifecycle.

View arXiv pageView PDFProject pageAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.27191 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.27191 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.27191 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Open-World Evaluations for Measuring Frontier AI Capabilities

arXiv cs.AI

This paper argues that traditional benchmarks both overestimate and underestimate frontier AI capabilities, and proposes 'open-world evaluations'—long-horizon, real-world tasks assessed qualitatively—as a complementary approach. The CRUX project is introduced, with a demonstration where an AI agent successfully published an iOS app to the App Store with minimal intervention.

An Empirical Study of Automating Agent Evaluation

arXiv cs.CL

This paper introduces EvalAgent, a system that automates the evaluation of AI agents by encoding domain-specific expertise, addressing the limitations of standard coding assistants in this task. It also presents AgentEvalBench, a benchmark for testing evaluation pipelines, and demonstrates significant improvements in evaluation reliability.

Frontier and Center: Who evaluates the evaluations? (12 minute read)

TLDR AI

Google Data Cloud's frontier AI team discusses a new approach to evaluating AI agents using information theory to create a meta-benchmark called Discovery Bench that measures how vague a query can be before an agent fails, providing a more nuanced map of agent capabilities than simple pass/fail exams.