Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists

Hugging Face Daily Papers Papers
llm-benchmark scientific-discovery capability-benchmark sda-bench evaluation ai-scientists

Summary

Introduces SDABench, a benchmark evaluating LLMs on six scientific analysis capabilities across five domains, finding models struggle with tasks requiring assumption selection and mechanistic reasoning.

Existing benchmarks for scientific data analysis evaluate LLMs primarily on code execution or workflow completion, overlooking that scientific analysis serves to support distinct types of scientific claims: hypothesis exploration, statistical inference, mechanistic explanation, each with different assumptions and validity criteria. We introduce SDABench, a benchmark that reorganizes evaluation around six capabilities (descriptive, exploratory, inferential, predictive, causal, and mechanistic) across five domains (Biology, Chemistry, Environment, Geography, Physics). SDABench comprises 527 real-data instances (SDA-Real) and 6000 synthetic instances (SDA-Synth), each in both multiple-choice and open-ended formats, constructed through an automated pipeline. Evaluating 15 representative LLMs, we find that models handle descriptive analysis well but degrade sharply on tasks requiring assumption selection, latent-process modeling, or mechanistic reasoning. SDABench further provides a five-stage error analysis framework that locates where LLMs fail: more advanced models more reliably identify the relevant scope and variables, but still struggle to select appropriate analytical procedures, model variable relationships, and draw valid conclusions.
Original Article
View Cached Full Text

Cached at: 07/16/26, 01:40 AM

Paper page - Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists

Source: https://huggingface.co/papers/2607.11079

Abstract

ExistingbenchmarksforscientificdataanalysisevaluateLLMsprimarilyoncodeexecutionorworkflowcompletion,overlookingthatscientificanalysisservestosupportdistincttypesofscientificclaims:hypothesisexploration,statisticalinference,mechanisticexplanation,eachwithdifferentassumptionsandvaliditycriteria.WeintroduceSDABench,abenchmarkthatreorganizesevaluationaroundsixcapabilities(descriptive,exploratory,inferential,predictive,causal,andmechanistic)acrossfivedomains(Biology,Chemistry,Environment,Geography,Physics).SDABenchcomprises527real-datainstances(SDA-Real)and6000syntheticinstances(SDA-Synth),eachinbothmultiple-choiceandopen-endedformats,constructedthroughanautomatedpipeline.Evaluating15representativeLLMs,wefindthatmodelshandledescriptiveanalysiswellbutdegradesharplyontasksrequiringassumptionselection,latent-processmodeling,ormechanisticreasoning.SDABenchfurtherprovidesafive-stageerroranalysisframeworkthatlocateswhereLLMsfail:moreadvancedmodelsmorereliablyidentifytherelevantscopeandvariables,butstillstruggletoselectappropriateanalyticalprocedures,modelvariablerelationships,anddrawvalidconclusions.

View arXiv pageView PDFAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.11079 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.11079 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.11079 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

SciR: A Controllable Benchmark for Scientific Reasoning in LLMs

arXiv cs.AI

SciR is a new controllable benchmark for evaluating LLMs on scientific reasoning including deduction, induction, and causal abduction, with parametric control over extraction and inference difficulty. Tests show both axes degrade performance across models, with reasoning models like DeepSeek-R1 outperforming instruct models on inference.

Do Benchmarks Underestimate LLM Performance? Evaluating Hallucination Detection With LLM-First Human-Adjudicated Assessment

arXiv cs.CL

This paper investigates whether standard benchmarks underestimate LLM performance by re-evaluating hallucination detection datasets using an LLM-first, human-adjudicated assessment method. The study finds that incorporating LLM reasoning into the adjudication process improves agreement and suggests that model-assisted re-evaluation yields more reliable benchmarks for ambiguity-prone tasks.

ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment

arXiv cs.AI

Introduces ForeSci, a temporally controlled benchmark for evaluating whether LLM agents can make forward-looking research judgments from historical evidence. It contains 500 tasks across four AI domains and shows that explicit evidence organization improves traceability but reveals a recurring evidence-decision decoupling.