Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists
Summary
Introduces SDABench, a benchmark evaluating LLMs on six scientific analysis capabilities across five domains, finding models struggle with tasks requiring assumption selection and mechanistic reasoning.
View Cached Full Text
Cached at: 07/16/26, 01:40 AM
Paper page - Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists
Source: https://huggingface.co/papers/2607.11079
Abstract
ExistingbenchmarksforscientificdataanalysisevaluateLLMsprimarilyoncodeexecutionorworkflowcompletion,overlookingthatscientificanalysisservestosupportdistincttypesofscientificclaims:hypothesisexploration,statisticalinference,mechanisticexplanation,eachwithdifferentassumptionsandvaliditycriteria.WeintroduceSDABench,abenchmarkthatreorganizesevaluationaroundsixcapabilities(descriptive,exploratory,inferential,predictive,causal,andmechanistic)acrossfivedomains(Biology,Chemistry,Environment,Geography,Physics).SDABenchcomprises527real-datainstances(SDA-Real)and6000syntheticinstances(SDA-Synth),eachinbothmultiple-choiceandopen-endedformats,constructedthroughanautomatedpipeline.Evaluating15representativeLLMs,wefindthatmodelshandledescriptiveanalysiswellbutdegradesharplyontasksrequiringassumptionselection,latent-processmodeling,ormechanisticreasoning.SDABenchfurtherprovidesafive-stageerroranalysisframeworkthatlocateswhereLLMsfail:moreadvancedmodelsmorereliablyidentifytherelevantscopeandvariables,butstillstruggletoselectappropriateanalyticalprocedures,modelvariablerelationships,anddrawvalidconclusions.
View arXiv pageView PDFAdd to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.11079 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.11079 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.11079 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery
The paper introduces HypoArena, a benchmark for evaluating LLMs' ability to proactively construct hypothesis spaces from incomplete evidence, and experiments on 15 frontier LLMs reveal capability stratification.
Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning
This survey examines recent progress in medical LLMs, presenting a dual-view approach that connects clinical practice with computational methods, and introduces a benchmark dataset for evaluating medical reasoning capabilities across 18 state-of-the-art models.
Turns out that many current science-based LLM benchmarks have flaws in their answers. When corrected, the LLM benchmark scores rose significantly.
The article discusses a research paper revealing that many science-based LLM benchmarks have incorrect answers, and when corrected, LLM performance scores rise significantly, suggesting current evaluations may underestimate AI capabilities in physics.
Ontology-Grounded, Reasoner-Verified Benchmarks for Evaluating LLM Reasoning in Scientific AI
The paper proposes a pipeline that automatically generates ontology-grounded, reasoner-verified multiple-choice benchmarks for evaluating LLM logical reasoning in scientific AI, validated on three ontologies (Pizza, PMDco, DOID) across six LLMs.
SciR: A Controllable Benchmark for Scientific Reasoning in LLMs
SciR is a new controllable benchmark for evaluating LLMs on scientific reasoning including deduction, induction, and causal abduction, with parametric control over extraction and inference difficulty. Tests show both axes degrade performance across models, with reasoning models like DeepSeek-R1 outperforming instruct models on inference.