Forecasting Scientific Progress with Artificial Intelligence

Hugging Face Daily Papers Papers
scientific-progress forecasting ai-evaluation benchmark temporal-prediction overconfidence

Summary

This paper introduces CUSP, a benchmark for evaluating AI systems' ability to forecast scientific progress, finding that current models show systematic overconfidence and domain-dependent limitations, failing to reliably predict scientific advances.

Artificial intelligence (AI) is increasingly embedded in scientific discovery, yet whether it can anticipate scientific progress remains unclear. To study this question, we introduce a temporally grounded evaluation framework for forecasting scientific progress under controlled knowledge constraints. We present CUSP (Cutoff-conditioned Unseen Scientific Progress), a multi-disciplinary and event-level benchmark that evaluates scientific forecasting in AI systems through feasibility assessment, mechanistic reasoning, generative solution design, and temporal prediction. Across 4,760 scientific events, we observe systematic and domain-dependent limitations in current frontier models. While models can identify plausible research directions from competing candidates, they fail to reliably predict whether scientific advances will be realized and systematically misestimate when they will occur. Performance is highly heterogeneous across domains, with the timing of AI progress more predictable than advances in biology, chemistry, and physics. Performance is largely insensitive to whether events occur before or after the training cutoff, suggesting these limitations cannot be explained solely by knowledge exposure in training data. Under controlled information access, additional pre-cutoff knowledge improves performance but does not close the gap to full-information settings, which becomes more pronounced for high-citation advances. Models also exhibit systematic overconfidence and strong response biases, indicating unreliable uncertainty estimation. Taken together, current AI systems fall short as predictive tools for scientific progress. Access to prior knowledge does not translate into reliable forecasting, and performance benefits more from post-event information than from forward-looking prediction.
Original Article
View Cached Full Text

Cached at: 05/22/26, 06:21 PM

Paper page - Forecasting Scientific Progress with Artificial Intelligence

Source: https://huggingface.co/papers/2605.22681

Abstract

Current AI systems demonstrate limited capability in predicting scientific progress, showing inconsistent performance across domains and systematic overconfidence in forecasts.

Artificial intelligence(AI) is increasingly embedded in scientific discovery, yet whether it can anticipatescientific progressremains unclear. To study this question, we introduce a temporally grounded evaluation framework for forecastingscientific progressunder controlled knowledge constraints. We present CUSP (Cutoff-conditioned UnseenScientific Progress), a multi-disciplinary and event-level benchmark that evaluatesscientific forecastingin AI systems throughfeasibility assessment,mechanistic reasoning,generative solution design, andtemporal prediction. Across 4,760 scientific events, we observe systematic anddomain-dependent limitationsin current frontier models. While models can identify plausible research directions from competing candidates, they fail to reliably predict whether scientific advances will be realized and systematically misestimate when they will occur. Performance is highly heterogeneous across domains, with the timing of AI progress more predictable than advances in biology, chemistry, and physics. Performance is largely insensitive to whether events occur before or after the training cutoff, suggesting these limitations cannot be explained solely by knowledge exposure in training data. Under controlled information access, additionalpre-cutoff knowledgeimproves performance but does not close the gap to full-information settings, which becomes more pronounced for high-citation advances. Models also exhibit systematic overconfidence and strong response biases, indicating unreliableuncertainty estimation. Taken together, current AI systems fall short as predictive tools forscientific progress. Access to prior knowledge does not translate into reliable forecasting, and performance benefits more frompost-event informationthan from forward-looking prediction.

View arXiv pageView PDFProject pageGitHub13Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.22681 in a model README.md to link it from this page.

Datasets citing this paper1

#### SeanWu25/CUSP Viewer• Updatedabout 12 hours ago • 4.76k • 253 • 8

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.22681 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

SciPaths: Forecasting Pathways to Scientific Discovery

arXiv cs.CL

Introduces SciPaths, a benchmark for forecasting the enabling contributions required to realize a target scientific discovery, and evaluates frontier and open-weight language models, finding significant room for improvement in reasoning backward from contributions to enabling building blocks.