Can frontier models forecast scientific progress?

Mostly no, but here is why.

This work looks at 4,760 scientific events across disciplines. Frontier models can identify plausible research directions when given options. They cannot reliably predict whether an advance will land, and they get the timeline wrong.

They suggest that this is a calibration problem, not a knowledge problem. Frontier models are confidently miscalibrated about whether and when scientific advances arrive.

Important grounding for any AI-scientist or research-planning agent that uses model forecasts to pick its agenda.

Paper: https://arxiv.org/abs/2605.22681

Learn to build effective AI agents in our academy: https://academy.dair.ai

Forecasting Scientific Progress with Artificial Intelligence

Source: https://arxiv.org/abs/2605.22681 View PDF

Abstract:Artificial intelligence (AI) is increasingly embedded in scientific discovery, yet whether it can anticipate scientific progress remains unclear. To study this question, we introduce a temporally grounded evaluation framework for forecasting scientific progress under controlled knowledge constraints. We present CUSP (Cutoff-conditioned Unseen Scientific Progress), a multi-disciplinary and event-level benchmark that evaluates scientific forecasting in AI systems through feasibility assessment, mechanistic reasoning, generative solution design, and temporal prediction. Across 4,760 scientific events, we observe systematic and domain-dependent limitations in current frontier models. While models can identify plausible research directions from competing candidates, they fail to reliably predict whether scientific advances will be realized and systematically misestimate when they will occur. Performance is highly heterogeneous across domains, with the timing of AI progress more predictable than advances in biology, chemistry, and physics. Performance is largely insensitive to whether events occur before or after the training cutoff, suggesting these limitations cannot be explained solely by knowledge exposure in training data. Under controlled information access, additional pre-cutoff knowledge improves performance but does not close the gap to full-information settings, which becomes more pronounced for high-citation advances. Models also exhibit systematic overconfidence and strong response biases, indicating unreliable uncertainty estimation. Taken together, current AI systems fall short as predictive tools for scientific progress. Access to prior knowledge does not translate into reliable forecasting, and performance benefits more from post-event information than from forward-looking prediction.

Submission history

From: Sean Wu [view email] **[v1]**Thu, 21 May 2026 16:23:36 UTC (26,940 KB)