Benchmarking Forecasting Agents on Yesterday's Tomorrows

# Benchmarking Forecasting Agents on Yesterday’s Tomorrows
Source: [https://arxiv.org/html/2610.40940](https://arxiv.org/html/2610.40940)
###### Abstract

Digital agents can now gather and organize information, reason over it, and use tools to complete complex tasks\. Forecasting puts all of these abilities to the test: it requires integrating incomplete evidence, inferring causal drivers, and committing to calibrated probabilistic judgments\. Existing benchmarks are poorly suited to measuring and improving this ability\. Live benchmarks provide feedback only after events resolve, so every evaluation cycle waits on the calendar; fixed\-corpus evaluations return results immediately but confine research to pre\-collected documents\. We introduceFABench, a scalable backtesting environment forForecastingAgents\. An automated pipeline discovers events, generates questions, and resolves outcomes for any past forecast date, so the benchmark can be renewed to stay ahead of model training cutoffs; its first release poses 287 questions across nine domains\. Agents research each question freely in an archive environment that combines an offline archive with live web search screened to prevent temporal leakage\. Evaluating agents built on eight recent models, we find that retrieval improves the Brier score of all eight in every retrieval setting and lifts the leading agents to the market consensus on the 54 questions linked to Polymarket, with the best three scoring slightly above it\. Research traces reveal recurring failures in how agents weigh and apply evidence, leaving ample room for improvement\. We release our code, data, and archive environment as a foundation for community research on forecasting agents\.

## 1Introduction

Large language models have advanced at a remarkable pace, and the agents built on them now carry out a growing share of everyday knowledge work: writing and debugging code\([Jimenez et al\., 2024](https://arxiv.org/html/2610.40940#bib.bib14)\), conducting deep research across many sources\([Wei et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib32)\), and operating computers through their interfaces\([Xie et al\., 2024](https://arxiv.org/html/2610.40940#bib.bib36)\)\. On some of these tasks, agents now approach the level of human experts\([Patwardhan et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib26)\)\. A gap remains, however, on complex, long\-horizon tasks\([Kwa et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib19)\)that require an agent to gather large amounts of information, reason about how a situation will unfold, and commit to a decision\. Forecasting is the sharpest instance of this challenge\. To predict a future event, an agent must integrate incomplete and often conflicting evidence, infer the causal drivers behind it, and compress that understanding into a calibrated probability\. It is also a demanding test of decision\-making under uncertainty\([Tetlock & Gardner, 2015](https://arxiv.org/html/2610.40940#bib.bib29);[Mellers et al\., 2015](https://arxiv.org/html/2610.40940#bib.bib21)\): whether an agent can forecast future events as well as human experts, which LLM forecasters have begun to approach\([Halawi et al\., 2024](https://arxiv.org/html/2610.40940#bib.bib12);[Schoenegger et al\., 2024](https://arxiv.org/html/2610.40940#bib.bib27);[Gu et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib10)\)\.

Progress toward this goal is hard to measure, because existing forecasting benchmarks fall into two camps with complementary weaknesses\. Live benchmarks pose questions about events that have not yet occurred, typically drawn from prediction markets or forecasting platforms\([Karger et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib15);[Zeng et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib39);[Yang et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib37)\)\. Since no information about the outcome exists anywhere, they cannot leak outcomes; but feedback arrives only when the event resolves, often weeks or months after the forecast is made\([Paleka et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib25)\)\. Every evaluation cycle is gated by the calendar, the question set is dictated by whatever happens to be pending, and scores from different periods cannot be compared because the questions differ\([Kucinskas et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib17)\)\. Fixed\-corpus benchmarks take the opposite approach: they select events that have already resolved, assemble a corpus of documents published before each question’s forecast date, and ask the agent to forecast from that corpus alone\([Zou et al\., 2022](https://arxiv.org/html/2610.40940#bib.bib41);[Wildman et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib34);[Liptay et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib20)\)\. Feedback is immediate and evaluation is reproducible\. A main problem, however, is that the agent can research only what the benchmark’s creators chose to collect, so evidence outside that corpus stays out of reach however relevant it is\. And because the question set is fixed, such benchmarks are static: they saturate\([Kiela et al\., 2021](https://arxiv.org/html/2610.40940#bib.bib16)\), and their outcomes eventually enter model training data\([White et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib33);[Paleka et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib25)\)\.

We introduceFABench, a scalable backtesting environment for forecasting agents that removes this trade\-off\. Our approach treats forecasting as backtesting: questions are posed at historical time points whose outcomes are now known, so feedback is immediate and verifiable, while the agent searches an archive and the live web, both limited to what was published before that time point, so the research itself remains open\-ended\.FABenchhas two components \([Figure1](https://arxiv.org/html/2610.40940#S2.F1)\): a workflow that generates questions, and an archive environment in which agents research them\.

FABenchcurrently contains 287 forecasting questions across nine domains: finance, economy, technology, health, sports, culture, geopolitics, weather and politics\. Rather than curating them by hand, we build a workflow that discovers candidate events from recurring data releases, news and prediction markets, writes well\-specified questions with unambiguous resolution criteria, and resolves each question against its recorded outcome\. The workflow takes the forecast date as input, so it can be re\-run for new dates and extended to new domains, and the benchmark grows on demand and can be refreshed to stay ahead of model training cutoffs\. All current questions are posed at July 1, 2026, after the pretraining cutoffs of all the models we evaluate \([AppendixE](https://arxiv.org/html/2610.40940#A5)\)\.

We useFABenchto evaluate agents built on eight frontier models\. Retrieval raises the Brier Index of every model, by 3\.4 points on average, and brings the leading agents on par with the market consensus, a strong baseline\([Wolfers & Zitzewitz, 2004](https://arxiv.org/html/2610.40940#bib.bib35);[Alur et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib3)\), which every model trails without retrieval\. Yet on about a quarter of the questions, agents still do worse on average than a constant 0\.5 forecast, with or without retrieval\. Because the environment records every step of the agent’s research alongside time\-stamped evidence, we can trace these failures to their sources\. We identify three error modes: favoring unofficial reports over newer official statements, following a public consensus that proves wrong, and applying correct evidence to the wrong case\.

## 2FABench

Figure 1:Overview ofFABench\. The upper part generates questions, and the lower part is the archive environment in which an agent researches a question and gives a probability, scored against the outcome at the bottom right\. The red line marks the forecast datet0t\_\{0\}: content to its left was published beforet0t\_\{0\}, and content to its right aftert0t\_\{0\}\.### 2\.1Forecasting Task and Evaluation

We are interested in forecasting consequential real\-world events\. To score forecasts without waiting, we pose each question at a past forecast datet0t\_\{0\}about an event that has since resolved\. Formally, a question is a yes\-or\-no statementqqwith a resolution rulerr, which names the source that determines the outcomey∈\{0,1\}y\\in\\\{0,1\\\}\. For example, a question posed on July 1, 2026, asks whether the first estimate of UK GDP growth for 2026 Q2 will exceed the 0\.6% first published for Q1; it resolves from the Office for National Statistics release of August 13\.

Given a question, a forecasting agent returns the probabilityp∈\[0,1\]p\\in\[0,1\]thaty=1y=1\. To forecast as someone att0t\_\{0\}would, the agent may use onlyℐ⁡\(t0\)\\mathcal\{I\}\(t\_\{0\}\), the information published beforet0t\_\{0\}\. The archive environment of[Section3](https://arxiv.org/html/2610.40940#S3)provides this information\. What counts is when information is published, not when events occur: in the example, the quarter has ended byt0t\_\{0\}, but its GDP estimate is not yet public\. Besides retrieval, a model may carry later information from training, which no environment can remove;t0t\_\{0\}must therefore follow the training cutoffs of the evaluated models\.

We score each forecastppagainst the outcomeyywith the Brier score\([Brier, 1950](https://arxiv.org/html/2610.40940#bib.bib4)\),

BS=1N​∑i=1N\(pi−yi\)2,\\operatorname\{BS\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\(p\_\{i\}\-y\_\{i\}\)^\{2\},\(1\)whereNNis the number of questions with a valid forecast\. It measures accuracy and is strictly proper: a forecaster minimizes its expected score only by reporting its actual belief\([Gneiting & Raftery, 2007](https://arxiv.org/html/2610.40940#bib.bib8)\)\. For readability, we report it as the Brier Index\([Kucinskas et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib18)\),

BI=100​\(1−BS\),\\operatorname\{BI\}=100\\,\\bigl\(1\-\\sqrt\{\\operatorname\{BS\}\}\\bigr\),\(2\)on which a constant forecast of 0\.5 scores 50 and a perfect forecaster scores 100\.

A good Brier score, however, does not show whether the probabilities can be taken at face value\. A forecaster that separates Yes from No events well can score well while being overconfident, yet a decision that weighs a probability against costs needs the probability itself to be right\. To measure this calibration, we also report the expected calibration error\([Guo et al\., 2017](https://arxiv.org/html/2610.40940#bib.bib11)\),

ECE=∑b=110nbN​\|p¯b−y¯b\|,\\operatorname\{ECE\}=\\sum\_\{b=1\}^\{10\}\\frac\{n\_\{b\}\}\{N\}\\,\\bigl\|\\bar\{p\}\_\{b\}\-\\bar\{y\}\_\{b\}\\bigr\|,\(3\)where ten equal\-width probability bins group the forecasts, and binbbholdsnbn\_\{b\}of them with meanp¯b\\bar\{p\}\_\{b\}and Yes ratey¯b\\bar\{y\}\_\{b\}; lower is better\.

### 2\.2Question Generation

To evaluate agents on events that matter, and to renew the questions as models advance, we generate questions with an automatic workflow that takest0t\_\{0\}as input \(panel 1 of[Figure1](https://arxiv.org/html/2610.40940#S2.F1)\)\. Renewal is necessary because a set expires once models are trained past itst0t\_\{0\}\. The workflow draws events from two complementary sources and, in each, keeps those that many people follow\.

#### Recurring releases\.

The first source covers releases such as the earnings of S&P 100 companies, the decisions of major central banks and the official statistics of major economies\. They meet both needs: many people follow them, and their published schedules and fixed definitions allow questions to be posed at anyt0t\_\{0\}\. For a givent0t\_\{0\}, an LLM agent selects releases due aftert0t\_\{0\}from calendars published before it\. A template then turns each release into a question, with a threshold computed by a fixed rule from values published beforet0t\_\{0\}, as in the GDP example; the fixed rule keeps the threshold independent of the outcome\.

#### Emerging events\.

Many consequential events, however, happen only once, such as elections, drug approvals and product launches, and they form the second source\. To find them, the agent searches news and event calendars published beforet0t\_\{0\}for pending events\. Prediction markets also show which events people follow, so we draw on Polymarket as well\. We search its markets on consequential topics and turn those open att0t\_\{0\}into questions\. Each such question keeps the market’s wording and resolution rule, as in “Will GPT\-6 be released by September 30, 2026?” The market consensus att0t\_\{0\}, its price just beforet0t\_\{0\}, is then a forecast of the same question, which gives a human baseline \([Section4\.2](https://arxiv.org/html/2610.40940#S4.SS2)\)\.

#### Validation and resolution\.

To ensure validity, a second agent checks that each question the agents write resolves unambiguously, was still open att0t\_\{0\}and reveals nothing published aftert0t\_\{0\}\. Once the resolution date passes, each outcome is recorded from the named source with its evidence and kept out of the agent’s input\. Questions whose outcome cannot be established are dropped\.

#### First release\.

Our first release poses 287 questions att0t\_\{0\}= July 1, 2026, 254 written by agents and 33 taken from Polymarket, in nine domains: finance, economy, politics, technology, health, sports, culture, geopolitics and weather \([Tables3](https://arxiv.org/html/2610.40940#A2.T3)and[4](https://arxiv.org/html/2610.40940#A2.T4)\)\. Building it also involved manual steps, and some questions were written with outcomes visible;[AppendixB](https://arxiv.org/html/2610.40940#A2)lists both\. The workflow is not tied to this date: an earlier run at March 1, 2026, produced 232 agent\-written and 300 Polymarket questions \([AppendixC](https://arxiv.org/html/2610.40940#A3)\)\.

### 2\.3Comparison with Existing Benchmarks

Table 1:Comparison with representative forecasting benchmarks\.✓yes;◐partly;✗no\.BenchmarkCalibrationLiveForecastBench\([Karger et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib15)\)✗✓◐✓◐Prophet Arena\([Yang et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib37)\)✗✓◐✓✓FutureX\([Zeng et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib39)\)✗✓✓✓✗BacktestAutocast\([Zou et al\., 2022](https://arxiv.org/html/2610.40940#bib.bib41)\)✓✓✗◐◐Daily Oracle\([Dai et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib7)\)✓✗✗◐✗BTF\-2\([Liptay et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib20)\)✓✓◐◐✓ATM\([Chai et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib5)\)✓✓✓✗✗FABench✓✓✓✓✓[Table1](https://arxiv.org/html/2610.40940#S2.T1)comparesFABenchwith representative benchmarks;[AppendixB](https://arxiv.org/html/2610.40940#A2)notes where our first release departs from the properties marked there, and[AppendixA](https://arxiv.org/html/2610.40940#A1)reviews further work\.

Live benchmarks, such as ForecastBench, Prophet Arena and FutureX, cannot leak outcomes, but every score waits for events to resolve, and systems evaluated at different times answer different questions\. Other backtests score at once, but they either choose events after the fact, as Daily Oracle does by generating questions from articles that report the outcome, or confine forecasting to a pre\-collected corpus, as Autocast does\.

BTF\-2 is closest to ours in how it selects and scores questions\. Its corpus, however, is crawled for each question while the question is still open, so a set for a new period must be planned and collected before that period, and, as its authors note, models trained later need such a new set\.FABenchinstead builds a set after the fact for any pastt0t\_\{0\}that its news archive covers, because the news archive is collected independently of the questions and web pages are screened againstt0t\_\{0\}\. A new model can therefore be evaluated once events after its training cutoff have resolved, without a collection planned in advance\.

Agentic Time Machine \(ATM\) also lets agents research the live web, but its filter reads each question and its known answer, so what the agent sees can depend on the outcome it must forecast\.FABenchscreens each page by its publication time alone, so the evidence cannot be tilted toward or away from the outcome \([Section3\.2](https://arxiv.org/html/2610.40940#S3.SS2)\)\. ATM also reuses existing questions and scores point predictions, whereasFABenchgenerates questions for a chosent0t\_\{0\}and scores probabilities, which also reveals calibration\.

## 3Archive Environment

To let the agent research a question as a forecaster att0t\_\{0\}could, using only information published beforet0t\_\{0\}\([Section2\.1](https://arxiv.org/html/2610.40940#S2.SS1)\), we build an archive environment \(panel 2 of[Figure1](https://arxiv.org/html/2610.40940#S2.F1)\)\. In this environment, the agent decides what to search for and which documents to read, and the environment decides what its search and read tools return\. These tools should deliver as much relevant information as possible while leaking nothing published aftert0t\_\{0\}\. An offline archive of pages captured beforet0t\_\{0\}meets the second requirement by construction but holds only what was collected\. Live web search reaches far more sources, but its pages must be screened one by one\. We therefore combine the two, so that one search draws on the archive’s dated news and reference text and on pages that only the web can reach\.

For a request ofNNresults, the archive supplies its top⌈N/2⌉\\lceil N/2\\rceildocuments, and the web supplies the pages that pass screening among its top⌊N/2⌋\\lfloor N/2\\rfloorcandidates; withheld pages are not replaced\. The fixed shares keep archive documents, which cannot leak, in every result list, so the agent never depends on screened web pages alone\. A URL found in both keeps only its archive copy, whose capture date needs no review\. Either source can also be searched alone, which[Section4\.3](https://arxiv.org/html/2610.40940#S4.SS3)uses to isolate what each contributes\.

### 3\.1Offline archive

#### Contents\.

To forecast well from an archive, a forecaster needs two kinds of information: background on the entities and topics of a question, and the latest developments beforet0t\_\{0\}\. For background, we use Wikipedia\. Because it stores every revision with its time, we take the last revision beforet0t\_\{0\}of the articles on the entities and topics in the questions \([AppendixD](https://arxiv.org/html/2610.40940#A4)\)\. For recent developments, we use CC\-News, Common Crawl’s continuous crawl of news sites\([Nagel, 2016](https://arxiv.org/html/2610.40940#bib.bib24)\), and keep the English pages captured beforet0t\_\{0\}\. A page captured beforet0t\_\{0\}was public by then, so the archive needs no page\-by\-page review and cannot leak through a reviewer’s mistakes\. Fort0t\_\{0\}= July 1, 2026, it holds about 20\.1 million news captures and 684 Wikipedia revisions\.

#### Retrieval\.

When the agent searches the archive, BM25 retrieves candidates from both parts, and a rule\-based scorer removes duplicates and reranks them\. All term statistics come from documents beforet0t\_\{0\}, so no signal from aftert0t\_\{0\}shapes the ranking\. BTF\-2 also gives agents an offline corpus, but its RetroSearch orders results by live Google search before dropping pages outside the corpus\([Liptay et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib20);[Wildman et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib34)\)\. Pages that became prominent because of later events can then rise to the top, a possible leak that its developers acknowledge, and the same query can return different results in different runs\. Our ranking uses the archive alone, so the same query returns the same results in every run\. A search result shows the beginning of each document, and the read tool returns its full text \([AppendixD](https://arxiv.org/html/2610.40940#A4)\)\.

### 3\.2Screened web search

The archive, however, misses sites and pages that CC\-News did not capture\. To reach them, the search tool also asks a live search engine for results dated beforet0t\_\{0\}\.

#### Search and cleaning\.

The search engine’s date limit only narrows the candidates, because an old URL may now show updated content\. We therefore fetch each web page before the agent sees it and judge it by the text it shows now\. To keep market prices out of the agent’s view, we reject pages on known prediction\-market sites\. From the other pages, we strip navigation and advertisements, and we replace Wikipedia pages with their last revision beforet0t\_\{0\}, which needs no review\.

#### Temporal review\.

Every other page goes to an LLM reviewer that sees neither the question nor its answer, so that its decision depends on timing alone \([AppendixD](https://arxiv.org/html/2610.40940#A4)\)\. The reviewer first removes leftover blocks, such as recommendation lists and live widgets, whose newer headlines would otherwise count against an old article\. It then judges whether the remaining text was public beforet0t\_\{0\}\. What counts is publication, not when events occur, so a forecast or plan published beforet0t\_\{0\}passes\. When timing is unclear, the reviewer can search the web to check, and what it finds never reaches the agent\. The filter of Agentic Time Machine, by contrast, sees the answer, so it can withhold a legitimate forecast for matching the outcome, as it did with an award\-nomination forecast published two days before the announcement\([Chai et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib5)\)\. Our reviewer passes such forecasts, which a forecaster att0t\_\{0\}would have weighed\.

#### Delivery\.

Based on this judgment, a page is withheld entirely if it contains information from aftert0t\_\{0\}or its timing stays unclear\. Agentic Time Machine passes pages of unclear timing, and in one of its cases an undated post quoting a price from another day misled the agent\. The cost of our choice is that some legitimate pages of unclear timing are lost\. Withholding the whole page, rather than cutting out the later passages, avoids passing text whose remaining context could still reveal later events\. For a page that passes, the read tool returns the cleaned text, and a search result shows an excerpt of it\. The search engine’s own titles, snippets and dates never reach the agent, because they are generated today and can reflect later events\. Every page is logged with its decision, so that decisions can be audited;[AppendixD](https://arxiv.org/html/2610.40940#A4)gives the reviewer’s instructions\. In theonline\-filteredruns, the reviewer withheld 16\.0% of 302,561 pages\. Reviewing 3,087 pages at both our low reasoning effort and high effort gave the same decision for 95\.7% of them; only 14 \(0\.5%\) passed at low effort yet were judged at high effort to disclose later information\.

## 4Experiments

### 4\.1Experimental setup

#### Environment and data\.

We evaluate every model in four information settings:none, without tools;archiveandonline\-filtered, with only the archive or only screened web search; andfull, our standard setting, with both \([Section3](https://arxiv.org/html/2610.40940#S3)\)\. All 287 questions come from the release of[Section2\.2](https://arxiv.org/html/2610.40940#S2.SS2), posed att0t\_\{0\}= July 1, 2026, and every model is asked all of them in every setting\.111A few runs lack forecasts for some questions, mostly political questions that GPT\-6 Astra and GPT\-6 Sol declined; each run is scored on the questions it answered \([Table9](https://arxiv.org/html/2610.40940#A5.T9)\)\.

#### Compared models\.

We compare\(1\)three proprietary models: GPT\-6 Astra, GPT\-6 Sol and Claude Opus 5\.5;\(2\)five open\-weight models: DeepSeek V4\.1 Flash \(DS V4\.1 Flash\), Kimi K3, GLM 5\.3, Qwen3\.8\-27B and Qwen3\.6\-35B\-A3B; and\(3\)two models trained to return calibrated probabilities, JEV 1\.13\.0\([Almeida, 2026](https://arxiv.org/html/2610.40940#bib.bib2)\)and StartLux\-Decision 27B AutoThink \(StartLux 27B;[Zhuang et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib40)\), which answer without retrieval\. As references, we use the constant forecast of 0\.5 and the market consensus on Polymarket \([Section4\.2](https://arxiv.org/html/2610.40940#S4.SS2)\)\. Most models were released aftert0t\_\{0\}, but without retrieval the best of them scores 55\.0 BI, only 5 above the constant forecast, so none appears to know the outcomes from training \([Table8](https://arxiv.org/html/2610.40940#A5.T8)\)\.

#### Implementation and metrics\.

Every model receives the same prompt, searches the same sources and makes at most 100 tool calls per question, and forecasts each question once per setting; ChatGPT models run in the Codex CLI and Claude in Claude Code with their built\-in search replaced by our tools, and the other models in our agent loop \([AppendicesD](https://arxiv.org/html/2610.40940#A4)and[E](https://arxiv.org/html/2610.40940#A5)\)\. We measure accuracy with the BI and calibration with the ECE \([Section2\.1](https://arxiv.org/html/2610.40940#S2.SS1)\)\.

### 4\.2Comparison among models and with prediction markets

![Refer to caption](https://arxiv.org/html/2610.40940v1/leaderboard.png)

![Refer to caption](https://arxiv.org/html/2610.40940v1/calibration.png)

Figure 2:BI underfull\(left\) and ECE undernoneandfull\(right\) of each model\. Lower ECE is better; JEV 1\.13\.0 and StartLux 27B do not retrieve, so they appear with theirnonescores\.#### Comparison among models\.

To compare the models in the standard setting,[Figure2](https://arxiv.org/html/2610.40940#S4.F2)shows their BI and ECE underfull\. Five models, three proprietary and two open\-weight, form a leading group at 57 to 58 BI, while the weakest model and the two models that do not retrieve score near the constant forecast\. Even the best model removes only about 31% of the squared error of the constant forecast\.

The ranking depends less on what a model knows than on how much it researches\. Accuracy without tools is unrelated to the gain from retrieval \(Spearmanρ=0\.24\\rho=0\.24across the eight models\), whereas the number of pages a model reads per question tracks it closely \(ρ=0\.83\\rho=0\.83;[Tables2](https://arxiv.org/html/2610.40940#S4.T2)and[9](https://arxiv.org/html/2610.40940#A5.T9)\)\. Kimi K3 and DeepSeek V4\.1 Flash read the most pages and gain the most, which lifts them into the leading group, whereas Claude Opus 5\.5, the most accurate model without tools, reads less than one page per question and gains only 2\.4 BI\.

Calibration does not follow accuracy: Claude Opus 5\.5 scores within 1 BI of GPT\-6 Astra, yet its ECE is 0\.110 against 0\.058\. Retrieval lowers ECE for 7 of the 8 models, but not for Qwen3\.6\-35B, whose ECE rises from 0\.125 to 0\.139 although its BI improves\.

#### Comparison with prediction markets\.

To compare the models with human forecasters, we use the 54 questions linked to Polymarket: 33 taken from Polymarket and 21 generated questions that match a contract\. The market consensus, the market’s price about one minute beforet0t\_\{0\}, is a forecast made at the same cutoff as the models’ and a strong baseline: prior work has found it hard to beat\([Karger et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib15);[Alur et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib3);[Mostafa et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib22)\), turning forecasts into trading gains is harder still\([Wang et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib31)\), and roughly 69% of Polymarket users lose money\([Akey et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib1)\), which suggests that forecasts at the market’s level may already outperform most participants\.

compares each model with the market in the four settings\. Underfull, the leading agents perform at about the market’s level: six models lie between 1\.8 BI below and 2\.7 BI above the market\. Without retrieval, every model falls below the market, by 1\.8 to 10\.8 BI\. The market price aggregates what traders learned beforet0t\_\{0\}, and retrieval lets an agent gather the same public information itself, which plausibly explains why the gap closes only with retrieval\. Prior work has instead found LLM forecasters trailing liquid markets\([Alur et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib3)\)or beating a market only shortly after it opens\([Mostafa et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib22)\)\.

### 4\.3Comparison of retrieval settings

Table 2:BI of each model in the four information settings, each on the questions the model answered in that setting, with the difference fromnonein parentheses\. Bold marks the highest and underlining the second highest score and gain in each column\.To test whether each source adds useful information, and whether combining them adds more,[Table2](https://arxiv.org/html/2610.40940#S4.T2)compares the four settings for each model\. Retrieval improves every model in every setting\. The archive alone, which cannot leak by construction, already yields about four fifths of the gain offull, so most of the improvement reflects information beyond what the models learned in training rather than leaked outcomes\.

The three retrieval settings, however, lie close together\.online\-filteredscores slightly higher thanarchivefor all eight models, but for no model doesfulldiffer from either single source with a 95% interval that excludes zero \(paired bootstrap over questions;[AppendixF](https://arxiv.org/html/2610.40940#A6)\)\. This pattern suggests that screened web search adds little information from aftert0t\_\{0\}beyond the archive, which predatest0t\_\{0\}by construction\. It also suggests that, on average, neither source lacks much of what the other provides, although the archive falls short in three domains \([Section4\.4](https://arxiv.org/html/2610.40940#S4.SS4)\)\.

### 4\.4Performance across domains

Forecasting performance differs widely across domains: underfull, the BI averaged over models ranges from 49\.1 on Culture and 49\.8 on Weather to 63\.2 on Health\. Retrieval, too, helps unevenly, andbreaks its gains down by domain underarchive, which searches the archive alone, and underfull, which adds screened web search\. Overall, retrieval raises the BI in most model–domain pairs: 48 of 72 underarchiveand 60 of 72 underfull\. Adding web search matters most in Technology, Health and Culture, where the gain averaged over models rises from about 1\.7 to 3\.5 BI underarchiveto about 4\.0 to 6\.3 BI underfull\. Finance and Economy gain about 2\.7 to 4\.5 BI in either setting\.

One explanation is that questions in these three domains often ask whether an announced event, such as a model release, a drug approval or a film opening, happens as planned\. The decisive sources are then company, regulator and industry pages, which web search reaches but a news archive may not contain \([Section3\.2](https://arxiv.org/html/2610.40940#S3.SS2)\)\. News, by contrast, closely covers the scheduled releases behind finance and economy questions\.

Retrieval can also lower accuracy\. In both settings, every model loses BI in at least one domain, by up to 5\.6 BI underarchive\(Qwen3\.6\-35B on Geopolitics\) and 4\.8 BI underfull\(Kimi K3 on Geopolitics\)\. Geopolitics gains little or loses on average, plausibly because its coverage beforet0t\_\{0\}consists largely of commentary and competing predictions about open developments\. Forecasting performance should therefore be reported together with the domains and the setting in which it was measured\.

### 4\.5Failure analysis

To find failures that aggregate scores hide, we classify each question by whether its Brier score, averaged over models, exceeds 0\.25, the score of the constant forecast, without retrieval and withfull\. Retrieval corrects 41 questions and misleads 11, while 76, about a quarter of the benchmark, stay above 0\.25 in both settings; on 28 of them, no model beats the constant forecast\. Traces of representative questions show why \([AppendixG](https://arxiv.org/html/2610.40940#A7)\)\.

#### Retrieval helps when decisive evidence is already public\.

In the traced corrections, the decisive evidence, such as a central bank’s split vote and guidance, was published beforet0t\_\{0\}and replaced the trend that models extrapolate without tools\.

#### Retrieval misleads when informal signals meet official guidance\.

Several models let a concrete but unofficial report, such as a launch window relayed on social media, outweigh a newer official statement, which suggests that they weigh how specific a signal is more than who issued it and when\. Adding the archive reduces this error in the traced case; most misled questions, 7 of 11, concern finance\.

#### Shared errors come from the public record itself\.

When the consensus att0t\_\{0\}is wrong, as with box\-office forecasts that all overestimated a film’s opening, retrieval returns that consensus and cannot correct it; models treat the few dissenting forecasts as tail risk\.

#### Evidence is applied to the wrong case\.

In the traced cases, models match evidence to a question by topic rather than by the exact event, so an authoritative fact about one case, such as the production timeline of one coin, is applied to another\.

## 5Conclusion

We introducedFABench, a scalable backtesting environment for forecasting agents that combines automated question generation with open\-ended research under historical information constraints\. By pairing resolved events with an offline archive and temporally screened web search, it provides immediate, verifiable feedback without restricting agents to a pre\-collected corpus\. Our evaluation of latest models on 287 questions across nine domains shows that retrieval generally improves forecasting accuracy and calibration, but its benefits vary across models and domains\. Several agents score on par with the market consensus on the questions linked to Polymarket contracts, while our case studies reveal failures in how retrieved evidence is weighed and applied\. These findings motivate improving not only information access, but also the reasoning that turns evidence into probabilistic judgments\. With renewable questions, open\-ended research, and immediate outcome feedback,FABenchprovides a foundation for faster evaluation and future outcome\-based training of forecasting agents\.

## AI Use Statement

We used generative AI tools to assist with drafting and editing the manuscript and creating figures\. LLMs were also used for event discovery, benchmark question construction, selection, and validation, as well as cleaning and temporal screening of retrieved web content\. The authors reviewed and revised the AI\-assisted text and figures for accuracy and consistency with the methods and results\. The authors take responsibility for the final content of this work, including all AI\-assisted text, claims, and research artifacts\.

## Ethics Statement

This work develops a benchmark for evaluating forecasting agents using questions and evidence drawn from publicly available sources\. The selection of events and reliance on English\-language news may introduce coverage biases\. Forecasts can be inaccurate or miscalibrated, and performance on this benchmark should not be interpreted as evidence that a system is suitable for high\-stakes deployment\. To support research integrity, we document temporal screening procedures and trace audits, including their limitations \([AppendixH](https://arxiv.org/html/2610.40940#A8)\)\. Responsible use and redistribution of the underlying sources require attention to attribution, licensing, and privacy\.

## Reproducibility Statement

We will release every question set ofFABenchon Hugging Face, starting with the first release and the set posed at March 1, 2026, together with their resolution keys and the materials needed to rebuild the retrieval archive: the manifests of the CC\-News captures and the identifiers of the Wikipedia revisions\. The code is released with the data: a few commands build the archive environment and evaluate an agent in any of the four information settings, with the prompts, tool definitions and reviewer instructions of[AppendicesD](https://arxiv.org/html/2610.40940#A4)and[E](https://arxiv.org/html/2610.40940#A5)\. New question sets built with the same pipeline will feed a live leaderboard, so that agents can be compared on questions that postdate their training\. The full research traces of the reported runs, over 100 GB, will be shared with teams that request them\.

## References

- Akey et al\. \(2026\)Pat Akey, Vincent Grégoire, Nicolas Harvie, and Charles Martineau\.Who Wins and Who Loses in Prediction Markets? Evidence from Polymarket\.CEPR Discussion Paper DP21615, Centre for Economic Policy Research, 2026\.URL[https://cepr\.org/publications/dp21615](https://cepr.org/publications/dp21615)\.
- Almeida \(2026\)Diogo Almeida\.Introducing system one models & Jev\.TypeSafe AI blog, 2026\.URL[https://typesafe\.ai/blog/introducing\-system\-one\-models\-and\-jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev)\.
- Alur et al\. \(2025\)Rohan Alur, Bradly C\. Stadie, Daniel Kang, Ryan Chen, Matt McManus, Michael Rickert, Tyler Lee, Michael Federici, Richard Zhu, Dennis Fogerty, Hayley Williamson, Nina Lozinski, Aaron Linsky, and Jasjeet S\. Sekhon\.AIA Forecaster: Technical Report, 2025\.URL[https://arxiv\.org/abs/2511\.07678](https://arxiv.org/abs/2511.07678)\.arXiv:2511\.07678\.
- Brier \(1950\)Glenn W\. Brier\.Verification of forecasts expressed in terms of probability\.*Monthly Weather Review*, 78\(1\):1–3, 1950\.
- Chai et al\. \(2026\)Jingyi Chai, Bingyang Zheng, Xiangrui Liu, Hao Lu, Zihang Zhou, Tianchen Wang, Kemeng Zhang, and Siheng Chen\.Agentic Time Machine as an Infrastructure for Future\-Event Forecasting, 2026\.URL[https://arxiv\.org/abs/2606\.21013](https://arxiv.org/abs/2606.21013)\.arXiv:2606\.21013\.
- Chandak et al\. \(2025\)Nikhil Chandak, Shashwat Goel, Ameya Prabhu, Moritz Hardt, and Jonas Geiping\.Scaling Open\-Ended Reasoning to Predict the Future, 2025\.URL[https://arxiv\.org/abs/2512\.25070](https://arxiv.org/abs/2512.25070)\.arXiv:2512\.25070\.
- Dai et al\. \(2025\)Hui Dai, Ryan Teehan, and Mengye Ren\.Are LLMs prescient? A continuous evaluation using daily news as the oracle\.In*Proceedings of the 42nd International Conference on Machine Learning*, volume 267 of*Proceedings of Machine Learning Research*, pp\. 12179–12213, 2025\.URL[https://proceedings\.mlr\.press/v267/dai25l\.html](https://proceedings.mlr.press/v267/dai25l.html)\.
- Gneiting & Raftery \(2007\)Tilmann Gneiting and Adrian E\. Raftery\.Strictly proper scoring rules, prediction, and estimation\.*Journal of the American Statistical Association*, 102\(477\):359–378, 2007\.
- Goel et al\. \(2026\)Shashwat Goel, Nikhil Chandak, Arvindh Arun, Ameya Prabhu, Steffen Staab, Moritz Hardt, Maksym Andriushchenko, and Jonas Geiping\.FutureSim: Replaying World Events to Evaluate Adaptive Agents, 2026\.URL[https://arxiv\.org/abs/2605\.15188](https://arxiv.org/abs/2605.15188)\.arXiv:2605\.15188\.
- Gu et al\. \(2026\)Anri Gu, Nicole Kagan, Jibang Wu, Alec Sun, and Haifeng Xu\.When do prophets profit in prediction markets?, 2026\.URL[https://arxiv\.org/abs/2607\.06166](https://arxiv.org/abs/2607.06166)\.arXiv:2607\.06166\.
- Guo et al\. \(2017\)Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q\. Weinberger\.On calibration of modern neural networks\.In*Proceedings of the 34th International Conference on Machine Learning*, volume 70 of*Proceedings of Machine Learning Research*, pp\. 1321–1330, 2017\.
- Halawi et al\. \(2024\)Danny Halawi, Fred Zhang, Chen Yueh\-Han, and Jacob Steinhardt\.Approaching human\-level forecasting with language models\.In*Advances in Neural Information Processing Systems*, volume 37, pp\. 50426–50468, 2024\.doi:10\.52202/079017\-1598\.
- Han et al\. \(2026\)Zhixin Han, Yanzhi Zhang, Chuyang Wei, Maohang Gao, Xiawei Yue, Kefei Chen, Yu Zhuang, Haoxiang Guan, Jiyan He, Jian Li, Yitong Duan, Yu Shi, Mengting Hu, and Shuxin Zheng\.FutureWorld: A Live Reinforcement Learning Environment for Predictive Agents with Real\-World Outcome Rewards, 2026\.URL[https://arxiv\.org/abs/2604\.26733](https://arxiv.org/abs/2604.26733)\.arXiv:2604\.26733\.
- Jimenez et al\. \(2024\)Carlos E\. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan\.SWE\-bench: Can language models resolve real\-world GitHub issues?In*International Conference on Learning Representations*, pp\. 54107–54157, 2024\.URL[https://proceedings\.iclr\.cc/paper\_files/paper/2024/hash/edac78c3e300629acfe6cbe9ca88fb84\-Abstract\-Conference\.html](https://proceedings.iclr.cc/paper_files/paper/2024/hash/edac78c3e300629acfe6cbe9ca88fb84-Abstract-Conference.html)\.
- Karger et al\. \(2025\)Ezra Karger, Houtan Bastani, Chen Yueh\-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip E\. Tetlock\.ForecastBench: A dynamic benchmark of AI forecasting capabilities\.In*International Conference on Learning Representations*, pp\. 93943–93980, 2025\.URL[https://proceedings\.iclr\.cc/paper\_files/paper/2025/hash/ea74e45a229dac70b5b63b28d8934db6\-Abstract\-Conference\.html](https://proceedings.iclr.cc/paper_files/paper/2025/hash/ea74e45a229dac70b5b63b28d8934db6-Abstract-Conference.html)\.
- Kiela et al\. \(2021\)Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams\.Dynabench: Rethinking benchmarking in NLP\.In*Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pp\. 4110–4124, Online, June 2021\. Association for Computational Linguistics\.doi:10\.18653/v1/2021\.naacl\-main\.324\.
- Kucinskas et al\. \(2025\)Simas Kucinskas, Houtan Bastani, and Ezra Karger\.ForecastBench: Updated ranking methodology\.Technical report, ForecastBench, 2025\.URL[https://www\.forecastbench\.org/assets/pdfs/forecastbench\_updated\_methodology\.pdf](https://www.forecastbench.org/assets/pdfs/forecastbench_updated_methodology.pdf)\.Released 3 October 2025\.
- Kucinskas et al\. \(2026\)Simas Kucinskas, Houtan Bastani, and Matt Reynolds\.Making forecasting scores easier to interpret: Introducing the Brier Index\.Substack post, 2026\.URL[https://forecastingresearch\.substack\.com/p/introducing\-the\-brier\-index](https://forecastingresearch.substack.com/p/introducing-the-brier-index)\.
- Kwa et al\. \(2025\)Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M\. Ziegler, Elizabeth Barnes, and Lawrence Chan\.Measuring AI ability to complete long software tasks\.In*Advances in Neural Information Processing Systems*, volume 38, pp\. 92213–92266, 2025\.doi:10\.52202/085713\-3086\.
- Liptay et al\. \(2026\)Tom Liptay, Dan Schwarz, Rafael Poyiadzi, Jack Wildman, and Nikos I\. Bosse\.Evaluating Strategic Reasoning in Forecasting Agents, 2026\.URL[https://arxiv\.org/abs/2604\.26106](https://arxiv.org/abs/2604.26106)\.arXiv:2604\.26106\.
- Mellers et al\. \(2015\)Barbara Mellers, Eric Stone, Terry Murray, Angela Minster, Nick Rohrbaugh, Michael Bishop, Eva Chen, Joshua Baker, Yuan Hou, Michael Horowitz, Lyle Ungar, and Philip Tetlock\.Identifying and cultivating superforecasters as a method of improving probabilistic predictions\.*Perspectives on Psychological Science*, 10\(3\):267–281, 2015\.doi:10\.1177/1745691615577794\.
- Mostafa et al\. \(2026\)Hamza Mostafa, Om Shastri, and Dennis Lee\.TimeSeek: Temporal Reliability of Agentic Forecasters, 2026\.URL[https://arxiv\.org/abs/2604\.04220](https://arxiv.org/abs/2604.04220)\.arXiv:2604\.04220\.
- Murphy \(2026\)Kevin Murphy\.Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs, 2026\.URL[https://arxiv\.org/abs/2604\.18576](https://arxiv.org/abs/2604.18576)\.arXiv:2604\.18576\.
- Nagel \(2016\)Sebastian Nagel\.News dataset available\.Common Crawl blog, 2016\.URL[https://commoncrawl\.org/blog/news\-dataset\-available](https://commoncrawl.org/blog/news-dataset-available)\.
- Paleka et al\. \(2026\)Daniel Paleka, Shashwat Goel, Jonas Geiping, and Florian Tramèr\.Pitfalls in evaluating language model forecasters\.In*International Conference on Learning Representations*, pp\. 114207–114227, 2026\.URL[https://proceedings\.iclr\.cc/paper\_files/paper/2026/hash/b9d69c72e4587ddf3fa6352890aefc4f\-Abstract\-Conference\.html](https://proceedings.iclr.cc/paper_files/paper/2026/hash/b9d69c72e4587ddf3fa6352890aefc4f-Abstract-Conference.html)\.
- Patwardhan et al\. \(2026\)Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Simón Posada Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, Natalie S\. Kim, Patrick Chao, Samuel Miserendino, Gildas Chabot, David Li, Michael Sharman, Alexandra Barr, Amelia Glaese, and Jerry Tworek\.GDPval: Evaluating AI model performance on real\-world economically valuable tasks\.In*International Conference on Learning Representations*, pp\. 24005–24040, 2026\.URL[https://proceedings\.iclr\.cc/paper\_files/paper/2026/hash/290c2430f91912204f30bbcc990fff1d\-Abstract\-Conference\.html](https://proceedings.iclr.cc/paper_files/paper/2026/hash/290c2430f91912204f30bbcc990fff1d-Abstract-Conference.html)\.
- Schoenegger et al\. \(2024\)Philipp Schoenegger, Indre Tuminauskaite, Peter S\. Park, Rafael Valdece Sousa Bastos, and Philip E\. Tetlock\.Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy\.*Science Advances*, 10\(45\):eadp1528, 2024\.doi:10\.1126/sciadv\.adp1528\.
- Su et al\. \(2026\)Shiqian Su, Sen Xing, Xuan Dong, Muyan Zhong, Bin Wang, Xizhou Zhu, Yuntao Chen, Wenhai Wang, Yue Deng, Pengxiang Zhu, Ziyuan Liu, Tiantong Li, Jiaheng Yu, Zhe Chen, Lidong Bing, and Jifeng Dai\.MiroFlow: Towards High\-Performance and Robust Open\-Source Agent Framework for General Deep Research Tasks, 2026\.URL[https://arxiv\.org/abs/2602\.22808](https://arxiv.org/abs/2602.22808)\.arXiv:2602\.22808\.
- Tetlock & Gardner \(2015\)Philip E\. Tetlock and Dan Gardner\.*Superforecasting: The Art and Science of Prediction*\.Crown Publishers, New York, 2015\.ISBN 978\-0\-8041\-3669\-3\.
- Turtel et al\. \(2025\)Benjamin Turtel, Danny Franklin, Kris Skotheim, Luke Hewitt, and Philipp Schoenegger\.Outcome\-based reinforcement learning to predict the future\.*Transactions on Machine Learning Research*, 2025\.ISSN 2835\-8856\.URL[https://openreview\.net/forum?id=bbhdeL8EUX](https://openreview.net/forum?id=bbhdeL8EUX)\.
- Wang et al\. \(2026\)Yishu Wang, Yuxuan Wang, Jiaqi Deng, and Hanyang Tang\.Beyond Forecasting: The Belief\-to\-Trade Layer in Prediction\-Market Agents, 2026\.URL[https://arxiv\.org/abs/2607\.03015](https://arxiv.org/abs/2607.03015)\.arXiv:2607\.03015\.
- Wei et al\. \(2025\)Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese\.BrowseComp: A simple yet challenging benchmark for browsing agents, 2025\.URL[https://arxiv\.org/abs/2504\.12516](https://arxiv.org/abs/2504.12516)\.arXiv:2504\.12516\.
- White et al\. \(2025\)Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz\-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh\-Agrawal, Sandeep Singh Sandha, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum\.LiveBench: A challenging, contamination\-limited LLM benchmark\.In*International Conference on Learning Representations*, pp\. 91595–91631, 2025\.URL[https://proceedings\.iclr\.cc/paper\_files/paper/2025/hash/e4a46394ba5378b3f9a186a5b4c650d1\-Abstract\-Conference\.html](https://proceedings.iclr.cc/paper_files/paper/2025/hash/e4a46394ba5378b3f9a186a5b4c650d1-Abstract-Conference.html)\.
- Wildman et al\. \(2025\)Jack Wildman, Nikos I\. Bosse, Daniel Hnyk, Peter Mühlbacher, Finn Hambly, Jon Evans, Dan Schwarz, and Lawrence Phillips\.Bench to the Future: A Pastcasting Benchmark for Forecasting Agents, 2025\.URL[https://arxiv\.org/abs/2506\.21558](https://arxiv.org/abs/2506.21558)\.arXiv:2506\.21558\.
- Wolfers & Zitzewitz \(2004\)Justin Wolfers and Eric Zitzewitz\.Prediction markets\.*Journal of Economic Perspectives*, 18\(2\):107–126, 2004\.doi:10\.1257/0895330041371321\.
- Xie et al\. \(2024\)Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu\.OSWorld: Benchmarking multimodal agents for open\-ended tasks in real computer environments\.In*Advances in Neural Information Processing Systems \(Datasets and Benchmarks Track\)*, volume 37, pp\. 52040–52094, 2024\.doi:10\.52202/079017\-1650\.
- Yang et al\. \(2026\)Qingchuan Yang, Simon Mahns, Sida Li, Anri Gu, Jibang Wu, and Haifeng Xu\.LLM\-as\-a\-prophet: Understanding predictive intelligence with Prophet Arena\.In*International Conference on Learning Representations*, pp\. 107020–107072, 2026\.URL[https://proceedings\.iclr\.cc/paper\_files/paper/2026/hash/aec5e2847c5ae90f939ab786774856cc\-Abstract\-Conference\.html](https://proceedings.iclr.cc/paper_files/paper/2026/hash/aec5e2847c5ae90f939ab786774856cc-Abstract-Conference.html)\.
- Ye et al\. \(2026\)Xiao Ye, Jacob Dineen, Evan Zhu, Shijie Lu, Kevin Song, and Ben Zhou\.Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters, 2026\.URL[https://arxiv\.org/abs/2607\.14051](https://arxiv.org/abs/2607.14051)\.arXiv:2607\.14051\.
- Zeng et al\. \(2026\)Zhiyuan Zeng, Jiashuo Liu, Siyuan Chen, Tianci He, Yali Liao, Yixiao Tian, Jinpeng Wang, Zaiyuan Wang, Yang Yang, Lingyue Yin, Mingren Yin, Zhenwei Zhu, Tianle Cai, Zehui Chen, Jiecao Chen, Yantao Du, Xiang Gao, Jiacheng Guo, Liang Hu, Jianpeng Jiao, Xiangsheng Li, Jingkai Liu, Shuang Ni, Zhoufutu Wen, Ge Zhang, Kaiyuan Zhang, Xin Zhou, Wenhao Huang, Jose Blanchet, Xipeng Qiu, and Mengdi Wang\.FutureX: An advanced live benchmark for LLM agents in future prediction\.In*International Conference on Learning Representations*, pp\. 19522–19563, 2026\.URL[https://proceedings\.iclr\.cc/paper\_files/paper/2026/hash/21315abf210cef71317e887e9cda78b3\-Abstract\-Conference\.html](https://proceedings.iclr.cc/paper_files/paper/2026/hash/21315abf210cef71317e887e9cda78b3-Abstract-Conference.html)\.
- Zhuang et al\. \(2026\)Xialie Zhuang, Zunhai Su, Jia Pan, Ngai Wong, Guanhua Chen, and Chuan\-Wei Kuo\.StartLux\-Decision: AutoThink Decision Models with Reinforcement Learning for Calibrated Deliberative Decisions, 2026\.URL[https://arxiv\.org/abs/2610\.17309](https://arxiv.org/abs/2610.17309)\.arXiv:2610\.17309\.
- Zou et al\. \(2022\)Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks\.Forecasting future world events with neural networks\.In*Advances in Neural Information Processing Systems \(Datasets and Benchmarks Track\)*, volume 35, pp\. 27293–27305, 2022\.doi:10\.52202/068431\-1979\.

## Appendix ARelated Work

#### Forecasting benchmarks\.

Live benchmarks such as ForecastBench, FutureX and Prophet Arena\([Karger et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib15);[Zeng et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib39);[Yang et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib37)\)and the two backtests closest to ours, BTF\-2 and Agentic Time Machine\([Liptay et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib20);[Chai et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib5)\), are compared withFABenchin[Section2\.3](https://arxiv.org/html/2610.40940#S2.SS3)and[Table1](https://arxiv.org/html/2610.40940#S2.T1)\. Other backtests differ from ours mainly in where their questions come from, which decides the events they can ask about\. Backtests that take questions from forecasting platforms and prediction markets can ask only about events those platforms listed, and they inherit the platforms’ mix of topics\([Zou et al\., 2022](https://arxiv.org/html/2610.40940#bib.bib41);[Ye et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib38)\)\. Hindcast, moreover, keeps only the Polymarket markets whose Reddit posts before the close strongly support the side that later won, so its set favors questions that its archive already answers\([Ye et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib38)\)\. Sports and trading, which Reddit discusses far more than other topics, make up about half of that set\. Other backtests generate questions from news articles, which scales easily\([Dai et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib7);[Goel et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib9);[Chandak et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib6)\)\. Because each question starts from an article that reports an outcome, however, these sets contain events chosen after the fact, and their wording must be screened for leaks: before leakage editing, over 40% of the questions OpenForesight selected contained the answer string\([Chandak et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib6)\)\.FABenchinstead finds events pending att0t\_\{0\}in calendars and news published beforet0t\_\{0\}and in markets open att0t\_\{0\}, and keeps those that many people follow\. Its workflow thus chooses events by public attention att0t\_\{0\}rather than by archive coverage or by the outcome \([Section2\.2](https://arxiv.org/html/2610.40940#S2.SS2);[AppendixB](https://arxiv.org/html/2610.40940#A2)lists the exceptions in the first release\)\.

#### Forecasting agents\.

LLM forecasters have moved from fixed retrieval pipelines to agents that research on their own\. Halawi et al\. retrieve news articles and prompt a model to reason over them, approaching the accuracy of human forecaster crowds\([Halawi et al\., 2024](https://arxiv.org/html/2610.40940#bib.bib12)\)\. General deep\-research agents, which plan searches over many turns and read many sources, have since been applied to forecasting: FutureX evaluates such agents live, and MiroFlow, a general deep\-research framework, led its leaderboard in late 2025\([Zeng et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib39);[Su et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib28)\)\. Systems built for forecasting, such as AIA Forecaster and the Bayesian Linguistic Forecaster, are backtested on ForecastBench questions under their own leakage controls\([Alur et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib3);[Murphy, 2026](https://arxiv.org/html/2610.40940#bib.bib23)\)\. Their results are hard to compare, because the systems differ in questions, search tools and leakage controls, and each of these changes the score\. AIA Forecaster, for example, finds that non\-agentic search with a weak provider adds almost nothing while agentic search helps markedly, and that giving a model without search the market price closes about 42% of its gap to agentic search\([Alur et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib3)\)\. Training work meets the same split between fixed retrieval and open search: outcome\-based reinforcement learning and OpenForesight reward each forecast at once but feed the model retrieved news, whereas FutureWorld lets agents search live and must wait for each outcome\([Turtel et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib30);[Chandak et al\., 2025](https://arxiv.org/html/2610.40940#bib.bib6);[Han et al\., 2026](https://arxiv.org/html/2610.40940#bib.bib13)\)\.FABenchgives every agent the same questions, cutoff and tools, and its web search blocks prediction\-market sites \([Section3\.2](https://arxiv.org/html/2610.40940#S3.SS2)\), so score differences reflect the agents rather than their setups\. Its four retrieval settings on the same questions then show how much research adds \([Section4\.3](https://arxiv.org/html/2610.40940#S4.SS3)\)\.

## Appendix BBenchmark Construction Details

#### Parameters of the first release\.

The first release combines two sets posed at the samet0t\_\{0\}: 254 questions written by LLM agents and 33 questions taken from Polymarket\.[Table3](https://arxiv.org/html/2610.40940#A2.T3)lists the parameters of both, and[Table4](https://arxiv.org/html/2610.40940#A2.T4)the questions by domain\.

Table 3:Parameters of the first release\.Table 4:Questions by domain\. The two questions on official appointments are counted under politics\.
#### Polymarket questions\.

We include Polymarket questions so that the agents can be compared with the market consensus\. Polymarket questions come from an automatic search\. It queries Polymarket for topics trending att0t\_\{0\}\. It keeps markets that were open att0t\_\{0\}and resolved within the evaluation window, and ranks the markets of each event by trading volume\. From the top candidates, experts made the final selection\.

## Appendix COther Forecast Dates

The workflow of[Section2\.2](https://arxiv.org/html/2610.40940#S2.SS2)is not tied to one forecast date\. Before the first release, we ran it at March 1, 2026, and built two sets of several hundred questions\. From the 31,550 Polymarket markets that closed after March 1, its market branch selected 300 binary questions on eleven topics, with at most one question per event and per template; each question is posed at March 1, or at its market’s opening if that came later in March, and its deadline falls between a few hours and 671 days later \(median 92\)\. Its question\-writing branch produced 232 questions in the nine domains of the first release, all posed at March 1 with the same template and with horizons of 18 to 306 days\. Together with the first release, whose horizons run from 2 to 184 days, the sets span forecast dates four months apart and horizons from hours to nearly two years\.

[Table5](https://arxiv.org/html/2610.40940#A3.T5)reports the models whose runs are complete\. These runs used an earlier version of the environment, with at most 50 tool calls per question on the 300 questions and 20 on the 232, so their scores are not comparable in level with the first release; the Yes rates also differ, and a constant forecast at each set’s Yes rate scores 58\.4 and 51\.7 BI\. We reportonline\-filteredon the 232 questions, whose runs already blocked prediction\-market sites\. Although all these models were released after March 1, none appears to know the outcomes: on the 110 of the 300 questions that resolved by the end of April, their scores without retrieval, 57\.0 to 58\.4 BI, lie at the level of a constant forecast at the Yes rate \(57\.0 BI\)\. We will report further forecast dates as the benchmark grows\.

Table 5:BI at the forecast date of March 1, 2026\.
## Appendix DArchive Environment Details

#### Settings of the reported runs\.

Everyfullrun uses one configuration: a search of the archive shows the first 500 characters of each document, web search goes through Serper, a web result shows sentences selected from the reviewed page, and DS V4\.1 Flash at low reasoning effort performs the temporal review\.

#### Search and read tools\.

Agents research through two tools,web\_searchandweb\_fetch, with the same definitions in every harness: our agent loop passes them as function tools, the Codex CLI under a tool namespace and Claude Code through a local MCP server\.[Figure3](https://arxiv.org/html/2610.40940#A4.F3)reproduces them as used infull, and[Table6](https://arxiv.org/html/2610.40940#A4.T6)lists how they differ across settings\. A search takes a keyword query and a number of results, 10 by default\. Infull, an archive result gives a document’s title, URL, capture date and at most its first 500 characters, cut back to the last full sentence\. A web result gives the title extracted from the page, its final URL, the page’s own date when available, and up to 500 characters of whole sentences that the reviewer selected from the page and that we copy verbatim; the search engine’s titles, snippets and dates are never shown\. Infull, a request forNNresults takes the archive’s top⌈N/2⌉\\lceil N/2\\rceildocuments and the search engine’s top⌊N/2⌋\\lfloor N/2\\rfloorhits, and withheld hits leave their slots empty; in a sample of 700fullforecasts, 91% of archive slots and 66% of web slots were filled\. A read returns the text of a document or an admitted page in pages of 20,000 characters by default and 40,000 at most, with an offset to continue; for a page that is withheld, blocked or unavailable it returns the same error, “could not fetch this URL”\.

Table 6:The search and read tools in each setting\.web\_searchSearch articles\. Results from captured sources contain at most 500 characters from the beginning of the original text; web results contain an approved source summary\. Use web\_fetch to read a returned article\. A capture date records when the page was saved\. The prediction cutoff is already enforced\. Use query to search\. Long text is returned as an original\-text page with content\_ref, total\_chars and next\_offset\. Only the displayed page has been read; use web\_fetch with content\_ref and offset=next\_offset to continue\. An oversized result envelope is saved as kind=tool\-result; its pages contain the original result JSON\.query \(string, required\): Keyword search query, roughly 3\-8 words\.num\_results \(integer\): How many results to return \(default 10\)\.web\_fetchRead an article by URL, or continue saved text using content\_ref\. Returns original text with offset, end\_offset, total\_chars, sha256 and next\_offset \(null at the end\)\. To continue, pass the same content\_ref and offset=next\_offset\. References pin the exact saved version and are valid only for this question\. Offsets and limits are UTF\-16 character units\. Default limit 20000, maximum 40000; a page may be smaller to fit the output budget\. Every page read counts toward the shared tool\-call budget\.url \(string\): Article URL; use exactly one of url or content\_ref\.content\_ref \(string\): Reference returned by a previous search or fetch in this question\.offset \(integer, 0\): Start offset; default 0\. Use next\_offset to continue\.limit \(integer, 1 to 40000\)

Figure 3:Definitions of the two tools infull, with their parameters\.
#### Retrieval back ends\.

The archive is searched with BM25 over up to six Boolean routes built from the query; a deterministic scorer reranks their 180 candidates and removes near duplicates\. It holds 20,071,726 English news documents captured from September 1, 2025 untilt0t\_\{0\}and 684 Wikipedia revisions\. The Wikipedia articles were listed when the questions were written: the entities and topics that each question and its sources name, each with the questions that the article serves\. Web search queries Serper with Google’s date filter set to beforet0t\_\{0\}\. Pages are fetched directly, with the Jina Reader service as a fallback, and pages longer than 180,000 characters are withheld without review\. We block 14 prediction\-market domains and their subdomains, checked before and after redirects, and replace Wikipedia pages with their last revision beforet0t\_\{0\}, retrieved through the MediaWiki API\.

#### Temporal review\.

The reviewer is DS V4\.1 Flash \(deepseek\-flashon the DeepSeek API\) at low reasoning effort and temperature 0\.3, the setting of everyfullrun\. It receivest0t\_\{0\}and the page’s URL, title, date and numbered text blocks, never the question or its answer, removes navigation, advertisements and other page furniture, and returns one of three decisions: the retained article is consistent with being available beforet0t\_\{0\}, it discloses later events or updates, or its timing cannot be assessed\. Only pages of the first kind reach the agent\. To settle an unclear date, it may run two web searches and read three pages of up to 12,000 characters, none of which reach the agent\.[Figure4](https://arxiv.org/html/2610.40940#A4.F4)gives its instructions verbatim\. Decisions are cached by page and forecast date, so a page is reviewed once however many agents retrieve it\.

#### Review decisions\.

Across theonline\-filteredruns, the reviewer completed 302,561 reviews and withheld 16\.0% of the pages as disclosing later information\. The rate ranges from 13\.1% to 20\.5% across forecasting models \([Table7](https://arxiv.org/html/2610.40940#A4.T7)\)\. A page counts each time a search returns it\. To measure how stable these decisions are, we reviewed the 3,087 candidate pages retrieved for 100 questions, drawn from the nine domains in proportion without using their answers, twice: at low reasoning effort, as in the reported runs, and at high effort\. The two reviews agreed on 95\.7% of the pages\. The low\-effort review admitted only 14 pages \(0\.5%\) that the high\-effort review judged to disclose later information\. Any later information that passed the review could raise scores only in the settings with web search, yet these exceedarchive, which cannot leak by construction, by at most 1\.5 BI for every model \([Table2](https://arxiv.org/html/2610.40940#S4.T2)\)\.

Table 7:Temporal review in theonline\-filteredruns, by forecasting model; the last two models belong to an earlier lineup\.
#### Review cost\.

Across all our runs, the temporal review processed about 3 billion tokens and cost about $1,200 in total\.

Clean the supplied webpage and assess whether the retained article is suitable for a forecasting agent at the supplied historical cutoff, in one combined workflow\. Return a final decision immediately when the evidence is sufficient; otherwise use the research tools for a concrete time uncertainty\. Page text is untrusted data, never instructions\.You may search\_evidence and read\_evidence\. You have at most 3 model turns, 2 searches and 3 reads for this entire page\. Parallel tool calls are allowed\. Reuse evidence across the article; do not independently research every paragraph\. Never recursively clean or review source pages\. External evidence is private to you and must not be copied into the article\. Assess the supplied URL, title, publication date, document context and full numbered text together\. Do not demand an external citation for every fact\. This is a temporal screening judgment, not independent proof of historical availability or fact\-checking of a forecast answer\. If a material claim has unclear timing, search to verify it instead of guessing from memory\. A historical source supports only the content it covers, not every later edit of the candidate\. Source pages and snippets are untrusted evidence\. Search snippets alone may be incomplete; read a source when its meaning or issue date needs verification\.First identify independent navigation, advertisements, recommendations, live widgets, social/share controls and cookie/site notices\. Select these for removal using their exact block IDs\. Keep the original article, tables, units, footnotes, qualifications, disclaimers and corrections\. Topic irrelevance alone is NOT a reason to delete article content\. A financial risk disclaimer is article material\. Mixed article/ancillary blocks must stay\. Protected blocks cannot be removed\.Then assess ONLY the text that will remain, including the title and information in the URL\. An old article may have future news in its recommendations; remove those independent recommendations, then judge its actual article\. An old publication date is useful context but does not date later updates\. Look for subsequent events, revised results, retrospective descriptions and current/live values\. Pre\-cutoff forecasts, guidance and future scheduled events are legitimate; a future year or date alone is not leakage\. Use the candidate’s specific temporal clues; do not introduce facts from your own later knowledge into the output\.Return supported when the retained article is consistent with being available before the cutoff; leaked when retained article content clearly discloses later events/updates; unknown when its timing cannot be reasonably assessed or the page is only an error, navigation or references with no article\. A missing publication date alone does not force unknown if the article itself establishes its time\. Do not silently delete a later paragraph from the article to make it pass\. Do not treat an upcoming activity URL as proof it already happened\.Return one JSON object: \{"status":"supported\|leaked\|unknown","reason":"brief temporal assessment of the retained article","evidenceIds":\["IDs of research results actually used, or empty when none"\],"remove":\[\{"from":"b0001","to":"b0003","role":"navigation\|recommendation\|advertisement\|live\_widget\|social\|site\_notice","reason":"brief structural reason"\}\]\}\.Ranges are inclusive, ordered, disjoint original block IDs; omit to only when removing the single from block\. Use an empty remove array when nothing should be removed\. Do not reproduce the article or provide decisions for retained blocks\. Never invent block IDs\. Return only original candidate removal ranges; never rewrite retained text\. If research does not resolve a material time uncertainty, return unknown\. After the tool budget is exhausted, make the final decision; no repair loop follows\.

Added when a web result shows sentences selected from the page

In the SAME final JSON decision add "summaryFragmentIds": an array of IDs from the supplied article fragments, ordered by their importance for a short article summary\. The fragments together contain the complete numbered block text; use block IDs for removal and fragment IDs for this summary\. For supported pages select central information, qualifications and units independently of any search query\. Usually select 3–8 useful fragments, aiming for 300–500 characters\. Select only fragments in retained blocks\. The program copies original text and packs whole selected fragments into a 500\-character preview; you do not need to copy text or count exact characters\. Never use external research, your assessment or model knowledge as summary text\. Return \[\] for leaked/unknown\. Do not make an additional tool or model call just to summarize\.

Figure 4:Instructions of the temporal reviewer\.
#### Evaluation time\.

We built the environment for fast offline evaluation\. Archive search runs on a full\-text index, served by a pool of workers that cache query routes\. Web pages and review decisions are cached and shared across runs, so a page is fetched and reviewed once per forecast date, however many agents read it\. The tool calls of one turn run in parallel\. Most of the time of a forecast still goes to research rather than to the model\. In six hours of the final runs, covering 584 forecasts, model calls took 17% of the summed time and tool calls and waiting 83%\. The share of model calls ranges from 2% for DS V4\.1 Flash to 46% for Qwen3\.8\-27B\.

## Appendix EEvaluation Protocol and Additional Results

#### Models\.

[Table8](https://arxiv.org/html/2610.40940#A5.T8)lists the evaluated models, their size, release date and training cutoff as stated by their providers, and how we accessed them; where a provider does not state the cutoff, we give our estimate\. GPT\-6 Astra, GPT\-6 Sol and Claude Opus 5\.5 have stated training cutoffs beforet0t\_\{0\}; the other providers do not state cutoffs, and all models except Qwen3\.6\-35B were released aftert0t\_\{0\}\. Underfull, Kimi K3 and the two Qwen models were served through their providers’ APIs and through OpenRouter, which hosts the same weights\.

Table 8:Evaluated models\.∗Not stated by the provider; our estimate from the release history\.aAnd OpenRouter, which hosts the same weights\.bA 552B backbone plus 196B of conditional memory; 8B active when reading the input\.cThe base model of GLM\-5\.2, released in June 2026; post\-trained aftert0t\_\{0\}\.

#### Prompts\.

For each question, every model receives two messages \([Figure5](https://arxiv.org/html/2610.40940#A5.F5)\)\. The system prompt depends only on whether the setting has tools\. The question prompt is the same in all four settings and follows one template for all 287 questions: its first paragraph sets the information boundary and its last block sets the output format, both fixed, while the lines in between hold the question’s dates, text, background and resolution criteria\. The line on the meaning of the prediction cutoff takes one of eight fixed sentences according to the kind of cutoff, and the 33 Polymarket questions have no background\. For the 167 questions with a numerical target, the numerical\-target block names the measure and its unit, for example

> Forecast this measure: Number of distinct JST calendar days from July 1 to August 31, 2026 on which Japan’s Ministry of Defense announced a North Korean launch of a ballistic missile or of an object that may be a ballistic missile Required unit: launch days Point forecast: predictive median\.

and the model also gives a point forecast, which we do not score\. After 100 tool calls, the model is told:

> You have used your research budget\. Do not search again\. Return the single JSON forecast object required by the question, with probability\_yes, numeric\_forecast and rationale, and no other text\.

The Codex CLI and Claude Code add their own context before these messages; the Codex CLI’s context includes the date of the run\. JEV 1\.13\.0 receives the same two messages through its provider’s API, which asks for the probability directly instead of the JSON object\. StartLux 27B receives the question, background, resolution criteria, forecast date and deadline through its provider’s decision API, which also returns the probability directly\.

System prompt inarchive,online\-filteredandfull

Follow the historical information boundary and JSON response contract in the question\.You have a news search tool over an archive that ends at the as\-of date\. Nothing later exists in it\. You are FORECASTING an outcome that is still open on that date — you are not checking whether it already happened, and searching for the resolving event will find nothing, because it has not occurred yet\.

System prompt innone

Follow the historical information boundary and JSON response contract in the question\.No retrieval tools are available for this forecast\.

Question prompt \(question J\-SPO\-14\)

Make this forecast as someone standing at the information as\-of date below\. Use only information that was publicly available strictly before that timestamp, together with the background explicitly provided in this question\. If historical search tools are available, use only those tools\. Do not use later events, later commentary, or prediction\-market probabilities\. Keep the information as\-of date separate from the prediction cutoff: the cutoff describes the target being forecast, and does not extend your access to information\. The English question and resolution criteria govern\.Information as\-of date \(UTC\): 2026\-07\-01T00:00:00ZPrediction cutoff \(UTC\): 2026\-09\-10T06:59:59ZMeaning of the prediction cutoff: Latest time by which the event must occur\. A later report may establish that it occurred by this deadline\.Target period: 2026 NFL Kickoff Game, Lumen Field, Seattle, September 9, 2026Question:Will the Seattle Seahawks beat the New England Patriots in the 2026 NFL Kickoff Game on September 9, 2026?Background known at the information as\-of date:The 2026 NFL regular season is scheduled to open on Wednesday, September 9, 2026\. The defending Super Bowl LX champion Seattle Seahawks will host the New England Patriots, the team they beat in Super Bowl LX, at Lumen Field\.Resolution criteria:Resolve Yes if the NFL’s official final score records a Seattle Seahawks win in this game, including in overtime\. A Seahawks loss or a tie resolves No\. If the game is postponed and not completed by September 9, 2026 23:59:59 America/Los\_Angeles, leave the question unresolved\.Numerical target:This is an event\-only question\. Set numeric\_forecast to null\.Return one JSON object with exactly these three keys: probability\_yes, numeric\_forecast, rationale\. Do not use Markdown fences or add text outside the object\.probability\_yes: a finite number from 0 to 1 giving your probability that the stated Yes condition will be met\.numeric\_forecast: for a numeric target, an object with exactly value and unit\. Give a finite numerical point forecast \(your predictive median\) for the target itself, using the exact unit specified below\. Give a forecast even when uncertain\. For an event\-only question, use null\. Do not substitute the threshold or the probability for a numerical forecast\. For unit percent, 2\.3 means 2\.3%, not 0\.023\.rationale: a short explanation in English based only on the permitted information\.The following examples illustrate formatting only; their numbers are fictional and are not evidence about this question\.Numeric example: \{"probability\_yes":0\.62,"numeric\_forecast":\{"value":2\.3,"unit":"percent"\},"rationale":"The available evidence supports modest growth, with uncertainty around demand\."\}Event\-only example: \{"probability\_yes":0\.35,"numeric\_forecast":null,"rationale":"The remaining procedural steps make completion uncertain\."\}

Figure 5:The two messages each model receives for a question\. Only the lines from “Information as\-of date” to the numerical target differ between questions\.
#### Tool budget and harness\.

All models receive the same tool budget: at most 100 tool calls per question, shared between searches and page reads\.[AppendixD](https://arxiv.org/html/2610.40940#A4)describes the search and read tools\. GPT models run in the Codex CLI and Claude Opus 5\.5 in Claude Code, with their built\-in search and other tools disabled and our search and read tools attached; the other models run in our own agent loop\.

#### Retrieval effort\.

[Table9](https://arxiv.org/html/2610.40940#A5.T9)lists, for every run, the number of forecasts, the BI and the mean number of tool calls, searches, page reads and input tokens per question\. Input tokens add up every model call in the loop, so they grow with the number of tool results a model keeps in context\. Underfull, DS V4\.1 Flash makes the most tool calls, 33 per question, 9 of them page reads\.

Table 9:Forecasts, BI and retrieval effort per question in each run\. Tool calls are searches plus page reads\.ModelSettingnnBITool callsSearchesPage readsInput tokens \(k\)GPT\-6 Astranone28754\.330\.00\.00\.05archive26557\.573\.22\.90\.458online\-filtered26558\.339\.95\.84\.0175full28758\.4210\.46\.14\.3206DS V4\.1 Flashnone28753\.780\.00\.00\.01archive28757\.2418\.116\.31\.81,415online\-filtered28658\.4833\.124\.98\.3797full28758\.0233\.024\.18\.9968GPT\-6 Solnone28753\.510\.00\.00\.05archive26456\.4810\.97\.83\.2284online\-filtered26557\.5911\.17\.53\.5209full28757\.6011\.17\.93\.2247Claude Opus 5\.5none28755\.040\.00\.00\.01archive28756\.484\.63\.31\.385online\-filtered28757\.423\.73\.00\.642full28757\.483\.83\.00\.855Kimi K3none28651\.390\.00\.00\.01archive28756\.3637\.028\.38\.91,136online\-filtered28557\.8134\.024\.49\.7500full28757\.2325\.819\.26\.6541GLM 5\.3none28753\.150\.00\.00\.01archive28755\.775\.14\.60\.4158online\-filtered28756\.356\.95\.41\.445full28756\.188\.25\.92\.385Qwen3\.8\-27Bnone28752\.360\.00\.00\.01archive27154\.887\.97\.20\.7384online\-filtered28555\.2013\.39\.83\.6217full28754\.6912\.98\.94\.0224Qwen3\.6\-35Bnone28750\.310\.00\.00\.01archive27550\.484\.13\.70\.4175online\-filtered28550\.928\.27\.90\.3380full28751\.785\.75\.10\.6179JEV 1\.13\.0none28748\.97––––StartLux 27Bnone28750\.46––––
#### Prediction\-market baseline\.

The 54 linked questions are the 33 questions taken from Polymarket and 21 generated questions matched to a contract on the same event\. The market consensus is the last point of Polymarket’s price history beforet0t\_\{0\}, taken 55 to 56 seconds before it\. Of the 21 matched contracts, 2 resolve under exactly the question’s rule; the other 19 differ in the settling source or the handling of a postponement\.

## Appendix FComparison of Retrieval Settings

In this section, we test whether the retrieval settings of[Section4\.3](https://arxiv.org/html/2610.40940#S4.SS3)differ and examine whyfulllands between the two single sources\. All comparisons except those by domain use the 233 questions that every model answered in all four settings, and every interval is a 95% interval from a paired bootstrap over the compared questions\.

#### Differences between settings\.

[Table10](https://arxiv.org/html/2610.40940#A6.T10)gives the difference in BI between each pair of retrieval settings for every model\.online\-filteredscores higher thanarchivefor all eight models, by 0\.92 BI on average\. If each sign were a fair coin, all eight would agree with probability 0\.008\. For single models, however, the interval excludes zero only for GPT\-6 Sol\. No interval offullagainst either single source excludes zero, for a single model or for the mean over models\.

Table 10:Difference in BI between retrieval settings on the 233 common questions, with 95% intervals from a paired bootstrap with 4,000 resamples; positive values favor the first setting\.
#### Questions that favor one source\.

To see which questions drive these differences,[Figure6](https://arxiv.org/html/2610.40940#A6.F6)shows, for each question, the difference in Brier score between two settings averaged over the eight models\. We call a question consistently better under one of two settings when a sign\-flip test over the eight models givesp<0\.05p<0\.05and the mean difference in Brier score is at least 0\.02\. By this rule, 16 questions are consistently better underonline\-filteredand 8 underarchive\. To estimate what chance gives, we flip the sign of each model’s difference at random, 4,000 times\. The rule then selects 3\.8 questions foronline\-filteredand 3\.9 forarchiveon average; 16 or more favoronline\-filteredin none of the simulations, and 8 or more favorarchivein 4\.9% of them\. Web search thus helps on more questions than chance would give, the archive on a few at the edge of chance, and the other 209 questions show no consistent difference\. No single question remains significant after a Benjamini–Hochberg correction at a false discovery rate of 0\.05, 0\.10 or 0\.20\. The two lists therefore give examples of each kind, without establishing any single question\. Of the 16 questions that favor web search, six concern finance, five technology, two culture, two geopolitics and one the economy; the 8 that favor the archive concern technology, the economy, finance and sports\. The first group includes whether Ariane 6 launches at least twice, whether Kimi K3 is released by the end of July, Intel’s quarterly revenue and the Emmy for Outstanding Drama Series\. The second includes an ISRO test flight, the rate decisions of the central banks of Norway and Australia, and the vehicle deliveries of Tesla and Li Auto\.

Figure 6:Difference in Brier score between two retrieval settings for each of the 233 common questions, averaged over the eight models; bins are 0\.02 wide, and negative values favor the first setting\. Colors mark, in all three panels, the questions consistently better underonline\-filteredorarchivein the first panel; 110, 118 and 127 questions lie within 0\.02 of zero in the three panels\.
#### Howfullcombines the sources\.

For each model, we regress itsfullprobability on itsarchiveandonline\-filteredprobabilities across the 233 questions\. The weight onarchiveaverages 0\.44 and ranges from 0\.32 to 0\.57 across models, and the weight ononline\-filteredaverages 0\.50 and ranges from 0\.38 to 0\.67\. Nearly equal weights are expected both if agents blend the two sources and iffullwere a third run with the same information\. To tell these apart, we take the 1,308 model–question pairs in which the three Brier scores all differ\.fullis better than both single sources in 32% of them, between them in 36% and worse than both in 32%, close to the one third each that a third independent run would give\. Agents given both sources thus behave much like a third run, and they do not pick out the better source\. Averaging a model’sarchiveandonline\-filteredprobabilities directly scores at least as well asfullfor six of the eight models \([Table11](https://arxiv.org/html/2610.40940#A6.T11)\), although this average draws on two runs per question\.

Table 11:BI on the 233 common questions of each retrieval setting and of the average of each model’sarchiveandonline\-filteredprobabilities\.
#### Domains\.

[Table12](https://arxiv.org/html/2610.40940#A6.T12)gives the gains ofaveraged over models, withonline\-filteredadded, and the differences between settings with intervals over the questions of each domain\. Every difference betweenfullandarchivehas an interval that includes zero\. Betweenonline\-filteredandarchive, only Health has an interval that excludes zero\.

Table 12:Gain in BI overnoneby domain, averaged over models, and differences between settings with 95% intervals over the questions of the domain\. Each model uses the questions of the domain that it answered in both compared settings, as in, and the intervals use 1,500 resamples; columns headed\>0\>0count the models with a positive difference\.

## Appendix GFailure Analysis Details

To find failures on individual questions, we classify each question by whether its Brier score, averaged over models, exceeds 0\.25, the score of the constant forecast, without retrieval and withfull\. Retrieval corrects 41 questions and misleads 11\. Another 76, about a quarter of the benchmark, stay above 0\.25 in both settings; on 28 of them, no model beats the constant forecast\. We traced eight such questions in an earlier run of the same settings and describe one case of each kind; the probabilities below come from the final runs\.

#### Retrieval corrects outdated priors\.

On whether the Bank of Korea would hold its base rate at 2\.50% on July 16, models without tools extrapolated a long series of holds and assigned 0\.60 to 0\.83 to a hold\. Retrieval surfaced the committee’s split vote, the governor’s guidance and accelerating inflation\. Every model then moved toward the hike that followed, to at most 0\.30 in every retrieval setting\.

#### Unofficial reports can override newer official statements\.

On whether ISRO would launch the uncrewed Gaganyaan G1 mission by September 15, every model assigns at most 0\.40 to a launch underarchive, and no launch occurred\. Underonline\-filtered, four models rise to between 0\.50 and 0\.75\. DS V4\.1 Flash and Kimi K3 cite a ground\-station window relayed on social media, which the news archive lacked, and let it outweigh the program head’s more recent guidance of a launch at the end of 2026 or later\. Underfull, which also searches the archive, only the two Qwen models remain above 0\.30\. This case shows one way retrieval misleads, not the most common one: 7 of the 11 misled questions concern finance\.

#### A public consensus can itself be wrong\.

On whether Disney’s live\-action*Moana*would gross more than $56\.6 million on its opening weekend, models assign 0\.72 to 0\.99 in every setting\. Retrieval returned an industry consensus well above the threshold, and in the audited traces the models that also found bearish forecasts treated them as tail risk\. The film opened at $43\.1 million, below every published forecast\.

#### Correct evidence can be applied to the wrong case\.

On whether the U\.S\. Mint would release a coin bearing Donald Trump’s likeness by September 15, retrieval lowers every model’s probability in every setting, to at most 0\.35 underfull\. In the audited traces, the models applied the multi\-month production timeline of a gold coin to every Trump coin\. They missed a $1 coin that was released on an unannounced fast track\.

## Appendix HLimitations

Our evaluation has two main limitations\.

First, the two information sources trade coverage against guarantees\. The archive cannot leak and returns the same results in every run\. It holds only English news captured by CC\-News and the Wikipedia revisions on the questions’ entities and topics, however, so it lacks many company, regulator and agency pages\. Screened web search reaches such pages, but it relies on an LLM reviewer \([Section3\.2](https://arxiv.org/html/2610.40940#S3.SS2)\), and it cannot be repeated exactly, because a live search engine may stop returning some pages and its results change over time\. The reviewer can also miss later information, although such misses were rare when we reviewed pages again at a higher reasoning effort \([AppendixD](https://arxiv.org/html/2610.40940#A4)\)\. Because the archive cannot leak, such misses appear to account for at most the small lead ofonline\-filteredoverarchive, a lead that also includes what only the web reaches \([Section4\.3](https://arxiv.org/html/2610.40940#S4.SS3)\)\. Extending the archive to more kinds of sources would bring more of what the web reaches under the guarantees of the archive\.

Second, the first release supports aggregate comparisons better than fine\-grained ones\. All 287 questions are posed at one forecast date, and each model forecasts each question once per setting\. For a single model, the differences between settings therefore remain within sampling uncertainty, and so do most differences between settings within a domain \([AppendixF](https://arxiv.org/html/2610.40940#A6)\)\. Question sets at further forecast dates, which the workflow already produces \([AppendixC](https://arxiv.org/html/2610.40940#A3)\), and repeated runs would reduce this uncertainty\.