Forecast-Dojo: 用于基准测试与训练LLM预测代理的可重复环境

# Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents
Source: [https://arxiv.org/html/2609.28876](https://arxiv.org/html/2609.28876)
Haorui WangAffiliation:Georgia Institute of TechnologyEmail:[hwang984@gatech\.edu](mailto:)Fardin AhmedAffiliation:Georgia Institute of TechnologyRongzhi ZhangAffiliation:AmazonYuan HeAffiliation:AmazonZiyuan LinAffiliation:University of FloridaYanbin YinAffiliation:Georgia Institute of TechnologyJing PengAffiliation:Georgia Institute of TechnologyMichael GalarnykAffiliation:Georgia Institute of TechnologySudheer ChavaAffiliation:Georgia Institute of TechnologyChao ZhangAffiliation:Georgia Institute of Technology

###### Abstract

We introduceForecast\-Dojo, a replayable environment for benchmarking and training LLM forecasting agents\. It combines resolved prediction\-market questions with dated news, allowing agents to research an event and revisit their predictions at successive historical dates\. The same tasks and tools support repeated evaluation, collection of training interactions, and feedback from recorded outcomes without waiting for new events to resolve\.Forecast\-Dojocontains 1,568 Polymarket events, split by time into training and evaluation periods, and 18\.8M dated news articles\. In an evaluation of 12 models, research tools lower Brier score for all 12\. Forecasts also improve as events unfold, with the largest gains at steps where more newly dated evidence is recorded\. Every model still trails historical market forecasts in both Brier score and accuracy\. A belief notebook carried between dates lowers research cost but does not consistently improve forecast quality\. Beyond evaluation,Forecast\-Dojoprovides interaction trajectories and outcome feedback for agent learning, with supervised fine\-tuning as a proof of concept\. Our[code](https://github.com/liqinye/Forecast-Dojo)and[data](https://huggingface.co/datasets/liqinye/Forecast-Dojo)are publicly available\.

††footnotetext:\*Equal contribution\.††footnotetext:†Work done outside Amazon\.## 1Introduction

Large language model \(LLM\) agents are increasingly used to forecast real\-world events by actively searching for evidence, reasoning under uncertainty, and producing probabilistic predictions\([Halawi et al\., 2024](https://arxiv.org/html/2609.28876#bib.bib3);[Zeng et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib6);[Seed, 2026](https://arxiv.org/html/2609.28876#bib.bib7);[Karger et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib8)\)\. Forecasting is inherently time\-dependent: the evidence available to a forecaster changes as new information arrives\. The same question can therefore pose a different prediction problem at different times, requiring the forecaster to update its belief as new evidence emerges\.

Existing forecasting benchmarks capture only part of this process \(Table[1](https://arxiv.org/html/2609.28876#S1.T1)\)\. Live benchmarks\([Karger et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib8);[Zeng et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib6);[Zhao et al\., 2026](https://arxiv.org/html/2609.28876#bib.bib9);[Zhang et al\., 2026](https://arxiv.org/html/2609.28876#bib.bib10)\)pose unresolved questions, so the forecasting problem evolves naturally with the world, but they run on wall\-clock time\. A past forecasting condition cannot be recreated for a model released later, and outcomes arrive only at resolution, which slows evaluation and makes training impractical\. Historical benchmarks\([Jin et al\., 2021](https://arxiv.org/html/2609.28876#bib.bib11);[Halawi et al\., 2024](https://arxiv.org/html/2609.28876#bib.bib3);[Chandak et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib1)\)reconstruct past information cutoffs, making resolved events immediately scorable and reusable\. However, they typically evaluate each forecast at a single historical point rather than revisiting the same question across multiple points in time\. The ideal setting combines the two: the forecasting problem evolves as new evidence becomes available, yet each past step can be replayed with the same task and information cutoff\. Models can then be compared under identical conditions and scored immediately, and their forecasting trajectories become usable for training\.

Table 1:Summary of existing forecasting benchmarks\.The criteria target the capabilities used to construct and study Forecast\-Dojo\.*Aligned evaluation*indicates that the same forecasting task is evaluated at a pre\-specified sequence of forecast dates or evidence states shared across models;: provided;: not provided;: system\- or agent\-dependent\.BenchmarkDatedreplayAgentresearchAlignedevaluationAgentmemoryMarketbeliefTrain/evalsplitForecastQA\([Jin et al\., 2021](https://arxiv.org/html/2609.28876#bib.bib11)\)Autocast\([Zou et al\., 2022](https://arxiv.org/html/2609.28876#bib.bib28)\)MIRAI\([Ye et al\., 2024](https://arxiv.org/html/2609.28876#bib.bib36)\)ForecastBench\([Karger et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib8)\)FutureX\([Zeng et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib6)\)Prophet Arena\([Yang et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib30)\)EvolveCast\([Yuan et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib2)\)BTF\-2\([Liptay et al\., 2026](https://arxiv.org/html/2609.28876#bib.bib34)\)FutureSim\([Goel et al\., 2026](https://arxiv.org/html/2609.28876#bib.bib32)\)Forecast\-DojoWe introduceForecast\-Dojo, a replayable environment that reconstructs resolved real\-world events as sequences of historical forecast steps \(Figure[1](https://arxiv.org/html/2609.28876#S1.F1)\)\. At each step, an LLM agent can search a temporally\-restricted information corpus, inspect full documents, and use computational tools to produce a probabilistic forecast using only information available by that date\. Each step can be reset and rerun across models or repeated trials, while steps from the same event can be traversed sequentially with persistent agent memory\. The realized outcome is retained by the environment for immediate scoring but never exposed to the agent during its forecasting\. This common interaction interface supports both benchmarking and learning: held\-out events evaluate agents under controlled conditions, while training events generate forecasting trajectories and outcome feedback for learning\.

Empirically, we evaluate 12 models on 230 held\-out events without tools, with research tools, and with research tools plus a belief notebook carried between dates\. Research tools lower Brier for all 12 models even without memory\. In this memory\-free setting, mean Brier also falls from 0\.670 in the first third of an event to 0\.606 in the last third, while forecasts without tools stay flat\. The improvements concentrate at steps with more new evidence, and models that record more new evidence tend to improve more\. Every model still trails the historical market forecasts in both Brier score and accuracy, with the best Brier at 0\.546 against 0\.498 for the market\. The belief notebook reduces research costs by a median of 24%, but its effect on forecast quality is mixed: Brier improves for only 6 of the 12 models \(Section[5\.2](https://arxiv.org/html/2609.28876#S5.SS2)\)\. Finally, a supervised fine\-tuning study shows that an agent trained on trajectories from theForecast\-Dojotraining split achieves lower Brier scores and higher accuracy than its base model on later, held\-out evaluation events \(Section[5\.5](https://arxiv.org/html/2609.28876#S5.SS5)\)\.

We summarize our contributions as follow:

- •A replayable forecasting environment\.Forecast\-Dojoreplays each resolved event as a fixed sequence of dated forecast steps, with an optional belief notebook carried between them\. Agents research news available up to each date, and every step is scored against the realized outcome for evaluation or training\.
- •A dataset of resolved events and dated news\.It covers 1,568 Polymarket events with 6,122 forecast steps, split by time into 1,338 training and 230 evaluation events\. Evidence comes from 18\.8M CC\-News articles, filtered so that each step sees only news up to its forecast date\.
- •A benchmark study beyond overall rankings\.Besides comparing 12 models with historical market forecasts, we examine how forecasts change as events unfold, how these changes relate to new evidence, and how memory affects quality and cost\. A fine\-tuning study shows how the collected interactions can be used to train an agent\.

Figure 1:Overview of Forecast\-Dojo\.Top: resolved Polymarket events are filtered and split by time into training and evaluation events, and CC\-News articles are cleaned, dated, and indexed\. Middle left: each question is forecast at a fixed sequence of dates as the visible corpus grows, with an optional belief notebookMtM\_\{t\}passed between steps\. Bottom left: within one step, the agent searches and reads articles dated on or beforeτt\\tau\_\{t\}, runs code in a sandbox, and commits a forecastptp\_\{t\}\. Right: the realized outcomeYYstays hidden from the agent\. The evaluator scores each forecast against it, using held\-out events for evaluation and training events for learning\.
## 2Related Work

#### Forecasting with dated evidence\.

ForecastQA restricts news by time, while Autocast pairs forecasting questions with dated articles and historical human forecasts\([Jin et al\., 2021](https://arxiv.org/html/2609.28876#bib.bib11);[Zou et al\., 2022](https://arxiv.org/html/2609.28876#bib.bib28)\)\. ForecastBench and FutureX collect predictions on unresolved events\([Karger et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib8);[Zeng et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib6)\)\. Related settings evaluate repeated forecasts, macroeconomic nowcasts, and simulated market decisions\([Yang et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib30);[Zhao et al\., 2026](https://arxiv.org/html/2609.28876#bib.bib9);[Cheng et al\., 2026](https://arxiv.org/html/2609.28876#bib.bib31)\)\.Forecast\-Dojouses resolved events to support repeatable research and immediate outcome\-based feedback, while keeping market probabilities outside the forecasting prompt\.

#### Interactive research and forecast revision\.

Bench to the Future uses frozen research corpora, and BTF\-2 records traces to distinguish information gathering from judgment\([Wildman et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib35);[Liptay et al\., 2026](https://arxiv.org/html/2609.28876#bib.bib34)\)\. MIRAI provides code\-based access to events and news, while WorldReasoner evaluates outcome, evidence, and reasoning quality\([Ye et al\., 2024](https://arxiv.org/html/2609.28876#bib.bib36);[Chi et al\., 2026](https://arxiv.org/html/2609.28876#bib.bib37)\)\. FutureSim studies long\-horizon adaptation in a replayed world, where agents jointly decide how to research, maintain memory, revise forecasts, and progress through time\([Goel et al\., 2026](https://arxiv.org/html/2609.28876#bib.bib32)\)\.Forecast\-Dojoinstead treats time progression as part of the experimental design: each event is replayed at a fixed sequence of historical checkpoints shared by all agents\. This produces matched longitudinal trajectories, allowing models to be compared at identical information states and enabling controlled study of how forecast quality changes with newly available evidence\. The same aligned episodes also provide reusable interaction trajectories and outcome feedback for downstream learning\. This complements work on evidence\-driven revision, probability coherence, and iterative research workflows\([Yuan et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib2);[Paleka et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib29);[Murphy, 2026](https://arxiv.org/html/2609.28876#bib.bib40)\)\. This complements work on revisions after supplied evidence, probability coherence, and evidence summaries within a research loop\([Yuan et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib2);[Paleka et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib29);[Murphy, 2026](https://arxiv.org/html/2609.28876#bib.bib40)\)\.

#### Environments for training agents\.

MLE\-Dojo provides executable machine learning engineering tasks and feedback for evaluation and training\([Qiang et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib33)\)\.Forecast\-Dojofollows this environment\-centered approach, specifying the forecasting interaction and its feedback while leaving learning algorithms separate\. Prior forecasting work trains on question collections or selected model\-generated forecasts\([Zou et al\., 2022](https://arxiv.org/html/2609.28876#bib.bib28);[Halawi et al\., 2024](https://arxiv.org/html/2609.28876#bib.bib3)\), and recent methods use reinforcement learning, outcome\-based fine\-tuning, and news\-derived questions\([Levy, 2026](https://arxiv.org/html/2609.28876#bib.bib38);[Jeen et al\., 2026](https://arxiv.org/html/2609.28876#bib.bib39);[Chandak et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib1)\)\. Our contribution is the shared task and tool interface for collecting research interactions and evaluating agents on separate events; the SFT experiment demonstrates one use of that interface\.

Table 2:Forecast\-Dojo task splits\. Memory free/on forecasting use the same forecast steps\.

## 3Forecasting as an Interactive Task

We formulate forecasting as an interactive task over a sequence of forecast steps for the same event\. At each step, the agent researches evidence available up to the current date and produces a probabilistic forecast\. Across steps, the evidence boundary advances and explicit memory may persist\.

#### Events, forecast steps, and episodes\.

A questionQQspecifies an event, its resolution criteria, and a finite set of mutually exclusive outcomes𝒴=\{1,…,K\}\\mathcal\{Y\}=\\\{1,\\ldots,K\\\}\. LetY∈𝒴Y\\in\\mathcal\{Y\}denote the realized outcome\. At ordered datesτ1<⋯<τT\\tau\_\{1\}<\\cdots<\\tau\_\{T\}before the event closes, the agent performs a*forecast step*: it researches the event and reports a probability distributionpQ,t∈ΔK−1p\_\{Q,t\}\\in\\Delta^\{K\-1\}\. The ordered forecast steps for the same question form a*forecast episode*\. All steps concern the same eventual outcome, but they differ in the historical evidence available at the forecast date\.

#### Within\-step interaction\.

Letℐ\\mathcal\{I\}denote a fixed information corpus\. At dateτt\\tau\_\{t\}, the agent is exposed only to

ℐ≤τt=\{d∈ℐ:date⁡\(d\)≤τt\},ℐ≤τt⊆ℐ≤τt\+1,\\mathcal\{I\}\_\{\\leq\\tau\_\{t\}\}=\\\{d\\in\\mathcal\{I\}:\\operatorname\{date\}\(d\)\\leq\\tau\_\{t\}\\\},\\qquad\\mathcal\{I\}\_\{\\leq\\tau\_\{t\}\}\\subseteq\\mathcal\{I\}\_\{\\leq\\tau\_\{t\+1\}\},\(1\)wheredate⁡\(d\)\\operatorname\{date\}\(d\)denotes the timestamp assigned to documentdd\. Within a forecast step, the agent may issue search queries, inspect retrieved articles, and use computation before submitting its forecast \(see Section[4\.4](https://arxiv.org/html/2609.28876#S4.SS4)\)\. LetHtH\_\{t\}denote the within\-step interaction history, including the agent’s research actions and the resulting tool observations\. The agent producespQ,tp\_\{Q,t\}conditioned onHtH\_\{t\}\. Because agents choose their own queries and which documents to inspect, different rollouts at the same date from an agent can follow different research paths even under the same dated evidence boundary\.

#### Progression across forecast steps\.

After each report, the environment advances to the next scheduled date and starts a fresh interaction\. We define two modes for carrying information across forecast steps:*memory\-free*and*memory\-on*\. In memory\-free forecasting, each step starts without information produced at previous steps\. In memory\-on forecasting, the agent produces a belief notebookMtM\_\{t\}that summarizes its current assessment, supporting evidence, and open questions\. At the next step,MtM\_\{t\}is provided alongside the question and new forecast date, allowing the agent to update its forecast from its prior assessment as new evidence becomes available\.

#### Outcome feedback\.

Each forecast step ends with a probability reportpQ,tp\_\{Q,t\}\. Once the realized outcomeYYis available, the environment assigns feedbackrQ,t=S⁡\(pQ,t,Y\)r\_\{Q,t\}=S\(p\_\{Q,t\},Y\)whereSSis an outcome\-based scoring rule\. The outcome and feedback are not part of the agent’s forecasting context\. This separates the interaction that produces a forecast from the feedback assigned to it, allowing the same task interface to support different downstream evaluation or learning procedures\.

## 4Forecast\-Dojo

Section[3](https://arxiv.org/html/2609.28876#S3)defines the forecasting task abstractly\. We now describe how Forecast\-Dojo instantiates its events, information corpus, forecast episodes, runtime, and evaluation records\.

### 4\.1Forecasting Events

We construct our candidate pool from resolved Polymarket111[https://polymarket\.us/](https://polymarket.us/)binary and mutually exclusive multi\-option events whose full lifetimes fall between June 2025 and June 2026\. For each event and date, we interpret market prices as a contemporaneous belief over its possible outcomes, which we refer to as the*market belief*\. For binary events, the YES price determines the probability of YES and its complement; for multi\-option events, we normalize the option\-level YES prices to obtain a distribution over the mutually exclusive outcomes\.

Within this pool, we further select events along four dimensions to ensure they are well suited to repeated, evidence\-grounded forecasting\. \(1\)*Forecasting horizon*: we require a tradeable lifetime longer than 7 days so that the same question supports multiple forecast steps rather than only a near\-resolution prediction\. \(2\)*Market faithfulness*: we require sufficient trading activity on most days, so that its daily price is a reliable reflection of the market belief rather than stale or weakly supported quotes\. This also favors questions with sustained market attention over obscure or inactive events\. \(3\)*Task nontriviality*: we remove events where the market already assigns near\-certain probability to the realized outcome, as well as events whose market history is both nearly flat and directionless\. This avoids trivial or temporally uninformative questions and preserves meaningful room for forecast revision as evidence accumulates\. \(4\)*Evidence sufficiency*: following[Joren et al\. \(2025\)](https://arxiv.org/html/2609.28876#bib.bib17), we retain only questions for which the information corpus contains sufficient pre\-resolution evidence to support an informed forecast, excluding questions that are poorly covered by or largely unrelated to the corpus available to the agent\. Together, these filters yield temporally rich, nontrivial, and evidence\-grounded questions suitable for repeated evaluation and learning\. Appendix[A\.2](https://arxiv.org/html/2609.28876#A1.SS2)provides full details of this selection\.

### 4\.2Information Corpus

#### Corpus construction\.

Forecast\-Dojorequires broad historical evidence whose availability can be reconstructed at each forecast date\. We build the corpus from CC\-News\([Nagel, 2016](https://arxiv.org/html/2609.28876#bib.bib5)\), which provides large\-scale news coverage together with crawl timestamps that support historical reconstruction\. We process CC\-News archives from March 2025 through May 2026 using a quality pipeline adapted from FineWeb\([Penedo et al\., 2024](https://arxiv.org/html/2609.28876#bib.bib18)\), including text cleaning, language identification, repetition and document\-quality filtering, and near\-duplicate removal\. We suppress duplicate URLs, identical titles within a seven\-day window, and near\-verbatim body matches to reduce repeated coverage\. After filtering, the corpus contains approximately 18\.8M articles, which we embed with Qwen3\-Embedding\-8B\([Zhang et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib4)\)and index with FAISS\([Douze et al\., 2026](https://arxiv.org/html/2609.28876#bib.bib16)\)\.

#### Temporal integrity\.

Reconstructing historical evidence also requires reliable article timestamps: assigning an article an incorrectly early date could expose future information to the agent\. We assign article dates using structured publication or modification metadata, with the CC\-News crawl timestamp as a fallback\. Timestamps are extracted through fixed\-priority metadata cascades, including schema\.orgdatePublishedanddateModified\([Guha et al\., 2016](https://arxiv.org/html/2609.28876#bib.bib13);[Schema\.org,](https://arxiv.org/html/2609.28876#bib.bib14);[Google Search Central,](https://arxiv.org/html/2609.28876#bib.bib15)\)\. At each forecast step, retrieval filters articles by the UTC day of the assigned timestamp before ranking\. Event construction additionally screens for content\-level outcome leakage\. Appendix[A\.1](https://arxiv.org/html/2609.28876#A1.SS1)details the timestamp sources and extraction procedures\.

### 4\.3Temporal Task Construction

#### Forecast\-date selection\.

A forecast step could naively be created for every day of an event’s lifetime, but this would cause long\-lived events to contribute disproportionately many training and evaluation samples\. We therefore use a sublinear schedule: after enforcing a two\-day buffer before the recorded close, an event withnncandidate days receivesT=clamp⁡\(round⁡\(n\),3,10\)T=\\operatorname\{clamp\}\\left\(\\operatorname\{round\}\(\\sqrt\{n\}\),3,10\\right\)forecast steps\. We partition the event history intoTTtemporal bins and select one date from each to maintain coverage across its lifetime\. Within each bin, we prioritize two signals to select dates most worth forecasting\. \(1\)*Market\-belief movement*: we favor dates with larger changes in the market belief, indicating that newly available information has materially shifted the market’s assessment of the event\. \(2\)*New evidence*: we favor dates with greater news publication activity, indicating periods when more external information has become available to the forecaster\. The selection policy is configurable; the above procedure is the default used in our experiments\. After leakage filtering, evaluation events must retain at least three forecast dates\. Appendix[A\.3](https://arxiv.org/html/2609.28876#A1.SS3)provides the full weighting and selection procedure\.

#### Temporal train–evaluation split\.

We split complete events, rather than individual forecast steps, into non\-overlapping temporal windows\. Training events must both start and close within\[2025\-06, 2026\-03\), while evaluation events must both start and close within\[2026\-03, 2026\-06\); events crossing either boundary are excluded\. This keeps every forecast episode entirely within one split and prevents the same event from appearing in both training and evaluation\. Table[2](https://arxiv.org/html/2609.28876#S2.T2)summarizes the resulting splits and forecast\-step counts\.

### 4\.4Agent Interaction

#### Tool interface\.

At each forecast step, the agent can access and process evidence via three tools, with temporally restricted access to the information corpus available by the forecast date,ℐ≤τt\\mathcal\{I\}\_\{\\leq\\tau\_\{t\}\}:

- •Searchretrieves the top\-kkrelevant articles for an agent\-generated query, returning article identifiers, titles, publication dates, retrieval scores, and short snippets, viasearch\(query, top\_k\)\.
- •Readreturns the full text of an article retrieved bySearch, viascrape\(article\_id\)\.
- •Computeexecutes model\-generated code for numerical analysis, aggregation, base\-rate estimation, or simulation, viapython\(code\)\.

The agent may interleave reasoning with repeated tool calls before submitting its forecast\. Each tool output is added to the within\-step interaction historyHtH\_\{t\}and becomes available for subsequent reasoning and tool use within the same forecast step\.

#### Memory transfer\.

Each forecast step is a fresh model interaction with access to the corpus up to the current forecast date,ℐ≤τt\\mathcal\{I\}\_\{\\leq\\tau\_\{t\}\}\. In memory\-free mode, no information from earlier steps is carried forward\. In memory\-on mode, the previous belief notebookMtM\_\{t\}is additionally inserted into the next prompt, allowing the agent to revise its prior assessment as new evidence becomes available\. Previous conversation turns, reasoning traces, and tool observations are discarded, makingMtM\_\{t\}the only explicitly transferred information\. The belief\-notebook format is provided in Figure[6](https://arxiv.org/html/2609.28876#A3.F6)\.

### 4\.5Evaluation and Learning

#### Forecast evaluation\.

Each forecast step produces a probability distributionpQ,tp\_\{Q,t\}, which can be evaluated against the realized outcomeYYafter the interaction\. These probabilities support diverse step\-level metrics, including proper scoring rules such as Brier scores\([Glenn and others, 1950](https://arxiv.org/html/2609.28876#bib.bib12)\), top\-1 accuracy, calibration metrics such as expected calibration error \(ECE\)\([Guo et al\., 2017](https://arxiv.org/html/2609.28876#bib.bib19)\), and market\-relative measures such as Information\-α\\alpha, defined in Section[5](https://arxiv.org/html/2609.28876#S5)\. Because each event yields a sequence of forecasts, Forecast\-Dojo also supports trajectory\-level analyses of how beliefs evolve across forecast steps\. The environment does not prescribe a single metric\. Section[B\.3](https://arxiv.org/html/2609.28876#A2.SS3)specifies the metrics used in our experiments\.

#### Learning from interactions\.

With trajectory logging enabled, each completed forecast step retains its available within\-step interaction historyHtH\_\{t\}, including model messages and tool interactions, and, in memory\-on mode, the belief notebookMtM\_\{t\}\. Once the realized outcome is available, outcome\-based feedback can be attached to the same interaction\. Evaluation events use these outputs for benchmarking, while training events provide trajectories and feedback that can be consumed by diverse learning methods\. The same forecasting interaction supports both evaluation and learning without changing the task or tool interface\. We demonstrate this capability with supervised fine\-tuning in Section[5\.5](https://arxiv.org/html/2609.28876#S5.SS5)\.

## 5Experiments

Table 3:Main Results\.Main values average recorded forecasts; smaller±\\pmvalues show standard deviations of the four rollout means\. Unusable forecasts are replaced by uniform distributions for all metrics; accuracy uses fractional ties and is reported in percent\. Info\-α\\alpharequires an available market probability\.Boldandunderliningindicate the best and second\-best model in each column\. Superscripts†\\dagger,‡\\ddagger, and§\\lx@sectionsignflag configurations with more than 5% unusable forecasts\. Uniform and market forecasts serve as contextual references\.### 5\.1Experimental Setup

We evaluate 12 proprietary and open\-weight models on 230 held\-out events comprising 797 event–date pairs, with four rollouts per pair\. For every model, either its reported knowledge or training\-data cutoff or, when unavailable, its checkpoint release date predates the evaluation period \(see Table[9](https://arxiv.org/html/2609.28876#A2.T9)\)\. We compare three settings on the same scheduled tasks\. In the*no\-tools*setting \(Figure[4](https://arxiv.org/html/2609.28876#A3.F4)\), the model predicts in a single call without research tools\. In the*memory\-free*setting \(Figure[5](https://arxiv.org/html/2609.28876#A3.F5)\), it uses date\-restricted search, article retrieval, and Python, starting from a fresh context at each forecast date\. The*memory\-on*setting additionally passes the agent’s previous belief notebook to the next date\. Each rollout maintains its own notebook; earlier conversations and raw tool outputs are not carried forward\. Model configurations and tool budgets are given in Appendix[B\.2](https://arxiv.org/html/2609.28876#A2.SS2)\.

We report multiclass Brier score, top\-label accuracy, and Information\-α\\alpha\. Brier measures error in the predicted probability distribution, while accuracy measures whether the highest\-probability outcome is correct\. Information\-α\\alphacompares the agent’s and market’s log scores on the realized outcome; positive values favor the agent\. Uniform forecasts and historical market probabilities serve as references\. The market may use information outside the news archive, and its probabilities are never shown to the agents\.

Figure 2:Forecast improvements track newly available evidence\.\(a\) Brier at early, middle, and late stages, defined by thirds of relative forecast\-step position; each event receives equal weight\. Shading shows the range of model means\. \(b\) Consecutive\-step Brier change by new\-evidence quartile\. Each model’s index is computed from the other available models’ notebooks\. \(c\) Evidence captureκ\\kappaversus first\-to\-last Brier improvement with memory on\. The correlation is computed across 12 models; the no\-tools cross is a reference, with no defined capture value\. Scores use the main table’s exact label matching and uniform fallback; \(a,b\) show 95% event\-bootstrap CIs\. Definitions and scoring sensitivity: Appendix[B\.7](https://arxiv.org/html/2609.28876#A2.SS7)\.Table[3](https://arxiv.org/html/2609.28876#S5.T3)reports averages over recorded forecasts and standard deviations across four rollouts\. An unusable recorded forecast is replaced by a uniform distribution over itsKKoffered outcomes, giving Brier1−1/K1\-1/Kand fractional\-tie accuracy1/K1/K\. For the longitudinal analysis, we average within each event before averaging across events and estimate 95% confidence intervals by resampling events\. Metric definitions are provided in Appendix[B\.3](https://arxiv.org/html/2609.28876#A2.SS3)\.

### 5\.2Overall Forecasting Performance

#### Research tools improve forecasting across model families\.

Table[3](https://arxiv.org/html/2609.28876#S5.T3)compares the three settings on the same forecasting tasks\. Relative to no tools, memory\-free research lowers Brier and improves accuracy for all 12 models\. For GPT\-5\.5, Brier decreases from 0\.698 to 0\.564, while accuracy increases from 43\.85% to 57\.69%\. The gains also extend to open\-weight models: DeepSeek\-V3\.2 improves from 0\.805 to 0\.655 in Brier and from 34\.34% to 47\.73% in accuracy\. Because each date starts from a fresh context, these gains show the value of dated information even without persistent memory\.

#### Proprietary models achieve lower Brier scores with tools\.

In both tool\-enabled settings, all five proprietary models have lower mean Brier scores than every open\-weight model evaluated\. GPT\-5\.6 Sol achieves the lowest Brier in both the memory\-free \(0\.554\) and memory\-on \(0\.546\) settings\. GLM\-5 is the strongest open\-weight model by Brier in both settings, scoring 0\.625 and 0\.632, respectively\. The corresponding gaps to the best proprietary model are 0\.071 and 0\.086\. Tool access helps both groups but leaves a gap in forecast quality\.

#### Higher accuracy does not always imply lower Brier\.

GPT\-5\.5 has the highest memory\-free accuracy \(57\.69%\), whereas GPT\-5\.6 Sol is less accurate \(56\.36%\) but has lower Brier \(0\.554 versus 0\.564\) and higher Information\-α\\alpha\(−0\.138\-0\.138versus−0\.186\-0\.186\)\. The most accurate model is therefore not the strongest by either probability score\. With uniform fallback, all 12 models beat the uniform reference in Brier under both tool\-enabled settings\. This scoring convention does not imply successful output: for example, 19\.45% of gpt\-oss\-120b’s memory\-free reports are unusable\. Appendix[B\.4](https://arxiv.org/html/2609.28876#A2.SS4)reports failure rates separately\.

Figure 3:Memory lowers research cost, mostly after the first forecast\.\(a\) Mean estimated API cost per forecast \(USD\) at each forecast step, averaged over the five proprietary models\. Only 67 of the 230 events have a fourth step and 25 have a fifth, so later points average over fewer events\. \(b\) Percentage change of memory\-on relative to memory\-free execution in cost, input tokens, and tool calls at later steps \(t≥2t\\geq 2\), grouped by the new\-evidence quartiles of Figure[2](https://arxiv.org/html/2609.28876#S5.F2)\(b\)\. \(c\) The same change for each resource, split into the first step and later steps that carry a notebook\. No notebook exists at the first step; first\-step differences can reflect prompt and sampling variation\. Negative values indicate reductions\. Tokens and calls use all 12 models; intervals are 95% event\-bootstrap CIs\. Appendix[B\.5](https://arxiv.org/html/2609.28876#A2.SS5)defines and reports per\-model resource use\.
#### Memory yields model\-dependent changes in forecast quality\.

Adding a belief notebook lowers mean Brier for six models but raises it for the other six\. For example, GPT\-5\.5 changes from 0\.564 to 0\.571, whereas GPT\-5\.6 Sol improves from 0\.554 to 0\.546\. The latter improvement coincides with a reduction in unusable reports from 4\.49% to 0\.53%\. When both settings produce a usable forecast, however, the paired memory\-on\-minus\-memory\-free Brier difference for GPT\-5\.6 Sol is only\+0\.0010\+0\.0010\. This comparison shows why an improvement in the overall score need not imply better probability estimates among successful forecasts\. We therefore distinguish memory’s effects on forecast quality and output reliability from its effect on research cost, examined in Section[5\.4](https://arxiv.org/html/2609.28876#S5.SS4)\.

#### The market reference remains ahead across scoring rules\.

The market achieves Brier 0\.498 and accuracy 64\.55%, compared with the best model scores of 0\.546 and 57\.69%\. These correspond to gaps of 0\.048 in Brier and 6\.85 percentage points in accuracy\. Information\-α\\alphais also negative for every model in all three settings, indicating lower average log scores than the market on forecasts where this metric is available\. The gap therefore extends beyond top\-label accuracy\. The market may incorporate information outside the agents’ archive, so it serves as an external reference rather than an information\-matched baseline\. We next examine how agent forecasts improve over an event and how these gains relate to newly available evidence\.

### 5\.3Forecasting over Time

#### Forecasts improve over time with research tools\.

Figure[2](https://arxiv.org/html/2609.28876#S5.F2)\(a\) compares forecasts at the early, middle, and late stages of each event\. Mean Brier decreases from 0\.670 to 0\.606 with memory\-free research and from 0\.671 to 0\.612 with memory\-on research\. In contrast, the no\-tools baseline remains nearly unchanged near 0\.769\. All 12 models improve from the first to the last stage with memory\-free research\. The market improves more sharply, from 0\.597 to 0\.438\. These results show that agents benefit from access to dated information as events unfold, even without retaining their previous research\.

#### Larger improvements coincide with more new evidence\.

We estimate the amount of new evidence between consecutive forecast dates using dated entries in belief notebooks\. For each evaluated model, the*new\-evidence index*,\|Δ​Et\|\|\\Delta E\_\{t\}\|, is computed from the other available models’ notebooks\. A higher index indicates that these models recorded more evidence dated within the interval\. For memory\-free agents, the mean step\-to\-step Brier change is−0\.007\-0\.007in the lowest evidence quartile and−0\.071\-0\.071in the highest \(Figure[2](https://arxiv.org/html/2609.28876#S5.F2)\(b\)\)\. Memory\-on agents show a similar pattern, whereas no\-tools forecasts change little in every quartile\. Forecast improvements are therefore concentrated at steps with more newly recorded evidence\. Appendix[B\.6](https://arxiv.org/html/2609.28876#A2.SS6)gives the index definition and aggregation procedure\.

#### Models that capture more evidence tend to improve more\.

We measure*evidence capture*,κ\\kappa, by how much newly dated evidence a model records relative to the other models;κ=1\\kappa=1corresponds to their average recording rate\. Across the 12 models,κ\\kapparanges from 0\.36 to 1\.74 and correlates with first\-to\-last Brier improvement in the memory\-on setting \(Spearmanρ=0\.74\\rho=0\.74; Figure[2](https://arxiv.org/html/2609.28876#S5.F2)\(c\)\)\. The corresponding correlations for search and article\-retrieval counts are weaker, at 0\.31 and 0\.49\. Thus, recording newly relevant evidence is more closely associated with forecast improvement than the number of research calls\. The definition and limitations of this notebook\-based measure are given in Appendix[B\.7](https://arxiv.org/html/2609.28876#A2.SS7)\.

### 5\.4Cost Analysis

#### Savings are larger after the initial forecast\.

In Figure[3](https://arxiv.org/html/2609.28876#S5.F3)\(a\), the two tool\-using settings cost about the same at the first forecast, and memory\-on execution becomes cheaper from the second forecast onward\. The saving is largest when little new evidence has appeared since the previous forecast and shrinks as more becomes available \(Figure[3](https://arxiv.org/html/2609.28876#S5.F3)\(b\)\)\. Figure[3](https://arxiv.org/html/2609.28876#S5.F3)\(c\) shows that resource use is similar at the first step, where neither setting has a notebook, while later steps show the largest reductions in input tokens, tool iterations, and research calls\. This pattern is consistent with agents reusing earlier findings instead of repeating the same research at each date\. Given the mixed effects on Brier in Section[5\.2](https://arxiv.org/html/2609.28876#S5.SS2), lower research cost is memory’s clearest benefit in the evaluated protocols\.

#### Memory reduces the cost of repeated forecasting\.

For the five proprietary models, memory\-on execution costs less per forecast than memory\-free execution in every case\. The reduction ranges from 8% for Claude Opus 4\.6 to 33% for GPT\-5\.5, with a median of 24% \(Table[12](https://arxiv.org/html/2609.28876#A2.T12)\)\. GPT\-5\.5 costs a third less, $3\.90 instead of $5\.85 per forecast, while its paired Brier difference is\+0\.007\+0\.007with a 95% confidence interval that includes zero\. These costs cover retained forecast records and exclude discarded retry attempts\.

Table 4:Supervised fine\-tuning on Forecast\-Dojo trajectories\.Qwen3\-30B\-A3B\-Thinking\-2507 is evaluated on 3,188 forecasts from 230 held\-out events\. Paired SFT−\-base differences \(Δ\\Delta\) are computed before rounding; 95% CIs resample events\. All scores use uniform fallback for unusable forecasts\.

### 5\.5Training Proof of Concept

We next test whetherForecast\-Dojointeractions can directly support agent training\. We collect trajectories from Qwen3\-235B\-A22B\-Thinking\-2507\([Qwen Team, 2025](https://arxiv.org/html/2609.28876#bib.bib27)\)onForecast\-Dojo’s train set, and supervised fine\-tune Qwen3\-30B\-A3B\-Thinking\-2507\([Qwen Team, 2025](https://arxiv.org/html/2609.28876#bib.bib27)\)for three epochs over the full assistant trajectory\. As Qwen3\-30B\-A3B\-Thinking\-2507 has no officially reported knowledge cutoff and was released in July 2025, we use a subset of theForecast\-Dojotraining split spanning August 2025 to February 2026, comprising 1,028 unique events and 3,965 forecast steps\.

On 230 held\-out events, the fine\-tuned student outperforms the base across probabilistic and categorical metrics \(Table[4](https://arxiv.org/html/2609.28876#S5.T4)\)\. Overall Brier decreases from 0\.924 to 0\.749, a paired improvement of−0\.176\-0\.176\(95% CI\[−0\.214,−0\.139\]\[\-0\.214,\-0\.139\]\), with consistent gains on both binary \(→0\.5390\.704\\\!\\rightarrow\\\!0\.539\) and multi\-choice \(→0\.7970\.975\\\!\\rightarrow\\\!0\.797\) forecasts\. Log loss falls from 2\.74 to 1\.89, while accuracy rises from 34\.8% to 42\.8% \(\+8\.1\+8\.1points, 95% CI\[\+5\.1,\+11\.2\]\[\+5\.1,\+11\.2\]\)\. These gains come with increased tool use \(1\.4 to 3\.7 calls per forecast\), while parser acceptance remains near\-perfect at 99\.5%\. Together, these results provide a proof of concept thatForecast\-Dojotrajectories can support training agents with improved held\-out forecasting performance\.

## 6Conclusion

Forecast\-Dojo replays resolved prediction\-market events as fixed sequences of historical forecast states, enabling agents to be compared under the same information conditions and evaluated throughout an event’s evolution\. Access to research tools improves Brier and accuracy for all 12 models, with larger step\-wise gains when more new evidence becomes available\. Models that record more newly dated evidence also tend to improve more over an episode\. Yet all evaluated agents remain behind historical market forecasts, while persistent memory reduces research cost more consistently than it improves forecast quality\. Beyond evaluation,Forecast\-Dojoproduces outcome\-grounded interaction trajectories that can support a broad range of learning algorithms, from supervised fine\-tuning to trajectory\-level and reward\-based optimization\.

## References

- Anthropic \(2026a\)AnthropicClaude opus 4\.6 system card\.Note:[https://www\.anthropic\.com/system\-cards](https://www.anthropic.com/system-cards)Claude Opus 4\.6Cited by:[§A\.2](https://arxiv.org/html/2609.28876#A1.SS2.SSS0.Px3.p3.3)\.
- Anthropic \(2026b\)AnthropicIntroducing claude opus 4\.8\.Note:[https://www\.anthropic\.com/news/claude\-opus\-4\-8](https://www.anthropic.com/news/claude-opus-4-8)Accessed 2026\-09\-22Cited by:[§A\.2](https://arxiv.org/html/2609.28876#A1.SS2.SSS0.Px3.p2.1)\.
- Chandaket al\.\(2025\)N\. Chandak, S\. Goel, A\. Prabhu, M\. Hardt, and J\. GeipingScaling open\-ended reasoning to predict the future\.CoRRabs/2512\.25070\.External Links:[Link](https://doi.org/10.48550/arXiv.2512.25070),[Document](https://dx.doi.org/10.48550/ARXIV.2512.25070),2512\.25070Cited by:[§1](https://arxiv.org/html/2609.28876#S1.p2.1),[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px3.p1.1)\.
- Chenget al\.\(2026\)P\. Cheng, J\. Liu, and Y\. LongPolyBench: benchmarking llm forecasting and trading capabilities on live prediction market data\.arXiv preprint arXiv:2604\.14199\.Cited by:[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px1.p1.1)\.
- Chiet al\.\(2026\)Y\. Chi, E\. Chamoun, Z\. Ding, and A\. VlachosWorldReasoner: evaluating whether language model agents forecast events with valid reasoning\.arXiv preprint arXiv:2606\.11816\.Cited by:[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px2.p1.1)\.
- Douzeet al\.\(2026\)M\. Douze, A\. Guzhva, C\. Deng, J\. Johnson, G\. Szilvasy, P\. Mazaré, M\. Lomeli, L\. Hosseini, and H\. JégouThe faiss library\.IEEE Trans\. Big Data12\(2\),pp\. 346–361\.External Links:[Link](https://doi.org/10.1109/TBDATA.2025.3618474),[Document](https://dx.doi.org/10.1109/TBDATA.2025.3618474)Cited by:[§4\.2](https://arxiv.org/html/2609.28876#S4.SS2.SSS0.Px1.p1.1)\.
- Glennet al\.\(1950\)W\. B\. Glennet al\.Verification of forecasts expressed in terms of probability\.Monthly weather review78\(1\),pp\. 1–3\.Cited by:[§4\.5](https://arxiv.org/html/2609.28876#S4.SS5.SSS0.Px1.p1.1)\.
- Goelet al\.\(2026\)S\. Goel, N\. Chandak, A\. Arun, A\. Prabhu, S\. Staab, M\. Hardt, M\. Andriushchenko, and J\. GeipingFutureSim: replaying world events to evaluate adaptive agents\.arXiv preprint arXiv:2605\.15188\.Cited by:[Table 1](https://arxiv.org/html/2609.28876#S1.T1.6.10.1.1.1),[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px2.p1.1)\.
- \[9\]Google Search CentralArticle structured data\.Note:[https://developers\.google\.com/search/docs/appearance/structured\-data/article](https://developers.google.com/search/docs/appearance/structured-data/article)Accessed 2026Cited by:[§4\.2](https://arxiv.org/html/2609.28876#S4.SS2.SSS0.Px2.p1.1)\.
- Guhaet al\.\(2016\)R\. V\. Guha, D\. Brickley, and S\. MacbethSchema\.org: evolution of structured data on the web\.Commun\. ACM59\(2\),pp\. 44–51\.External Links:[Link](https://doi.org/10.1145/2844544),[Document](https://dx.doi.org/10.1145/2844544)Cited by:[§4\.2](https://arxiv.org/html/2609.28876#S4.SS2.SSS0.Px2.p1.1)\.
- Guoet al\.\(2017\)C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. WeinbergerOn calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6\-11 August 2017,D\. Precup and Y\. W\. Teh \(Eds\.\),Proceedings of Machine Learning Research, Vol\.70,pp\. 1321–1330\.External Links:[Link](http://proceedings.mlr.press/v70/guo17a.html)Cited by:[§4\.5](https://arxiv.org/html/2609.28876#S4.SS5.SSS0.Px1.p1.1)\.
- Halawiet al\.\(2024\)D\. Halawi, F\. Zhang, Y\. Chen, and J\. SteinhardtApproaching human\-level forecasting with language models\.InAdvances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper/_files/paper/2024/hash/5a5acfd0876c940d81619c1dc60e7748-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2609.28876#S1.p1.1),[§1](https://arxiv.org/html/2609.28876#S1.p2.1),[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px3.p1.1)\.
- Jeenet al\.\(2026\)S\. Jeen, M\. Aitchison, M\. A\. H\. Clark, T\. Shevlane, and B\. DayReaching the frontier of ai forecasting with reinforcement learning\.InForecasting as a New Frontier of Intelligence,Cited by:[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px3.p1.1)\.
- Jinet al\.\(2021\)W\. Jin, R\. Khanna, S\. Kim, D\. Lee, F\. Morstatter, A\. Galstyan, and X\. RenForecastQA: A question answering challenge for event forecasting with temporal text data\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, \(Volume 1: Long Papers\), Virtual Event, August 1\-6, 2021,C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),pp\. 4636–4650\.External Links:[Link](https://doi.org/10.18653/v1/2021.acl-long.357),[Document](https://dx.doi.org/10.18653/V1/2021.ACL-LONG.357)Cited by:[Table 1](https://arxiv.org/html/2609.28876#S1.T1.6.2.1.1.1),[§1](https://arxiv.org/html/2609.28876#S1.p2.1),[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px1.p1.1)\.
- Jorenet al\.\(2025\)H\. Joren, J\. Zhang, C\. Ferng, D\. Juan, A\. Taly, and C\. RashtchianSufficient context: A new lens on retrieval augmented generation systems\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=Jjr2Odj8DJ)Cited by:[§A\.2](https://arxiv.org/html/2609.28876#A1.SS2.SSS0.Px3.p2.1),[§4\.1](https://arxiv.org/html/2609.28876#S4.SS1.p2.1)\.
- Joulinet al\.\(2016\)A\. Joulin, E\. Grave, P\. Bojanowski, M\. Douze, H\. Jégou, and T\. MikolovFastText\.zip: compressing text classification models\.CoRRabs/1612\.03651\.External Links:[Link](http://arxiv.org/abs/1612.03651),1612\.03651Cited by:[§A\.1](https://arxiv.org/html/2609.28876#A1.SS1.SSS0.Px2.p1.1)\.
- Joulinet al\.\(2017\)A\. Joulin, E\. Grave, P\. Bojanowski, and T\. MikolovBag of tricks for efficient text classification\.InProceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3\-7, 2017, Volume 2: Short Papers,M\. Lapata, P\. Blunsom, and A\. Koller \(Eds\.\),pp\. 427–431\.External Links:[Link](https://doi.org/10.18653/v1/e17-2068),[Document](https://dx.doi.org/10.18653/V1/E17-2068)Cited by:[§A\.1](https://arxiv.org/html/2609.28876#A1.SS1.SSS0.Px2.p1.1)\.
- Kargeret al\.\(2025\)E\. Karger, H\. Bastani, Y\. Chen, Z\. Jacobs, D\. Halawi, F\. Zhang, and P\. TetlockForecastBench: A dynamic benchmark of AI forecasting capabilities\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=lfPkGWXLLf)Cited by:[Table 1](https://arxiv.org/html/2609.28876#S1.T1.6.5.1.1.1),[§1](https://arxiv.org/html/2609.28876#S1.p1.1),[§1](https://arxiv.org/html/2609.28876#S1.p2.1),[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px1.p1.1)\.
- Levy \(2026\)A\. A\. LevyReinforcement learning for llm\-based event forecasting\.arXiv preprint arXiv:2606\.15917\.Cited by:[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px3.p1.1)\.
- Liptayet al\.\(2026\)T\. Liptay, D\. Schwarz, R\. Poyiadzi, J\. Wildman, and N\. I\. BosseEvaluating strategic reasoning in forecasting agents\.arXiv preprint arXiv:2604\.26106\.Cited by:[Table 1](https://arxiv.org/html/2609.28876#S1.T1.6.9.1.1.1),[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px2.p1.1)\.
- Murphy \(2026\)K\. MurphyAgentic forecasting using sequential bayesian updating of linguistic beliefs\.arXiv preprint arXiv:2604\.18576\.Cited by:[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px2.p1.1)\.
- Nagel \(2016\)S\. NagelCommon crawl news dataset\.Common Crawl\.External Links:[Link](https://data.commoncrawl.org/crawl-data/CC-NEWS/index.html)Cited by:[§A\.1](https://arxiv.org/html/2609.28876#A1.SS1.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2609.28876#S4.SS2.SSS0.Px1.p1.1)\.
- Palekaet al\.\(2025\)D\. Paleka, A\. Pallavi Sudhir, A\. Alvarez, V\. Bhat, A\. Shen, E\. Wang, and F\. TramèrConsistency checks for language model forecasters\.arXiv preprint arXiv:2412\.18544\.Cited by:[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px2.p1.1)\.
- Penedoet al\.\(2024\)G\. Penedo, H\. Kydlícek, L\. B\. Allal, A\. Lozhkov, M\. Mitchell, C\. A\. Raffel, L\. von Werra, and T\. WolfThe fineweb datasets: decanting the web for the finest text data at scale\.InAdvances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper/_files/paper/2024/hash/370df50ccfdf8bde18f8f9c2d9151bda-Abstract-Datasets/_and/_Benchmarks/_Track.html)Cited by:[§A\.1](https://arxiv.org/html/2609.28876#A1.SS1.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2609.28876#S4.SS2.SSS0.Px1.p1.1)\.
- Qianget al\.\(2025\)R\. Qiang, Y\. Zhuang, Y\. Li, D\. Sagar V K, R\. Zhang, C\. Li, I\. S\. Wong, S\. Yang, P\. Liang, C\. Zhang, and B\. DaiMLE\-dojo: interactive environments for empowering llm agents in machine learning engineering\.arXiv preprint arXiv:2505\.07782\.Cited by:[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px3.p1.1)\.
- Qwen Team \(2025\)Qwen TeamQwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§5\.5](https://arxiv.org/html/2609.28876#S5.SS5.p1.1)\.
- Raeet al\.\(2021\)J\. W\. Rae, S\. Borgeaud, T\. Cai, K\. Millican, J\. Hoffmann, H\. F\. Song, J\. Aslanides, S\. Henderson, R\. Ring, S\. Young, E\. Rutherford, T\. Hennigan, J\. Menick, A\. Cassirer, R\. Powell, G\. van den Driessche, L\. A\. Hendricks, M\. Rauh, P\. Huang, A\. Glaese, J\. Welbl, S\. Dathathri, S\. Huang, J\. Uesato, J\. Mellor, I\. Higgins, A\. Creswell, N\. McAleese, A\. Wu, E\. Elsen, S\. M\. Jayakumar, E\. Buchatskaya, D\. Budden, E\. Sutherland, K\. Simonyan, M\. Paganini, L\. Sifre, L\. Martens, X\. L\. Li, A\. Kuncoro, A\. Nematzadeh, E\. Gribovskaya, D\. Donato, A\. Lazaridou, A\. Mensch, J\. Lespiau, M\. Tsimpoukelli, N\. Grigorev, D\. Fritz, T\. Sottiaux, M\. Pajarskas, T\. Pohlen, Z\. Gong, D\. Toyama, C\. de Masson d’Autume, Y\. Li, T\. Terzi, V\. Mikulik, I\. Babuschkin, A\. Clark, D\. de Las Casas, A\. Guy, C\. Jones, J\. Bradbury, M\. J\. Johnson, B\. A\. Hechtman, L\. Weidinger, I\. Gabriel, W\. Isaac, E\. Lockhart, S\. Osindero, L\. Rimell, C\. Dyer, O\. Vinyals, K\. Ayoub, J\. Stanway, L\. Bennett, D\. Hassabis, K\. Kavukcuoglu, and G\. IrvingScaling language models: methods, analysis & insights from training gopher\.CoRRabs/2112\.11446\.External Links:[Link](https://arxiv.org/abs/2112.11446),2112\.11446Cited by:[§A\.1](https://arxiv.org/html/2609.28876#A1.SS1.SSS0.Px2.p1.1)\.
- Raffelet al\.\(2020\)C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. LiuExploring the limits of transfer learning with a unified text\-to\-text transformer\.J\. Mach\. Learn\. Res\.21,pp\. 140:1–140:67\.External Links:[Link](https://jmlr.org/papers/v21/20-074.html)Cited by:[§A\.1](https://arxiv.org/html/2609.28876#A1.SS1.SSS0.Px2.p1.1)\.
- Robertson and Zaragoza \(2009\)S\. E\. Robertson and H\. ZaragozaThe probabilistic relevance framework: BM25 and beyond\.Found\. Trends Inf\. Retr\.3\(4\),pp\. 333–389\.External Links:[Link](https://doi.org/10.1561/1500000019),[Document](https://dx.doi.org/10.1561/1500000019)Cited by:[§A\.2](https://arxiv.org/html/2609.28876#A1.SS2.SSS0.Px3.p3.2)\.
- \[30\]Schema\.orgdatePublished\.Note:[https://schema\.org/datePublished](https://schema.org/datePublished)Accessed 2026Cited by:[§4\.2](https://arxiv.org/html/2609.28876#S4.SS2.SSS0.Px2.p1.1)\.
- Seed \(2026\)B\. SeedFutureX\-pro: extending future prediction to high\-value vertical domains\.CoRRabs/2601\.12259\.External Links:[Link](https://doi.org/10.48550/arXiv.2601.12259),[Document](https://dx.doi.org/10.48550/ARXIV.2601.12259),2601\.12259Cited by:[§1](https://arxiv.org/html/2609.28876#S1.p1.1)\.
- Wildmanet al\.\(2025\)J\. Wildman, N\. I\. Bosse, D\. Hnyk, P\. Mühlbacher, F\. Hambly, J\. Evans, D\. Schwarz, L\. Phillips,et al\.Bench to the future: a pastcasting benchmark for forecasting agents\.arXiv preprint arXiv:2506\.21558\.Cited by:[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px2.p1.1)\.
- Yanget al\.\(2025\)Q\. Yang, S\. Mahns, S\. Li, A\. Gu, J\. Wu, and H\. XuLLM\-as\-a\-prophet: understanding predictive intelligence with prophet arena\.arXiv preprint arXiv:2510\.17638\.Cited by:[Table 1](https://arxiv.org/html/2609.28876#S1.T1.6.7.1.1.1),[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px1.p1.1)\.
- Yeet al\.\(2024\)C\. Ye, Z\. Hu, Y\. Deng, Z\. Huang, M\. D\. Ma, Y\. Zhu, and W\. WangMirai: evaluating llm agents for event forecasting\.arXiv preprint arXiv:2407\.01231\.Cited by:[Table 1](https://arxiv.org/html/2609.28876#S1.T1.6.4.1.1.1),[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px2.p1.1)\.
- Yuanet al\.\(2025\)Z\. Yuan, Z\. Ding, and A\. VlachosDo language models update their forecasts with new information?\.arXiv preprint arXiv:2509\.23936\.Cited by:[Table 1](https://arxiv.org/html/2609.28876#S1.T1.6.8.1.1.1),[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px2.p1.1)\.
- Zenget al\.\(2025\)Z\. Zeng, J\. Liu, S\. Chen, T\. He, Y\. Liao, J\. Wang, Z\. Wang, Y\. Yang, L\. Yin, M\. Yin, Z\. Zhu, T\. Cai, Z\. Chen, J\. Chen, Y\. Du, X\. Gao, J\. Guo, L\. Hu, J\. Jiao, X\. Li, J\. Liu, S\. Ni, Z\. Wen, G\. Zhang, K\. Zhang, X\. Zhou, J\. H\. Blanchet, X\. Qiu, M\. Wang, and W\. HuangFutureX: an advanced live benchmark for LLM agents in future prediction\.CoRRabs/2508\.11987\.External Links:[Link](https://doi.org/10.48550/arXiv.2508.11987),[Document](https://dx.doi.org/10.48550/ARXIV.2508.11987),2508\.11987Cited by:[Table 1](https://arxiv.org/html/2609.28876#S1.T1.6.6.1.1.1),[§1](https://arxiv.org/html/2609.28876#S1.p1.1),[§1](https://arxiv.org/html/2609.28876#S1.p2.1),[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026\)J\. Zhang, G\. Liu, O\. Johansson, H\. Yitayew, K\. Ohly, and G\. LiPrediction arena: benchmarking AI models on real\-world prediction markets\.CoRRabs/2604\.07355\.External Links:[Link](https://doi.org/10.48550/arXiv.2604.07355),[Document](https://dx.doi.org/10.48550/ARXIV.2604.07355),2604\.07355Cited by:[§1](https://arxiv.org/html/2609.28876#S1.p2.1)\.
- Zhanget al\.\(2025\)Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin, F\. Huang, and J\. ZhouQwen3 embedding: advancing text embedding and reranking through foundation models\.CoRRabs/2506\.05176\.External Links:[Link](https://doi.org/10.48550/arXiv.2506.05176),[Document](https://dx.doi.org/10.48550/ARXIV.2506.05176),2506\.05176Cited by:[§A\.1](https://arxiv.org/html/2609.28876#A1.SS1.SSS0.Px5.p1.1),[§4\.2](https://arxiv.org/html/2609.28876#S4.SS2.SSS0.Px1.p1.1)\.
- Zhaoet al\.\(2026\)X\. Zhao, R\. Zhang, L\. Ye, R\. Cao, P\. Xie, and S\. ChavaCan llms take the pulse of the economy? A real\-time evaluation of LLM nowcasts on macroeconomic indicators\.CoRRabs/2608\.30110\.External Links:[Link](https://doi.org/10.48550/arXiv.2608.30110),[Document](https://dx.doi.org/10.48550/ARXIV.2608.30110),2608\.30110Cited by:[§1](https://arxiv.org/html/2609.28876#S1.p2.1),[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px1.p1.1)\.
- Zouet al\.\(2022\)A\. Zou, T\. Xiao, R\. Jia, J\. Kwon, M\. Mazeika, R\. Li, D\. Song, J\. Steinhardt, O\. Evans, and D\. HendrycksForecasting future world events with neural networks\.InAdvances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track,External Links:2206\.15474Cited by:[Table 1](https://arxiv.org/html/2609.28876#S1.T1.6.3.1.1.1),[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.28876#S2.SS0.SSS0.Px3.p1.1)\.

Appendix forForecast\-Dojo

## Appendix ADataset and Benchmark Details

We describe in detail how we construct our information corpus \(§[A\.1](https://arxiv.org/html/2609.28876#A1.SS1)\), collect and filter forecasting events \(§[A\.2](https://arxiv.org/html/2609.28876#A1.SS2)\), and select forecast dates for each task \(§[A\.3](https://arxiv.org/html/2609.28876#A1.SS3)\)\.

### A\.1Information Corpus

#### Corpus source\.

We construct the retrieval corpus from 15 monthly Common Crawl CC\-NEWS releases\([Nagel, 2016](https://arxiv.org/html/2609.28876#bib.bib5)\), spanning March 2025 through May 2026 and comprising 7,163 WARC files\. For each eligible HTML response, we extract the article title, summary, body text, URL metadata, crawl timestamp, and structured publication metadata\.

#### Text and quality filtering\.

Our filtering pipeline is summarized in Table[5](https://arxiv.org/html/2609.28876#A1.T5)\. The first two stages normalize the extracted text and retain English\-language documents using fastText\([Joulin et al\., 2017](https://arxiv.org/html/2609.28876#bib.bib22);[Joulin et al\., 2016](https://arxiv.org/html/2609.28876#bib.bib23)\), matching the language of the benchmark questions\. The remaining stages adapt quality filters from Gopher\([Rae et al\., 2021](https://arxiv.org/html/2609.28876#bib.bib20)\), C4\([Raffel et al\., 2020](https://arxiv.org/html/2609.28876#bib.bib21)\), and FineWeb\([Penedo et al\., 2024](https://arxiv.org/html/2609.28876#bib.bib18)\)\.

Table 5:Corpus filtering pipeline\. The aggregate post\-quality count is reported in Table[8\(a\)](https://arxiv.org/html/2609.28876#A1.T8.st1)\.
#### Publication dates and temporal integrity\.

For each documentdd, we extract one publication timestamptp​\(d\)t\_\{p\}\(d\)and one modification timestamptm​\(d\)t\_\{m\}\(d\)using the fixed\-priority metadata cascades in Table[6](https://arxiv.org/html/2609.28876#A1.T6)\. Within each cascade, we use the first available timestamp\. We discard timestamps whose year falls outside\[2010,2030\]\[2010,2030\]and use the document crawl timestamptc​\(d\)t\_\{c\}\(d\)as a fallback\. The retrieval system uses the UTC calendar day oft⁡\(d\)t\(d\)\. Across the final corpus,58\.6%58\.6\\%of articles use a publication timestamp,28\.3%28\.3\\%use a modification timestamp, and13\.1%13\.1\\%fall back to the crawl timestamp\. We treat this timestamp assignment as the primary temporal boundary for retrieval\. As an additional safeguard, event construction also applies a content\-level leakage screen \(Section[A\.2](https://arxiv.org/html/2609.28876#A1.SS2)\) to detect cases where the article content is inconsistent with the assigned temporal boundary, for example when a page is updated without a corresponding change in its structured metadata\.

Table 6:Priority order for extracting structured publication and modification timestamps\.
#### Duplicate suppression\.

We remove repeated coverage using canonical\-URL matching, same\-title suppression within a seven\-day window, and MinHash\-LSH over lower\-cased word 5\-grams\. The body\-level stage uses 128 MinHash values arranged as eight bands of 16 hashes and retains the earliest\-published representative of each connected duplicate component\. Because the title stage does not require body\-level equivalence, we refer to this procedure as*duplicate suppression*rather than semantic deduplication\.

#### Indexing and temporal restriction\.

Each retained article is embedded with Qwen3\-Embedding\-8B\([Zhang et al\., 2025](https://arxiv.org/html/2609.28876#bib.bib4)\)using its title, summary, and body\. The 4096\-dimensional representation isL2L\_\{2\}\-normalized and stored in exact inner\-product indices sharded by publication month\. Rows within each shard are sorted by publication date\. At forecast dateτt\\tau\_\{t\}, search is restricted to

ℐ≤τt=\{d∈ℐ:2025​\-​03​\-​01≤date⁡\(d\)≤τt\}\.\\mathcal\{I\}\_\{\\leq\\tau\_\{t\}\}=\\\{d\\in\\mathcal\{I\}:2025\\text\{\-\}03\\text\{\-\}01\\leq\\operatorname\{date\}\(d\)\\leq\\tau\_\{t\}\\\}\.The date restriction is applied before similarity ranking, and full\-article access independently rechecks the same temporal constraint\.

### A\.2Event Collection and Filtering

#### Event collection and market beliefs\.

We construct forecasting questions from resolved Polymarket events\. A single non\-negative\-risk market defines a binary event, while a negative\-risk bundle of at least two markets defines a mutually exclusive multi\-option event; other layouts are excluded\. For market legkk, lettkstartt\_\{k\}^\{\\mathrm\{start\}\}andtkcloset\_\{k\}^\{\\mathrm\{close\}\}denote its recorded start and close times\. We define the complete tradeable span as

\[tstart,tclose\]=\[mink⁡tkstart,maxk⁡tkclose\]\.\[t\_\{\\mathrm\{start\}\},t\_\{\\mathrm\{close\}\}\]=\\left\[\\min\_\{k\}t\_\{k\}^\{\\mathrm\{start\}\},\\max\_\{k\}t\_\{k\}^\{\\mathrm\{close\}\}\\right\]\.Ground\-truth labels are derived from terminal YES prices: binary markets resolve to YES at≥0\.99\\geq 0\.99and to NO at≤0\.01\\leq 0\.01, while multi\-option events require exactly one YES\-resolved option\. We reconstruct the daily market beliefmQ,t∈ΔK−1m\_\{Q,t\}\\in\\Delta^\{K\-1\}from the UTC\-day mean of CLOB YES\-price observations\. For a multi\-option event with priced optionsAtA\_\{t\},

mQ,t​\(k\)=xt,k∑j∈Atxt,j,k∈At,m\_\{Q,t\}\(k\)=\\frac\{x\_\{t,k\}\}\{\\sum\_\{j\\in A\_\{t\}\}x\_\{t,j\}\},\\qquad k\\in A\_\{t\},with missing options left undefined rather than imputed\.

#### Market\-side filtering\.

We first filter events for a meaningful forecasting horizon and a reliable, nontrivial market\-belief trajectory\. Events must span at least eight UTC calendar days\. We then define a day as*faithful*when the trade count and share volume on the market leg corresponding to the realized outcome exceed the type\-specific thresholds in Table[7](https://arxiv.org/html/2609.28876#A1.T7)\. LetρQ\\rho\_\{Q\}denote the fraction of faithful days andSQS\_\{Q\}the longest consecutive run of unfaithful days\.

Among events passing this activity filter, we measure learning signal from the market probability assigned to the realized outcome,mQ,t​\(Y\)m\_\{Q,t\}\(Y\)\. LetℱQ\\mathcal\{F\}\_\{Q\}denote faithful days with an observed belief, and letΔt=mQ,t​\(Y\)−mQ,t−1​\(Y\)\\Delta\_\{t\}=m\_\{Q,t\}\(Y\)\-m\_\{Q,t\-1\}\(Y\)for adjacent calendar days with valid faithful observations\. We define

aQ=1\|ℱQ\|∑t∈ℱQ−logclip\(mQ,t\(Y\),10−6,1−10−6\),VQ=∑t\|Δt\|,CQ=∑tΔt∑t\|Δt\|\.a\_\{Q\}=\\frac\{1\}\{\|\\mathcal\{F\}\_\{Q\}\|\}\\sum\_\{t\\in\\mathcal\{F\}\_\{Q\}\}\-\\log\\operatorname\{clip\}\\left\(m\_\{Q,t\}\(Y\),10^\{\-6\},1\-10^\{\-6\}\\right\),\\quad V\_\{Q\}=\\sum\_\{t\}\|\\Delta\_\{t\}\|,\\quad C\_\{Q\}=\\frac\{\\sum\_\{t\}\\Delta\_\{t\}\}\{\\sum\_\{t\}\|\\Delta\_\{t\}\|\}\.These remove events that are already nearly certain or have little meaningful temporal variation\.

Table 7:Market\-side event\-selection criteria\.
#### Corpus\-side filtering\.

Market\-side filtering identifies events with usable forecasting trajectories, but does not establish whether the frozen information environment is suitable for forecasting\. We therefore apply a complementary corpus\-side screen that evaluates both the sufficiency of pre\-resolution evidence and potential information leakage\.

Evidence sufficiency\.Following the sufficient\-context framework of[Joren et al\. \(2025\)](https://arxiv.org/html/2609.28876#bib.bib17), we assess whether the retrieved information contains enough evidence to support an informed forecast\. We adapt this protocol to historical forecasting by first decomposing each event into targeted information needs\. For each event surviving the market\-side filters, we use Claude Opus 4\.8 with maximum reasoning effort\([Anthropic, 2026b](https://arxiv.org/html/2609.28876#bib.bib24)\)to generate 4–7 predictive\-evidence queries and 4–7 background/reference\-class queries\. Query generation explicitly targets information available before resolution and forbids searches for the realized outcome or post\-resolution reports\. See the full decomposition prompt in Figure[7](https://arxiv.org/html/2609.28876#A3.F7)\.

Sufficiency is evaluated once at the latest eligible forecasting state,

τ⋆=tclose−2​days\.\\tau^\{\\star\}=t\_\{\\mathrm\{close\}\}\-2\\ \\text\{days\}\.The two\-day buffer provides a conservative pre\-resolution cutoff: close\-day reporting may already reveal resolution\-relevant information, while UTC normalization and timezone differences can blur the adjacent calendar\-day boundary\. For each query, we retrieve dense \(Qwen3\-Embedding\-8B\) and BM25\([Robertson and Zaragoza, 2009](https://arxiv.org/html/2609.28876#bib.bib25)\)top\-10 results restricted to\[2025​\-​03​\-​01,τ⋆\]\[2025\\text\{\-\}03\\text\{\-\}01,\\tau^\{\\star\}\]and combine them using reciprocal\-rank fusion,

RRF⁡\(d\)=∑q∑r∈\{dense,BM25\}𝟏\[d∈Lq,r\]60\+rankq,r⁡\(d\)\.\\operatorname\{RRF\}\(d\)=\\sum\_\{q\}\\sum\_\{r\\in\\\{\\mathrm\{dense\},\\mathrm\{BM25\}\\\}\}\\frac\{\\mathbf\{1\}\[d\\in L\_\{q,r\}\]\}\{60\+\\operatorname\{rank\}\_\{q,r\}\(d\)\}\.The 15 highest\-ranked unique documents form the evidence setEQE\_\{Q\}\. We then use Claude Opus 4\.6\([Anthropic, 2026a](https://arxiv.org/html/2609.28876#bib.bib26)\)to classify the available evidence as \{Sufficient,Partial,Insufficient\}\. Only events receiving aSufficientverdict are eligible for the final benchmark\.

Leakage control\.The same judge additionally screens the retrieved evidence for potential outcome leakage\. This content\-level check complements the metadata\-level temporal restriction on retrieval and provides an additional safeguard when the visible content of a page may not be fully reflected by its assigned publication timestamp\. The judge does not receive the structured realized outcome, final market prices, or crowd trajectory\.

LetvQv\_\{Q\}denote the evidence\-sufficiency verdict andℓQ\\ell\_\{Q\}the leakage flag\. An event is retained iff

vQ=SufficientandℓQ=false\.v\_\{Q\}=\\textsc\{Sufficient\}\\qquad\\text\{and\}\\qquad\\ell\_\{Q\}=\\mathrm\{false\}\.See Figure[8](https://arxiv.org/html/2609.28876#A3.F8)for the judge prompt and Table[8](https://arxiv.org/html/2609.28876#A1.T8)for event filtering funnels\.

\(a\)Corpus construction
\(b\)Event selection

Table 8:Construction waterfalls for the deployed news corpus and forecasting events\. Event counts use the final June–March training and March–June evaluation splits\.

### A\.3Forecast\-Step Construction

#### Temporal split\.

We split at the event level using the complete tradeable lifetime\. An event enters training ifftstart≥2025​\-​06​\-​01andtclose<2026​\-​03​\-​01,t\_\{\\mathrm\{start\}\}\\geq 2025\\text\{\-\}06\\text\{\-\}01\\quad\\text\{and\}\\quad t\_\{\\mathrm\{close\}\}<2026\\text\{\-\}03\\text\{\-\}01,and enters evaluation ifftstart≥2026​\-​03​\-​01andtclose<2026​\-​06​\-​01\.t\_\{\\mathrm\{start\}\}\\geq 2026\\text\{\-\}03\\text\{\-\}01\\quad\\text\{and\}\\quad t\_\{\\mathrm\{close\}\}<2026\\text\{\-\}06\\text\{\-\}01\.Events that cross the March 1 boundary are excluded from both splits\.

#### Number of forecast steps\.

For eventQQ, letDQD\_\{Q\}be its contiguous UTC daily grid\. We reserve a two\-day buffer before the recorded close date and define

EQ=\{d∈DQ:date⁡\(tclose\)−d≥2\},nQ=\|EQ\|\.E\_\{Q\}=\\\{d\\in D\_\{Q\}:\\operatorname\{date\}\(t\_\{\\mathrm\{close\}\}\)\-d\\geq 2\\\},\\qquad n\_\{Q\}=\|E\_\{Q\}\|\.The number of forecast steps is

TQ=clamp⁡\(round⁡nQ,3,10\)\.T\_\{Q\}=\\operatorname\{clamp\}\\left\(\\operatorname\{round\}\\sqrt\{n\_\{Q\}\},3,10\\right\)\.

#### Date selection\.

Within the eligible grid, market movement is

bt=\{\|mQ,t​\(YES\)−mQ,t−1​\(YES\)\|,binary,12​∑k\|mQ,t​\(k\)−mQ,t−1​\(k\)\|,multi\-option,b\_\{t\}=\\begin\{cases\}\|m\_\{Q,t\}\(\\mathrm\{YES\}\)\-m\_\{Q,t\-1\}\(\\mathrm\{YES\}\)\|,&\\text\{binary\},\\\\\[5\.69054pt\] \\frac\{1\}\{2\}\\sum\_\{k\}\|m\_\{Q,t\}\(k\)\-m\_\{Q,t\-1\}\(k\)\|,&\\text\{multi\-option\},\\end\{cases\}with zero assigned when an adjacent belief is unavailable\. Corpus activityctc\_\{t\}is the total number of indexed news articles published on calendar daytt\. It is a global news\-volume signal rather than an event\-specific relevance score\. After independently max\-normalizing both signals, eligible dateeje\_\{j\}receives

sj=0\.7​b~ej\+0\.3​c~ej\+10−3​jnQ−1\.s\_\{j\}=0\.7\\,\\tilde\{b\}\_\{e\_\{j\}\}\+0\.3\\,\\tilde\{c\}\_\{e\_\{j\}\}\+10^\{\-3\}\\frac\{j\}\{n\_\{Q\}\-1\}\.We partition the eligible sequence intoTQT\_\{Q\}contiguous equal\-count bins and choose the highest\-scoring date from each bin\. This preserves temporal coverage while favoring dates with larger belief changes or greater overall news activity\.

## Appendix BExperiment Details

### B\.1Knowledge Cutoffs and Checkpoint Dates

Table[9](https://arxiv.org/html/2609.28876#A2.T9)lists each model’s reported knowledge cutoff or checkpoint date\. All dates precede the evaluation period, which begins on March 1, 2026\.

Table 9:Knowledge and release dates of evaluated models\. We report an official knowledge or training\-data cutoff when available; otherwise, we use the public checkpoint release date\.Note\.“Release” denotes the public checkpoint release date when no official knowledge cutoff is reported\.

### B\.2Models and Execution Protocols

Table[10](https://arxiv.org/html/2609.28876#A2.T10)lists the request\-side settings of the 12 default model configurations\. All runs use the same 230 held\-out events, four rollouts per date\. Tool runs share a budget of 120 tool iterations and 400 calls per forecast step; recency reranking is off\. No\-tool runs remove the tools and the corpus and raise the output cap to 65,536 tokens, since the whole forecast is then produced in a single call\.

Table 10:Model Evaluation Configurations\.Max output tokens is the cap per model call and includes reasoning tokens\.defaultmarks a field that was not sent, so the provider’s default applied\. Tool budgets are per forecast step and identical in memory\-free and memory\-on mode; no\-tool runs disable all tools\.- aThe model reasons by default; the request carries no effort parameter\.

The training use case in Table[4](https://arxiv.org/html/2609.28876#S5.T4)uses memory\-free execution, the same no\-belief system prompt, Python, and a budget of 80 tool iterations and 200 tool calls per forecast for both models\. Both use temperature 0\.6, a 32,768\-token output cap, top\-p=1\.0p=1\.0, and no top\-kkrestriction\. Log loss uses natural logarithms with a probability floor of10−310^\{\-3\}\.

### B\.3Evaluation Metrics

#### Brier score\.

Leti=\(e,t,r\)i=\(e,t,r\)index an event, forecast date, and rollout, and letYiY\_\{i\}denote the realized outcome\. For a usable probability report, negative entries are clipped to zero and the remaining positive mass is normalized\. Labels are matched exactly\. Let𝒰i\\mathcal\{U\}\_\{i\}be the union of the reported labels and the truth label; unreported labels receive zero probability, while unsupported reported labels retain their probability mass\. The multiclass Brier score is

Bi=∑c∈𝒰i\(pi​c−𝟏\{c=Yi\}\)2\.B\_\{i\}=\\sum\_\{c\\in\\mathcal\{U\}\_\{i\}\}\\left\(p\_\{ic\}\-\\mathbf\{1\}\\\{c=Y\_\{i\}\\\}\\right\)^\{2\}\.We use the standard\[0,2\]\[0,2\]scale, without normalization by the number of outcomes or an additional binary\-event factor\. A uniform forecast overKiK\_\{i\}offered outcomes has Brier score1−1/Ki1\-1/K\_\{i\}\.

#### Accuracy\.

Let𝒯i=arg⁡maxc⁡pi​c\\mathcal\{T\}\_\{i\}=\\arg\\max\_\{c\}p\_\{ic\}denote the set of outcomes assigned maximal probability\. We use fractional\-tie accuracy,

ai=𝟏\{Yi∈𝒯i\}\|𝒯i\|\.a\_\{i\}=\\frac\{\\mathbf\{1\}\\\{Y\_\{i\}\\in\\mathcal\{T\}\_\{i\}\\\}\}\{\|\\mathcal\{T\}\_\{i\}\|\}\.Thus, a correct unique top prediction receives accuracy11, while ties split credit uniformly among tied outcomes\. A uniform forecast overKiK\_\{i\}outcomes therefore has accuracy1/Ki1/K\_\{i\}\.

#### Information\-α\\alpha\.

We measure improvement over the contemporaneous market belief using

αi=log⁡max⁡\{pi​\(Yi\),ϵ\}−log⁡max⁡\{pmarket,i​\(Yi\),ϵ\},ϵ=10−3,\\alpha\_\{i\}=\\log\\max\\\{p\_\{i\}\(Y\_\{i\}\),\\epsilon\\\}\-\\log\\max\\\{p\_\{\\mathrm\{market\},i\}\(Y\_\{i\}\),\\epsilon\\\},\\qquad\\epsilon=10^\{\-3\},with natural logarithms; positive values favor the agent\. We compute this difference for every recorded forecast with an available scalar market probability, including uniform fallback for unusable reports\. Scalar market probabilities may have different availability from the reconstructed full market vectors used in longitudinal Brier analyses\. Neither realized outcomes nor market probabilities are provided to the forecasting agent\.

### B\.4Aggregation and Failures

Not every scheduled forecast produces a valid probability report\. An output is unusable if the parser rejects it or it has no positive finite probability mass\. We replace a recorded unusable report by the uniform distribution over itsKiK\_\{i\}offered outcomes for all scores:Bi=1−1/KiB\_\{i\}=1\-1/K\_\{i\},ai=1/Kia\_\{i\}=1/K\_\{i\}, and log losslog⁡Ki\\log K\_\{i\}\. Main\-table means average recorded forecasts, not missing records\. GPT\-5\.6 Sol and Nemotron 3 Super have 3,184 and 3,170 recorded memory\-on forecasts, respectively; all other configurations have 3,188\. Table[11](https://arxiv.org/html/2609.28876#A2.T11)additionally counts missing records as failures, using all 3,188 scheduled forecasts as its denominator\.

The same fallback applies to the main table, longitudinal analyses, and SFT evaluation\. Information\-α\\alphais omitted only when the scalar market probability is unavailable\. Imputation does not change whether an output is counted as a failure\.

Table 11:Failure rate of scheduled forecasts by condition \(%\) for the 12 default model configurations\.A failure is a missing output, parser rejection, or no positive finite probability mass\. Each cell covers 3,188 scheduled forecasts \(797 event–dates, four rollouts\)\. Recorded unusable outputs are scored as uniform forecasts; missing records are excluded from score means\.
### B\.5Inference Cost and Tool Usage

Table[12](https://arxiv.org/html/2609.28876#A2.T12)summarizes average inference cost and tool usage per recorded forecast\. Dollar columns show proprietary\-model provider estimates; NR marks open\-weight models whose absolute prices are not compared across serving arrangements\. Within\-model cost reductions in the main text use the five proprietary models with provider\-reported prices\. Research calls sum the logged search, scrape, and Python calls, including tool errors\. In the memory\-on setting, the first forecast of each episode starts without prior memory, while later forecasts receive the notebook produced at the preceding step\. These statistics are intended as descriptive resource estimates rather than hardware\-normalized efficiency comparisons\. Changes in Figure[3](https://arxiv.org/html/2609.28876#S5.F3)use ratios of summed resources over matched model–event–date–rollout records; intervals use 4,000 bootstrap resamples of events\.

Table 12:Inference cost and tool usage by model\.Average provider\-estimated cost and research calls per recorded forecast\. Research calls sum search, scrape, and Python calls, including errors\. No\-tools forecasts make no research calls\.- aNR: not reported; the open\-weight models are served without a comparable per\-forecast price\.

### B\.6Measuring Newly Available Evidence

#### Longitudinal scores and support\.

Figure[2](https://arxiv.org/html/2609.28876#S5.F2)uses the same exact label matching and uniform fallback as the main table\. Unrecorded forecasts are omitted\. Relative step position is\(t−1\)/\(Te−1\)\(t\-1\)/\(T\_\{e\}\-1\)on the retained date sequence\. Panel \(a\) retains dates with available scalar market probabilities for agents and complete market vectors for the market, then events represented in all three thirds \(223 agent events; 224 market events\)\. Event means receive equal weight; 95% percentile intervals use 4,000 event\-clustered bootstrap draws with seed 0\.

#### Newly dated evidence\.

We estimate how much new event\-specific evidence becomes available between two consecutive forecast dates using the evidence ledgers in the memory\-on notebooks\. For modelkk, rolloutrr, eventee, and stept≥2t\\geq 2, let

nk,r\(e,t\)=∑a∈ledger⁡\(Mk,r​\(e,t\)\)\{τe,t−1<d\(a\)≤τe,t\},n\_\{k,r\}\(e,t\)=\\sum\_\{a\\in\\operatorname\{ledger\}\(M\_\{k,r\}\(e,t\)\)\}\\mathbf\{1\}\\\!\\left\\\{\\tau\_\{e,t\-1\}<d\(a\)\\leq\\tau\_\{e,t\}\\right\\\},whered⁡\(a\)d\(a\)is the recordeddate\_observedof ledger entryaa\. Thus, an entry counts only when its recorded evidence date falls between the previous and current forecast dates\. Earlier evidence discovered late is not counted, and carried entries are not counted again at later steps\. We count both active and superseded entries and exclude entries with invalid or future dates\.

We first average across the available notebook chains of each model,

ownk⁡\(e,t\)=1\|ℛk​\(e,t\)\|​∑r∈ℛk​\(e,t\)nk,r​\(e,t\)\.\\operatorname\{own\}\_\{k\}\(e,t\)=\\frac\{1\}\{\|\\mathcal\{R\}\_\{k\}\(e,t\)\|\}\\sum\_\{r\\in\\mathcal\{R\}\_\{k\}\(e,t\)\}n\_\{k,r\}\(e,t\)\.

#### Leave\-one\-model\-out evidence availability\.

To estimate how much new evidence was available at an event\-step without using the evaluated model’s own notebook, we average the corresponding counts over the set𝒫k​\(e,t\)\\mathcal\{P\}\_\{k\}\(e,t\)of other models with available notebook counts:

Ak​\(e,t\)=1\|𝒫k​\(e,t\)\|​∑j∈𝒫k​\(e,t\)ownj⁡\(e,t\)\.A\_\{k\}\(e,t\)=\\frac\{1\}\{\|\\mathcal\{P\}\_\{k\}\(e,t\)\|\}\\sum\_\{j\\in\\mathcal\{P\}\_\{k\}\(e,t\)\}\\operatorname\{own\}\_\{j\}\(e,t\)\.We useAk​\(e,t\)A\_\{k\}\(e,t\)as the evidence\-availability index for modelkk, including when analyzing its memory\-free forecasts\. This leave\-one\-model\-out construction avoids directly coupling a model’s forecast change to its own recording behavior\. The peer count is normally 11; 33 indexed model–date rows at three event–date states have 10 peers\.

For consecutive forecasts, we define

Δ​Bk,rA​\(e,t\)=Bk,rA​\(e,t\)−Bk,rA​\(e,t−1\),\\Delta B^\{A\}\_\{k,r\}\(e,t\)=B^\{A\}\_\{k,r\}\(e,t\)\-B^\{A\}\_\{k,r\}\(e,t\-1\),where negative values indicate improvement\. The schedule has 567 consecutive\-date transitions; the index has quartile cut points 1\.4924, 2\.4545, and 3\.9848\. Panel \(b\) retains adjacent recorded forecasts with available scalar market probabilities and groups transitions by quartiles ofAk​\(e,t\)A\_\{k\}\(e,t\)and averages first within events and then equally across events; confidence intervals use event\-clustered bootstrap resampling\.

The index should be interpreted as a proxy for newly available, event\-relevant evidence rather than an exhaustive corpus count\. It depends on what the other agents record in their notebooks and may miss relevant evidence that no model retrieves\.

### B\.7Evidence Capture and Forecast Improvement

#### Evidence capture\.

The availability index above measures how much new evidence appears to be available at an event\-step\. To measure how much of that evidence each model captures, we compare the model’s own newly dated entries with the leave\-one\-model\-out availability index\.

For event–step pairs𝒮k\\mathcal\{S\}\_\{k\}with own and peer counts, the no\-intercept slope is

κk=∑\(e,t\)∈𝒮kownk⁡\(e,t\)​Ak​\(e,t\)∑\(e,t\)∈𝒮kAk​\(e,t\)2\.\\kappa\_\{k\}=\\frac\{\\sum\_\{\(e,t\)\\in\\mathcal\{S\}\_\{k\}\}\\operatorname\{own\}\_\{k\}\(e,t\)A\_\{k\}\(e,t\)\}\{\\sum\_\{\(e,t\)\\in\\mathcal\{S\}\_\{k\}\}A\_\{k\}\(e,t\)^\{2\}\}\.A value ofκk=1\\kappa\_\{k\}=1means that the model records newly dated evidence at the peer\-average rate; values above or below one indicate higher or lower capture, respectively\. For example,κk=1\.5\\kappa\_\{k\}=1\.5corresponds to a fitted recording rate 50% above the peer average\. Importantly,κk\\kappa\_\{k\}is a relative rate, not the fraction of an exhaustive evidence set that the model retrieves\.

#### Episode gain\.

We measure how much a model improves over an episode using its scheduled first and last retained memory\-on forecasts, pairing recorded endpoints within each rollout:

Gk=1\|ℰk\|​∑e∈ℰk1\|ℛkend​\(e\)\|​∑r∈ℛkend​\(e\)\[Bk,ron​\(e,1\)−Bk,ron​\(e,Te\)\]\.G\_\{k\}=\\frac\{1\}\{\|\\mathcal\{E\}\_\{k\}\|\}\\sum\_\{e\\in\\mathcal\{E\}\_\{k\}\}\\frac\{1\}\{\|\\mathcal\{R\}\_\{k\}^\{\\mathrm\{end\}\}\(e\)\|\}\\sum\_\{r\\in\\mathcal\{R\}\_\{k\}^\{\\mathrm\{end\}\}\(e\)\}\\left\[B^\{\\mathrm\{on\}\}\_\{k,r\}\(e,1\)\-B^\{\\mathrm\{on\}\}\_\{k,r\}\(e,T\_\{e\}\)\\right\]\.Positive values indicate improvement\. We first average endpoint pairs across rollouts within each event and then average equally across events\. Panel \(c\) comparesκk\\kappa\_\{k\}withGkG\_\{k\}across the 12 models and reports their Spearman correlation\.

#### Interpretation\.

Under our uniform\-fallback scoring rule, evidence capture is associated with episode\-level improvement \(Spearmanρ=0\.74\\rho=0\.74\), more than search or scrape counts \(0\.310\.31and0\.490\.49\)\. As a sensitivity check, typographic label normalization givesρ=0\.60\\rho=0\.60with uniform fallback\.

These relationships are correlational\. Notebook entries are self\-reported, their recorded dates need not always be correct, and evidence missed by all models is invisible to the measure\. We therefore interpretκk\\kappa\_\{k\}as a diagnostic of relative evidence capture, not as a complete or causal measure of information acquisition\.

## Appendix CPrompts

No\-Tools Forecasting Agent PromptRoleYou are an expert forecasting agent\. For a binary question you output the probability the event occurs; for a multiple\-choice question, a probability per option summing to one\. Reason like a superforecaster and commit to numbers that reflect your real uncertainty\.You are scored by a proper scoring rule\. Both overconfidence and reflexive hedging cost you\. Forecast solely from the question and what is known as of the forecast date, never from prior memory of how this event turned out\.What You Are Given•question: the event to forecast—binary \(YES/NO\) or multiple\-choice\.•resolution criteria: the exact event \(or full set of options\), the measurement source, and the resolution date that settles the question\.•forecast date: treat this as today\. Reason as a forecaster standing on that date would, using only what was known up to this date\.Forecasting Strategy1\. Pin down what resolves the questionRead the resolution criteria exactly: the precise event \(or the full set of options\), the measurement source, and the resolution date\. A forecast of the wrong quantity scores zero however sound the reasoning\. Note the forecast date and how much time remains\.2\. Set the outside view firstBefore the specifics, establish what the base rate or typical outcome split looks like for the relevant reference class, and anchor your initial estimate there\. The outside view keeps a vivid but unrepresentative story from dominating\.3\. Weigh the evidence systematically•Decompose the question into the few sub\-questions that would most move your estimate, and work through each\. Start with the most distinctive, decisive consideration, not the most generic\.•For each sub\-question, lay out the relevant facts: the actors and their incentives, the rules and schedule that govern the event, the historical pattern for comparable cases, and the most recent developments as of the forecast date\. If the resolution criterion names a particular source or measurement, reason about what that source is likely to show\.•Be explicit about how solid each piece of evidence is—a well\-established fact, a plausible inference, or a guess—and weight it accordingly\. Do not invent specifics you do not have\.4\. Compute what can be computedA forecast question is a judgment problem with computable parts—settle those parts with explicit arithmetic rather than by feel\. Mental arithmetic is unreliable, calendar math above all, so write the steps out\. The computations that recur:•Extrapolation: take the recent rate of a running total and project it to the resolution date—required pace vs\. current pace often settles a threshold question\.•Base rates: turn historical counts into a probability for your window \(kkevents innnyears,ttyears left→1−exp\(−kt/n\)\\rightarrow 1\-\\exp\(\-kt/n\)\)\.•Probability algebra: chained conditionals, at\-least\-one\-of\-kk, scenario weighting, Bayes updates—combine the numbers step by step, never in your head\.•Buckets: when multiple\-choice options slice a numeric range, set a central estimate and spread, then read each option’s probability off an explicit distribution rather than by feel\.Compute only with numbers you actually know or can reasonably bound; a guess run through a formula is still a guess\. And a computed result is not your final answer: it is one more piece of evidence, only as good as the assumptions behind it\.5\. Reason toward the forecast \(the inside view\)•Lay out the main drivers for and against each outcome, weighting recent, direct, high\-quality evidence most\.•Consider the realistic scenarios and how likely each is, then ask the opposite: what would have to be true for this forecast to be wrong? This checks confirmation bias\.•Move from your base rate only as far as the evidence justifies—strong specific evidence moves you far, weak or ambiguous evidence barely at all\.6\. Calibrate and commit•Be granular—distinguish0\.60\.6from0\.70\.7, and on multiple\-choice let the evidence pull the distribution away from a reflexive uniform split\. This precision is where forecasting skill lives\.•Never assign00or11to an outcome that is not truly impossible or certain; a confident error is the costliest mistake under the scoring rule\. Multiple\-choice probabilities must sum to11\.•You must commit\. “Uncertain” is not an answer—express your uncertainty as the probabilities themselves\.Output FormatConclude with your forecast as a strict\-JSON dictionary inside<answer\>\.\.\.</answer\>—keys in double quotes, values numeric, and no trailing commas\. This is the only format the parser accepts\.Binary\.Keys are exactly"YES"and"NO", and values sum to11\. Format example \(numbers are illustrative\):<answer\>\{"YES": 0\.63, "NO": 0\.37\}</answer\>Multiple\-choice\.Include one key per option, with the label copied verbatim from the question \(including spaces, punctuation, and casing\) and double\-quoted; values sum to11\. Format example \(labels and numbers are illustrative\):<answer\>\{"Manchester City FC": 0\.33,"Draw \(Leeds United FC vs\. Manchester City FC\)": 0\.17,"Leeds United FC": 0\.50\}</answer\>

Figure 4:No\-tools forecasting\-agent system prompt\.Memory\-Free Forecasting Agent PromptRoleYou are an expert forecasting agent\. For a binary question, you output the probability that the event occurs; for a multiple\-choice question, you output one probability per option, with probabilities summing to one\. Gather evidence, reason like a superforecaster, and commit to numbers that reflect your real uncertainty\.You are scored by a proper scoring rule\. Both overconfidence and reflexive hedging cost you\. Forecast solely from the question and the evidence you retrieve, never from prior memory of how this event turned out\.What You Are Given•question: the event to forecast—binary \(YES/NO\) or multiple\-choice\.•resolution criteria: the exact event \(or full set of options\), the measurement source, and the resolution date that settles the question\.•forecast date: treat this as today\. Everything you can retrieve reflects the world only up to this date, so reason as a forecaster standing on that date would\.Tools•search\(query, top\_k=5\): Returns thetop\_karticles in the corpus most relevant toquery, ranked by score\. Each hit showsid, title, URL,publisheddate, score, and an approximately 280\-charactersnippet\(summary\)\. Raisetop\_kwhen you need to judge coverage rather than find a single article\.•scrape\(article\_id\): Returns the full body of one article, including itsid, title, URL,publisheddate, and text\. Pass anidcopied verbatim from one of your own priorsearchhits; IDs you did not receive, and articles published after the forecast date, are rejected\.•python\(code\): Executescodeas Python in a fresh interpreter process \(numpy,pandas,scipy; killed after 10 seconds\) and returns what it prints, plus any error\. Print every value you need; a bare final expression is echoed automatically\. Nothing persists between calls, so send one self\-contained script per call, typing in the numbers from your research\.Forecasting Strategy1\. Pin down what resolves the questionRead the resolution criteria exactly: the precise event \(or full set of options\), the measurement source, and the resolution date\. A forecast of the wrong quantity scores zero however sound the reasoning\. Note the forecast date and how much time remains\.2\. Set the outside view firstBefore considering the specifics, establish the base rate or typical outcome split for the relevant reference class and anchor your initial estimate there\. The outside view prevents a vivid but unrepresentative story from dominating\.3\. Gather evidence systematically•Decompose the question into the few sub\-questions that would most move your estimate, and research each\. Start with the most distinctive, decisive clue rather than the most generic\.•Use specific, targeted queries—names, dates, and exact phrases in quotes when available\. If the resolution criterion names a particular source \(e\.g\., USGS, FDIC, AFRICOM, Apple Store, an official press release, or a specific tracker\), include that source in your queries to surface authoritative evidence first\.•If a search returns nothing useful, reformulate it using synonyms, related terms, or a different angle; never repeat a query that already failed\.Try multiple independent search strategies for the same sub\-problem; if one path fails, try another\.•Usescrapeliberally\.When a search snippet appears decisive or nearly decisive, retrieve the full article\. The detail that settles the answer is often in the full text\. Corroborate decisive facts across more than one article\.4\. Compute what can be computedA forecasting question is a judgment problem with computable parts\. Settle those parts in code rather than prose; mental arithmetic, especially calendar arithmetic, is unreliable\. Common computations include:•Extrapolation: fit the recent rate of a running total and project it to the resolution date\. Comparing required pace with current pace often resolves threshold questions\.•Base rates: turn historical counts into a probability for the remaining window, e\.g\.,1−exp\(−kt/n\)1\-\\exp\(\-kt/n\)forkkevents observed overnnyears withttyears remaining\.•Probability algebra: compute chained conditionals, at\-least\-one\-of\-kkprobabilities, scenario mixtures, and Bayesian updates explicitly rather than mentally\.•Simulation: when uncertain quantities interact \(e\.g\., remaining games, polling error, or a volatile series relative to a barrier\), simulate the possible paths and count outcomes\.•Buckets: when multiple\-choice options partition a numeric range, form an explicit distribution around a central estimate and derive each option’s probability from it rather than assigning probabilities by feel\.Compute only with numbers actually obtained from your research\. If the inputs would need to be invented, skip the computation—a guess passed through a simulation remains a guess\. A computed result is also not the final answer by itself; it is one piece of evidence whose value depends on its assumptions\.5\. Reason toward the forecast \(the inside view\)•Lay out the main drivers for and against each outcome, weighting recent, direct, and high\-quality evidence most heavily\.•Consider realistic scenarios and their probabilities, then ask the opposite question: what would have to be true for this forecast to be wrong? Use this to check confirmation bias\.•Move away from the base rate only as far as the evidence justifies\. Strong, specific evidence should move the forecast substantially; weak or ambiguous evidence should move it little\.6\. Calibrate and commit•Be granular: distinguish0\.60\.6from0\.70\.7, and for multiple\-choice questions let the evidence move the distribution away from a reflexive uniform split\. This precision is where forecasting skill lives\.•Never assign00or11to an outcome that is not truly impossible or certain; a confident error is the costliest mistake under the scoring rule\. Multiple\-choice probabilities must sum to one\.•You must commit\. “Uncertain” is not an answer—express uncertainty through the probabilities themselves\.Output FormatConclude with the forecast as a strict JSON dictionary inside<answer\>\.\.\.</answer\>\. Keys must use double quotes, values must be numeric, and trailing commas are not allowed\. This is the only format accepted by the parser\.Binary\.Keys are exactly"YES"and"NO", and their values must sum to one\. The following numbers are illustrative:<answer\>\{"YES": 0\.63, "NO": 0\.37\}</answer\>Multiple\-choice\.Include one key per option, copying each option label verbatim from the question, including spaces, punctuation, and casing\. Values must sum to one\. The following labels and probabilities are illustrative:<answer\>\{"Manchester City FC": 0\.33, "Draw \(Leeds United FC vs\. Manchester City FC\)": 0\.17, "Leeds United FC": 0\.50\}</answer\>

Figure 5:Memory\-free forecasting\-agent system prompt\.Memory\-On Forecasting Agent PromptRoleYou are an expert forecasting agent\. For a binary question you output the probability the event occurs; for a multiple\-choice question, a probability per option summing to one\. Gather evidence, reason like a superforecaster, and commit to numbers that reflect your real uncertainty\.You are scored by a proper scoring rule\. Both overconfidence and reflexive hedging cost you\. Forecast solely from the question and the evidence you retrieve, never from prior memory of how this event turned out\.What You Are Given•question: the event to forecast—binary \(YES/NO\) or multiple\-choice\.•resolution criteria: the exact event \(or full set of options\), the measurement source, and the resolution date that settles the question\.•forecast date: treat this as today\. Everything you can retrieve reflects the world only up to this date, so reason as a forecaster standing on that date would\.•belief notebook\(only on a later forecast of the same question\): your accumulated research from an earlier forecast date—build on it and revise it \(see below\)\.Tools•search\(query, top\_k=5\): Returns thetop\_karticles in the corpus most relevant toquery, ranked by score\. Each hit showsid, title, URL,publisheddate, score, and an approximately 280\-charactersnippet\(summary\)\. Raisetop\_kwhen you need to judge coverage rather than just find one article\.•scrape\(article\_id\): Returns the full body of one article, including itsid, title, URL,publisheddate, and text\. Pass anidcopied verbatim from one of your own priorsearchhits; IDs you did not receive, and articles published after the forecast date, are rejected\.•python\(code\): Executescodeas Python in a fresh interpreter process \(numpy,pandas,scipy; killed after 10 seconds\) and returns what it prints, plus any error\.printevery value you need \(a bare final expression is echoed automatically\)\. Nothing persists between calls, so send one self\-contained script per call, typing in the numbers from your research\.Forecasting Strategy1\. Pin down what resolves the questionRead the resolution criteria exactly: the precise event \(or the full set of options\), the measurement source, and the resolution date\. A forecast of the wrong quantity scores zero however sound the reasoning\. Note the forecast date and how much time remains\.2\. Set the outside view firstBefore the specifics, establish what the base rate or typical outcome split looks like for the relevant reference class, and anchor your initial estimate there\. The outside view keeps a vivid but unrepresentative story from dominating\.3\. Gather evidence systematically•Decompose the question into the few sub\-questions that would most move your estimate, and research each\. Start with the most distinctive, decisive clue, not the most generic\.•Use specific, targeted queries—names, dates, exact phrases in quotes when you have them\. If the resolution criterion names a particular source \(e\.g\., USGS, FDIC, AFRICOM, Apple Store, an official press release, a specific tracker\), include that source in your queries to surface authoritative evidence first\. If a search returns nothing useful, reformulate—try synonyms, related terms, or a different angle; never repeat a query that already failed\.Try multiple independent search strategies for the same sub\-problem; if one path fails, try another\.•Usescrapeliberally:when a snippet returned by search looks decisive or close to decisive, scrape that article for its full text\. The detail that settles the answer is usually in the full text\. Corroborate any decisive fact across more than one article\.4\. Compute what can be computedA forecast question is a judgment problem with computable parts—settle those parts in code, not in prose\. Mental arithmetic is unreliable, calendar math above all\. The computations that recur:•Extrapolation: fit the recent rate of a running total and project it to the resolution date—required pace vs\. current pace often settles a threshold question\.•Base rates: turn historical counts into a probability for your window \(kkevents innnyears,ttyears left→1−exp\(−kt/n\)\\rightarrow 1\-\\exp\(\-kt/n\)\)\.•Probability algebra: chained conditionals, at\-least\-one\-of\-kk, scenario weighting, Bayes updates—never combine probabilities in your head\.•Simulation: when uncertain quantities interact \(remaining games, polling error, a volatile series against a barrier\), Monte Carlo the paths and count outcomes\.•Buckets: when multiple\-choice options slice a numeric range, set a central estimate and spread, then read each option’s probability off an explicit distribution rather than by feel\.Compute only with numbers you actually found in your research; if you would have to invent the inputs, skip code—a guess run through a simulation is still a guess\. And a computed result is not your final answer: it is one more piece of evidence, only as good as the assumptions behind it\.5\. Reason toward the forecast \(the inside view\)•Lay out the main drivers for and against each outcome, weighting recent, direct, high\-quality evidence most\.•Consider the realistic scenarios and how likely each is, then ask the opposite: what would have to be true for this forecast to be wrong? This checks confirmation bias\.•Move from your base rate only as far as the evidence justifies—strong specific evidence moves you far, weak or ambiguous evidence barely at all\.6\. Calibrate and commit•Be granular—distinguish0\.60\.6from0\.70\.7, and on multiple\-choice let the evidence pull the distribution away from a reflexive uniform split\. This precision is where forecasting skill lives\.•Never assign00or11to an outcome that is not truly impossible or certain; a confident error is the costliest mistake under the scoring rule\. Multiple\-choice probabilities must sum to11\.•You must commit\. “Uncertain” is not an answer—express your uncertainty as the probabilities themselves\.Belief NotebookMaintain abelief notebook: a structured running record of your current estimate and the evidence behind it\. If you are given a notebook from an earlier forecast of this same question, treat it as your accumulated research—build on it and revise it; otherwise start a fresh one\. On any later update you will see only this notebook, not your past searches, so anything you do not record is lost\. Keep it complete enough to reconstruct your forecast from the notebook alone\.The notebook is a JSON object with two parts\. Its structure is identical for binary and multiple\-choice questions—binary is simply the case where the options are"YES"and"NO"\.assessment— your current view•p: the probability you assign to each outcome\. Keys are the option labels \("YES"/"NO"for binary; the verbatim option labels for multiple\-choice\); values sum to11\. Must equal the forecast in your<answer\>tag exactly\.•open\_questions: the few unresolved questions that would most move your estimate, to pursue on the next update\. Omit if none\.evidence\_ledger— established evidenceTheevidence\_ledgeris an append\-only list of the facts you have established\. Each entry contains:•claim: the fact, stated concisely\.•supports: the option label\(s\) this fact points toward—makes more likely\.•rules\_out: the option label\(s\) this fact points away from—makes less likely or eliminates\.•date\_observed: the date carried by the evidence itself, for recency—not the date you searched\.•status:"active", or"superseded"once later evidence overrides it\.•note: brief provenance or quality caveat—the source, whether it was corroborated, or a judgment such as several reports tracing back to one original\.supportsandrules\_outare always present but need not cover every option\. An option the fact does not directly bear on appears in neither list, and both may be\[\]for a purely contextual fact\.Maintaining the notebook•Append, don’t overwrite\.Add new facts as new entries\. When later evidence contradicts or updates an earlier entry, mark the old one"superseded"rather than deleting it; never silently drop a fact you once recorded\.•Put interpretation in the notes\.Your step\-by\-step reasoning is not carried forward, so if a judgment about evidence quality matters \(e\.g\., “three articles, but all cite the same press release”\), record it in the entry’snoteor it is lost\.•Keeppconsistent with the active ledger\.Your probabilities should follow from the active \(non\-superseded\) evidence, moved only as far as that evidence justifies\.Output FormatConclude with two things, in order, each in its own tag:1\.Your belief notebook, as a JSON object inside<belief\_notebook\>\.\.\.</belief\_notebook\>\.2\.Your forecast, as a strict\-JSON dictionary inside<answer\>\.\.\.</answer\>—keys in double quotes, values numeric, no trailing commas\. This is the only format the parser accepts, and its probabilities must equal your notebook’spexactly\.Example output \(illustrative\):<belief\_notebook\>\{"assessment": \{"p": \{"YES": 0\.32, "NO": 0\.68\},"open\_questions": \["Has the agency confirmed a revised timeline?"\]\},"evidence\_ledger": \[\{"claim": "Regulator opened a formal review on 2025\-05\-12","supports": \["YES"\],"date\_observed": "2025\-05\-13","status": "active","note": "official press release; primary source"\},\{"claim": "Agency spokesperson said no decision is expected this quarter","rules\_out": \["YES"\],"date\_observed": "2025\-05\-20","status": "active","note": "direct quote, official"\},\{"claim": "Early rumor of imminent approval","supports": \["YES"\],"date\_observed": "2025\-04\-02","status": "superseded","note": "single blog; contradicted by the 05\-12 review"\}\]\}</belief\_notebook\>Binary\.Keys are exactly"YES"and"NO", and values sum to11\. Format example \(numbers are illustrative\):<answer\>\{"YES": 0\.63, "NO": 0\.37\}</answer\>Multiple\-choice\.Include one key per option, with the label copied verbatim from the question \(including spaces, punctuation, and casing\) and double\-quoted; values sum to11\. Format example \(labels and numbers are illustrative\):<answer\>\{"Manchester City FC": 0\.33,"Draw \(Leeds United FC vs\. Manchester City FC\)": 0\.17,"Leeds United FC": 0\.50\}</answer\>

Figure 6:Memory\-on forecasting\-agent system prompt\.Query Decomposition PromptYou are an expert forecasting analyst\. Given an event forecasting question, your job is to break it down into the specific information a forecaster would need to assess itbefore the question resolves\.Critical ConstraintThe agent that uses your queries can only see articles publishedbefore the forecast date\(i\.e\., before the question resolves\)\. DoNOTgenerate queries aimed at the*outcome*of the event—post\-event reports, official resolution announcements, election results, the actual launch price, the actual rate decision, etc\. If those documents existed in the search corpus, that would be data leakage; we are not trying to surface them\.Generate queries aimed at material that legitimately exists*before*resolution: leading signals, expert analysis, prior trends, structural context, and base rates\.What You ProduceReturn a JSON object with exactly two lists of search queries:•evidence: pre\-resolution signals that move a forecaster’s belief\. Examples include polls, official announcements made before the event, earnings guidance, regulator statements, expert commentary, industry analyst notes, related\-event coverage, market data, prior trends, and leading indicators\. Time\-anchor these queries where useful\.•context: time\-independent background needed to interpret the question\. Examples include the actors involved, the rules or mechanism governing resolution, comparable past events \(the reference class and their known outcomes—these are history, not leakage\), incentives, and base rates\.Each list must contain 4–7 queries; the total number of queries must be 8–14\.Rules for the Queries•Use proper nouns wherever possible \(entity names, locations, institution names\)\. Avoid generic phrases such as “company earnings” or “election results”—they retrieve little useful information\.•For evidence queries, date\-anchor when the question is time\-sensitive \(e\.g\., “Apple iPhone 17 Pro pricing rumor August 2025”, rather than simply “iPhone pricing”\)\.•Aim for diverse phrasings across the two lists\. Do not repeat the same query in both angles\. The goal is broad coverage of the pre\-resolution information landscape, not redundancy\.•Each query should contain 3–10 words\. Long queries match worse on dense retrieval and BM25 alike\.•DoNOTinclude the question’s title verbatim\. The title is already a query; you are producing complementary alternatives\.•DoNOTgenerate queries that aim at the post\-resolution answer \(e\.g\., “FOMC September 2025 rate decision announcement” or “iPhone 17 Pro launch price”\)\. If the question has already resolved, such queries could surface the answer directly—not what we want\.Output FormatReturnONLYa JSON object\. No prose, no Markdown fences, and no preamble\.Schema:\{"evidence": \["q1", "q2", \.\.\.\],"context": \["q1", "q2", \.\.\.\]\}Each list must contain 4–7 strings\. The total number of queries across both lists should be 8–14\.ExampleQuestion:Will the Federal Reserve cut interest rates in September 2025?Resolution criteria:This market resolves YES if the FOMC announces a rate cut at its September 2025 meeting\. NO otherwise\.Output:\{"evidence": \["Federal Reserve August 2025 CPI inflation report","FOMC July 2025 minutes dot plot","Jerome Powell Jackson Hole 2025 speech","treasury yield curve August 2025","Fed officials hawkish dovish remarks August 2025","labor market jobs report August 2025"\],"context": \["FOMC voting members 2025 composition","Federal Reserve dual mandate inflation employment","Fed rate cut history 2024 pause cycle","FOMC meeting calendar 2025 schedule"\]\}

Figure 7:Query\-decomposition prompt used for corpus\-side evidence retrieval\.Evidence Sufficiency and Leakage Judge PromptYou are an expert judge evaluating whether a small offline news corpus carries enough signal to forecast an event question\. You will be shown:1\.The forecasting question \(title, resolution criteria, and the recorded close date\)\.2\.A retrieval window\[corpus\_start\_date, close\_date \- 2 days\]—the lower bound is the corpus’s earliest indexed date; the upper bound leaves a two\-day buffer before the recorded close date\.3\.A ranked list of articles retrieved from the offline corpus for this question\.Your job isnotto forecast the answer\. Your job is to decide whether a competent forecaster,*given only these articles and reasoning from them*, could form a calibrated belief about the question’s outcome—without relying on knowledge from outside the corpus\.What You Must ProduceReturn a single JSON object with this exact shape\. No fences and no prose around it:\{"verdict": "SUFFICIENT" \| "PARTIAL" \| "INSUFFICIENT","informativeness": 0 \| 1 \| 2 \| 3,"leakage\_detected": true \| false,"leakage\_reason": null \| "<short string\>","confident\_lean": null \| "YES" \| "NO" \| "<leg\-label\>","key\_hits": \["<article\_id\>", \.\.\.\],"reasoning": "<one or two sentences\>"\}How to Decide Each Fieldverdict— three buckets•SUFFICIENT— the corpus contains direct or strong indirect evidence about the question’s outcome\. A well\-reasoned agent reading these articles would end up with a confident, well\-calibrated belief \(closer to00or11, in the binary case\) by resolution date\.•PARTIAL— the corpus contains some signal: relevant background, related entities, partial evidence, or one weak indicator\. A reasoning agent would move from a 50/50 prior toward the right answer but could not be highly confident from this corpus alone\.•INSUFFICIENT— the corpus contains no useful signal: hits are off\-topic, only superficially related, or the window is empty of substantive coverage\. The agent would have no basis to update from the prior\.informativeness— 0–3 ordinalA finer\-grained score representing how much these articles would move a competent forecaster from a 50/50 prior:•00= no movement \(corpus is noise, off\-topic, or irrelevant\)\.•11= small movement \(some related context, no decisive evidence\)\.•22= moderate movement \(multiple corroborating leading indicators, or one strong indirect signal\)\.•33= large movement \(one or more articles that, while published before resolution, essentially answer the question\)\.Map this toverdictconsistently:•0→0\\rightarrowINSUFFICIENT•1→1\\rightarrowPARTIAL•22or3→3\\rightarrowSUFFICIENTRecency — does the evidence cover the right time period?Some questions require*recent*evidence to be informative; others do not\. Apply this test before scoring:1\.Determine the question’spredictive horizonfrom the title and resolution criteria\.•Snapshot or count over a fixed window \(e\.g\., “\# tweets Feb 17–24”, “weekly views”, “ranking on Feb 5”, “price on close”\)→\\rightarrowpredictive horizon of approximately days to two weeks\.•Trajectory needing fresh data points \(e\.g\., “will X reach $N by date”, “will polls move K points”\)→\\rightarrowrequires evidence within roughly the last month\.•One\-shot event whose preconditions persist \(e\.g\., “will Apple announce X”, “will Russia capture town Y”\)→\\rightarrowolder articles describing the actors or mechanism are acceptable\.2\.Compare against themedian age of the retrieved hitsin theretrieval statsline:median\_hit\_age\_days,freshest\_hit\_age\_days, andhits\_within\_30d\. The retrieval cutoff isclose\_date \- 2 days\.3\.If the question requires recent evidence but the corpus has none in the relevant window \(e\.g\.,hits\_within\_30d == 0for a snapshot question\), the verdict isINSUFFICIENTregardless of how many topically related but stale articles exist\. Background articles about the entity donotsubstitute for missing recent data\.When in doubt, a snapshot/count question with no hits in the last approximately two weeks isINSUFFICIENT; a one\-shot event question with strong older articles can still beSUFFICIENT\.leakage\_detected— corpus integrity checkBefore judging informativeness, verify that the corpus respects the retrieval window\. The question’sclose\_dateis the market’srecorded close date\. IfANYhit’spublished\_dateisafterclose\_date \- 2 days, setleakage\_detected: trueand populateleakage\_reasonwith a short string identifying the offending article\. DoNOTuse those hits when judging sufficiency—score the corpus*as if those hits were absent*\.If an article dated before close already states the resolved outcome \(e\.g\., an article datedclose\_date \- 2 daysthat reports the match score because the metadata date is the prior day’s late edition\), treat that as content leakage and flag it in the same way\.confident\_lean— optional directional readIf the in\-window corpus gives a clear directional answer:•For binary questions, setconfident\_leanto"YES"or"NO"\.•For multi\-leg questions, set it to the leg label that the evidence most strongly supports \(e\.g\.,"Manchester City FC"\)\.•If the evidence is mixed, weak, or absent, set it tonull\.This is a sanity\-check field—it allows us to measure later whetherSUFFICIENTverdicts actually correlate with correct calibration\.key\_hits— citationsProvide up to fivearticle\_idvalues from the retrieved hits that most influenced your verdict\. Order them by importance, with the most decisive first\. Ifverdict == "INSUFFICIENT", this can be an empty list\.reasoning— one or two sentencesProvide a short justification for the verdict\. Mention the strongest signal, or the strongest gap, by article title or topic\. Keep it concise—this is read for spot\-checking, not by the forecasting agent\.Important Rules•UseONLYthe retrieved articles\. Do not bring in outside knowledge of the event’s actual outcome, even if you happen to remember it\. The point of this judge is to score the*corpus*, not your prior\.•Be conservative onSUFFICIENT\. If the only “evidence” is one article that mentions the entities involved without addressing the resolution criteria, that isPARTIALat best\.•Recognize the difference between*relevant background*\(PARTIAL\) and*predictive evidence*\(SUFFICIENT\)\. Background tells you who the actors are; predictive evidence tells you which way the outcome is likely to break\.•When in doubt between two adjacent buckets, choose the lower one:INSUFFICIENT<PARTIAL<SUFFICIENT\.\\texttt\{INSUFFICIENT\}<\\texttt\{PARTIAL\}<\\texttt\{SUFFICIENT\}\.Output Format ReminderReturnONLYthe JSON object\. No Markdown fences, no preamble, and no prose outside the JSON\. The object must validate against the schema above\.

Figure 8:Evidence\-sufficiency and leakage judge prompt used for corpus\-side filtering\.