逻辑一致的准确预测

arXiv cs.LG 论文

摘要

一篇论文提出了一种监督型大语言模型,通过基于预测模型的推理过程进行微调,联合消解相关预测之间的逻辑不一致,使采用检索增强的 GPT-5.1 预测模型在 ForecastBench 上的布里尔评分改善了 9.3%,同时将结构性不一致减少了 82–90%。

arXiv:2611.17735v1 Announce Type: new Abstract: Agentic forecasting systems based on large language models (LLMs) are now competitive with expert human forecasters, but these systems still suffer from a critical limitation: the inability to produce forecasts that are logically consistent. For example, forecasts for related events often fail to satisfy the axioms of probability, which in turn implies that related forecasts cannot simultaneously be correct and undermines their utility for decision makers. Enforcing consistency is not trivial -- naive approaches (e.g., projection onto the set of consistent forecasts) can degrade accuracy. We argue that the key to overcoming the apparent tradeoff between consistency and accuracy lies in the reasoning which supports each forecast. We finetune a supervisory language model that leverages this reasoning to jointly reconcile inconsistencies while improving the accuracy of related forecasts. On ForecastBench, our method improves the Brier score of a retrieval-augmented GPT-5.1 forecaster by 9.3% while reducing measures of structural inconsistency by 82-90%. We interrogate our method via a comprehensive series of baselines and ablations, and find that no alternative method matches our gains in consistency and accuracy simultaneously.
查看原文
查看缓存全文

缓存时间: 2026/11/16 08:45

# Logically Consistent Accurate Forecasts
Source: [https://arxiv.org/html/2611.17735](https://arxiv.org/html/2611.17735)
Rohan AlurAffiliation:Bridgewater AIA LabsAffiliation:Massachusetts Institute of Technology\*Equal contribution\.Manish RaghavanAffiliation:Massachusetts Institute of Technology\*Equal contribution\.Devavrat ShahAffiliation:Massachusetts Institute of Technology\*Equal contribution\.

###### Abstract

Agentic forecasting systems based on large language models \(LLMs\) are now competitive with expert human forecasters, but these systems still suffer from a critical limitation: the inability to produce forecasts that are logically*consistent*\. For example, forecasts for related events often fail to satisfy the axioms of probability, which in turn implies that related forecasts cannot simultaneously be correct and undermines their utility for decision makers\. Enforcing consistency is not trivial — naive approaches \(e\.g\., projection onto the set of consistent forecasts\) can degrade accuracy\. We argue that the key to overcoming the apparent tradeoff between consistency and accuracy lies in the reasoning which supports each forecast\. We finetune a supervisory language model that leverages this reasoning to jointly reconcile inconsistencies while improving the accuracy of related forecasts\. On ForecastBench\[[1](https://arxiv.org/html/2611.17735#bib.bibx1)\], our method improves the Brier score of a retrieval\-augmented GPT\-5\.1 forecaster by9\.3%9\.3\\%while reducing measures of structural inconsistency by8282–90%90\\%\. We interrogate our method via a comprehensive series of baselines and ablations, and find that no alternative method matches our gains in consistency and accuracy simultaneously\.

## 1Introduction

Agentic forecasting systems based on large language models \(LLMs\) have improved rapidly in recent years\[[1](https://arxiv.org/html/2611.17735#bib.bibx1),[2](https://arxiv.org/html/2611.17735#bib.bibx2),[3](https://arxiv.org/html/2611.17735#bib.bibx3)\], but important limitations remain\. In this work, we focus on a particular shortcoming: the inability to maintain logical*consistency*across related forecasts\. To illustrate this challenge, suppose we ask an LLM\-based forecasting system to generate two separate election forecasts: what is the probability a candidate wins by at least1%1\\%of the vote, and what is the probability the same candidate wins by at least5%5\\%of the vote?

We can immediately see that the true probabilities must satisfyℙ⁡\(wins≥1%\)≥ℙ⁡\(wins≥5%\)\\mathbb\{P\}\(\\texttt\{wins $\\geq 1\\%$\}\)\\geq\\mathbb\{P\}\(\\texttt\{wins $\\geq 5\\%$\}\), as the latter necessarily implies the former\. However, forecasting systems are typically queried independently on each input, and will commonly violate even elementary probability axioms such as this one\. This can be viewed as a special case of modern agentic LLM\-based systems’ more general failure to maintain a coherent*world model*, even when outputs appear correct both in isolation and on average\[[4](https://arxiv.org/html/2611.17735#bib.bibx4),[5](https://arxiv.org/html/2611.17735#bib.bibx5),[6](https://arxiv.org/html/2611.17735#bib.bibx6)\]\. We discuss additional related work in Appendix[A](https://arxiv.org/html/2611.17735#A1)\.

We argue that this type of inconsistency severely undermines the utility of LLM\-based forecasts: forecasts are most valuable insofar as they inform decisions, and it is unclear how to reconcile estimates that are in direct conflict with each other\. Unfortunately, while detecting forecast inconsistencies can be straightforward, correcting these inconsistencies without sacrificing the accuracy of forecasts is not — see the example in Appendix[B](https://arxiv.org/html/2611.17735#A2)for a concrete illustration\.

In this work we address the challenge of improving logical consistency across questions without trading off accuracy\. Our central insight is to examine the LLM forecaster’s reasoning when reconciling inconsistent forecasts\. We show empirically that this approach can significantly improve both logical consistency and accuracy, overcoming the apparent tradeoff faced by alternative methods\.

Given target eventto forecast1\. QuestionGenerate structurallyrelated questions2\. Related question clusterRun the agentic LLM forecasteron each question, producingprobabilities and reasoning3\. Upstream forecastsJointly reconcile allforecasts given the cluster’sprobabilities and reasoning4\. Refined forecastsWill candidate X win by at least 1%?nested \(stronger\)X wins by≥5%\{\\geq\}\\,5\\%?original questionX wins by≥1%\{\\geq\}\\,1\\%?nested \(weaker\)X wins \(by any margin\)?complementX does*not*win by≥1%\{\\geq\}\\,1\\%?partitionX wins by<1%\{<\}\\,1\\%?partitionX loses?50%because …40%because …62%because …72%because …15%because …25%because …nesting:50%\>40%50\\%\>40\\%→\\rightarrow30%<47%30\\%<47\\%complements:sum to112%112\\%→\\rightarrowsum to99%99\\%partition:sum to80%80\\%→\\rightarrowsum to99%99\\%30%47%58%52%11%41%

Figure 1:Overview of our method, illustrated with a stylized example\. Given a target question to forecast, we generate a cluster of related questions with structural relationships to the original\. We then run the upstream agentic LLM forecaster on all questions in the cluster, obtaining initial probability forecasts and reasoning traces for each question\. Finally, the crux of our method is to jointly reconcile the forecasts within each cluster using the initial probabilities and reasoning traces\.
## 2Setup

Forecasting events\.LetΩ\\Omegadenote the elementary outcomes of interest, and letℱ\\mathcal\{F\}denote theσ\\sigma\-algebra overΩ\\Omega\. For example,Ω\\Omegamay denote the set of possible outcomes for the U\.S\. Congressional elections, andℱ\\mathcal\{F\}may contain the events these outcomes induce \(e\.g\., that candidate X wins by Y points\)\. Letℙ⋆\\mathbb\{P\}^\{\\star\}denote the distribution over outcomesΩ\\Omega\. We are interested in forecastingn≥1n\\geq 1events, which we denote asEi⊂Ω,Ei∈ℱE\_\{i\}\\subset\\Omega,E\_\{i\}\\in\\mathcal\{F\}fori∈\[n\]≡\{1,…,n\}i\\in\[n\]\\equiv\\\{1,\\dots,n\\\}\. That is, we seek to output the marginal probabilitiespi⋆=ℙ⋆​\(Ei\)p^\{\\star\}\_\{i\}=\\mathbb\{P\}^\{\\star\}\(E\_\{i\}\)fori∈\[n\]i\\in\[n\]\.

Estimates and evidence\.For each eventEiE\_\{i\}, letqiq\_\{i\}denote the question text describing it\. We are provided with a corresponding*probability estimate*pi\(0\)∈\[0,1\]p^\{\(0\)\}\_\{i\}\\in\[0,1\]and a set of accompanying*reasoning*ziz\_\{i\}\. In this work we consider\(pi\(0\),zi\)\(p^\{\(0\)\}\_\{i\},z\_\{i\}\)that are generated by an upstream agentic forecasting system that has been queried withqiq\_\{i\}\. Thus,ziz\_\{i\}denotes the evidence collected by the agentic forecaster \(e\.g\., via open\-ended web search\) and the reasoning it generates over this evidence for eventEiE\_\{i\};pi\(0\)p^\{\(0\)\}\_\{i\}then denotes the upstream forecaster’s estimated probability forEiE\_\{i\}\. We let𝐩\(0\)=\[pi\(0\):i∈\[n\]\]\\mathbf\{p\}^\{\(0\)\}=\[p^\{\(0\)\}\_\{i\}:i\\in\[n\]\]and𝐳=\[𝐳i:i∈\[n\]\]\\mathbf\{z\}=\[\\mathbf\{z\}\_\{i\}:i\\in\[n\]\]\. We treat these quantities as fixed, exogenously provided inputs throughout\.

Structural constraints and prior information\.A primary focus of our approach is to leverage relationships between the events of interest, both as inputs to our methodology and for the evaluation thereof\. We express these relationships as constraints on the set of allowed probability vectors; for example, the relationshipEi⊂EjE\_\{i\}\\subset E\_\{j\}can be expressed as the constraintpi⋆≤pj⋆p^\{\\star\}\_\{i\}\\leq p^\{\\star\}\_\{j\}\. We consider three types of structural constraints, which we denote by𝒞\\mathcal\{C\},𝒫\\mathcal\{P\}, andℳ\\mathcal\{M\}\.111The complement constraints \(𝒞\\mathcal\{C\}\) can be expressed as a special case of the partition constraints \(𝒫\\mathcal\{P\}\)\. We separate them here to match our experimental setup\.Complements\(𝒞\\mathcal\{C\}\): for a pair\(i,j\)\(i,j\)withEi=EjcE\_\{i\}=E\_\{j\}^\{c\}, it must be thatpi⋆\+pj⋆=1p^\{\\star\}\_\{i\}\+p^\{\\star\}\_\{j\}=1\.Partitions\(𝒫\\mathcal\{P\}\): more generally, forJ⊆\[n\]J\\subseteq\[n\], if⋃j∈JEj=Ω\\bigcup\_\{j\\in J\}E\_\{j\}=\\OmegaandEj∩Ej′=∅E\_\{j\}\\cap E\_\{j^\{\\prime\}\}=\\emptysetfor allj≠j′∈Jj\\neq j^\{\\prime\}\\in J, then∑j∈Jpj⋆=1\\sum\_\{j\\in J\}p^\{\\star\}\_\{j\}=1\. That is, ifJJdenotes a partition of the outcome space, the corresponding forecasts for the events inJJmust sum to11\.Nesting\(ℳ\\mathcal\{M\}\): for a chainEi1⊆⋯⊆EikE\_\{i\_\{1\}\}\\subseteq\\cdots\\subseteq E\_\{i\_\{k\}\}, we havepi1⋆≤⋯≤pik⋆p^\{\\star\}\_\{i\_\{1\}\}\\leq\\cdots\\leq p^\{\\star\}\_\{i\_\{k\}\}\.

Goal and evaluation\.Given\(qi,pi\(0\),zi:i∈\[n\]\)\(q\_\{i\},p^\{\(0\)\}\_\{i\},z\_\{i\}:i\\in\[n\]\), our goal is to transform the estimates𝐩\(0\)\\mathbf\{p\}^\{\(0\)\}into refined forecasts𝐩^=\[p^i:i∈\[n\]\]\\hat\{\\mathbf\{p\}\}=\[\\hat\{p\}\_\{i\}:i\\in\[n\]\]\. Ideally these refined forecasts will be more accurate:𝐩^\\hat\{\\mathbf\{p\}\}will be closer to𝐩⋆\\mathbf\{p\}^\{\\star\}than𝐩\(0\)\\mathbf\{p\}^\{\(0\)\}is\. The primary complication, as in all binary forecasting tasks, is that we never observe the true probabilities𝐩⋆\\mathbf\{p\}^\{\\star\}\. We do however observe the realized outcomesYi=𝟏​\{Ei​occurs\}Y\_\{i\}=\\mathbf\{1\}\\\{E\_\{i\}\\text\{ occurs\}\\\}\. By definition, we havepi⋆=ℙ⋆​\(Yi=1\)=ℙ⋆​\(Ei\)p^\{\\star\}\_\{i\}=\\mathbb\{P\}^\{\\star\}\(Y\_\{i\}=1\)=\\mathbb\{P\}^\{\\star\}\(E\_\{i\}\)\.

We measure performance via the Brier Score: for𝐩∈\[0,1\]n\\mathbf\{p\}\\in\[0,1\]^\{n\}and𝐘∈\{0,1\}n\\mathbf\{Y\}\\in\\\{0,1\\\}^\{n\},𝖡𝗋𝗂𝖾𝗋⁡\(𝐩,𝐘\)=1n​∑i∈\[n\]\(pi−Yi\)2\\mathsf\{Brier\}\(\\mathbf\{p\},\\mathbf\{Y\}\)=\\frac\{1\}\{n\}\\sum\_\{i\\in\[n\]\}\(p\_\{i\}\-Y\_\{i\}\)^\{2\}\. In addition to accuracy, we also evaluate the degree to which the forecasts𝐩^\\hat\{\\mathbf\{p\}\}conform to the structural constraints above\. Specifically, we compute themagnitude of constraint violationswith respect to𝒞,𝒫\\mathcal\{C\},\\mathcal\{P\}, andℳ\\mathcal\{M\}\. The precise evaluation metrics are defined in Appendix[C\.2](https://arxiv.org/html/2611.17735#A3.SS2)\.

## 3Method: imposing consistency while improving accuracy

Our method consists of three stages, as shown in Figure[1](https://arxiv.org/html/2611.17735#S1.F1)\. First, for each event of interestEiE\_\{i\}, we produce a set of related events via*synthetic clustering*, described in Section[3\.1](https://arxiv.org/html/2611.17735#S3.SS1)\. Each resulting cluster consists of the targetEiE\_\{i\}along with events that are structurally related toEiE\_\{i\}via the constraint families𝒞\\mathcal\{C\},𝒫\\mathcal\{P\}andℳ\\mathcal\{M\}\. Next, we use an upstream LLM forecaster to generate an initial probability estimate and associated reasoning for each event in the cluster\. Finally, we train a small language model to examine these estimates and reasoning jointly and produce an updated forecastp^i\\hat\{p\}\_\{i\}for each event, described in Section[3\.2](https://arxiv.org/html/2611.17735#S3.SS2)\. Note that this final step is the crux of our method and is agnostic to the particular clustering procedure or upstream LLM forecaster used\.

### 3\.1Synthetic clustering

Each input questionqiq\_\{i\}is assigned to a cluster consisting of \(a\) itself and \(b\) synthetically generated, structurally related variants of itself\. For each input questionqiq\_\{i\}, we prompt Claude Opus 4\.7\[[7](https://arxiv.org/html/2611.17735#bib.bibx7)\]to generate related questions corresponding to the constraint families𝒞\\mathcal\{C\},𝒫\\mathcal\{P\}andℳ\\mathcal\{M\}\. In particular, for eachqiq\_\{i\}we generate \(a\)q¯i\\bar\{q\}\_\{i\}which describes the complement ofqiq\_\{i\}, \(b\) siblingsqi1,…,qikq\_\{i\}^\{1\},\\dots,q\_\{i\}^\{k\}which, together withqiq\_\{i\}, form a mutually exclusive and collectively exhaustive partition of the outcome space \(i\.e\., exactly one of the eventsqi,qi1,…,qikq\_\{i\},q\_\{i\}^\{1\},\\dots,q\_\{i\}^\{k\}must occur\),222Complement questions in \(a\) are a special case of event\-space partitions, so we requirek≥2k\\geq 2in \(b\)\.and \(c\) a nested sequence of eventsq~i1,…,q~iℓ\\tilde\{q\}\_\{i\}^\{1\},\\dots,\\tilde\{q\}\_\{i\}^\{\\ell\}which, together withqiq\_\{i\}, form a chain of increasingly general events \(i\.e\., each event implies the next, so that their probabilities must be monotonically nondecreasing\)\. Appendix[D](https://arxiv.org/html/2611.17735#A4)contains further details including our validation of the correctness of these clusters\.

Let the setJiJ\_\{i\}index the synthetic questions associated withqiq\_\{i\}\. Applying the augmentation across all questions yields the index setI=\[n\]∪⋃i=1nJiI=\[n\]\\cup\\bigcup\_\{i=1\}^\{n\}J\_\{i\}\. We defineCi=\{i\}∪JiC\_\{i\}=\\\{i\\\}\\cup J\_\{i\}as the index set for the cluster corresponding toqiq\_\{i\}\. The set of clusters\{Ci\}i=1n\\\{C\_\{i\}\\\}\_\{i=1\}^\{n\}partitions the augmented index setII\.

### 3\.2Learning algorithm

Our learning algorithm maps the augmented inputs\{\(qj,zj,pj\(0\)\)\}j∈I\\\{\(q\_\{j\},z\_\{j\},p^\{\(0\)\}\_\{j\}\)\\\}\_\{j\\in I\}to a set of refined forecasts\{p^j\}j∈I\\\{\\hat\{p\}\_\{j\}\\\}\_\{j\\in I\}\. For eachj∈Ij\\in I, letC⁡\(j\)C\(j\)denote the cluster containingjj\.

To generate a refined forecast forqjq\_\{j\}, we construct a prompt containing the text, reasoning and estimates for every question in the corresponding clusterC⁡\(j\)C\(j\), and we setp^j\\hat\{p\}\_\{j\}to the probability assigned to the “yes” token after renormalizing the model’s distribution over the “yes” and “no” tokens\. Additional details are given in Appendix[C](https://arxiv.org/html/2611.17735#A3)\.

Loss function\.We refine forecasts by finetuning Qwen3\.6\-35B\-A3B\[[8](https://arxiv.org/html/2611.17735#bib.bibx8)\]to minimize an approximately equal weighted sum of three terms – Brier score, logical consistency penalties, and squared deviation from the upstream forecasts \(a regularization term\)\. The consistency penalty uses squared violations of complement and partition constraints, and hinge violations of monotonicity constraints\.

We do*not*directly observe outcomes for the synthetic questions \(although some of these outcomes can be deduced\)\. Thus, Brier score is computed over the original questions only, while consistency and proximity terms are computed over both original and synthetic questions\. Appendix[C](https://arxiv.org/html/2611.17735#A3)contains full details, including exact definitions and hyperparameter settings\.

## 4Empirical results

Data\.We source data from ForecastBench\[[1](https://arxiv.org/html/2611.17735#bib.bibx1)\], a popular forecasting benchmark of binary questions drawn from prediction markets and well\-maintained datasets\.

We construct our training set as the union of unique events released by ForecastBench from 03/02/2025 through 02/15/2026\. From this set, we keep only those that resolve strictly before 03/01/2026, and use the questions released from 03/01/2026 through 04/26/2026 as our validation set\. This prevents foreknowledge of events in the validation set from bleeding into the training objective\. Similarly, we only keep validation set questions which resolve strictly before 05/10/2026, and use questions released from 05/10/2026 through 07/05/2026 as our canonical test set\. Importantly, this test set consists only of questions published after the release \(and therefore beyond the training cutoff\) of Qwen3\.6\-35B\-A3B\[[8](https://arxiv.org/html/2611.17735#bib.bibx8)\]\.

Following the evaluation methodology for ForecastBench\[[1](https://arxiv.org/html/2611.17735#bib.bibx1)\], we compute each evaluation metric separately for two groups, the market sources and dataset sources, and then report their equal\-weighted average\. Our test set has 243 market questions and 1,344 dataset questions\. For efficiency, our training and validation sets sample a random subset of their respective eligible dataset questions, such that the number of dataset questions in each is equal to the number of market questions\. See Appendix[E](https://arxiv.org/html/2611.17735#A5)for further details\.

Upstream forecaster\.We use a retrieval\-augmented GPT\-5\.1\[[9](https://arxiv.org/html/2611.17735#bib.bibx9)\]forecaster\. All of our training, validation, and test questions are released after the knowledge cutoff date of GPT\-5\.1\[[10](https://arxiv.org/html/2611.17735#bib.bibx10)\]\. We use Perplexity as a search provider, and use their API to filter retrieved news sources to those published before the release date of each question\[[11](https://arxiv.org/html/2611.17735#bib.bibx11)\]\. Appendix[F](https://arxiv.org/html/2611.17735#A6)describes additional details\.

Results\.We summarize our primary results in Table[1](https://arxiv.org/html/2611.17735#S4.T1)below\. Our method improves the Brier score and all forms of consistency measured on the test set across three independent seeds\. The improvement in every metric is significant at the5%5\\%level, detailed in Appendix[I](https://arxiv.org/html/2611.17735#A9)\. We report results for each seed separately in Appendix[G](https://arxiv.org/html/2611.17735#A7), and evaluation metric definitions are in Appendix[C\.2](https://arxiv.org/html/2611.17735#A3.SS2)\.

Table 1:Our method’s headline results\.*Raw*denotes the upstream forecaster\.*Ours*reports the mean±\\pmsample standard deviation of our method across three seeds\. Lower is better for every metric\.%Δ\\%\\Deltareports the relative percentage change of*Ours*with respect to*Raw*\.Table 2:Comparison of our method with baseline methods, showing the relative change \(%Δ\\%\\Delta\) in Brier score and consistency violations with respect to*Raw*for each method\. Negative values are better, and the best value in each column is in bold\.Baselines and ablations\.We conduct a series of baselines and ablations\. Our method is statistically better in accuracy, consistency, or both metrics, as compared with every alternative method tested\. Table[2](https://arxiv.org/html/2611.17735#S4.T2)summarizes performance across baseline methods\. For example, since we are effectively using additional rollouts of the upstream forecaster when considering related questions’ forecasts, we compare against a simple ensemble over ten independent rollouts for the same event\.

Table[7](https://arxiv.org/html/2611.17735#A8.T7)in Appendix[H\.2](https://arxiv.org/html/2611.17735#A8.SS2)provides the analogous summary of ablation methods – a notable ablation is the removal of forecaster reasoning traces from our method, where the Brier change shifts significantly from−9\.32%\-9\.32\\%to−2\.67%\-2\.67\\%\. Appendix[H](https://arxiv.org/html/2611.17735#A8)describes each baseline and ablation, and Appendix[I](https://arxiv.org/html/2611.17735#A9)describes significance testing\.

## 5Discussion and limitations

In this work, we show that forecasts from LLM\-based agentic systems can be post\-processed to improve logical consistency without sacrificing accuracy\. Our key insight is to leverage the forecaster’s own reasoning to reconcile inconsistent forecasts\. However, our approach has several limitations\. The first is scale: our method relies on a relatively small open\-source language model, and practical constraints — both cost and runtime — preclude us from scaling our method to impose consistency across large numbers of related forecasts\. Relatedly, our methodology also assumes that relationships between events are provided exogenously rather than learned directly from the available data, and only handles three types of structural constraints\.

These limitations indicate multiple avenues for future research\. First, a natural extension of our work would consider a broader set of possible relationships, including “soft” constraints on the relationships between forecasts\. A second promising direction is to train an auxiliary model to perform this labeling directly, indicating which events should be related and how\. Finally, scaling our approach to larger sets of related events — whether via simply scaling the effective inference compute, or via algorithmic improvements to allow for e\.g\., iterative refinement of forecasts or inference\-time retrieval over related forecasts — would significantly broaden the applicability of our methods\.

## References

- \[1\]Ezra Karger, Houtan Bastani, Chen Yueh\-Han, Zachary Jacobs, Danny Halawi, Fred Zhang and Philip Tetlock“ForecastBench: A dynamic benchmark of AI forecasting capabilities”In*arXiv preprint arXiv:2409\.19839*, 2024
- \[2\]Danny Halawi, Fred Zhang, Chen Yueh\-Han and Jacob Steinhardt“Approaching Human\-Level Forecasting with Language Models”, 2024eprint:arXiv:2402\.18563
- \[3\]Rohan Alur et al\.“AIA Forecaster: Technical Report”, 2025arXiv:[2511\.07678 \[cs\.AI\]](https://arxiv.org/abs/2511.07678)
- \[4\]Keyon Vafa, Justin Chen, Ashesh Rambachan, Jon Kleinberg and Sendhil Mullainathan“Evaluating the world model implicit in a generative model”, 2024arXiv:[2406\.03689 \[cs\.CL\]](https://arxiv.org/abs/2406.03689)
- \[5\]Keyon Vafa, Peter Chang, Ashesh Rambachan and Sendhil Mullainathan“What has a foundation model found? Using inductive bias to probe for world models”, 2025arXiv:[2507\.06952 \[cs\.LG\]](https://arxiv.org/abs/2507.06952)
- \[6\]Marina Mancoridis, Bec Weeks, Keyon Vafa and Sendhil Mullainathan“Potemkin understanding in large language models”, 2025arXiv:[2506\.21521 \[cs\.CL\]](https://arxiv.org/abs/2506.21521)
- \[7\]Anthropic“System Card: Claude Opus 4\.7”, 2026URL:[https://www\-cdn\.anthropic\.com/037f06850df7fbe871e206dad004c3db5fd5034verb0/Claude](https://www-cdn.anthropic.com/037f06850df7fbe871e206dad004c3db5fd5034verb0/Claude)
- \[8\]Qwen Team“Qwen3\.6\-35B\-A3B: Agentic Coding Power, Now Open to All”, 2026URL:[https://qwen\.ai/blog?id=qwen3\.6\-35b\-a3b](https://qwen.ai/blog?id=qwen3.6-35b-a3b)
- \[9\]OpenAI“GPT\-5\.1 Instant and GPT\-5\.1 Thinking System Card Addendum”, 2025URL:[https://cdn\.openai\.com/pdf/4173ec8d\-1229\-47db\-96de\-06d87147e07e/5\_1\_sverbystem\_card\.pdf](https://cdn.openai.com/pdf/4173ec8d-1229-47db-96de-06d87147e07e/5_1_sverbystem_card.pdf)
- \[10\]OpenAI“GPT\-5\.1”, 2025URL:[https://developers\.openai\.com/api/docs/models/gpt\-5\.1](https://developers.openai.com/api/docs/models/gpt-5.1)
- \[11\]Perplexity AI“Search Date and Time Filters”URL:[https://docs\.perplexity\.ai/docs/search/filters/date\-time\-filters](https://docs.perplexity.ai/docs/search/filters/date-time-filters)
- \[12\]Michael Lawrence, Paul Goodwin, Marcus O’Connor and Dilek \\"Onkal“Judgmental forecasting: A review of progress over the last 25years” Twenty five years of forecastingIn*International Journal of Forecasting*22\.3, 2006, pp\. 493–518DOI:[10\.1016/j\.ijforecast\.2006\.03\.007](https://dx.doi.org/10.1016/j.ijforecast.2006.03.007)
- \[13\]Philip Tetlock“Expert political judgment: How good is it? How can we know?”, 2005
- \[14\]Philip Tetlock and Dan Gardner“Superforecasting: The art and science of prediction”, 2015
- \[15\]Robert Fildes and H\.O\. Stekler“The state of macroeconomic forecasting”In*Journal of Macroeconomics*, 2002
- \[16\]Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans and Dan Hendrycks“Forecasting Future World Events with Neural Networks”, 2022eprint:arXiv:2206\.15474
- \[17\]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li and Peter\. Liu“Exploring the Limits of Transfer Learning with a Unified Text\-to\-Text Transformer”, 2019eprint:arXiv:1910\.10683
- \[18\]Daniel Khashabi, Yeganeh Kordi and Hannaneh Hajishirzi“UnifiedQA\-v2: Stronger Generalization via Broader Cross\-Format Training”, 2022eprint:arXiv:2202\.12359
- \[19\]OpenAI et al\.“GPT\-4 Technical Report”, 2023eprint:arXiv:2303\.08774
- \[20\]Philipp Schoenegger and Peter\. Park“Large Language Model Prediction Capabilities: Evidence from a Real\-World Forecasting Tournament”, 2023eprint:arXiv:2310\.13014
- \[21\]Mahdi Abolghasemi, Odkhishig Ganbold and Kristian Rotaru“Humans vs Large Language Models: Judgmental Forecasting in an Era of Advanced AI”, 2023eprint:arXiv:2312\.06941
- \[22\]Anthropic“Model card and evaluations for Claude models”, 2023URL:[https://www\-cdn\.anthropic\.com/bd2a28d2535bfb0494cc8e2a3bf135d2e752322verb6/Model\-Card\-Claude\-2\.pdf](https://www-cdn.anthropic.com/bd2a28d2535bfb0494cc8e2a3bf135d2e752322verb6/Model-Card-Claude-2.pdf)
- \[23\]Qi Yan, Raihan Seraj, Jiawei He, Lili Meng and Tristan Sylvain“AutoCast\+\+: Enhancing World Event Prediction with Zero\-shot Ranking\-based Context Retrieval”, 2023eprint:arXiv:2310\.01880
- \[24\]Qingchuan Yang, Simon Mahns, Sida Li, Anri Gu, Jibang Wu and Haifeng Xu“LLM\-as\-a\-Prophet: Understanding Predictive Intelligence with Prophet Arena”, 2025eprint:arXiv:2510\.17638
- \[25\]Zhiyuan Zeng et al\.“FutureX: An advanced live benchmark for LLM agents in future prediction”, 2025arXiv:[2508\.11987 \[cs\.AI\]](https://arxiv.org/abs/2508.11987)
- \[26\]Metaculus“Q1 AI Benchmarking Results: Pro Forecasters Crush Bots” Metaculus notebook, published May 2025,[https://www\.metaculus\.com/notebooks/38673/q1\-ai\-benchmarking\-results/](https://www.metaculus.com/notebooks/38673/q1-ai-benchmarking-results/), 2025URL:[https://www\.metaculus\.com/notebooks/38673/q1\-ai\-benchmarking\-results/](https://www.metaculus.com/notebooks/38673/q1-ai-benchmarking-results/)
- \[27\]Elvis Hsieh, Preston Fu and Jonathan Chen“Reasoning and tools for human\-level forecasting”, 2024arXiv:[2408\.12036 \[cs\.LG\]](https://arxiv.org/abs/2408.12036)
- \[28\]Zhixin Han et al\.“FutureWorld: A live reinforcement learning environment for predictive agents with real\-world outcome rewards”, 2026arXiv:[2604\.26733 \[cs\.AI\]](https://arxiv.org/abs/2604.26733)
- \[29\]Philipp Schoenegger, Indre Tuminauskaite, Peter Park, Rafael Valdece Bastos and Philip Tetlock“Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy”In*Science Advances*10\.45American Association for the Advancement of Science, 2024, pp\. eadp1528
- \[30\]Yuante Li, Yicheng Tao, Kate Zhang, Taozhi Wang, Gefei Gu and Yaxin Zhou“Diverse evidence, better forecasts: Multi\-agent deliberation under information asymmetry”, 2026arXiv:[2607\.01661 \[cs\.AI\]](https://arxiv.org/abs/2607.01661)
- \[31\]Benjamin Turtel, Danny Franklin and Philipp Schoenegger“LLMs can teach themselves to better predict the future”In*arXiv preprint arXiv:2502\.05253*, 2025
- \[32\]Benjamin Turtel, Danny Franklin, Kris Skotheim, Luke Hewitt and Philipp Schoenegger“Outcome\-based reinforcement learning to predict the future”, 2025arXiv:[2505\.17989 \[cs\.LG\]](https://arxiv.org/abs/2505.17989)
- \[33\]Scott Jeen, Matthew Aitchison and Mantic“Training LLMs to Predict World Events” https://thinkingmachines\.ai/news/training\-llms\-to\-predict\-world\-events/In*Thinking Machines Lab: News*, 2026
- \[34\]Lukas Fluri, Daniel Paleka and Florian Tramèr“Evaluating Superhuman Models with Consistency Checks”, 2023eprint:arXiv:2306\.09983
- \[35\]Sarah Pratt, Seth Blumberg, Pietro Carolino and Meredith Morris“Can Language Models Use Forecasting Strategies?”, 2024eprint:arXiv:2406\.04446
- \[36\]Daniel Paleka, Abhimanyu Sudhir, Alejandro Alvarez, Vineeth Bhat, Adam Shen, Evan Wang and Florian Tramèr“Consistency Checks for Language Model Forecasters”, 2024eprint:arXiv:2412\.18544
- \[37\]Suproteem Sarkar and Keyon Vafa“Lookahead bias in pretrained language models”In*SSRN Electron\. J\.*Elsevier BV, 2024
- \[38\]Perplexity AI“Perplexity Search API”URL:[https://docs\.perplexity\.ai/docs/search/quickstart](https://docs.perplexity.ai/docs/search/quickstart)
- \[39\]W\. James and Charles Stein“Estimation with quadratic loss”In*Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics*University of California Press, 1961, pp\. 361–379URL:[https://projecteuclid\.org/euclid\.bsmsp/1200512173](https://projecteuclid.org/euclid.bsmsp/1200512173)
- \[40\]A\.\. Baranchik“A Family of Minimax Estimators of the Mean of a Multivariate Normal Distribution”In*The Annals of Mathematical Statistics*41\.2Institute of Mathematical Statistics, 1970, pp\. 642–645DOI:[10\.1214/aoms/1177697104](https://dx.doi.org/10.1214/aoms/1177697104)
- \[41\]Bradley Efron and Carl Morris“Data Analysis Using Stein’s Estimator and its Generalizations”In*Journal of the American Statistical Association*70\.350Taylor & Francis, 1975, pp\. 311–319DOI:[10\.1080/01621459\.1975\.10479864](https://dx.doi.org/10.1080/01621459.1975.10479864)

## Appendix

## Appendix ARelated work

The present work builds on a rich literature in*judgmental forecasting*, which studies the application of human judgment to forecasting tasks\[[12](https://arxiv.org/html/2611.17735#bib.bibx12),[13](https://arxiv.org/html/2611.17735#bib.bibx13),[14](https://arxiv.org/html/2611.17735#bib.bibx14)\]\. We focus in particular on binary forecasting, in which the forecaster is presented with a hypothetical future event and tasked with estimating the probability that this event will occur\. Judgmental forecasting is often discussed in contrast to statistical forecasting; while the two can often play complementary roles, judgmental forecasting is particularly valuable in domains which lack sufficient data to build statistical models, or in which the available data does not fully capture the relevant dynamics\[[15](https://arxiv.org/html/2611.17735#bib.bibx15),[12](https://arxiv.org/html/2611.17735#bib.bibx12)\]\.

Recent advancements in general\-purpose large language models \(LLMs\) have heralded a significant uptick in work which seeks to*automate*judgmental forecasting\. Most modern work on LLM\-based forecasters can be traced to the Autocast benchmark\[[16](https://arxiv.org/html/2611.17735#bib.bibx16)\], which found that early language models like T5\[[17](https://arxiv.org/html/2611.17735#bib.bibx17)\]and UnifiedQA\-v2\[[18](https://arxiv.org/html/2611.17735#bib.bibx18)\]dramatically underperform aggregated human forecasts\. Follow\-up work similarly found that OpenAI’s GPT\-4 model\[[19](https://arxiv.org/html/2611.17735#bib.bibx19)\]underperforms aggregated human forecasts\[[20](https://arxiv.org/html/2611.17735#bib.bibx20),[21](https://arxiv.org/html/2611.17735#bib.bibx21)\]\. However, later work by\[[2](https://arxiv.org/html/2611.17735#bib.bibx2)\]showed that developing an appropriate harness around GPT\-4 \(or other similarly capable models, like the Claude\-2 series\[[22](https://arxiv.org/html/2611.17735#bib.bibx22)\]\) could approach – though not exceed – the performance of the human crowd\.

There are now a wide variety of benchmarks for evaluating the performance of LLM\-based forecasters, including the aforementioned Autocast\[[16](https://arxiv.org/html/2611.17735#bib.bibx16)\], the subsequent Autocast\+\+ benchmark\[[23](https://arxiv.org/html/2611.17735#bib.bibx23)\], ProphetArena\[[24](https://arxiv.org/html/2611.17735#bib.bibx24)\], ForecastBench\[[1](https://arxiv.org/html/2611.17735#bib.bibx1)\], FutureX\[[25](https://arxiv.org/html/2611.17735#bib.bibx25)\], the Metaculus Forecasting series\[[26](https://arxiv.org/html/2611.17735#bib.bibx26)\], and others\. These benchmarks vary significantly in their methodology — most notably, whether they are static or dynamically update over time, and whether each forecasting question includes the necessary context \(e\.g\., snippets from relevant news articles\) or whether the forecaster is charged with retrieving this context itself\. In this work we focus on ForecastBench\[[1](https://arxiv.org/html/2611.17735#bib.bibx1)\], a dynamically updating benchmark which includes only minimal context for each event\.

Our work builds directly on a complementary literature focused on methodology for LLM\-based forecasters\. The first line of work focuses on*retrieval*, or the process by which a forecasting agent gathers relevant context to inform its forecast\.\[[27](https://arxiv.org/html/2611.17735#bib.bibx27)\]extend the static retrieval pipeline of\[[2](https://arxiv.org/html/2611.17735#bib.bibx2)\]to allow forecasting agents to dynamically execute their own web search via tool calling;\[[28](https://arxiv.org/html/2611.17735#bib.bibx28)\]further develop a live reinforcement learning environment for training LLM\-based agents to perform open\-ended research before producing forecasts\. A separate line of work, inspired by earlier work on human superforecasters\[[14](https://arxiv.org/html/2611.17735#bib.bibx14)\], considers ensembling or deliberation among multiple language models to produce more accurate forecasts\[[29](https://arxiv.org/html/2611.17735#bib.bibx29),[30](https://arxiv.org/html/2611.17735#bib.bibx30)\]\. Finally,\[[31](https://arxiv.org/html/2611.17735#bib.bibx31),[32](https://arxiv.org/html/2611.17735#bib.bibx32),[33](https://arxiv.org/html/2611.17735#bib.bibx33)\]propose techniques for directly training language models to output more accurate \(and better calibrated\) forecasts after being provided with the appropriate context\.

Finally, the present work is perhaps most closely related to those which move beyond evaluating forecast accuracy to instead study the*consistency*of language model forecasts\.\[[34](https://arxiv.org/html/2611.17735#bib.bibx34)\]and\[[35](https://arxiv.org/html/2611.17735#bib.bibx35)\]show that early language models failed to produce self\-consistent forecasts; for example, by forecasting that sports records will evolve non\-monotonically over time, or that language model forecasts for an event and its negation fail to sum to11\.\[[36](https://arxiv.org/html/2611.17735#bib.bibx36)\]propose the use of such consistency checks as an evaluation measure in its own right, and show that consistency correlates with forecast accuracy\. Critically, consistency across forecasts can be evaluated instantaneously, while the accuracy of contemporaneous forecasts can only be evaluated later, once the outcomes for the corresponding events are known\. The primary focus of our work is to directly impose consistency on related forecasts without degrading their accuracy\.

## Appendix BA toy example

We work through a simple toy example to illustrate both the challenges and opportunities entailed by reasoning directly about the relationships between events\.

Recall our running example, in which we seek to refine forecasts related to U\.S\. election outcomes\. LetE1E\_\{1\}be the event\{candidate X wins by at least1%\}\\\{\\texttt\{candidate X wins by at least $1\\%$\}\\\}andE2E\_\{2\}be\{candidate X wins by at least5%\}\\\{\\texttt\{candidate X wins by at least $5\\%$\}\\\}\. Suppose the upstream forecaster gives forecasts ofp1\(0\)=0\.40p^\{\(0\)\}\_\{1\}=0\.40andp2\(0\)=0\.50p^\{\(0\)\}\_\{2\}=0\.50, respectively\.

Even in this highly stylized setting, we can immediately infer that at least one of these forecasts is incorrect\. In particular, becauseE2E\_\{2\}necessarily impliesE1E\_\{1\}, the true probabilities should satisfyp2⋆≤p1⋆p^\{\\star\}\_\{2\}\\leq p^\{\\star\}\_\{1\}, but the corresponding estimates violate this constraint\. This kind of incoherence is the rule rather than the exception: LLM\-based forecasters are queried independently for each event, and have no mechanism for enforcing even elementary forms of consistency across related forecasts\. As discussed in Appendix[A](https://arxiv.org/html/2611.17735#A1), this can be viewed as an instance of language models’ more general failure to form coherent internal models of the world\.

While detecting this kind of inconsistency is straightforward, it is far less clear what to do about it\. We could impose the constraint by settingp^1←p2\(0\)\\hat\{p\}\_\{1\}\\leftarrow p^\{\(0\)\}\_\{2\}, but this arbitrarily privilegesp2\(0\)p^\{\(0\)\}\_\{2\}overp1\(0\)p^\{\(0\)\}\_\{1\}\. Why not setp^2←p1\(0\)\\hat\{p\}\_\{2\}\\leftarrow p^\{\(0\)\}\_\{1\}instead? More generally, the feasible set\{p^1,p^2∣p^2≤p^1,0≤p^1,p^2≤1\}\\\{\\hat\{p\}\_\{1\},\\hat\{p\}\_\{2\}\\mid\\hat\{p\}\_\{2\}\\leq\\hat\{p\}\_\{1\},0\\leq\\hat\{p\}\_\{1\},\\hat\{p\}\_\{2\}\\leq 1\\\}only weakly constrains the set of allowed forecasts\. In the interest of fairness, we could perform a Euclidean projection onto this set, setting𝐩^=arg⁡minp2≤p1⁡‖𝐩−𝐩\(0\)‖2=\(0\.45,0\.45\)\\hat\{\\mathbf\{p\}\}=\\arg\\min\_\{p\_\{2\}\\leq p\_\{1\}\}\\\|\\mathbf\{p\}\-\\mathbf\{p\}^\{\(0\)\}\\\|\_\{2\}=\(0\.45,0\.45\)\. However, whether this is an improvement depends entirely on the unobserved value of𝐩⋆\\mathbf\{p\}^\{\\star\}\. If𝐩⋆=\(0\.50,0\.40\)\\mathbf\{p\}^\{\\star\}=\(0\.50,0\.40\), we have improved both forecasts; if𝐩⋆=\(0\.40,0\.30\)\\mathbf\{p\}^\{\\star\}=\(0\.40,0\.30\), thenp1\(0\)p^\{\(0\)\}\_\{1\}was correct and we have needlessly corrupted it\. The same will be true of any projection method which is agnostic as to the relative quality ofp1\(0\)p^\{\(0\)\}\_\{1\}andp2\(0\)p^\{\(0\)\}\_\{2\}\.

This example reveals two things\. First, consistency is often desirable in its own right: forecasts are most valuable as inputs to decision making, and mutually contradictory numbers offer little clarity\. Second, even if consistency is a secondary concern, examining both forecasts at once reveals an incoherence that we could not have discovered by examining either one in isolation\. This in turn offers an opportunity to improve the*accuracy*of our forecasts\. However, as this example also reveals, naive projections run the risk of simply “stealing” strength \(rather than sharing it\) across related forecasts\.

We argue that the missing ingredient is precisely the one which distinguishes LLM\-based forecasts from other forms of statistical forecasting: the evidence and reasoning that accompany each estimate\. The two forecasts will in general reflect different, possibly competing perspectives\. For example, the reasoning forp1\(0\)p^\{\(0\)\}\_\{1\}may focus on nomination procedures and intra\-party coalition management, while the reasoning forp2\(0\)p^\{\(0\)\}\_\{2\}may focus on macroeconomic conditions and general election polling\. By reasoning over the union of this evidence, we can in principle reconcile it into a single, coherent view of the world\. We develop such a method in this paper\.

## Appendix CLearning process

Below we describe additional learning process details, building on Section[3\.2](https://arxiv.org/html/2611.17735#S3.SS2)\.

### C\.1Extracting forecasts

Instead of directly prompting the model to return a probability, we instruct the model to output “yes” if the event described inqjq\_\{j\}is going to occur and “no” otherwise\.

We then obtain the refined forecast

p^j=Pθ​\(“Yes”∣promptj\)Pθ​\(“Yes”∣promptj\)\+Pθ​\(“No”∣promptj\)\\hat\{p\}\_\{j\}=\\frac\{P\_\{\\theta\}\(\\text\{\`\`Yes''\}\\mid\\mathrm\{prompt\}\_\{j\}\)\}\{P\_\{\\theta\}\(\\text\{\`\`Yes''\}\\mid\\mathrm\{prompt\}\_\{j\}\)\+P\_\{\\theta\}\(\\text\{\`\`No''\}\\mid\\mathrm\{prompt\}\_\{j\}\)\}
whereθ\\thetadenotes the model parameters\. We use this formulation, rather thanPθ​\(“Yes”∣promptj\)P\_\{\\theta\}\(\\text\{\`\`Yes''\}\\mid\\mathrm\{prompt\}\_\{j\}\)directly, to exclude the \(typically negligible\) probability mass the model assigns to the rest of the token vocabulary\.

### C\.2Metric Definitions and Loss function

Letℬ⊆I\\mathcal\{B\}\\subseteq Idenote the set of indices in a batch of training examples, and letNℬ=\|ℬ\|N\_\{\\mathcal\{B\}\}=\|\\mathcal\{B\}\|denote the size of the batch\. We batch by full clusters to ensure that all intra\-cluster relations are present in the same forward pass\. Importantly, we do not directly observe outcomes for the synthetic questions\. Thus, the synthetic questions enter the training objective only through the consistency and proximity terms, which we detail below\. We evaluate the Brier score over the original set of questionsq1,…,qnq\_\{1\},\\ldots,q\_\{n\}\.

Let

ℛ=\{i∈ℬ:∃Yi∈\{0,1\}​associated with​i\}\\mathcal\{R\}=\\\{i\\in\\mathcal\{B\}:\\exists Y\_\{i\}\\in\\\{0,1\\\}~\\mbox\{associated with\}~i\\\}be the subset of events with observed outcomes, and letNℛ=\|ℛ\|N\_\{\\mathcal\{R\}\}=\|\\mathcal\{R\}\|be the number of such events\. Letθ\\thetadenote the model parameters and letp^i=p^i​\(θ\)\\hat\{p\}\_\{i\}=\\hat\{p\}\_\{i\}\(\\theta\)denote the refined probability the model, parameterized byθ\\theta, assigns to eventii\. The loss function combines three terms\.

The first term is the Brier score, which evaluates the accuracy of the forecasts:

ℒbrier​\(θ,ℬ\)=1Nℛ​∑i∈ℛ\(p^i−Yi\)2\.\\mathcal\{L\}\_\{\\mathrm\{brier\}\}\(\\theta;\\mathcal\{B\}\)=\\frac\{1\}\{N\_\{\\mathcal\{R\}\}\}\\sum\_\{i\\in\\mathcal\{R\}\}\(\\hat\{p\}\_\{i\}\-Y\_\{i\}\)^\{2\}\.
The second term is the proximity term, which regularizes the refined forecasts towards the upstream estimates:

ℒprox​\(θ,ℬ\)=1Nℬ​∑i∈ℬ\(p^i−pi\(0\)\)2\.\\mathcal\{L\}\_\{\\mathrm\{prox\}\}\(\\theta;\\mathcal\{B\}\)=\\frac\{1\}\{N\_\{\\mathcal\{B\}\}\}\\sum\_\{i\\in\\mathcal\{B\}\}\(\\hat\{p\}\_\{i\}\-p^\{\(0\)\}\_\{i\}\)^\{2\}\.
The third term is the consistency term, which penalizes violations of the constraint families𝒞\\mathcal\{C\},𝒫\\mathcal\{P\}, andℳ\\mathcal\{M\}\. Let𝒞ℬ\\mathcal\{C\}\_\{\\mathcal\{B\}\},𝒫ℬ\\mathcal\{P\}\_\{\\mathcal\{B\}\}, andℳℬ\\mathcal\{M\}\_\{\\mathcal\{B\}\}denote the subsets of the corresponding constraint families whose members all appear in the batchℬ\\mathcal\{B\}\. For example,

𝒞ℬ=\{\(a,b\)∈𝒞:a,b∈ℬ\}\.\\mathcal\{C\}\_\{\\mathcal\{B\}\}=\\\{\(a,b\)\\in\\mathcal\{C\}:a,b\\in\\mathcal\{B\}\\\}\.Each constraint family contributes the following terms:

ℓ𝒞​\(a,b\)\\displaystyle\\ell\_\{\\mathcal\{C\}\}\(a,b\)=\(p^a\+p^b−1\)2,\\displaystyle=\(\\hat\{p\}\_\{a\}\+\\hat\{p\}\_\{b\}\-1\)^\{2\},\(a,b\)∈𝒞ℬ,\\displaystyle\(a,b\)\\in\\mathcal\{C\}\_\{\\mathcal\{B\}\},ℓ𝒫​\(S\)\\displaystyle\\ell\_\{\\mathcal\{P\}\}\(S\)=\(∑i∈Sp^i−1\)2,\\displaystyle=\\left\(\\sum\_\{i\\in S\}\\hat\{p\}\_\{i\}\-1\\right\)^\{2\},S∈𝒫ℬ,\\displaystyle S\\in\\mathcal\{P\}\_\{\\mathcal\{B\}\},ℓℳ​\(m\)\\displaystyle\\ell\_\{\\mathcal\{M\}\}\(m\)=1km−1​∑r=1km−1\[p^ir−p^ir\+1\]\+,\\displaystyle=\\frac\{1\}\{k\_\{m\}\-1\}\\sum\_\{r=1\}^\{k\_\{m\}\-1\}\[\\hat\{p\}\_\{i\_\{r\}\}\-\\hat\{p\}\_\{i\_\{r\+1\}\}\]\_\{\+\},m=\(i1,…,ikm\)∈ℳℬ,\\displaystyle m=\(i\_\{1\},\\ldots,i\_\{k\_\{m\}\}\)\\in\\mathcal\{M\}\_\{\\mathcal\{B\}\},where complement and partition constraints penalize squared deviations from the constraint set, while monotone chains use a hinge loss \(\[⋅\]\+=max⁡\{⋅,0\}\[\\cdot\]\_\{\+\}=\\max\\\{\\cdot,0\\\}\)\. The monotone hinge loss is averaged over thekm−1k\_\{m\}\-1adjacent members of each chain so that its magnitude is invariant to chain length\.

We weight each term by the corresponding hyperparametersα𝒞,α𝒫,αℳ≥0\\alpha\_\{\\mathcal\{C\}\},\\alpha\_\{\\mathcal\{P\}\},\\alpha\_\{\\mathcal\{M\}\}\\geq 0and define

Ncons=\|𝒞ℬ\|\+\|𝒫ℬ\|\+\|ℳℬ\|\.N\_\{\\mathrm\{cons\}\}=\|\\mathcal\{C\}\_\{\\mathcal\{B\}\}\|\+\|\\mathcal\{P\}\_\{\\mathcal\{B\}\}\|\+\|\\mathcal\{M\}\_\{\\mathcal\{B\}\}\|\.This gives the consistency loss

ℒcons​\(θ,ℬ\)=1Ncons​\(α𝒞​∑\(a,b\)∈𝒞ℬℓ𝒞​\(a,b\)\+α𝒫​∑S∈𝒫ℬℓ𝒫​\(S\)\+αℳ​∑m∈ℳℬℓℳ​\(m\)\)\.\\mathcal\{L\}\_\{\\mathrm\{cons\}\}\(\\theta;\\mathcal\{B\}\)=\\frac\{1\}\{N\_\{\\mathrm\{cons\}\}\}\\left\(\\alpha\_\{\\mathcal\{C\}\}\\sum\_\{\(a,b\)\\in\\mathcal\{C\}\_\{\\mathcal\{B\}\}\}\\ell\_\{\\mathcal\{C\}\}\(a,b\)\+\\alpha\_\{\\mathcal\{P\}\}\\sum\_\{S\\in\\mathcal\{P\}\_\{\\mathcal\{B\}\}\}\\ell\_\{\\mathcal\{P\}\}\(S\)\+\\alpha\_\{\\mathcal\{M\}\}\\sum\_\{m\\in\\mathcal\{M\}\_\{\\mathcal\{B\}\}\}\\ell\_\{\\mathcal\{M\}\}\(m\)\\right\)\.We adopt the convention thatℒcons​\(θ,ℬ\)=0\\mathcal\{L\}\_\{\\mathrm\{cons\}\}\(\\theta;\\mathcal\{B\}\)=0whenNcons=0N\_\{\\mathrm\{cons\}\}=0\.

Finally, the training loss is a convex combination of the preceding three terms, with weightswbrier,wprox,wcons≥0w\_\{\\mathrm\{brier\}\},w\_\{\\mathrm\{prox\}\},w\_\{\\mathrm\{cons\}\}\\geq 0satisfyingwbrier\+wprox\+wcons=1w\_\{\\mathrm\{brier\}\}\+w\_\{\\mathrm\{prox\}\}\+w\_\{\\mathrm\{cons\}\}=1:

ℒ⁡\(θ,ℬ\)=wbrier​ℒbrier​\(θ,ℬ\)\+wprox​ℒprox​\(θ,ℬ\)\+wcons​ℒcons​\(θ,ℬ\)\.\\mathcal\{L\}\(\\theta;\\mathcal\{B\}\)=w\_\{\\mathrm\{brier\}\}\\mathcal\{L\}\_\{\\mathrm\{brier\}\}\(\\theta;\\mathcal\{B\}\)\+w\_\{\\mathrm\{prox\}\}\\mathcal\{L\}\_\{\\mathrm\{prox\}\}\(\\theta;\\mathcal\{B\}\)\+w\_\{\\mathrm\{cons\}\}\\mathcal\{L\}\_\{\\mathrm\{cons\}\}\(\\theta;\\mathcal\{B\}\)\.

### C\.3Experiment details and hyperparameters

- •Base model:Qwen3\.6\-35B\-A3B\.
- •Adapter:LoRA with rank3232\.
- •Prompt rendering:Qwen chat template with thinking disabled\.
- •Optimizer:Adam with a constant learning rate of10−510^\{\-5\}\.
- •Epochs:22\.
- •Batch Size:Target batch size of1616events\. We greedily pack complete clusters into each batch, so the realized size of a batch may be smaller than1616\(or larger in the case of a single oversized cluster\)\.
- •Loss weights:\(wbrier,wprox,wcons\)=\(0\.33,0\.34,0\.33\)\(w\_\{\\mathrm\{brier\}\},w\_\{\\mathrm\{prox\}\},w\_\{\\mathrm\{cons\}\}\)=\(0\.33,0\.34,0\.33\)\.
- •Consistency scale weights:\(α𝒞,α𝒫,αℳ\)=\(1,1,5\)\(\\alpha\_\{\\mathcal\{C\}\},\\alpha\_\{\\mathcal\{P\}\},\\alpha\_\{\\mathcal\{M\}\}\)=\(1,1,5\)for complement, multi\-way partition, and monotonicity constraints, respectively\.
- •Checkpoint selection:We evaluate on the validation set every 20 optimization steps\. As described in Section[4](https://arxiv.org/html/2611.17735#S4), we follow ForecastBench’s methodology of separately computing evaluation metrics on the two source types \(market and dataset sources\) and then computing their equal\-weighted average\. Thus, for each checkpoint, we compute the validation loss separately for market and dataset question clusters\. LetΘ\\Thetadenote the set of evaluated checkpoints, and let𝒱market\\mathcal\{V\}\_\{\\mathrm\{market\}\}and𝒱dataset\\mathcal\{V\}\_\{\\mathrm\{dataset\}\}denote the full set of validation questions in market and dataset clusters respectively\. We select the checkpoint used for test\-set evaluation as θ⋆∈arg⁡minθ∈Θ⁡\[12​ℒ​\(θ,𝒱market\)\+12​ℒ​\(θ,𝒱dataset\)\]\.\\theta^\{\\star\}\\in\\arg\\min\_\{\\theta\\in\\Theta\}\\left\[\\tfrac\{1\}\{2\}\\mathcal\{L\}\(\\theta;\\mathcal\{V\}\_\{\\mathrm\{market\}\}\)\+\\tfrac\{1\}\{2\}\\mathcal\{L\}\(\\theta;\\mathcal\{V\}\_\{\\mathrm\{dataset\}\}\)\\right\]\.
- •Random seed:31,32,3331,32,33, used for adapter initialization and training order shuffling\.

## Appendix DSynthetic question generation

Here we expand on synthetic question generation as described in Section[3\.1](https://arxiv.org/html/2611.17735#S3.SS1)\. Recall thatJiJ\_\{i\}indexes the synthetic questions associated withqiq\_\{i\}\. We can partitionJiJ\_\{i\}into subsets which correspond to the constraint families𝒞\\mathcal\{C\},𝒫\\mathcal\{P\}, andℳ\\mathcal\{M\}as follows:

Ji=Ji𝒞∪Ji𝒫∪Jiℳ,J\_\{i\}=J\_\{i\}^\{\\mathcal\{C\}\}\\cup J\_\{i\}^\{\\mathcal\{P\}\}\\cup J\_\{i\}^\{\\mathcal\{M\}\},whereJi𝒞J\_\{i\}^\{\\mathcal\{C\}\}indexes the complementq¯i\\bar\{q\}\_\{i\},Ji𝒫J\_\{i\}^\{\\mathcal\{P\}\}the siblingsqi1,…,qikq\_\{i\}^\{1\},\\dots,q\_\{i\}^\{k\}, andJiℳJ\_\{i\}^\{\\mathcal\{M\}\}the nested chainq~i1,…,q~iℓ\\tilde\{q\}\_\{i\}^\{1\},\\dots,\\tilde\{q\}\_\{i\}^\{\\ell\}\.

These subsets correspond directly to the corresponding constraint families:Ji𝒞J\_\{i\}^\{\\mathcal\{C\}\}contributes the complement pair\(i,j\)∈𝒞\(i,j\)\\in\\mathcal\{C\},Ji𝒫J\_\{i\}^\{\\mathcal\{P\}\}gives the partition constraint\{i\}∪Ji𝒫∈𝒫\\\{i\\\}\\cup J\_\{i\}^\{\\mathcal\{P\}\}\\in\\mathcal\{P\}, andJiℳJ\_\{i\}^\{\\mathcal\{M\}\}gives the monotone constraints on\{i\}∪Jiℳ\\\{i\\\}\\cup J\_\{i\}^\{\\mathcal\{M\}\}inℳ\\mathcal\{M\}\.

We validate the correctness of these synthetically generated questions with an LLM judge, retaining only those the judge deems valid\. To confirm the judge’s accuracy, we randomly sample5050judge\-approved clusters for each type of relationship and manually review them\.

This review discovered no false negatives — events which did not have the desired relationship to the original question but were not flagged by the judge\. We can thus upper bound the false negative rate for the \(hypothetical\) superpopulation at5\.82%5\.82\\%for questions corresponding to each constraint set\. We obtain this value simply by solving\(1−p\)50=0\.05\(1\-p\)^\{50\}=0\.05, which calculates the maximum false negative rate consistent with observing00false negatives over5050i\.i\.d\. samples\. Alternatively, we can pool all150150reviewed cases to bound the false negative rate at1\.98%1\.98\\%over all synthetic questions \(under the assumption that questions corresponding to the three constraint types are equally represented in our hypothetical superpopulation\)\. We conclude from this exercise that synthetic clustering behaves as desired\.

Table 3:LLM\-judge filtering and manual\-audit summary for synthetic relationship families in the test set\. The final column reports the one\-sided95%95\\%confidence bound on the judge’s false\-pass rate\.
## Appendix EForecasting data

In this section we elaborate on our description in Section[4](https://arxiv.org/html/2611.17735#S4)about our data sourced from ForecastBench\[[1](https://arxiv.org/html/2611.17735#bib.bibx1)\], a popular benchmark for evaluating the performance of LLM\-based forecasters\. As discussed in Appendix[A](https://arxiv.org/html/2611.17735#A1), ForecastBench is a dynamic benchmark which regularly publishes new questions about future events\. This is intended in part to enable evaluation after the model’s pretraining “knowledge cutoff”, which otherwise contaminates retrospective back testing\[[37](https://arxiv.org/html/2611.17735#bib.bibx37)\]\.

Each question in the dataset prompts the model to forecast whether a particular event will occur by a particular date \(the “resolution date”\)\. Questions span a wide variety of topics, including sports, economics, elections and public policy, technology, and others\. They are sourced from a combination of trusted, well\-maintained datasets \(e\.g\., the FRED economic database, Yahoo\! Finance\) and public prediction market platforms \(e\.g\., Manifold Markets and Polymarket\)\. The former are deemed dataset questions and the latter deemed market questions\. Performance metrics are measured as a equal\-weighted average over the market subset and the dataset subset of questions\. We defer to\[[1](https://arxiv.org/html/2611.17735#bib.bibx1)\]for a full treatment of the benchmark construction\.

We describe the specific release dates and cutoff criteria we use for each of our training, validation, and test sets in Section[4](https://arxiv.org/html/2611.17735#S4)\. The same underlying question can be contained in multiple releases, in essence asking the same question at slightly different forecasting dates\. Dataset questions can additionally contain multiple resolution\-date variants of the same underlying question in a single release\. In our test set, we deduplicate instances of the former but not the latter\. We randomly select one release date to keep for each underlying question, and then keep all of its resolution\-date variants in that release\. We use random number generator seed 33\.

For our training and validation sets, we do additional deduplication and subsampling for efficiency\. First, we randomly select only one release date and one resolution date for each underlying question, deduplicating both types of repetitive questions mentioned above\. Then, as described in Section[4](https://arxiv.org/html/2611.17735#S4), we construct our training and validation sets to have an equal number of market and dataset source questions\. Since dataset questions outnumber market questions in both sets, we randomly sample a subset of eligible dataset questions for each set, stratifying by original source such that source proportions are maintained\. For random number generation, we use seeds 31 and 32 for training and validation respectively\.

## Appendix FUpstream forecaster

In this section we provide more details on our upstream forecaster, building on Section[4](https://arxiv.org/html/2611.17735#S4)\.

Retrieval\.We give GPT\-5\.1 access to agentic web search through the Perplexity API\[[38](https://arxiv.org/html/2611.17735#bib.bibx38)\]\. To prevent foreknowledge leakage, we restrict search results to sources published before the release date of each question by setting fieldssearch\_before\_date\_filterandlast\_updated\_before\_filterto that date\. As an additional foreknowledge prevention measure, we also explicitly disallow prediction market domains usingsearch\_domain\_filter\.

Prompt\.We use GPT\-5\.1\[[9](https://arxiv.org/html/2611.17735#bib.bibx9)\]with the following system prompt:

> You are a well\-calibrated forecaster\. Estimate P\(YES\) as of the forecast date under the question text and resolution criteria, the latter of which overrides the former if they conflict\.Answer from the perspective of the forecast date, as if it were the present; do not use knowledge of events or outcomes after that date, even when the forecast date is in the past\.Your estimate will be evaluated by its accuracy \(e\.g\., Brier score\), rewarding precision and penalizing overconfidence\.Return these structured fields:\- probability: P\(YES\) as a number from 0 to 1\- reasoning: a brief explanation of the estimate\- factors\_for: 1 to 5 concise reasons supporting YES\- factors\_against: 1 to 5 concise reasons supporting NO\- key\_uncertainties: 1 to 5 concise unresolved drivers

We use the following user prompt template:

> Forecast date: <forecast date\>Resolution date: <resolution date\>Question:<question\>Resolution criteria:<resolution criteria\>Background information:<background information\>

## Appendix GDetailed results

Table[4](https://arxiv.org/html/2611.17735#A7.T4)shows our headline results broken down by seed\.

Table 4:Per\-seed breakdown of our main results shown in Table[1](https://arxiv.org/html/2611.17735#S4.T1)\.*Raw*denotes the upstream forecaster and is shared across seeds; lower is better for every metric\. Each column reports the metric value with its relative change from Raw in parentheses \(negative is better\)\. Brier score is computed over the resolved original questions, whereas consistency metrics are computed over the corresponding augmented clusters\. Each metric is an equal\-weighted average over the243243\-question market subset and the1,3441,344\-question dataset subset\.We report the three\-seed mean separately for market and dataset questions in Table[5](https://arxiv.org/html/2611.17735#A7.T5)\.

Table 5:Our main results by question source\. Each entry reports Raw→\\rightarrowModel, with the relative change from Raw in parentheses\. Model values are means across the three training seeds\. Lower is better for every metric\.Below we also show the Brier change of the complement questions, since their outcomes can be derived\. We include a seed\-level breakdown\. Averaged across seeds, our method reduces complement Brier score by20\.05%20\.05\\%\. This improvement is statistically significant under a paired bootstrap test \(two\-sidedp=0\.0002p=0\.0002with 10,000 bootstrap replicates\)\. See Appendix[I](https://arxiv.org/html/2611.17735#A9)for uncertainty quantification methodology\.

Table 6:Brier score over the complement questions in our test set, computed as an equal\-weighted average over the243243\-question market subset and1,3441,344\-question dataset subset\.*Raw*denotes the upstream forecaster and is shared across seeds\.Δ\\Deltais Model minus Raw, so negative values indicate improvement\. The average row is the mean of the per\-seed squared losses\.
## Appendix HBaselines and ablations

### H\.1Baselines

We consider five baseline methods\. As Table[2](https://arxiv.org/html/2611.17735#S4.T2)illustrates, none of the baseline methods match our method in both consistency and accuracy improvements, and Appendix[I](https://arxiv.org/html/2611.17735#A9)demonstrates that the gap is statistically significant\. Here we provide a brief description of each baseline\. More detailed results for each baseline are provided in Appendix[J](https://arxiv.org/html/2611.17735#A10)\.

Ensemble scaling\.Recall that, for each eventEiE\_\{i\}, the input to our method consists of reasoningziz\_\{i\}along with the probability estimatepi\(0\)p^\{\(0\)\}\_\{i\}output by an upstream forecaster\. We ask whether producing an ensemble over ten independent rollouts for the*same*event is sufficient to reproduce our results\. This baseline is motivated by\[[3](https://arxiv.org/html/2611.17735#bib.bibx3)\], which attributes substantial improvements in accuracy to this form of ensembling\. Because our method also requires additional rollouts of the upstream forecaster,333Across our1,5871,587test clusters, the mean cluster size is7\.237\.23questions \(with a standard deviation of1\.631\.63\), including the original question\.ensemble scaling tests whether its gains arise merely from a similar form of “test\-time scaling,” rather than from meaningfully leveraging relationships between different events\.

James–Stein shrinkage\.Next, we consider whether our method can be distinguished from a statistical correction which simply regularizes each estimate toward a fixed point, discarding the associated reasoning as well as the nature of the relationships between events\. In particular, we consider positive\-part James\-Stein shrinkage, which we implement by regularizing each initial estimatepi\(0\)p^\{\(0\)\}\_\{i\}toward the mean of the corresponding clusterp¯c\\bar\{p\}\_\{c\}:

p^i=\(1−λc\)​pi\(0\)\+λc​p¯c,λc=min⁡\{1,max⁡\{0,\(kc−3\)​σ¯c2∑i∈c\(pi\(0\)−p¯c\)2\}\},\\hat\{p\}\_\{i\}=\(1\-\\lambda\_\{c\}\)p^\{\(0\)\}\_\{i\}\+\\lambda\_\{c\}\\bar\{p\}\_\{c\},\\qquad\\lambda\_\{c\}=\\min\\left\\\{1,\\max\\left\\\{0,\\frac\{\(k\_\{c\}\-3\)\\bar\{\\sigma\}\_\{c\}^\{2\}\}\{\\sum\_\{i\\in c\}\(p^\{\(0\)\}\_\{i\}\-\\bar\{p\}\_\{c\}\)^\{2\}\}\\right\\\}\\right\\\},
wherekc=\|c\|k\_\{c\}=\|c\|is the cluster size\. To estimateσ¯c2\\bar\{\\sigma\}\_\{c\}^\{2\}, we use the same ten independent rollouts of each question that we used for the ensemble scaling baseline\. Letpi​rp\_\{ir\}denote the forecast from rolloutr∈\[10\]r\\in\[10\]for eventEiE\_\{i\}\. Recall thatpi\(0\)=pi​1p^\{\(0\)\}\_\{i\}=p\_\{i1\}\. Letp¯i=110​∑r=110pi​r\\bar\{p\}\_\{i\}=\\frac\{1\}\{10\}\\sum\_\{r=1\}^\{10\}p\_\{ir\}\. We estimate the varianceσ¯c2\\bar\{\\sigma\}\_\{c\}^\{2\}as

σ^i2=1\(10−1\)​∑r=110\(pi​r−p¯i\)2,σ¯c2=1kc​∑i∈cσ^i2\.\\hat\{\\sigma\}\_\{i\}^\{2\}=\\frac\{1\}\{\(10\-1\)\}\\sum\_\{r=1\}^\{10\}\(p\_\{ir\}\-\\bar\{p\}\_\{i\}\)^\{2\},\\qquad\\bar\{\\sigma\}\_\{c\}^\{2\}=\\frac\{1\}\{k\_\{c\}\}\\sum\_\{i\\in c\}\\hat\{\\sigma\}\_\{i\}^\{2\}\.We consider shrinking towards zero as opposed to the cluster mean, as well as heteroskedastic and log\-odds variants of the James–Stein estimator in Appendix[J\.2](https://arxiv.org/html/2611.17735#A10.SS2)\.

Platt scaling\.Next we consider*Platt Scaling*, which instead*extremizes*forecasts towards either00or11\. This approach is motivated by the empirical observation that language models tend to hedge binary forecasts towards0\.50\.5, and are otherwise subject to various forms of miscalibration\[[3](https://arxiv.org/html/2611.17735#bib.bibx3),[35](https://arxiv.org/html/2611.17735#bib.bibx35)\]\. Following\[[3](https://arxiv.org/html/2611.17735#bib.bibx3)\], we set

p^i=σ⁡\(3​logit⁡\(pi\(0\)\)\)\.\\hat\{p\}\_\{i\}=\\sigma\\\!\\left\(\\sqrt\{3\}\\operatorname\{logit\}\(p^\{\(0\)\}\_\{i\}\)\\right\)\.
Like the James\-Stein shrinkage baseline, this statistical correction operates only on the estimatespi\(0\)p^\{\(0\)\}\_\{i\}and discards the rest of the input\.

Projection onto the consistent set\.We consider a Euclidean projection of the estimatespi\(0\)p^\{\(0\)\}\_\{i\}onto the set of consistent forecasts\. Let𝒦⊆\[0,1\]\|I\|\\mathcal\{K\}\\subseteq\[0,1\]^\{\|I\|\}denote the set of forecasts which satisfy the constraints in𝒞,𝒫\\mathcal\{C\},\\mathcal\{P\}andℳ\\mathcal\{M\}\. We compute

𝐩^=arg⁡min𝐩∈𝒦⁡‖𝐩−𝐩\(0\)‖22\.\\hat\{\\mathbf\{p\}\}=\\arg\\min\_\{\\mathbf\{p\}\\in\\mathcal\{K\}\}\\\|\\mathbf\{p\}\-\\mathbf\{p\}^\{\(0\)\}\\\|\_\{2\}^\{2\}\.
This projection enforces the constraints via the smallest possible perturbation to the input estimates\. As discussed in Appendix[B](https://arxiv.org/html/2611.17735#A2), this kind of projection does not privilege one estimate over another; in particular, it does not leverage the reasoning𝐳\\mathbf\{z\}to distinguish which estimates might be more accurate than others\. We report additional variations of this baseline in Appendix[J\.4](https://arxiv.org/html/2611.17735#A10.SS4)\.

Zero\-shot LLM\.Finally, we prompt GPT\-5\.1 \(the same LLM which underpins the upstream forecaster\) with each cluster of questions, their initial forecasts, and all the explicit consistency relationships to enforce, and we ask for updated probabilities for all the questions that are reconciled to be consistent\. The full prompt is provided in Appendix[J\.5](https://arxiv.org/html/2611.17735#A10.SS5)\.

### H\.2Ablations

We now ablate the three primary components of our methodology: providing the model with cross\-question context, imposing an explicit consistency objective, and incorporating reasoning traces from the upstream forecaster\. These ablation experiments reveal that each of these three ingredients is necessary to achieve our improvements in both accuracy and consistency across forecasts\. We also substitute our finetuned model with GPT\-5\.1 off\-the\-shelf and find that it underperforms our finetuned model\. Below we provide a brief description of each ablation\. We summarize the results of these ablation experiments in Table[7](https://arxiv.org/html/2611.17735#A8.T7)below\. Additional details for each ablation are provided in Appendix[K](https://arxiv.org/html/2611.17735#A11)\.

No cross\-question context\.We first consider removing the model’s ability to reconcile context across distinct but related questions\. Instead, given a target eventEiE\_\{i\}and corresponding questionqiq\_\{i\}, we retain the original upstream forecast\(pi\(0\),zi\)\(p\_\{i\}^\{\(0\)\},z\_\{i\}\)and generate nine additional independent rollouts\(pi\(r\),zi\(r\)\)\(p\_\{i\}^\{\(r\)\},z\_\{i\}^\{\(r\)\}\)forr∈\{1,…,9\}r\\in\\\{1,\\ldots,9\\\}for the*same*event\. Each set of ten forecasts forms a distinct cluster, allowing our model to reconcile the reasoning across multiple upstream forecasts while preventing it from leveraging relationships between distinct events\. The loss function substitutes the cross\-question consistency term for the average of the squared penalties on each pairwise difference between predicted probabilities within the cluster\. For a clean comparison with our main experiment and to isolate the effect of cross\-question context from ensembling \(which is a separate baseline discussed in Appendix[H\.1](https://arxiv.org/html/2611.17735#A8.SS1)\), we compute the updated Brier score using only the updated forecastp^i\\hat\{p\}\_\{i\}corresponding to the original upstream forecastpi\(0\)p\_\{i\}^\{\(0\)\}, rather than ensembling the updated forecasts from all ten rollouts\. We measure consistency across related questions by similarly generating nine additional rollouts of each synthetic question, running them through this ablation model, and calculating consistency metrics on thep^i\\hat\{p\}\_\{i\}estimates of each original question and its synthetic related questions\.

No explicit consistency objective\.Next we consider whether the model can learn to produce mutually coherent forecasts without an explicit supervision signal for consistency\. In particular, we remove the consistency term from the loss function while maintaining the overall weight of the regularization term, which gives the following modified training objective:

ℒ⁡\(θ,ℬ\)=0\.66​ℒbrier​\(θ,ℬ\)\+0\.34​ℒprox​\(θ,ℬ\)\.\\mathcal\{L\}\(\\theta;\\mathcal\{B\}\)=0\.66\\,\\mathcal\{L\}\_\{\\mathrm\{brier\}\}\(\\theta;\\mathcal\{B\}\)\+0\.34\\,\\mathcal\{L\}\_\{\\mathrm\{prox\}\}\(\\theta;\\mathcal\{B\}\)\.
The training and evaluation setup is otherwise as described in Section[4](https://arxiv.org/html/2611.17735#S4)\. In particular, the model is still presented with estimates and reasoning traces across distinct but related questions, and can in principle learn to produce consistent forecasts as a consequence of maximizing accuracy\.

No reasoning traces\.Next, we revoke the model’s ability to inspect the reasoning produced by the upstream forecaster\. In this experiment, we simply remove the traceziz\_\{i\}from each cluster while otherwise retaining the structure of the prompt\. The model still retains access to the questionsqiq\_\{i\}and the corresponding estimatespi\(0\)p^\{\(0\)\}\_\{i\}, and can therefore learn both to identify inconsistent forecasts and to perform statistical corrections\. However, it cannot inspect the evidence used to produce each estimate to decide*how*to reconcile inconsistent forecasts\.

Frontier LLM\.Finally, we consider the use of a frontier language model\. We again choose OpenAI’s GPT\-5\.1\[[9](https://arxiv.org/html/2611.17735#bib.bibx9)\]model to remain consistent with the upstream forecaster\. We prompt this model identically to our own method, with additional instructions to directly provide a probability estimate\. Recall, as described in Section[3\.2](https://arxiv.org/html/2611.17735#S3.SS2)that our method infers these probabilities from the logits corresponding to the “YES” and “NO” tokens; this is infeasible for GPT\-5\.1, which does expose these logits\.

Table 7:The relative improvement of each ablation method with respect to*Raw*; negative values indicate better performance\.Oursdenotes our method, with metrics averaged over three seeds\.

## Appendix ISignificance testing

We compute pairwise bootstrapped p\-values between our model and the raw upstream forecasts, as well as between our model and each baseline method, following the methodology of\[[1](https://arxiv.org/html/2611.17735#bib.bibx1)\]and\[[3](https://arxiv.org/html/2611.17735#bib.bibx3)\]\. This test based on pairwise differences is more informative than a traditional bootstrapped confidence interval around Brier score because in the latter case most of the variation is driven by inter\-question variability, and two models’ confidence intervals can look quite similar even if one model strictly outperforms the other on every question\.

We compute the difference in Brier score between our method and each alternative method for each question in the test set\. We then bootstrap sample with replacementN=10,000N=10\{,\}000times, stratifying by market\-question subset and dataset\-question subset, such that each sample is the same size as the test set and contains243243market questions and1,3441,344dataset questions\. We compute the average difference in Brier score for each bootstrap sample to get a distribution of averages \(when we compute the average, we separately calculate the market subset average and the dataset subset average, and then equal\-weight their combination\)\. We recenter this distribution at zero by subtracting average of averages \(the average of the bootstrap sample average differences\)\. Then, we compute the two\-sided p\-value of our original mean of Brier score differences \(equal\-weighted between market and dataset sources\) on this recentered bootstrap distribution\.

We similarly compute the paired bootstrap p\-value between our method and each alternative method for each consistency metric\. We use the same bootstrap samples across all metrics – clusters are sampled as a unit to preserve intra\-cluster relationships\.

#### Method vs\. raw\.

As Table[8](https://arxiv.org/html/2611.17735#A9.T8)illustrates, our method is significantly better in all metrics as compared with the raw upstream forecasts\.

Table 8:Two\-sided paired\-bootstrap p\-values comparing our method with the raw upstream forecasts on the test set\. Bold entries are significant at the 5% level; the superscript P indicates that our method has the lower metric\.
#### Baselines\.

Our method is significantly better on all computed metrics compared with James–Stein shrinkage and Platt scaling\. Our method has significantly lower consistency violations on all three measures as compared with ensemble scaling\. For the two methods that impose the constraints exactly, consistent\-set projection and the zero\-shot LLM, our method has significantly lower Brier score\.

Table 9:Two\-sided paired\-bootstrap p\-values comparing our method with each baseline on the test set\. Bold entries are significant at the 5% level; the superscript indicates whether our method \(P\) or the baseline \(B\) has the lower metric\.
#### Ablations\.

Our method has significantly lower Brier score than the no\-reasoning\-trace ablation\. It has significantly lower consistency violations than the no\-consistency\-loss, no\-cross\-question\-context, and the frontier LLM ablation\.

Table 10:Two\-sided paired\-bootstrap p\-values comparing our method with each ablation on the test set\. Bold entries are significant at the 5% level; the superscript indicates whether our method \(P\) or the ablation \(A\) has the lower metric\.

## Appendix JAdditional baseline details

Here we provide additional details on the baseline methods mentioned in Appendix[H\.1](https://arxiv.org/html/2611.17735#A8.SS1)\.

### J\.1Ensemble scaling

We investigate whether the increased number of independent upstream LLM forecaster runs that are now feeding each individual updated forecast is responsible for our model’s gains\. As a baseline, we run ten times as many independent upstream forecaster runs for each original question and take their ensemble average\. This results in Brier gains and consistency gains, but they are not as strong as our method\. See Section[I](https://arxiv.org/html/2611.17735#A9)for significance testing\.

Table 11:Arithmetic mean of ten independent GPT\-5\.1 \+ Perplexity forecasts on the test set\.
### J\.2Shrinkage

We investigate whether our method is merely implementing a form of regularization via shrinkage\. Assuming we are given the same clusters of related questions as described in section[3\.1](https://arxiv.org/html/2611.17735#S3.SS1), we can directly apply James\-Stein style shrinkage\[[39](https://arxiv.org/html/2611.17735#bib.bibx39)\]to the forecasts of each question in the cluster\.

For each cluster, we shrink each initial forecastpi\(0\)p^\{\(0\)\}\_\{i\}toward the cluster meanp¯c\\bar\{p\}\_\{c\}:

p^i=\(1−λc\)​pi\(0\)\+λc​p¯c\.\\hat\{p\}\_\{i\}=\(1\-\\lambda\_\{c\}\)p\_\{i\}^\{\(0\)\}\+\\lambda\_\{c\}\\bar\{p\}\_\{c\}\.We use the classical positive\-part James\-Stein coefficient\[[40](https://arxiv.org/html/2611.17735#bib.bibx40)\], which assumes a common variance across questions within each cluster:

λc=min⁡\{1,max⁡\{0,\(kc−3\)​σ¯c2∑i∈c\(pi\(0\)−p¯c\)2\}\}\.\\lambda\_\{c\}=\\min\\left\\\{1,\\max\\left\\\{0,\\frac\{\(k\_\{c\}\-3\)\\bar\{\\sigma\}\_\{c\}^\{2\}\}\{\\sum\_\{i\\in c\}\(p\_\{i\}^\{\(0\)\}\-\\bar\{p\}\_\{c\}\)^\{2\}\}\\right\\\}\\right\\\}\.Herekck\_\{c\}is the number of questions in clusterccandσ¯c2\\bar\{\\sigma\}\_\{c\}^\{2\}is the average sample variance across ten rollouts for each question in the cluster\.

We see that shrinkage toward the cluster mean does not meaningfully improve Brier score\.

Table 12:Classical positive\-part James\-Stein shrinkage on synthetic\-question clusters\.The properties of our initial probability forecasts don’t satisfy homoskedastic assumptions, or assumptions on being unbounded in value\. Below we report heteroskedastic empirical Bayes and log\-odds variations of this experiment\.

#### Heteroskedastic empirical Bayes\.

We also test a heteroskedastic empirical Bayes form of the shrinkage estimator\[[41](https://arxiv.org/html/2611.17735#bib.bibx41)\]\. Questions can have different levels of uncertainty, and we have empirical estimates of those uncertainty levels from the ten independent runs of each question\. For each cluster, we shrink each initial forecastpi\(0\)p\_\{i\}^\{\(0\)\}toward the cluster meanp¯c\\bar\{p\}\_\{c\}:

p^i=\(1−λi\)​pi\(0\)\+λi​p¯c,λi=σ^i2σ^i2\+τ^c2\.\\hat\{p\}\_\{i\}=\(1\-\\lambda\_\{i\}\)p\_\{i\}^\{\(0\)\}\+\\lambda\_\{i\}\\bar\{p\}\_\{c\},\\qquad\\lambda\_\{i\}=\\frac\{\\hat\{\\sigma\}\_\{i\}^\{2\}\}\{\\hat\{\\sigma\}\_\{i\}^\{2\}\+\\hat\{\\tau\}\_\{c\}^\{2\}\}\.Hereσ^i2\\hat\{\\sigma\}\_\{i\}^\{2\}is the sample variance across the ten rollouts for questionii, andτ^c2\\hat\{\\tau\}\_\{c\}^\{2\}is an estimate of the inter\-question dispersion within clustercc,

τ^c2=max⁡\(0,Sc2−σ^2¯c\),Sc2=1kc−1​∑i∈c\(pi\(0\)−p¯c\)2,\\hat\{\\tau\}\_\{c\}^\{2\}=\\max\\\!\\big\(0,\\;S\_\{c\}^\{2\}\-\\overline\{\\hat\{\\sigma\}^\{2\}\}\_\{c\}\\big\),\\qquad S\_\{c\}^\{2\}=\\frac\{1\}\{k\_\{c\}\-1\}\\sum\_\{i\\in c\}\\big\(p\_\{i\}^\{\(0\)\}\-\\bar\{p\}\_\{c\}\\big\)^\{2\},whereSc2S\_\{c\}^\{2\}is the observed dispersion of the raw forecasts within clusterccandσ^2¯c\\overline\{\\hat\{\\sigma\}^\{2\}\}\_\{c\}is the average intra\-question sampling noise in that cluster\.

Table 13:Heteroskedastic empirical Bayes shrinkage on synthetic\-question clusters\.
#### Log\-odds classical James\-Stein\.

The main shrinkage baseline in Table[12](https://arxiv.org/html/2611.17735#A10.T12)applies classical positive\-part James\-Stein shrinkage directly to the initial probabilitiespi\(0\)p\_\{i\}^\{\(0\)\}\. We also try applying the same classical shrinkage rule to the log\-odds,logit⁡\(pi\(0\)\)\\operatorname\{logit\}\(p\_\{i\}^\{\(0\)\}\), which are unbounded\.

Table 14:Log\-odds classical positive\-part James\-Stein shrinkage on synthetic\-question clusters\.The heteroskedastic empirical Bayes and log\-odds variants do not meaningfully improve performance, so the shrinkage baseline’s failure does not appear to be specific to the probability\-scale classical James\-Stein form\.

#### Shrinking towards zero\.

Finally, we try shrinking probabilities towards zero instead of the cluster mean\. LetKKbe the total number of questions in the test set, including synthetic questions\. We apply one global shrinkage coefficient,

p^i=\(1−λ\)​pi\(0\),λ=min⁡\{1,\(K−2\)​σ¯2∑i=1K\(pi\(0\)\)2\}\.\\hat\{p\}\_\{i\}=\(1\-\\lambda\)p\_\{i\}^\{\(0\)\},\\qquad\\lambda=\\min\\left\\\{1,\\frac\{\(K\-2\)\\bar\{\\sigma\}^\{2\}\}\{\\sum\_\{i=1\}^\{K\}\(p\_\{i\}^\{\(0\)\}\)^\{2\}\}\\right\\\}\.HereK=11,472K=11\{,\}472andσ¯2\\bar\{\\sigma\}^\{2\}is the average sample variance of forecasts across all questions\. Table[15](https://arxiv.org/html/2611.17735#A10.T15)shows the performance of shrinking towards zero, which yields similar performance to the other forms of shrinkage shown\.

Table 15:Global classical positive\-part James–Stein shrinkage toward the fixed probability target zero\.

### J\.3Platt scaling

In the opposite direction to shrinkage, there’s the question of whether we are simply improving the calibration of predictions by extremizing them, since LLMs tend to hedge towards 0\.5\[[3](https://arxiv.org/html/2611.17735#bib.bibx3)\]\. Following the fixed\-parameter convention used in\[[3](https://arxiv.org/html/2611.17735#bib.bibx3)\], we apply Platt scaling with coefficient3\\sqrt\{3\}directly to each initial ensemble mean forecast:

p^i=σ⁡\(3​logit⁡\(pi\(0\)\)\)\.\\hat\{p\}\_\{i\}=\\sigma\\\!\\left\(\\sqrt\{3\}\\operatorname\{logit\}\(p\_\{i\}^\{\(0\)\}\)\\right\)\.This is an independent per\-question statistical correction; it does not use question text, reasoning traces, clusters, or consistency constraints\. As Table[16](https://arxiv.org/html/2611.17735#A10.T16)illustrates, Platt scaling is ineffective at improving either Brier score or consistency, making most metrics worse than the raw forecasts\.

Table 16:Fixed\-parameter Platt scaling with coefficient3\\sqrt\{3\}\.
### J\.4Projection onto consistent solution set

We investigate whether our method is merely benefiting from the constraint of the probability axioms it is enforcing\. We can more simply apply these consistency constraints by minimizing the L2 deviation from the initial probabilities for each question while enforcing constraints\. While projection enforces perfect consistency, the Brier improvement is minimal\. Appendix[I](https://arxiv.org/html/2611.17735#A9)illustrates that this Brier improvement is statistically worse than that of our method\.

Table 17:Projection onto the consistent solution set with synthetic structural constraints enforced exactly\.
### J\.5Zero\-shot LLM

Finally, we consider the zero\-shot performance of GPT\-5\.1\. Instead of using strict projection, we ask the same model used for the upstream forecaster, GPT\-5\.1, to update the probabilities given the consistency constraints\. We use the following system prompt:

> You are given a cluster of forecasting questions and an initial P\(YES\) for each question\. Update all probabilities jointly using only the supplied question information and the explicit consistency constraints\.Reason about the questions as a coherent joint forecast\. Reassess each initial probability in light of the event details, the other events in the cluster, and their logical relationships\. Among probability assignments that satisfy every constraint, choose the values that best represent your substantive judgment of the events’ likelihoods\. The initial probabilities are estimates rather than fixed values; do not make arbitrary adjustments merely to force consistency\.The constraint block is authoritative:\- A complement constraint means the two listed probabilities must sum to 1\.\- A MECE constraint means the listed mutually exclusive and collectively exhaustive probabilities must sum to 1\.\- A monotonic chain P\(Event A\) <= P\(Event B\) means the probabilities must be non\-decreasing from left to right\.Return one updated probability for every event, in the exact numbered event order\. Satisfy all listed constraints simultaneously\. Return only the required JSON object and no explanation\.

The user prompt template is:

> \[Cluster of <N\> forecasting events\]\[Event 1\]Question: <question text\>Resolution date: <resolution date\>Background information: <background information\>Resolution criteria: <resolution criteria\>Initial predicted P\(YES\): <initial probability, rounded to four decimal places\>\.\.\.\[Event N\]Question: <question text\>Resolution date: <resolution date\>Background information: <background information\>Resolution criteria: <resolution criteria\>Initial predicted P\(YES\): <initial probability, rounded to four decimal places\>\[Consistency constraints\]\[Complement constraints: each pair must sum to 1\]\- P\(Event <i\>\) \+ P\(Event <j\>\) = 1\[MECE constraints: each group must sum to 1\]\- P\(Event <i\>\) \+ \.\.\. \+ P\(Event <j\>\) = 1\[Monotonic chains: probabilities must be non\-decreasing\]\- P\(Event <i\>\) <= \.\.\. <= P\(Event <j\>\)Return exactly <N\> probabilities in Event 1 through Event <N\> order\.

We enforce the output to be a JSON object of the form "probabilities": \[…\]\|\. We check that the array has lengthNNand had values in\[0,1\]\[0,1\]through downstream validation\.

Table 18:Zero\-shot GPT\-5\.1 joint\-reconciliation baseline on synthetic\-question clusters\.The zero\-shot LLM satisfies all three constraint families up to numerical precision, but worsens Brier score by3\.98%3\.98\\%relative to the raw forecasts\.

## Appendix KAdditional ablation details

In this section we provide detailed results for each ablation mentioned in Appendix[H\.2](https://arxiv.org/html/2611.17735#A8.SS2), including per\-seed metrics when applicable\. The methodology for each ablation is described in Appendix[H\.2](https://arxiv.org/html/2611.17735#A8.SS2), and significance testing is shown in Appendix[I](https://arxiv.org/html/2611.17735#A9)\.

#### Ablation: no consistency loss

Table 19:Ablation removing the consistency term from the loss on synthetic\-question clusters\.Table 20:Per\-seed breakdown of Table[19](https://arxiv.org/html/2611.17735#A11.T19)\. Each column reports the metric value with its relative change from Raw in parentheses \(negative is better\)\.
#### Ablation: no reasoning traces

Table 21:Ablation removing reasoning traces from prompts on synthetic\-question clusters\.Table 22:Per\-seed breakdown of Table[21](https://arxiv.org/html/2611.17735#A11.T21)\. Each column reports the metric value with its relative change from Raw in parentheses \(negative is better\)\.
#### Ablation: no cross\-question context

Table 23:No cross\-question context ablation that uses clusters containing ten independent runs of the same question, as opposed to clusters with related questions\.Table 24:Per\-seed breakdown of Table[23](https://arxiv.org/html/2611.17735#A11.T23)\. Each column reports the metric value with its relative change from Raw in parentheses \(negative is better\)\.
#### Ablation: frontier LLM

Table 25:Ablation replacing the finetuned model with GPT\-5\.1 off\-the\-shelf\.

相似文章

将未来行为预测作为学习任务

arXiv cs.AI

本文提出了 Behavior Forecasters,一种从推理轨迹中预测 LRM 未来行为(如答案一致性和输入敏感性)的学习方法,以更低的成本超越了 GPT-5.4 和 Claude Opus 4.6。

将预测未来行为作为一项学习任务

Hugging Face Daily Papers

本文提出训练行为预测器,从单条推理轨迹预测大型推理模型的输出,在计算成本更低的情况下优于GPT-5.4和Claude Opus-4.6等大型语言模型,绕过了传统的可解释性方法。