Competence-Gated Pooling of Language Models and Priors for Event Forecasting

Hugging Face Daily Papers Papers

Summary

This paper introduces competence-gated pooling to determine when language models should influence forecasts that already have external predictions, improving accuracy by learning domain-specific weights and deferring when external sources are stronger.

In hybrid forecasting, a language model is often one of several available signals. A system may already have a market, crowd, or statistical forecast and must decide whether the model adds useful information or should be ignored. The relevant target is therefore not standalone model accuracy, but relative competence, defined as the model's marginal value beyond the available external forecast. Under Brier loss, we characterize when model disagreement can improve an external forecast and derive the gain from using domain-specific rather than global pooling weights. We then introduce a competence gate that estimates domain-level source weights from resolved outcomes, shrinks uncertain estimates toward a global weight, and recalibrates the pooled forecast. Across 2,357 resolved binary questions and five language models, the gate improves the main external baseline from 0.0771 to 0.0732 Brier and significantly outperforms global forecast combinations. The gain remains significant under leakage controls and against a leakage-safe time-series prior on the pooled structured set, with separate evidence on FRED. In contrast, the gate gives no significant improvement on the official ForecastBench market subset, where it largely defers to the market. Across four Qwen models, verbal confidence does not reliably identify when the model outperforms the external forecast, while outcome-estimated competence supports better abstention decisions. These results provide a practical approach for selective model use based on measured marginal value.
Original Article
View Cached Full Text

Cached at: 09/14/26, 10:37 PM

Paper page - Competence-Gated Pooling of Language Models and Priors for Event Forecasting

Source: https://huggingface.co/papers/2609.12101 When should an LLM actually influence a forecast that already has a market, crowd, or statistical prediction?

Most forecasting work asks how accurate an LLM is in isolation. We instead study relative competence: the model’s marginal value beyond an already-available forecast.

We introduce competence-gated pooling, which learns domain-specific source weights from resolved outcomes and defers when the external source is stronger. Across 2,357 resolved forecasting questions, the gate improves Brier score from 0.0771 → 0.0732 and significantly outperforms global forecast combinations. On a strong live-market subset, however, it correctly finds no significant gain from adding the LLM.

We also find that self-reported LLM confidence does not reliably predict when the model beats the external forecast—while outcome-estimated competence enables substantially better selective prediction.

Takeaway: don’t ask only “Is the model good?” Ask “Does it add information beyond what we already know?”

Similar Articles

Forecasting Future Behavior as a Learning Task

Hugging Face Daily Papers

This paper proposes training Behavior Forecasters to predict large reasoning model outputs from single trajectories, outperforming large language models like GPT-5.4 and Claude Opus-4.6 at lower computational cost, bypassing traditional explainability methods.

Competing Learning in Hybrid Architectures

Hugging Face Daily Papers

The paper identifies a Competing Learning phenomenon in hybrid language models where recurrence and attention layers compete to learn the same functions, and proposes a staged training procedure that improves language modeling loss, common-sense reasoning, and retrieval tasks on models up to 1B parameters.

Co-Evolving Acting and Forecasting for Language Agents

Hugging Face Daily Papers

CoPE 论文提出通过在成功轨迹上训练多步动作目标,使语言智能体的行动与预测能力协同演化,从而在不增加推理开销的情况下提升样本效率,相比 GRPO 与 ECHO 在多项基准上取得更强表现。