What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness
Summary
This paper probes internal representations of LLM forecasters to improve calibration and assess faithfulness of chain-of-thought reasoning, finding that probes achieve better calibration and act as lie detectors.
View Cached Full Text
Cached at: 07/10/26, 06:12 AM
# What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness Source: [https://arxiv.org/abs/2607.08046](https://arxiv.org/abs/2607.08046) [View PDF](https://arxiv.org/pdf/2607.08046) > Abstract:Large language models fine\-tuned for forecasting can be accurate yet poorly calibrated, and their chain\-of\-thought \(CoT\) reasoning may not faithfully reflect the evidence behind a forecast\. We ask whether internal representations offer a more direct window into both\. Working with Eternis\-Forecaster 8B on OpenForesight, we train representation\-pooling probes on intermediate activations and find they achieve substantially better calibration; a result that also holds for GLM\-4\.7\-Flash and GLM\-4\.5\-Air\. We then assess CoT faithfulness through evidence ablation and diversionary injection: removing an influential source in the prompt often changes the model's forecast while leaving the reasoning trace untouched\. The same probes function as lie detectors: their activations track behavioral shifts far better than the reasoning trace does, and they also predict the direction of change in 84% of cases, including when the CoT conceals the perturbation's influence\. Finally, forced answering reveals that forecasts are largely fixed before reasoning begins: a single pre\-reasoning pass recovers the committed answer and confidence, and routing questions by the spread of this pre\-set answer distribution saves 30\-47% of generated tokens, with no loss of accuracy\. Together, these results establish probing internal representations as a practical tool for calibrating, auditing, and triaging language model forecasters and reasoning models more broadly\. ## Submission history From: Raphael Sarfati \[[view email](https://arxiv.org/show-email/25252c81/2607.08046)\] **\[v1\]**Thu, 9 Jul 2026 01:52:09 UTC \(858 KB\)
Similar Articles
Making LLMs tell you how confident they really are through probe-targeted fine tuning.[R]
This research presents probe-targeted fine-tuning (LoRA) to make LLMs verbally express their internal confidence, achieving causal control over confidence outputs and demonstrating that models often know when they are right or wrong but fail to articulate it.
LLMs know when they are wrong. I made a fix relating to Anthropic's new "global workspace" paper [R]
The author presents a method to make LLMs verbalize calibrated confidence by using a linear probe on mid-layer states and a small trained bridge to confidence logits, requiring only 200 labeled examples and no weight modification. This is linked to Anthropic's global workspace paper explaining the know-say gap.
Don't Go Breaking My LLM: The Impact of Pruning Attention Layers on Explanation Faithfulness and Confidence Calibration
This paper studies how pruning attention layers in LLMs affects explanation faithfulness and confidence calibration, finding that accuracy often remains high but interpretability and reliability degrade, highlighting a misalignment between model confidence, interpretability, and accuracy.
Faithful uncertainty in LLM agents: calibration vs utility tradeoff in practice[D]
A practitioner discusses the calibration vs. utility tradeoff in LLM agents, sharing experience with a verifier-based pipeline that reduces hallucinated tool calls by ~60% but introduces latency costs and drops easy correct answers.
Do LLMs Really Know What They Don't Know? Internal States Mainly Reflect Knowledge Recall Rather Than Truthfulness
This paper challenges the assumption that LLMs can reliably distinguish between hallucinated and factual outputs through internal signals, arguing that internal states primarily reflect knowledge recall rather than truthfulness. The authors propose a taxonomy of hallucinations (associated vs. unassociated) and show that associated hallucinations exhibit hidden-state geometries overlapping with factual outputs, making standard detection methods ineffective.