Forecasting Future Behavior as a Learning Task

# Forecasting Future Behavior as a Learning Task
Source: [https://arxiv.org/html/2606.11445](https://arxiv.org/html/2606.11445)
Mosh Levy Bar\-Ilan University Constellation moshe0110@gmail\.com&Yoav Goldberg Bar\-Ilan University Allen Institute for AI &Asa Cooper Stickland UK AI Security Institute

###### Abstract

Trust in an AI system is often anchored by explanations of how it works, which one then uses to forecast its behavior on new inputs\. For large reasoning models \(LRMs\), this conventional route is particularly difficult to follow: explanation methods for single token generations do not naturally generalize to long trajectories, and the trajectories themselves are often not faithful when read as natural language\. We propose an alternative that bypasses the explanation step: treat behavior forecasting as a learnable task and train*Behavior Forecasters*that operates on a single reasoning trajectory to make the same forecasts one would typically seek from an explanation\. The forecaster’s training data is obtained by querying the LRM with no human annotation, and its inference is done in a single forward pass\. We instantiate this approach on two tasks: how likely the LRM is to repeat its answer on re\-runs, and how removing parts of the input changes its answer\. We evaluate this approach on both tasks across three diverse reasoning datasets and find that trained Behavior Forecasters are more accurate than GPT\-5\.4 and Claude Opus 4\.6 reading the same trajectories as naive readers, at a small fraction of their inference cost\. We find that fine\-tuning the backbone end\-to\-end and initializing it from the target LRM are each necessary for strong performance\. These results show that the reasoning trajectory carries information about the LRM’s future behavior that goes beyond what naive reading conveys\.

††Code and data:[https://github\.com/Mosh0110/behavior\-forecasters](https://github.com/Mosh0110/behavior-forecasters)## 1Introduction

Relying on an AI system requires anticipating how it will behave on new inputs\. Conventionally, this anticipation is anchored by explanations of the model’s computation, widely treated as a device for forecasting111We use*forecasting*rather than*predicting*to distinguish between the action of stating something about a model’s future behavior and the operation of a neural network producing its output\.how it will behave on new inputs or in new scenarios\(Hempel and Oppenheim,[1948](https://arxiv.org/html/2606.11445#bib.bib46); Douglas,[2009](https://arxiv.org/html/2606.11445#bib.bib47); Jacoviet al\.,[2021](https://arxiv.org/html/2606.11445#bib.bib48); Doshi\-Velez and Kim,[2017](https://arxiv.org/html/2606.11445#bib.bib60); Lipton,[2018](https://arxiv.org/html/2606.11445#bib.bib136)\)\. This view is reflected in the way researchers in machine learning operationalize and evaluate explanations by the accuracy of the forecasts they enable\(Miller,[2019](https://arxiv.org/html/2606.11445#bib.bib10)\)\. The most basic granularity at which such forecasts can drive deployment decisions is the individual input, for example how consistent the model’s answer is across reruns or how it would change under counterfactual variants of the input\(Ribeiroet al\.,[2016](https://arxiv.org/html/2606.11445#bib.bib132); Lundberg and Lee,[2017](https://arxiv.org/html/2606.11445#bib.bib133); Sundararajanet al\.,[2017](https://arxiv.org/html/2606.11445#bib.bib134); Alvarez\-Melis and Jaakkola,[2018](https://arxiv.org/html/2606.11445#bib.bib5); Barredo Arrietaet al\.,[2020](https://arxiv.org/html/2606.11445#bib.bib135)\)\. These per\-input behavior forecasts are the focus of this paper \(Section[2\.2](https://arxiv.org/html/2606.11445#S2.SS2)\)\.

Explanations that enable such forecasts are particularly difficult to obtain for large reasoning models \(LRMs\) that emit a long sequence of reasoning tokens before producing a final answer \(e\.g\., O1\(OpenAI,[2024](https://arxiv.org/html/2606.11445#bib.bib32)\), R1\(DeepSeek\-AIet al\.,[2025](https://arxiv.org/html/2606.11445#bib.bib34)\)\) \(Section[2\.3](https://arxiv.org/html/2606.11445#S2.SS3)\)\. Methods developed for a single forward pass tell us about the generation of one token, and these methods do not naturally extend to the long reasoning trajectories these models emit\. The reasoning tokens themselves, despite their natural\-language appearance, are often not a faithful account of the computation that produced the answer \(see Section[2\.3\.1](https://arxiv.org/html/2606.11445#S2.SS3.SSS1)for supporting literature\), so*naive reading*of the trajectory may lead to wrong forecasts\. This leaves two unsatisfying options for behavior forecasting on LRMs: read the tokens naively and accept an unreliable picture, or disregard them\.

In this paper, we propose an alternative to making such forecasts via explanations: train an external model to forecast future behavior directly from a single reasoning trajectory \(Section[3](https://arxiv.org/html/2606.11445#S3)\)\. We call this trained model a*Behavior Forecaster*: it relies on the tokens in the reasoning trajectory to carry information about the underlying computation, but does not require this information to be recoverable by reading the tokens as text\(Levyet al\.,[2025b](https://arxiv.org/html/2606.11445#bib.bib1)\)\. We treat behavior forecasting as a supervised learning problem in which the training data is generated at scale without human annotation by querying the LRM itself, producing ground\-truth behavioral labels\.

![Refer to caption](https://arxiv.org/html/2606.11445v1/x1.png)Figure 1:Behavior forecasting from a single reasoning trajectory\. Given one observed trajectory of the target LRM \(prompt, reasoning, answer\), a trained Behavior Forecaster predicts a property of the LRM’s future behavior in one forward pass: how likely the LRM is to repeat the answer \(rerun consistency\), or how likely each prompt part is to have caused the answer \(counterfactual sensitivity\)\. The Behavior Forecaster is trained on many such trajectories with behavioral labels obtained by querying the LRM\. A frontier LLM reading the same trajectory naively relies on the text being faithful, does not scale as it requires a lot more compute, and yields less accurate predictions\.Given one reasoning trajectory of the target LRM on the input of interest, a Behavior Forecaster predicts in one forward pass a statistic of the LRM’s future behavior; we instantiate it on two such statistics:*rerun consistency*, computed over reruns on the same input, and*counterfactual sensitivity*, computed over perturbations of the input\. We initialize the forecaster from the target LRM, attach a task\-specific prediction head, and train them jointly\.

We show that the Behavior Forecaster successfully learns both tasks \(Section[4](https://arxiv.org/html/2606.11445#S4)\)\. We evaluate the approach on both tasks across three reasoning datasets\. The trained Behavior Forecaster is more accurate than GPT\-5\.4 and Claude Opus 4\.6 reading the same trajectory naively, while consuming less than1/10,0001/10\{,\}000their estimated compute\. We also show that forecasters trained on two datasets transfer to a held\-out third dataset with at most600600fine\-tuning steps, on both tasks, and that within a single dataset family they generalize to held\-out variants without any fine\-tuning\. The counterfactual\-sensitivity forecaster also transfers to predicting whether the LRM relied on a user\-provided hint, which is a common way to evaluate the faithfulness of the reasoning text\(Turpinet al\.,[2023](https://arxiv.org/html/2606.11445#bib.bib51); Chenet al\.,[2025](https://arxiv.org/html/2606.11445#bib.bib120); Chua and Evans,[2025](https://arxiv.org/html/2606.11445#bib.bib67); Marioriyadet al\.,[2025](https://arxiv.org/html/2606.11445#bib.bib52)\)\.

We then ablate the forecaster to identify which architectural and training choices matter \(Section[5](https://arxiv.org/html/2606.11445#S5)\)\. Training only the prediction head, initializing from a randomized backbone, or removing the reasoning tokens from the input all reduce performance, and the best arrangement of prompt, reasoning, and answer in the input depends on the task\.

Our findings suggest that core local properties of LRM behavior, normally probed by expensive resampling or by unreliable reading of the reasoning tokens, can be forecast from a single observed trajectory\. We see this as motivation to study behavior forecasting as a learnable task in its own right\. More broadly, these findings suggest that the reasoning trajectory carries information about the LRM’s future behavior beyond what naive reading conveys, inviting us to treat reasoning trajectories as data with learnable patterns rather than as natural language to be read\.

## 2Behavior forecasting for LRMs

In this section we motivate behavior forecasting as a route to trust in AI models, formalize the task this paper addresses, and discuss why current approaches motivate a new method for LRMs\.

### 2\.1Trust through behavior forecasting

Trust in AI models rests on the user’s ability to forecast how they will behave on future inputs\(Lee and See,[2004](https://arxiv.org/html/2606.11445#bib.bib150); Hoff and Bashir,[2015](https://arxiv.org/html/2606.11445#bib.bib151); Jacoviet al\.,[2021](https://arxiv.org/html/2606.11445#bib.bib48); Zhouet al\.,[2026](https://arxiv.org/html/2606.11445#bib.bib144)\)\. The concept considered as “explanation” is widely seen as a device for making such forecasts about a system\(Hempel and Oppenheim,[1948](https://arxiv.org/html/2606.11445#bib.bib46); Douglas,[2009](https://arxiv.org/html/2606.11445#bib.bib47)\): knowing how a system operates enables forecasting how it will behave on new inputs or in new scenarios\. Researchers in machine learning adopt the same view, operationalizing and evaluating explanations by the accuracy of the forecasts they enable\(Doshi\-Velez and Kim,[2017](https://arxiv.org/html/2606.11445#bib.bib60); Miller,[2019](https://arxiv.org/html/2606.11445#bib.bib10)\)\.

### 2\.2Forecasting behavior from a single trajectory

We focus on forecasting properties of a model’s behavior on a given input, such as how consistent the model’s answer is across reruns or how sensitive its answer is to counterfactual perturbations of the input, from a single observed run\(Ribeiroet al\.,[2016](https://arxiv.org/html/2606.11445#bib.bib132); Lundberg and Lee,[2017](https://arxiv.org/html/2606.11445#bib.bib133); Sundararajanet al\.,[2017](https://arxiv.org/html/2606.11445#bib.bib134)\)\. Working at this granularity is useful in practice for two reasons\. First, per\-input signals can directly drive per\-query decisions such as abstention, flagging, or routing to human review, which are sometimes required from deployed AI systems\(European Parliament and Council of the European Union,[2024](https://arxiv.org/html/2606.11445#bib.bib169)\)\. These decisions must be made at scale, which rules out collecting multiple resamples online\. And second, larger behavioral testing, debugging, and auditing pipelines can aggregate these local measurements across inputs to characterize the system’s overall behavior\(Ribeiroet al\.,[2020](https://arxiv.org/html/2606.11445#bib.bib4); Bhattet al\.,[2020](https://arxiv.org/html/2606.11445#bib.bib154)\)\. These pipelines run at scale too, which rules out expensive per\-input measurements: the per\-input cost compounds across every input the pipeline covers\.

##### Formal setup\.

LetMMbe a stochastic LRM\. On a promptPP, one execution ofMMproduces an observed trajectoryτ=\(P,R,A\)\\tau=\(P,R,A\), whereRRis the LRM’s reasoning tokens andAAis the final answer\. A behavior forecasting problem specifies \(i\) a set of related prompts\{P′\}\\\{P^\{\\prime\}\\\}aroundPP\(which may bePPitself for reruns\), and \(ii\) a targetb​\(M,P,A\)b\(M,P,A\)defined as a statistic ofMM’s answer distribution under these related prompts, conditioned on the observed answerAA\. The task is: given one observed trajectoryτ∼M​\(P\)\\tau\\sim M\(P\), forecastb​\(M,P,A\)b\(M,P,A\)without runningMMon any related promptP′P^\{\\prime\}\.

We instantiate this definition on two tasks\.

- •Rerun consistency:the related prompt isPPitself \(P′=PP^\{\\prime\}=P\);b​\(M,P,A\)b\(M,P,A\)is the probability that another run produces the same answerAA\.
- •Counterfactual sensitivity:for each segmentssin the setS​\(P\)S\(P\)of removable segments ofPP, the related prompt is the perturbationP′=P−sP^\{\\prime\}=P\_\{\-s\}that removesssfromPP;b​\(M,P,A\)b\(M,P,A\)has one component perss, equal to how much removingssreduces the probability thatMMproducesAA\.

In both, target values lie in\[0,1\]\[0,1\]and are derived fromMM’s induced answer distribution\. The task is therefore regression against probability\-derived targets\.

### 2\.3Why we need a new approach for LRMs

There are two natural approaches to behavior forecasting on an LRM, and both run into limitations that motivate a different approach: reading the reasoning trajectory as an explanation of the LRM’s computation \(Section[2\.3\.1](https://arxiv.org/html/2606.11445#S2.SS3.SSS1)\), and applying standard behavior forecasting tools \(Section[2\.3\.2](https://arxiv.org/html/2606.11445#S2.SS3.SSS2)\)\.

#### 2\.3\.1Naive reading of the reasoning trajectory is often unreliable

A behavior forecast asks, for a given input, how confidently the LRM arrived at its answer \(rerun consistency\) and which input clauses it actually used \(counterfactual sensitivity\)\.

The reasoning trajectory is the natural candidate source for these outcomes in an LRM\(Korbaket al\.,[2025](https://arxiv.org/html/2606.11445#bib.bib118); Guanet al\.,[2025](https://arxiv.org/html/2606.11445#bib.bib12)\)\. If the trajectory’s natural\-language surface faithfully tracked the LRM’s computation, recovering each outcome would reduce to a classical natural language processing task on that surface text: confidence detection on the trajectory for rerun consistency, and tracing how the trajectory’s reasoning steps connect input segments to the final answer for counterfactual sensitivity\. We refer to this approach as*naive reading*: interpreting the trajectory through generic English semantics, without learned familiarity with the patterns the target LRM uses in its reasoning tokens\. Both human readers and other LRMs read naively in this sense\. But the natural\-language surface of the trajectory is often not a faithful account of the computation behind the answer\.

##### Omission\.

LRMs omit factors from the natural\-language reasoning that shape the final answer\. Planted cues such as biased answer positions, user hints, and stereotype signals shift predictions without appearing in the natural\-language reasoning\(Turpinet al\.,[2023](https://arxiv.org/html/2606.11445#bib.bib51); Yeeet al\.,[2024](https://arxiv.org/html/2606.11445#bib.bib75); Chua and Evans,[2025](https://arxiv.org/html/2606.11445#bib.bib67); Arcuschinet al\.,[2025](https://arxiv.org/html/2606.11445#bib.bib53); Marioriyadet al\.,[2025](https://arxiv.org/html/2606.11445#bib.bib52); Lindseyet al\.,[2025](https://arxiv.org/html/2606.11445#bib.bib38); Mirtaheri and Belkin,[2026](https://arxiv.org/html/2606.11445#bib.bib143); Young,[2026](https://arxiv.org/html/2606.11445#bib.bib147); Bachmannet al\.,[2026](https://arxiv.org/html/2606.11445#bib.bib117)\)\.

##### Semantic mismatch\.

Even the steps that do appear in the natural\-language reasoning can diverge from the computation the LRM does to produce the answer\. The surface natural language can be illegible to humans and AI naive readers while the LRM still arrives at correct answers\(Jose,[2025](https://arxiv.org/html/2606.11445#bib.bib116)\), injecting changes into the natural\-language reasoning often leaves the final answer unchanged\(Lanhamet al\.,[2023](https://arxiv.org/html/2606.11445#bib.bib56); Paulet al\.,[2024](https://arxiv.org/html/2606.11445#bib.bib42)\), and causal mediation analysis shows that LRMs do not use their intermediate steps in the way a human reader would predict\(Levyet al\.,[2025a](https://arxiv.org/html/2606.11445#bib.bib127)\)\.

Chenet al\.\([2025](https://arxiv.org/html/2606.11445#bib.bib120)\)andHanet al\.\([2026](https://arxiv.org/html/2606.11445#bib.bib149)\)also show that faithfulness is not guaranteed to improve with model scale or more performant models\. Naive reading is therefore unlikely to reliably deliver the outcomes we are after, motivating a learned approach to behavior forecasting\.

#### 2\.3\.2Standard behavior forecasting methods are not applicable

Standard behavior forecasting methods are unlikely to be useful for forecasting LRMs behavior either\. The long, stochastic trajectories LRMs produce make these methods either prohibitively expensive or blind to the trajectory itself\.

##### Resampling methods are prohibitively expensive\.

Resampling underlies existing methods for forecasting future model behavior\(Joneset al\.,[2025](https://arxiv.org/html/2606.11445#bib.bib145); Serranoet al\.,[2026](https://arxiv.org/html/2606.11445#bib.bib146)\)\. It is straightforward in principle but impractical in deployment\. Estimating either rerun consistency fromNNreruns or counterfactual sensitivity fromNNperturbations multiplies the already high cost of one long reasoning run byNN\.

##### Single\-location probes miss the trajectory\.

Standard attribution and uncertainty signals all explain the generation of a single fixed position in the model’s output\. Gradient\-based input attributions like integrated gradients\(Sundararajanet al\.,[2017](https://arxiv.org/html/2606.11445#bib.bib134)\)require a differentiable path from input to output, but the LRM computational graph involves discrete sampling steps that break the gradient flow; in LLMs the common substitute is the attention from the final\-answer tokens back to the prompt\(Chuanget al\.,[2024](https://arxiv.org/html/2606.11445#bib.bib162); Jain and Wallace,[2019](https://arxiv.org/html/2606.11445#bib.bib163)\)\. Token\-probability uncertainty signals\(Manakulet al\.,[2023](https://arxiv.org/html/2606.11445#bib.bib160); Farquharet al\.,[2024](https://arxiv.org/html/2606.11445#bib.bib161)\)score the final\-answer logits in the same way\. None of these registers what happens inside the reasoning trajectory, where much of the LRM’s computation actually unfolds\.

## 3Method

Section[2](https://arxiv.org/html/2606.11445#S2)showed that standard tools for behavior forecasting either break down or become prohibitively expensive on LRMs, and that naive reading of the reasoning trajectory is unlikely to faithfully answer behavior forecasting questions\. We propose an alternative: we train a model to forecast behavioral properties of the LRM from a single observed reasoning trajectory\. We call this trained model a*Behavior Forecaster*\(or simply*forecaster*\)\. We first describe the learning setup \(Section[3\.1](https://arxiv.org/html/2606.11445#S3.SS1)\), then the per\-task supervision \(Section[3\.2](https://arxiv.org/html/2606.11445#S3.SS2)\), and finally the Behavior Forecaster architecture \(Section[3\.3](https://arxiv.org/html/2606.11445#S3.SS3)\)\.

### 3\.1Behavior forecasting as a learning task

We treat behavior forecasting as a supervised learning task: from one observed reasoning trajectory, predict a target behavioral statistic of the LRM\. This relies on the trajectory carrying information about the underlying computation, but does not require this information to be readable as natural\-language text\(Levyet al\.,[2025b](https://arxiv.org/html/2606.11445#bib.bib1)\)\. Framing it this way lets behavior forecasting benefit from data and compute\(Sutton,[2019](https://arxiv.org/html/2606.11445#bib.bib8)\)\.

To generate training data, for each promptPPwe first obtain one observed trajectoryτ∼M​\(P\)\\tau\\sim M\(P\)with answerAA\. We then run the LRM on the related executions defined by the forecasting problem and use those executions to estimate the target statisticb​\(M,P,A\)b\(M,P,A\)\.

We train a Behavior Forecaster to map the observed trajectory to the target statistic in inference\. The expensive executions are needed only for data generation\. The cost gap relative to resampling at inference is large: estimating the rerun\-consistency label for one input from a fresh resample requires additional trajectories from the LRM, each up to thousands of tokens generated autoregressively, while the Behavior Forecaster needs one forward pass over the already\-observed trajectory, orders of magnitude less compute and far lower latency at deployment\.

### 3\.2Task\-specific supervision

We estimate the answer probabilities enteringb​\(M,P,A\)b\(M,P,A\)empirically: for each promptPPwe run the LRM1010times onPP, and for counterfactual sensitivity we additionally run the LRM1010times on each perturbed version ofPP, all with the LRM’s recommended sampling settings \(temperature0\.60\.6, top\-pp0\.950\.95\)\. We extract a final answer from each trajectory and treat two runs as producing the same answer when their extracted answers match, rather than requiring identical output text\. A sample is labeled only if at least55of its runs yields an extractable answer; full details are in Appendix[A](https://arxiv.org/html/2606.11445#A1)\.

##### Rerun consistency\.

For each run on a prompt, the run\-level target is the fraction of the prompt’s other valid runs that produce the same extracted answer; this estimates the chance that a fresh run on the same prompt would reproduce the answer\.

##### Counterfactual sensitivity\.

For each promptPP, we generate one perturbed promptP−sP\_\{\-s\}per segments∈S​\(P\)s\\in S\(P\); the segmentation scheme is dataset\-specific\. We retain only prompts whose answerAAappears in at least70%70\\%of the1010original runs, so labels reflect omission effects rather than base sampling noise\. The segment\-level label measures how much removingssreduces the rate at which the LRM producesAA, normalized to\[0,1\]\[0,1\]so a value of11means removingssalways changes the answer\. Every token in segmentssshares this score; the per\-segment prediction aggregates the forecaster’s per\-token outputs within the segment\.

### 3\.3Behavior Forecaster architecture

The Behavior Forecaster’s backbone shares its architecture with the target LRM and initializes from the LRM’s weights, so it starts with representations already attuned to the reasoning tokens\. We use the same kind of backbone for both tasks, training a separate copy per task and pairing each with a task\-specific input arrangement and head\.

##### Counterfactual sensitivity: prompt\-echo arrangement\.

For counterfactual sensitivity, the task is one score per segment of the prompt; we aggregate per\-token outputs into a per\-segment score \(Appendix[B\.1](https://arxiv.org/html/2606.11445#A2.SS1)\)\. The natural placement for these per\-token predictions is at the prompt positions\. But the backbone is a standard autoregressive decoder: a prediction head at a prompt\-position token attends only to earlier tokens, which excludes the reasoningRRand the answerAA\. Without those, the head has none of the trajectory’s evidence about how the LRM used each prompt segment\. We therefore append a copy of the prompt after the observed trajectory and produce per\-token predictions at the echoed\-prompt positions\. A per\-token MLP head produces one prediction at each echoed\-prompt position, and the per\-segment score in\[0,1\]\[0,1\]is obtained by aggregating these within each segment\.

##### Rerun consistency: cross\-attention pooling\.

For rerun consistency, the target is one score per trajectory, with no prompt\-side structure to attribute to\. Trajectories vary in length, so we pool the backbone’s hidden states into a fixed\-size representation that a simple MLP head can map to one scalar in\[0,1\]\[0,1\]\. We pool with cross\-attention over a fixed set of learned query vectors; the queries attend over all trajectory positions, so the pooled representation can draw on evidence from anywhere in the trajectory\. Architectural details \(number of query vectors, head sizes, etc\.\) are in Appendix[B\.2](https://arxiv.org/html/2606.11445#A2.SS2)\.

## 4Behavior Forecasting is learnable

![Refer to caption](https://arxiv.org/html/2606.11445v1/figures/sec4_learnability_olmo_qwen35_2b_original_style_grid.png)Figure 2:A trained Behavior Forecaster is more accurate than two frontier naive readers and the standard single\-location probes, at a small fraction of the readers’ inference cost\. Rows show target LRM: OLMo\-3\-7B\-Think \(top; FEVEROUS, RuleTaker, TreeCut\) and Qwen3\.5\-2B \(bottom; FEVEROUS, RuleTaker\)\. For Qwen3\.5\-2B, TreeCut is omitted because its trajectories were too long for our available generation resources\. Each panel plots the forecaster’s test Spearman correlation over training steps; horizontal lines mark the two frontier naive readers \(GPT\-5\.4, Claude Opus 4\.6\), the task\-specific single\-location probe on the target LRM \(attention attribution for counterfactual sensitivity, answer prediction likelihood for rerun consistency\), and a random\-prediction baseline\. Final Spearman and Pearson values for the OLMo Behavior Forecaster and both naive readers are reported in Table[9](https://arxiv.org/html/2606.11445#A3.T9); per\-dataset OLMo values, with Behavior Forecaster entries taken from the last saved checkpoint, are reported in Table[10](https://arxiv.org/html/2606.11445#A3.T10); cluster\-bootstrap confidence intervals and paired permutation tests for the OLMo comparison are reported in Appendix[D](https://arxiv.org/html/2606.11445#A4); single\-location probe details are in Appendix[C\.2](https://arxiv.org/html/2606.11445#A3.SS2)\.We evaluate Behavior Forecasters on both tasks across three reasoning datasets\. A Behavior Forecaster trained on the pooled three datasets is more accurate than both strong naive readers and the standard single\-location probes, at a small fraction of the readers’ inference cost \(Section[4\.2](https://arxiv.org/html/2606.11445#S4.SS2)\); the same comparison holds on a second target LRM, Qwen3\.5\-2B\. It also generalizes beyond its training data \(Section[4\.3](https://arxiv.org/html/2606.11445#S4.SS3)\): within a single dataset family the forecaster generalizes zero\-shot to a held\-out variant, and across datasets a forecaster trained on two of the three adapts to the held\-out third with target\-side fine\-tuning\. Finally, we test whether a forecaster trained for counterfactual sensitivity can initialize a new Behavior Forecaster for a related hint\-sensitivity target \(Section[4\.4](https://arxiv.org/html/2606.11445#S4.SS4)\)\.

### 4\.1Experimental setup

##### Target LRMs and datasets\.

Our main target LRM is OLMo\-3\-7B\-Think\(Team Olmoet al\.,[2025](https://arxiv.org/html/2606.11445#bib.bib168)\), whose training data and training pipeline are fully released; this lets us build evaluation sets from data we can verify the LRM was not trained on, and lets follow\-up work that depends on knowing the LRM’s training data or pipeline build directly on the same target\. For the main learnability comparison \(Section[4\.2](https://arxiv.org/html/2606.11445#S4.SS2)\) we additionally evaluate on Qwen3\.5\-2B\(Team,[2026](https://arxiv.org/html/2606.11445#bib.bib35)\)as a second target LRM, to check that the gap against naive readers is not specific to OLMo; all other experiments use OLMo only\. We use three reasoning datasets that cover substantively different tasks: mathematical word problems \(TreeCut\(Ouyang,[2025](https://arxiv.org/html/2606.11445#bib.bib167)\)\), factual claims grounded in Wikipedia \(FEVEROUS\(Alyet al\.,[2021](https://arxiv.org/html/2606.11445#bib.bib141)\)\), and synthetic logical reasoning \(RuleTaker\(Clarket al\.,[2020](https://arxiv.org/html/2606.11445#bib.bib166)\)\)\. Full data construction and splits details, and per\-dataset sample counts are in Appendix[A\.1](https://arxiv.org/html/2606.11445#A1.SS1)\.

##### Training\.

We initialize the Behavior Forecaster from the target LRM’s weights and train it end\-to\-end with AdamW, weight decay0\.010\.01, and a linear\-decay schedule with100100warmup steps and a minimum learning rate of1×10−61\{\\times\}10^\{\-6\}\. We use an effective batch size of3232and train for22epochs\. Counterfactual sensitivity uses learning rate1×10−51\{\\times\}10^\{\-5\}and a token\-level MLP head; rerun consistency uses learning rate5×10−55\{\\times\}10^\{\-5\}and a sequence\-level cross\-attention MLP head\. All the Behavior Forecasters in the paper were trained on44NVIDIA H200 GPUs in under2424hours\. Architectural hyperparameters and the transfer fine\-tuning budget for Table[2](https://arxiv.org/html/2606.11445#S4.T2)are in Appendix[B\.2](https://arxiv.org/html/2606.11445#A2.SS2)\. All results in the paper, except for the main learnability figure \(Figure[2](https://arxiv.org/html/2606.11445#S4.F2)\), are evaluated on a checkpoint selected by validation loss; the main figure plots the test correlation across training steps directly\.

##### Naive reader baselines\.

We compare against frontier LRMs \(Claude Opus 4\.6 and GPT\-5\.4\) used as strong naive readers that receive the reasoning trajectory as input and predict the target property from its surface meaning\. These readers come well\-equipped for what the reading task reduces to, but they have not learned how the target LRM uses its reasoning tokens\.Maslejet al\.\([2025](https://arxiv.org/html/2606.11445#bib.bib88)\)report that LRMs perform at or above human level on the kind of basic text\-understanding tasks naive reading reduces to\. In practice, researchers already use frontier LLMs as readers of reasoning trajectories, deploying them to judge whether a target LRM’s trajectory tracks its underlying computation\(Chenet al\.,[2025](https://arxiv.org/html/2606.11445#bib.bib120); Young,[2026](https://arxiv.org/html/2606.11445#bib.bib147)\)\. The Behavior Forecaster and naive readers differ in their exposure to the target LRM’s reasoning trajectories: the Behavior Forecaster is trained on those trajectories and so can learn the patterns the LRM uses to encode information in its reasoning tokens\. These readers are larger than the Behavior Forecaster and use their own extended reasoning, making them substantially more expensive to run; any margin a Behavior Forecaster shows against them is therefore a conservative measure of the trained\-versus\-naive gap\. For each task we tested three prompt variants on a small pilot slice and selected, per reader, the variant with the strongest pilot correlation\. Full prompts and selected variants are in Appendix[C\.1](https://arxiv.org/html/2606.11445#A3.SS1)\.

##### Metric\.

We report Spearman correlation in the main text, which depends only on the rank order of predictions and is insensitive to compression, shift, or any monotone rescaling of the predicted scale\. Additional results using Pearson correlation are in the appendix\.

### 4\.2Trained Behavior Forecasters are more accurate than untrained baselines

##### Naive readers\.

The trained Behavior Forecaster is more accurate than strong naive readers \(GPT\-5\.4 and Claude Opus 4\.6\) reading the same trajectories, at a small fraction of their inference cost, on both target LRMs \(Figure[2](https://arxiv.org/html/2606.11445#S4.F2)\)\. On Qwen3\.5\-2B the Behavior Forecaster reaches Spearman0\.6530\.653on counterfactual sensitivity \(vs0\.4170\.417for GPT\-5\.4 and0\.5220\.522for Claude Opus 4\.6\) and0\.7400\.740on rerun consistency \(vs0\.2240\.224and0\.2670\.267respectively\)\. We also find that a same\-LRM reader control, in which OLMo\-3\-7B\-Think reads its own observed trajectory, is substantially weaker than both the trained Behavior Forecaster and the frontier naive readers \(Appendix[C\.3](https://arxiv.org/html/2606.11445#A3.SS3)\)\.

##### Single\-location probes\.

The Behavior Forecaster also outperforms the single\-location probes from Section[2\.3\.2](https://arxiv.org/html/2606.11445#S2.SS3.SSS2)applied to the target LRM: final\-answer attention to candidate input segments for counterfactual sensitivity\(Chuanget al\.,[2024](https://arxiv.org/html/2606.11445#bib.bib162); Jain and Wallace,[2019](https://arxiv.org/html/2606.11445#bib.bib163)\)and final\-answer log\-likelihood for rerun consistency\(Manakulet al\.,[2023](https://arxiv.org/html/2606.11445#bib.bib160); Farquharet al\.,[2024](https://arxiv.org/html/2606.11445#bib.bib161)\)\. Both clear the random\-prediction baseline but remain too weak to be useful in practice \(on the OLMo target,0\.1600\.160and0\.1290\.129respectively; on Qwen3\.5\-2B,0\.0160\.016and0\.2060\.206\), well below the Behavior Forecaster on both targets and below the naive readers \(Appendix[C\.2](https://arxiv.org/html/2606.11445#A3.SS2)\)\.

### 4\.3Generalization to new data

We test how a trained Behavior Forecaster generalizes to data it was not trained on, in two settings\.

##### Same\-family check\.

We train a Behavior Forecaster on three subsets of the FEVEROUS dataset, holding out the numerical subset entirely, and then evaluate it directly on FEVEROUS numerical examples with no fine\-tuning\. The Behavior Forecaster remains ahead of both frontier naive readers on this held\-out subset for both tasks \(Table[1](https://arxiv.org/html/2606.11445#S4.T1)\)\.

##### Transfer across datasets\.

We hold out one of our three datasets \(FEVEROUS, RuleTaker, or TreeCut\), train a source Behavior Forecaster on the remaining two, and fine\-tune it on the held\-out target for at most∼600\{\\sim\}600steps\. Transfer improves performance on every held\-out target for both tasks, though rerun consistency remains weaker on TreeCut than on FEVEROUS or RuleTaker after adaptation \(Table[2](https://arxiv.org/html/2606.11445#S4.T2)\)\.

Table 1:Same\-family transfer within FEVEROUS, for the OLMo\-3\-7B\-Think Behavior Forecaster\. The Behavior Forecaster is trained on the non\-numerical FEVEROUS subsets and evaluated on FEVEROUS numerical without target\-side fine\-tuning\.Table 2:Transfer to held\-out datasets, for the OLMo\-3\-7B\-Think Behavior Forecaster\. The*source*Behavior Forecaster is trained on the two other datasets and evaluated on the held\-out target without target\-side training; the*adapted*Behavior Forecaster is the fine\-tuned checkpoint\. Src and Adp denote source and adapted\.

### 4\.4Transfer across behavioral targets

We also test transfer to a new behavioral target\. Starting from the counterfactual\-sensitivity Behavior Forecaster, we adapt to hint\-sensitivity forecasting: given a single trajectory produced with an explicit answer hint, predict the probability that the target LRM’s no\-hint answer would differ from the observed hinted answer\. The test split holds out two strong hint templates and contains610610trajectories from122122FEVEROUS source examples\. As shown in Table[3](https://arxiv.org/html/2606.11445#S4.T3), the transferred forecaster obtains the best rank correlation on this surface, outperforming both direct hint\-sensitivity training from the target LRM and frontier naive readers that see the same hinted prompt, observed answer, and full completion\.

Table 3:Transfer to hint\-sensitivity forecasting on held\-out FEVEROUS hint templates, for the OLMo\-3\-7B\-Think Behavior Forecaster\. The direct Behavior Forecaster is trained on the hint\-sensitivity surface from the target LRM initialization; the transferred Behavior Forecaster initializes from a counterfactual\-sensitivity model and is then adapted to the same hint\-sensitivity surface\. All methods are evaluated on the same610610held\-out trajectories\.

## 5Ablating the Behavior Forecaster

We now ask which parts of the Behavior Forecaster make its learning possible: the input arrangement, the pretrained initialization, and end\-to\-end backbone training\. We vary the input arrangement \(which of promptPP, reasoningRR, and answerAAare present and in what order\), whether the backbone is initialized from the target LRM or randomly initialized with the same architecture, and whether the backbone is frozen while only the prediction head is trained\. Table[4](https://arxiv.org/html/2606.11445#S5.T4)reports the resulting test correlations\.

Table 4:Behavior Forecaster variants across input arrangement, initialization, and backbone training\. Test Spearman and Pearson for OLMo\-backed ablations\.PP,RR, andAAdenote prompt, reasoning tokens, and final answer\. “OLMo init” is short for OLMo\-3\-7B\-Think initialization\. The bold row in each task is the main Behavior Forecaster used in Section[4](https://arxiv.org/html/2606.11445#S4)\.##### The useful arrangement is task\-specific\.

Counterfactual sensitivity is strongest with the prompt\-echoPP–RR–AA–PParrangement, while rerun consistency is strongest with the plainPP–RR–AAarrangement; swapping one task’s preferred arrangement for the other’s hurts performance on both\. Removing the reasoning tokens hurts both tasks substantially, so the reasoning tokens themselves, not just the prompt and answer, carry the signal the Behavior Forecaster uses\.

##### Initialization from the target LRM helps\.

On both tasks, initializing from OLMo\-3\-7B\-Think is stronger than a randomly initialized backbone with the same architecture\. The gap is larger for counterfactual sensitivity than for rerun consistency\. This suggests that pretrained representations contribute beyond what the architecture alone provides\.

##### An external trained model beats attaching only a head to the LRM\.

Attaching only a prediction head to the original LRM, i\.e\., freezing the OLMo backbone and training only the head, performs well below the fully fine\-tuned Behavior Forecaster \(a separately trained copy of the backbone with its own head\)\. The original LRM’s hidden states already contain useful signal, but adapting a separate model end\-to\-end on top of those states substantially improves extraction of that signal\.

## 6Related work

##### Probing internal states for current\-run properties\.

A line of work trains probes on the internal states of language models to predict properties of the model’s behavior on the current input\. Such methods predict whether a generated statement is factual\(Azaria and Mitchell,[2023](https://arxiv.org/html/2606.11445#bib.bib164); Burnset al\.,[2022](https://arxiv.org/html/2606.11445#bib.bib59)\), whether the model will hallucinate\(Kossenet al\.,[2024](https://arxiv.org/html/2606.11445#bib.bib159); Kadavathet al\.,[2022](https://arxiv.org/html/2606.11445#bib.bib152)\), whether the model will refuse a request\(Arditiet al\.,[2024](https://arxiv.org/html/2606.11445#bib.bib165)\), and broader behavioral attributes such as honesty or power\-seeking tendencies\(Zouet al\.,[2023](https://arxiv.org/html/2606.11445#bib.bib139)\); earlier probing work also recovers behaviorally relevant signals from the residual stream more broadly\(Alain and Bengio,[2017](https://arxiv.org/html/2606.11445#bib.bib81); Belroseet al\.,[2023](https://arxiv.org/html/2606.11445#bib.bib138); Karvonenet al\.,[2025](https://arxiv.org/html/2606.11445#bib.bib131)\)\. A growing line of work applies the same idea inside LRM trajectories: probes on hidden states at intermediate steps predict whether the current intermediate answer is correct, supporting early exit\(Zhanget al\.,[2025](https://arxiv.org/html/2606.11445#bib.bib158)\); attention probes decode the same run’s eventual final answer well before it appears in the generated text\(Boppanaet al\.,[2026](https://arxiv.org/html/2606.11445#bib.bib157)\); and mechanistic analysis finds that in\-context concepts are progressively refined across the trajectory, with steering experiments confirming that this refinement causally drives accuracy\(Kharlapenkoet al\.,[2026](https://arxiv.org/html/2606.11445#bib.bib156)\)\. All of these targets concern the current run: properties of the trajectory or its eventual answer\. Our work targets the LRM’s future behavior instead: rerun consistency and counterfactual sensitivity describe the distribution over future executions related to the input, not properties of the realized one\.

##### Resampling\-based analysis of LRM trajectories\.

Prior work estimates the causal effect of individual reasoning steps on the answer distribution by repeatedly resampling and intervening on the trajectory\(Bogdanet al\.,[2025](https://arxiv.org/html/2606.11445#bib.bib114); Macaret al\.,[2025](https://arxiv.org/html/2606.11445#bib.bib155)\)\. For deployment\-time per\-query decisions, the per\-input cost of these methods is prohibitive: dozens to hundreds of fresh trajectories, each thousands of forward passes long\. We treat the resampling cost as a one\-time investment to label training data, and train a Behavior Forecaster that approximates the same statistics from a single observed trajectory\.

## 7Limitations

##### OOD tasks generalization\.

We test transfer across datasets from the same behavioral tasks in our study, but we do not establish how well Behavior Forecasters generalize to OOD tasks that differ substantially from those used for training\. A stronger test would require training on a much broader pool of tasks\. This is costly because each labeled example requires many additional target\-LRM generations, and those generations can be long and compute\-intensive\. As a result, our findings provide a promising first indication that this approach can work, but deployment would require broader training coverage\.

##### More faithful future LRMs could close the gap\.

Swaroopet al\.\([2025](https://arxiv.org/html/2606.11445#bib.bib14)\)andBakeret al\.\([2025](https://arxiv.org/html/2606.11445#bib.bib13)\)propose explicit methods for training LRMs to produce more faithful chains of thought\. A more faithful target LRM could close part of the gap between a trained Behavior Forecaster and naive reading, since more of the relevant computation would already be visible at the surface\.

## 8Conclusion

Trust in an AI system rests on the ability to forecast how it will behave, which is particularly challenging for LRMs \(Section[2\.3](https://arxiv.org/html/2606.11445#S2.SS3)\)\. We formulated behavior forecasting as its own learning task and introducedBehavior Forecasters, models that forecast behavioral properties of the target LRM from a single observed reasoning trajectory\. We trained Behavior Forecasters on rerun consistency and counterfactual sensitivity, with labels generated by querying the LRM itself\.

The task is learnable: a trained Behavior Forecaster outperforms strong naive readers and single\-location probes, generalizes across datasets with fine\-tuning and across same\-family variants without it, and relies on initialization from the target LRM, end\-to\-end fine\-tuning, and a task\-specific input arrangement\.

Our results show that the trajectory holds more information about the LRM’s future behavior than on surface level\. We hope this motivates further study of behavior forecasting as a learning task: any LRM behavioral property that can be automatically labeled is a candidate target for low\-cost forecasting at deployment scale\.

## References

- Understanding intermediate layers using linear classifier probes\.Cited by:[§6](https://arxiv.org/html/2606.11445#S6.SS0.SSS0.Px1.p1.1)\.
- D\. Alvarez\-Melis and T\. S\. Jaakkola \(2018\)On the robustness of interpretability methods\.InICML Workshop on Human Interpretability in Machine Learning,External Links:[Link](https://arxiv.org/abs/1806.08049)Cited by:[§1](https://arxiv.org/html/2606.11445#S1.p1.1)\.
- R\. Aly, Z\. Guo, M\. Schlichtkrull, J\. Thorne, A\. Vlachos, C\. Christodoulopoulos, O\. Cocarascu, and A\. Mittal \(2021\)FEVEROUS: fact extraction and VERification over unstructured and structured information\.InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks,External Links:[Link](https://arxiv.org/abs/2106.05707)Cited by:[§A\.1](https://arxiv.org/html/2606.11445#A1.SS1.p1.1),[§4\.1](https://arxiv.org/html/2606.11445#S4.SS1.SSS0.Px1.p1.1)\.
- I\. Arcuschin, J\. Janiak, R\. Krzyzanowski, S\. Rajamanoharan, N\. Nanda, and A\. Conmy \(2025\)Chain\-of\-thought reasoning in the wild is not always faithful\.arXiv preprint arXiv:2503\.08679\.Cited by:[§2\.3\.1](https://arxiv.org/html/2606.11445#S2.SS3.SSS1.Px1.p1.1)\.
- A\. Arditi, O\. Obeso, A\. Syed, D\. Paleka, N\. Panickssery, W\. Gurnee, and N\. Nanda \(2024\)Refusal in language models is mediated by a single direction\.External Links:2406\.11717,[Link](https://arxiv.org/abs/2406.11717)Cited by:[§6](https://arxiv.org/html/2606.11445#S6.SS0.SSS0.Px1.p1.1)\.
- A\. Azaria and T\. Mitchell \(2023\)The internal state of an llm knows when it’s lying\.External Links:2304\.13734,[Link](https://arxiv.org/abs/2304.13734)Cited by:[§6](https://arxiv.org/html/2606.11445#S6.SS0.SSS0.Px1.p1.1)\.
- G\. Bachmann, Y\. Jiang, S\. M\. M\. Dezfooli, and M\. Nabi \(2026\)The potential of cot for reasoning: a closer look at trace dynamics\.External Links:2602\.14903,[Link](https://arxiv.org/abs/2602.14903)Cited by:[§2\.3\.1](https://arxiv.org/html/2606.11445#S2.SS3.SSS1.Px1.p1.1)\.
- B\. Baker, J\. Huizinga, L\. Gao, Z\. Dou, M\. Y\. Guan, A\. Madry, W\. Zaremba, J\. Pachocki, and D\. Farhi \(2025\)Monitoring reasoning models for misbehavior and the risks of promoting obfuscation\.External Links:2503\.11926,[Link](https://arxiv.org/abs/2503.11926)Cited by:[§7](https://arxiv.org/html/2606.11445#S7.SS0.SSS0.Px2.p1.1)\.
- A\. Barredo Arrieta, N\. Díaz\-Rodríguez, J\. Del Ser, A\. Bennetot, S\. Tabik, A\. Barbado, S\. García, S\. Gil\-Lopez, D\. Molina, R\. Benjamins, R\. Chatila, and F\. Herrera \(2020\)Explainable artificial intelligence \(XAI\): concepts, taxonomies, opportunities and challenges toward responsible AI\.Information Fusion58,pp\. 82–115\.Cited by:[§1](https://arxiv.org/html/2606.11445#S1.p1.1)\.
- N\. Belrose, Z\. Furman, L\. Smith, D\. Halawi, I\. Ostrovsky, L\. McKinney, S\. Biderman, and J\. Steinhardt \(2023\)Eliciting latent predictions from transformers with the tuned lens\.arXiv preprint arXiv:2303\.08112\.Cited by:[§6](https://arxiv.org/html/2606.11445#S6.SS0.SSS0.Px1.p1.1)\.
- U\. Bhatt, A\. Xiang, S\. Sharma, A\. Weller, A\. Taly, Y\. Jia, J\. Ghosh, R\. Puri, J\. M\. F\. Moura, and P\. Eckersley \(2020\)Explainable machine learning in deployment\.InProceedings of the 2020 Conference on Fairness, Accountability, and Transparency \(FAT\*\),pp\. 648–657\.Cited by:[§2\.2](https://arxiv.org/html/2606.11445#S2.SS2.p1.1)\.
- P\. C\. Bogdan, U\. Macar, N\. Nanda, and A\. Conmy \(2025\)Thought anchors: which llm reasoning steps matter?\.External Links:2506\.19143,[Link](https://arxiv.org/abs/2506.19143)Cited by:[§6](https://arxiv.org/html/2606.11445#S6.SS0.SSS0.Px2.p1.1)\.
- S\. Boppana, A\. Ma, M\. Loeffler, R\. Sarfati, E\. Bigelow, A\. Geiger, O\. Lewis, and J\. Merullo \(2026\)Reasoning theater: disentangling model beliefs from chain\-of\-thought\.External Links:2603\.05488,[Link](https://arxiv.org/abs/2603.05488)Cited by:[§6](https://arxiv.org/html/2606.11445#S6.SS0.SSS0.Px1.p1.1)\.
- C\. Burns, H\. Ye, D\. Klein, and J\. Steinhardt \(2022\)Discovering latent knowledge in language models without supervision\.arXiv preprint arXiv:2212\.03827\.External Links:[Link](https://arxiv.org/abs/2212.03827)Cited by:[§6](https://arxiv.org/html/2606.11445#S6.SS0.SSS0.Px1.p1.1)\.
- Y\. Chen, J\. Benton, A\. Radhakrishnan, J\. Uesato, C\. Denison, J\. Schulman, A\. Somani, P\. Hase, M\. Wagner, F\. Roger, V\. Mikulik, S\. R\. Bowman, J\. Leike, J\. Kaplan, and E\. Perez \(2025\)Reasoning models don’t always say what they think\.External Links:2505\.05410,[Link](https://arxiv.org/abs/2505.05410)Cited by:[§1](https://arxiv.org/html/2606.11445#S1.p5.2),[§2\.3\.1](https://arxiv.org/html/2606.11445#S2.SS3.SSS1.Px2.p2.1),[§4\.1](https://arxiv.org/html/2606.11445#S4.SS1.SSS0.Px3.p1.1)\.
- J\. Chua and O\. Evans \(2025\)Are deepseek r1 and other reasoning models more faithful?\.External Links:2501\.08156,[Link](https://arxiv.org/abs/2501.08156)Cited by:[§1](https://arxiv.org/html/2606.11445#S1.p5.2),[§2\.3\.1](https://arxiv.org/html/2606.11445#S2.SS3.SSS1.Px1.p1.1)\.
- Y\. Chuang, L\. Qiu, C\. Hsieh, R\. Krishna, Y\. Kim, and J\. Glass \(2024\)Lookback lens: detecting and mitigating contextual hallucinations in large language models using only attention maps\.Note:EMNLP 2024 main conference long paperExternal Links:2407\.07071,[Link](https://arxiv.org/abs/2407.07071)Cited by:[§C\.2](https://arxiv.org/html/2606.11445#A3.SS2.SSS0.Px2.p1.4),[§2\.3\.2](https://arxiv.org/html/2606.11445#S2.SS3.SSS2.Px2.p1.1),[§4\.2](https://arxiv.org/html/2606.11445#S4.SS2.SSS0.Px2.p1.4)\.
- P\. Clark, O\. Tafjord, and K\. Richardson \(2020\)Transformers as soft reasoners over language\.InProceedings of the Twenty\-Ninth International Joint Conference on Artificial Intelligence \(IJCAI\),External Links:2002\.05867,[Link](https://arxiv.org/abs/2002.05867)Cited by:[§A\.1](https://arxiv.org/html/2606.11445#A1.SS1.p1.1),[§4\.1](https://arxiv.org/html/2606.11445#S4.SS1.SSS0.Px1.p1.1)\.
- DeepSeek\-AI, D\. Guo, D\. Yang, and H\. Z\. et al\. \(2025\)DeepSeek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.External Links:[Link](https://arxiv.org/abs/2501.12948)Cited by:[§1](https://arxiv.org/html/2606.11445#S1.p2.1)\.
- F\. Doshi\-Velez and B\. Kim \(2017\)Towards a rigorous science of interpretable machine learning\.arXiv preprint arXiv:1702\.08608\.External Links:[Link](https://arxiv.org/abs/1702.08608)Cited by:[§1](https://arxiv.org/html/2606.11445#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.11445#S2.SS1.p1.1)\.
- H\. Douglas \(2009\)Reintroducing prediction to explanation\.Philosophy of Science76,pp\. 444 – 463\.External Links:[Document](https://dx.doi.org/10.1086/648111),[Link](https://api.semanticscholar.org/CorpusID:120278496)Cited by:[§1](https://arxiv.org/html/2606.11445#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.11445#S2.SS1.p1.1)\.
- European Parliament and Council of the European Union \(2024\)Regulation \(eu\) 2024/1689 laying down harmonised rules on artificial intelligence \(artificial intelligence act\)\.Note:Article 14: Human oversight\. Official Journal of the European Union, 12 July 2024External Links:[Link](https://eur-lex.europa.eu/eli/reg/2024/1689/oj)Cited by:[§2\.2](https://arxiv.org/html/2606.11445#S2.SS2.p1.1)\.
- S\. Farquhar, J\. Kossen, L\. Kuhn, and Y\. Gal \(2024\)Detecting hallucinations in large language models using semantic entropy\.Nature630,pp\. 625–630\.External Links:[Document](https://dx.doi.org/10.1038/s41586-024-07421-0),[Link](https://www.nature.com/articles/s41586-024-07421-0)Cited by:[§C\.2](https://arxiv.org/html/2606.11445#A3.SS2.SSS0.Px1.p1.2),[§2\.3\.2](https://arxiv.org/html/2606.11445#S2.SS3.SSS2.Px2.p1.1),[§4\.2](https://arxiv.org/html/2606.11445#S4.SS2.SSS0.Px2.p1.4)\.
- M\. Y\. Guan, M\. Wang, M\. Carroll, Z\. Dou, A\. Y\. Wei, M\. Williams, B\. Arnav, J\. Huizinga, I\. Kivlichan, M\. Glaese, J\. Pachocki, and B\. Baker \(2025\)Monitoring monitorability\.arXiv preprint arXiv:2512\.18311\.Cited by:[§2\.3\.1](https://arxiv.org/html/2606.11445#S2.SS3.SSS1.p2.1)\.
- Y\. Han, Y\. Lee, and J\. Do \(2026\)RFEval: benchmarking reasoning faithfulness under counterfactual reasoning intervention in large reasoning models\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=2Gc8aj0afg)Cited by:[§2\.3\.1](https://arxiv.org/html/2606.11445#S2.SS3.SSS1.Px2.p2.1)\.
- C\. G\. Hempel and P\. Oppenheim \(1948\)Studies in the logic of explanation\.Philosophy of Science15,pp\. 135 – 175\.External Links:[Document](https://dx.doi.org/10.1086/286983),[Link](https://api.semanticscholar.org/CorpusID:16924146)Cited by:[§1](https://arxiv.org/html/2606.11445#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.11445#S2.SS1.p1.1)\.
- K\. A\. Hoff and M\. Bashir \(2015\)Trust in automation: integrating empirical evidence on factors that influence trust\.Human Factors57\(3\),pp\. 407–434\.External Links:[Link](https://doi.org/10.1177/0018720814547570)Cited by:[§2\.1](https://arxiv.org/html/2606.11445#S2.SS1.p1.1)\.
- A\. Jacovi, A\. Marasović, T\. Miller, and Y\. Goldberg \(2021\)Formalizing trust in artificial intelligence: prerequisites, causes and goals of human trust in ai\.InProceedings of the 2021 ACM conference on fairness, accountability, and transparency,pp\. 624–635\.Cited by:[§1](https://arxiv.org/html/2606.11445#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.11445#S2.SS1.p1.1)\.
- S\. Jain and B\. C\. Wallace \(2019\)Attention is not explanation\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),Minneapolis, Minnesota,pp\. 3543–3556\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1357),[Link](https://aclanthology.org/N19-1357/)Cited by:[§C\.2](https://arxiv.org/html/2606.11445#A3.SS2.SSS0.Px2.p1.4),[§2\.3\.2](https://arxiv.org/html/2606.11445#S2.SS3.SSS2.Px2.p1.1),[§4\.2](https://arxiv.org/html/2606.11445#S4.SS2.SSS0.Px2.p1.4)\.
- E\. Jones, M\. Tong, J\. Mu, M\. Mahfoud, J\. Leike, R\. Grosse, J\. Kaplan, W\. Fithian, E\. Perez, and M\. Sharma \(2025\)Forecasting rare language model behaviors\.arXiv preprint arXiv:2502\.16797\.Cited by:[§2\.3\.2](https://arxiv.org/html/2606.11445#S2.SS3.SSS2.Px1.p1.3)\.
- A\. Jose \(2025\)Reasoning models sometimes output illegible chains of thought\.External Links:2510\.27338,[Link](https://arxiv.org/abs/2510.27338)Cited by:[§2\.3\.1](https://arxiv.org/html/2606.11445#S2.SS3.SSS1.Px2.p1.1)\.
- S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Hatfield\-Dodds, N\. DasSarma, E\. Tran\-Johnson, S\. Johnston, S\. El\-Showk, A\. Jones, N\. Elhage, T\. Hume, A\. Chen, Y\. Bai, S\. Bowman, S\. Fort, D\. Ganguli, D\. Hernandez, J\. Jacobson, J\. Kernion, S\. Kravec, L\. Lovitt, K\. Ndousse, C\. Olsson, S\. Ringer, D\. Amodei, T\. Brown, J\. Clark, N\. Joseph, B\. Mann, S\. McCandlish, C\. Olah, and J\. Kaplan \(2022\)Language models \(mostly\) know what they know\.arXiv preprint arXiv:2207\.05221\.Cited by:[§6](https://arxiv.org/html/2606.11445#S6.SS0.SSS0.Px1.p1.1)\.
- A\. Karvonen, J\. Chua, C\. Dumas, K\. Fraser\-Taliente, S\. Kantamneni, J\. Minder, E\. Ong, A\. S\. Sharma, D\. Wen, O\. Evans, and S\. Marks \(2025\)Activation oracles: training and evaluating llms as general\-purpose activation explainers\.External Links:2512\.15674,[Link](https://arxiv.org/abs/2512.15674)Cited by:[§6](https://arxiv.org/html/2606.11445#S6.SS0.SSS0.Px1.p1.1)\.
- D\. Kharlapenko, A\. Stolfo, A\. Conmy, M\. Sachan, and Z\. Jin \(2026\)Fluid representations in reasoning models\.External Links:2602\.04843,[Link](https://arxiv.org/abs/2602.04843)Cited by:[§6](https://arxiv.org/html/2606.11445#S6.SS0.SSS0.Px1.p1.1)\.
- T\. Korbak, M\. Balesni, E\. Barnes, Y\. Bengio, J\. Benton, J\. Bloom, M\. Chen, A\. Cooney, A\. Dafoe, A\. Dragan, S\. Emmons, O\. Evans, D\. Farhi, R\. Greenblatt, D\. Hendrycks, M\. Hobbhahn, E\. Hubinger, G\. Irving, E\. Jenner, D\. Kokotajlo, V\. Krakovna, S\. Legg, D\. Lindner, D\. Luan, A\. Mądry, J\. Michael, N\. Nanda, D\. Orr, J\. Pachocki, E\. Perez, M\. Phuong, F\. Roger, J\. Saxe, B\. Shlegeris, M\. Soto, E\. Steinberger, J\. Wang, W\. Zaremba, B\. Baker, R\. Shah, and V\. Mikulik \(2025\)Chain of thought monitorability: a new and fragile opportunity for ai safety\.External Links:2507\.11473,[Link](https://arxiv.org/abs/2507.11473)Cited by:[§2\.3\.1](https://arxiv.org/html/2606.11445#S2.SS3.SSS1.p2.1)\.
- J\. Kossen, J\. Han, M\. Razzak, L\. Schut, S\. Malik, and Y\. Gal \(2024\)Semantic entropy probes: robust and cheap hallucination detection in llms\.External Links:2406\.15927,[Link](https://arxiv.org/abs/2406.15927)Cited by:[§6](https://arxiv.org/html/2606.11445#S6.SS0.SSS0.Px1.p1.1)\.
- T\. Lanham, A\. Chen, A\. Radhakrishnan, B\. Steiner, C\. Denison, D\. Hernandez, D\. Li, E\. Durmus, E\. Hubinger, J\. Kernion, K\. Lukošiūtė, K\. Nguyen, N\. Cheng, N\. Joseph, N\. Schiefer, O\. Rausch, R\. Larson, S\. McCandlish, S\. Kundu, S\. Kadavath, S\. Yang, T\. Henighan, T\. Maxwell, T\. Telleen\-Lawton, T\. Hume, Z\. Hatfield\-Dodds, J\. Kaplan, J\. Brauner, S\. R\. Bowman, and E\. Perez \(2023\)Measuring faithfulness in chain\-of\-thought reasoning\.arXiv preprint arXiv:2307\.13702\.External Links:[Link](https://arxiv.org/abs/2307.13702)Cited by:[§2\.3\.1](https://arxiv.org/html/2606.11445#S2.SS3.SSS1.Px2.p1.1)\.
- J\. D\. Lee and K\. A\. See \(2004\)Trust in automation: designing for appropriate reliance\.Human Factors46\(1\),pp\. 50–80\.External Links:[Link](https://doi.org/10.1518/hfes.46.1.50.30392)Cited by:[§2\.1](https://arxiv.org/html/2606.11445#S2.SS1.p1.1)\.
- M\. Levy, Z\. Elyoseph, and Y\. Goldberg \(2025a\)Humans perceive wrong narratives from ai reasoning texts\.External Links:2508\.16599,[Link](https://arxiv.org/abs/2508.16599)Cited by:[§2\.3\.1](https://arxiv.org/html/2606.11445#S2.SS3.SSS1.Px2.p1.1)\.
- M\. Levy, Z\. Elyoseph, S\. Ravfogel, and Y\. Goldberg \(2025b\)State over tokens: characterizing the role of reasoning tokens\.arXiv preprint arXiv:2512\.12777\.Cited by:[§1](https://arxiv.org/html/2606.11445#S1.p3.1),[§3\.1](https://arxiv.org/html/2606.11445#S3.SS1.p1.1)\.
- J\. Lindsey, W\. Gurnee, E\. Ameisen, B\. Chen, A\. Pearce, N\. L\. Turner, C\. Citro, D\. Abrahams, S\. Carter, B\. Hosmer, J\. Marcus, M\. Sklar, A\. Templeton, T\. Bricken, C\. McDougall, H\. Cunningham, T\. Henighan, A\. Jermyn, A\. Jones, A\. Persic, Z\. Qi, T\. B\. Thompson, S\. Zimmerman, K\. Rivoire, T\. Conerly, C\. Olah, and J\. Batson \(2025\)On the biology of a large language model\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2025/attribution-graphs/biology.html)Cited by:[§2\.3\.1](https://arxiv.org/html/2606.11445#S2.SS3.SSS1.Px1.p1.1)\.
- Z\. C\. Lipton \(2018\)The mythos of model interpretability\.Queue16\(3\),pp\. 31–57\.Cited by:[§1](https://arxiv.org/html/2606.11445#S1.p1.1)\.
- S\. M\. Lundberg and S\. Lee \(2017\)A unified approach to interpreting model predictions\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§1](https://arxiv.org/html/2606.11445#S1.p1.1),[§2\.2](https://arxiv.org/html/2606.11445#S2.SS2.p1.1)\.
- U\. Macar, P\. C\. Bogdan, S\. Rajamanoharan, and N\. Nanda \(2025\)Thought branches: interpreting llm reasoning requires resampling\.External Links:2510\.27484,[Link](https://arxiv.org/abs/2510.27484)Cited by:[§6](https://arxiv.org/html/2606.11445#S6.SS0.SSS0.Px2.p1.1)\.
- P\. Manakul, A\. Liusie, and M\. Gales \(2023\)SelfCheckGPT: zero\-resource black\-box hallucination detection for generative large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 9004–9017\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.557),[Link](https://aclanthology.org/2023.emnlp-main.557/)Cited by:[§C\.2](https://arxiv.org/html/2606.11445#A3.SS2.SSS0.Px1.p1.2),[§2\.3\.2](https://arxiv.org/html/2606.11445#S2.SS3.SSS2.Px2.p1.1),[§4\.2](https://arxiv.org/html/2606.11445#S4.SS2.SSS0.Px2.p1.4)\.
- A\. Marioriyad, S\. Adim, N\. Alighardashi, M\. S\. Banghshah, and M\. H\. Rohban \(2025\)Unspoken hints: accuracy without acknowledgement in llm reasoning\.External Links:2509\.26041,[Link](https://arxiv.org/abs/2509.26041)Cited by:[§1](https://arxiv.org/html/2606.11445#S1.p5.2),[§2\.3\.1](https://arxiv.org/html/2606.11445#S2.SS3.SSS1.Px1.p1.1)\.
- N\. Maslej, L\. Fattorini, R\. Perrault, Y\. Gil, V\. Parli, N\. Kariuki, E\. Capstick, A\. Reuel, E\. Brynjolfsson, J\. Etchemendy, K\. Ligett, T\. Lyons, J\. Manyika, J\. C\. Niebles, Y\. Shoham, R\. Wald, T\. Walsh, A\. Hamrah, L\. Santarlasci, J\. B\. Lotufo, A\. Rome, A\. Shi, and S\. Oak \(2025\)Artificial intelligence index report 2025\.External Links:2504\.07139,[Link](https://arxiv.org/abs/2504.07139)Cited by:[§4\.1](https://arxiv.org/html/2606.11445#S4.SS1.SSS0.Px3.p1.1)\.
- T\. Miller \(2019\)Explanation in artificial intelligence: insights from the social sciences\.Artificial Intelligence267,pp\. 1–38\.External Links:[Document](https://dx.doi.org/10.1016/j.artint.2018.07.007),[Link](https://doi.org/10.1016/j.artint.2018.07.007)Cited by:[§1](https://arxiv.org/html/2606.11445#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.11445#S2.SS1.p1.1)\.
- P\. Mirtaheri and M\. Belkin \(2026\)Catching rationalization in the act: detecting motivated reasoning before and after CoT via activation probing\.External Links:2603\.17199,[Link](https://arxiv.org/abs/2603.17199)Cited by:[§2\.3\.1](https://arxiv.org/html/2606.11445#S2.SS3.SSS1.Px1.p1.1)\.
- OpenAI \(2024\)OpenAI o1 system card\.Note:Accessed: 2024External Links:[Link](https://openai.com/index/openai-o1-system-card/)Cited by:[§1](https://arxiv.org/html/2606.11445#S1.p2.1)\.
- J\. Ouyang \(2025\)Treecut: a synthetic unanswerable math word problem dataset for llm hallucination evaluation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 1073–1085\.Cited by:[§A\.1](https://arxiv.org/html/2606.11445#A1.SS1.p1.1),[§4\.1](https://arxiv.org/html/2606.11445#S4.SS1.SSS0.Px1.p1.1)\.
- D\. Paul, R\. West, A\. Bosselut, and B\. Faltings \(2024\)Making reasoning matter: measuring and improving faithfulness of chain\-of\-thought reasoning\.External Links:2402\.13950,[Link](https://arxiv.org/abs/2402.13950)Cited by:[§2\.3\.1](https://arxiv.org/html/2606.11445#S2.SS3.SSS1.Px2.p1.1)\.
- M\. T\. Ribeiro, S\. Singh, and C\. Guestrin \(2016\)“Why should I trust you?”: explaining the predictions of any classifier\.InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,pp\. 1135–1144\.Cited by:[§1](https://arxiv.org/html/2606.11445#S1.p1.1),[§2\.2](https://arxiv.org/html/2606.11445#S2.SS2.p1.1)\.
- M\. T\. Ribeiro, T\. Wu, C\. Guestrin, and S\. Singh \(2020\)Beyond accuracy: behavioral testing of NLP models with CheckList\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 4902–4912\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.442)Cited by:[§2\.2](https://arxiv.org/html/2606.11445#S2.SS2.p1.1)\.
- A\. Serrano, W\. Xing, D\. Lindner, and E\. Jenner \(2026\)Frontier models can take actions at low probabilities\.arXiv preprint arXiv:2603\.02202\.Cited by:[§2\.3\.2](https://arxiv.org/html/2606.11445#S2.SS3.SSS2.Px1.p1.3)\.
- M\. Sundararajan, A\. Taly, and Q\. Yan \(2017\)Axiomatic attribution for deep networks\.InProceedings of the 34th International Conference on Machine Learning,pp\. 3319–3328\.Cited by:[§1](https://arxiv.org/html/2606.11445#S1.p1.1),[§2\.2](https://arxiv.org/html/2606.11445#S2.SS2.p1.1),[§2\.3\.2](https://arxiv.org/html/2606.11445#S2.SS3.SSS2.Px2.p1.1)\.
- R\. Sutton \(2019\)The bitter lesson\.Note:[http://www\.incompleteideas\.net/IncIdeas/BitterLesson\.html](http://www.incompleteideas.net/IncIdeas/BitterLesson.html)Cited by:[§3\.1](https://arxiv.org/html/2606.11445#S3.SS1.p1.1)\.
- A\. Swaroop, A\. Nallani, S\. Uboweja, A\. Uzdenova, M\. Nguyen, K\. Zhu, S\. Dev, A\. Panda, V\. Sharma, and M\. Chaudhary \(2025\)FRIT: using causal importance to improve chain\-of\-thought faithfulness\.External Links:2509\.13334,[Link](https://arxiv.org/abs/2509.13334)Cited by:[§7](https://arxiv.org/html/2606.11445#S7.SS0.SSS0.Px2.p1.1)\.
- Team Olmo, A\. Ettinger, A\. Bertsch, B\. Kuehl, D\. Graham, D\. Heineman, D\. Groeneveld, F\. Brahman, F\. Timbers, H\. Ivison, J\. Morrison, J\. Poznanski, K\. Lo, L\. Soldaini, M\. Jordan, M\. Chen, M\. Noukhovitch, N\. Lambert, P\. Walsh, P\. Dasigi, R\. Berry, S\. Malik, S\. Shah, S\. Geng, S\. Arora, S\. Gupta, T\. Anderson, T\. Xiao, T\. Murray, T\. Romero, V\. Graf, A\. Asai, A\. Bhagia, A\. Wettig, A\. Liu, A\. Rangapur, C\. Anastasiades, C\. Huang, D\. Schwenk, H\. Trivedi, I\. Magnusson, J\. Lochner, J\. Liu, L\. J\. V\. Miranda, M\. Sap, M\. Morgan, M\. Schmitz, M\. Guerquin, M\. Wilson, R\. Huff, R\. Le Bras, R\. Xin, R\. Shao, S\. Skjonsberg, S\. Z\. Shen, S\. S\. Li, T\. Wilde, V\. Pyatkin, W\. Merrill, Y\. Chang, Y\. Gu, Z\. Zeng, A\. Sabharwal, L\. Zettlemoyer, P\. W\. Koh, A\. Farhadi, N\. A\. Smith, and H\. Hajishirzi \(2025\)Olmo 3\.External Links:2512\.13961,[Link](https://arxiv.org/abs/2512.13961)Cited by:[§4\.1](https://arxiv.org/html/2606.11445#S4.SS1.SSS0.Px1.p1.1)\.
- Q\. Team \(2026\)Qwen3\.5: accelerating productivity with native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§4\.1](https://arxiv.org/html/2606.11445#S4.SS1.SSS0.Px1.p1.1)\.
- M\. Turpin, J\. Michael, E\. Perez, and S\. R\. Bowman \(2023\)Language models don’t always say what they think: unfaithful explanations in chain\-of\-thought prompting\.InAdvances in Neural Information Processing Systems,Vol\.36\.Note:Paper examining the faithfulness of chain\-of\-thought explanations in language modelsExternal Links:[Link](https://arxiv.org/abs/2305.04388)Cited by:[§1](https://arxiv.org/html/2606.11445#S1.p5.2),[§2\.3\.1](https://arxiv.org/html/2606.11445#S2.SS3.SSS1.Px1.p1.1)\.
- E\. Yee, A\. Li, C\. Tang, Y\. H\. Jung, R\. Paturi, and L\. Bergen \(2024\)Dissociation of faithful and unfaithful reasoning in llms\.arXiv preprint arXiv:2405\.15092\.Cited by:[§2\.3\.1](https://arxiv.org/html/2606.11445#S2.SS3.SSS1.Px1.p1.1)\.
- R\. J\. Young \(2026\)Lie to me: how faithful is chain\-of\-thought reasoning in reasoning models?\.External Links:2603\.22582,[Link](https://arxiv.org/abs/2603.22582)Cited by:[§2\.3\.1](https://arxiv.org/html/2606.11445#S2.SS3.SSS1.Px1.p1.1),[§4\.1](https://arxiv.org/html/2606.11445#S4.SS1.SSS0.Px3.p1.1)\.
- A\. Zhang, Y\. Chen, J\. Pan, C\. Zhao, A\. Panda, J\. Li, and H\. He \(2025\)Reasoning models know when they’re right: probing hidden states for self\-verification\.External Links:2504\.05419,[Link](https://arxiv.org/abs/2504.05419)Cited by:[§6](https://arxiv.org/html/2606.11445#S6.SS0.SSS0.Px1.p1.1)\.
- L\. Zhou, P\. A\.M\. Casares, F\. Martínez\-Plumed, J\. Burden, R\. Burnell, L\. Cheke, C\. Ferri, A\. Marcoci, B\. Mehrbakhsh, Y\. Moros\-Daval, S\. Ó hÉigeartaigh, D\. Rutar, W\. Schellaert, K\. Voudouris, and J\. Hernández\-Orallo \(2026\)Predictable artificial intelligence\.Artificial Intelligence353,pp\. 104491\.External Links:ISSN 0004\-3702,[Document](https://dx.doi.org/10.1016/j.artint.2026.104491),[Link](https://www.sciencedirect.com/science/article/pii/S0004370226000172)Cited by:[§2\.1](https://arxiv.org/html/2606.11445#S2.SS1.p1.1)\.
- A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren, A\. Pan, X\. Yin, M\. Mazeika, A\. Dombrowski, S\. Goel, N\. Li, M\. J\. Byun, Z\. Wang, A\. Mallen, S\. Basart, S\. Koyejo, D\. Song, M\. Fredrikson, J\. Z\. Kolter, and D\. Hendrycks \(2023\)Representation engineering: a top\-down approach to ai transparency\.External Links:2310\.01405,[Link](https://arxiv.org/abs/2310.01405)Cited by:[§6](https://arxiv.org/html/2606.11445#S6.SS0.SSS0.Px1.p1.1)\.

## Appendix AData and label construction

### A\.1Datasets and splits

The paper\-facing experiments use three reasoning datasets: FEVEROUS\(Alyet al\.,[2021](https://arxiv.org/html/2606.11445#bib.bib141)\), RuleTaker\(Clarket al\.,[2020](https://arxiv.org/html/2606.11445#bib.bib166)\), and TreeCut\(Ouyang,[2025](https://arxiv.org/html/2606.11445#bib.bib167)\)\. FEVEROUS includes multiple subsets; we take its four largest labeled subsets and combine them into a single dataset for splitting and balancing\.

We balance by dataset so that no single dataset dominates training and so that headline metrics are not driven by the largest available pool\. For counterfactual sensitivity, the per\-dataset cap is2,0002\{,\}000original samples where available; RuleTaker contributes its full filtered pool of1,6281\{,\}628samples\. For rerun consistency, each dataset contributes1,5001\{,\}500original samples\. The combined FEVEROUS budget is split across its four subsets in proportion to the available pool of each subset, which preserves the natural relative size of each subset within the FEVEROUS budget\.

##### Trajectory length\.

We cap the length of the target LRM’s generation \(reasoning tokens plus final answer\) at sampling time\. We chose this cap to be long enough that the great majority of trajectories complete naturally and short enough that each item fits within the per\-item memory budget of our training hardware\. We use8,0008\{,\}000tokens\. Across the paper\-facing datasets, between98\.6%98\.6\\%and99\.8%99\.8\\%of generated runs reach a final answer within the cap\.

##### Splits\.

The main learnability runs use a sample\-level95/0/595/0/5train/test split with seed4242\. Runs that require validation\-based checkpoint selection use an85/10/585/10/5train/val/test split derived from the same held\-out test surface\.

##### Counterfactual sensitivity counts\.

For each original sample we sample the target LRM1010times on the original prompt and1010times on each perturbed prompt\. A sample is kept only if it has at least55valid runs of the original prompt and an original\-answer consistency of at least0\.70\.7, so its labels are built on a sufficiently stable original answer\. Each kept sample carries multiple segment\-level labels, one per kept removable segment\.

Table[5](https://arxiv.org/html/2606.11445#A1.T5)reports the per\-dataset sample counts\.

Table 5:Counterfactual sensitivity sample counts for the85/10/585/10/5split\. FEVEROUS is treated as one dataset, with its budget distributed across its four subsets\.
##### Metric aggregation\.

For counterfactual sensitivity, each retained segment label is one evaluation item\. We compute Spearman and Pearson correlations over the flattened vector of segment\-level predictions and labels, rather than first averaging predictions within each original sample\. Consequently, samples with more retained segments contribute more evaluation items to the reported correlation\. Table[6](https://arxiv.org/html/2606.11445#A1.T6)reports the corresponding held\-out sample counts and segment\-level evaluation entries\.

Table 6:Evaluation units for counterfactual sensitivity correlations\. Each segment entry contributes one prediction–label pair to Spearman and Pearson\.
##### Rerun consistency counts\.

For each original sample we sample the target LRM1010times on the prompt\. We require at least55valid runs for a sample to be labeled, so each label is built from a sufficient sample size\. At training and evaluation time, the Behavior Forecaster consumes55observed runs per sample as separate inputs\. This separates the runs used to estimate each label \(up to1010, post\-filter\) from the runs actually fed to the Behavior Forecaster \(55, the inference\-time constant we hold fixed across samples\)\.

Table[7](https://arxiv.org/html/2606.11445#A1.T7)reports per\-dataset sample counts\.

Table 7:Rerun consistency per\-dataset sample counts in the85/10/585/10/5split\.

### A\.2Answer extraction

The label\-construction rules in the rest of this appendix treat two runs as producing the same answer when their dataset\-specific extracted values are equal, not when their full output strings match\. We apply the parser to the LRM’s post\-</think\>answer span only\. For each dataset we apply a short ordered list of regular expressions and use the first match; if none matches, the run is marked extraction\-failed and excluded from the labeled run set\.

##### FEVEROUS \(multiple choice\)\.

The three FEVEROUS verdicts \(Supports,Refutes,Not enough info\) are presented as a multiple\-choice question with letters A–C\. The parser looks, in order, for: \(i\) a boxed letter, e\.g\.\\boxed\{A\}; \(ii\) an explicit “\(final \)?answer/choice \(is\|:\) X” phrasing; \(iii\) a single letter on its own line; \(iv\) a trailing letter at end of text\. The extracted value is the matched letter\.

##### RuleTaker \(true/false\)\.

The parser looks, in order, for: \(i\) an explicit “label: true\|false” phrasing; \(ii\) the last occurrence of a standalonetrueorfalsetoken in the answer span\. The extracted value is the matched token, lowercased\.

##### TreeCut \(numeric or unknown\)\.

The parser first checks for an explicitUnknownanswer using “\(final \)?answer \(is\|:\) unknown” or a standaloneunknownat the end of the answer span\. Otherwise it looks for, in order: \(i\) a boxed numeric value, e\.g\.\\boxed\{16\.5\}; \(ii\) an “answer/result \(is\|:\)nn” phrasing; \(iii\) a number at end of text; and as a last resort the last numeric literal anywhere in the answer span\. Two numeric values are treated as the same when they are exactly equal underfloatcomparison;Unknownis its own equivalence class\.

### A\.3Counterfactual sensitivity label construction

For each original promptPPwe define a dataset\-specific set of removable segmentsS​\(P\)S\(P\)and create one perturbed promptP−sP\_\{\-s\}for eachs∈S​\(P\)s\\in S\(P\)\. A run is valid if it has no generation failure, has an extractable answer, reaches the closing</think\>tag, and has non\-empty post\-</think\>answer text\. For each kept prompt, we identify the answerAAproduced by at least0\.70\.7of the1010valid original runs; prompts without such an answer are dropped\.

Letcorigc\_\{\\mathrm\{orig\}\}be the fraction of valid original runs producingAA, andc−sc\_\{\-s\}the fraction of valid perturbed runs onP−sP\_\{\-s\}producingAA\. The segment label is

AF​\(s\)=clip​\(corig−c−scorig,0,1\)\.\\mathrm\{AF\}\(s\)=\\mathrm\{clip\}\\\!\\left\(\\frac\{c\_\{\\mathrm\{orig\}\}\-c\_\{\-s\}\}\{c\_\{\\mathrm\{orig\}\}\},\\,0,\\,1\\right\)\.A perturbed instance is kept only if it has at least55valid runs\. A sample can survive with only a subset of its segment labels; missing segments contribute no supervision during training\. This per\-segment retention rule is intentional: it lets us keep a sample’s well\-supported segments without forcing every segment to meet the run\-count threshold\.

##### Omission units\.

Table[8](https://arxiv.org/html/2606.11445#A1.T8)lists the dataset\-specific unit removed to form one perturbed instance\.

Table 8:Omission units used to construct counterfactual sensitivity perturbations\.

### A\.4Rerun consistency label construction

A run is valid if it has no generation failure and has an extractable answer\. If runiihas extracted answeraia\_\{i\}, andcic\_\{i\}of thennvalid runs on the same input have extracted answeraia\_\{i\}, the run\-level target is

ℓcons​\(i\)=ci−1n−1\.\\ell\_\{\\mathrm\{cons\}\}\(i\)=\\frac\{c\_\{i\}\-1\}\{n\-1\}\.This estimates the likelihood that another run on the same input will produce the same answer as runii\.

If any of a sample’s runs hits the trajectory length cap, we discard the entire sample rather than dropping individual long runs\. Discarding the whole sample preserves the exact run set its labels were computed on, so we do not need to recompute labels after the length filter\.

## Appendix BBehavior Forecaster training and architecture

### B\.1Training objectives

##### Counterfactual sensitivity\.

Counterfactual sensitivity uses the dataset\-defined set of removable segmentsS​\(P\)S\(P\)during training\. For an observed trajectoryτ=\(P,R,A\)\\tau=\(P,R,A\)in the prompt\-echo arrangement, the Behavior Forecaster emits one logitztz\_\{t\}at each token position in the echoed prompt region\. For each retained segments∈S​\(P\)s\\in S\(P\), we build a binary maskms,tm\_\{s,t\}over the echo\-region positions corresponding to that segment\. The Behavior Forecaster first pools logits within the segment,

z¯s=∑tms,t​zt∑tms,t,a^s=σ​\(z¯s\)\.\\bar\{z\}\_\{s\}=\\frac\{\\sum\_\{t\}m\_\{s,t\}z\_\{t\}\}\{\\sum\_\{t\}m\_\{s,t\}\},\\qquad\\hat\{a\}\_\{s\}=\\sigma\(\\bar\{z\}\_\{s\}\)\.The targetasa\_\{s\}is the clipped attributable\-fraction label from Section[A\.3](https://arxiv.org/html/2606.11445#A1.SS3)\. The loss is binary cross\-entropy on the pooled segment logit,

ℒcf=1\|𝒮valid\|​∑s∈𝒮valid\[−as​log⁡σ​\(z¯s\)−\(1−as\)​log⁡\(1−σ​\(z¯s\)\)\]\.\\mathcal\{L\}\_\{\\mathrm\{cf\}\}=\\frac\{1\}\{\|\\mathcal\{S\}\_\{\\mathrm\{valid\}\}\|\}\\sum\_\{s\\in\\mathcal\{S\}\_\{\\mathrm\{valid\}\}\}\\left\[\-a\_\{s\}\\log\\sigma\(\\bar\{z\}\_\{s\}\)\-\(1\-a\_\{s\}\)\\log\(1\-\\sigma\(\\bar\{z\}\_\{s\}\)\)\\right\]\.Thus each retained segment contributes one loss term, even though the Behavior Forecaster produces token\-level logits\. This objective assumes the removable segments and prompt\-alignment masks are provided by the dataset construction; the Behavior Forecaster is trained to score the supplied segments, not to discover the segment boundaries\. At evaluation time we use the same pooled predictions and compute correlations over the flattened set of retained segment labels\.

##### Rerun consistency\.

Rerun consistency is a trajectory\-level prediction problem and has no segment masks\. For each observed trajectoryii, the Behavior Forecaster emits one scalar logitziz\_\{i\}and predictionℓ^i=σ​\(zi\)\\hat\{\\ell\}\_\{i\}=\\sigma\(z\_\{i\}\)\. We train with mean\-squared error against the continuous run\-level consistency label:

ℒrerun=1N​∑i=1N\(ℓ^i−ℓcons​\(i\)\)2\.\\mathcal\{L\}\_\{\\mathrm\{rerun\}\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\(\\hat\{\\ell\}\_\{i\}\-\\ell\_\{\\mathrm\{cons\}\}\(i\)\\right\)^\{2\}\.Each of the55observed runs per sample is treated as a separate training and evaluation item\. All rerun\-consistency results reported in the paper use this continuous objective\.

### B\.2Architectural hyperparameters and transfer fine\-tuning budget

##### Backbone and heads\.

The OLMo\-3\-7B\-Think backbone has hidden sized=4096d=4096\. The token\-level head used for counterfactual sensitivity is a22\-layer MLP with hidden sizes\[512,256\]\[512,256\]applied at each echoed\-prompt position, producing one logit per token\. The sequence\-level head used for rerun consistency pools the backbone’s hidden states with cross\-attention over1212learned query vectors of dimensiondd\(3232attention heads, dropout0\.10\.1\), then concatenates the pooled vectors and passes them through a22\-layer MLP with hidden sizes\[1024,256\]\[1024,256\]and GELU activations between layers \(each layer is Linear→\\toLayerNorm→\\toGELU→\\toDropout\) to produce one scalar logit\.

##### Transfer fine\-tuning budget for Table[2](https://arxiv.org/html/2606.11445#S4.T2)\.

For each held\-out target dataset, we initialize from the source Behavior Forecaster \(trained on the other two datasets with the same recipe as Section[4](https://arxiv.org/html/2606.11445#S4)\) and fine\-tune the entire model on the target’s train split with the same optimizer, weight decay, schedule, effective batch size, and per\-task learning rate as the main runs\. Counterfactual sensitivity uses the95/0/595/0/5surface and the final checkpoint after22epochs of target\-side fine\-tuning; rerun consistency uses an85/10/585/10/5surface for22epochs and selects by best validation loss\.

## Appendix CBaselines and probes

### C\.1Naive reader prompts

Each reader is given three pieces of information per evaluation example: the original input the target LRM saw, the target LRM’s observed final answer for that run, and the target LRM’s full completion text for that run\. The reader returns a continuous score in\[0,1\]\[0,1\]and is not trained or calibrated on the task\.

For each task we tested three prompt variants on a100100\-sample pilot slice of the evaluation set and selected, per reader model, the variant with the highest pilot Spearman correlation\. This per\-reader selection is deliberate: it gives each reader its strongest configuration before we compare against the trained Behavior Forecaster\.

##### Counterfactual sensitivity prompts\.

All three variants ask the reader to score, for each candidate segment, how strongly removing that segment would shift the target LRM’s final answer away from the observed answer\. The variants differ only in framing wording\. GPT\-5\.4 used variant v2 \(Box[C\.1](https://arxiv.org/html/2606.11445#A3.SS1.SSS0.Px1)\); Claude Opus 4\.6 used variant v3 \(Box[C\.1](https://arxiv.org/html/2606.11445#A3.SS1.SSS0.Px1)\)\. Variant v1 frames the score as “how strongly removing the segment would reduce the chance of the same final answer” and is included in the released prompt set\.

Counterfactual sensitivity, variant v3 \(Claude Opus 4\.6\)Below is a problem and one observed run produced by an AI model\. The problem is partitioned into candidate segments\.You will see the original problem, the full completion from that run, the observed final answer, and the candidate segments\.Your task is to estimate, for each candidate segment, how strongly that segment supports the observed answer: if you removed the segment, how likely is the model to move away from the observed answer?Technical details: target model and sampling settings \(temperature=0\.6, top\_p=0\.95\) are provided in the prompt\. Score must be clipped to \[0,1\]\. Answer 0 if removing the segment would still leave the observed answer more likely than a different answer\. Answer 1 if removing the segment would make a different answer more likely than the observed answer\.Return one floating\-point score for each segment as JSON: \{"scores":\{"1":\.\.\.,"2":\.\.\.\}\}, with exactly one entry per candidate segment\.Original problem, observed final answer, observed full completion, and the candidate segment list are interpolated into the prompt as separate fields\.

Counterfactual sensitivity, variant v2 \(GPT\-5\.4\)Below is one observed run from an AI model on a problem\. The problem text has been divided into candidate segments\.You will see the original problem, the model’s observed full completion, the observed final answer, and the candidate segments\.For each candidate segment, imagine removing only that segment from the problem\. Predict how much that removal would reduce the chance that the model gives the same final answer as in the observed run\.Technical details and score range match variant v3\. Answer 0 if, after removing the segment, the same final answer is more likely than a different final answer; answer 1 if a different final answer is more likely\.Return one floating\-point score per segment as JSON, with exactly one entry per candidate segment\.

##### Rerun consistency prompts\.

All three variants ask the reader to forecast how often the observed final answer would recur on reruns, differing only in how the score is framed\. GPT\-5\.4 and Claude Opus 4\.6 both used variant \(a\), the direct stability prompt \(Box[C\.1](https://arxiv.org/html/2606.11445#A3.SS1.SSS0.Px2)\)\. Variant \(b\) reframes the target as “the probability that another run on the same input would produce the same final answer” and variant \(c\) as “the expected fraction of many reruns whose final answer would match the observed final answer”; both are included in the released prompt set\.

Rerun consistency, variant \(a\) \(both readers\)You are forecasting how stable a reasoning model’s final answer is under rerunning\.You are given the exact input the model saw, the model’s observed final answer on this run, and the model’s full completion on this run\.Task: predict the stability score of this observed answer on a scale from 0 to 1\. 0 means this answer would be unique among repeated runs on the same input; 1 means essentially all repeated runs would give this same answer; intermediate values mean this answer would recur on some but not all reruns\.Predict answer stability under rerunning, not correctness\. Base your prediction only on the observed input and this observed run\. Return only a single decimal number between 0 and 1, with no explanation\.

##### Behavior Forecaster and naive reader test metrics\.

Table[9](https://arxiv.org/html/2606.11445#A3.T9)reports the OLMo\-3\-7B\-Think test Spearman and Pearson correlations summarized by the top row of Figure[2](https://arxiv.org/html/2606.11445#S4.F2)\. Behavior Forecaster numbers are the test results used for the paper\-facing comparison\.

Table 9:OLMo\-3\-7B\-Think test Spearman and Pearson correlation for the trained Behavior Forecaster and two frontier naive readers on each task\.Table[10](https://arxiv.org/html/2606.11445#A3.T10)breaks the OLMo\-3\-7B\-Think comparison down by dataset family\. The Behavior Forecaster is more accurate than both naive readers in aggregate and on most per\-dataset cells\.

Table 10:Per\-dataset test Spearman/Pearson correlations for the OLMo\-3\-7B\-Think comparison in Figure[2](https://arxiv.org/html/2606.11445#S4.F2)\. Behavior Forecaster values are computed from the last saved checkpoints: step16681668for counterfactual sensitivity and step13361336for rerun consistency\. For counterfactual sensitivity, the paper\-facing plotted checkpoint is step16001600; the last saved checkpoint differs only in the fourth decimal place in the pooled metric\. FEVEROUS aggregates the four FEVEROUS subsets, matching the three\-dataset framing used in the main text\.

### C\.2Single\-location probe baselines on the target LRM

The dashed single\-location probes in Figure[2](https://arxiv.org/html/2606.11445#S4.F2)are direct adaptations of standard attribution and uncertainty signals to each task: each reads a single fixed location of the frozen target LRM and produces a per\-instance score, with no further training, threshold tuning, or calibration\. They are computed on the same test surfaces as the main figure\.

##### Answer likelihood for rerun consistency\.

For each observed trajectoryτi=\(Pi,Ri,Ai\)\\tau\_\{i\}=\(P\_\{i\},R\_\{i\},A\_\{i\}\), we teacher\-force the stored prompt and completion through OLMo\-3\-7B\-Think and score only the final\-answer tokens after the first closing</think\>tag\. The reported score is the mean final\-answer log probability,

silik=1\|T​\(Ai\)\|​∑t∈T​\(Ai\)log⁡pM​\(xt∣x<t\),s\_\{i\}^\{\\mathrm\{lik\}\}=\\frac\{1\}\{\|T\(A\_\{i\}\)\|\}\\sum\_\{t\\in T\(A\_\{i\}\)\}\\log p\_\{M\}\(x\_\{t\}\\mid x\_\{<t\}\),whereT​\(Ai\)T\(A\_\{i\}\)is the set of final\-answer token positions\. Higher values mean lower answer perplexity and therefore higher predicted rerun consistency\. This baseline follows the broad use of token probabilities as uncertainty or hallucination signals in generation baselines\(Manakulet al\.,[2023](https://arxiv.org/html/2606.11445#bib.bib160); Farquharet al\.,[2024](https://arxiv.org/html/2606.11445#bib.bib161)\); here we intentionally use the simplest answer\-only version rather than semantic clustering or resampling\.

##### Answer\-to\-input attention for counterfactual sensitivity\.

For each labeled input segment, we teacher\-force the observed trajectory through OLMo\-3\-7B\-Think with attention outputs enabled\. We then use the final\-answer tokens as queries and score a candidate segment by the last\-layer attention mass assigned to that segment’s prompt tokens:

sjattn=1H​\|T​\(A\)\|​∑h=1H∑t∈T​\(A\)∑u∈T​\(Sj\)αt,u\(L,h\)\.s\_\{j\}^\{\\mathrm\{attn\}\}=\\frac\{1\}\{H\|T\(A\)\|\}\\sum\_\{h=1\}^\{H\}\\sum\_\{t\\in T\(A\)\}\\sum\_\{u\\in T\(S\_\{j\}\)\}\\alpha^\{\(L,h\)\}\_\{t,u\}\.HereT​\(Sj\)T\(S\_\{j\}\)is the token span of segmentjj,α\(L,h\)\\alpha^\{\(L,h\)\}is headhhin the final transformer layer, and higher values mean that final\-answer tokens attend more to the segment\. This sits in the standard family of attention\-based attribution probes for context use\(Chuanget al\.,[2024](https://arxiv.org/html/2606.11445#bib.bib162); Jain and Wallace,[2019](https://arxiv.org/html/2606.11445#bib.bib163)\), applied here at the natural readout point in an LRM: from the final\-answer tokens back to the prompt\.

Table 11:Single\-location probe baselines on the target LRM, used as dashed lines in Figure[2](https://arxiv.org/html/2606.11445#S4.F2)\. Both are evaluated on the same test surfaces as the main figure\.

### C\.3Target\-LRM naive reader control

As an additional diagnostic, we evaluate OLMo\-3\-7B\-Think itself as a naive reader of its own observed trajectory\. This baseline tests whether the target LRM can recover the future\-behavior signal by prompted reading alone, without any training or calibration\. In the stateless variant, one prompt contains the original input, the observed OLMo completion, the observed final answer, and the same forecasting task given to the other naive readers\. In the stateful variant, we replay the original input as the user message and the observed OLMo completion as the assistant message, then append a follow\-up user message asking for the forecast\.

Table 12:Appendix\-only target\-LRM naive reader control\. OLMo\-3\-7B\-Think is prompted to read its own observed trajectory and predict the target behavioral property without training\.

## Appendix DStatistical uncertainty estimates

For the main learnability comparison on OLMo\-3\-7B\-Think, we estimate uncertainty by cluster bootstrapping over original samples, stratified by dataset family\. This treats an original sample, not a flattened segment or observed run, as the resampling unit\. We use5,0005\{,\}000bootstrap resamples for confidence intervals\. For paired tests, we compare the Behavior Forecaster and each naive reader on exactly matched prediction rows, resampling the same sample clusters for both systems\. We also run a one\-sided paired permutation test with10,00010\{,\}000permutations, swapping the two systems’ predictions together at the original\-sample cluster level and using the directional alternative that the Behavior Forecaster has higher correlation than the naive reader\. Table[13](https://arxiv.org/html/2606.11445#A4.T13)reports Spearman uncertainty for the main metric\.

Table 13:Cluster\-bootstrap uncertainty and one\-sided paired permutation tests for the main Spearman results\. CIs are95%95\\%cluster\-bootstrap intervals over original samples\.For transfer, the unit of replication is the held\-out target dataset, so the analysis is necessarily low\-power with three targets per task\. The adapted source improves over the unadapted source for all three held\-out targets on both tasks\. The mean adapted\-minus\-source Spearman gap is0\.2280\.228with bootstrap CI \[0\.1960\.196,0\.2730\.273\] for counterfactual sensitivity and0\.2680\.268with CI \[0\.0750\.075,0\.4330\.433\] for rerun consistency; the one\-sided sign\-test value isp=0\.125p=0\.125in both cases because there are only three targets\. We treat ablation rows as descriptive single\-run comparisons rather than significance tests, since they do not have repeated seeds or saved paired predictions for every variant\.