Reasoning Errors Have a Region and a Direction in the Residual-Stream Trajectory of LLMs

# Reasoning Errors Have a Region and a Direction in the Residual-Stream Trajectory of LLMs
Source: [https://arxiv.org/html/2608.05660](https://arxiv.org/html/2608.05660)
Hamed Damirchi1,2Ignacio Meza De la Jara1,2,3Damith Ranasinghe2,3Yuhang Liu1,2,4Javen Shi1,2,41Australian Institute for Machine Learning2Adelaide University3Naval Group Pacific4Responsible AI Research Centre, Australia\{firstname\.lastname\}@adelaide\.edu\.au

###### Abstract

As language models are increasingly used for tasks that require verifiable reasoning, reliably distinguishing sound reasoning from flawed reasoning has become an important practical problem\. Recent trajectory\-based methods seek this signal in layerwise residual\-stream displacements, which capture how representations change while attenuating some stable, token\-specific information\. However, displacement omits the state from which an update originates, whereas restoring the full state risks reintroducing shortcut\-prone information\. We identify this trade\-off and propose a three\-stream detector that combines motion with two restricted views of location\. A coarse region reader based on vector quantization and a fine direction reader over normalized multi\-layer states\. This design restores enough state context to interpret the motion without returning to full\-state probing\. On reasoning benchmarks unseen during training, our method improves selection accuracy by up to 12% over the displacement\-only state of the art and 21% over single\-layer probing baselines\. Although trained only on reasoning benchmarks, it also reads factual completion and fact verification, ahead of every detector we compare against, which places the signal on correctness rather than on a kind of reasoning\. Ablations further show that motion, region, and direction provide complementary signals\. These results suggest that reasoning validity is better read from state\-conditioned motion than from either static states or decontextualized trajectories alone\.

![Refer to caption](https://arxiv.org/html/2608.05660v1/x1.png)Figure 1:Three\-stream validity detector\.Motion, quantized region, and direction signals are fused to classify candidate answers as valid or invalid\.## 1Introduction

Language models are increasingly relied on for problems where reaching the correct answer requires reasoning, from a mathematical proof\(Ayoubet al\.[2026](https://arxiv.org/html/2608.05660#bib.bib4)\)to the clinical reasoning behind a treatment recommendation\(Dekonincket al\.[2026](https://arxiv.org/html/2608.05660#bib.bib5)\)\. A user who cannot recheck every step is trusting that the model reasons soundly, and that it can tell a sound line of argument from a flawed one\. That distinction is hard to audit from the answer alone\. A wrong answer can read as fluently as a correct one, and the confidence a model assigns it does not track whether it is right\(Sunet al\.[2025](https://arxiv.org/html/2608.05660#bib.bib6)\)\. One place to look for a signal of reasoning soundness is in the internal representations the model forms as it reads it\.

A common way to probe these representations is to train a linear classifier on activations from a single layer, treating each activation as a point and asking whether a linear boundary separates correct from incorrect answers\(Azizianet al\.[2025](https://arxiv.org/html/2608.05660#bib.bib53)\)\. However, successful classification does not by itself establish that the probe has detected whether the underlying reasoning is sound\. A single\-layer representation may simultaneously contain two types of signal: 1\)reasoning\-related signalsthat reflect the model’s reasoning computation, and 2\)label\-correlated signalsthat are predictive of the label but need not reflect reasoning validity, e\.g\., the wording, format, and topic of the input, token identity, and other dataset\-specific regularities\(Brickenet al\.[2023](https://arxiv.org/html/2608.05660#bib.bib51); Liuet al\.[2025](https://arxiv.org/html/2608.05660#bib.bib55)\)\. In this context, linear probing may exploit label\-correlated signals because it is trained solely to minimize classification error\. One empirical indication of this vulnerability is the poor cross\-benchmark generalization of linear probes\. A probe trained on one benchmark may fail when the wording, format, or topic changes\(Azizianet al\.[2025](https://arxiv.org/html/2608.05660#bib.bib53); Postmus and Abreu[2024](https://arxiv.org/html/2608.05660#bib.bib57); Rimskyet al\.[2024](https://arxiv.org/html/2608.05660#bib.bib48)\)\.

Recent trajectory\-based methods seek a more reliable readout by shifting attention from what information is present in a single\-layer activation to how the representation changes across layers\. Truth as a Trajectory\(Damirchiet al\.[2026](https://arxiv.org/html/2608.05660#bib.bib68)\), for example, represents this evolution through layerwise displacements, the difference between the activation at one layer and that at the next\. These displacements are unrolled across tokens and layers rather than reading the representation from a single layer\. Intuitively, recent work suggests that signals remain recoverable from representations across layers\(Nikolaouet al\.[2025](https://arxiv.org/html/2608.05660#bib.bib1)\), and the linear representation hypothesis further suggests that signals remain approximately linearly decodable across layers\(Gurnee and Tegmark[2024](https://arxiv.org/html/2608.05660#bib.bib2); Turneret al\.[2023](https://arxiv.org/html/2608.05660#bib.bib3)\)\. Consequently, label\-correlated signals that remain stable across adjacent layers may be suppressed by differencing, as their shared contributions cancel in the subtraction\. Consistent with this intuition, trajectory\-based methods have been shown to generalize better across benchmarks than single\-layer probes\(Damirchiet al\.[2026](https://arxiv.org/html/2608.05660#bib.bib68)\)\.

#### Challenges\.

While differencing in trajectory\-based methods may suppress some label\-correlated signals, it may also remove reasoning\-related signals, particularly those encoding the model’s reasoning state\. Differencing suppresses information that remains stable across adjacent layers, and such stability need not separate label\-correlated signals from reasoning\-related ones\. Such reasoning\-state information is important for interpreting changes across layers, since the meaning of a representational change may depend on the state in which it occurs\. As shown in Figure[2](https://arxiv.org/html/2608.05660#S1.F2), supporting and contradicting contexts place otherwise matched statements in different regions of the representation space, indicating that layer states retain context\-dependent validity information\. This observation motivates restoring part of the state alongside the displacement\. The key challenge is therefore to determine what state information, and how much of it, should be restored\. Restoring too little may leave cross\-layer changes difficult to interpret, whereas restoring the full state may reintroduce the label\-correlated signals that differencing was intended to suppress\. This creates a state\-restoration trade\-off, which we examine in Sec\.[3](https://arxiv.org/html/2608.05660#S3)\.

![Refer to caption](https://arxiv.org/html/2608.05660v1/x2.png)Figure 2:Context changes the state of the same statement\. We vary only the context and observe that the resulting states occupy distinct regions of a UMAP projection, with supporting and contradicting contexts landing in different places\.
#### Contributions\.

To address the challenges above, we propose a framework that retains cross\-layer changes while selectively restoring reasoning\-related state information\. It has three components\. A motion reader follows trajectory\-based methods and reads layerwise displacements, preserving their advantage in suppressing some label\-correlated signals\.

The other two are complementary state readers\. The region reader projects normalized multi\-layer states into a learned codebook and replaces each continuous state with its nearest codebook entry, retaining only a coarse state location\. The direction reader instead preserves the normalized state directions across selected answer tokens and layers, retaining finer state distinctions while discarding magnitude and excluding the remaining tokens and layers\. Together, the three readers balance the state\-restoration trade\-off by retaining cross\-layer motion while restoring only the state information needed to interpret it\. A reader designed to restrict state information need not remain restricted once trained, so we show that the restriction holds post\-training\. Our contributions are summarized as follows:

- •We identify a state\-restoration trade\-off\.Trajectory\-based differencing may discard reasoning\-related state information, whereas restoring the full state may reintroduce label\-correlated signals \(Sec\.[3](https://arxiv.org/html/2608.05660#S3)\)\.
- •We address it with two complementary state readers\.A region reader restores coarse state location through codebook quantization, and a direction reader keeps finer state information as normalized per\-layer directions \(Sec\.[4](https://arxiv.org/html/2608.05660#S4)\)\.
- •What the readers restore transfers to benchmarks they never saw\.On reasoning benchmarks unseen during training, our method improves selection accuracy by up to 21% over linear probing and 12% over the displacement\-only state of the art \(Sec\.[6](https://arxiv.org/html/2608.05660#S6)\)\. Trained on reasoning alone, the same detector also outperforms every detector we compare against on unseen factual\-completion and fact\-verification benchmarks, which places what the streams read on correctness rather than on a kind of reasoning\.

## 2Related Work

Work on internal states commonly emphasizes one of two readings\. One treats a hidden state as a static point and asks whether correctness is encoded as a direction at some layer\. The other treats the forward pass as a process and reads how the state changes across layers\. The first asks where the computation is; the second asks how it moves\. Our work joins these readings\. We read the change, as the dynamic view does, together with a compact reading of the location that differencing discards\. We organize prior work around these two readings and the point where ours brings them together\.

### 2\.1Static Linear Representations

Linear representations are a central view in interpretability, where high\-level properties appear as directions in activation space\(Parket al\.[2023](https://arxiv.org/html/2608.05660#bib.bib31); Elhageet al\.[2022](https://arxiv.org/html/2608.05660#bib.bib32)\)\. This view motivates tools with different goals\. Linear probes test whether a property is linearly decodable\(Belinkov[2022](https://arxiv.org/html/2608.05660#bib.bib33)\), sparse autoencoders seek sparse, interpretable features\(Hubenet al\.[2024](https://arxiv.org/html/2608.05660#bib.bib56)\), and contrast\-consistent search finds directions without labels\(Burnset al\.[2022](https://arxiv.org/html/2608.05660#bib.bib35)\)\. Applied specifically to correctness, linear decoding yields a geometry of truth, a direction at a chosen layer that separates correct from incorrect answers and can steer behavior when intervened on\(Marks and Tegmark[2024](https://arxiv.org/html/2608.05660#bib.bib78)\)\. Two things limit it\. The direction is found by an unprincipled layer\-by\-layer search, and it is read from a static point that mixes what the model computes with the surface form of the input\. A probe can therefore rely on dataset\-specific vocabulary or format rather than a signal that transfers\. Directions fitted for different tasks can turn out close to orthogonal\(Azizianet al\.[2025](https://arxiv.org/html/2608.05660#bib.bib53)\), an effect attributed to the polysemantic nature of activations\(Lindseyet al\.[2025](https://arxiv.org/html/2608.05660#bib.bib54)\)\. We read the displacement trajectory rather than a static point\. Differencing attenuates stable surface information, and the compact location streams restore context that the displacement drops\. Together, they recover correctness structure that a single\-layer direction does not\.

### 2\.2Transformers as Trajectories

A second view reads the forward pass as a dynamical system\. The residual updatehℓ\+1=hℓ\+f​\(hℓ\)h\_\{\\ell\+1\}=h\_\{\\ell\}\+f\(h\_\{\\ell\}\)motivates an Euler\-integration view of depth\(Chenet al\.[2018](https://arxiv.org/html/2608.05660#bib.bib37); Luet al\.[2019](https://arxiv.org/html/2608.05660#bib.bib38)\), so the discrete layer\-wise states can be read as a trajectory through activation space, and attention has been modeled as interacting particles that cluster over depth\(Geshkovskiet al\.[2023](https://arxiv.org/html/2608.05660#bib.bib39)\)\. Recent work models the residual stream of a deployed model this way and reports that its effective rank collapses across depth\(Fernando and Guitchounts[2025](https://arxiv.org/html/2608.05660#bib.bib50)\)\. A related line reads depth as its own axis, decoding each layer’s state\(Belroseet al\.[2025](https://arxiv.org/html/2608.05660#bib.bib69)\)or learning a recurrence over the stack of layers rather than over tokens\(Xuet al\.[2024](https://arxiv.org/html/2608.05660#bib.bib71)\)\. The former decodes intermediate states, while the latter processes depth as a sequence\. Truth as a Trajectory\(Damirchiet al\.[2026](https://arxiv.org/html/2608.05660#bib.bib68)\)follows the dynamical view for correctness, reading the layer\-to\-layer displacement so the signal transfers across datasets\. We keep that motion but show that it under\-determines the computation\. The location streams instead read a set of selected depths without a recurrence over depth\.

### 2\.3The Geometry of Reasoning

A separate line of work also studies reasoning through the geometry of a trajectory and shares some of our vocabulary, but it addresses a more structured setting\.Zhouet al\.\([2025](https://arxiv.org/html/2608.05660#bib.bib40)\)split a passage of reasoning into parts, average the final layer’s activations across each part into one vector, and read the sequence as a flow whose velocity tracks logical structure and whose position tracks the topic, a demonstration carried out on deductions built for the study\. Their flow keeps only the final layer of each part and requires a passage already laid out as a chain of steps\. This construction can describe the geometry of a clean deduction, but it requires a problem\-specific segmentation and evaluation setting\. Our detector instead reads supplied question\-answer candidates as they occur in commonly used benchmarks, with motion from their residual\-stream displacements and compact location streams from their states\. This lets one detector train on a reasoning benchmark and select correct candidates on unseen reasoning benchmarks and factuality tasks, without requiring an explicit chain of reasoning or a task\-specific segmentation\.

## 3The Trade\-Off in Restoring States

Differencing retains cross\-layer changes but discards the state in which they occur\. On ARC\-Challenge\(Clarket al\.[2018](https://arxiv.org/html/2608.05660#bib.bib58)\), a linear probe on one mid\-network state selects the correct answer on68\.6%68\.6\\%of items, compared with62\.2%62\.2\\%for the displacement reader\. The state therefore holds correctness\-relevant information that differencing does not pass on\. But restoring the state also exposes answer wording\. At the probe’s layer, a last\-token state gives wording reconstructionR2=0\.086R^\{2\}\{=\}0\.086and retrieval2\.262\.26, compared with0\.0070\.007and1\.011\.01for displacement; pooling every answer token raises these to0\.1910\.191and3\.133\.13while losing0\.90\.9points of selection accuracy\. The probe’s donor advantage also falls to0\.90\.9points on held\-out reasoning benchmarks \(Table[1](https://arxiv.org/html/2608.05660#S6.T1)\)\. Figure[3](https://arxiv.org/html/2608.05660#S3.F3)shows the resulting trade\-off\. We therefore restore only the state information needed to interpret cross\-layer changes, rather than the full representation\.

![Refer to caption](https://arxiv.org/html/2608.05660v1/x3.png)Figure 3:The trade\-off the detector has to solve\.Each reader is placed by how often it selects the correct answer on ARC\-Challenge, and by how much of that answer’s wording a held\-out regression recovers from what it hands its decision head\. Ours is marked at whichever of its three streams exposes the most wording\. The two raw\-state readers are measured at layer1414, and accuracy for the displacement reader and for ours is the in\-distribution cell of Table[1](https://arxiv.org/html/2608.05660#S6.T1)\.
## 4A Three\-Reader Framework

The previous section motivates a principle ofminimally sufficient state restoration, under which a detector retains cross\-layer changes while exposing only the state information needed to interpret them\. Figure[1](https://arxiv.org/html/2608.05660#S0.F1)shows the three\-reader framework that instantiates it\. A motion reader captures cross\-layer changes through layerwise displacements, preserving the central advantage of trajectory\-based methods\. A region reader and a direction reader then selectively restore complementary forms of state information at different levels of detail\. Their outputs are combined to produce a single validity score for each candidate answer\.

#### Setup\.

We work in the selection setting\. A model is shown a question and a set of candidate answers, one correct and the rest not\. For each candidate we pass the question and that candidate through the model and record its residual\-stream activations, leaving the model unchanged\. Writeht,ℓ∈ℝdh\_\{t,\\ell\}\\in\\mathbb\{R\}^\{d\}for the activation at tokenttand layerℓ\\ell, so that across theLLlayers of one pass these states form a trajectory through activation space\. The detector scores each candidate from that trajectory, and selects the highest\-scoring one\.

### 4\.1Motion Reader for Cross\-Layer Changes

The motion reader is the model of prior work\(Damirchiet al\.[2026](https://arxiv.org/html/2608.05660#bib.bib68)\), which we take as it stands\. It reads the trajectory through its displacement,

dt,ℓ=ht,ℓ\+1−ht,ℓ,d\_\{t,\\ell\}=h\_\{t,\\ell\+1\}\-h\_\{t,\\ell\},\(1\)the update that layerℓ\\ellwrites into the residual stream\. Differencing attenuates the persistent, token\-specific content the stream already holds, which is what lets the signal transfer across datasets\(Elhageet al\.[2023](https://arxiv.org/html/2608.05660#bib.bib70)\)\. Stacking the displacements over every token and layer gives one sequence, and a bidirectional LSTM reads it and returns its final state as the motion embeddingcmotc\_\{\\mathrm\{mot\}\}\. Nothing in this reading is new, and nothing about it is changed\.

### 4\.2Shared Representation for State Restoration

The two location readers begin by giving up a state’s size\. The magnitude of the residual stream grows with depth, so states drawn from different layers do not share a scale, and a reader that kept it would pass that growth rather than the content\. What they keep is the direction,

ut,ℓ=ht,ℓ/∥ht,ℓ∥,u\_\{t,\\ell\}=h\_\{t,\\ell\}/\\lVert h\_\{t,\\ell\}\\rVert,\(2\)and throughout, the direction of a state means this orientation, where the state itself points, not the direction the trajectory travels in, which the motion already records\. Both are confined to the answer, since the question is common to every candidate, with𝒜\\mathcal\{A\}its tokens andt⋆t^\{\\star\}its final token\. Both are also restricted to a fixed windowℒ\\mathcal\{L\}ofm=6m=6layers in the middle\-to\-late range of the network\. Early layers group tokens by surface form rather than by what they mean\(Nepalet al\.[2025](https://arxiv.org/html/2608.05660#bib.bib19)\), late layers work in the space of the token about to be emitted\(Yaoet al\.[2024](https://arxiv.org/html/2608.05660#bib.bib20); Duttaet al\.[2024](https://arxiv.org/html/2608.05660#bib.bib21); Xieet al\.[2024](https://arxiv.org/html/2608.05660#bib.bib22)\), and between the two, representations come loose from the wording that produced them\(Menget al\.[2022](https://arxiv.org/html/2608.05660#bib.bib23)\), which is the surface detail the previous section required a reading to leave behind\. Our probe baseline agrees, settling between layers1313and1616of3232for every donor when free to choose\. The window is fixed in advance rather than selected on results, and Table[19](https://arxiv.org/html/2608.05660#A15.T19)sweeps it\. At an answer token, every layer’s direction passes through one projectionWWshared across depth, and the results are concatenated,

gt=\[W​ut,ℓ1;…;W​ut,ℓm\]\.g\_\{t\}=\\big\[\\,Wu\_\{t,\\ell\_\{1\}\}\\,;\\,\\dots\\,;\\,Wu\_\{t,\\ell\_\{m\}\}\\,\\big\]\.\(3\)Normalization is what allowsWWto be shared, since with magnitudes left in each slot would track a layer’s depth rather than its content, and depth survives because each layer keeps its own slot\. The two share this construction but not its weights, each fitting its ownWW, and differ only in how much ofgtg\_\{t\}each keeps\.

### 4\.3Region Reader for Coarse State Restoration

The region reader keeps only which part of the state space an answer occupies, not where inside that part it falls\. A learned codebook holdsKKentries, trained jointly with the detector, andKKsets the resolution\. More entries pass a finer location, fewer a coarser one\.

The reader uses the answer’s final token, where the answer has settled\. Its projected directionsgt⋆g\_\{t^\{\\star\}\}pass through an encoder intoz=Enc​\(gt⋆\)z=\\mathrm\{Enc\}\(g\_\{t^\{\\star\}\}\), then the code is replaced by the nearest entry\{ek\}k=1K\\\{e\_\{k\}\\\}\_\{k=1\}^\{K\},

q​\(z\)=ek⋆,k⋆=arg⁡mink⁡∥z−ek∥\.q\(z\)=e\_\{k^\{\\star\}\},\\qquad k^\{\\star\}=\\arg\\min\_\{k\}\\,\\lVert z\-e\_\{k\}\\rVert\.\(4\)The head receivesq​\(z\)q\(z\)ascregc\_\{\\mathrm\{reg\}\}, so two states assigned the same entry arrive as the same input and no continuous code passes through the bottleneck\. Quantization is hard for the head, but learning uses the straight\-through estimatez~=z\+sg​\[ek⋆−z\]\\tilde\{z\}=z\+\\mathrm\{sg\}\[\\,e\_\{k^\{\\star\}\}\-z\\,\]\(van den Oordet al\.[2017](https://arxiv.org/html/2608.05660#bib.bib79)\)\. We initialize from data, update entries by moving average, reseed unused entries, and use a commitment term to prevent codebook collapse\(Lancuckiet al\.[2020](https://arxiv.org/html/2608.05660#bib.bib80)\); Appendix[M](https://arxiv.org/html/2608.05660#A13)reports code usage\.

### 4\.4Direction Reader for Fine State Restoration

A region names the part of space a state fell in and says nothing else, so a correct and an incorrect answer landing in the same region reach the head as one answer\. The direction reader keeps what that discards, where the state sits inside the space rather than the name of the part it occupies\.

It therefore keepsgtg\_\{t\}whole, recording where the state points at each depth inℒ\\mathcal\{L\}separately rather than compressing the depths into one label\. It has already given up the magnitude, every layer outsideℒ\\mathcal\{L\}and every token outside the answer, so its narrowness comes from what it declines to read\.

A small MLP, shared across answer tokens, maps eachgtg\_\{t\}topt=MLP​\(gt\)p\_\{t\}=\\mathrm\{MLP\}\(g\_\{t\}\)\. Nothing in the concatenation relates one depth to another, and the MLP is what lets a pattern visible only across depths be represented\. An answer is a span rather than a point, so the reading combines two summaries of it,

cdir=Wc​\[pt⋆;1\|𝒜\|​∑t∈𝒜pt\]\.c\_\{\\mathrm\{dir\}\}=W\_\{c\}\\big\[\\,p\_\{t^\{\\star\}\}\\,;\\,\\tfrac\{1\}\{\|\\mathcal\{A\}\|\}\\textstyle\\sum\_\{t\\in\\mathcal\{A\}\}p\_\{t\}\\,\\big\]\.\(5\)The final token holds the state the answer settles on, and the mean holds the states it passed through on the way\.

### 4\.5Three Readers, One Decision

Each reader returns one embedding\. The three are concatenated and passed to an MLP headffthat outputs a validity score,

y^=σ​\(f​\(\[cmot;cdir;creg\]\)\),\\hat\{y\}=\\sigma\\big\(f\(\[\\,c\_\{\\mathrm\{mot\}\}\\,;\\,c\_\{\\mathrm\{dir\}\}\\,;\\,c\_\{\\mathrm\{reg\}\}\\,\]\)\\big\),\(6\)and among a question’s candidates the one with the highesty^\\hat\{y\}is selected\. All parts train together from scratch on the pairwise objective, with the commitment term added to the loss\.

The embeddings from the readers are concatenated\. Note that these embeddings are not symmetric in how they treat depth, since the motion passes through the recurrence of an LSTM in layer order while the two location readings treat the depths inℒ\\mathcal\{L\}as an unordered set\.

## 5Experimental Setup

#### Models and features\.

All base\-model weights remain frozen\. We use Llama\-3\.1\-8B\(Grattafioriet al\.[2024](https://arxiv.org/html/2608.05660#bib.bib72)\), Qwen2\.5\-14B\(Yanget al\.[2025b](https://arxiv.org/html/2608.05660#bib.bib73)\), and Qwen3\-30B\-A3B\(Yanget al\.[2025a](https://arxiv.org/html/2608.05660#bib.bib74)\), spanning two families,3232and4848layers, and a mixture of experts with33B active parameters\. One forward pass per candidate records residual states at every layer and token; every reader uses these activations\. Appendix[A](https://arxiv.org/html/2608.05660#A1)gives the model depths and extraction procedure\.

#### Datasets\.

We use ARC\-Challenge and ARC\-Easy\(Clarket al\.[2018](https://arxiv.org/html/2608.05660#bib.bib58)\), OpenBookQA\(Mihaylovet al\.[2018](https://arxiv.org/html/2608.05660#bib.bib61)\), CommonsenseQA\(Talmoret al\.[2019](https://arxiv.org/html/2608.05660#bib.bib63)\), Social IQa\(Sapet al\.[2019](https://arxiv.org/html/2608.05660#bib.bib65)\), and HellaSwag\(Zellerset al\.[2019](https://arxiv.org/html/2608.05660#bib.bib60)\)as donors, the one benchmark a detector trains on; evaluate reasoning transfer on those six benchmarks plus Story Cloze\(Mostafazadehet al\.[2016](https://arxiv.org/html/2608.05660#bib.bib62)\)and MMLU\(Hendryckset al\.[2021](https://arxiv.org/html/2608.05660#bib.bib75)\); and test factual transfer on FACTOR\-wiki, FACTOR\-news, FACTOR\-expert\(Muhlgayet al\.[2024](https://arxiv.org/html/2608.05660#bib.bib76)\), and VitaminC\(Schusteret al\.[2021](https://arxiv.org/html/2608.05660#bib.bib77)\)\. TriviaQA and HaluEval are excluded\. Appendix[B](https://arxiv.org/html/2608.05660#A2)gives the dataset details\.

#### Task and metric\.

Each item supplies one correct candidate and one or more incorrect ones\. We train on one donor benchmark and score each candidate from a frozen\-model forward pass\. Selection accuracy is the fraction of items whose correct candidate outscores every incorrect candidate\. We compare against a linear probe on one validation\-selected layer and the motion\-only reader\(Damirchiet al\.[2026](https://arxiv.org/html/2608.05660#bib.bib68)\), using identical items and splits\. Appendix[C](https://arxiv.org/html/2608.05660#A3)gives the training recipe\.

## 6Experiments

Two questions come first\. Trained on a single reasoning benchmark, does restoring the location improve the detection of reasoning errors on benchmarks held out from training, and does the same detector identify factual errors it was never shown? The rest ask where the gain comes from\.

### 6\.1Does restoring the location improve transfer to unseen reasoning benchmarks?

We train a detector on one reasoning benchmark and evaluate it on the rest\. Table[1](https://arxiv.org/html/2608.05660#S6.T1)reports every training donor and method, one column per evaluation benchmark, with the per\-row average, the in\-distribution score \(ID\), and the out\-of\-distribution average over the unseen targets \(OOD\)\. Among the three trained readers the full model is best on4747of the4848cells, the lone exception MMLU under the HellaSwag donor, where the motion\-only reader edges ahead\. Figure[5](https://arxiv.org/html/2608.05660#A7.F5)in the appendix maps that gain cell by cell, the difference between our full model and a motion\-only reader\. The gain is spread across the grid rather than concentrated in a few donor and target pairs, and it is largest on transfer into HellaSwag\. Its out\-of\-distribution transfer improves on the linear probe by1010to2121points across donors, and on the motion\-only model of prior work by77to1212, with its largest margin over prior work,1212points, on the ARC\-Challenge donor\. Restoring the location is what moves the signal onto reasoning the detector was not trained on\. Across training seeds, the full model’s per\-donor reasoning means move by at most about a point, far under the gap to either baseline \(Table[8](https://arxiv.org/html/2608.05660#A8.T8)\)\. Three controls place that gain on the design rather than on what came with it\. Scaling the motion reader to four times the width does not close it, removing magnitude from the motion reader does not reproduce it, and the streams give back less of an answer’s wording than the state a probe reads \(Appendix[K](https://arxiv.org/html/2608.05660#A11)\)\.

Table 1:Cross\-task transfer of reasoning\-error detection, selection accuracy\. Each detector trains on the row benchmark and is scored on every column\.Avgis the row mean over the eight targets,IDthe score on the training benchmark, andOODthe mean over the unseen ones, so every off\-diagonal cell is zero\-shot transfer\. Best per cell in bold, among the three trained methods\. Base Model is the frozen model’s own answer likelihood, with no donor and no ID\. On Llama\-3\.1\-8B, Ours is the three\-seed mean \(Table[8](https://arxiv.org/html/2608.05660#A8.T8)\)\.Finding\.What the location streams restore is not specific to the benchmark it was learned on\. Every donor yields a detector that reads reasoning it never trained on more accurately than both the motion\-only method and a probe\.

### 6\.2Does what the detector reads extend beyond reasoning?

Nothing in training touches factual claims, yet a reasoning\-trained detector identifies factual errors it was never shown\. Table[2](https://arxiv.org/html/2608.05660#S6.T2)puts reasoning\-trained detectors on two factual benchmarks of different form, with no further training\. In FACTOR\(Muhlgayet al\.[2024](https://arxiv.org/html/2608.05660#bib.bib76)\)the wrong option is a minimal single\-fact edit of a correct completion\. In VitaminC\(Schusteret al\.[2021](https://arxiv.org/html/2608.05660#bib.bib77)\)a claim is checked against a piece of evidence\. We report the three donors that transfer best on reasoning, with the grid for every donor in Table[9](https://arxiv.org/html/2608.05660#A9.T9)\. The full model is the best of the three trained readers on all four targets, ahead of the motion\-only model by99to1919points, and that lead holds in every one of the twelve donor cells behind the means\. The frozen model’s own likelihood sits beside them, and it reads the FACTOR sets above the full model while trailing it by1818points on reasoning, since a single\-fact edit of a fluent completion is close to an unlikely one whereas a reasoning error is not \(Appendix[F](https://arxiv.org/html/2608.05660#A6)\)\.

The two baselines trade places across the four targets\. The probe reads FACTOR\-expert and VitaminC better than the displacement reader does, while the displacement reader leads on the two completion sets\. The full model is above both on every target, so its reach does not depend on which baseline suits a given set\. Averaging over all six donors rather than three narrows the gain but leaves it positive, since a donor that transfers weakly on reasoning reaches the factual sets weakly too\. Some benchmarks we considered were read accurately by every method, ours included\. We exclude TriviaQA and HaluEval because a shortest\-answer heuristic, without access to the model, reaches88\.888\.8and95\.995\.9selection accuracy; every reported target is below chance under the same heuristic \(Appendix[D](https://arxiv.org/html/2608.05660#A4)\)\. A linear read trained on each factual set peaks at layers1212to1616\(Figure[6](https://arxiv.org/html/2608.05660#A9.F6)\), inside the band the location streams already read\. Neither result is particular to Llama\-3\.1\-8B, and on a dense Qwen2\.5\-14B and a Qwen3\-30B\-A3B mixture of experts the full model stays the best of the three readers on both axes, by88to1414points on reasoning and1111to1616on the factual sets \(Appendix[N](https://arxiv.org/html/2608.05660#A14)\)\.

Table 2:The reach into factual errors, selection accuracy\. Reasoning\-trained detectors, scored zero\-shot on four factual targets \(full test sets\) and averaged over the three donors that transfer best on reasoning \(ARC\-Challenge, ARC\-Easy, CommonsenseQA\)\. Length picks the shortest candidate, and Base Model, set apart on the right, is the frozen model’s own answer likelihood\. Bold marks the best trained detector\. Single seed; per\-donor cells in Table[9](https://arxiv.org/html/2608.05660#A9.T9)\.Finding\.What the location streams read is closer to correctness than to any particular kind of reasoning\. A detector trained on reasoning alone, and never shown a factual claim, picks the true completion out of minimal edits and checks a claim against its evidence\.

### 6\.3Which added stream does the work?

Table[3](https://arxiv.org/html/2608.05660#S6.T3)adds one location stream at a time to the motion base, on a single donor\. Each addition lifts the motion stream’s performance on its own, the direction stream by33to2020points across the eight reasoning targets and the region stream by44to1515on seven of them, leaving CommonsenseQA where it found it\. The two together read best on every target\. Removing the direction stream from that pair costs0\.80\.8to5\.75\.7points and removing the region stream at most2\.12\.1, though on HellaSwag the pair and the direction stream alone are level\.The gain rests on neither stream alone\.Whether the two recover the same answers is a question these accuracies do not settle\. A late\-fusion variant that scores each stream on its own answers it, where every stream uniquely recovers items the other two miss and a per\-item oracle stands33to2121points above the trained combination \(Appendix[J](https://arxiv.org/html/2608.05660#A10)\)\. Counting which streams read an item together shows the region reading to sit almost inside the direction reading on reasoning but not on VitaminC, so which readings agree is set by the kind of question asked\.

Table 3:Component ablation, selection accuracy, adding the direction stream, the region stream, or both to the motion base, on the ARC\-Challenge donor and the eight reasoning targets\. All four columns come from one seed, so Ours here is a single run rather than the three\-seed mean of Table[1](https://arxiv.org/html/2608.05660#S6.T1)\. Best per row in bold\.
### 6\.4What does the region code hold?

The region stream hands the head one of128128entries and nothing else\. What a single entry is worth depends on how much an answer’s location says about whether it is correct\.

Before any compression, a linear classifier on the raw residual\-stream state at layer1414, scored on held\-out items, tells a correct completion from an incorrect one on FACTOR\-expert at an AUC of0\.920\.92, against0\.50\.5for a read that says nothing\. On FACTOR\-wiki the same classifier reaches0\.610\.61\(Figure[4](https://arxiv.org/html/2608.05660#S6.F4)\), so how much location holds depends on the benchmark\.

Part of that signal survives compression to a single entry\. Predicting correctness from the region alone beats the base rate on all four sets \(Table[15](https://arxiv.org/html/2608.05660#A13.T15)\)\. The same table counts the entries each set uses, and Appendix[M](https://arxiv.org/html/2608.05660#A13)maps which ones\. No entry belongs to the factual sets alone, and every entry a factual answer takes is one a reasoning answer takes as well\.The detector was never given a vocabulary for factual answers and did not need one, which is part of why it reads errors it was not trained on\.

![Refer to caption](https://arxiv.org/html/2608.05660v1/x4.png)Figure 4:A held\-out linear read of the raw residual\-stream state at layer1414, out\-of\-fold, on two FACTOR domains, against0\.50\.5for a read that says nothing\. The figure shows the state itself and not the learned region code, so it measures the location signal available to be compressed rather than what the region stream keeps\. Layer1414is the depth the probe baseline most often selects\.Table 4:The region code as a shared vocabulary\. Distinct codes used of the128128, and the accuracy of predicting correctness from the region alone \(majority vote\) against the base rate\. One donor’s codebook \(ARC\-Challenge\), single seed, up to800800items per dataset; the majority vote is a lower bound on the code’s signal\.
### 6\.5Where in the network does the location have to be read?

The middle third of the stack gives the highest reasoning mean,75\.675\.6, compared with63\.763\.7early and69\.569\.5late, while all layers give the highest factual mean,63\.763\.7\. Six layers in the middle third also read higher than all3232on reasoning,75\.675\.6against74\.074\.0, so which depths are read matters more than how many\. We retain the window fixed before these runs rather than selecting it on the sweep\. Appendix[O](https://arxiv.org/html/2608.05660#A15)gives the controlled comparison and its implications for the reasoning and factual axes\.

## 7Limitations

Our detector is evaluated only by selecting among supplied candidates, not by judging whether a single answer is correct\. It needs the residual stream, a forward pass per candidate, a labeled donor benchmark, and separate training for each model\. Its performance is bounded by how much the model state itself distinguishes correct from incorrect answers, and this bound varies across target sets\. The location streams retain direction but discard activation magnitude, which may contain correctness signal\. Finally, late fusion does not use all information in the three streams\. The gap between the trained combination and a per\-item oracle indicates room for a learned gate that selects or weighs streams for each item\. Appendix[P](https://arxiv.org/html/2608.05660#A16)gives the detailed evidence and discussion\.

## 8Conclusion

A displacement records how a computation moves and drops where it sits\. We restore that location beside the motion at two resolutions, a discrete region and a per\-layer direction, and read all three at the answer\. Kept narrow by design, the two added streams are6\.7%6\.7\\%of the motion reader’s parameters and give back little of an answer’s wording\. Trained on one reasoning benchmark and scored on benchmarks held out from training, the detector improves on a linear probe by1010to2121points of selection accuracy and on the displacement reader by77to1212\. Trained on reasoning alone, it identifies factual errors it was never shown, both minimal single\-fact edits and claims checked against their evidence\. Both results hold on a larger dense model and on a mixture of experts\. The three readings are right about different items, and the region an answer occupies is a coordinate that reasoning and factual errors share\. Where a computation sits and how it moves are separable readings of one trajectory, and correctness is legible in both\.

## References

- Structured clinical approach to enable large language models to be used for improved clinical diagnosis and explainable reasoning\.Communications Medicine6\(1\),pp\. 86\.External Links:[Document](https://dx.doi.org/10.1038/s43856-025-01348-x)Cited by:[§1](https://arxiv.org/html/2608.05660#S1.p1.1)\.
- W\. Azizian, M\. Kirchhof, E\. Ndiaye, L\. Béthune, M\. Klein, P\. Ablin, and marco cuturi \(2025\)The geometries of truth are orthogonal across tasks\.InICML 2025 Workshop on Reliable and Responsible Foundation Models,External Links:[Link](https://openreview.net/forum?id=FdfvGu5rM5)Cited by:[§1](https://arxiv.org/html/2608.05660#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.05660#S2.SS1.p1.1)\.
- Y\. Belinkov \(2022\)Probing classifiers: promises, shortcomings, and advances\.Computational Linguistics48\(1\),pp\. 207–219\.Cited by:[§2\.1](https://arxiv.org/html/2608.05660#S2.SS1.p1.1)\.
- N\. Belrose, I\. Ostrovsky, L\. McKinney, Z\. Furman, L\. Smith, D\. Halawi, S\. Biderman, and J\. Steinhardt \(2025\)Eliciting latent predictions from transformers with the tuned lens\.External Links:2303\.08112,[Link](https://arxiv.org/abs/2303.08112)Cited by:[§2\.2](https://arxiv.org/html/2608.05660#S2.SS2.p1.1)\.
- T\. Bricken, A\. Templeton, J\. Batson, B\. Chen, A\. Jermyn, T\. Conerly, N\. L\. Turner, C\. Anil, C\. Denison, A\. Askell, R\. Lasenby, Y\. Wu, S\. Kravec, N\. Schiefer, T\. Maxwell, N\. Joseph, A\. Tamkin, K\. Nguyen, B\. McLean, J\. E\. Burke, T\. Hume, S\. Carter, T\. Henighan, and C\. Olah \(2023\)Towards monosemanticity: decomposing language models with dictionary learning\.Note:https://transformer\-circuits\.pub/2023/monosemantic\-featuresTransformer Circuits ThreadCited by:[§1](https://arxiv.org/html/2608.05660#S1.p2.1)\.
- C\. Burns, H\. Ye, D\. Klein, and J\. Steinhardt \(2022\)Discovering latent knowledge in language models without supervision\.arXiv preprint arXiv:2212\.03827\.Cited by:[§2\.1](https://arxiv.org/html/2608.05660#S2.SS1.p1.1)\.
- R\. T\. Q\. Chen, Y\. Rubanova, J\. Bettencourt, and D\. Duvenaud \(2018\)Neural ordinary differential equations\.Advances in Neural Information Processing Systems31\.Cited by:[§2\.2](https://arxiv.org/html/2608.05660#S2.SS2.p1.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[Appendix B](https://arxiv.org/html/2608.05660#A2.p2.1),[§3](https://arxiv.org/html/2608.05660#S3.p1.10),[§5](https://arxiv.org/html/2608.05660#S5.SS0.SSS0.Px2.p1.1)\.
- H\. Damirchi, Imezadelajara, E\. Abbasnejad, A\. Shamsi, Z\. Zhang, and J\. Q\. Shi \(2026\)Truth as a trajectory: what internal representations reveal about large language model reasoning\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 44774–44790\.External Links:[Link](https://aclanthology.org/2026.acl-long.2073/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.2073),ISBN 979\-8\-89176\-390\-6Cited by:[Appendix C](https://arxiv.org/html/2608.05660#A3.SS0.SSS0.Px2.p1.2),[§1](https://arxiv.org/html/2608.05660#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.05660#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2608.05660#S4.SS1.p1.3),[§5](https://arxiv.org/html/2608.05660#S5.SS0.SSS0.Px3.p1.1)\.
- J\. Dekoninck, I\. Petrov, K\. Minchev, M\. Balunovic, M\. Vechev, M\. Marinov, M\. Drencheva, L\. Konova, M\. Shumanov, K\. Tsvetkov, N\. Drenchev, L\. Todorov, K\. Nikolova, N\. Georgiev, V\. Kalinkova, and M\. Ismoldayev \(2026\)The open proof corpus: a large\-scale study of llm\-generated mathematical proofs\.External Links:2506\.21621,[Link](https://arxiv.org/abs/2506.21621)Cited by:[§1](https://arxiv.org/html/2608.05660#S1.p1.1)\.
- S\. Dutta, J\. Singh, S\. Chakrabarti, and T\. Chakraborty \(2024\)How to think step\-by\-step: a mechanistic understanding of chain\-of\-thought reasoning\.External Links:2402\.18312Cited by:[§4\.2](https://arxiv.org/html/2608.05660#S4.SS2.p1.8)\.
- N\. Elhage, T\. Hume, C\. Olsson, N\. Schiefer, T\. Henighan, S\. Kravec, Z\. Hatfield\-Dodds, R\. Lasenby,et al\.\(2022\)Toy models of superposition\.arXiv preprint arXiv:2209\.10652\.Cited by:[§2\.1](https://arxiv.org/html/2608.05660#S2.SS1.p1.1)\.
- N\. Elhage, R\. Lasenby, and C\. Olah \(2023\)Privileged bases in the transformer residual stream\.Note:Transformer Circuits ThreadExternal Links:[Link](https://transformer-circuits.pub/2023/privileged-basis/index.html)Cited by:[§4\.1](https://arxiv.org/html/2608.05660#S4.SS1.p1.2)\.
- J\. Fernando and G\. Guitchounts \(2025\)Transformer dynamics: a neuroscientific approach to interpretability of large language models\.External Links:2502\.12131,[Link](https://arxiv.org/abs/2502.12131)Cited by:[§2\.2](https://arxiv.org/html/2608.05660#S2.SS2.p1.1)\.
- B\. Geshkovski, C\. Letrouit, Y\. Polyanskiy, and P\. Rigollet \(2023\)The emergence of clusters in self\-attention dynamics\.arXiv preprint arXiv:2305\.05465\.Cited by:[§2\.2](https://arxiv.org/html/2608.05660#S2.SS2.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur,et al\.\(2024\)The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§5](https://arxiv.org/html/2608.05660#S5.SS0.SSS0.Px1.p1.3)\.
- W\. Gurnee and M\. Tegmark \(2024\)Language models represent space and time\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 2483–2503\.Cited by:[§1](https://arxiv.org/html/2608.05660#S1.p3.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021\)Measuring massive multitask language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[Appendix B](https://arxiv.org/html/2608.05660#A2.p3.1),[§5](https://arxiv.org/html/2608.05660#S5.SS0.SSS0.Px2.p1.1)\.
- R\. Huben, H\. Cunningham, L\. R\. Smith, A\. Ewart, and L\. Sharkey \(2024\)Sparse autoencoders find highly interpretable features in language models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=F76bwRSLeK)Cited by:[§2\.1](https://arxiv.org/html/2608.05660#S2.SS1.p1.1)\.
- A\. Lancucki, J\. Chorowski, G\. Sanchez, R\. Marxer, N\. Chen, H\. J\. G\. A\. Dolfing, S\. Khurana, T\. Alumäe, and A\. Laurent \(2020\)Robust training of vector quantized bottleneck models\.In2020 International Joint Conference on Neural Networks \(IJCNN\),pp\. 1–7\.External Links:[Document](https://dx.doi.org/10.1109/IJCNN48605.2020.9207145)Cited by:[§4\.3](https://arxiv.org/html/2608.05660#S4.SS3.p2.6)\.
- J\. Lindsey, W\. Gurnee, E\. Ameisen, B\. Chen, A\. Pearce, N\. L\. Turner, C\. Citro, D\. Abrahams, S\. Carter, B\. Hosmer, J\. Marcus, M\. Sklar, A\. Templeton, T\. Bricken, C\. McDougall, H\. Cunningham, T\. Henighan, A\. Jermyn, A\. Jones, A\. Persic, Z\. Qi, T\. B\. Thompson, S\. Zimmerman, K\. Rivoire, T\. Conerly, C\. Olah, and J\. Batson \(2025\)On the biology of a large language model\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2025/attribution-graphs/biology.html)Cited by:[§2\.1](https://arxiv.org/html/2608.05660#S2.SS1.p1.1)\.
- Y\. Liu, D\. Gong, Y\. Cai, E\. Gao, Z\. Zhang, B\. Huang, M\. Gong, A\. van den Hengel, and J\. Q\. Shi \(2025\)I predict therefore i am: is next token prediction enough to learn human\-interpretable concepts from data?\.External Links:2503\.08980,[Link](https://arxiv.org/abs/2503.08980)Cited by:[§1](https://arxiv.org/html/2608.05660#S1.p2.1)\.
- Y\. Lu, Z\. Li, D\. He, Z\. Sun, B\. Dong, T\. Qin, L\. Wang, and T\. Liu \(2019\)Understanding and improving transformer from a multi\-particle dynamic system perspective\.arXiv preprint arXiv:1906\.02762\.Cited by:[§2\.2](https://arxiv.org/html/2608.05660#S2.SS2.p1.1)\.
- S\. Marks and M\. Tegmark \(2024\)The geometry of truth: emergent linear structure in large language model representations of true/false datasets\.External Links:2310\.06824,[Link](https://arxiv.org/abs/2310.06824)Cited by:[§2\.1](https://arxiv.org/html/2608.05660#S2.SS1.p1.1)\.
- K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov \(2022\)Locating and editing factual associations in GPT\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2202\.05262Cited by:[§4\.2](https://arxiv.org/html/2608.05660#S4.SS2.p1.8)\.
- T\. Mihaylov, P\. Clark, T\. Khot, and A\. Sabharwal \(2018\)Can a suit of armor conduct electricity? a new dataset for open book question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp\. 2381–2391\.Cited by:[Appendix B](https://arxiv.org/html/2608.05660#A2.p2.1),[§5](https://arxiv.org/html/2608.05660#S5.SS0.SSS0.Px2.p1.1)\.
- N\. Mostafazadeh, N\. Chambers, X\. He, D\. Parikh, D\. Batra, L\. Vanderwende, P\. Kohli, and J\. Allen \(2016\)A corpus and cloze evaluation for deeper understanding of commonsense stories\.InProceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 839–849\.Cited by:[Appendix B](https://arxiv.org/html/2608.05660#A2.p3.1),[§5](https://arxiv.org/html/2608.05660#S5.SS0.SSS0.Px2.p1.1)\.
- D\. Muhlgay, O\. Ram, I\. Magar, Y\. Levine, N\. Ratner, Y\. Belinkov, O\. Abend, K\. Leyton\-Brown, A\. Shashua, and Y\. Shoham \(2024\)Generating benchmarks for factuality evaluation of language models\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),Y\. Graham and M\. Purver \(Eds\.\),St\. Julian’s, Malta,pp\. 49–66\.External Links:[Link](https://aclanthology.org/2024.eacl-long.4/),[Document](https://dx.doi.org/10.18653/v1/2024.eacl-long.4)Cited by:[Appendix B](https://arxiv.org/html/2608.05660#A2.p4.1),[§5](https://arxiv.org/html/2608.05660#S5.SS0.SSS0.Px2.p1.1),[§6\.2](https://arxiv.org/html/2608.05660#S6.SS2.p1.3)\.
- A\. Nepal, S\. Shrestha, A\. Shrestha, M\. Kim, J\. Naghiyev, R\. Shwartz\-Ziv, and K\. Ross \(2025\)Layer importance for mathematical reasoning is forged in pre\-training and invariant after post\-training\.InNeurIPS 2025 Workshop on Mathematical Reasoning and AI \(MATH\-AI\),External Links:2506\.22638Cited by:[§4\.2](https://arxiv.org/html/2608.05660#S4.SS2.p1.8)\.
- G\. Nikolaou, T\. Mencattini, D\. Crisostomi, A\. Santilli, Y\. Panagakis, and E\. Rodolà \(2025\)Language models are injective and hence invertible\.arXiv preprint arXiv:2510\.15511\.Cited by:[§1](https://arxiv.org/html/2608.05660#S1.p3.1)\.
- K\. Park, Y\. J\. Choe, and V\. Veitch \(2023\)The linear representation hypothesis and the geometry of large language models\.arXiv preprint arXiv:2311\.03658\.Cited by:[§2\.1](https://arxiv.org/html/2608.05660#S2.SS1.p1.1)\.
- J\. Postmus and S\. Abreu \(2024\)Steering large language models using conceptors: improving addition\-based activation engineering\.InMINT: Foundation Model Interventions,External Links:[Link](https://openreview.net/forum?id=gyAnAq16HC)Cited by:[§1](https://arxiv.org/html/2608.05660#S1.p2.1)\.
- N\. Rimsky, N\. Gabrieli, J\. Schulz, M\. Tong, E\. Hubinger, and A\. Turner \(2024\)Steering llama 2 via contrastive activation addition\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15504–15522\.External Links:[Link](https://aclanthology.org/2024.acl-long.828/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.828)Cited by:[§1](https://arxiv.org/html/2608.05660#S1.p2.1)\.
- M\. Sap, H\. Rashkin, D\. Chen, R\. Le Bras, and Y\. Choi \(2019\)Social iqa: commonsense reasoning about social interactions\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),pp\. 4463–4473\.Cited by:[Appendix B](https://arxiv.org/html/2608.05660#A2.p2.1),[§5](https://arxiv.org/html/2608.05660#S5.SS0.SSS0.Px2.p1.1)\.
- T\. Schuster, A\. Fisch, and R\. Barzilay \(2021\)Get your vitamin C\! robust fact verification with contrastive evidence\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,K\. Toutanova, A\. Rumshisky, L\. Zettlemoyer, D\. Hakkani\-Tur, I\. Beltagy, S\. Bethard, R\. Cotterell, T\. Chakraborty, and Y\. Zhou \(Eds\.\),Online,pp\. 624–643\.External Links:[Link](https://aclanthology.org/2021.naacl-main.52/),[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.52)Cited by:[Appendix B](https://arxiv.org/html/2608.05660#A2.p4.1),[§5](https://arxiv.org/html/2608.05660#S5.SS0.SSS0.Px2.p1.1),[§6\.2](https://arxiv.org/html/2608.05660#S6.SS2.p1.3)\.
- F\. Sun, N\. Li, K\. Wang, and L\. Goette \(2025\)Large language models are overconfident and amplify human bias\.External Links:2505\.02151,[Link](https://arxiv.org/abs/2505.02151)Cited by:[§1](https://arxiv.org/html/2608.05660#S1.p1.1)\.
- A\. Talmor, J\. Herzig, N\. Lourie, and J\. Berant \(2019\)CommonsenseQA: a question answering challenge targeting commonsense knowledge\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),pp\. 4149–4158\.Cited by:[Appendix B](https://arxiv.org/html/2608.05660#A2.p2.1),[§5](https://arxiv.org/html/2608.05660#S5.SS0.SSS0.Px2.p1.1)\.
- A\. M\. Turner, L\. Thiergart, G\. Leech, D\. Udell, J\. J\. Vazquez, U\. Mini, and M\. MacDiarmid \(2023\)Steering language models with activation engineering\.arXiv preprint arXiv:2308\.10248\.Cited by:[§1](https://arxiv.org/html/2608.05660#S1.p3.1)\.
- A\. van den Oord, O\. Vinyals, and k\. kavukcuoglu \(2017\)Neural discrete representation learning\.InAdvances in Neural Information Processing Systems,I\. Guyon, U\. V\. Luxburg, S\. Bengio, H\. Wallach, R\. Fergus, S\. Vishwanathan, and R\. Garnett \(Eds\.\),Vol\.30,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/file/7a98af17e63a0ac09ce2e96d03992fbc-Paper.pdf)Cited by:[§4\.3](https://arxiv.org/html/2608.05660#S4.SS3.p2.6)\.
- Z\. Xie, J\. Guo, T\. Yu, and S\. Li \(2024\)Calibrating reasoning in language models with internal consistency\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2405\.18711Cited by:[§4\.2](https://arxiv.org/html/2608.05660#S4.SS2.p1.8)\.
- H\. Xu, Y\. Song, Q\. Liu, J\. van Genabith, and D\. Xiong \(2024\)Rewiring the transformer with depth\-wise LSTMs\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),N\. Calzolari, M\. Kan, V\. Hoste, A\. Lenci, S\. Sakti, and N\. Xue \(Eds\.\),Torino, Italia,pp\. 14122–14133\.External Links:[Link](https://aclanthology.org/2024.lrec-main.1231/)Cited by:[§2\.2](https://arxiv.org/html/2608.05660#S2.SS2.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao,et al\.\(2025a\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§5](https://arxiv.org/html/2608.05660#S5.SS0.SSS0.Px1.p1.3)\.
- A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu,et al\.\(2025b\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§5](https://arxiv.org/html/2608.05660#S5.SS0.SSS0.Px1.p1.3)\.
- Y\. Yao, N\. Zhang, Z\. Xi, M\. Wang, Z\. Xu, S\. Deng, and H\. Chen \(2024\)Knowledge circuits in pretrained transformers\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2405\.17969Cited by:[§4\.2](https://arxiv.org/html/2608.05660#S4.SS2.p1.8)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 4791–4800\.Cited by:[Appendix B](https://arxiv.org/html/2608.05660#A2.p2.1),[§5](https://arxiv.org/html/2608.05660#S5.SS0.SSS0.Px2.p1.1)\.
- Y\. Zhou, Y\. Wang, X\. Yin, S\. Zhou, and A\. R\. Zhang \(2025\)The geometry of reasoning: flowing logics in representation space\.arXiv preprint arXiv:2510\.09782\.Cited by:[§2\.3](https://arxiv.org/html/2608.05660#S2.SS3.p1.1)\.

Supplementary Material for Reasoning Errors Have a Region and a Direction in the Residual\-Stream Trajectory of LLMs

## Appendix AModels and Feature Extraction

Table[5](https://arxiv.org/html/2608.05660#A1.T5)lists the three models and the depths each detector reads\. All three are base models rather than instruction\-tuned ones, and every weight stays frozen\. The six depths sit at the same fractions of network depth in every model, spanning the middle\-to\-late band Sec\.[4](https://arxiv.org/html/2608.05660#S4)motivates, so a3232\-layer and a4848\-layer model are read at matching relative positions\.

Table 5:The three models\. Llama\-3\.1\-8B and Qwen2\.5\-14B are dense; Qwen3\-30B\-A3B is a mixture of experts with3030B parameters of which33B are active per token\. Width is the residual\-stream dimension\. The depths are the layers at which the direction and region streams read the state\.For each question and each candidate answer we run one forward pass over the question followed by that candidate, and record the residual\-stream state at the output of every layer for every token, which gives one array of shape\[ntokens×nlayers,d\]\[\\,n\_\{\\mathrm\{tokens\}\}\\times n\_\{\\mathrm\{layers\}\},\\,d\\,\]per candidate, stored in half precision\. Nothing is written back into the model, so the same activations serve every method we compare\.

The three streams read different parts of that array\. The motion stream reads the displacements over the whole token\-and\-layer grid, all tokens and all layers\. The direction and region streams read only the answer’s tokens, at the depths in Table[5](https://arxiv.org/html/2608.05660#A1.T5); where an answer runs longer than6464tokens,𝒜\\mathcal\{A\}is its last6464\. Llama\-3\.1\-8B is scored on the full test set of every target\. On the two Qwen models each target is scored on up to2,0002\{,\}000items, and the same items are used for every method\.

## Appendix BDatasets

Our evaluation is built around generalization\. Each detector trains on a single benchmark and is then scored on eight reasoning benchmarks and four factual ones, on the same splits for every method\. We call that one benchmark the detector’s donor, since it alone supplies training data and every other benchmark in the suite is held out from it\.

Six of the reasoning benchmarks serve as donors in turn, and they ask for different forms of reasoning\. ARC\-Challenge and ARC\-Easy\(Clarket al\.[2018](https://arxiv.org/html/2608.05660#bib.bib58)\)are grade\-school science questions, the challenge partition holding the ones that retrieval baselines fail\. OpenBookQA\(Mihaylovet al\.[2018](https://arxiv.org/html/2608.05660#bib.bib61)\)pairs a science fact with the everyday knowledge needed to apply it\. CommonsenseQA\(Talmoret al\.[2019](https://arxiv.org/html/2608.05660#bib.bib63)\)asks about ordinary objects and situations, Social IQa\(Sapet al\.[2019](https://arxiv.org/html/2608.05660#bib.bib65)\)about people’s motives and reactions, and HellaSwag\(Zellerset al\.[2019](https://arxiv.org/html/2608.05660#bib.bib60)\)about which continuation of a described event is plausible\. Transfer between these is transfer between kinds of reasoning\. We cap each training set at10,00010\{,\}000items so that no benchmark supplies an order of magnitude more data than another, which lets a difference between two donors be read as a property of the benchmark rather than of its size\.

The other two reasoning benchmarks are conventionally used for evaluation rather than training, and we use them the same way\. The Story Cloze test\(Mostafazadehet al\.[2016](https://arxiv.org/html/2608.05660#bib.bib62)\)asks which of two endings completes a short everyday story, and MMLU\(Hendryckset al\.[2021](https://arxiv.org/html/2608.05660#bib.bib75)\)spans5757academic and professional subjects\. Neither resembles what a donor supplies\. They extend the evaluation in two directions the training benchmarks do not reach, narrative coherence and breadth of subject matter\.

If what the location streams read is a signal about correctness rather than about reasoning tasks, it should reach errors of another kind, so the suite ends with two factual paradigms that no detector trains on\. In FACTOR\(Muhlgayet al\.[2024](https://arxiv.org/html/2608.05660#bib.bib76)\)every wrong option is a minimal edit of a correct completion, changing one fact and leaving the rest standing, across three domains, wiki, news and expert\. VitaminC\(Schusteret al\.[2021](https://arxiv.org/html/2608.05660#bib.bib77)\)instead pairs a claim with a piece of evidence and asks whether the evidence bears the claim out\. The two put a factual error in front of the detector in different forms\. Two benchmarks common in this area, TriviaQA and HaluEval, are left out of the suite, since both can be scored well above chance without consulting the model \(Appendix[D](https://arxiv.org/html/2608.05660#A4)\)\.

## Appendix CTask, Metric, Baselines and Training

#### Task and metric\.

Each item in every dataset gives one correct answer and one or more incorrect ones\. We form a positive instance from each, the question with its correct answer, and one negative per incorrect answer\. Our metric is selection accuracy, the fraction of items whose correct answer outscores every incorrect answer of that item, with chance at1/\(N\+1\)1/\(N\{\+\}1\)for an item withNNincorrect answers\.

#### Baselines\.

We compare against two baselines on the same splits and metric\. The first is a linear probe on a single layer’s activations, the standard static reader, with the layer selected by the donor’s own held\-out validation selection accuracy, which lands between layers1313and1616for every donor \(Appendix[E](https://arxiv.org/html/2608.05660#A5)\)\. The second is the motion stream alone, the displacement LSTM of prior work\(Damirchiet al\.[2026](https://arxiv.org/html/2608.05660#bib.bib68)\), which is our model with the two location streams removed\. The gap between that baseline and the full model is what restoring the location adds\.

#### Implementation\.

The motion stream is a two\-layer LSTM with128128hidden units\. The direction stream projects each layer’s state to88dimensions and the region stream to6464, and the region codebook holdsK=128K\{=\}128entries\. All parts train jointly from scratch with Adam at learning rate10−310^\{\-3\}under a cosine schedule, batch size1616with gradient accumulation of44\. A tenth of each donor’s training data is held out for validation\. We train for seven epochs and keep the checkpoint with the best validation accuracy, and the same recipe is used on all three models\. Each seed sets the weight initialization and the data\-order shuffle and leaves the data itself unchanged\. The main table’s full\-model rows are the mean over three training seeds \(Table[8](https://arxiv.org/html/2608.05660#A8.T8)\), and each other analysis states its seed and donor status in its caption\.

#### Compute and software\.

Extraction and training run on NVIDIA RTX A6000 GPUs with4848GB of memory, under Ubuntu, in PyTorch2\.72\.7with Transformers4\.524\.52and h5py3\.133\.13\. The activations of each dataset are written once to HDF5 in half precision and read back by every method, so no base model is run twice\.

## Appendix DSurface Confounds and Excluded Benchmarks

Two benchmarks common in this area, TriviaQA and HaluEval, are absent from our suite\. A benchmark whose wrong answers are systematically longer than its right ones can be scored without consulting the model, and Table[6](https://arxiv.org/html/2608.05660#A4.T6)reports a heuristic that selects the shortest candidate\. On TriviaQA the correct answer is shorter by3\.33\.3tokens on average and on HaluEval by11\.611\.6, so the rule reaches88\.888\.8and95\.995\.9\. A score on either would say more about how the benchmark was built than about what a detector reads, so we set both aside\. Every benchmark we report sits below chance under the same rule\. FACTOR and VitaminC are length\-clean by construction, since a wrong option is a minimal edit of the right one\.

Length is not the only route\. HaluEval’s correct answers are human\-written and its incorrect ones model\-generated, so its two classes differ in authorship as well as in correctness\. A bag\-of\-words classifier over the answer text alone, with no access to the model, separates them at an AUC of0\.9660\.966\. Either measurement on its own would be enough to set the benchmark aside\.

Table 6:Selection accuracy×100\\times 100of a heuristic that selects the shortest candidate, with no access to the model\. Chance is1/\(N\+1\)1/\(N\{\+\}1\)averaged over items, so it tracks each item’s candidate count\. Every reported target sits below chance; the two excluded benchmarks are solved by length alone\.
## Appendix EThe Linear Probe Baseline

The probe reads one layer, so the layer must be chosen fairly\. We select it by the donor’s own held\-out validation selection accuracy, sweeping all3232layers, which lands in a stable mid\-network band for every donor \(Table[7](https://arxiv.org/html/2608.05660#A5.T7)\)\. Selecting the layer on a single factual set instead is noisy and lands earlier in the network, at layers99to1313, because donor\-to\-factual selection accuracy is low \(2828to4646\) and its argmax is weakly identified\. An earlier criterion that selected the layer on TriviaQA and HaluEval is retired with those benchmarks\. On the two Qwen models the same criterion sweeps all4848layers of each\.

Table 7:The probe’s validation\-selected layer per donor and its cross\-target reasoning mean, selection accuracy×100\\times 100\. The choice is deterministic given the donor split\.
## Appendix FThe Model’s Own Likelihood

The likelihood baseline reads no internal state\. For each question and candidate we run one forward pass over the frozen base model and score the candidate by the log\-probability it assigns to the answer’s tokens alone, taking the highest scoring candidate as the pick\. Nothing is trained, and the prompts and the items are the ones every other row is scored on\.

On reasoning the model’s own likelihood is the weakest reader on average in Table[1](https://arxiv.org/html/2608.05660#S6.T1)\. It averages55\.155\.1over the eight targets, where the probe averages59\.559\.5across donors, the displacement reader64\.764\.7, and the full model73\.173\.1, which puts it1818points below the full model\.What the detector reads here is therefore a signal from the representations that shows how the base model can distinguish the validity of candidates despite token space not allowing for this signal to come through\.

The factual sets go the other way\. The likelihood reads all three FACTOR sets above the full model, by6\.06\.0,14\.114\.1and5\.15\.1points \(Table[2](https://arxiv.org/html/2608.05660#S6.T2)\), and it does so under every scoring variant we ran\. FACTOR’s wrong options are single\-fact edits of an otherwise fluent completion, so whether a completion is factual and whether it is likely nearly coincide, and a likelihood scorer is reading the property the benchmark is built from\. The one reasoning target it reads well is HellaSwag, at79\.079\.0, which is a completion benchmark of the same shape\. VitaminC asks something else, whether a claim holds given a supplied passage, and there the full model leads,67\.167\.1against65\.165\.1\. That margin is2\.02\.0points on one set, and it holds on two of the three donors\.Reasoning correctness is not as legible in the model’s token space \(what the model says\) as it is in how it computes, while the factual correctness of a minimal edit largely is\.Note that our readers are trained on reasoning, and the assertion here is based on generalization to factual benchmarks from reasoning datasets\.

Three variants of that score come off the same forward pass, the total log\-probability of the answer, its per\-token mean, and a per\-character normalization\. We report the per\-token mean\. A total inherits the answer\-length shortcut this suite is built to exclude, the same shortcut that sets TriviaQA and HaluEval aside in Appendix[D](https://arxiv.org/html/2608.05660#A4), so headlining it would hold the baseline to a looser standard than the paper holds its own targets\. The ordering reported on the factual sets holds under all three\. This baseline is measured on Llama\-3\.1\-8B, so the cross\-model tables carry no such row\.

## Appendix GPer\-Cell Transfer Gain

Figure[5](https://arxiv.org/html/2608.05660#A7.F5)reads Table[1](https://arxiv.org/html/2608.05660#S6.T1)as a map of the gain from restoring the location, the full model minus the motion\-only reader in every cell of the transfer grid\. The gain is positive on all but one of the4848cells and largest on transfer into HellaSwag\.

![Refer to caption](https://arxiv.org/html/2608.05660v1/x5.png)Figure 5:Per\-cell gain from restoring the location, Ours minus the motion\-only reader on the transfer grid of Table[1](https://arxiv.org/html/2608.05660#S6.T1)\(selection accuracy×100\\times 100\)\. Blue is a gain for the full model, red a loss\. Outlined cells are in\-distribution \(donor equals target\)\. The gain is broad and largest on transfer into HellaSwag, with a single negative cell, MMLU under the HellaSwag donor\.
## Appendix HResult Stability Across Seeds

Across three training seeds \(weight initialization and data\-order shuffle, identical data\), the full model’s results move little\. Table[8](https://arxiv.org/html/2608.05660#A8.T8)reports the cross\-target reasoning mean and its spread for each training donor\. The spread is a tenth of a point on the steadiest donors and about a point at its widest, on HellaSwag and ARC\-Easy, far below the roughly eight\-point margin the full model holds over the motion\-only baseline and the larger margin over the probe\. The ordering over the baselines is therefore not seed\-fragile\.

Table 8:Seed stability of the full model on the reasoning axis, selection accuracy×100\\times 100\. For each training donor, the cross\-target reasoning mean across three training seeds \(weight initialization and data\-order shuffle, identical data\), as mean±\\pmstandard deviation with the observed range\. The probe is deterministic given the split\.
## Appendix IFactual Transfer per Donor

Table[9](https://arxiv.org/html/2608.05660#A9.T9)breaks the factual means into donor cells\. The full model beats the motion\-only model in all twelve cells, by5\.75\.7to30\.130\.1points, and beats the probe in eleven of twelve, the exception FACTOR\-expert under the ARC\-Easy donor, where the probe reads0\.40\.4higher on a236236\-item set\. The reach into factual errors requires a donor that itself transfers well on reasoning, and a donor that transfers weakly on reasoning reaches these sets weakly too\.

Table 9:Per\-donor factual transfer, selection accuracy×100\\times 100, single seed, full test sets\. The FACTOR\-expert rows rest on236236items, so their per\-donor cells are noisy; read the means in the main table\.An in\-domain linear read of each factual set, sweeping the six extracted layers, peaks at layer1212on FACTOR\-wiki \(52\.652\.6\), layer1616on FACTOR\-news \(61\.561\.5\) and FACTOR\-expert \(97\.997\.9\), and layer1212on VitaminC \(89\.089\.0\)\. The FACTOR\-expert ceiling rests on a held\-out split of about4747items and is optimistic\. The sweep covers only the six extracted depths, so it locates which of those carries the most in\-domain signal rather than showing the band itself to be the best available \(Figure[6](https://arxiv.org/html/2608.05660#A9.F6)\)\.

![Refer to caption](https://arxiv.org/html/2608.05660v1/x6.png)Figure 6:In\-domain per\-layer selection accuracy of a linear read on each factual set, sweeping the six extracted depths\. Every set peaks in the mid\-network band, layers1212to1616\. The FACTOR\-expert curve rests on about4747held\-out items and is optimistic\. Chance is2525\.
## Appendix JThe Late\-Fusion Decomposition

The detector of Sec\.[4](https://arxiv.org/html/2608.05660#S4)joins the three readings by concatenating their embeddings and passing them to a single head, which returns one score and no per\-stream quantity\. Reading a stream on its own therefore needs a model built to expose one\. We train a variant that leaves every stream unchanged and replaces the joint head with one linear head per stream\. Writingcsc\_\{s\}for the embedding of streams∈\{mot,dir,reg\}s\\in\\\{\\mathrm\{mot\},\\mathrm\{dir\},\\mathrm\{reg\}\\\}, each head returns a scalar,

os=ws⊤​cs\+bs\.o\_\{s\}=w\_\{s\}^\{\\top\}c\_\{s\}\+b\_\{s\}\.\(7\)Each logit is standardized on its own,

o~s=os−μsvs\+ϵ,\\tilde\{o\}\_\{s\}=\\frac\{o\_\{s\}\-\\mu\_\{s\}\}\{\\sqrt\{v\_\{s\}\+\\epsilon\}\},\(8\)whereμs\\mu\_\{s\}andvsv\_\{s\}are the mean and variance of that stream’s logit, taken over the batch during training and from running estimates at evaluation\. Nothing is learned in this step, neither a scale nor a shift\. The three standardized logits are then combined by a learned weight,

y^=σ​\(∑sαs​o~s\),α=softmax​\(a\),a∈ℝ3,\\hat\{y\}=\\sigma\\Big\(\\textstyle\\sum\_\{s\}\\alpha\_\{s\}\\tilde\{o\}\_\{s\}\\Big\),\\qquad\\alpha=\\mathrm\{softmax\}\(a\),\\quad a\\in\\mathbb\{R\}^\{3\},\(9\)withaainitialized so that the three weights start equal\. All parts train jointly from scratch on the pairwise objective, under the recipe of Appendix[C](https://arxiv.org/html/2608.05660#A3)\.

The standardization is what makes the per\-stream reading worth taking\. Left out, one head can settle at a near\-constant logit while the others grow, and that stream then scores near the floor for reasons of scale rather than of signal\. What it does to a stream is a single affine map with a positive scale, so it cannot reorder the candidates of an item\. It changes what the heads learn, not how a stream ranks the answers in front of it\.

Three quantities are read from the trained variant\. A stream reads an item correctly when its own logit on the correct candidate exceeds its logit on every incorrect candidate of that item, which gives the per\-stream columns of Table[10](https://arxiv.org/html/2608.05660#A10.T10)and the correctness sets behind Figures[8](https://arxiv.org/html/2608.05660#A10.F8)and[9](https://arxiv.org/html/2608.05660#A10.F9)\. The fused column is the model’s own combined score\. The oracle counts an item correct when at least one of the three streams reads it correctly, which bounds what a rule choosing between the streams item by item could reach; it is not attainable without already knowing the answer\.

The variant is a measurement device and not a second proposal\. On the ARC\-Challenge donor’s own test set it reaches75\.175\.1against74\.774\.7for the concatenation model of Sec\.[4](https://arxiv.org/html/2608.05660#S4)at the same seed, so the streams are separated inside a detector of the same strength\. Its solo numbers describe what each stream can support on its own, not a component of the concatenation model’s score\.

#### Are the streams redundant or complementary?

The results in Table[10](https://arxiv.org/html/2608.05660#A10.T10)show the direction stream to be the strongest single reader on the two reasoning sets and on FACTOR\-expert, while on VitaminC motion reads better than either location stream\. We also report an oracle over the three, which counts an item correct when any one of them selects the right candidate and so bounds what choosing between the streams item by item could reach\. It clears the best single stream on all four sets, so signal sits in the three that no one of them recovers alone\. Figure[7](https://arxiv.org/html/2608.05660#A10.F7)shows where that signal is, with every stream, motion included, uniquely recovering items the other two miss, and motion alone rescuing281281items on VitaminC that neither location stream reads correctly\. Concatenation already turns part of that into accuracy, passing the best single stream on three of the four sets\. On FACTOR\-expert it does not, reading1212points below the direction stream alone\.Each reading is right about different items, so the three together hold more about correctness than any reader here recovers from them, and the33to2121points between the trained fusion and the oracle measure how much\.Closing that gap asks for more than another rule for combining the three, since gating and per\-step injection score within0\.0050\.005AUC of concatenation \(Appendix[K](https://arxiv.org/html/2608.05660#A11)\)\. The useful quantity is which of them to trust on a given item\.

![Refer to caption](https://arxiv.org/html/2608.05660v1/x7.png)Figure 7:Division of labor, the count of items each stream alone reads correctly when the other two miss it, under the late\-fusion decomposition of one trained detector\. One donor \(ARC\-Challenge\), single seed, up to2,0002\{,\}000items per dataset\.Table 10:Division of labor, selection accuracy\. Each stream read on its own, the trained fusion of the three, and a per\-item oracle over the three, under the late\-fusion decomposition of this appendix\. Bold marks the best single stream\. One donor \(ARC\-Challenge\), single seed, up to2,0002\{,\}000items per dataset\.
#### Where do the streams’ correct answers overlap?

Each stream recovers items the other two miss\. How the three group on the rest is a separate question, and Figure[8](https://arxiv.org/html/2608.05660#A10.F8)answers it by counting the items every combination of streams reads correctly\. On every dataset the largest group is the one all three read, so the streams are not reading disjoint properties but one property with different reach\. Beyond that core, the two reasoning sets and FACTOR\-expert are alike\. What the region reading recovers there sits almost entirely inside what direction recovers, since it seldom reads an item alone and seldom reads one alongside motion that direction misses, so the coarse reading adds little the fine one does not already hold\. VitaminC does not behave that way\. Region reads items on its own several times more often than on any of the other three sets, and the items it reads together with motion while direction misses them are the largest pair there\. Figure[9](https://arxiv.org/html/2608.05660#A10.F9)shows the same sets item by item, where the wide band of agreement on the reasoning benchmarks gives way to a divided one\.The coarse reading is not a blurred copy of the fine one, and which readings agree is set by the kind of question asked rather than by a fixed order among the three\.The component ablation of Table[3](https://arxiv.org/html/2608.05660#S6.T3)is scored on reasoning alone, which is where the two location readings overlap most, so it reads the region stream where it has least of its own to add\.

![Refer to caption](https://arxiv.org/html/2608.05660v1/x8.png)Figure 8:Overlap of the items each stream reads correctly, under the late\-fusion decomposition of one trained detector\. Each bar counts the items that a given combination of streams all read right, so the single\-stream bars reproduce Figure[7](https://arxiv.org/html/2608.05660#A10.F7)\. The two largest groups of each panel are shaded dark\. M motion, D direction, R region\. One donor \(ARC\-Challenge\), single seed, up to2,0002\{,\}000items per dataset\.![Refer to caption](https://arxiv.org/html/2608.05660v1/x9.png)Figure 9:The same per\-stream correctness sets, item by item, colored by stream and sorted so shared correct\-and\-incorrect patterns run together\. Each column is one item, and a stream’s row is filled where that stream reads the item correctly on its own\. One donor \(ARC\-Challenge\), single seed, up to2,0002\{,\}000items per dataset\.

## Appendix KDesign Checks and Controls

#### Fusion\.

On an earlier two\-stream form of the model, concatenation, per\-item gating and per\-step injection land within0\.0050\.005held\-out AUC of one another on the reasoning targets we retain, once checkpoints are selected by validation AUC \(ARC\-Challenge donor0\.7320\.732,0\.7360\.736,0\.7340\.734\), while a unified attention head trails at0\.6750\.675\. We keep concatenation\. It puts nothing learned between the readings and the score, so what a reading contributes can be measured without also accounting for a gate whose own behaviour varies from item to item\.

#### Region codebook health\.

A naive codebook collapses onto a handful of entries, and the region stream then stops contributing\. Seeding the entries from the first batch of data, updating them by an exponential moving average, reseeding unused entries, and a commitment term holding the encoder to its choice hold the codebook’s perplexity near5050of the128128entries, where a fully collapsed codebook would read11, and turn the region stream from a small loss into a gain on the factual axis\. The region stream is therefore worth having only if its codebook stays healthy\.

#### Direction only\.

The residual\-stream norm grows with depth, so raw states from different layers do not share a scale, and the location streams therefore read unit\-normalized directions\. Class differences in the final\-layer norm exist but are dataset\-dependent, and reading the unnormalized state in place of the direction scored lower in our development runs \(see Limitations\)\.

#### Is the gain just added capacity?

The two location streams add0\.310\.31M parameters to the motion reader,6\.7%6\.7\\%of its LSTM \(Appendix[L](https://arxiv.org/html/2608.05660#A12)\)\. Small as that is, the gain might come from those parameters rather than from location\. We train both readers on the ARC\-Challenge donor at hidden widths128128,256256and512512\(Table[11](https://arxiv.org/html/2608.05660#A11.T11)\)\. The full model leads at every width, by99to1111points on the reasoning mean and1313to1515on the factual one\. Width does little for either\. The full model moves by0\.30\.3points over a fourfold increase, while the motion\-only reader gains1\.71\.7points at256256and gives most of them back at512512\. At that width its25\.725\.7M parameters read ten points below the full model’s5\.15\.1M\.The gap is the design, not the size\.

Table 11:Scaling both readers to hidden widths128128,256256, and512512on the ARC\-Challenge donor, selection accuracy\. Reasoning is the mean over four reasoning sets \(ARC\-Challenge, ARC\-Easy, OpenBookQA, CommonsenseQA\), Factual over FACTOR and VitaminC\. Width scales each reader’s recurrent state, and the full model’s stream widths with it; the codebook holds128128entries at every width\. Single seed, up to2,0002\{,\}000items per target\.
#### Is the gain just normalization?

The location streams read direction\-only states, so the gain might come from dropping magnitude rather than from restoring location\. We take the magnitude away from the motion reader as well, retraining it on unit\-length displacements \(Table[12](https://arxiv.org/html/2608.05660#A11.T12)\)\. On the ARC\-Challenge donor that costs1111points on the reasoning mean and leaves ARC\-Easy where it was\. On the factual mean, the normalized reader loses1010points on the first donor and gains2\.52\.5on the second, and the full model is ahead of both motion readers on each\.The gain is location, not normalization\.

Table 12:Removing magnitude from the motion reader, selection accuracy\. Motion \(raw\) is the displacement\-only baseline; Motion \(normalized\) reads unit\-length displacements, like the location streams; Ours is the full model\. Reasoning is the mean over four reasoning sets, Factual over FACTOR and VitaminC\. Single seed, up to2,0002\{,\}000items per target\.
#### Is the gain just the wording?

The state the location streams read holds an answer’s wording, so a stream could pass that wording on in place of location\. We repeat both measurements on the streams the detector has learned \(Table[13](https://arxiv.org/html/2608.05660#A11.T13)\)\. The first asks how much of the variation in an answer’s wording a regression can predict from a stream’s embedding, where zero would mean the embedding says nothing about it\. The three reach0\.0090\.009at most\. The second asks whether embeddings that sit near one another belong to answers that share wording, where11would mean they are no more alike than randomly chosen answers\. The three reach1\.321\.32at most\. On the same two measurements the state a single\-layer probe reads reaches0\.0860\.086and2\.262\.26, and pooling that state over the whole answer raises it to0\.1910\.191and3\.133\.13\. Between the two location readings the order is the one the design predicts, the direction stream passing more than the region code\.What the streams restore is location, not the wording the raw state holds\.

Table 13:How much of the correct answer’s wording each representation gives back, on the ARC\-Challenge test set \(1,1721\{,\}172items, one correct answer each\)\.R2R^\{2\}is held\-out reconstruction of a latent semantic embedding of the answer text and Retrieval is the nearest\-neighbor text\-similarity ratio, so lower is safer in both columns\. Dim is the width of the representation read\. The raw\-state rows are the residual stream at layer1414, the depth the probe reads, which is the basis the motivation uses\. One donor \(ARC\-Challenge\), single seed\.

## Appendix LParameter Counts

The detector is small, and the location it restores is smaller still\. Table[14](https://arxiv.org/html/2608.05660#A12.T14)counts the trainable parameters of each stream on Llama\-3\.1\-8B\. The motion stream is the two\-layer bidirectional LSTM over the40964096\-dimensional displacements, at4\.724\.72M parameters, and the displacement\-only baseline of prior work is exactly this LSTM with a small classification head,4\.764\.76M in total\. The two location streams we add are far lighter\. The direction stream, one shared projection of each layer’s state to a compact coordinate followed by a small aggregator, is37\.537\.5K parameters\. The region stream, whose bulk is a single projection of each state to6464dimensions before the codebook, is277277K parameters, alongside a128128\-entry codebook maintained as a moving\-average buffer rather than by gradient\.Together the two location streams add0\.310\.31M parameters,6\.7%6\.7\\%of the motion LSTM, so restoring the location costs a small fraction of reading the motion\.The detector as a whole is small against the frozen model it reads\. Llama\-3\.1\-8B has about8\.08\.0B parameters, so the full detector, at5\.085\.08M, is0\.06%0\.06\\%of the model it runs on, the same overhead as the displacement\-only method it builds on\. The counts scale with the model’s residual\-stream width\.

Table 14:Trainable parameter counts on Llama\-3\.1\-8B; the four components sum to the full detector, which runs on top of the frozen base model\. The motion stream is the displacement bi\-LSTM of prior work, and the displacement\-only baseline is that LSTM with its own head \(4,755,7134\{,\}755\{,\}713parameters\)\. The two location streams together add314,176314\{,\}176parameters,6\.7%6\.7\\%of the motion LSTM, and the full detector is0\.06%0\.06\\%of the base model it reads\. The region stream’s128128\-entry codebook is a moving\-average buffer and is not counted here\.
## Appendix MRegion\-Code Usage

Figure[10](https://arxiv.org/html/2608.05660#A13.F10)shows which of the128128region codes each dataset uses\. The factual rows concentrate in a few bright bands that all lie inside the reasoning rows’ support, and no code is factual\-only, which is the usage pattern behind the shared\-vocabulary reading in the main text\.

Before quantization, a held\-out linear read of the raw state reaches AUC0\.920\.92on FACTOR\-expert and0\.610\.61on FACTOR\-wiki \(Figure[4](https://arxiv.org/html/2608.05660#S6.F4)\)\. Table[15](https://arxiv.org/html/2608.05660#A13.T15)shows that part of this signal survives as a discrete region code\.

Table 15:Region\-code signal: the number of used codes and accuracy from a code\-only majority vote\.![Refer to caption](https://arxiv.org/html/2608.05660v1/Figures/region_code_heatmap.png)Figure 10:Region\-code usage by dataset, one row per dataset over the128128codebook entries\. Factual sets concentrate on a subset of the codes reasoning sets use\. One donor’s codebook \(ARC\-Challenge\), up to800800items per dataset\.
## Appendix NCross\-Model Full Results

Both results in the body come from Llama\-3\.1\-8B\. We rerun the pipeline on two more models, a dense Qwen2\.5\-14B and a Qwen3\-30B\-A3B mixture of experts, training the same three readers on the same three donors, with the layer set read in each model given in Appendix[A](https://arxiv.org/html/2608.05660#A1)\. On reasoning the displacement reader leads the probe by2\.72\.7points on Llama\-3\.1\-8B, the two are level on Qwen2\.5\-14B, and on the mixture of experts the displacement reader trails by2\.52\.5\(Table[16](https://arxiv.org/html/2608.05660#A14.T16)\)\. The full model is the best of the three readers on reasoning and on factual transfer for all three models\. Its margin over whichever baseline is stronger for a given model is88to1414points on reasoning and1111to1616on the two factual sets\.The lead is not inherited from the displacement reader\.

Tables[17](https://arxiv.org/html/2608.05660#A14.T17)and[18](https://arxiv.org/html/2608.05660#A14.T18)give the per\-donor grids behind the cross\-model summary \(Table[16](https://arxiv.org/html/2608.05660#A14.T16)\)\. The full model is the best of the three readers in every donor cell of both grids, with one exception on the mixture of experts, where the motion\-only reader edges it on ARC\-Challenge under the ARC\-Easy donor\. Both models train on the three shared donors and read six evenly spaced layers, single seed, up to2,0002\{,\}000items per target\.

Table 16:Cross\-model summary, selection accuracy, cross\-donor means over the three shared donors \(ARC\-Challenge, ARC\-Easy, CommonsenseQA\)\. Reasoning is the mean over the eight reasoning targets, Factual over FACTOR\-wiki and VitaminC\. Single seed; the two Qwen models are scored on up to2,0002\{,\}000items per target\. Best per model in bold\.Table 17:Cross\-model reasoning transfer, selection accuracy×100\\times 100, per donor and evaluation target\. Each detector trains on the donor row and is evaluated on every target column\. Best per cell in bold; Ours is best in every cell except one \(Qwen3\-30B\-A3B, ARC\-Easy donor, ARC\-Challenge target\)\. Single seed, up to2,0002\{,\}000items per target\.Table 18:Cross\-model factual transfer, selection accuracy×100\\times 100, reasoning\-trained detectors evaluated zero\-shot on two length\-clean factual sets\. Best per cell in bold\. Single seed, up to2,0002\{,\}000items per target\.
## Appendix OWhere in the Network Does the Location Have to Be Read?

The six depths the two location streams read are taken from the literature on the roles layers play at different depths of the network, rather than from any measurement on the trained detector\. Early layers group tokens by surface form, late layers work in the space of the token about to be emitted, and representations come loose from the wording in between\. That settles a band and not a set of layers within it\. We retrain the full model five times on the ARC\-Challenge donor, changing only the depths the two location streams read \(Table[19](https://arxiv.org/html/2608.05660#A15.T19)\)\. Three arms take one third of the stack each, a fourth reads every layer, and the fifth is our window\.

Depth is the largest lever in the method’s configuration\. A motion\-only reader trained the same way reaches63\.163\.1on the reasoning mean, and reading the early third takes it to63\.763\.7, so the first third of the network is worth0\.60\.6points of location\. The middle third is worth12\.512\.5\. Across the three bands the score rises and then falls,63\.763\.7early,75\.675\.6middle and69\.569\.5late, which is the shape the three roles predict\. Six layers of the middle third also read higher than all3232,75\.675\.6against74\.074\.0, so which depths are read matters more than how many\. Reading all3232layers changes the parameter count as well, so that arm speaks to how many depths are needed and not to which\.

Our window is not the best of the five\. The middle third reads3\.03\.0points above it on the reasoning mean and is ahead on1111of the1212targets, and the ordering holds when both arms are retrained\. We keep our window because it was fixed before any of these runs and every number in this paper is measured with it\.The gain rests on reading the middle of the network, and not on a window selected to produce it\.The two axes want different depths\. Reading all3232layers gives the best factual mean,63\.763\.7against the middle third’s61\.361\.3, and most of that sits on FACTOR\-expert,72\.072\.0against64\.464\.4\. The factual sets use the late, output\-adjacent layers that reasoning does not need, which is the same place the model’s own likelihood reads them from\.

Table 19:The depths the location streams read, selection accuracy×100\\times 100, on the ARC\-Challenge donor and the full test sets\. The first three arms and ours each read six layers spread across the band shown and are identical in size; the last reads every layer and is not\. Reasoning is the mean over the eight reasoning targets and Factual over the four factual ones\. All five arms share one training configuration and one evaluation basis and differ only in the depths the two location streams read\. Our window is retrained here under that shared configuration rather than taken from Table[1](https://arxiv.org/html/2608.05660#S6.T1), so the five arms are directly comparable and the comparison is read within the sweep\. Best per column in bold\.
## Appendix PDetailed Limitations

Our detector is only ever scored on selection among the candidates an item supplies\. It is never asked whether a single answer standing on its own is correct, so we cannot say how its score behaves as an absolute judgment\. It needs the model’s residual stream, one forward pass per candidate, and a labeled donor benchmark, and it is trained for the model it reads\.

Everything the detector reads comes from the model, so it is bounded by what the model itself distinguishes\. Where the state holds little about whether an answer is correct, no reading of the trajectory supplies it\. A linear read of the raw state tells a correct completion from an incorrect one at an AUC of0\.920\.92on FACTOR\-expert and0\.610\.61on FACTOR\-wiki \(Figure[4](https://arxiv.org/html/2608.05660#S6.F4)\), and every method we compare, ours included, reads FACTOR\-wiki lower than FACTOR\-expert\. Reading the model is also what makes the bound measurable\. The same read reports how much a model holds about correctness on a new set before a detector is trained for it\. The streams read activations a forward pass already produces, so the three readings are available wherever a model scores an answer, and not only in analysis after the fact\.

The location streams keep the direction of a state and set its magnitude aside, so what they pass is an orientation on the unit sphere rather than a position in activation space\. Reading the unnormalized state in place of the direction scored lower in our development runs, so the magnitude is dropped rather than modeled\. Some of the correctness signal may sit there, and a reading that keeps it as a quantity of its own is untested\.

Our fusion does not use everything the three readings hold\. A per\-item oracle over them stands33to2121points of selection accuracy above the trained combination on the four sets we decompose \(Appendix[J](https://arxiv.org/html/2608.05660#A10), Table[10](https://arxiv.org/html/2608.05660#A10.T10)\), which bounds what a rule choosing between the streams item by item could add\. Closing that gap asks for a better gate over the three streams rather than another fixed combining rule, since per\-item gating and per\-step injection score within0\.0050\.005AUC of concatenation \(Appendix[K](https://arxiv.org/html/2608.05660#A11)\)\. What such a gate would add beyond accuracy is a reading of the detector itself\. Ours reports which stream is right only after the fact, through a variant trained to expose one, whereas a gate that weighs the three on each item would state at prediction time which reading a decision rests on, and so whether the motion of a computation or the coarse or fine location of its state settles a given case\. A detector that reports that is an instrument for asking which aspect of the model’s computation a judgment depends on\.