雅可比透镜的数学理论

# A Mathematical Theory of the Jacobian Lens
Source: [https://arxiv.org/html/2610.31955](https://arxiv.org/html/2610.31955)
Mitchell OstrowAffiliation:Department of Brain and Cognitive SciencesAffiliation:Massachusetts Institute of TechnologyEmail:[ostrow@mit\.edu](mailto:)Daoyuan QianAffiliation:Department of Brain and Cognitive SciencesAffiliation:Massachusetts Institute of TechnologyIla FieteAffiliation:Department of Brain and Cognitive SciencesAffiliation:Massachusetts Institute of Technology

###### Abstract

Recent work from Anthropic introduced a new interpretability tool, called the Jacobian lens \(J\-lens,[Gurnee et al\., 2026](https://arxiv.org/html/2610.31955#bib.bib1)\)\. This method identified linear directions in the hidden states of frontier language models that had properties reminiscent of working memory\-like representations\. However, it is unclear what mechanistic properties of such features make them detectable to the J\-lens\. To this end, we introduce a mathematical theory of the J\-lens, and show that it preferentially identifies*coherent*features – directions that act like leading, spatially coherent Lyapunov modes\. Small perturbations along these directions amplify across layers, remain roughly constant across token positions, and are consistent across different contexts\. While we are agnostic to whether these properties are necessarily associated with a “Global Workspace”, the analytically derived preferences of the J\-lens nevertheless explain prior empirical applied to both model classes\. We validate our analysis on a Bayes filter and a toy transformer, each optimally predicting next tokens emitted from a Hidden Markov Model \(HMM\)\. We show how modulating task parameters that change the coherence of features in the Bayes filter correspondingly and predictively modulates the response of the J\-lens \. We also introduce of a family of J\-lenses built to emphasize different temporal scales of information processing in LLMs, which we callω\\omega\-J\-lenses\.

## 1Introduction

In a recent paper,[Gurnee et al\. \(2026\)](https://arxiv.org/html/2610.31955#bib.bib1)introduced a novel interpretability technique that they claimed could identify an internal ‘workspace’ in large language models \(LLMs\), which had properties reminiscent of the Global Workspace\([Baars, 1988](https://arxiv.org/html/2610.31955#bib.bib4);[Dehaene et al\., 1998](https://arxiv.org/html/2610.31955#bib.bib5)\)\. The object they primarily study is the ‘Jacobian lens’, or ‘J\-lens’\. This interpretability tool is designed in the spirit of other ‘lenses’\([nostalgebraist, 2020](https://arxiv.org/html/2610.31955#bib.bib19);[Belrose et al\., 2023](https://arxiv.org/html/2610.31955#bib.bib20)\), which seek to decode transformer hidden state activations into language by applying the unembedding matrix of the model to these activations \(potentially after a learned affine map, as in the tuned lens\)\. These lenses benefit interpretability by mapping hidden states to tokens, which are in the space of language and hopefully more interpretable\. Decoding intermediate layers this way has shown that predictions are refined gradually across depth\([nostalgebraist, 2020](https://arxiv.org/html/2610.31955#bib.bib19);[Belrose et al\., 2023](https://arxiv.org/html/2610.31955#bib.bib20)\), that multilingual models pass through an English\-like latent space before producing the target language\([Wendler et al\., 2024](https://arxiv.org/html/2610.31955#bib.bib21)\), that false few\-shot demonstrations derail predictions only after a critical layer\([Halawi et al\., 2024](https://arxiv.org/html/2610.31955#bib.bib22)\), and that a single hidden state already carries information about tokens several positions ahead\([Pal et al\., 2023](https://arxiv.org/html/2610.31955#bib.bib23)\)\. The trajectory of tuned\-lens predictions across layers can also detect malicious inputs\([Belrose et al\., 2023](https://arxiv.org/html/2610.31955#bib.bib20)\)\.

The Jacobian lens \(J\-lens,[Gurnee et al\., 2026](https://arxiv.org/html/2610.31955#bib.bib1)\) continues this line of work by introducing a new transfer operator onto the residual stream vector before the application of the unembedding matrix and mapping to tokens\. This operator is the Jacobian of the transformer, averaged over tokens\. In their paper, the motivation for the usage of the Jacobian𝑲\\bm\{K\}is based on standard results in calculus: A Jacobian describes how a small perturbation in the input of a function results in a change in the output\. When the Jacobian is studied in an LLM, it provides directions in the residual stream that are “verbalizable”\([Gurnee et al\., 2026](https://arxiv.org/html/2610.31955#bib.bib1)\), which is to say that it identifies how perturbations change the linguistic output of the model\. However, this property holds only in the local neighborhood of the point on which the Jacobian is defined, and the J\-lens is defined by averaging over prompts, tokens, and future positions in the output layer\. An average Jacobian has no reason to behave like any of its elements\. Indeed, if the Jacobians of a system form a curved manifold, the average Jacobian can easily lie off\-manifold\. More generally, well\-known features that describe the behavior of linear operators \(e\.g\., singular vectors and values, which are nonlinear functions of a matrix\) can deviate grossly from those of individual Jacobians under averaging \(Appendix[A](https://arxiv.org/html/2610.31955#A1)\)\. In Figs\.[1](https://arxiv.org/html/2610.31955#S1.F1)and[10](https://arxiv.org/html/2610.31955#A2.F10)we visually demonstrate this phenomenon for four open source LLMs\. What, then, does the J\-lens capture?

Figure 1:Low\-dimensional projection \(principal component analysis, PCA\) of the top 100 singular values for individual \(black\) and mean \(red\) Jacobians of four different LLMs \(Appendix[B](https://arxiv.org/html/2610.31955#A2)\)\. Each Jacobian is evaluated with input at an intermediate layer chosen to be halfway through the model and output at the final layer\. Prompts are sampled from a range of datasets with different topics \(Wikipedia, Python code, math, news, etc\.\)\. Jacobian spectra are plotted in Fig\.[9](https://arxiv.org/html/2610.31955#A2.F9)\.Recent work\([Yan et al\., 2026](https://arxiv.org/html/2610.31955#bib.bib24)\)has attempted to analyze the J\-lens as an approximate least squares map from the hidden state to the output using Stein’s lemma\([Stein, 1981](https://arxiv.org/html/2610.31955#bib.bib25)\)\. However, Stein’s lemma only applies when the input data is sampled from a Gaussian distribution, and transformer hidden states are notoriously non\-Gaussian, with heavy\-tailed outlier features and a few activations that are orders of magnitude larger than the rest\([Dettmers et al\., 2022](https://arxiv.org/html/2610.31955#bib.bib26);[Sun et al\., 2024](https://arxiv.org/html/2610.31955#bib.bib27)\)\. Rather, the J\-lens is most directly related to a branch of dynamical systems theory calledLyapunov theory, which describes how perturbations grow or shrink in spatiotemporal dynamical systems\.

##### The Jacobian lens\.

Let𝒉ℓ,t​\(x\)∈ℝd\\bm\{h\}\_\{\\ell,t\}\(x\)\\in\\mathbb\{R\}^\{d\}be the activation at layerℓ\\elland token positionttin a fixed sequencexx, and letLLbe the final layer\. Writeτ\\tauto represent a lag in token positions,HHfor the number of lags averaged over, andc=\(x,t\)c=\(x,t\)for a context\. Define

𝑲⁡\(c,ℓ,τ\)\\displaystyle\\bm\{K\}\(c,\\ell,\\tau\):=∂𝒉L,t\+τ​\(x\)∂𝒉ℓ,t​\(x\),\\displaystyle:=\\frac\{\\partial\\bm\{h\}\_\{L,t\+\\tau\}\(x\)\}\{\\partial\\bm\{h\}\_\{\\ell,t\}\(x\)\},\(1\)𝑮c​\(ℓ,H\)\\displaystyle\\bm\{G\}\_\{c\}\(\\ell,H\):=1H​∑τ=0H−1𝑲⁡\(c,ℓ,τ\)\\displaystyle:=\\frac\{1\}\{H\}\\sum\_\{\\tau=0\}^\{H\-1\}\\bm\{K\}\(c,\\ell,\\tau\)𝑱H​\(ℓ\)\\displaystyle\\bm\{J\}\_\{H\}\(\\ell\):=𝔼c​𝑮c​\(ℓ,H\)\.\\displaystyle:=\\mathbb\{E\}\_\{c\}\\bm\{G\}\_\{c\}\(\\ell,H\)\.
The lens is then applied to a state, along with the unembedding matrix:

lens⁡\(𝒉ℓ,t\)=softmax⁡\(𝑾U​norm​\(𝑱H​\(ℓ\)​𝒉ℓ,t\)\)\\mathrm\{lens\}\(\\bm\{h\}\_\{\\ell,t\}\)=\\mathrm\{softmax\}\\big\(\\bm\{W\}\_\{U\}\\,\\mathrm\{norm\}\(\\bm\{J\}\_\{H\}\(\\ell\)\\,\\bm\{h\}\_\{\\ell,t\}\)\\big\)\(2\)where𝑾U\\bm\{W\}\_\{U\}is the model’s unembedding matrix\.

##### Lyapunov exponents\.

Letδ​𝒉​\(T\)\\delta\\bm\{h\}\(T\)be a perturbation of a state evolved forTTsteps under the dynamics\. Its Lyapunov exponent\([Oseledets, 1968](https://arxiv.org/html/2610.31955#bib.bib6);[Eckmann and Ruelle, 1985](https://arxiv.org/html/2610.31955#bib.bib7)\)measures asymptotic growth of the perturbation:

χ=limT→∞1T​log⁡‖δ​𝒉​\(T\)‖‖δ​𝒉​\(0\)‖,δ​𝒉​\(0\)≠𝟎,\\chi=\\lim\_\{T\\to\\infty\}\\frac\{1\}\{T\}\\log\\frac\{\\left\\lVert\\delta\\bm\{h\}\(T\)\\right\\rVert\}\{\\left\\lVert\\delta\\bm\{h\}\(0\)\\right\\rVert\},\\qquad\\delta\\bm\{h\}\(0\)\\neq\\bm\{0\},\(3\)when the limit exists\. At finiteTT, the finite\-time Lyapunov exponents are

χi​\(T\)=1T​log⁡si​\(T\),\\chi\_\{i\}\(T\)=\\frac\{1\}\{T\}\\log s\_\{i\}\(T\),\(4\)wheresi​\(T\)s\_\{i\}\(T\)are singular values of the fullTT\-step Jacobian\. For dynamical systems, this is typically computed by chaining with multiplication each one\-step Jacobian with iterative renormalization\([Benettin et al\., 1980](https://arxiv.org/html/2610.31955#bib.bib9);[Engelken et al\., 2023](https://arxiv.org/html/2610.31955#bib.bib11)\)\. Spatiotemporal versions also resolve how a perturbation travels across space: the convective Lyapunov exponent\([Deissler and Kaneko, 1987](https://arxiv.org/html/2610.31955#bib.bib8);[Jiotsa et al\., 2013](https://arxiv.org/html/2610.31955#bib.bib12)\)measures the growth of a perturbation in a reference frame that is moving in space\. Hydrodynamic Lyapunov modes\([Eckmann and Gat, 2000](https://arxiv.org/html/2610.31955#bib.bib13)\)are long\-wavelength Lyapunov vectors whose exponents approach zero as their wavenumberkkapproaches zero\.

For transformers, depth is the evolution coordinate \(TT\) and token position the spatial coordinate\. A hydrodynamic interpretation requires identifying long wavelength with growth near zero\. We’ll show here that the J\-space is tightly linked to Lyapunov theory\. Under the assumptions below, the J\-lens favors spatially smooth, slowly varying responses, because it by construction integrates out the fast dynamics, implicitly behaving like a low\-pass filter in space\. With this connection in hand, we show that many of the results in the original paper follow naturally\. Note that this is not an argument that it does not identify a Global Workspace per se, but rather to shed light on what this method is designed to capture and why that may or may not align with the philosophical conception of the Global Workspace\.

We begin by constructing a model that establishes this connection, and subsequently relax assumptions of the model\.

##### Shared and context\-specific modes\.

Suppose our Jacobians can be decomposed as follows:

𝑲⁡\(c,ℓ,τ\)\\displaystyle\\bm\{K\}\(c,\\ell,\\tau\)=∑ieT​χi\+τ​γi​𝒖i​𝒗i∗⏟𝑺⁡\(ℓ,τ\):shared modes\+∑jeT​χc​j\+τ​γc​j​𝒖c​j​𝒗c​j∗⏟𝑹⁡\(c,ℓ,τ\):context\-specific modes,\\displaystyle=\\underbrace\{\\sum\_\{i\}e^\{T\\chi\_\{i\}\+\\tau\\gamma\_\{i\}\}\\,\\bm\{u\}\_\{i\}\\bm\{v\}\_\{i\}^\{\*\}\}\_\{\\bm\{S\}\(\\ell,\\tau\):\\,\\text\{shared modes\}\}\+\\underbrace\{\\sum\_\{j\}e^\{T\\chi\_\{cj\}\+\\tau\\gamma\_\{cj\}\}\\,\\bm\{u\}\_\{cj\}\\bm\{v\}\_\{cj\}^\{\*\}\}\_\{\\bm\{R\}\(c,\\ell,\\tau\):\\,\\text\{context\-specific modes\}\},\(5\)withT=L−ℓT=L\-\\ellthe number of layers between the read\-in layer and the output, and every mode quantity depends onℓ\\ell, which we suppress\.γi=μi\+i​θi\\gamma\_\{i\}=\\mu\_\{i\}\+\\mathrm\{i\}\\theta\_\{i\}describes the cross\-token evolution dynamics\. The shared input directions𝒗i\\bm\{v\}\_\{i\}and output directions𝒖i\\bm\{u\}\_\{i\}are each orthonormal\. Each mode has two finite\-time Lyapunov exponents:χi∈ℝ\\chi\_\{i\}\\in\\mathbb\{R\}in depth andμi∈ℝ\\mu\_\{i\}\\in\\mathbb\{R\}in token position, whileθi\\theta\_\{i\}is the mode’s rotation per token\. At a fixed\(ℓ,τ\)\(\\ell,\\tau\), any matrix admits a singular value decomposition \(SVD\)\. Sharing directions across contexts, factoring depth \(Fig\.[2](https://arxiv.org/html/2610.31955#S1.F2)a\) and lag \(Fig\.[2](https://arxiv.org/html/2610.31955#S1.F2)b\) in this way, and treating the two exponents as independent are additional assumptions of this illustrative model\. A Lyapunov mode is a mode that evolves according to a Lyapunov exponent, and hence is defined by its input direction𝒗i\\bm\{v\}\_\{i\}\. Averaging over lags and contexts splits the lens into a shared part and an averaged remainder,𝑱H=𝑺H\+𝔼c​𝑹c\\bm\{J\}\_\{H\}=\\bm\{S\}\_\{H\}\+\\mathbb\{E\}\_\{c\}\\bm\{R\}\_\{c\}, in which shared modeiiis weighted byeT​χi​fH​\(γi\)e^\{T\\chi\_\{i\}\}f\_\{H\}\(\\gamma\_\{i\}\), with

fH​\(γ\)=1H​∑τ=0H−1eτ​γf\_\{H\}\(\\gamma\)=\\frac\{1\}\{H\}\\sum\_\{\\tau=0\}^\{H\-1\}e^\{\\tau\\gamma\}whereHHdenotes the maximum lag between input and output token positions \(Appendix[C](https://arxiv.org/html/2610.31955#A3)\)\. We also assume the context\-specific response is no larger than the shared one:𝔼c​‖𝑹c‖22≤M2\\mathbb\{E\}\_\{c\}\\left\\lVert\\bm\{R\}\_\{c\}\\right\\rVert\_\{2\}^\{2\}\\leq M^\{2\}, whereM=‖𝑺H‖2M=\\left\\lVert\\bm\{S\}\_\{H\}\\right\\rVert\_\{2\}\.

##### Isolation of slow modes\.

The future averagingfH​\(γ\)f\_\{H\}\(\\gamma\)suppresses modes that decay rapidly or oscillate quickly\. Forμi≤0\\mu\_\{i\}\\leq 0,\|fH​\(γi\)\|≤1\|f\_\{H\}\(\\gamma\_\{i\}\)\|\\leq 1\. Slow modes withH​\|μi\|,H​\|θi\|≪1H\|\\mu\_\{i\}\|,H\|\\theta\_\{i\}\|\\ll 1retainfH​\(γi\)≈1f\_\{H\}\(\\gamma\_\{i\}\)\\approx 1, while rapid decay or oscillation is attenuated\. Another way to see this is to consider the Fourier decomposition of a responsey⁡\(τ\)y\(\\tau\)over the window:

y~H​\(ω\)\\displaystyle\\widetilde\{y\}\_\{H\}\(\\omega\)=∑τ=0H−1e−i​ω​τ​y​\(τ\),\\displaystyle=\\sum\_\{\\tau=0\}^\{H\-1\}e^\{\-\\mathrm\{i\}\\omega\\tau\}y\(\\tau\),\(6\)1H​y~H​\(0\)\\displaystyle\\frac\{1\}\{H\}\\widetilde\{y\}\_\{H\}\(0\)=1H​∑τ=0H−1y⁡\(τ\)\.\\displaystyle=\\frac\{1\}\{H\}\\sum\_\{\\tau=0\}^\{H\-1\}y\(\\tau\)\.\(7\)Thus averaging over time extracts the zero\-frequency coefficient of the finite window, thereby identifying slowly varying responses in space \(token position\)\. Similarly, time \(layers\) weights modes byeT​χie^\{T\\chi\_\{i\}\}, thereby favoring growing or slowly decaying modes\. This is not necessarily a strong weighting, especially closer to the output when the Jacobian becomes increasingly aligned with the identity map\.

##### Incoherent directions are suppressed\.

For a unit input direction𝒗\\bm\{v\}and independent contextsc,c′c,c^\{\\prime\}, the averaged remainder is‖\(𝔼c​𝑹c\)​𝒗‖22=𝔼c,c′​⟨𝑹c​𝒗,𝑹c′​𝒗⟩\\left\\lVert\(\\mathbb\{E\}\_\{c\}\\bm\{R\}\_\{c\}\)\\bm\{v\}\\right\\rVert\_\{2\}^\{2\}=\\mathbb\{E\}\_\{c,c^\{\\prime\}\}\\langle\\bm\{R\}\_\{c\}\\bm\{v\},\\bm\{R\}\_\{c^\{\\prime\}\}\\bm\{v\}\\rangle, which quantifies the expected overlap between two contexts’ responses to the same input \(Appendix[C](https://arxiv.org/html/2610.31955#A3)\)\. Suppose this overlap is small,sup‖𝒗‖=1𝔼c,c′​\|⟨𝑹c​𝒗,𝑹c′​𝒗⟩\|≤η​𝔼c​‖𝑹c‖22\\sup\_\{\\left\\lVert\\bm\{v\}\\right\\rVert=1\}\\mathbb\{E\}\_\{c,c^\{\\prime\}\}\|\\langle\\bm\{R\}\_\{c\}\\bm\{v\},\\bm\{R\}\_\{c^\{\\prime\}\}\\bm\{v\}\\rangle\|\\leq\\eta\\,\\mathbb\{E\}\_\{c\}\\left\\lVert\\bm\{R\}\_\{c\}\\right\\rVert\_\{2\}^\{2\}with0≤η<10\\leq\\eta<1\. Taking the largest growth over unit inputs and assuming bounded scales of the modes,

‖𝔼c​𝑹c‖22=sup‖𝒗‖=1𝔼c,c′​⟨𝑹c​𝒗,𝑹c′​𝒗⟩≤sup‖𝒗‖=1𝔼c,c′​\|⟨𝑹c​𝒗,𝑹c′​𝒗⟩\|≤η​𝔼c​‖𝑹c‖22≤η​M2=η​‖𝑺H‖22\.\\boxed\{\\begin\{aligned\} \\left\\lVert\\mathbb\{E\}\_\{c\}\\bm\{R\}\_\{c\}\\right\\rVert\_\{2\}^\{2\}&=\\sup\_\{\\left\\lVert\\bm\{v\}\\right\\rVert=1\}\\mathbb\{E\}\_\{c,c^\{\\prime\}\}\\langle\\bm\{R\}\_\{c\}\\bm\{v\},\\bm\{R\}\_\{c^\{\\prime\}\}\\bm\{v\}\\rangle\\\\ &\\leq\\sup\_\{\\left\\lVert\\bm\{v\}\\right\\rVert=1\}\\mathbb\{E\}\_\{c,c^\{\\prime\}\}\|\\langle\\bm\{R\}\_\{c\}\\bm\{v\},\\bm\{R\}\_\{c^\{\\prime\}\}\\bm\{v\}\\rangle\|\\\\ &\\leq\\eta\\,\\mathbb\{E\}\_\{c\}\\left\\lVert\\bm\{R\}\_\{c\}\\right\\rVert\_\{2\}^\{2\}\\leq\\eta M^\{2\}=\\eta\\left\\lVert\\bm\{S\}\_\{H\}\\right\\rVert\_\{2\}^\{2\}\.\\end\{aligned\}\}\(8\)Thus for small overlapη\\eta, averaging suppresses the incoherent response relative to the shared response\. Since𝑱H=𝑺H\+𝔼c​𝑹c\\bm\{J\}\_\{H\}=\\bm\{S\}\_\{H\}\+\\mathbb\{E\}\_\{c\}\\bm\{R\}\_\{c\}, its leading singular directions approach those of𝑺H\\bm\{S\}\_\{H\}asη→0\\eta\\to 0\.

##### Relaxing exact shared modes\.

The above model shows that if spatially \(in a transformer, across tokens\) and contextually coherent Lyapunov modes exist, these modes dominate in the Jacobian lens\. However, exact shared directions need not exist\. We show next under a more general construction that the J\-lens directions preferentially select Lyapunov modes that \(a\) maximally grow over depthTT\(magnitudeaa\), \(b\) are maximally coherent across token lagsτ\\tau\(lag coherenceρτ\\rho\_\{\\tau\}\), and \(c\) are maximally coherent across different contextscc\(context coherenceρc\\rho\_\{c\}\)\. Using the same SVD notation, write

𝑮c=𝔼τ​𝑲​\(c,ℓ,τ\)=∑jsc​j​𝒖c​j​𝒗c​j∗,sc​1≥sc​2≥⋯≥0\.\\bm\{G\}\_\{c\}=\\mathbb\{E\}\_\{\\tau\}\\bm\{K\}\(c,\\ell,\\tau\)=\\sum\_\{j\}s\_\{cj\}\\bm\{u\}\_\{cj\}\\bm\{v\}\_\{cj\}^\{\*\},\\qquad s\_\{c1\}\\geq s\_\{c2\}\\geq\\cdots\\geq 0\.\(9\)For a unit input direction𝒗\\bm\{v\}, define

a⁡\(𝒗\)\\displaystyle a\(\\bm\{v\}\):=𝔼c,τ​‖𝑲⁡\(c,ℓ,τ\)​𝒗‖,\\displaystyle:=\\mathbb\{E\}\_\{c,\\tau\}\\left\\lVert\\bm\{K\}\(c,\\ell,\\tau\)\\,\\bm\{v\}\\right\\rVert,\(10\)ρτ​\(𝒗\)\\displaystyle\\rho\_\{\\tau\}\(\\bm\{v\}\):=𝔼c​‖𝑮c​𝒗‖𝔼c,τ​‖𝑲⁡\(c,ℓ,τ\)​𝒗‖∈\[0,1\],\\displaystyle:=\\frac\{\\mathbb\{E\}\_\{c\}\\left\\lVert\\bm\{G\}\_\{c\}\\bm\{v\}\\right\\rVert\}\{\\mathbb\{E\}\_\{c,\\tau\}\\left\\lVert\\bm\{K\}\(c,\\ell,\\tau\)\\,\\bm\{v\}\\right\\rVert\}\\in\[0,1\],ρc​\(𝒗\)\\displaystyle\\rho\_\{c\}\(\\bm\{v\}\):=‖𝔼c,τ​𝑲​\(c,ℓ,τ\)​𝒗‖𝔼c​‖𝑮c​𝒗‖∈\[0,1\]\.\\displaystyle:=\\frac\{\\left\\lVert\\mathbb\{E\}\_\{c,\\tau\}\\bm\{K\}\(c,\\ell,\\tau\)\\,\\bm\{v\}\\right\\rVert\}\{\\mathbb\{E\}\_\{c\}\\left\\lVert\\bm\{G\}\_\{c\}\\bm\{v\}\\right\\rVert\}\\in\[0,1\]\.We call these thecoherence parametersfor directions in the Jacobian lens\. We use the 2\-norm for each parameter here, but other norms could reasonably apply as well\.a⁡\(𝒗\)a\(\\bm\{v\}\)is the average response magnitude,ρτ​\(𝒗\)\\rho\_\{\\tau\}\(\\bm\{v\}\)is the lag coherence of a response, andρc​\(𝒗\)\\rho\_\{c\}\(\\bm\{v\}\)is the context coherence\. These coherence parameters can be understood intuitively through the lens of spatiotemporal Lyapunov modes:aadescribes the canonical notion of a Lyapunov exponent, here how a direction grows over time \(layers\)\.ρτ\\rho\_\{\\tau\}describes how activity along a direction oscillates across spatial \(token\) positions, withρτ=1\\rho\_\{\\tau\}=1indicating no oscillation\. Lastly,ρc\\rho\_\{c\}describes how consistently direction is read in and written out across different contexts \(instantiations of Jacobian\), given the same vector\.

We illustrate what each coherence parameter measures in Fig\.[2](https://arxiv.org/html/2610.31955#S1.F2)\. Intuitively, magnitude measures the total effect of a perturbation along a feature direction across both layers and time, thereby capturing how persistent or relevant it is\. Lag coherence measures how the response of a perturbation changes across token positions, for which large values are approximately static, such as the overall topic of a document, whereas incoherent features change rapidly, such as word type\. Context coherence measures whether a perturbation along a direction responds similarly in the same way for different contexts, and how prior context modulates its effect\. For example, adding in a new rule such as “today is opposite day” into context requires reprocessing each feature with this in mind\. Context\-coherent directions are processed consistently across contexts, while incoherent directions change their behavior\.

Figure 2:Schematic of the three coherence parameters\.\(left\)Schematic of largeχ\>0\\chi\>0versus smallχ<0\\chi<0magnitude directions\. A perturbation grows across layers in the red direction, but shrinks in the blue direction\.\(middle\)Schematic of lag\-coherent \(black\) versus lag\-incoherent \(gray\) feature directions across token positions\.\(right\)Context\-coherent directions evolve similarly across layers in contexts, while context\-incoherent directions evolve differently\.Defineρc=0\\rho\_\{c\}=0,ρτ=0\\rho\_\{\\tau\}=0when their denominators are zero\. The three factors telescope via multiplication to yield‖𝑱H​𝒗‖\\left\\lVert\\bm\{J\}\_\{H\}\\bm\{v\}\\right\\rVert\. The leading singular value of the lens is therefore

σ1​\(𝑱H\)=max‖𝒗‖=1⁡ρc​\(𝒗\)​ρτ​\(𝒗\)​a​\(𝒗\)\.\\boxed\{\\sigma\_\{1\}\(\\bm\{J\}\_\{H\}\)=\\max\_\{\\left\\lVert\\bm\{v\}\\right\\rVert=1\}\\rho\_\{c\}\(\\bm\{v\}\)\\,\\rho\_\{\\tau\}\(\\bm\{v\}\)\\,a\(\\bm\{v\}\)\.\}\(11\)Subsequent directions maximize the same quantity subject to orthogonality to prior input directions𝒗1,…,𝒗i−1\\bm\{v\}\_\{1\},\\dots,\\bm\{v\}\_\{i\-1\}\. This shows that the J\-lens preferentially upweights directions with the largest combinations of magnitudesa⁡\(𝒗\)a\(\\bm\{v\}\)and coherencesρτ​\(𝒗\)​ρc​\(𝒗\)\\rho\_\{\\tau\}\(\\bm\{v\}\)\\,\\rho\_\{c\}\(\\bm\{v\}\)\. In the notation of Eq\.[5](https://arxiv.org/html/2610.31955#S1.E5), the J\-lens selects the modes with the largest growth across layersχi\\chi\_\{i\}\(magnitude\), spatial changes that do not oscillate much \(θi≈0\\theta\_\{i\}\\approx 0\), and𝒖i\\bm\{u\}\_\{i\},𝒗i\\bm\{v\}\_\{i\}that change minimally across contexts \(context coherence\)\.[Gurnee et al\. \(2026\)](https://arxiv.org/html/2610.31955#bib.bib1)found that the J\-space summarized only a small portion of the variance\. This observation is easily explained by recognizing that variance explained is unrelated to each of the factors in Eq\.[11](https://arxiv.org/html/2610.31955#S1.E11), which we formalize now:

###### Corollary 1\.1\.

A direction selected by Jacobian averaging may explain arbitrarily little or arbitrarily much activation variance\. Consequently, the singular values of𝐉H\\bm\{J\}\_\{H\}do not determine the variance explained by its selected directions\.

The proof is a two\-dimensional counterexample and is given in Appendix[D](https://arxiv.org/html/2610.31955#A4)\.

### 1\.1Mapping coherent modes onto tokens

The analysis above characterizes the singular modes selected by Jacobian averaging, whereas the J\-space used in the original paper is constructed from token\-indexed directions𝒋q\\bm\{j\}\_\{q\}and is therefore not simply the leading singular subspace of𝑱H\\bm\{J\}\_\{H\}\. These descriptions can nevertheless be connected exactly\. Let𝑱H=𝑼​𝚺​𝑽∗\\bm\{J\}\_\{H\}=\\bm\{U\}\\bm\{\\Sigma\}\\bm\{V\}^\{\*\}, and let𝒘q\\bm\{w\}\_\{q\}denote the unembedding vector for tokenqq\. The corresponding J\-lens direction in the input residual stream is

𝒋q:=𝑱H∗​𝒘q=𝑽​𝚺​𝑼∗​𝒘q=∑iσi​⟨𝒖i,𝒘q⟩​𝒗i\.\\bm\{j\}\_\{q\}:=\\bm\{J\}\_\{H\}^\{\*\}\\bm\{w\}\_\{q\}=\\bm\{V\}\\bm\{\\Sigma\}\\bm\{U\}^\{\*\}\\bm\{w\}\_\{q\}=\\sum\_\{i\}\\sigma\_\{i\}\\langle\\bm\{u\}\_\{i\},\\bm\{w\}\_\{q\}\\rangle\\bm\{v\}\_\{i\}\.\(12\)Thus every token\-indexed J\-lens direction is a mixture of the right singular modes𝒗i\\bm\{v\}\_\{i\}: the singular valueσi\\sigma\_\{i\}weights the strength of the corresponding coherent response channel, while⟨𝒖i,𝒘q⟩\\langle\\bm\{u\}\_\{i\},\\bm\{w\}\_\{q\}\\ranglemeasures how strongly that channel’s downstream response aligns with the token’s unembedding direction\. This connection matters because the J\-space is operationally defined by approximating an activation as a sparse non\-negative combination of the directions𝒋q\\bm\{j\}\_\{q\}\([Gurnee et al\., 2026](https://arxiv.org/html/2610.31955#bib.bib1)\)\. The sparse decomposition therefore selects a small number of semantically labeled mixtures of the coherent modes that survive averaging\. In this sense, the singular structure of𝑱H\\bm\{J\}\_\{H\}supplies the underlying dynamical geometry, while the unembedding and sparse non\-negative decomposition translate that geometry into the token\-level “concepts” reported by the J\-lens\.

### 1\.2Families of Jacobian Lenses

Our analysis suggests that the Jacobian lens can be modified to extract different types of information\. Here, we focus on the modulation of lag coherence selectivity\. Eq\.[7](https://arxiv.org/html/2610.31955#S1.E7)highlights that averaging over time points is equivalent to a Fourier projection onto the zero\-frequency mode\. Modulating theω\\omegaparameter in this projection \(or even generalizing to band\-pass filtering\) induces a family of lenses we callω\\omega\-J\-lenses, whereω∈\[−π,π\]\\omega\\in\[\-\\pi,\\pi\]given the Nyquist limit for sampling frequency11/ token\. We demonstrate their selectivity in our toy task in Fig\.[6](https://arxiv.org/html/2610.31955#S4.F6)\.

## 2Toy Model

We designed a simple toy task to study the J\-lens for models performing next\-token prediction, and demonstrate its behavior on optimal Bayes filters as well as trained transformers\. Our task, inspired by recent work studying how transformers learn to track hidden states\([Shai et al\., 2024](https://arxiv.org/html/2610.31955#bib.bib3)\), is a hidden Markov model \(HMM\) with several independent latent statesX\(i\)X^\{\(i\)\}, each arranged on a ring ofnin\_\{i\}discrete states:X\(i\)∈\{0,1,…,ni−1\}X^\{\(i\)\}\\in\\\{0,1,\\dots,n\_\{i\}\-1\\\}\. We design the dynamics to be the simplest model that can selectively modulate each of the coherence factors via the task parameters\. The dynamics of each latent state are \(suppressing the latent index for clarity\):

Xt\+1=\{Xt\+v\+Dtmodnw\.p\.​λUtw\.p\.​1−λX\_\{t\+1\}=\\begin\{cases\}X\_\{t\}\+v\+D\_\{t\}\\mod n\\quad\\text\{w\.p\. \}\\lambda\\\\ U\_\{t\}\\quad\\text\{w\.p\. \}1\-\\lambda\\end\{cases\}\(13\)Hereλ\\lambdais called the state persistence parameter,Dt∈\{−1,0,1\}D\_\{t\}\\in\\\{\-1,0,1\\\}with probabilitiesq,1−2​q,qq,1\-2q,qandUt∼Uniform⁡\{0,…,n−1\}U\_\{t\}\\sim\\mathrm\{Uniform\}\\\{0,\\dots,n\-1\\\}\. We will see how changingλ\\lambdamodulates magnitudeaaby determining for how many positions a token is useful to estimate the state of the latents\. The integervvis the velocity of the state on the ring,θ=2​π​v/n\\theta=2\\pi v/nis its angular velocity, andDtD\_\{t\}is a diffusion term\. Lag coherence measures whether a state drives all future states in the same way, andvvmodulates this by increasing misalignment across positions\. With a fully observed state at all timepoints, however, the model is not forced to integrate over time \(filtering\), which would make the Jacobian00across all timepoints\. To force integration, we reveal the states with probabilityε\\varepsilon, via another random variableBt∈\{0,1\}B\_\{t\}\\in\\\{0,1\\\}\. Each latent state is combined into a token by concatenating state index into annn\-ary variable and converting that to a token indexYtY\_\{t\}, making a total of1\+∏ini1\+\\prod\_\{i\}n\_\{i\}tokens, where the additional token is a null token \(states unobserved\)\. In the following, we use three latents withni∈\{2,8\}n\_\{i\}\\in\\\{2,8\\\}for simplicity\. We will show how this model setup allows us to selectively modulate the magnitudeaa\(throughλ\\lambda\) and lag coherenceρτ\\rho\_\{\\tau\}\(throughθ\\theta\) of the Jacobian lens vectors\. Note that this toy model is easily solved by a transformer, and indeed even the embeddings learn to represent the latent states\. The main result of our analysis is that we cana priori predict the order of the modes and their approximate coherence parameters in the Jacobian lens of the transformer\.

### 2\.1Introducing contextual dependency

So far, all tokens across all random sequences for a given model are integrated the same way\. This means that thecontext coherenceρc\\rho\_\{c\}of the Jacobian lens vectors cannot be modulated\. To introduce this, we add a new variable called akeystateWt∈\{−1,\+1\}W\_\{t\}\\in\\\{\-1,\+1\\\}where the observed stateYt=Xt\+n2​𝟏​\(Wt−1=−1\)Y\_\{t\}=X\_\{t\}\+\\frac\{n\}\{2\}\\mathbf\{1\}\(W\_\{t\-1\}=\-1\), andP⁡\(Wt=1\)=\(1\+ψ\)/2P\(W\_\{t\}=1\)=\(1\+\\psi\)/2, with key biasψ∈\[0,1\]\\psi\\in\[0,1\]\. This state is always fully observed, forcing the models to look back one timestep to identify how to infer the state of the observed token at the present timestep\. This directly modulates context coherence because it forces the read\-in and read\-out directions of a feature to change dependent on the context\. Each keyed state has its own independent key, making\(1\+2​∏ini\)​2\|𝒦\|\(1\+2\\prod\_\{i\}n\_\{i\}\)\\,2^\{\|\\mathcal\{K\}\|\}tokens, where𝒦\\mathcal\{K\}is the set of latents with keys\.

## 3The Bayes Filter and its Jacobian Lens

Prior work\([Shai et al\., 2024](https://arxiv.org/html/2610.31955#bib.bib3)\)has found that transformers trained on HMMs learned to represent optimal Bayesian beliefs of the hidden states in their residual stream\. We follow this line of work by deriving the Bayes filter for the above HMM, its Jacobian lens, and then show that the Jacobian lens captures belief statesordered by their coherence parametersin the transformer as well\. Because each state exists on a ring, we account for rotational symmetry by studying the Fourier modesϕj​\(x\)=e2​π​i​j​x/n\\phi\_\{j\}\(x\)=e^\{2\\pi\\mathrm\{i\}jx/n\}\. The Bayes filter optimally integrates the given evidence into abelief state, which is defined as the expected value of a state given the history:

mt\(j\)\\displaystyle m^\{\(j\)\}\_\{t\}:=𝔼\[ϕj\(Xt\+1\)∣Y1,…,Yt\]\\displaystyle:=\\mathbb\{E\}\[\\phi\_\{j\}\(X\_\{t\+1\}\)\\mid Y\_\{1\},\\dots,Y\_\{t\}\]\(14\)m0\(j\)\\displaystyle m^\{\(j\)\}\_\{0\}:=0\.\\displaystyle:=0\.\(15\)
The belief state for a given Fourier mode of a given state is updated by unkeyed evidence as

mt\(j\)=αj​\[\(1−Bt\)​mt−1\(j\)\+Bt​ϕj​\(Xt\)\]m^\{\(j\)\}\_\{t\}=\\alpha\_\{j\}\\big\[\(1\-B\_\{t\}\)m^\{\(j\)\}\_\{t\-1\}\+B\_\{t\}\\phi\_\{j\}\(X\_\{t\}\)\\big\]\(16\)whereαj=λ​cj​ei​θj\\alpha\_\{j\}=\\lambda c\_\{j\}e^\{\\mathrm\{i\}\\theta\_\{j\}\}, withθj=j​θ\\theta\_\{j\}=j\\thetathe rotation of modejjper step andcj=1−2​q\+2​q​cos⁡\(2​π​j/n\)c\_\{j\}=1\-2q\+2q\\cos\(2\\pi j/n\)the diffusion factor \(Appendix[E](https://arxiv.org/html/2610.31955#A5)\)\. This update rule is locally linear, either updating with the expected evolution operatorαj\\alpha\_\{j\}if no evidence is observed, and replacing the state with the observation otherwise\. The graphical model of the task and the Bayes filter are schematized in Fig\.[3](https://arxiv.org/html/2610.31955#S3.F3)\.

Figure 3:Graphical models of the task\(a\)and the Bayes filter\(b\)\.\(a\)Three latentsXt\(i\)X^\{\(i\)\}\_\{t\}evolve independently on their rings\. The reveal variableBtB\_\{t\}sets whether the tokenYtY\_\{t\}shows the latents, and in the context sweep the keyWt−1W\_\{t\-1\}sets whether the shown state is flipped\.\(b\)The beliefmtm\_\{t\}is the previous beliefmt−1m\_\{t\-1\}propagated byα\\alpha, or, whenBt=1B\_\{t\}=1, the revealed state fromYtY\_\{t\}propagated byα\\alpha\.### 3\.1Keyed Filters

Keyed states are observed differently, but the key stateWt−1W\_\{t\-1\}is directly observable at every token and is invertible\. This means that the Fourier mode of the evidence is directly related to the state:ϕj​\(Yt\)=Wt−1​ϕj​\(Xt\)\\phi\_\{j\}\(Y\_\{t\}\)=W\_\{t\-1\}\\phi\_\{j\}\(X\_\{t\}\)\. The Bayes filter can be similarly extended, as

mt\(j\)=αj​\[\(1−Bt\)​mt−1\(j\)\+Bt​𝟏​\(Wt−1=1\)​ϕj​\(Yt\)\+Bt​𝟏​\(Wt−1=−1\)​Wt−1​ϕj​\(Yt\)\]m^\{\(j\)\}\_\{t\}=\\alpha\_\{j\}\\big\[\(1\-B\_\{t\}\)m^\{\(j\)\}\_\{t\-1\}\+B\_\{t\}\\mathbf\{1\}\(W\_\{t\-1\}=1\)\\phi\_\{j\}\(Y\_\{t\}\)\+B\_\{t\}\\mathbf\{1\}\(W\_\{t\-1\}=\-1\)W\_\{t\-1\}\\phi\_\{j\}\(Y\_\{t\}\)\\big\]\(17\)
The first of the latter two terms accounts for the setting in which the key does not change the state, and the second accounts for the changed state by inverting the rotation to recover the state\. This forces the predictive model to change where it reads in token information and writes to the predicted next token, thereby modulating the contextual coherence of the dynamics\.

### 3\.2The Jacobian Lens

In this section we derive the three factors of the J\-lens for the Bayes filter \(magnitudea\(j\)a^\{\(j\)\}, lag coherenceρτ\(j\)\\rho\_\{\\tau\}^\{\(j\)\}and context coherenceρc\(j\)\\rho\_\{c\}^\{\(j\)\}\) and show how the task parameters set them\. We focus on one latent and one Fourier modejj: the Jacobian of modejjof the belief att\+τt\+\\tauwith respect to modejjof the evidence at the revealed sourcett, in contextcc,

Kj​\(c,τ\)=∂mt\+τ\(j\)∂ϕj​\(Xt\)\.K\_\{j\}\(c,\\tau\)=\\frac\{\\partial m^\{\(j\)\}\_\{t\+\\tau\}\}\{\\partial\\phi\_\{j\}\(X\_\{t\}\)\}\.
This object captures all of the dynamics, as the latents are independent and the filter acts on each Fourier mode separately\. Thus the Jacobians between different latents or different modes are zero, and the lens is block diagonal, making each mode its own separate singular vector\. The piecewise\-linear components of the dynamics combine to form

Kj​\(c,τ\)=αjτ\+1​∏k=1τ\(1−Bt\+k\)=αjτ\+1​Aτ,K\_\{j\}\(c,\\tau\)=\\alpha\_\{j\}^\{\\tau\+1\}\\prod\_\{k=1\}^\{\\tau\}\(1\-B\_\{t\+k\}\)=\\alpha\_\{j\}^\{\\tau\+1\}A\_\{\\tau\},\(18\)where the product is one if no evidence has been observed across all tokens from positiontttot\+τt\+\\tau, and zero otherwise\.

Letrj:=\|αj\|​\(1−ε\)r\_\{j\}:=\\left\\lvert\\alpha\_\{j\}\\right\\rvert\(1\-\\varepsilon\)be the memory of modejjacross one step, whereε=P⁡\(Bt=1\)\\varepsilon=P\(B\_\{t\}=1\), and letN⁡\(c\)≥1N\(c\)\\geq 1be the number of tokens until the next reveal, so thatP⁡\(N=k\)=\(1−ε\)k−1​εP\(N=k\)=\(1\-\\varepsilon\)^\{k\-1\}\\varepsilon\. Summing over lags gives∑τ=1HKj​\(c,τ\)=αj​Sj,N⁡\(c\)\\sum\_\{\\tau=1\}^\{H\}K\_\{j\}\(c,\\tau\)=\\alpha\_\{j\}S\_\{j,N\(c\)\}, whereSj,N:=∑τ=1min⁡\(N−1,H\)αjτS\_\{j,N\}:=\\sum\_\{\\tau=1\}^\{\\min\(N\-1,H\)\}\\alpha\_\{j\}^\{\\tau\}is the response accumulated before the next reveal\. This sum matches the lens of Eq\. \([1](https://arxiv.org/html/2610.31955#S1.E1)\) up to the averaging factor1/H1/H, which scales every singular value equally and so leaves their order unchanged\. For largeHHthe three coherence parameters are \(Appendix[E\.1](https://arxiv.org/html/2610.31955#A5.SS1)\)

a\(j\)\\displaystyle a^\{\(j\)\}=\|αj\|​rj1−rj,\\displaystyle=\\frac\{\|\\alpha\_\{j\}\|r\_\{j\}\}\{1\-r\_\{j\}\},\(19\)ρτ\(j\)\\displaystyle\\rho\_\{\\tau\}^\{\(j\)\}=\(1−rj\)​𝔼c​\|Sj,N\|rj,\\displaystyle=\\frac\{\(1\-r\_\{j\}\)\\,\\mathbb\{E\}\_\{c\}\\left\\lvert S\_\{j,N\}\\right\\rvert\}\{r\_\{j\}\},ρc\(j\)\\displaystyle\\rho\_\{c\}^\{\(j\)\}=\|ψ\|​rj\|1−rj​ei​θj\|​𝔼c​\|Sj,N\|,\\displaystyle=\|\\psi\|\\frac\{r\_\{j\}\}\{\\left\\lvert 1\-r\_\{j\}e^\{\\mathrm\{i\}\\theta\_\{j\}\}\\right\\rvert\\,\\mathbb\{E\}\_\{c\}\\left\\lvert S\_\{j,N\}\\right\\rvert\},where𝔼c​\|Sj,N\|\\mathbb\{E\}\_\{c\}\\left\\lvert S\_\{j,N\}\\right\\rvertis the series given in Eq\. \([43](https://arxiv.org/html/2610.31955#A5.E43)\)\. Whenθj=0\\theta\_\{j\}=0, every term ofSj,NS\_\{j,N\}is real and positive, soρτ\(j\)=1,ρc\(j\)=\|ψ\|\\rho\_\{\\tau\}^\{\(j\)\}=1,\\rho\_\{c\}^\{\(j\)\}=\|\\psi\|and the persistenceλ\\lambdachanges only the magnitudea\(j\)a^\{\(j\)\}\. A nonzero angular velocityθ\\thetarotates successive terms ofSj,NS\_\{j,N\}so that they partly cancel, which lowers the lag coherenceρτ\(j\)\\rho\_\{\\tau\}^\{\(j\)\}\.

### 3\.3Connection to the Transformer

The Jacobian lens on the transformer is defined by𝑾U​𝑱H\\bm\{W\}\_\{U\}\\bm\{J\}\_\{H\}, where𝑾U\\bm\{W\}\_\{U\}is the unembedding matrix\. Given that the activity after the unembedding layer maps to tokens via a softmax, the J\-lens on the transformer studies the effect of perturbations on the log probabilities, rather than the probabilities of each state as in the Bayes filter\. The difference on the Jacobian is a factor of∂log⁡p⁡\(Xt\)=∂p⁡\(Xt\)/p⁡\(Xt\)\\partial\\log p\(X\_\{t\}\)=\\partial p\(X\_\{t\}\)/p\(X\_\{t\}\), which is an effective reweighting by a positive factor\. This reweighting, however, preserves the order of each factor \(aa,ρτ\\rho\_\{\\tau\},ρc\\rho\_\{c\}\), thus the qualitative ordering as predicted by the theory should transfer to the transformer\.

## 4Numerical Results

In the following three sections, we first introduce the parameter that is modulated across multiple independent latent states in the HMM \(three, across all sections\) to induce selective changes in either magnitudeaa, lag coherenceρτ\\rho\_\{\\tau\}, or context coherenceρc\\rho\_\{c\}\. We then apply the Bayes filter to validate the analytical theory\. Next, we train a two\-layer decoder\-only transformer\([Vaswani et al\., 2017](https://arxiv.org/html/2610.31955#bib.bib14)\)with residual dimension 128 and 4 heads per layer to predict the next token given the history \(see Appendix[F](https://arxiv.org/html/2610.31955#A6)for further details\)\. Our primary result is a validation that the Jacobian lensunsupervisedlyidentifies independent modes associated with the latent states, ordered by the selectively modulated coherence factor\. To do so, we fit an encoding model that maps from the Fourier modes \(two per latent state forsin\\sinandcos\\cos\) of a latent to the residual stream, and compute the coherence parameters for these directions\. We also compute the total cosine similarity between this encoding subspace and the Jacobian lens singular vectors𝒗i\\bm\{v\}\_\{i\}\(input side\) and𝒖i\\bm\{u\}\_\{i\}\(output side\) \(Appendices[G](https://arxiv.org/html/2610.31955#A7)and[H](https://arxiv.org/html/2610.31955#A8)\)\. We also repeat this process on decoding vectors\. Finally, we show that we can predict the persistenceλ0∗\\lambda\_\{0\}^\{\*\}at which two latents swap places, given different angular velocitiesθ\\theta\.

### 4\.1Selectively modulating magnitude

Figure 4:The J\-lens captures the selective modulation of magnitudeaain the Bayes filter and transformers\.\(a\)Cartoon of the task feature that is modulated: a stateXt−1X\_\{t\-1\}on the ring is kept with probabilityλ\\lambda\(persistence\), and sampled uniformly otherwise\.\(b\)Learning curves for the transformer, in comparison to the optimal Bayes filter\.\(c\)J\-lens factors of each latent as a function of persistenceλ\\lambda: lens weight \(the singular valuea​ρτ​ρca\\,\\rho\_\{\\tau\}\\,\\rho\_\{c\}\) and magnitudeaa, both relative to their largest value, then lag coherenceρτ\\rho\_\{\\tau\}and context coherenceρc\\rho\_\{c\}\. Open circles: Bayes filter\. Blue circles and orange squares: transformer layers 0 and 1\. Large markers show the mean across seeds and small markers show single seeds\. Red line: theory \(Eq\.[19](https://arxiv.org/html/2610.31955#S3.E19)\)\.We first modulate the persistenceλ\\lambdaby designing a hidden Markov model \(HMM\) with three latent states, each on a ring, withλ=0\.8,0\.9,0\.95\\lambda=0\.8,0\.9,0\.95respectively\. We set the latent state velocityv=0v=0, which holdsρτ\\rho\_\{\\tau\}andρc\\rho\_\{c\}fixed while selectively modulating the magnitudeaaof each latent \(Fig\.[4](https://arxiv.org/html/2610.31955#S4.F4)a\)\. All models learn to predict the next token as well as the optimal Bayes filter \(Fig\.[4](https://arxiv.org/html/2610.31955#S4.F4)b\)\. On the Bayes filter, the measured J\-lens factors match the theory: the lens weight‖𝐉H​𝐱‖2\|\|\\mathbf\{J\}\_\{H\}\\mathbf\{x\}\|\|\_\{2\}tracks the magnitudeaa, whileρτ\\rho\_\{\\tau\}andρc\\rho\_\{c\}stay fixed \(Fig\.[4](https://arxiv.org/html/2610.31955#S4.F4)c\)\. The J\-lens applied to the transformer has similar factors at each layer\. As predicted, the J\-lens directions identify latents in order of their magnitudeaa\(Fig\.[13](https://arxiv.org/html/2610.31955#A10.F13)\)\. Across all seeds, this is true for both the input direction𝒗i\\bm\{v\}\_\{i\}\(the right singular vector of the J\-lens, which reads in the residual stream at layerℓ\\ell\) and the output direction𝒖i\\bm\{u\}\_\{i\}\(the left singular vector, which writes to the output\)\. The rank order predicted by the theory is validated on the Bayes filter and transformer \(Figs\.[11](https://arxiv.org/html/2610.31955#A10.F11)and[12](https://arxiv.org/html/2610.31955#A10.F12)\)\. The results are also consistent when utilizing decoding vectors for each latent, rather than encoding vectors \(Fig\.[14](https://arxiv.org/html/2610.31955#A10.F14)\), albeit with slightly smaller alignment values\.

### 4\.2Selectively modulating lag coherence

Figure 5:The J\-lens captures the selective modulation of lag coherenceρτ\\rho\_\{\\tau\}in the Bayes filter and transformers\.\(a\)The velocityvvof the state along the ring, with angular velocityθ=2​π​v/n\\theta=2\\pi v/n, modulates lag coherenceρτ\\rho\_\{\\tau\}\.\(b\)Learning curves of the transformer, as in Fig\.[4](https://arxiv.org/html/2610.31955#S4.F4)\.\(c\)J\-lens factors as a function of angular velocityθ\\theta, as in Fig\.[4](https://arxiv.org/html/2610.31955#S4.F4)\.Next, we selectively modulated the lag coherenceρτ\\rho\_\{\\tau\}by changing the velocityvvof each latent on the ring \(Fig\.[5](https://arxiv.org/html/2610.31955#S4.F5)a\)\. We placed states on a ring of 8 indices, and selected three latents again with velocitiesv=0v=0\(perfectly coherent\),22, and44\(maximally incoherent\), that is,θ=0,π/2,π\\theta=0,\\pi/2,\\pi\. Our theory again predicts that the J\-lens should identify these latents in order of their lag coherenceρτ\\rho\_\{\\tau\}\. As in the prior section, we found that for transformers trained to optimality \(Fig\.[5](https://arxiv.org/html/2610.31955#S4.F5)b\), this played out empirically across all seeds \(Fig\.[15](https://arxiv.org/html/2610.31955#A10.F15)\)\. The lag coherenceρτ\\rho\_\{\\tau\}of the J\-lens applied to the transformer on each latent separately was similarly modulated most by the velocity, whileaaandρc\\rho\_\{c\}were less sensitive \(Fig\.[5](https://arxiv.org/html/2610.31955#S4.F5)c\)\. Again, the results are also consistent when utilizing decoding vectors for each latent, rather than encoding vectors \(Fig\.[16](https://arxiv.org/html/2610.31955#A10.F16)\)\.

#### 4\.2\.1ω\\omega\-lens family

As described in Sec\.[1\.2](https://arxiv.org/html/2610.31955#S1.SS2), we also evaluated theω\\omega\-J\-lenses on the lag coherence sweep for both transformer layers and the Bayes filter\. We swept theω\\omegaparameter across the whole range, and found that it preferentially upweighted latent directions with lag coherence parameters \(tuned viaθ\\thetain the task\) that maximally aligned with theω\\omega\(Fig\.[6](https://arxiv.org/html/2610.31955#S4.F6)\)\. This suggests that theω\\omega\-J\-lens family can be used to selectively identify rapidly changing information across tokens \(like syntax\) or slowly changing information, by sweepingω\\omega\.

Figure 6:Theω\\omega\-J\-lens selectively upweights latent directions with lag coherences aligned withω\\omega\.Solid line indicates the theoretical curves \(optimally rescaled from the Bayes filter theory to fit the transformers\), points indicate real sampled data\. Colors \(denoting latent index\) are determined by alignment with latent encoding direction\.

### 4\.3Selectively modulating context coherence

Lastly, we selectively modulate context coherenceρc\\rho\_\{c\}by introducing a key state to latent states withn=2n=2states \(Sec\.[2\.1](https://arxiv.org/html/2610.31955#S2.SS1)\)\. In this HMM, an observed state is flipped to the opposite state with probability\(1−ψ\)/2\(1\-\\psi\)/2\. Holding all other parameters besidesψ\\psifixed, the context coherence can be modulated independently, sinceρc=ψ\\rho\_\{c\}=\\psifor the Bayes filter \(Appendix[F](https://arxiv.org/html/2610.31955#A6)\)\. We selectψi=0,0\.75,1\\psi\_\{i\}=0,0\.75,1, ranging from maximally incoherent \(0\) to coherent \(1\), and train a transformer on this 3\-latent HMM\. The J\-lens factors of the Bayes filter match the theory, withρc\\rho\_\{c\}trackingψ\\psi\(Fig\.[7](https://arxiv.org/html/2610.31955#S4.F7)\)\. The Bayes filter J\-lens identifies the latents in the predicted order, and the transformer does as well \(Fig\.[17](https://arxiv.org/html/2610.31955#A10.F17)\)\.

This lens is noisier than the previous analyses, potentially due to the increase in noise in the observed states added by the key variable\. In both layers, several J\-lens directions also align with patterns other than the latent belief states, mainly the reveal stateBtB\_\{t\}\(whether or not the latent states are observed\) of the present token and the key stateWt−1W\_\{t\-1\}\(Fig\.[19](https://arxiv.org/html/2610.31955#A10.F19)\)\. This suggests that the J\-lens can identify other important features in the latent space in an unsupervised fashion\. The lens factors \(Fig\.[7](https://arxiv.org/html/2610.31955#S4.F7)c\) show that both transformer layers compress the range of context coherence:ρc\\rho\_\{c\}is much higher than in the Bayes filter for the incoherent latent \(ψ=0\\psi=0\) and lower for the coherent ones\. The highρc\\rho\_\{c\}atψ=0\\psi=0suggests that transformers may have a bias for extracting context\-independent information from data\. Finally, the results remain consistent when utilizing decoding vectors for each latent, rather than encoding vectors \(Fig\.[18](https://arxiv.org/html/2610.31955#A10.F18)\)\.

Figure 7:The J\-lens captures the selective modulation of context coherenceρc\\rho\_\{c\}in the Bayes filter and transformers\.\(a\)For keyed features, the true state is revealed in the token with probability\(1\+ψ\)/2\(1\+\\psi\)/2\. Otherwise, the state is revealed inverted, so thatρc=ψ\\rho\_\{c\}=\\psi\.\(b\)Learning curves of the transformer, as in Fig\.[4](https://arxiv.org/html/2610.31955#S4.F4)\.\(c\)J\-lens factors as a function of key biasψ\\psi, as in Fig\.[4](https://arxiv.org/html/2610.31955#S4.F4)\.Figure 8:Predictive theory of crossing of latents in the J\-lens\.\(a\)Empirical singular valuesσ\\sigma\(circles\) and analytically derived values in the large\-HHlimit \(line\) for the J\-lens applied to a Bayes filter with two latents \(L0, L1\)\.λ0\\lambda\_\{0\}is swept for L0, whileλ1\\lambda\_\{1\}is held fixed for L1\. Vertical dotted line denotes the predicted crossing pointλ0∗\\lambda\_\{0\}^\{\*\}\.\(b\)Predicted crossing pointλ0∗\\lambda\_\{0\}^\{\*\}\(Eq\. \([46](https://arxiv.org/html/2610.31955#A5.E46)\)\) and empirical crossing point \(circles\) as a function of the angular velocityθ1\\theta\_\{1\}of L1\.\(c\)Difference in singular valuesσ0−σ1\\sigma\_\{0\}\-\\sigma\_\{1\}as a function ofλ0−λ0∗\\lambda\_\{0\}\-\\lambda\_\{0\}^\{\*\}, for the transformer trained on the same HMM as in \(a\)\. Vertical solid line denotes the mean crossing point across seeds, gray shading denotes standard deviation\. Each curve is averaged over44variations ofθ1\\theta\_\{1\}\(circles in \(b\)\)\.
### 4\.4Theory predicts the crossing of two latents

To further emphasize that our theory can predict the ordering of features in the J\-lens, we show that our theory can predict when two latents are swapped in the Bayes filter, and that this predicted crossing point roughly translates to the transformer\. We simulate an HMM with no keys here for simplicity\. For two latents with angular velocitiesθ0=0\\theta\_\{0\}=0andθ1\\theta\_\{1\}, our theory predicts a persistence valueλ0∗\\lambda\_\{0\}^\{\*\}for a givenλ1\\lambda\_\{1\}andθ1\\theta\_\{1\}, at which two latents swap order in the J\-lens \(Appendix[E\.1](https://arxiv.org/html/2610.31955#A5.SS1.SSS0.Px4), Eq\. \([46](https://arxiv.org/html/2610.31955#A5.E46)\)\)\. On the Bayes filter, this theory plays out extremely cleanly \(Fig\.[8](https://arxiv.org/html/2610.31955#S4.F8)a,b\) across a range ofλ1\\lambda\_\{1\}andθ1\\theta\_\{1\}values, with the theory exactly predicting the crossing pointλ0∗\\lambda\_\{0\}^\{\*\}of two latents\. Applied to the transformer, our results generalize with small bias \(forλ∈\[0,1\]\\lambda\\in\[0,1\], as the difference in singular valuesσ0−σ1\\sigma\_\{0\}\-\\sigma\_\{1\}grows monotonically withλ0\\lambda\_\{0\}, and the error isO⁡\(0\.01\)O\(0\.01\)\) and in opposite directions at each layer \(Fig\.[8](https://arxiv.org/html/2610.31955#S4.F8)c\)\. Together, this suggests that even though the Bayes filter theory does not perfectly translate to the transformer, as expected, the predictions are still quite informative\.

## 5Discussion

We have shown that the Jacobian lens, as introduced by[Gurnee et al\. \(2026\)](https://arxiv.org/html/2610.31955#bib.bib1), identifies modes in any temporally and spatially varying dynamical system that jointly maximize three different statistical features:

##### Magnitudeaa\.

Magnitude measures how a state grows or shrinks across a ‘temporal’ dimension\. In transformers, this growth is across layers, with depth exponentχ\\chi\. However, this could be defined differently in other systems\. Magnitude aligns most directly with the standard notion of a Lyapunov exponent, which typically measures growth across time in a dynamical system\.

##### Lag coherenceρτ\\rho\_\{\\tau\}\.

Lag coherence measures how a feature is modulated across ‘spatial’ lagsτ\\tau\. In transformers, this is measured across token positions, where a mode decays at rateμ\\muand rotates at frequencyθ\\theta\. Cross\-token dynamics in language models have previously been studied as multi\-particle systems\([Lu et al\., 2019](https://arxiv.org/html/2610.31955#bib.bib15);[Geshkovski et al\., 2023](https://arxiv.org/html/2610.31955#bib.bib16)\), in which each token \(particle\) has a particular spatial position and interacts with others via self\-attention\. The most coherent spatial modes are those with zero spatial frequency and little decay \(θ,μ≈0\\theta,\\mu\\approx 0\)\. In Lyapunov theory, coherent directions relate to the measurement of hydrodynamic Lyapunov modes, which are exactly the spatially coherent Lyapunov vectors\.

##### Context coherenceρc\\rho\_\{c\}\.

Context coherence measures how much the dynamics are modulated across different states or time periods\. Roughly, maximizing context coherence identifies modes that areautonomous\(ρc≈1\\rho\_\{c\}\\approx 1\), which is to sayindependent with respect to the context\. The transformer architecture is highly context\-dependent, whereas autonomous dynamical systems are not\. That the transformer learned features with much higherρc\\rho\_\{c\}than the Bayes filter for the context\-incoherent latent \(Fig\.[7](https://arxiv.org/html/2610.31955#S4.F7)c\) suggests that transformers might learn to modulate contextual information via context\-independent dynamics, which could be a fundamental mechanism for symbolic processing\. This further suggests that it may be viable to study transformers more generally as dynamical systems\([Ostrow et al\., 2024](https://arxiv.org/html/2610.31955#bib.bib18)\)just like recurrent neural networks\([Sussillo and Barak, 2013](https://arxiv.org/html/2610.31955#bib.bib17)\)\.

Our analysis explains a number of phenomena in the original J\-lens paper\([Gurnee et al\., 2026](https://arxiv.org/html/2610.31955#bib.bib1)\)\. As described previously, high\-magnitude, lag\-coherent, and context\-coherent directions do not have to be large variance \(Corollary[1\.1](https://arxiv.org/html/2610.31955#S1.Thmtheorem1)\)\. This is commensurate with[Gurnee et al\. \(2026\)](https://arxiv.org/html/2610.31955#bib.bib1)finding that the J\-space is never more than 10% of activation variance\. Lag coherence preference explains why, when Claude was asked to ‘concentrate on X while solving Y’\([Gurnee et al\., 2026](https://arxiv.org/html/2610.31955#bib.bib1)\), the J\-lens identified features corresponding to X, as these features would be prevalent and coherent across all token positions\. A preference for contextually coherent directions explains why averaging identifies concepts that are broadly verbalizable, not specific to a single prompt\.

Our work aligns with other results in mechanistic interpretability\([Shai et al\., 2024](https://arxiv.org/html/2610.31955#bib.bib3)\)that study how toy transformers represent belief state geometry in their latent state and learn to decompose independent factors of the generative process into orthogonal subspaces\([Shai et al\., 2026](https://arxiv.org/html/2610.31955#bib.bib2)\)\. However, this work has generally relied on linear decoding of these belief states, which requiresa prioriknowledge of the optimal belief states at each timepoint\. Our results here suggest that the J\-lens could be a way to identify these belief states in an unsupervised manner, which would be effective for scaling to real\-world behaviors\.

This analysis also suggests that a family of Jacobian lenses could be designed to capture different types of information within transformers\. For example, local information could be emphasized by averaging over high\-frequency Fourier modes, rather than low\-frequency, as we showed in Fig\.[6](https://arxiv.org/html/2610.31955#S4.F6)\. This could be relevant for isolating and understanding fast\-changing information, such as those on the level of syntax that change word\-by\-word\. Contextually incoherent information could be isolated by identifying Jacobian modes that minimize context coherenceρc\\rho\_\{c\}, rather than maximize it\. Given recent work on estimating Jacobians from dynamical data\([Eisen et al\., 2026](https://arxiv.org/html/2610.31955#bib.bib10)\), theω\\omega\-J\-lenses could productively be applied to biological neural data as well\. Together, our results clarify the mathematical nature of the J\-lens, which is necessary for such a useful and efficient interpretability tool\.

##### Limitations\.

Here, we only study toy models, in which the task was relatively easy to learn and the coherence parameters of the task were easily manipulable\.

## References

- Baars \(1988\)B\. J\. BaarsA cognitive theory of consciousness\.Cambridge University Press\.Cited by:[§1](https://arxiv.org/html/2610.31955#S1.p1.1)\.
- Belroseet al\.\(2023\)N\. Belrose, I\. Ostrovsky, L\. McKinney, Z\. Furman, L\. Smith, D\. Halawi, S\. Biderman, and J\. SteinhardtEliciting latent predictions from transformers with the tuned lens\.arXiv preprint arXiv:2303\.08112\.Cited by:[§1](https://arxiv.org/html/2610.31955#S1.p1.1)\.
- Benettinet al\.\(1980\)G\. Benettin, L\. Galgani, A\. Giorgilli, and J\. StrelcynLyapunov Characteristic Exponents for smooth dynamical systems and for hamiltonian systems; a method for computing all of them\. Part 1: Theory\.Meccanica15\(1\),pp\. 9–20\.External Links:ISSN 1572\-9648,[Link](https://doi.org/10.1007/BF02128236),[Document](https://dx.doi.org/10.1007/BF02128236)Cited by:[§1](https://arxiv.org/html/2610.31955#S1.SS0.SSS0.Px2.p1.3)\.
- Chalkidiset al\.\(2022\)I\. Chalkidis, A\. Jana, D\. Hartung, M\. Bommarito, I\. Androutsopoulos, D\. M\. Katz, and N\. AletrasLexGLUE: a benchmark dataset for legal language understanding in English\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics,pp\. 4310–4330\.Cited by:[10th item](https://arxiv.org/html/2610.31955#A2.I1.i10.p1.1)\.
- Chenet al\.\(2021\)Y\. Chen, Y\. Liu, L\. Chen, and Y\. ZhangDialogSum: a real\-life scenario dialogue summarization dataset\.InFindings of the Association for Computational Linguistics: ACL\-IJCNLP 2021,pp\. 5062–5074\.Cited by:[7th item](https://arxiv.org/html/2610.31955#A2.I1.i7.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[6th item](https://arxiv.org/html/2610.31955#A2.I1.i6.p1.1)\.
- Dehaeneet al\.\(1998\)S\. Dehaene, M\. Kerszberg, and J\. ChangeuxA neuronal model of a global workspace in effortful cognitive tasks\.Proceedings of the National Academy of Sciences95\(24\),pp\. 14529–14534\.Cited by:[§1](https://arxiv.org/html/2610.31955#S1.p1.1)\.
- Deissler and Kaneko \(1987\)R\. J\. Deissler and K\. KanekoVelocity\-dependent Lyapunov exponents as a measure of chaos for open\-flow systems\.Physics Letters A119\(8\),pp\. 397–402\.Cited by:[§1](https://arxiv.org/html/2610.31955#S1.SS0.SSS0.Px2.p1.3)\.
- Dettmerset al\.\(2022\)T\. Dettmers, M\. Lewis, Y\. Belkada, and L\. ZettlemoyerLLM\.int8\(\): 8\-bit matrix multiplication for transformers at scale\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§1](https://arxiv.org/html/2610.31955#S1.p3.1)\.
- Eckmann and Gat \(2000\)J\. Eckmann and O\. GatHydrodynamic Lyapunov modes in translation\-invariant systems\.Journal of Statistical Physics98\(3\),pp\. 775–798\.Cited by:[§1](https://arxiv.org/html/2610.31955#S1.SS0.SSS0.Px2.p1.3)\.
- Eckmann and Ruelle \(1985\)J\. Eckmann and D\. RuelleErgodic theory of chaos and strange attractors\.Reviews of Modern Physics57\(3\),pp\. 617–656\.Cited by:[§1](https://arxiv.org/html/2610.31955#S1.SS0.SSS0.Px2.p1.1)\.
- Eisenet al\.\(2026\)A\. J\. Eisen, M\. Ostrow, S\. Chandra, L\. Kozachkov, E\. Miller, and I\. FieteCharacterizing control between interacting subsystems with deep jacobian estimation\.Advances in Neural Information Processing Systems38,pp\. 51361–51406\.Cited by:[§5](https://arxiv.org/html/2610.31955#S5.SS0.SSS0.Px3.p4.1)\.
- Engelkenet al\.\(2023\)R\. Engelken, F\. Wolf, and L\. F\. AbbottLyapunov spectra of chaotic recurrent neural networks\.Phys\. Rev\. Res\.5,pp\. 043044\.External Links:[Document](https://dx.doi.org/10.1103/PhysRevResearch.5.043044),[Link](https://link.aps.org/doi/10.1103/PhysRevResearch.5.043044)Cited by:[§1](https://arxiv.org/html/2610.31955#S1.SS0.SSS0.Px2.p1.3)\.
- Geshkovskiet al\.\(2023\)B\. Geshkovski, C\. Letrouit, Y\. Polyanskiy, and P\. RigolletA mathematical perspective on Transformers\.arXiv preprint arXiv:2312\.10794\.Cited by:[§5](https://arxiv.org/html/2610.31955#S5.SS0.SSS0.Px2.p1.1)\.
- Gurneeet al\.\(2026\)W\. Gurnee, N\. Sofroniew, A\. Pearce, M\. Piotrowski, I\. Kauvar, R\. Chen, A\. Soligo, P\. Bogdan, E\. Ong, R\. Wang, B\. Thompson, D\. Abrahams, S\. Kantamneni, E\. Ameisen, J\. Batson, and J\. LindseyVerbalizable representations form a global workspace in language models\.arXiv preprint arXiv:2607\.15495\.Cited by:[§1](https://arxiv.org/html/2610.31955#S1.SS0.SSS0.Px6.p3.2),[§1\.1](https://arxiv.org/html/2610.31955#S1.SS1.p1.2),[§1](https://arxiv.org/html/2610.31955#S1.p1.1),[§1](https://arxiv.org/html/2610.31955#S1.p2.1),[§5](https://arxiv.org/html/2610.31955#S5.SS0.SSS0.Px3.p2.1),[§5](https://arxiv.org/html/2610.31955#S5.p1.1),[Abstract](https://arxiv.org/html/2610.31955#abstract1.1)\.
- Halawiet al\.\(2024\)D\. Halawi, J\. Denain, and J\. SteinhardtOverthinking the truth: understanding how language models process false demonstrations\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2610.31955#S1.p1.1)\.
- Hermannet al\.\(2015\)K\. M\. Hermann, T\. Kočiský, E\. Grefenstette, L\. Espeholt, W\. Kay, M\. Suleyman, and P\. BlunsomTeaching machines to read and comprehend\.InAdvances in Neural Information Processing Systems,Vol\.28\.Cited by:[8th item](https://arxiv.org/html/2610.31955#A2.I1.i8.p1.1)\.
- Jiotsaet al\.\(2013\)K\. Jiotsa, A\. Jiotsa, A\. Politi, and A\. TorciniConvective Lyapunov spectra\.Journal of Physics A: Mathematical and Theoretical46\(25\),pp\. 254013\.External Links:[Document](https://dx.doi.org/10.1088/1751-8113/46/25/254013)Cited by:[§1](https://arxiv.org/html/2610.31955#S1.SS0.SSS0.Px2.p1.3)\.
- Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.InInternational Conference on Learning Representations,Cited by:[Appendix F](https://arxiv.org/html/2610.31955#A6.p3.1)\.
- Luet al\.\(2021\)S\. Lu, D\. Guo, S\. Ren, J\. Huang, A\. Svyatkovskiy, A\. Blanco, C\. Clement, D\. Drain, D\. Jiang, D\. Tang, G\. Li, L\. Zhou, L\. Shou, L\. Zhou, M\. Tufano, M\. Gong, M\. Zhou, N\. Duan, N\. Sundaresan, S\. K\. Deng, S\. Fu, and S\. LiuCodeXGLUE: a machine learning benchmark dataset for code understanding and generation\.InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks,Cited by:[3rd item](https://arxiv.org/html/2610.31955#A2.I1.i3.p1.1)\.
- Luet al\.\(2019\)Y\. Lu, Z\. Li, D\. He, Z\. Sun, B\. Dong, T\. Qin, L\. Wang, and T\. LiuUnderstanding and improving Transformer from a multi\-particle dynamic system point of view\.arXiv preprint arXiv:1906\.02762\.Cited by:[§5](https://arxiv.org/html/2610.31955#S5.SS0.SSS0.Px2.p1.1)\.
- Meyeret al\.\(2024\)Y\. Meyer, M\. Emadi, D\. Nathawani, L\. Ramaswamy, K\. Boyd, M\. Van Segbroeck, M\. Grossman, P\. Mlocek, and D\. NewberrySynthetic\-text\-to\-SQL: a synthetic dataset for training language models to generate SQL queries from natural language prompts\.Note:[https://huggingface\.co/datasets/gretelai/synthetic\_text\_to\_sql](https://huggingface.co/datasets/gretelai/synthetic_text_to_sql)Cited by:[4th item](https://arxiv.org/html/2610.31955#A2.I1.i4.p1.1)\.
- nostalgebraist \(2020\)nostalgebraistInterpreting GPT: the logit lens\.Note:LessWrong,[https://www\.lesswrong\.com/posts/AcKRB8wDpdaN6v6ru/interpreting\-gpt\-the\-logit\-lens](https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens)Cited by:[§1](https://arxiv.org/html/2610.31955#S1.p1.1)\.
- Oseledets \(1968\)V\. I\. OseledetsA multiplicative ergodic theorem\. Lyapunov characteristic numbers for dynamical systems\.Transactions of the Moscow Mathematical Society19,pp\. 197–231\.Cited by:[§1](https://arxiv.org/html/2610.31955#S1.SS0.SSS0.Px2.p1.1)\.
- Ostrowet al\.\(2024\)M\. Ostrow, A\. Eisen, and I\. FieteDelay embedding theory of neural sequence models\.arXiv preprint arXiv:2406\.11993\.Cited by:[§5](https://arxiv.org/html/2610.31955#S5.SS0.SSS0.Px3.p1.1)\.
- Palet al\.\(2023\)K\. Pal, J\. Sun, A\. Yuan, B\. C\. Wallace, and D\. BauFuture lens: anticipating subsequent tokens from a single hidden state\.InProceedings of the 27th Conference on Computational Natural Language Learning \(CoNLL\),External Links:[Document](https://dx.doi.org/10.18653/v1/2023.conll-1.37)Cited by:[§1](https://arxiv.org/html/2610.31955#S1.p1.1)\.
- Pasteret al\.\(2024\)K\. Paster, M\. Dos Santos, Z\. Azerbayev, and J\. BaOpenWebMath: an open dataset of high\-quality mathematical web text\.InInternational Conference on Learning Representations,Cited by:[5th item](https://arxiv.org/html/2610.31955#A2.I1.i5.p1.1)\.
- Seeet al\.\(2017\)A\. See, P\. J\. Liu, and C\. D\. ManningGet to the point: summarization with pointer\-generator networks\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics,pp\. 1073–1083\.Cited by:[8th item](https://arxiv.org/html/2610.31955#A2.I1.i8.p1.1)\.
- Shaiet al\.\(2026\)A\. Shai, L\. Amdahl\-Culleton, C\. L\. Christensen, H\. R\. Bigelow, F\. Rosas, A\. Boyd, E\. A\. Alt, K\. J\. Ray, and P\. M\. RiechersTransformers learn factored representations\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=flDGB9On0h)Cited by:[§5](https://arxiv.org/html/2610.31955#S5.SS0.SSS0.Px3.p3.1)\.
- Shaiet al\.\(2024\)A\. S\. Shai, S\. E\. Marzen, L\. Teixeira, A\. G\. Oldenziel, and P\. M\. RiechersTransformers represent belief state geometry in their residual stream\.Advances in Neural Information Processing Systems37,pp\. 75012–75034\.Cited by:[§2](https://arxiv.org/html/2610.31955#S2.p1.1),[§3](https://arxiv.org/html/2610.31955#S3.p1.1),[§5](https://arxiv.org/html/2610.31955#S5.SS0.SSS0.Px3.p3.1)\.
- Stein \(1981\)C\. M\. SteinEstimation of the mean of a multivariate normal distribution\.The Annals of Statistics9\(6\),pp\. 1135–1151\.Cited by:[§1](https://arxiv.org/html/2610.31955#S1.p3.1)\.
- Suet al\.\(2024\)J\. Su, M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. LiuRoFormer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.Cited by:[Appendix F](https://arxiv.org/html/2610.31955#A6.p1.1)\.
- Sunet al\.\(2024\)M\. Sun, X\. Chen, J\. Z\. Kolter, and Z\. LiuMassive activations in large language models\.InConference on Language Modeling \(COLM\),Cited by:[§1](https://arxiv.org/html/2610.31955#S1.p3.1)\.
- Sussillo and Barak \(2013\)D\. Sussillo and O\. BarakOpening the black box: low\-dimensional dynamics in high\-dimensional recurrent neural networks\.Neural Computation25\(3\),pp\. 626–649\.Cited by:[§5](https://arxiv.org/html/2610.31955#S5.SS0.SSS0.Px3.p1.1)\.
- Tuggeneret al\.\(2020\)D\. Tuggener, P\. von Däniken, T\. Peetz, and M\. CieliebakLEDGAR: a large\-scale multi\-label corpus for text classification of legal provisions in contracts\.InProceedings of the Twelfth Language Resources and Evaluation Conference,pp\. 1235–1241\.Cited by:[10th item](https://arxiv.org/html/2610.31955#A2.I1.i10.p1.1)\.
- Tunstallet al\.\(2022\)L\. Tunstall, L\. von Werra, and T\. WolfNatural language processing with transformers\.O’Reilly Media\.Cited by:[2nd item](https://arxiv.org/html/2610.31955#A2.I1.i2.p1.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§4](https://arxiv.org/html/2610.31955#S4.p1.1)\.
- Wendleret al\.\(2024\)C\. Wendler, V\. Veselovsky, G\. Monea, and R\. WestDo llamas work in English? on the latent language of multilingual transformers\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§1](https://arxiv.org/html/2610.31955#S1.p1.1)\.
- Wikimedia Foundation \(2023\)Wikimedia FoundationWikimedia downloads: wikipedia dump of 2023\-11\-01\.Note:[https://dumps\.wikimedia\.org](https://dumps.wikimedia.org/)Cited by:[1st item](https://arxiv.org/html/2610.31955#A2.I1.i1.p1.1)\.
- Yanet al\.\(2026\)S\. Yan, K\. Ding, C\. Tan, Q\. Chen, W\. Wang, X\. Li, and Z\. LingShort horizons and sparse concepts: a mathematical view of the readout in the J\-lens\.arXiv preprint arXiv:2608\.25347\.Cited by:[§1](https://arxiv.org/html/2610.31955#S1.p3.1)\.

## Appendix AJacobian singular values under averaging

FixU,VU,Vand singular valuessj=e−β​js\_\{j\}=e^\{\-\\beta j\}forj=1,…,dj=1,\\dots,d, withS=diag⁡\(s1,…,sd\)S=\\mathrm\{diag\}\(s\_\{1\},\\dots,s\_\{d\}\)\. For each samplei=1,…,Ni=1,\\dots,N, draw a random permutationPiP\_\{i\}and set

Xi=U​Pi​S​Pi⊤​V⊤\.X\_\{i\}\\;=\\;UP\_\{i\}SP\_\{i\}^\{\\top\}V^\{\\top\}\.\(20\)EveryXiX\_\{i\}has the same singular valuese−β,e−2​β,…,e−β​de^\{\-\\beta\},e^\{\-2\\beta\},\\dots,e^\{\-\\beta d\}\. SinceU,VU,Vare shared,

X¯=1N​∑iXi=U⁡\(1N​∑iPi​S​Pi⊤\)​V⊤=U​S¯​V⊤,\\bar\{X\}\\;=\\;\\frac\{1\}\{N\}\\sum\_\{i\}X\_\{i\}\\;=\\;U\\Big\(\\frac\{1\}\{N\}\\sum\_\{i\}P\_\{i\}SP\_\{i\}^\{\\top\}\\Big\)V^\{\\top\}\\;=\\;U\\bar\{S\}V^\{\\top\},\(21\)so the singular values ofX¯\\bar\{X\}are the diagonal entries ofS¯\\bar\{S\},

S¯k​k=1N​∑isπi​\(k\)\.\\bar\{S\}\_\{kk\}\\;=\\;\\frac\{1\}\{N\}\\sum\_\{i\}s\_\{\\pi\_\{i\}\(k\)\}\.\(22\)Eachsπi​\(k\)s\_\{\\pi\_\{i\}\(k\)\}is a uniform draw from\{s1,…,sd\}\\\{s\_\{1\},\\dots,s\_\{d\}\\\}, so

𝔼​S¯k​k=s¯=1d​∑j=1de−β​j,Var​S¯k​k=Var⁡\(s\)N→N→∞0\.\\mathbb\{E\}\\,\\bar\{S\}\_\{kk\}\\;=\\;\\bar\{s\}\\;=\\;\\frac\{1\}\{d\}\\sum\_\{j=1\}^\{d\}e^\{\-\\beta j\},\\qquad\\mathrm\{Var\}\\,\\bar\{S\}\_\{kk\}\\;=\\;\\frac\{\\mathrm\{Var\}\(s\)\}\{N\}\\;\\xrightarrow\{N\\to\\infty\}\\;0\.\(23\)Every singular value ofX¯\\bar\{X\}therefore converges to the same values¯\\bar\{s\}\. EachXiX\_\{i\}has the decaying spectrume−β​ke^\{\-\\beta k\}, but the meanX¯\\bar\{X\}has a flat spectrum:X¯→s¯​U​V⊤\\bar\{X\}\\to\\bar\{s\}\\,UV^\{\\top\}\.

## Appendix BReal Jacobians of LLMs

For each model, we computed per\-token Jacobians of the residual stream on 96 text prompts drawn from 16 public datasets, six prompts each, all from Hugging Face:

- •Wikipedia, 2023\-11\-01 dump, in English, Russian, Arabic, Chinese, Japanese and Swahili \(wikimedia/wikipedia;[Wikimedia Foundation, 2023](https://arxiv.org/html/2610.31955#bib.bib30)\);
- •Python code from CodeParrot\-clean \(codeparrot/codeparrot\-clean;[Tunstall et al\., 2022](https://arxiv.org/html/2610.31955#bib.bib31)\);
- •JavaScript functions from the CodeXGLUE code\-to\-text benchmark \(google/code\_x\_glue\_ct\_code\_to\_text;[Lu et al\., 2021](https://arxiv.org/html/2610.31955#bib.bib32)\);
- •SQL schemas with queries from Gretel’s synthetic text\-to\-SQL dataset \(gretelai/synthetic\_text\_to\_sql;[Meyer et al\., 2024](https://arxiv.org/html/2610.31955#bib.bib33)\);
- •mathematical web text from OpenWebMath \(open\-web\-math/open\-web\-math;[Paster et al\., 2024](https://arxiv.org/html/2610.31955#bib.bib34)\);
- •grade\-school math word problems with worked solutions from GSM8K \(openai/gsm8k;[Cobbe et al\., 2021](https://arxiv.org/html/2610.31955#bib.bib35)\);
- •dialogues from DialogSum \(knkarthick/dialogsum;[Chen et al\., 2021](https://arxiv.org/html/2610.31955#bib.bib36)\);
- •news articles from CNN/DailyMail, version 3\.0\.0 \(abisee/cnn\_dailymail;[Hermann et al\., 2015](https://arxiv.org/html/2610.31955#bib.bib37);[See et al\., 2017](https://arxiv.org/html/2610.31955#bib.bib38)\);
- •
- •contract clauses from LEDGAR, via LexGLUE \(coastalcph/lex\_glue;[Tuggener et al\., 2020](https://arxiv.org/html/2610.31955#bib.bib39);[Chalkidis et al\., 2022](https://arxiv.org/html/2610.31955#bib.bib40)\);
- •

Prompts were drawn at random, with a fixed seed, from the first 200 training examples of each dataset\. Only texts of at least 400 characters were kept, and each was truncated to its first 57 tokens\. We computed the exact Jacobian from the middle block of all models to the last block, one token ahead:

Jc,t=∂hL−1\(c\)​\(t\+1\)∂hℓ\(c\)​\(t\)∈ℝd×d\.J\_\{c,t\}\\;=\\;\\frac\{\\partial\\,h\_\{L\-1\}^\{\(c\)\}\(t\+1\)\}\{\\partial\\,h\_\{\\ell\}^\{\(c\)\}\(t\)\}\\;\\in\\;\\mathbb\{R\}^\{d\\times d\}\.\(24\)Jacobians were computed by reverse\-mode differentiation, with float32 weights and eager attention, givingN=96×8=768N=96\\times 8=768matrices per model\. The mean Jacobian is their element\-wise average,

J¯=1N​∑c,tJc,t\.\\bar\{J\}\\;=\\;\\frac\{1\}\{N\}\\sum\_\{c,t\}J\_\{c,t\}\.\(25\)whereccdenotes context andttdenotes position in the text\. To visualize the distribution of these matrices, we computed each Jacobian’s top 100 singular values, suppressing the rest as they may be noisy\. We rescaled the singular value distribution by taking a logarithm, so that lower values can contribute in dimensionality reduction, and normalized the distribution for all Jacobians\. We then fit and applied PCA to the individual Jacobians, then projected the mean Jacobian into this space\.

Figure 9:Jacobian singular value spectra of four LLMs, with estimated power laws\.Individual Jacobian curve is averaged over all points in Fig\.[1](https://arxiv.org/html/2610.31955#S1.F1), with shading denoting standard deviation\.Figure 10:Low\-dimensional projection of the left and right singular vectors of Jacobians\.The top 100 left and right singular vectors, ranked by singular values as in Fig\.[1](https://arxiv.org/html/2610.31955#S1.F1), are projected into two dimensions by computing the subspace distance on Grassmannians‖Pi−Pj‖F=2​∑ksin2⁡θk\|\|P\_\{i\}\-P\_\{j\}\|\|\_\{F\}=\\sqrt\{2\\sum\_\{k\}\\sin^\{2\}\\theta\_\{k\}\}, whereθk\\theta\_\{k\}are the principal subspace angles andPiP\_\{i\}is the projector matrix defined by the singular vectors,Pi=Ui​UiTP\_\{i\}=U\_\{i\}U\_\{i\}^\{T\}\. This pairwise distance matrix is reduced to two dimensions and visualized with PCA\. \(a\) Left singular vectors\. \(b\) Right singular vectors\.
## Appendix CThe shared\-mode model

This appendix provides the averaging steps and assumptions behind the shared\-mode model of Eq\. \([5](https://arxiv.org/html/2610.31955#S1.E5)\)\.

##### Averaging\.

The shared component has singular strengthseT​χi\+τ​μie^\{T\\chi\_\{i\}\+\\tau\\mu\_\{i\}\}\. Suppressingℓ\\ellin the notation, averaging gives

𝑱H\\displaystyle\\bm\{J\}\_\{H\}=𝑺H\+𝔼c​𝑹c,\\displaystyle=\\bm\{S\}\_\{H\}\+\\mathbb\{E\}\_\{c\}\\bm\{R\}\_\{c\},𝑺H\\displaystyle\\bm\{S\}\_\{H\}=∑igi​𝒖i​𝒗i∗,\\displaystyle=\\sum\_\{i\}g\_\{i\}\\bm\{u\}\_\{i\}\\bm\{v\}\_\{i\}^\{\*\},\(26\)𝑹c\\displaystyle\\bm\{R\}\_\{c\}=1H​∑τ=0H−1𝑹⁡\(c,ℓ,τ\),\\displaystyle=\\frac\{1\}\{H\}\\sum\_\{\\tau=0\}^\{H\-1\}\\bm\{R\}\(c,\\ell,\\tau\),gi\\displaystyle g\_\{i\}=eT​χi​fH​\(γi\),\\displaystyle=e^\{T\\chi\_\{i\}\}f\_\{H\}\(\\gamma\_\{i\}\),fH​\(γ\)\\displaystyle f\_\{H\}\(\\gamma\)=1H​∑τ=0H−1eτ​γ=\{\(1−eH​γ\)/\(H⁡\(1−eγ\)\),eγ≠1,1,eγ=1\.\\displaystyle=\\frac\{1\}\{H\}\\sum\_\{\\tau=0\}^\{H\-1\}e^\{\\tau\\gamma\}=\\begin\{cases\}\(1\-e^\{H\\gamma\}\)/\(H\(1\-e^\{\\gamma\}\)\),&e^\{\\gamma\}\\neq 1,\\\\ 1,&e^\{\\gamma\}=1\.\\end\{cases\}The shared singular strengths after averaging are\|gi\|\|g\_\{i\}\|\. Assume bounded accumulated depth growth and non\-growing lag dynamics:

Tχi≤Γ,μi≤0,‖𝑺H‖2=M≤eΓ,𝔼c‖𝑹c‖22≤M2\.\\begin\{gathered\}T\\chi\_\{i\}\\leq\\Gamma,\\qquad\\mu\_\{i\}\\leq 0,\\\\ \\left\\lVert\\bm\{S\}\_\{H\}\\right\\rVert\_\{2\}=M\\leq e^\{\\Gamma\},\\qquad\\mathbb\{E\}\_\{c\}\\left\\lVert\\bm\{R\}\_\{c\}\\right\\rVert\_\{2\}^\{2\}\\leq M^\{2\}\.\\end\{gathered\}\(27\)These assumptions prevent the context\-specific response from having arbitrarily greater magnitude than the shared response scale\. Spectral dominance additionally requires shared strengths above the averaged remainder, as shown in Eq\. \([8](https://arxiv.org/html/2610.31955#S1.E8)\)\.

##### Overlap of context\-specific responses\.

For a unit input perturbation𝒗\\bm\{v\}and independent context drawsc,c′c,c^\{\\prime\},

‖\(𝔼c​𝑹c\)​𝒗‖22=𝒗∗​𝔼c,c′​\[𝑹c∗​𝑹c′\]​𝒗=𝔼c,c′​⟨𝑹c​𝒗,𝑹c′​𝒗⟩\.\\displaystyle\\left\\lVert\(\\mathbb\{E\}\_\{c\}\\bm\{R\}\_\{c\}\)\\bm\{v\}\\right\\rVert\_\{2\}^\{2\}=\\bm\{v\}^\{\*\}\\mathbb\{E\}\_\{c,c^\{\\prime\}\}\[\\bm\{R\}\_\{c\}^\{\*\}\\bm\{R\}\_\{c^\{\\prime\}\}\]\\bm\{v\}=\\mathbb\{E\}\_\{c,c^\{\\prime\}\}\\langle\\bm\{R\}\_\{c\}\\bm\{v\},\\bm\{R\}\_\{c^\{\\prime\}\}\\bm\{v\}\\rangle\.\(28\)Here⟨𝑨,𝑩⟩F=tr⁡\(𝑨∗​𝑩\)\\langle\\bm\{A\},\\bm\{B\}\\rangle\_\{F\}=\\operatorname\{tr\}\(\\bm\{A\}^\{\*\}\\bm\{B\}\)and∗\*denotes conjugate transpose\. Thus growth is the expected Frobenius overlap of the Jacobians restricted to the same input direction\. The unprojected overlap⟨𝑹c,𝑹c′⟩F\\langle\\bm\{R\}\_\{c\},\\bm\{R\}\_\{c^\{\\prime\}\}\\rangle\_\{F\}sums over input directions instead\.

Write𝑹c=∑jsc​jR​𝒖c​jR​\(𝒗c​jR\)∗\\bm\{R\}\_\{c\}=\\sum\_\{j\}s^\{R\}\_\{cj\}\\bm\{u\}^\{R\}\_\{cj\}\(\\bm\{v\}^\{R\}\_\{cj\}\)^\{\*\}, usings,𝒖,𝒗s,\\bm\{u\},\\bm\{v\}for singular values, output directions, and input directions\. Then

⟨𝑹c​𝒗,𝑹c′​𝒗⟩=∑j,ksc​jR​sc′​kR⏟magnitude​\(𝒗∗​𝒗c​jR\)​\(\(𝒗c′​kR\)∗​𝒗\)⏟input alignment​\(\(𝒖c​jR\)∗​𝒖c′​kR\)⏟output alignment\.\\langle\\bm\{R\}\_\{c\}\\bm\{v\},\\bm\{R\}\_\{c^\{\\prime\}\}\\bm\{v\}\\rangle=\\sum\_\{j,k\}\\underbrace\{s^\{R\}\_\{cj\}s^\{R\}\_\{c^\{\\prime\}k\}\}\_\{\\text\{magnitude\}\}\\underbrace\{\(\\bm\{v\}^\{\*\}\\bm\{v\}^\{R\}\_\{cj\}\)\(\(\\bm\{v\}^\{R\}\_\{c^\{\\prime\}k\}\)^\{\*\}\\bm\{v\}\)\}\_\{\\text\{input alignment\}\}\\underbrace\{\(\(\\bm\{u\}^\{R\}\_\{cj\}\)^\{\*\}\\bm\{u\}^\{R\}\_\{c^\{\\prime\}k\}\)\}\_\{\\text\{output alignment\}\}\.\(29\)The boundsup‖𝒗‖=1𝔼c,c′​\|⟨𝑹c​𝒗,𝑹c′​𝒗⟩\|≤η​𝔼c​‖𝑹c‖22\\sup\_\{\\left\\lVert\\bm\{v\}\\right\\rVert=1\}\\mathbb\{E\}\_\{c,c^\{\\prime\}\}\|\\langle\\bm\{R\}\_\{c\}\\bm\{v\},\\bm\{R\}\_\{c^\{\\prime\}\}\\bm\{v\}\\rangle\|\\leq\\eta\\,\\mathbb\{E\}\_\{c\}\\left\\lVert\\bm\{R\}\_\{c\}\\right\\rVert\_\{2\}^\{2\}used in Eq\. \([8](https://arxiv.org/html/2610.31955#S1.E8)\) therefore requires that, on average, the input or output directions of the context\-specific modes are poorly aligned across contexts\.

##### Response\-scale bound\.

Since‖𝑱H​𝒗‖=ρc​\(𝒗\)​ρτ​\(𝒗\)​a​\(𝒗\)\\left\\lVert\\bm\{J\}\_\{H\}\\bm\{v\}\\right\\rVert=\\rho\_\{c\}\(\\bm\{v\}\)\\,\\rho\_\{\\tau\}\(\\bm\{v\}\)\\,a\(\\bm\{v\}\)anda⁡\(𝒗\)≤𝔼c,τ​‖𝑲⁡\(c,ℓ,τ\)‖2a\(\\bm\{v\}\)\\leq\\mathbb\{E\}\_\{c,\\tau\}\\left\\lVert\\bm\{K\}\(c,\\ell,\\tau\)\\right\\rVert\_\{2\}for a unit input𝒗\\bm\{v\},

‖𝑱H​𝒗‖≤ρc​\(𝒗\)​ρτ​\(𝒗\)​𝔼c,τ​‖𝑲⁡\(c,ℓ,τ\)‖2\.\\left\\lVert\\bm\{J\}\_\{H\}\\bm\{v\}\\right\\rVert\\leq\\rho\_\{c\}\(\\bm\{v\}\)\\,\\rho\_\{\\tau\}\(\\bm\{v\}\)\\,\\mathbb\{E\}\_\{c,\\tau\}\\left\\lVert\\bm\{K\}\(c,\\ell,\\tau\)\\right\\rVert\_\{2\}\.\(30\)At a fixed response scale, a direction’s contribution to the lens is therefore capped by the product of its two coherences\.

## Appendix DProof of Corollary[1\.1](https://arxiv.org/html/2610.31955#S1.Thmtheorem1)

###### Proof\.

Let𝒆1,𝒆2\\bm\{e\}\_\{1\},\\bm\{e\}\_\{2\}be orthonormal directions, and consider the linear response model

𝑲⁡\(c,ℓ,τ\)=s⁡\(𝒆1​𝒆1∗\+ξc​\(−1\)τ​𝒆2​𝒆2∗\),s\>0,\\bm\{K\}\(c,\\ell,\\tau\)=s\\left\(\\bm\{e\}\_\{1\}\\bm\{e\}\_\{1\}^\{\*\}\+\\xi\_\{c\}\(\-1\)^\{\\tau\}\\bm\{e\}\_\{2\}\\bm\{e\}\_\{2\}^\{\*\}\\right\),\\qquad s\>0,whereξc\\xi\_\{c\}takes values±1\\pm 1with equal probability\. The first direction is perfectly coherent across contexts and constant across future lags; the second changes sign across contexts and oscillates rapidly across lags\. Thus

𝑱H=𝔼c​1H​∑τ=0H−1𝑲⁡\(c,ℓ,τ\)=s​𝒆1​𝒆1∗\.\\bm\{J\}\_\{H\}=\\mathbb\{E\}\_\{c\}\\frac\{1\}\{H\}\\sum\_\{\\tau=0\}^\{H\-1\}\\bm\{K\}\(c,\\ell,\\tau\)=s\\,\\bm\{e\}\_\{1\}\\bm\{e\}\_\{1\}^\{\*\}\.The unique nonzero singular value isss, and its input direction is𝒆1\\bm\{e\}\_\{1\}\.

Independently, choose centered input activations, independent ofξc\\xi\_\{c\}, with covariance

𝑪=β​𝒆1​𝒆1∗\+𝒆2​𝒆2∗,β\>0\.\\bm\{C\}=\\beta\\,\\bm\{e\}\_\{1\}\\bm\{e\}\_\{1\}^\{\*\}\+\\bm\{e\}\_\{2\}\\bm\{e\}\_\{2\}^\{\*\},\\qquad\\beta\>0\.These responses are realized by the context\-dependent linear maps𝒉↦𝑲⁡\(c,ℓ,τ\)​𝒉\\bm\{h\}\\mapsto\\bm\{K\}\(c,\\ell,\\tau\)\\bm\{h\}\. The fraction of activation variance captured by projection onto the selected direction is

𝒆1∗​𝑪​𝒆1tr⁡𝑪=β1\+β⟶0as​β→0\.\\frac\{\\bm\{e\}\_\{1\}^\{\*\}\\bm\{C\}\\bm\{e\}\_\{1\}\}\{\\operatorname\{tr\}\\bm\{C\}\}=\\frac\{\\beta\}\{1\+\\beta\}\\longrightarrow 0\\quad\\text\{as \}\\beta\\to 0\.Interchanging the two covariance eigenvalues makes this fraction approach11, without changing the Jacobians, their coherence or slowness, or the singular values of𝑱H\\bm\{J\}\_\{H\}\.

By \([12](https://arxiv.org/html/2610.31955#S1.E12)\), every nonzero token\-indexed J\-lens direction in this example is proportional to𝒆1\\bm\{e\}\_\{1\}\. Hence even the full span of the J\-lens directions can capture arbitrarily little activation variance\. Jacobian averaging favors coherent, slowly varying responses, weighted by their response magnitudes\. Those properties do not require the corresponding activations to have large variance\. ∎

## Appendix EBayes filter derivations

Any distributionpponℤn\\mathbb\{Z\}\_\{n\}is defined by its modes𝔼p​ϕj​\(X\)\\mathbb\{E\}\_\{p\}\\phi\_\{j\}\(X\):

p⁡\(x\)=1n​∑j=0n−1𝔼p​\[ϕj​\(X\)\]​ϕj​\(x\)¯\.p\(x\)=\\frac\{1\}\{n\}\\sum\_\{j=0\}^\{n\-1\}\\mathbb\{E\}\_\{p\}\[\\phi\_\{j\}\(X\)\]\\,\\overline\{\\phi\_\{j\}\(x\)\}\.\(31\)
###### Lemma E\.1\(Expected dynamic\)\.

Forj≠0j\\neq 0,

𝔼⁡\[ϕj​\(Xt\+1\)∣Xt\]=αj​ϕj​\(Xt\),αj:=λ​cj​ei​θj,cj:=1−2​q\+2​q​cos⁡\(2​π​j/n\)\.\\mathbb\{E\}\[\\phi\_\{j\}\(X\_\{t\+1\}\)\\mid X\_\{t\}\]=\\alpha\_\{j\}\\,\\phi\_\{j\}\(X\_\{t\}\),\\qquad\\alpha\_\{j\}:=\\lambda\\,c\_\{j\}\\,e^\{\\mathrm\{i\}\\theta\_\{j\}\},\\qquad c\_\{j\}:=1\-2q\+2q\\cos\(2\\pi j/n\)\.\(32\)

###### Proof\.

The expectation of a mode is𝔼​ϕj=1n​∑yϕj​\(y\)=0\\mathbb\{E\}\\phi\_\{j\}=\\frac\{1\}\{n\}\\sum\_\{y\}\\phi\_\{j\}\(y\)=0\. One update step gives

ϕj​\(Xt\+v\+Dt\)=ϕj​\(Xt\)​ei​θj​e2​π​i​j​Dt/n\\displaystyle\\phi\_\{j\}\(X\_\{t\}\+v\+D\_\{t\}\)=\\phi\_\{j\}\(X\_\{t\}\)\\,e^\{\\mathrm\{i\}\\theta\_\{j\}\}e^\{2\\pi\\mathrm\{i\}jD\_\{t\}/n\}\(33\)and

𝔼e2​π​i​j​Dt/n=1−2q\+q\(e2​π​i​j/n\+e−2πij/n\)=cj\.\\displaystyle\\mathbb\{E\}e^\{2\\pi\\mathrm\{i\}jD\_\{t\}/n\}=1\-2q\+q\\left\(e^\{2\\pi\\mathrm\{i\}j/n\}\+e^\{\-2\\pi\\mathrm\{i\}j/n\}\\right\)=c\_\{j\}\.\(34\)∎

### E\.1Derivation of the J\-lens factors

We use the notation of Section[3\.2](https://arxiv.org/html/2610.31955#S3.SS2):Kj​\(c,τ\)=αjτ\+1​AτK\_\{j\}\(c,\\tau\)=\\alpha\_\{j\}^\{\\tau\+1\}A\_\{\\tau\},rj=\|αj\|​\(1−ε\)r\_\{j\}=\\left\\lvert\\alpha\_\{j\}\\right\\rvert\(1\-\\varepsilon\), andN⁡\(c\)N\(c\)the number of tokens until the next reveal, withP⁡\(N=k\)=\(1−ε\)k−1​εP\(N=k\)=\(1\-\\varepsilon\)^\{k\-1\}\\varepsilon\. ThenAτ=1A\_\{\\tau\}=1forτ<N\\tau<Nand00fromNNonward, so𝔼c​Aτ=\(1−ε\)τ\\mathbb\{E\}\_\{c\}A\_\{\\tau\}=\(1\-\\varepsilon\)^\{\\tau\}\.

##### Magnitude\.

Since\|Kj​\(c,τ\)\|=\|αj\|τ\+1​Aτ\\left\\lvert K\_\{j\}\(c,\\tau\)\\right\\rvert=\\left\\lvert\\alpha\_\{j\}\\right\\rvert^\{\\tau\+1\}A\_\{\\tau\},

a\(j\)=\|αj\|​∑τ=1Hrjτ=\|αj\|​rj​\(1−rjH\)1−rj\.a^\{\(j\)\}=\\left\\lvert\\alpha\_\{j\}\\right\\rvert\\sum\_\{\\tau=1\}^\{H\}r\_\{j\}^\{\\tau\}=\\left\\lvert\\alpha\_\{j\}\\right\\rvert\\,\\frac\{r\_\{j\}\(1\-r\_\{j\}^\{H\}\)\}\{1\-r\_\{j\}\}\.\(35\)

##### Lag coherence\.

For one mode, the lag coherence is

ρτ\(j\):=𝔼c​\|1H​∑τ=1HKj​\(c,τ\)\|𝔼c​1H​∑τ=1H\|Kj​\(c,τ\)\|,\\rho\_\{\\tau\}^\{\(j\)\}:=\\frac\{\\mathbb\{E\}\_\{c\}\\left\\lvert\\frac\{1\}\{H\}\\sum\_\{\\tau=1\}^\{H\}K\_\{j\}\(c,\\tau\)\\right\\rvert\}\{\\mathbb\{E\}\_\{c\}\\frac\{1\}\{H\}\\sum\_\{\\tau=1\}^\{H\}\\left\\lvert K\_\{j\}\(c,\\tau\)\\right\\rvert\},\(36\)whose denominator isa\(j\)/Ha^\{\(j\)\}/H\. Summing the Jacobian over lags,

∑τ=1HKj​\(c,τ\)=αj​Sj,N⁡\(c\),Sj,N:=∑τ=1min⁡\(N−1,H\)αjτ,\\sum\_\{\\tau=1\}^\{H\}K\_\{j\}\(c,\\tau\)=\\alpha\_\{j\}\\,S\_\{j,N\(c\)\},\\qquad S\_\{j,N\}:=\\sum\_\{\\tau=1\}^\{\\min\(N\-1,H\)\}\\alpha\_\{j\}^\{\\tau\},\(37\)whereSj,NS\_\{j,N\}is the update accumulated over the steps until the state is next revealed\. The lag coherence is therefore

ρτ\(j\)=\|αj\|​𝔼c​\|Sj,N\|\|αj\|​∑τ=1Hrjτ=\(1−rj\)​𝔼c​\|Sj,N\|rj​\(1−rjH\),\\rho\_\{\\tau\}^\{\(j\)\}=\\frac\{\\left\\lvert\\alpha\_\{j\}\\right\\rvert\\mathbb\{E\}\_\{c\}\\left\\lvert S\_\{j,N\}\\right\\rvert\}\{\\left\\lvert\\alpha\_\{j\}\\right\\rvert\\sum\_\{\\tau=1\}^\{H\}r\_\{j\}^\{\\tau\}\}=\\frac\{\(1\-r\_\{j\}\)\\mathbb\{E\}\_\{c\}\\left\\lvert S\_\{j,N\}\\right\\rvert\}\{r\_\{j\}\(1\-r\_\{j\}^\{H\}\)\},\(38\)where the expectation over the reveal time has the closed form

𝔼c​\|Sj,N⁡\(c\)\|=∑k=1H\(1−ε\)k−1​ε​\|Sj,k\|\+\(1−ε\)H​\|Sj,H\+1\|\.\\mathbb\{E\}\_\{c\}\\left\\lvert S\_\{j,N\(c\)\}\\right\\rvert=\\sum\_\{k=1\}^\{H\}\(1\-\\varepsilon\)^\{k\-1\}\\varepsilon\\,\\left\\lvert S\_\{j,k\}\\right\\rvert\+\(1\-\\varepsilon\)^\{H\}\\left\\lvert S\_\{j,H\+1\}\\right\\rvert\.\(39\)

##### Context coherence\.

The mean of the accumulated response is

𝔼c​Sj,N⁡\(c\)=∑τ=1Hαjτ​\(1−ε\)τ=∑τ=1H\(rj​ei​θj\)τ,\|𝔼c​Sj,N⁡\(c\)\|=rj​\|1−\(rj​ei​θj\)H\|\|1−rj​ei​θj\|\.\\mathbb\{E\}\_\{c\}S\_\{j,N\(c\)\}=\\sum\_\{\\tau=1\}^\{H\}\\alpha\_\{j\}^\{\\tau\}\(1\-\\varepsilon\)^\{\\tau\}=\\sum\_\{\\tau=1\}^\{H\}\\left\(r\_\{j\}e^\{\\mathrm\{i\}\\theta\_\{j\}\}\\right\)^\{\\tau\},\\qquad\\left\\lvert\\mathbb\{E\}\_\{c\}S\_\{j,N\(c\)\}\\right\\rvert=\\frac\{r\_\{j\}\\left\\lvert 1\-\(r\_\{j\}e^\{\\mathrm\{i\}\\theta\_\{j\}\}\)^\{H\}\\right\\rvert\}\{\\left\\lvert 1\-r\_\{j\}e^\{\\mathrm\{i\}\\theta\_\{j\}\}\\right\\rvert\}\.\(40\)The context coherence

ρc\(j\):=\|𝔼c​∑τ=1HKj​\(c,τ\)\|𝔼c​\|∑τ=1HKj​\(c,τ\)\|=\|𝔼c​Sj,N\|𝔼c​\|Sj,N\|=rj​\|1−\(rj​ei​θj\)H\|\|1−rj​ei​θj\|​𝔼c​\|Sj,N\|\\rho\_\{c\}^\{\(j\)\}:=\\frac\{\\left\\lvert\\mathbb\{E\}\_\{c\}\\sum\_\{\\tau=1\}^\{H\}K\_\{j\}\(c,\\tau\)\\right\\rvert\}\{\\mathbb\{E\}\_\{c\}\\left\\lvert\\sum\_\{\\tau=1\}^\{H\}K\_\{j\}\(c,\\tau\)\\right\\rvert\}=\\frac\{\\left\\lvert\\mathbb\{E\}\_\{c\}S\_\{j,N\}\\right\\rvert\}\{\\mathbb\{E\}\_\{c\}\\left\\lvert S\_\{j,N\}\\right\\rvert\}=\\frac\{r\_\{j\}\\left\\lvert 1\-\(r\_\{j\}e^\{\\mathrm\{i\}\\theta\_\{j\}\}\)^\{H\}\\right\\rvert\}\{\\left\\lvert 1\-r\_\{j\}e^\{\\mathrm\{i\}\\theta\_\{j\}\}\\right\\rvert\\,\\mathbb\{E\}\_\{c\}\\left\\lvert S\_\{j,N\}\\right\\rvert\}\(41\)follows by dividing Eq\. \([40](https://arxiv.org/html/2610.31955#A5.E40)\) by Eq\. \([39](https://arxiv.org/html/2610.31955#A5.E39)\)\.

##### Long\-window limit\.

Since\|αj\|=λ​\|cj\|<1\|\\alpha\_\{j\}\|=\\lambda\|c\_\{j\}\|<1, the accumulated responses are bounded,\|Sj,k\|≤\|αj\|/\(1−\|αj\|\)\|S\_\{j,k\}\|\\leq\|\\alpha\_\{j\}\|/\(1\-\|\\alpha\_\{j\}\|\), so the boundary term\(1−ε\)H​\|Sj,H\+1\|\(1\-\\varepsilon\)^\{H\}\|S\_\{j,H\+1\}\|in Eq\. \([39](https://arxiv.org/html/2610.31955#A5.E39)\) and the factorsrjHr\_\{j\}^\{H\}in Eqs\. \([35](https://arxiv.org/html/2610.31955#A5.E35)\), \([38](https://arxiv.org/html/2610.31955#A5.E38)\), \([40](https://arxiv.org/html/2610.31955#A5.E40)\) and \([41](https://arxiv.org/html/2610.31955#A5.E41)\) vanish geometrically asH→∞H\\to\\infty\. The magnitude and the mean accumulated response become

a\(j\)=\|αj\|​rj1−rj,𝔼c​Sj,N=rj​ei​θj1−rj​ei​θj,a^\{\(j\)\}=\\frac\{\|\\alpha\_\{j\}\|\\,r\_\{j\}\}\{1\-r\_\{j\}\},\\qquad\\mathbb\{E\}\_\{c\}S\_\{j,N\}=\\frac\{r\_\{j\}e^\{\\mathrm\{i\}\\theta\_\{j\}\}\}\{1\-r\_\{j\}e^\{\\mathrm\{i\}\\theta\_\{j\}\}\},\(42\)and, substitutingSj,k=αj​\(1−αjk−1\)/\(1−αj\)S\_\{j,k\}=\\alpha\_\{j\}\(1\-\\alpha\_\{j\}^\{k\-1\}\)/\(1\-\\alpha\_\{j\}\)into Eq\. \([39](https://arxiv.org/html/2610.31955#A5.E39)\),

𝔼c​\|Sj,N\|=ε​\|αj\|\|1−αj\|​∑k=0∞\(1−ε\)k​\|1−αjk\|\.\\mathbb\{E\}\_\{c\}\|S\_\{j,N\}\|=\\frac\{\\varepsilon\\,\|\\alpha\_\{j\}\|\}\{\|1\-\\alpha\_\{j\}\|\}\\sum\_\{k=0\}^\{\\infty\}\(1\-\\varepsilon\)^\{k\}\\bigl\|1\-\\alpha\_\{j\}^\{k\}\\bigr\|\.\(43\)The coherences areρτ\(j\)=\(1−rj\)​𝔼c​\|Sj,N\|/rj\\rho^\{\(j\)\}\_\{\\tau\}=\(1\-r\_\{j\}\)\\,\\mathbb\{E\}\_\{c\}\|S\_\{j,N\}\|/r\_\{j\}andρc\(j\)=rj/\(\|1−rj​ei​θj\|​𝔼c​\|Sj,N\|\)\\rho^\{\(j\)\}\_\{c\}=r\_\{j\}/\\bigl\(\|1\-r\_\{j\}e^\{\\mathrm\{i\}\\theta\_\{j\}\}\|\\,\\mathbb\{E\}\_\{c\}\|S\_\{j,N\}\|\\bigr\)\. For generalθj\\theta\_\{j\}, the modulus\|1−αjk\|\|1\-\\alpha\_\{j\}^\{k\}\|prevents a closed form for Eq\. \([43](https://arxiv.org/html/2610.31955#A5.E43)\), but it cancels from their product and from the singular value of Eq\. \([11](https://arxiv.org/html/2610.31955#S1.E11)\),

ρτ\(j\)​ρc\(j\)=1−rj\|1−rj​ei​θj\|,σ\(j\):=a\(j\)​ρτ\(j\)​ρc\(j\)=\|αj\|​\|𝔼c​Sj,N\|=\|αj\|​rj\|1−rj​ei​θj\|,\\rho^\{\(j\)\}\_\{\\tau\}\\rho^\{\(j\)\}\_\{c\}=\\frac\{1\-r\_\{j\}\}\{\|1\-r\_\{j\}e^\{\\mathrm\{i\}\\theta\_\{j\}\}\|\},\\qquad\\sigma^\{\(j\)\}:=a^\{\(j\)\}\\,\\rho^\{\(j\)\}\_\{\\tau\}\\rho^\{\(j\)\}\_\{c\}=\|\\alpha\_\{j\}\|\\,\\bigl\|\\mathbb\{E\}\_\{c\}S\_\{j,N\}\\bigr\|=\\frac\{\|\\alpha\_\{j\}\|\\,r\_\{j\}\}\{\|1\-r\_\{j\}e^\{\\mathrm\{i\}\\theta\_\{j\}\}\|\},\(44\)so only the split ofσ\(j\)\\sigma^\{\(j\)\}between lag and context coherence requires the series\. The split is closed whenαj\\alpha\_\{j\}is real\. Then1−αjk\>01\-\\alpha\_\{j\}^\{k\}\>0for allkk, soSj,kS\_\{j,k\}has the sign ofαj\\alpha\_\{j\}for everykkand𝔼c​\|Sj,N\|=\|𝔼c​Sj,N\|\\mathbb\{E\}\_\{c\}\|S\_\{j,N\}\|=\|\\mathbb\{E\}\_\{c\}S\_\{j,N\}\|\. For a static latent \(θ=0\\theta=0\),𝔼c​\|Sj,N\|=rj/\(1−rj\)\\mathbb\{E\}\_\{c\}\|S\_\{j,N\}\|=r\_\{j\}/\(1\-r\_\{j\}\)andρτ\(j\)=ρc\(j\)=1\\rho^\{\(j\)\}\_\{\\tau\}=\\rho^\{\(j\)\}\_\{c\}=1: the singular value is pure magnitude\. For an alternating mode \(θj=π\\theta\_\{j\}=\\pi\),𝔼c​\|Sj,N\|=rj/\(1\+rj\)\\mathbb\{E\}\_\{c\}\|S\_\{j,N\}\|=r\_\{j\}/\(1\+r\_\{j\}\),ρc\(j\)=1\\rho^\{\(j\)\}\_\{c\}=1andρτ\(j\)=\(1−rj\)/\(1\+rj\)\\rho^\{\(j\)\}\_\{\\tau\}=\(1\-r\_\{j\}\)/\(1\+r\_\{j\}\): the context coherence is perfect, and the lag coherence carries the whole loss from rotation\. For intermediateθj\\theta\_\{j\}both coherences are below one, with their product fixed by Eq\. \([44](https://arxiv.org/html/2610.31955#A5.E44)\)\.

##### Singular\-value switching\.

For the Bayes filter in belief units, the latents are independent and share only the reveal, so the lens is block diagonal over latents \(Section[3\.2](https://arxiv.org/html/2610.31955#S3.SS2)\) and each latent’s modejjcontributes a degenerate pair of singular values equal toσ\(j\)\\sigma^\{\(j\)\}\. Singular values of different latents therefore cross rather than avoid each other, and the top direction of the lens switches from one latent to the other exactly where their singular values are equal\. Compare two latents, labeled00and11as in Fig\.[8](https://arxiv.org/html/2610.31955#S4.F8), on modej=1j=1, which carries the largest singular value here\. We fixj=1j=1and index by latentiiinstead:αi=λi​c1​ei​θi\\alpha\_\{i\}=\\lambda\_\{i\}c\_\{1\}e^\{\\mathrm\{i\}\\theta\_\{i\}\},ri=\(1−ε\)​\|αi\|r\_\{i\}=\(1\-\\varepsilon\)\|\\alpha\_\{i\}\|, andσi\\sigma\_\{i\}is the corresponding singular value\. Herec1c\_\{1\}is the diffusion factor of modej=1j=1, shared by both latents\. By Eq\. \([44](https://arxiv.org/html/2610.31955#A5.E44)\), the switch occurs when

\|α0\|2\|1−r0​ei​θ0\|=\|α1\|2\|1−r1​ei​θ1\|\.\\frac\{\|\\alpha\_\{0\}\|^\{2\}\}\{\|1\-r\_\{0\}e^\{\\mathrm\{i\}\\theta\_\{0\}\}\|\}=\\frac\{\|\\alpha\_\{1\}\|^\{2\}\}\{\|1\-r\_\{1\}e^\{\\mathrm\{i\}\\theta\_\{1\}\}\|\}\.\(45\)Take latent 0 static \(θ0=0\\theta\_\{0\}=0\) with its persistenceλ0\\lambda\_\{0\}swept, and latent 1 fixed\. Letκ:=σ1/\(1−ε\)=\|α1\|2/\|1−r1​ei​θ1\|\\kappa:=\\sigma\_\{1\}/\(1\-\\varepsilon\)=\|\\alpha\_\{1\}\|^\{2\}/\|1\-r\_\{1\}e^\{\\mathrm\{i\}\\theta\_\{1\}\}\|\. Then Eq\. \([45](https://arxiv.org/html/2610.31955#A5.E45)\) is the quadratic\|α0\|2\+\(1−ε\)​κ​\|α0\|−κ=0\|\\alpha\_\{0\}\|^\{2\}\+\(1\-\\varepsilon\)\\kappa\\,\|\\alpha\_\{0\}\|\-\\kappa=0, whose positive root gives the critical persistence

λ0∗=12​c1​\[\(1−ε\)2​κ2\+4​κ−\(1−ε\)​κ\]\.\\lambda\_\{0\}^\{\*\}=\\frac\{1\}\{2c\_\{1\}\}\\Bigl\[\\sqrt\{\(1\-\\varepsilon\)^\{2\}\\kappa^\{2\}\+4\\kappa\}\-\(1\-\\varepsilon\)\\kappa\\Bigr\]\.\(46\)This closed form needsθ0=0\\theta\_\{0\}=0; otherwise the equation is quartic and has no simple root\. The singular valueσ0\\sigma\_\{0\}rises monotonically from00withλ0\\lambda\_\{0\}\. The static latent therefore takes the top direction forλ0\>λ0∗\\lambda\_\{0\}\>\\lambda\_\{0\}^\{\*\}and loses it below, and a switch exists within the sweep wheneverσ0\|λ0=1\>σ1\\sigma\_\{0\}\|\_\{\\lambda\_\{0\}=1\}\>\\sigma\_\{1\}\. The rotating latent’s handicap enters only through\|1−r1​ei​θ1\|\|1\-r\_\{1\}e^\{\\mathrm\{i\}\\theta\_\{1\}\}\|, the inverse of its product of coherences up to the factor1−r11\-r\_\{1\}, which is smallest forθ1=0\\theta\_\{1\}=0and grows with\|θ1\|\|\\theta\_\{1\}\|up toπ\\pi\. A faster\-turning latent is thus overtaken at a lowerλ0∗\\lambda\_\{0\}^\{\*\}\.

## Appendix FTransformer Training Methods

Each model is a standard decoder\-only transformer with causal self\-attention and rotary position embeddings \(RoPE;[Su et al\., 2024](https://arxiv.org/html/2610.31955#bib.bib28)\) applied to the queries and keys only during attention\. The vocabulary sizeVVis defined by the HMM:V=83\+1=513V=8^\{3\}\+1=513for three latents on rings of88\(every combination of revealed places, plus the null token\) andV=\(1\+2⋅23\)​23=136V=\(1\+2\\cdot 2^\{3\}\)\\,2^\{3\}=136for the context sweep \(null or plain or keyed reveal of three±1\\pm 1latents, times the three key bits\)\.

We write𝒉ℓ,t∈ℝd\\bm\{h\}\_\{\\ell,t\}\\in\\mathbb\{R\}^\{d\}for the residual stream at positionttbefore blockℓ\+1\\ell\+1:𝒉0,t\\bm\{h\}\_\{0,t\}is the token embedding,𝒉1,t\\bm\{h\}\_\{1,t\}the stream between the two blocks, and𝒉2,t\\bm\{h\}\_\{2,t\}the output of the last block \(soL=2L=2\), which the final LayerNorm and𝑾U\\bm\{W\}\_\{U\}read\. The J\-lens is read atℓ=0\\ell=0andℓ=1\\ell=1\.

We trained each model with cross\-entropy on next\-token prediction, with AdamW\([Loshchilov and Hutter, 2019](https://arxiv.org/html/2610.31955#bib.bib29)\)for20,00020\{,\}000steps with learning rate10−310^\{\-3\}and weight decay0\.010\.01, on uniform batches of6464sequences\. We compared it with two references: the Bayes filter, which is the optimal predictor, and a memoryless predictor that uses the stationary distribution of the process\.

### F\.1Data for all sweeps

For each sweep and seed we draw500,000500\{,\}000training sequences of192192tokens from the respective HMM process, and200200test sequences\. All sweeps use a reveal probability ofε=0\.2\\varepsilon=0\.2\. We ran 8 model seeds per task, jittering task parameters for each seed around a fixed mean\. For the magnitude sweep, we set the number of states per latentn=8n=8, velocityv=0v=0, diffusion probabilityq=1/4q=1/4and persistenceλi=0\.8,0\.9,0\.95\\lambda\_\{i\}=0\.8,0\.9,0\.95\. For the lag sweep, we setn=8n=8,v=4,2,0v=4,2,0,q=1/4q=1/4,λ=0\.9\\lambda=0\.9\. For the context coherence sweep we setn=2n=2,v=0v=0,q=0q=0,λ=0\.95\\lambda=0\.95, and key probabilitiesψi=0,0\.75,1\\psi\_\{i\}=0,0\.75,1\.

These parameter sweeps selectively modulate magnitude, lag coherence and context coherence:

##### Magnitude sweep\.

We only differλi\\lambda\_\{i\}, hence allSN\>0S\_\{N\}\>0and therefore𝔼c​\|SN\|=𝔼c​SN\\mathbb\{E\}\_\{c\}\|S\_\{N\}\|=\\mathbb\{E\}\_\{c\}S\_\{N\}and soρτ=1\\rho\_\{\\tau\}=1, and likewiseρc=1\\rho\_\{c\}=1\. Recall that

ai=λi​c1​ri​\(1−riH\)1−ria\_\{i\}=\\lambda\_\{i\}c\_\{1\}\\frac\{r\_\{i\}\(1\-r\_\{i\}^\{H\}\)\}\{1\-r\_\{i\}\}\(47\)withri=λi​c1​\(1−ε\)r\_\{i\}=\\lambda\_\{i\}c\_\{1\}\(1\-\\varepsilon\), giving magnitude parameters of0\.82,1\.23,1\.500\.82,1\.23,1\.50forλ=0\.8,0\.9,0\.95\\lambda=0\.8,0\.9,0\.95\.

##### Lag sweep\.

We have thatλ,c1,ε\\lambda,c\_\{1\},\\varepsilonare shared, hence the magnitude is fixed across all latents\. Onlyθ\\thetavaries, which variesαi=λ​c1​ei​θi\\alpha\_\{i\}=\\lambda c\_\{1\}e^\{\\mathrm\{i\}\\theta\_\{i\}\}, and withSk=αi​1−αik−11−αiS\_\{k\}=\\alpha\_\{i\}\\frac\{1\-\\alpha\_\{i\}^\{k\-1\}\}\{1\-\\alpha\_\{i\}\}, we have thatρτ=1,0\.35,0\.24\\rho\_\{\\tau\}=1,0\.35,0\.24forθi=0,π/2,π\\theta\_\{i\}=0,\\pi/2,\\pi\. Forρc\\rho\_\{c\}, the sign ofSNS\_\{N\}is shared across all contexts forθ=0,π\\theta=0,\\piand soρc=1\\rho\_\{c\}=1\. Forπ/2\\pi/2, the value is slightly less, atρc=0\.945\\rho\_\{c\}=0\.945\.

##### Context sweep\.

For the context sweep, we modifiedψi\\psi\_\{i\}only\. This keepsρτ\\rho\_\{\\tau\}and magnitudeaafixed, whileρc=ψ\\rho\_\{c\}=\\psi, as𝔼c​\[Wt−1​SN\]=ψ​𝔼c​SN\\mathbb\{E\}\_\{c\}\[W\_\{t\-1\}S\_\{N\}\]=\\psi\\mathbb\{E\}\_\{c\}S\_\{N\}and𝔼​\|Wt−1​SN\|=𝔼c​SN\\mathbb\{E\}\|W\_\{t\-1\}S\_\{N\}\|=\\mathbb\{E\}\_\{c\}S\_\{N\}\.

## Appendix GComputing alignment of J\-lens vectors and latent state representations

We compute the Jacobian lens on10001000sampled sequences from the task HMM after a 32\-token burn\-in period, for sequences of length192192\. We computed the Jacobian only on the tokens that revealed latent states, totaling about19,00019\{,\}000sources\. We averaged overH=128H=128lags\. The Jacobian was computed with forward\-mode autodifferentiation and validated with a split\-halves analysis\. For the Bayes filter, the input is separated by latent, and the output is the predicted distribution over each latent separately as well\.

We computed the singular value decomposition𝑱H=𝑼​𝚺​𝑽∗\\bm\{J\}\_\{H\}=\\bm\{U\}\\bm\{\\Sigma\}\\bm\{V\}^\{\*\}using standard linear algebra routines in Python\. The left singular vectors𝑼\\bm\{U\}are the output directions that the Jacobian writes, while the right singular vectors𝑽\\bm\{V\}are the directions that the model reads\. They are ordered by the singular values on the diagonal of𝚺\\bm\{\\Sigma\}\.

## Appendix HAlignment between J\-lens vectors and latent vectors

We identify the relationship between the J\-lens vectors and the latent state features by identifying an encoding and decoding subspace for each latent state modeϕj​\(X\(i\)\)\\phi\_\{j\}\(X^\{\(i\)\}\)\(the Fourier modejjof latent stateii\), and computing the overlap between the J\-lens vector and these subspaces\. Specifically, we takensn\_\{s\}sequences and, at the last time step of each, recordpplatent state patterns \(e\.g\.ℜ⁡ϕ1​\(X\(1\)\)\\Re\\,\\phi\_\{1\}\(X^\{\(1\)\}\)andℑ⁡ϕ1​\(X\(1\)\)\\Im\\,\\phi\_\{1\}\(X^\{\(1\)\}\), the real and imaginary parts of the first Fourier mode of latent state11\)\. We stack the centered residual activity vectors into𝒁∈ℝns×d\\bm\{Z\}\\in\\mathbb\{R\}^\{n\_\{s\}\\times d\}and the centered patterns into𝚽∈ℝns×p\\bm\{\\Phi\}\\in\\mathbb\{R\}^\{n\_\{s\}\\times p\}, which has two columns \(real and imaginary parts\) for each latent mode\. These are the patterns on which encoding and decoding regression is done\.

We compute the encoding direction of each pattern independently via

𝑬=diag​\(𝚽⊤​𝚽\)−1​𝚽⊤​𝒁,\\bm\{E\}=\\mathrm\{diag\}\(\\bm\{\\Phi\}^\{\\top\}\\bm\{\\Phi\}\)^\{\-1\}\\bm\{\\Phi\}^\{\\top\}\\bm\{Z\},\(48\)which is a map from patterns to activity; the diagonal means that each pattern is regressed independently, rather than jointly\. For a decoding analysis, we reverse the process, computing

𝑫i=\(𝒁⊤​𝒁\+ν​𝑰\)−1​𝒁⊤​𝑴i,\\bm\{D\}\_\{i\}=\(\\bm\{Z\}^\{\\top\}\\bm\{Z\}\+\\nu\\bm\{I\}\)^\{\-1\}\\bm\{Z\}^\{\\top\}\\bm\{M\}\_\{i\},\(49\)whereν\\nuis a ridge penalty and𝑴i∈ℝns×2\\bm\{M\}\_\{i\}\\in\\mathbb\{R\}^\{n\_\{s\}\\times 2\}holds the Bayes filter beliefs of latentii, for example𝔼⁡\[ϕ1​\(Xt\(i\)\)∣Y≤t\]\\mathbb\{E\}\[\\phi\_\{1\}\(X^\{\(i\)\}\_\{t\}\)\\mid Y\_\{\\leq t\}\], split into real and imaginary parts\.

On the output side, the patterns are directly readable from the tokens\. Each token represents a particular state of all latents, so we can directly align the Fourier mode patterns broadcast to the output singular vectors\. That is, each output singular vector is inℝV\\mathbb\{R\}^\{V\}, and for each Fourier mode of each latent we construct a vector inℝV\\mathbb\{R\}^\{V\}by setting the entry of each token tocos⁡\(2​π​x/n\)\\cos\(2\\pi x/n\)\(orsin\\sin\), wherexxis the state of that latent in the token\. Then, the encoding analysis simply comes down to projection of the J\-lens singular vector onto the plane spanned by each pattern individually\.

We compute alignment by measuring how much a unit vector of the J\-lens lies in the span of a group of encoding or decoding directions, which in practice are taken to be the vectors corresponding to a single mode of a latent state \(typically two vectors corresponding to the sine and cosine directions\)\. We define the overlap of a unit lens vector𝒗\\bm\{v\}by projecting it onto the span of thekik\_\{i\}reference vectors corresponding to the relevant feature:

Oi​\(𝒗\)=‖𝑸i⊤​𝒗‖,O\_\{i\}\(\\bm\{v\}\)=\\left\\lVert\\bm\{Q\}\_\{i\}^\{\\top\}\\bm\{v\}\\right\\rVert,\(50\)where the columns of𝑸i\\bm\{Q\}\_\{i\}are an orthonormal basis for thekik\_\{i\}reference vectors, constructed with a QR decomposition\. Under this construction, a random unit direction indddimensions has an expected squared projection ofki/dk\_\{i\}/donto akik\_\{i\}\-dimensional subspace, making the chance level of overlapki/d\\sqrt\{k\_\{i\}/d\}, withki=2k\_\{i\}=2for Fourier modes andd=128d=128\. We assign each J\-lens direction to the latent with the largest overlap\.

## Appendix IValidating singular value switching theory

The switching analysis \(Appendix[E\.1](https://arxiv.org/html/2610.31955#A5.SS1.SSS0.Px4)\) makes two predictions\. First, in the large\-HHlimit the singular value of each latent’s first Fourier mode isσi=\|αi\|​ri/\|1−ri​ei​θi\|\\sigma\_\{i\}=\\left\\lvert\\alpha\_\{i\}\\right\\rvert r\_\{i\}/\\left\\lvert 1\-r\_\{i\}e^\{\\mathrm\{i\}\\theta\_\{i\}\}\\right\\rvert\(Eq\. \([44](https://arxiv.org/html/2610.31955#A5.E44)\)\)\. Second, for a static latent 0 with swept persistenceλ0\\lambda\_\{0\}and a fixed latent 1, the two singular values cross, and the top J\-lens direction switches from latent 1 to latent 0, at the persistenceλ0∗\\lambda\_\{0\}^\{\*\}of Eq\. \([46](https://arxiv.org/html/2610.31955#A5.E46)\)\. We test both predictions on the Bayes filter, and the second on transformers trained across the switch\.

##### Task\.

We use the toy\-model HMM with two unkeyed latents on rings ofn=8n=8states, diffusion probabilityq=1/4q=1/4and reveal probabilityε=0\.2\\varepsilon=0\.2, which givesV=82\+1=65V=8^\{2\}\+1=65tokens\. Latent 0 is static \(v0=0v\_\{0\}=0\) and its persistenceλ0\\lambda\_\{0\}is swept\. Latent 1 has fixed persistenceλ1=0\.95\\lambda\_\{1\}=0\.95and velocityv1∈\{1,2,3,4\}v\_\{1\}\\in\\\{1,2,3,4\\\}, that is, angular velocityθ1=π​v1/4∈\{π/4,π/2,3​π/4,π\}\\theta\_\{1\}=\\pi v\_\{1\}/4\\in\\\{\\pi/4,\\pi/2,3\\pi/4,\\pi\\\}, which are all the nonzero turns a ring of88allows, up to direction\.

##### Theory\.

The predicted singular values are Eq\. \([44](https://arxiv.org/html/2610.31955#A5.E44)\), evaluated for each latent on modej=1j=1\. The predicted switch point is Eq\. \([46](https://arxiv.org/html/2610.31955#A5.E46)\), which depends only on latent 1 and the shared parameters, not onλ0\\lambda\_\{0\}\. The series for𝔼c​\|Sj,N\|\\mathbb\{E\}\_\{c\}\\left\\lvert S\_\{j,N\}\\right\\rvert\(Eq\. \([43](https://arxiv.org/html/2610.31955#A5.E43)\)\), needed for the individual coherences but not forσi\\sigma\_\{i\}, is truncated at200200terms\.

##### Measuring the singular values\.

We compute the J\-lens as in Appendix[G](https://arxiv.org/html/2610.31955#A7), on10001000sampled sequences of192192tokens with a3232\-token burn\-in, using only the revealed source tokens, and sum overH=64H=64lags\. The first Fourier mode of each latent is a plane \(its real and imaginary parts\), so each latent owns two singular directions\. We assign the J\-lens directions to latents by the overlap of their output side with each latent’s mode\-1 pattern \(Appendix[H](https://arxiv.org/html/2610.31955#A8)\), and takeσi\\sigma\_\{i\}to be the mean of the two singular values assigned to latentii\. For the Bayes filter the two are degenerate \(Appendix[E\.1](https://arxiv.org/html/2610.31955#A5.SS1.SSS0.Px4)\)\.

### I\.1Bayes filter

##### Singular values\.

We sweepλ0∈\[0\.55,0\.95\]\\lambda\_\{0\}\\in\[0\.55,0\.95\]in steps of0\.0250\.025atθ1=π/2\\theta\_\{1\}=\\pi/2, and compare the measuredσ0\\sigma\_\{0\}andσ1\\sigma\_\{1\}with Eq\. \([44](https://arxiv.org/html/2610.31955#A5.E44)\)\. The same sampled sequences are reused at everyλ0\\lambda\_\{0\}, so that sampling noise does not vary along the sweep\.

##### Switch point\.

For eachθ1\\theta\_\{1\}we measureσ0\\sigma\_\{0\}andσ1\\sigma\_\{1\}at seven values ofλ0\\lambda\_\{0\}, spaced0\.020\.02apart and centered on Eq\. \([46](https://arxiv.org/html/2610.31955#A5.E46)\)\. We locate the crossing by linear interpolation oflog⁡\(σ0/σ1\)\\log\(\\sigma\_\{0\}/\\sigma\_\{1\}\), which increases monotonically withλ0\\lambda\_\{0\}, and compare it with Eq\. \([46](https://arxiv.org/html/2610.31955#A5.E46)\)\.

### I\.2Transformer

##### Training\.

For eachθ1\\theta\_\{1\}we train transformers \(Appendix[F](https://arxiv.org/html/2610.31955#A6)\) on a grid of1111persistencesλ0=λ0∗\+Δ\\lambda\_\{0\}=\\lambda\_\{0\}^\{\*\}\+\\Delta, withΔ\\Deltafrom−0\.24\-0\.24to0\.160\.16in steps of0\.040\.04andλ0∗\\lambda\_\{0\}^\{\*\}from Eq\. \([46](https://arxiv.org/html/2610.31955#A5.E46)\)\. The grid is wide because the transformer’s switch need not sit at the filter’s\. At every grid point we train55models from independent seeds,220220models in total, and compute each model’s J\-lens at layersℓ=0\\ell=0andℓ=1\\ell=1on10001000freshly sampled sequences\. The transformer’s lens is in token\-score units, so its singular values are on a different scale from the filter’s \(Section[3\.2](https://arxiv.org/html/2610.31955#S3.SS2)\)\. We therefore compare only the location of the crossing with the theory\.

##### Estimating the crossing\.

For each model we take the differenceΔ​σ=σ0−σ1\\Delta\\sigma=\\sigma\_\{0\}\-\\sigma\_\{1\}and compute its mean and standard deviation over seeds at each grid point\. We then smooth these with a Savitzky–Golay filter \(a least\-squares quadratic over a window of55grid points\)\. The crossingλ^0∗\\hat\{\\lambda\}\_\{0\}^\{\*\}is where the smoothed meanμ\\muchanges sign, found by linear interpolation on that grid step, and we propagate the spread across seeds into an uncertainty on the crossing as

δ​λ^0∗=s⁡\(λ^0∗\)\|d​μ/d​λ0\|​\(λ^0∗\)\.\\delta\\hat\{\\lambda\}\_\{0\}^\{\*\}=\\frac\{s\(\\hat\{\\lambda\}\_\{0\}^\{\*\}\)\}\{\\left\\lvert\\mathrm\{d\}\\mu/\\mathrm\{d\}\\lambda\_\{0\}\\right\\rvert\(\\hat\{\\lambda\}\_\{0\}^\{\*\}\)\}\.\(51\)We compareλ^0∗±δ​λ^0∗\\hat\{\\lambda\}\_\{0\}^\{\*\}\\pm\\delta\\hat\{\\lambda\}\_\{0\}^\{\*\}at each layer with Eq\. \([46](https://arxiv.org/html/2610.31955#A5.E46)\)\.

## Appendix JSupplementary Results

![Refer to caption](https://arxiv.org/html/2610.31955v1/fig_supp_theory_vs_lens.png)Figure 11:Validation of the J\-lens theory for the Bayes filter\. X\-axis denotes the value predicted by the theory \(Eq\. \([19](https://arxiv.org/html/2610.31955#S3.E19)\)\), while the y\-axis denotes the value measured from the lens, for magnitudeaa, lag coherenceρτ\\rho\_\{\\tau\}and context coherenceρc\\rho\_\{c\}\(columns\) in each sweep \(rows, colored byλ\\lambda,θ\\thetaorψ\\psi\)\. There is only one visible data point for the fixed parameters in each sweep because the HMM is designed to fix these metrics in the Bayes filter\.![Refer to caption](https://arxiv.org/html/2610.31955v1/fig_supp_ranks.png)Figure 12:Validation of the J\-lens theory rank prediction for the Bayes filter and the transformer\. X\-axis denotes the predicted rank order of the singular vectors corresponding to each particular latent \(colored by their task parameterλ\\lambda,θ\\thetaorψ\\psi\), while the y\-axis denotes the measured rank in the J\-lens\.![Refer to caption](https://arxiv.org/html/2610.31955v1/fig_supp_lens_directions_magnitude.png)Figure 13:Alignment of the J\-lens directions with the encoding directions in the magnitude sweep\.AlignmentOiO\_\{i\}of the J\-lens singular vectors \(solid: input side𝒗i\\bm\{v\}\_\{i\}, hatched: output side𝒖i\\bm\{u\}\_\{i\}\), ranked by singular value, with the encoding directions𝑬\\bm\{E\}of each latent \(colored by persistenceλ\\lambda\) and with other patterns of the HMM \(gray\)\. Columns show the Bayes filter belief state \(left\), transformer layer 0 \(middle\) and layer 1 \(right\)\. Dashed line: alignment of a random direction\.Figure 14:Alignment of the J\-lens vectors with the decoding vectors for each latent in the magnitude sweep\.\(a\)Test decoding accuracy \(R2R^\{2\}\) for the Bayes filter belief statemtm\_\{t\}on each layer of the transformer, for each latent in the sweep\.\(b, c\)AlignmentOiO\_\{i\}of the J\-lens directions𝒗i\\bm\{v\}\_\{i\}with each latent’s own decoding directions𝑫i\\bm\{D\}\_\{i\}\(colored\) and with the other latents’ decoding directions \(gray\), for transformer layer 0 \(b\) and layer 1 \(c\)\.![Refer to caption](https://arxiv.org/html/2610.31955v1/fig_supp_lens_directions_lag.png)Figure 15:Alignment of the J\-lens directions with the encoding directions in the lag coherence sweep\.Latents are colored by angular velocityθ\\theta\. Panels as in Fig\.[13](https://arxiv.org/html/2610.31955#A10.F13)\.Figure 16:Alignment of the J\-lens vectors with the decoding vectors for each latent in the lag coherence sweep\.\(a\)Test decoding accuracy \(R2R^\{2\}\) for the Bayes filter belief statemtm\_\{t\}on each layer of the transformer, for each latent in the sweep\.\(b, c\)AlignmentOiO\_\{i\}of the J\-lens directions𝒗i\\bm\{v\}\_\{i\}with each latent’s own decoding directions𝑫i\\bm\{D\}\_\{i\}\(colored\) and with the other latents’ decoding directions \(gray\), for transformer layer 0 \(b\) and layer 1 \(c\)\.![Refer to caption](https://arxiv.org/html/2610.31955v1/fig_supp_lens_directions_context.png)Figure 17:Alignment of the J\-lens directions with the encoding directions in the context coherence sweep\.Latents are colored by key biasψ\\psi\. Panels as in Fig\.[13](https://arxiv.org/html/2610.31955#A10.F13)\.Figure 18:Alignment of the J\-lens vectors with the decoding vectors for each latent in the context coherence sweep\.\(a\)Test decoding accuracy \(R2R^\{2\}\) for the Bayes filter belief statemtm\_\{t\}on each layer of the transformer, for each latent in the sweep\.\(b, c\)AlignmentOiO\_\{i\}of the J\-lens directions𝒗i\\bm\{v\}\_\{i\}with each latent’s own decoding directions𝑫i\\bm\{D\}\_\{i\}\(colored\) and with the other latents’ decoding directions \(gray\), for transformer layer 0 \(b\) and layer 1 \(c\)\.Figure 19:J\-lens direction alignment with other features of the HMM that are not the latent states in the context sweep\.Black denotes the reveal stateBtB\_\{t\}: whether or not the latent state is revealed in the token, and the colors denote the key stateWt−1W\_\{t\-1\}\(whether the latent state is flipped\) for each key biasψ\\psi\.