驾驭不动点曲线：连续思维机中的图像分类

# Riding the Fixed-Point Curve: \titlebreakImage Classification in the Continuous Thought Machine
Source: [https://arxiv.org/html/2610.19220](https://arxiv.org/html/2610.19220)
\\jmlrpages

###### Abstract

The Continuous Thought Machine \(CTM\) is a recurrent network that attends to inputs and makes predictions based on the synchronization between its neurons\. On tasks spanning many domains, it displays emergent thought processes that have not been explained\. We report a number of findings about how the CTM classifies images, focusing on ImageNet\. Each thinking step is an attracting but strongly non\-normal map exhibiting balanced amplification \(Murphy & Miller, 2009\); while attraction is already present at CTM initialization, training highly promotes non\-normality\. By Banach’s theorem, the neural state has a locally unique attracting fixed point, which moves at every step, and the thought trajectory is the curve these fixed points trace, followed one tick behind\. Most of that curve’s motion is shared across images, and visible as a previously reported traveling wave of activity across neurons\. The attention schedule along the thought is largely generic: shuffling queries across images at every tick costs only 7% of classification accuracy, while image\-independent increments predict the next query withR2=0\.98R^\{2\}=0\.98\(held out\)\. Our results suggest that the CTM has found a single strategy to process all ImageNet images\.

††proceedings:NeurIPS: Accepted at NeurIPS††year:2026††workshop:Workshop on Symmetry and Geometry in Neural Representations \(NeurReps\), non\-archival## 1Introduction

The Continuous Thought Machine\([Darlow et al\., 2025](https://arxiv.org/html/2610.19220#bib.bib3)\)evolves a neural state for a fixed number of internal ticks against frozen image features, choosing at each tick where to attend and reading a prediction from the*synchronization*between neurons — a decayed second moment of their co\-activations\. It has been applied to classification, mazes, parity, sorting and reinforcement learning, and displays emergent behaviour, including low\-frequency traveling waves across its neurons, that has not been explained\. We take the official ImageNet checkpoint, without further training, and ask what kind of object the internal trajectory is\.

The CTM’s neural synchronization latent acts like a memory: when it is held fixed, a tick is an ordinary map whose local behaviour is read from one Jacobian; that map is contractive in spectrum but strongly non\-normal, so it has a locally unique attracting fixed point that the state never quite reaches\. Because the memory changes each tick, the target moves, and inference is a curve of moving fixed points followed one step behind \(§[3](https://arxiv.org/html/2610.19220#S3)\)\. Most of the curve’s motion is shared across images, and attention follows a schedule that is shared as well \(§[4](https://arxiv.org/html/2610.19220#S4)\)\. Ordering neurons by the phase of that curve reproduces a traveling wave of neural activity reported by the CTM authors \(§[5](https://arxiv.org/html/2610.19220#S5)\), and the image’s class rides on the curve from the first tick \(§[6](https://arxiv.org/html/2610.19220#S6)\)\. We then turn to the memory’s role in sharpening the prediction \(§[7](https://arxiv.org/html/2610.19220#S7)\) and to the contrast with smaller MNIST and CIFAR models \(§[8](https://arxiv.org/html/2610.19220#S8)\)\.

## 2The Continuous Thought Machine

We use the official ImageNet CTM \(D=4096D\{=\}4096neurons,T=50T\{=\}50ticks, a ResNet backbone,L=196L\{=\}196attention tokens\); it reaches0\.720\.72top\-1 on our validation subsample \(one image from each of the10001000ImageNet classes of the validation set\)\. All ImageNet results in this paper are on this subsample or subsets of it, except for class identity \(§[6](https://arxiv.org/html/2610.19220#S6)\), which uses its own sample of5050images per class; sample sizes and 95% intervals for every headline number are in Appendix[D](https://arxiv.org/html/2610.19220#A4)\.

We follow the notation of[Darlow et al\. \(2025\)](https://arxiv.org/html/2610.19220#bib.bib3), generally writing matrices in upper\-case and vectors in lower\-case, with tick indices as superscripts\. As illustrated in Fig\.[1](https://arxiv.org/html/2610.19220#S2.F1), the image is first passed through a convolutional backbone network, which extracts features into a key/value bank K:

K=proj⁡\(flat⁡\(bϕ​\(x\)\)\)∈ℝL×d,K\\;=\\;\\mathrm\{proj\}\\big\(\\mathrm\{flat\}\(b\_\{\\phi\}\(x\)\)\\big\)\\;\\in\\;\\mathbb\{R\}^\{L\\times d\},\(1\)this is a matrix ofLLtokens of widthdd\. K is computed once and held fixed: the model never re\-observes the input\. It then runsTTinternal ticks againstKK, carrying a vector of post\-activationszt∈ℝDz^\{t\}\\in\\mathbb\{R\}^\{D\}—DDneurons in total — from a learned, input\-independent initial valuez1z^\{1\}\. One tick proceeds in four steps\.

![Refer to caption](https://arxiv.org/html/2610.19220v1/figures/fig1_architecture.png)Figure 1:One tick of the CTM\.After a single backbone pass the model iterates against a frozen token bank: a queryqtq^\{t\}read from the action\-side synchronization slice drives attention; a synapse model mixes the attention outputoto^\{t\}with the stateztz^\{t\}; per\-neuron networks produce the next statezt\+1z^\{t\+1\}; the synchronization matrixSt\+1S^\{t\+1\}is updated and its output slice is read to logitsyt\+1y^\{t\+1\}\.### Attending\.

The current post\-activations are summarized by the action\-side latent representationSactiontS^\{t\}\_\{\\mathrm\{action\}\}\(selected entries ofStS^\{t\}\), which a linear mapWinW\_\{\\mathrm\{in\}\}turns into a queryqtq^\{t\}that modulates the data by cross\-attention:

qt=Win​Sactiont∈ℝd,ot=softmax⁡\(qt​K⊤d\)​K∈ℝd\.q^\{t\}=W\_\{\\mathrm\{in\}\}\\,S^\{t\}\_\{\\mathrm\{action\}\}\\;\\in\\;\\mathbb\{R\}^\{d\},\\qquad o^\{t\}\\;=\\;\\mathrm\{softmax\}\\\!\\left\(\\frac\{q^\{t\}K^\{\\\!\\top\}\}\{\\sqrt\{d\}\}\\right\)K\\;\\in\\;\\mathbb\{R\}^\{d\}\.\(2\)This is the sole coupling between the thought and the image:KKis fixed, and the query depends on the trajectory only throughSactiontS^\{t\}\_\{\\mathrm\{action\}\}\.

### Updating the state\.

The attention output is concatenated with the current post\-activations and passed through the*synapse model*fθsynf\_\{\\theta\_\{\\mathrm\{syn\}\}\}, a single gated linear layer shared across neurons, giving the*pre\-activations*at∈ℝDa^\{t\}\\in\\mathbb\{R\}^\{D\}; this is the only step in which neurons exchange information\. The*neuron\-level models*gθdg\_\{\\theta\_\{d\}\}map pre\-activations to the next post\-activations, each neuron seeing only its own historyA\(i\)t∈ℝMA^\{t\}\_\{\(i\)\}\\in\\mathbb\{R\}^\{M\}through its own private MLP\. Cross\-neuron mixing is thus confined tofθsynf\_\{\\theta\_\{\\mathrm\{syn\}\}\}, whilegθdg\_\{\\theta\_\{d\}\}gives each neuron an individual temporal response — so the timing of neurons relative to one another becomes a carrier of information, which is what the synchronization matrix later reads:

at=fθsyn​\(\[ot;zt\]\),At=\[at−M\+1,…,at\]∈ℝD×M,zt\+1=gθd​\(At\)\.a^\{t\}=f\_\{\\theta\_\{\\mathrm\{syn\}\}\}\\big\(\[\\,o^\{t\};\\,z^\{t\}\\,\]\\big\),\\qquad A^\{t\}=\\big\[a^\{t\-M\+1\},\\dots,a^\{t\}\\big\]\\in\\mathbb\{R\}^\{D\\times M\},\\qquad z^\{t\+1\}=g\_\{\\theta\_\{d\}\}\\big\(A^\{t\}\\big\)\.\(3\)

### Accumulating the memory\.

The neural statesz1,…,ztz^\{1\},\\dots,z^\{t\}, stacked intoZt∈ℝt×DZ^\{t\}\\in\\mathbb\{R\}^\{t\\times D\}are used to build the*synchronization matrix*: it is the running second moment between neurons,

St=1t​\(Zt\)⊤​Zt=1t​∑τ≤tzτ​\(zτ\)⊤∈ℝD×D\.S^\{t\}\\;=\\;\\tfrac\{1\}\{t\}\\big\(Z^\{t\}\\big\)^\{\\\!\\top\}Z^\{t\}\\;=\\;\\tfrac\{1\}\{t\}\\sum\_\{\\tau\\leq t\}z^\{\\tau\}\\big\(z^\{\\tau\}\\big\)^\{\\\!\\top\}\\;\\in\\;\\mathbb\{R\}^\{D\\times D\}\.\(4\)

### Reading a decision from it\.

The model does not storeZtZ^\{t\}: it keepsStS^\{t\}as a running accumulator, with an exponential decayri​j=exp⁡\(−δi​j\)r\_\{ij\}=\\exp\(\-\\delta\_\{ij\}\)learned per neuron pair — one scalar each, fixed after training and identical for every input and every tick\. From the synchronization matrixStS^\{t\}, two latent representations are extracted,SouttS^\{t\}\_\{\\mathrm\{out\}\}andSactiontS^\{t\}\_\{\\mathrm\{action\}\}, which are simply fixed slices of its entries and respectively determine the prediction \(out\) and the next features to attend to \(action\)\. The out\-side latent representationSouttS^\{t\}\_\{\\mathrm\{out\}\}is read out by a linear mapWoutW\_\{\\mathrm\{out\}\}to that tick’s logitsyty^\{t\}and certainty𝒞t\\mathcal\{C\}^\{t\}, and the action\-sideSactiontS^\{t\}\_\{\\mathrm\{action\}\}is projected through a linear layer to form the next query:

yt=Wout​Soutt,𝒞t=1−H⁡\(yt\),qt\+1=Win​Sactiont\.y^\{t\}=W\_\{\\mathrm\{out\}\}\\,S^\{t\}\_\{\\mathrm\{out\}\},\\qquad\\mathcal\{C\}^\{t\}=1\-H\(y^\{t\}\),\\qquad q^\{t\+1\}=W\_\{\\mathrm\{in\}\}\\,S^\{t\}\_\{\\mathrm\{action\}\}\.\(5\)
Att=1t=1there is no history to summarize:z1z^\{1\}and the initial pre\-activation trace are learned parameters identical for every image, so the first attention read is the model’s learned prior over where to look\. This matters later: because every trajectory starts from the same point, where a trajectory ends and the direction it travelled in are two descriptions of the same thing\.

### Training objective\.

TTis the thought budget: fixed in advance, identical at training and inference, with no halting mechanism and no early exit\. The loss is a*dual\-tick*cross\-entropy, averaging the cross\-entropy at whichever tick had the lowest loss and at whichever had the highest certainty, both chosen per input\. Nothing weights later ticks, requires an answer beforeTT, or rewards the synchronization statistic directly\. Throughout, each sample’s decision tick is its tick of highest certainty,tdec=arg⁡maxt⁡𝒞tt\_\{\\mathrm\{dec\}\}=\\arg\\max\_\{t\}\\mathcal\{C\}^\{t\}, as in[Darlow et al\. \(2025\)](https://arxiv.org/html/2610.19220#bib.bib3), and every decision\-dependent quantity is computed over\[1,tdec\]\[1,t\_\{\\mathrm\{dec\}\}\]\. Sample sizes differ between analyses because some \(materialized Jacobians, Schur forms\) are expensive; the sample size is stated with each result\.

### Design choices specific to ImageNet\.

Under the official ImageNet checkpoint the synchronization memory uses*random pairing*: rather than forming the fullD×DD\\times Dmatrix, a fixed set of neuron pairs\(i,j\)\(i,j\)is drawn once at initialization and only those entries are tracked, so the action and output slices are long vectors rather than a dense matrix\. The checkpoint tracks81968196pairs for the output slice and20482048for the action slice, againstD=4096D=4096neurons\. The sampling is broad rather than selective: a neuron takes part in4\.04\.0output pairs on average and98%98\\%of neurons appear in at least one, so the readout draws on the population as a whole rather than on a privileged subset\. None of this affects any state\-space quantity used here\.

### What is known about how the CTM forms decisions\.

The original CTM paper reports several findings on ImageNet\-1K\([Darlow et al\., 2025](https://arxiv.org/html/2610.19220#bib.bib3)\)at a budget of 50 ticks\. Most images pass a0\.80\.8*certainty*threshold within ten ticks with a long tail out to fifty, so the budget could in principle be halted early, as in adaptive\-computation\-time recurrent networks\([Graves, 2016](https://arxiv.org/html/2610.19220#bib.bib7)\); the logits are unusually well calibrated without post\-hoc adjustment; and reading the prediction at the most certain tick beats reading the current one beyond roughly fifteen ticks, while averaging logits does worst—so the model passes through low\-certainty, sometimes incorrect states before settling\. That certainty rises—equivalently, that output entropy falls—along the thought in most cases is therefore already known, and is in fact built into the CTM’s objective\. The attention trajectory over image features also emerges through training, and visual inspection shows diverse, multiscale dynamics with low\-frequency traveling waves across a learned feature map arising with no driving signal—a phenomenon with a long history in cortex\([Muller et al\., 2018](https://arxiv.org/html/2610.19220#bib.bib12)\)\. One further property of the released ImageNet checkpoint bears on what follows: its learned decay rates are negligible over the5050\-tick budget, soSoutS\_\{\\text\{out\}\}is in effect a running average of the state’s co\-activations, a deterministic quadratic functional ofz1,…,ztz^\{1\},\\dots,z^\{t\}\. We therefore analyze the stateztz^\{t\}, where the recurrence’s learned per\-tick maps and its fixed points live\.

## 3An attracting, non\-normal fixed\-point curve

What governs the state’s motion? One way to study recurrent network dynamics is to locate the fixed points of the recurrence and linearize there\([Sussillo and Barak, 2013](https://arxiv.org/html/2610.19220#bib.bib15)\)—we hold the memory context𝒜t\\mathcal\{A\}^\{t\}fixed, which turns the tick into an ordinary mapzt\+1=f⁡\(zt,𝒜t\)z^\{t\+1\}=f\(z^\{t\};\\mathcal\{A\}^\{t\}\)on state space\. We reconstruct that map exactly \(a single reconstructed step reproduces the model’s next state to<10−6<\\\!10^\{\-6\}\) and iterate it: it converges at30553055of30723072image–tick pairs from256256images \(residual<10−6<\\\!10^\{\-6\}; App\.[A](https://arxiv.org/html/2610.19220#A1)\), and at every one of these fixed pointsz⋆z^\{\\star\}the JacobianJJ, read by autograd power iteration, has spectral radius well below one \(ρ⁡\(J\)=0\.15\\rho\(J\)=0\.15median, 95% interval0\.140\.14–0\.160\.16, max0\.950\.95\)\. By Banach’s theorem, applied in an adapted norm \(App\.[A](https://arxiv.org/html/2610.19220#A1)\), the fixed point is locally unique and attracting: the state has one well\-defined target at each tick\. Beyond the neighbourhood the theorem covers,34563456restarts from random states and from other images’ states all converged to the same fixed point, and we found no second basin \(App\.[A](https://arxiv.org/html/2610.19220#A1)\)\.

The map is nevertheless strongly*non\-normal*: although every eigenvalue decays, the largest singular value exceeds one \(σmax​\(J\)=3\.2\\sigma\_\{\\max\}\(J\)=3\.2median, up to7474\), so it briefly stretches a few directions even while contracting overall \(Fig\.[2](https://arxiv.org/html/2610.19220#S3.F2)a\)\. Estimation details, the per\-fixed\-point ratioσmax​\(J\)/ρ​\(J\)\\sigma\_\{\\max\}\(J\)/\\rho\(J\), and a finite\-difference confirmation of the stretching are in Appendix[A](https://arxiv.org/html/2610.19220#A1)\.

![Refer to caption](https://arxiv.org/html/2610.19220v1/figures/fig2_spectrum_imagenet.png)Figure 2:Attracting but non\-normal, and one tick late\.\(a\)At each fixed point the spectral radiusρ⁡\(J\)\\rho\(J\)stays below one \(the map is stable and the fixed point unique\) while the operator normσmax​\(J\)\\sigma\_\{\\max\}\(J\)exceeds one \(it is not a Euclidean contraction\); the gap is the non\-normality\.\(b\)State and fixed point projected on the leading axis of the fixed\-point curve, withzt−1⋆z^\{\\star\}\_\{t\-1\}overlaid: the state’s trajectory is the fixed\-point curve delayed by one tick\.\(c\)The same claim measured: the mean distance‖zt−zt−k⋆‖\\\|z^\{t\}\-z^\{\\star\}\_\{t\-k\}\\\|to the fixed pointkkticks earlier, against candidate lagkk, is minimized atk=1k=1\(5\.95\.9, against41\.441\.4atk=0k=0and40\.740\.7atk=2k=2\)\.### What training adds\.

Attraction itself comes with the architecture: an untrained CTM also hasρ⁡\(J\)<1\\rho\(J\)<1at every fixed point \(median0\.050\.05\), as does a*random core*—the trained backbone with the recurrence at initialization, so the recurrence sees the same image features \(median0\.240\.24\)\. Against this matched\-input control, training lowers the spectral radius \(0\.24→0\.150\.24\\to 0\.15\) and raises the non\-normalityσmax​\(J\)/ρ​\(J\)\\sigma\_\{\\max\}\(J\)/\\rho\(J\)from about77to about2323, while the untrained map is a plain Euclidean contraction \(σmax​\(J\)<1\\sigma\_\{\\max\}\(J\)<1everywhere\)\. Training also concentrates each neuron model’s sensitivity on the newest entry of its pre\-activation window, whereztz^\{t\}enters \(2\.6×2\.6\\timesits uniform share, against uniform at initialization\), so the contraction is not an artifact ofztz^\{t\}entering only one ofMMwindow slots\.

### One tick behind\.

Because the memory changes with every tick, the target moves, and the trajectory is simply the curve these fixed points trace, followed one update behind \(Fig\.[2](https://arxiv.org/html/2610.19220#S3.F2)b\)\. The state is not on the current fixed point but on the previous one: the mean distance‖zt−zt−k⋆‖\\\|z^\{t\}\-z^\{\\star\}\_\{t\-k\}\\\|is minimized atk=1k=1and is then seven times smaller than the distance to the current fixed point \(5\.95\.9versus41\.441\.4, over128128images; Fig\.[2](https://arxiv.org/html/2610.19220#S3.F2)c\)\. The offset points backwards along the fixed point’s own motion \(mediancos⁡\(zt−zt⋆,zt−1⋆−zt⋆\)=0\.995\\cos\(z^\{t\}\-z^\{\\star\}\_\{t\},\\,z^\{\\star\}\_\{t\-1\}\-z^\{\\star\}\_\{t\}\)=0\.995\) and measures a full step of it \(median ratio1\.011\.01\), so the state trails the moving attractor by one update rather than sitting on it \(Fig\.[3](https://arxiv.org/html/2610.19220#S3.F3)shows the trailing state\)\. This lag is largely structural:ztz^\{t\}is one contraction step fromzt−1⋆z^\{\\star\}\_\{t\-1\}, and the shrink factor across one step \(median0\.110\.11over15361536image–tick pairs\) is whatρ⁡\(J\)\\rho\(J\)at the same fixed points predicts \(median0\.140\.14\); what is not structural is how far the fixed point moves each tick, which keeps the state a full step behind \(App\.[A](https://arxiv.org/html/2610.19220#A1)\)\.

![Refer to caption](https://arxiv.org/html/2610.19220v1/figures/fig3_fixed_point_curve_imagenet.png)Figure 3:The fixed\-point curve\.The recurrence’s residual field on the plane spanned by the path of successive fixed points, shown across ticks for one held\-out image\. Color is residual magnitude, streamlines the in\-plane flow; the red curve is the path of the fixed pointz⋆z^\{\\star\}, the star its current location, and the black dot the state, which stays within about one step of the moving attractor\.
### The non\-normality is balanced amplification\.

Schur\-triangularizingJ=U​T​U∗J=UTU^\{\*\}, almost the whole operator lives in the strictly upper\-triangular, feed\-forward part: that fraction — which equals Henrici’s departure from normality — is0\.990\.99, against0\.710\.71for a Ginibre matrix and0\.930\.93for the random core \(equivalently, the eigenvalues carry2\.0%2\.0\\%of∥J∥F2\\lVert J\\rVert\_\{F\}^\{2\}, against14%14\\%for the random core\), while the antisymmetric part sits at the random\-matrix level\. Withρ⁡\(J\)<1\\rho\(J\)<1, this triad of near\-maximal feed\-forward non\-normality, stable eigenvalues and chance\-level rotation is the structure[Murphy and Miller \(2009\)](https://arxiv.org/html/2610.19220#bib.bib13)identified as*balanced amplification*: a hidden feed\-forward chain that amplifies activity in transit while every mode decays \(App\.[B](https://arxiv.org/html/2610.19220#A2)\)\.

## 4Attention follows a shared schedule

Where the model attends is a program shared across images, not an image\-specific search\. Attention is selective: it concentrates on about9696of the196196tokens \(exponential of its entropy\), with its most\-attended token carrying11×11\\timesthe uniform weight\.Ablations\(Table[1](https://arxiv.org/html/2610.19220#S4.T1)\): freezing the query at its first\-tick value, so the model never advances its schedule, removes34%34\\%of accuracy, and flattening attention removes most of it \(−80%\-80\\%\); handing each image*another image’s*query at the matching tick costs only7%7\\%\(10001000images\)\. The progression is essential; its image\-specific content matters far less\.Factorization:substituting the population\-average query at each tick reproduces the image’s own attention map at cosine0\.9360\.936\(Fig\.[4](https://arxiv.org/html/2610.19220#S4.F4)\) — a generic schedule pointed at the image’s features\.Prediction:the current query plus the shared average increment forecasts the next attention map at cosine0\.9960\.996, against0\.9480\.948for assuming attention stays put \(400400held\-out images\): the next glance is predictable before the model takes it \(App\.[C](https://arxiv.org/html/2610.19220#A3)\)\. This is a one\-step\-ahead forecast: the predictor consumes the current query, so it anticipates the next glance from the current one, not the whole scan from the image alone\.

What the schedule does to the curve\.Most of the fixed point’s motion is common to all images: the mean curve over images carries77%77\\%of its squared motion per tick\. Rolling the model out with an imposed query and recomputing the fixed points \(9696images\), freezing the query at its first\-tick value leaves97%97\\%of the fixed point’s motion, both overall and in each image’s own part\. Handing each image another image’s schedule moves its curve about a third of the way towards that image’s curve \(0\.350\.35of the distance between the two\), while each step keeps the direction of the image’s own \(cosine0\.860\.86once the shared curve is removed, against0\.000\.00with the other image’s step\)\. The schedule thus shifts each image’s curve, which matters for the decision, while the curve’s motion comes mostly from the recurrence and its memory\.

Table 1:Query/attention ablations on ImageNet \(top\-1 attdect\_\{\\mathrm\{dec\}\}; chance0\.0010\.001\)\. Freezing the query’s schedule is costly; replacing its image\-specific content costs little\.n=1000n\{=\}1000images; 95% intervals in Appendix[D](https://arxiv.org/html/2610.19220#A4)\. Without attention the image never enters after the backbone andz1z^\{1\}is a learned constant, so every image follows the same trajectory and receives the same class; accuracy is that class’s frequency in the sample\.![Refer to caption](https://arxiv.org/html/2610.19220v1/figures/fig4_attention_scan_imagenet.png)Figure 4:The attention scan is a shared schedule\.For two held\-out images \(input, first column\), the model’s true attention \(first row of each pair\) and the attention produced by the*shared*population\-average query \(second row\), across ticks; one color scale per image\. The shared schedule recovers the image\-specific scan\.
## 5The fixed point curve is a traveling wave

The curve \(§[3](https://arxiv.org/html/2610.19220#S3)\) accounts for a phenomenon the CTM authors report: low\-frequency traveling waves across a UMAP of activation profiles, arising with no driving signal\. They are the moving fixed point made visible\. Ordering theDDneurons by the phase of the fixed\-point trajectoryz⋆​\(At\)z^\{\\star\}\(A^\{t\}\)or by their own phase recovers the same traveling wave \(Fig\.[5](https://arxiv.org/html/2610.19220#S5.F5), left vs middle\)\. This ordering recovers the authors’ activation\-phase ordering \(peak\-tick Spearman0\.820\.82, circular phase correlation0\.880\.88; medians over128128images\) — as expected, since the state tracksz⋆z^\{\\star\}within about one step, so activation phase is essentially fixed\-point phase\. The neurons’ peak activation times are spread across essentially the whole window — standard deviation15\.015\.0of5050ticks for both the fixed points and the activations, against≈14\.4\{\\approx\}14\.4for a uniform spread — so neurons traverse the shared motion in staggered phase rather than together, which is what makes the front travel rather than stand\. Becauseρ⁡\(J\)<1\\rho\(J\)<1the fixed point moves smoothly and the resulting pattern is a transient propagating pulse rather than a sustained standing wave, matching the low\-frequency, driverless character of the reported phenomenon\.

![Refer to caption](https://arxiv.org/html/2610.19220v1/figures/fig5_traveling_wave_imagenet.png)Figure 5:The traveling wave is the moving fixed point\.One held\-out image; neurons ordered by the phase ofz⋆​\(At\)z^\{\\star\}\(A^\{t\}\)\.Left:activationsztz^\{t\}over ticks — the reported wave\.Right:the fixed points under the same ordering, indistinguishable from the left\.That the two panels match follows largely from the one\-step lag of §[3](https://arxiv.org/html/2610.19220#S3)\. The wave is not, however, coupled to the attention schedule: across128128images, the cross\-validated canonical correlation between the fixed point’s leading temporal modes and the schedule’s \(0\.520\.52\) does not exceed a permutation null \(95th percentile0\.540\.54;p=0\.17p=0\.17\)\. The fixed\-point trajectory is also higher\-dimensional than a single front: its top\-two temporal modes hold only0\.380\.38of the variance, on average over these images\.

### The cascade account, and why it fails\.

The balanced\-amplification structure of §[3](https://arxiv.org/html/2610.19220#S3)suggests a compelling alternative account of the waves\. If the tick map is an almost\-pure feed\-forward cascade that also organizes the observed dynamics, then the traveling waves should appear when neurons are ordered by their position in that cascade\. They do not\. Ordering neurons by their position inJJ’s feed\-forward chain — each neuron’s mean index in the Schur basisJ=U​T​U∗J=UTU^\{\*\}, weighted by its participation∑kUi​k2\\sum\_\{k\}U\_\{ik\}^\{2\}, usingJJat the mid\-trajectory tick — peak activation time does not advance: the rank correlation between cascade position and peak tick is−0\.08\-0\.08over allDDneurons, and between cascade position and activation phase\+0\.01\+0\.01\(medians over1616images\)\. The reason is structural:JJ— and with it its Schur ordering — is redefined at every tick, so no single instantaneous cascade can organize a trajectory that runs over all ticks; and because the state sits on the moving fixed point, the bulk of the activation is carried by where that fixed point is, while the balanced\-amplification transient governs only the one\-step residual\. The traveling waves are thus a signature of the moving fixed\-point curve, not of the instantaneous feed\-forward cascade; the balanced\-amplification structure ofJJstands as a property of the operator whose functional role is left open\.

## 6The curve carries class identity

Every trajectory starts from the same learnedz1z^\{1\}, so class identity must enter the curve along the way\. It does so at once\.

### Images and recordings\.

5050classes with5050validation images each \(25002500images\), drawn at random from the10001000ImageNet classes\. The model is rolled out once per image, recording the stateztz^\{t\}, the fixed pointzt⋆z^\{\\star\}\_\{t\}\(iterated to relative tolerance10−510^\{\-5\}\) and the readout memorySouttS^\{t\}\_\{\\mathrm\{out\}\}at1515ticks\.

### When class becomes decodable\.

At each tick, a linear probe \(standardization,128128principal components, multinomial logistic regression;55\-fold cross\-validation\) decodes class from each representation\. With about4040training images per class the probes are a lower bound: the model’s own readout, trained on all of ImageNet, reaches0\.920\.92where the probes plateau near0\.860\.86\. Comparisons are therefore made between representations under the same probe\. The fixed point leads:0\.82±0\.020\.82\\pm 0\.02at tick11\(5050\-way; chance0\.020\.02\) against0\.79±0\.020\.79\\pm 0\.02forztz^\{t\}and0\.79±0\.030\.79\\pm 0\.03forSouttS^\{t\}\_\{\\mathrm\{out\}\}, and above the token\-bank mean of the image features \(0\.75±0\.020\.75\\pm 0\.02\)\. The state’s accuracy at tickt\+1t\{\+\}1tracks the fixed point’s at ticktt\(0\.810\.81vs0\.820\.82,0\.830\.83vs0\.840\.84,0\.840\.84vs0\.850\.85over ticks11–44\), the one\-tick lag of §[3](https://arxiv.org/html/2610.19220#S3)seen in class information\. The fixed point and the state peak near0\.860\.86by tick44–66, andSouttS^\{t\}\_\{\\mathrm\{out\}\}near0\.870\.87by tick1010; thereafterSouttS^\{t\}\_\{\\mathrm\{out\}\}holds its level whilezt⋆z^\{\\star\}\_\{t\}andztz^\{t\}drift slightly lower \(to about0\.830\.83\), with transient dips \(ticks2525and3636\) that we do not interpret\. The model’s readout, restricted to the5050classes, rises from0\.730\.73at tick11to90%90\\%of its plateau at tick33, as the memory accumulates, and plateaus near0\.920\.92from tick1313\.

### Where class sits relative to the curve\.

We writezi,t⋆=ct\+dk⁡\(i\),t\+ri,tz^\{\\star\}\_\{i,t\}=c\_\{t\}\+d\_\{k\(i\),t\}\+r\_\{i,t\}, withctc\_\{t\}the mean over images \(the shared curve\),dk,td\_\{k,t\}the offset of classkk, andri,tr\_\{i,t\}the image residual\. Over images and ticks, the shared curve carries0\.660\.66of the variance, class offsets0\.1050\.105\(against0\.0070\.007when labels are shuffled\) and the image residual0\.240\.24\. The class offsets lie almost entirely outside the curve’s leading two\-dimensional plane \(0\.5%0\.5\\%of their variance inside it, against0\.05%0\.05\\%for a random plane\): classes do not occupy different regions of the curve but ride the same curve, displaced sideways\. The offsets are not fixed in direction either: their cosine with the final offset is near zero at first, about0\.60\.6over ticks66–1616, falls to about0\.10\.1near tick2525, and recovers by the end\. At the final tick the mean class offset is0\.210\.21of the distance travelled along the curve\.

## 7What the synchronization memory does

Across the ticks the model grows more certain: normalized predictive entropy \(H/ln⁡1000H/\\ln 1000\) falls from0\.520\.52to0\.050\.05\(raw3\.563\.56to0\.320\.32nats\) by the last tick, most of it within the first ten ticks \(mediantdec=39t\_\{\\mathrm\{dec\}\}=39of5050over10001000images; Fig\.[6](https://arxiv.org/html/2610.19220#S7.F6)a\)\. This much is expected—certainty rising over the thought is reported by the CTM’s authors and is written into their loss \(§[2](https://arxiv.org/html/2610.19220#S2.SS0.SSS0.Px7)\)\. Our question is what produces it, and the answer is the readout’s memory\. The decision is read from the synchronization matrixSouttS^\{t\}\_\{\\mathrm\{out\}\}, a decay\-weighted running average of pairwise co\-activations \(Eq\.[4](https://arxiv.org/html/2610.19220#S2.E4)\); in this checkpoint the learned decays are effectively off over the budget, so it is close to a uniform running mean of the state’s co\-activations\. Like a memory it*accumulates and settles*: its per\-tick increment∥Δ​Soutt∥\\lVert\\Delta S^\{t\}\_\{\\mathrm\{out\}\}\\rVertfalls and then stays low, even as the state stays live—∥Δ​zt∥\\lVert\\Delta z^\{t\}\\rVertdoes not shrink but grows, the traveling activity of §[5](https://arxiv.org/html/2610.19220#S5)\. Splitting each tick’s change in entropy into the change that would occur if the memory were frozen and only the state moved, and the change caused by the memory updating with the state held fixed, the memory term accounts for all of the drop over\[1,tdec\]\[1,t\_\{\\mathrm\{dec\}\}\]\(it exceeds it:−0\.43\-0\.43against a total of−0\.32\-0\.32in normalized entropy,256256images; 95% interval for its share111111–159%159\\%\), while the state’s own motion does not help and slightly raises entropy \(\+0\.11\+0\.11\); to first order its step is nearly orthogonal to the entropy\-reducing direction \(\|cos⁡\(Δ​zt,−∇zH\)\|<0\.06\|\\\!\\cos\(\\Delta z^\{t\},\-\\nabla\_\{z\}H\)\|<0\.06at every tick\)\.

![Refer to caption](https://arxiv.org/html/2610.19220v1/figures/fig6_memory_controls_imagenet.png)Figure 6:What sharpens the prediction\.\(a\)Normalized entropy over ticks, intact and under each manipulation \(512512images\)\.\(b\)Fraction of images whose predicted class differs from its final value\.### Causal tests\.

On512512images \(Fig\.[6](https://arxiv.org/html/2610.19220#S7.F6)\), three manipulations separate what the memory does from what the state does\.*\(i\) Frozen state\.*If the state is held fixed after tickkkwhile the memory keeps accumulating it, entropy keeps falling fork=4k\{=\}4\(to0\.0220\.022at tick5050, below the intact0\.0470\.047\), yet top\-1 at the last tick collapses from0\.730\.73to0\.280\.28; fork=10k\{=\}10or2020entropy stays roughly level and final top\-1 falls to0\.020\.02–0\.030\.03\. Accumulating one repeated state makes the readout confident but wrong: the decision is read from the spread of states along the trajectory, not from any single state\.*\(ii\) Shuffled ticks\.*Feeding the same states to the memory in random order changes intermediate entropy but not the final readout \(0\.720\.72either way\), as expected with inert decays: the memory is an order\-free average of co\-activations\.*\(iii\) Fixed norm\.*Rescaling the logits to a common norm removes the descent: normalized entropy stays between0\.030\.03and0\.120\.12from the first tick on\. The entropy fall is therefore largely the growth of the accumulated readout’s magnitude, better read as a growing gain than as evidence accumulating\. The decision itself settles earlier: the predicted class still differs from its final value for38%38\\%of images at tick44,18%18\\%at tick88,12%12\\%at tick1616and8%8\\%at tick4040\. Without attention, every image receives the same class \(here “football helmet”\), sincez1z^\{1\}is a learned constant and no image information enters\.

## 8Smaller models

TwoD=128D\{=\}128models, one on MNIST and one on CIFAR\-10, separate what is intrinsic to CTM inference from what ImageNet elicits\. Both have a dense synchronization matrix: unlike the ImageNet checkpoint’s random pairing, every tracked entry is a pair among the first1616neurons, giving action and output slices of136136entries each\. The CIFAR model uses a ResNet\-9 backbone, which lifts its accuracy to0\.840\.84from0\.640\.64with the original backbone and lets us treat it as a competent classifier rather than a toy\.

### The curve is universal, the transient amplification is not\.

Measured the same way as the ImageNet model, both have a spectral radius near a tenth at the fixed point \(ρ⁡\(J\)=0\.10\\rho\(J\)=0\.10on MNIST,0\.0680\.068on CIFAR\), but a largest singular value below one throughout on MNIST \(σmax​\(J\)=0\.49\\sigma\_\{\\max\}\(J\)=0\.49, max0\.770\.77\) and in the median on CIFAR \(0\.360\.36, max3\.063\.06; about one CIFAR fixed point in ten does stretch\)—these maps are Euclidean contractions, in contrast to the ImageNet map’s several\-fold stretch \(Table[2](https://arxiv.org/html/2610.19220#S8.T2)\)\. They are nonetheless non\-normal: already at initialization \(normal share∑\|λ\|2/∥J∥F2≈0\.29\\sum\|\\lambda\|^\{2\}/\\lVert J\\rVert\_\{F\}^\{2\}\\approx 0\.29, against0\.500\.50for a Ginibre matrix\), and more so after training \(0\.160\.16on MNIST,0\.0950\.095on CIFAR, against0\.0200\.020on ImageNet\)\. Training thus increases non\-normality at every scale, more at larger scale, but only on ImageNet does it reach transient amplification\.

Table 2:Cross\-model comparison\. The attracting curve \(ρ⁡\(J\)<1\\rho\(J\)<1\) is universal; transient amplification \(σmax​\(J\)\>1\\sigma\_\{\\max\}\(J\)\\\!\>\\\!1\) and query dependence emerge only on ImageNet\. “attn\.” is the entropy of the head\-averaged attention weights over image patches, as a fraction of its maximumln⁡L\\ln L\(11= uniform\); “frozen” and “shuffled” are the accuracy changes from freezing the query at its first\-tick value and from giving each image another image’s query\.
### No attention program, and a briefly aligned descent\.

Their attention is close to uniform on MNIST \(H/Hmax=0\.97H/H\_\{\\max\}=0\.97, about4444of4949patches\) and moderately selective on CIFAR \(0\.900\.90, about4242of6464\), but on both, freezing or swapping the query moves accuracy by at most a few tenths of a percent, and if anything upwards \(frozen/swapped:\+0\.0%\+0\.0\\%/\+0\.0%\+0\.0\\%on MNIST,\+0\.3%\+0\.3\\%/\+0\.2%\+0\.2\\%on CIFAR\), so the query plays no real role: where they attend does not depend on the thought\. Only removing or flattening attention hurts\. Entropy falls reliably on both, and there the state’s step does line up with the entropy gradient over part of the thought, then turns away from it while entropy keeps falling; the memory term takes over past the crossing\. On CIFAR the alignment is immediate and brief \(peak\+0\.69\+0\.69at tick11, crossing at tick55\), while on MNIST it builds up and lasts longer \(peak\+0\.65\+0\.65at tick99, crossing at tick1717\); on ImageNet the early, aligned phase is absent, and the memory carries the descent from the start\. In all cases the learned forgetting rates confirm that the memory keeps integrating rather than saturating: the earliest contributions still carry substantial weight at the decision tick \(median retention0\.900\.90on MNIST,0\.630\.63on CIFAR\)\. Together these place the small models cleanly: the same skeleton, without the transient amplification or the attention program that scale brings\.

## 9Discussion

CTM inference on ImageNet tracks a curve of fixed points, largely shared across images, while attention follows a schedule that is shared as well\. The attracting, non\-normal maps we uncover suggest how the CTM transiently amplifies activity while staying stable\. Non\-normal recurrence as a substrate for transient amplification and memory has been studied in neural circuits\([Ganguli et al\., 2008](https://arxiv.org/html/2610.19220#bib.bib4);[Murphy and Miller, 2009](https://arxiv.org/html/2610.19220#bib.bib13);[Goldman, 2009](https://arxiv.org/html/2610.19220#bib.bib5);[Hennequin et al\., 2014](https://arxiv.org/html/2610.19220#bib.bib9);[Bondanelli and Ostojic, 2020](https://arxiv.org/html/2610.19220#bib.bib2)\)and built into trained recurrent networks\([Kerg et al\., 2019](https://arxiv.org/html/2610.19220#bib.bib10)\); locating and linearizing around the fixed points of a trained network follows[Sussillo and Barak \(2013\)](https://arxiv.org/html/2610.19220#bib.bib15);[Golub and Sussillo \(2018\)](https://arxiv.org/html/2610.19220#bib.bib6);[Maheswaranathan et al\. \(2019\)](https://arxiv.org/html/2610.19220#bib.bib11)\. What differs here is that the map, and with it the fixed point, is redefined at every tick\. On ImageNet, where the CTM attends next can be predicted one step ahead, and the model might gain freedom from image\-dependent decay rates, a variant of fast\-weight memories\([Ba et al\., 2016](https://arxiv.org/html/2610.19220#bib.bib1);[Schlag et al\., 2021](https://arxiv.org/html/2610.19220#bib.bib14)\)\. The results are for one checkpoint, a subsample of validation images, and image classification\.

## References

- Ba et al\. \[2016\]Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z\. Leibo, and Catalin Ionescu\.Using fast weights to attend to the recent past\.In*Advances in Neural Information Processing Systems*, 2016\.
- Bondanelli and Ostojic \[2020\]Giulio Bondanelli and Srdjan Ostojic\.Coding with transient trajectories in recurrent neural networks\.*PLoS Computational Biology*, 16\(2\):e1007655, 2020\.
- Darlow et al\. \[2025\]Luke Darlow, Ciaran Regan, Sebastian Risi, Jeffrey Seely, and Llion Jones\.Continuous thought machines\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025\.arXiv:2505\.05522\.
- Ganguli et al\. \[2008\]Surya Ganguli, Dongsung Huh, and Haim Sompolinsky\.Memory traces in dynamical systems\.*Proceedings of the National Academy of Sciences*, 105\(48\):18970–18975, 2008\.
- Goldman \[2009\]Mark S\. Goldman\.Memory without feedback in a neural network\.*Neuron*, 61\(4\):621–634, 2009\.
- Golub and Sussillo \[2018\]Matthew D\. Golub and David Sussillo\.FixedPointFinder: A Tensorflow toolbox for identifying and characterizing fixed points in recurrent neural networks\.*Journal of Open Source Software*, 3\(31\):1003, 2018\.
- Graves \[2016\]Alex Graves\.Adaptive computation time for recurrent neural networks\.*arXiv preprint arXiv:1603\.08983*, 2016\.
- Hannagan et al\. \[2018\]T\. Hannagan, A\. Nieder, P\. Viswanathan, and S\. Dehaene\.A random\-matrix theory of the number sense\.*Philosophical Transactions of the Royal Society B*, 373\(1740\):20170253, 2018\.[10\.1098/rstb\.2017\.0253](https://doi.org/10.1098/rstb.2017.0253)\.
- Hennequin et al\. \[2014\]Guillaume Hennequin, Tim P\. Vogels, and Wulfram Gerstner\.Optimal control of transient dynamics in balanced networks supports generation of complex movements\.*Neuron*, 2014\.
- Kerg et al\. \[2019\]Giancarlo Kerg, Kyle Goyette, Maximilian Puelma Touzel, Gauthier Gidel, Eugene Vorontsov, Yoshua Bengio, and Guillaume Lajoie\.Non\-normal recurrent neural network \(nnRNN\): learning long time dependencies while improving expressivity with transient dynamics\.In*Advances in Neural Information Processing Systems*, 2019\.
- Maheswaranathan et al\. \[2019\]Niru Maheswaranathan, Alex H\. Williams, Matthew D\. Golub, Surya Ganguli, and David Sussillo\.Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics\.In*Advances in Neural Information Processing Systems*, 2019\.
- Muller et al\. \[2018\]Lyle Muller, Frédéric Chavane, John Reynolds, and Terrence J\. Sejnowski\.Cortical travelling waves: mechanisms and computational principles\.*Nature Reviews Neuroscience*, 19\(5\):255–268, 2018\.
- Murphy and Miller \[2009\]Brendan K\. Murphy and Kenneth D\. Miller\.Balanced amplification: a new mechanism of selective amplification of neural activity patterns\.*Neuron*, 61\(4\):635–648, 2009\.
- Schlag et al\. \[2021\]Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber\.Linear transformers are secretly fast weight programmers\.In*International Conference on Machine Learning*, 2021\.
- Sussillo and Barak \[2013\]David Sussillo and Omri Barak\.Opening the black box: low\-dimensional dynamics in high\-dimensional recurrent neural networks\.*Neural Computation*, 25\(3\):626–649, 2013\.
- Trefethen and Embree \[2005\]Lloyd N\. Trefethen and Mark Embree\.*Spectra and Pseudospectra: The Behavior of Nonnormal Matrices and Operators*\.Princeton University Press, 2005\.

## Appendix AMethod and robustness

### Computing the Jacobian of one thought cycle\.

To study the CTM’s neural dynamics we freeze the memory at a chosen tick\. With the image tokens, the memory accumulators, and the short pre\-activation history held fixed, one tick becomes an ordinary map from states to states,zt\+1=f⁡\(zt,𝒜t\)z^\{t\+1\}=f\(z^\{t\};\\mathcal\{A\}^\{t\}\), where𝒜t\\mathcal\{A\}^\{t\}bundles everything held constant\. We reconstruct this map from the model’s own components and verify that a single reconstructed step reproduces the true next state to within numerical error\. Iterating the map gives its fixed point; the Jacobian at the fixed point, obtained by automatic differentiation, gives its local behavior\. From that Jacobian we read two numbers by power iteration: the spectral radius, which governs whether the map settles, and the largest singular value, which measures how much it can stretch a direction in a single step\.

Objects\.Fix a tick and its context𝒜\\mathcal\{A\}\. The frozen mapf⁡\(⋅,𝒜\):ℝD→ℝDf\(\\cdot;\\mathcal\{A\}\):\\mathbb\{R\}^\{D\}\\to\\mathbb\{R\}^\{D\}of §[2](https://arxiv.org/html/2610.19220#S2)isC1C^\{1\}\. A*fixed point*z⋆=z⋆​\(𝒜\)z^\{\\star\}=z^\{\\star\}\(\\mathcal\{A\}\)solvesz⋆=f⁡\(z⋆,𝒜\)z^\{\\star\}=f\(z^\{\\star\};\\mathcal\{A\}\); the*Jacobian*there isJ=∂f/∂z\|z⋆J=\\partial f/\\partial z\\big\|\_\{z^\{\\star\}\}, with*spectral radius*ρ⁡\(J\)=maxi⁡\|λi​\(J\)\|\\rho\(J\)=\\max\_\{i\}\|\\lambda\_\{i\}\(J\)\|and*operator norm*σmax​\(J\)=λmax​\(J⊤​J\)\\sigma\_\{\\max\}\(J\)=\\sqrt\{\\lambda\_\{\\max\}\(J^\{\\top\}J\)\}\. The map is*non\-normal*whenJ⊤​J≠J​J⊤J^\{\\top\}J\\neq JJ^\{\\top\}, which permitsσmax​\(J\)\>1≥ρ⁡\(J\)\\sigma\_\{\\max\}\(J\)\>1\\geq\\rho\(J\): transient stretching of some directions despite every eigenvalue decaying\.

Banach’s fixed\-point theorem\.If\(X,∥⋅∥\)\(X,\\\|\\cdot\\\|\)is complete andg:X→Xg:X\\to Xis a contraction —‖g⁡\(u\)−g⁡\(v\)‖≤L​‖u−v‖\\\|g\(u\)\-g\(v\)\\\|\\leq L\\\|u\-v\\\|for someL<1L<1and allu,vu,v— thengghas a unique fixed pointx⋆x^\{\\star\}, and the iterationuk\+1=g⁡\(uk\)u\_\{k\+1\}=g\(u\_\{k\}\)converges to it geometrically,‖uk−x⋆‖≤Lk​‖u0−x⋆‖\\\|u\_\{k\}\-x^\{\\star\}\\\|\\leq L^\{k\}\\\|u\_\{0\}\-x^\{\\star\}\\\|\.

### How we apply it\.

We do*not*have a global Euclidean contraction:σmax​\(J\)\>1\\sigma\_\{\\max\}\(J\)\>1, soffcan push nearby points apart in Euclidean norm, and Banach does not apply in∥⋅∥2\\\|\\cdot\\\|\_\{2\}\. Local uniqueness and attraction follow instead from the spectral radius, through a standard fact \(Householder\): for any matrixJJand anyε\>0\\varepsilon\>0there is a vector norm whose induced operator norm satisfies‖J‖⋆≤ρ⁡\(J\)\+ε\\\|J\\\|\_\{\\star\}\\leq\\rho\(J\)\+\\varepsilon\. Since we measureρ⁡\(J\)=0\.15<1\\rho\(J\)=0\.15<1, choosingε\\varepsilonwithρ⁡\(J\)\+ε<1\\rho\(J\)\+\\varepsilon<1yields a norm∥⋅∥⋆\\\|\\cdot\\\|\_\{\\star\}in whichJJis a contraction; becauseffisC1C^\{1\}, it is then a contraction in∥⋅∥⋆\\\|\\cdot\\\|\_\{\\star\}on a neighborhood ofz⋆z^\{\\star\}, and Banach applies there\. The fixed point is therefore locally unique and the iteration converges geometrically in∥⋅∥⋆\\\|\\cdot\\\|\_\{\\star\}— matching the observation that iterating from the current state reaches residual<10−6<\\\!10^\{\-6\}at99\.4%99\.4\\%of image–tick pairs \(see Estimation below\)\. The non\-normality \(σmax​\(J\)≫ρ⁡\(J\)\\sigma\_\{\\max\}\(J\)\\gg\\rho\(J\)\) is exactly why convergence need not be monotone in the*Euclidean*norm, and why the state lags the moving fixed point by about one step even while it is strongly attracted in∥⋅∥⋆\\\|\\cdot\\\|\_\{\\star\}\.

Establishing non\-normality\.A stability check based on random directions is therefore misleading: it averagesff’s action over directions, reporting something close to the*typical*stretch — about0\.070\.07here, comfortably below one — and never samples the few directions stretched most\. We confirm the stretching is genuine, not an artifact of differentiation, by perturbingz⋆z^\{\\star\}along the single most\-stretched direction and measuring the effect by finite differences \(over3232fixed points, the finite\-difference stretch and the autograd operator norm agree to a median ratio of1\.0001\.000; median stretch2\.72\.7\)\. Stability of an unrolled map like this must be read fromρ⁡\(J\)\\rho\(J\)at the fixed point, not from an average stretch\.

Estimation\.We estimateρ⁡\(J\)\\rho\(J\)andσmax​\(J\)\\sigma\_\{\\max\}\(J\)two ways that agree in magnitude: power iteration atz⋆z^\{\\star\}\(used for Fig\.[2](https://arxiv.org/html/2610.19220#S3.F2)and the mediansρ⁡\(J\)≈0\.15\\rho\(J\)\\\!\\approx\\\!0\.15,σmax​\(J\)≈3\.2\\sigma\_\{\\max\}\(J\)\\\!\\approx\\\!3\.2, over30553055fixed points\), and exact eigen\- and singular\-value decompositions of the fully materializedJJ\(used for the null comparisons of Appendix[B](https://arxiv.org/html/2610.19220#A2); medians over6060image–tick pairs,2020images at ticks88,2424and4040\)\. At the remaining1717of the30723072image–tick pairs \(at ticks44,4444and4848\) the iteration does not settle within200200steps, nor, for the four we examined, within20002000; these pairs have no fixed point to analyze and are excluded from all spectral statistics\. We quote the non\-normality as the median of the*per\-fixed\-point*ratioσmax​\(J\)/ρ⁡\(J\)≈24\\sigma\_\{\\max\}\(J\)/\\rho\(J\)\\\!\\approx\\\!24\(95% interval2323–2424\) under power iteration; this exceeds the ratio of the marginal medians \(3\.2/≈213\.2/0\.15\\\!\\approx\\\!21\) because the ratio is right\-skewed across ticks\. The materialized route gives a somewhat higher median \(≈30\\approx\\\!30\) on its own sample of6060fixed points; we use it only for the null comparisons, where measurement and nulls are computed identically\.

### Other basins\.

Banach’s theorem establishes local uniqueness only: withσmax​\(J\)\>1\\sigma\_\{\\max\}\(J\)\>1there is no global Euclidean contraction\. For6464images at ticks88,2424and4040, we iterated the frozen map from random states at the typical scale ofztz^\{t\}and at three times it, and from other images’ states at the same tick \(11521152runs of each kind\)\. All34563456runs converged, and all reached the fixed point obtained fromztz^\{t\}\(median relative distance2×10−72\\times 10^\{\-7\}\)\. We found no second basin\.

### The lag against the contraction rate\.

Over15361536image–tick pairs \(128128images\), the distance tozt−1⋆z^\{\\star\}\_\{t\-1\}shrinks across one step by a median factor0\.110\.11\(interquartile range0\.060\.06–0\.220\.22\), againstρ⁡\(J\)=0\.14\\rho\(J\)=0\.14\(0\.080\.08–0\.330\.33\) at the same fixed points and0\.100\.10for the linearized map∥J​e∥/∥e∥\\lVert Je\\rVert/\\lVert e\\rVert; per pair, shrink andρ⁡\(J\)\\rho\(J\)are correlated \(Spearman0\.660\.66\)\. The lag is the contraction at work\. The fixed point itself moves by a median3737per tick,1\.081\.08times the pre\-step distance; this motion, set mainly by the memory \(§[4](https://arxiv.org/html/2610.19220#S4)\), is the part of the trajectory the contraction does not imply\.

## Appendix BBalanced amplification: the Schur form of the tick map

### Pseudospectra and a random\-matrix null\.

The object behind the transient stretching is theε\\varepsilon\-pseudospectrum ofJJ\[[Trefethen and Embree, 2005](https://arxiv.org/html/2610.19220#bib.bib16)\]: becauseJJis non\-normal, its pseudospectrum reaches well outside the unit disk even though its spectrum sits at radiusρ⁡\(J\)≈0\.15\\rho\(J\)\\\!\\approx\\\!0\.15, which is what allows the one\-step Euclidean growth and the lag\. Repeated multiplication by a fixed*random*matrix is on its own enough to organize structured neural codes\[[Hannagan et al\., 2018](https://arxiv.org/html/2610.19220#bib.bib8)\], so a random matrix is the natural yardstick for an operator applied repeatedly\. To ask what produces this non\-normality, we compare the measuredσmax​\(J\)/ρ​\(J\)\\sigma\_\{\\max\}\(J\)/\\rho\(J\)\(of order3030\) against two same\-size nulls\. A Ginibre matrix \(i\.i\.d\. Gaussian entries\) givesσmax​\(J\)/ρ⁡\(J\)≈2\\sigma\_\{\\max\}\(J\)/\\rho\(J\)\\\!\\approx\\\!2by the circular law, so a generic*light\-tailed*random matrix is far more normal thanJJ\.

### What produces the non\-normality\.

An entry\-shuffle ofJJ—which preservesJJ’s heavy\-tailed entry distribution but destroys its learned arrangement—gives a medianσmax​\(J\)/ρ​\(J\)\\sigma\_\{\\max\}\(J\)/\\rho\(J\)of about1313over6060image–tick pairs, already six times the Gaussian value; the measured≈30\\approx\\\!30adds a further factor of about2\.32\.3on top\. Both ingredients therefore matter: the heavy\-tailed entry*distribution*accounts for most of the gap from a Gaussian matrix, while the specific learned*arrangement*still contributes a substantial additional amount\. Either wayJJis sharply distinguished from a Gaussian matrix of the same size\. Part of the non\-normality is architectural—an untrained CTM givesσmax​\(J\)/ρ⁡\(J\)≈7\\sigma\_\{\\max\}\(J\)/\\rho\(J\)\\approx 7—and training multiplies it; training increases it on the small models too, but less \(§[8](https://arxiv.org/html/2610.19220#S8)\)\. \(Medians over6060image–tick pairs\.\)

### Balanced amplification\.

To probe the*form*of the non\-normality we Schur\-triangularize the materializedJ=U​T​U∗J=UTU^\{\*\}and report three medians \(Table[3](https://arxiv.org/html/2610.19220#A2.T3)\): the feed\-forward fraction∥triu⁡\(T,1\)∥F/∥J∥F\\lVert\\mathrm\{triu\}\(T,1\)\\rVert\_\{F\}/\\lVert J\\rVert\_\{F\}\(mass in the strictly upper\-triangular, feed\-forward part\), which equals Henrici’s departure from normalitydep⁡\(J\)=∥J∥F2−∑i\|λi\|2/∥J∥F\\mathrm\{dep\}\(J\)=\\sqrt\{\\lVert J\\rVert\_\{F\}^\{2\}\-\\sum\_\{i\}\|\\lambda\_\{i\}\|^\{2\}\}/\\lVert J\\rVert\_\{F\}; and the antisymmetric fraction∥\(J−J⊤\)/2∥F/∥J∥F\\lVert\(J\-J^\{\\top\}\)/2\\rVert\_\{F\}/\\lVert J\\rVert\_\{F\}\. Against the same Ginibre and entry\-shuffle nulls, and against the random core \(the trained backbone with the recurrence at initialization\),JJis near\-maximally feed\-forward \(0\.990\.99vs0\.710\.71/0\.810\.81/0\.930\.93\) with a chance\-level rotational part \(0\.710\.71\)—the balanced\-amplification signature of[Murphy and Miller \[2009\]](https://arxiv.org/html/2610.19220#bib.bib13); reading a recurrent operator’s hidden feed\-forward organization off its Schur form follows[Goldman \[2009\]](https://arxiv.org/html/2610.19220#bib.bib5)\. This structure is visible directly in the materialized operator \(Fig\.[7](https://arxiv.org/html/2610.19220#A2.F7)\)\.

### What the Schur matrix does not show\.

Two features of it are conventions rather than findings, and we read neither\. The Schur vectors are each defined only up to sign—U↦U​DU\\mapsto UDwithD=diag⁡\(±1\)D=\\mathrm\{diag\}\(\\pm 1\)sendsTi​j↦di​dj​Ti​jT\_\{ij\}\\mapsto d\_\{i\}d\_\{j\}T\_\{ij\}—so the sign pattern is gauge\-dependent, and we plot\|Ti​j\|\|T\_\{ij\}\|, which is not\. The eigenvalue ordering along the diagonal is likewise arbitrary, and reordering it yields a differentUUand a differentTTrather than a permutation of the same entries; the placement of individual modes, including the densely coupled leading rows and trailing columns, therefore carries no information\. What does not depend on either convention is the diagonal itself—the eigenvalue set in any ordering—and hence the diagonal/off\-diagonal split that the feed\-forward fraction measures\.

Table 3:Balanced\-amplification descriptors of the tick\-map JacobianJJatz⋆z^\{\\star\}\(medians over image–tick pairs\), against same\-size random nulls and the random core \(trained backbone, recurrence at initialization; feed\-forward fraction only\)\. The Schur feed\-forward fraction and Henrici’s departure from normality coincide and are reported together\.JJis a near\-maximal feed\-forward non\-normal operator whose rotational part is at chance\.![Refer to caption](https://arxiv.org/html/2610.19220v1/figures/fig7_schur_imagenet.png)Figure 7:The tick map is a feed\-forward cascade in its Schur basis\.Magnitudes\|Ti​j\|\|T\_\{ij\}\|\(log scale\) of the real Schur form of the materialized JacobianJ=U​T​U∗J=UTU^\{\*\}\(one held\-out image, mid\-trajectory tick\)\. The Schur form is upper triangular by construction; what is informative is where the mass sits within it\. For a normal operatorTTwould be diagonal, the whole norm carried by the eigenvalues\. Here almost none is: the strictly upper\-triangular part accounts for a fraction0\.990\.99of∥J∥F\\lVert J\\rVert\_\{F\}\(Table[3](https://arxiv.org/html/2610.19220#A2.T3)\), so the operator is nearly all feed\-forward coupling between Schur modes rather than modal decay\. Magnitudes are shown, and individual mode positions should not be read as structure; see the text\.

## Appendix CThe attention schedule in detail

The attention map comes apart into a shared schedule, which depends only on the tick, and the image’s features, which decide where that schedule lands\. Discarding each image’s own query and substituting the population\-average query at the matching tick still recovers the image’s true attention map at cosine similarity0\.9360\.936\(400400held\-out images; Figs\.[4](https://arxiv.org/html/2610.19220#S4.F4)and[8](https://arxiv.org/html/2610.19220#A3.F8)\)\. Although each image sets its own starting query, the*way*the query advances is nearly common to all images, so a simple predictor—the current query plus the shared average increment \(R2=0\.98R^\{2\}=0\.98held out\)—forecasts the next query, and with it the next attention map \(cosine0\.9960\.996versus0\.9480\.948for assuming attention stays put\)\. Predicting the schedule thus predicts where the model will look\.

A note on what this does and does not let one skip at run time\. Predicting the next attention map ahead of time needs only the image’s features and the stored schedule; it does not need to run the recurrence\. Predicting the actual class still does, because the decision is read from the state the recurrence produces\. The query’s direct effect on entropy, measured as a one\-step counterfactual, is modest—it is the routing of attention, compounded over many ticks and captured by the ablations, that makes the query matter\.

![Refer to caption](https://arxiv.org/html/2610.19220v1/figures/fig8_shared_schedule_imagenet.png)Figure 8:The shared schedule\.Left:alignment of each image’s query with the population\-average query at each tick,cos⁡\(qit,q¯t\)\\cos\(q^\{t\}\_\{i\},\\bar\{q\}^\{t\}\)\.Right:the attention computed from the shared, population\-average query matches each image’s true attention\.
## Appendix DHeadline numbers

All numbers use the official checkpoint without training; intervals are bootstrap 95%, and accuracies are computed per image on one common sample of10001000images \(Table[4](https://arxiv.org/html/2610.19220#A4.T4)\)\.

Table 4:Headline numbers, with the number of images \(or image–tick pairs\) each rests on and a bootstrap 95% interval\. Accuracies are computed per image on one common sample\.