Cross-Trajectory Chimera Interventions Reveal Dissociable Roles of Weight Magnitude and Direction in Grokking

# Cross-Trajectory Chimera Interventions Reveal Dissociable Roles of Weight Magnitude and Direction in Grokking
Source: [https://arxiv.org/html/2607.06628](https://arxiv.org/html/2607.06628)
###### Abstract

Which properties of a partially trained network are causally portable to a*different*, independently trained network, and which are not? Interventions on a single training trajectory cannot answer this: they can show that a property is*necessary*for an outcome within one run, but not whether it is*portable*across runs\. We introduce*cross\-trajectory chimera interventions*: given two networks trained from different random seeds on the same task, we decompose each weight vector into a magnitude \(norm\) and a direction \(unit vector\), recombine the magnitude of one run with the direction of another to form a*chimera*, and continue training\. Across two modular\-arithmetic tasks that exhibit grokking, we find a clean dissociation\. The*direction*carries a transferable component associated with the final circuit’s spectral identity: implanting a donor’s direction while keeping the recipient’s norm drives the continued run toward the donor’s eventual circuit in40/4040/40independent recombinations, and this transfer is specific to the donor’s content rather than to the mere angular displacement \(an angle\-matched random control produces no such shift\)\. Interpolating the direction along the geodesic between two runs reveals that this transfer is*threshold\-like*rather than a continuous blend\. Most notably, cross\-trajectory intervention reveals that transferability itself depends on the recipient’s dynamical state: the interpolation threshold at which identity flips from recipient\-like to donor\-like is predicted by the recipient’s weight norm, separating perfectly by norm class across all2020pairs and both tasks \(joint exact permutation probability1\.9×10−41\.9\{\\times\}10^\{\-4\}\)\. We localize the threshold to a resolution of±1/64\\pm 1/64using an adaptive bisection procedure that we contribute as a reusable measurement tool for stability\-under\-protocol interventions\. By contrast, the*norm*carries only a modest, spatially distributed effect on grokking delay and no measurable identity information\. Together these results map a concrete geometric division of labour: direction indexes which solution a trajectory approaches, while norm governs how susceptible that identity is to being overwritten\.

## 1Introduction

Grokking—the delayed onset of generalization long after a network has fit its training data—has become a controlled setting for asking how neural networks transition from memorization to structured solutions\(Poweret al\.,[2022](https://arxiv.org/html/2607.06628#bib.bib1); Nandaet al\.,[2023](https://arxiv.org/html/2607.06628#bib.bib2)\)\. A large body of recent work intervenes on a single training run to probe this transition: freezing, rescaling, or projecting weights along a trajectory and observing the effect on delay or on the emerging circuit\(Nandaet al\.,[2023](https://arxiv.org/html/2607.06628#bib.bib2); Varmaet al\.,[2023](https://arxiv.org/html/2607.06628#bib.bib4)\)\. Such within\-trajectory interventions, however, cannot answer a distinct question: of everything a partially trained checkpoint contains, which parts are*portable*—able to determine an outcome when transplanted into a*different*, independently trained run? Within\-trajectory interventions can establish that a property is*necessary*for an outcome inside one run; they cannot establish whether that property is*portable*—able to produce the outcome when placed in a different run’s context\. Distinguishing these two questions is our organizing concern throughout the paper: we do not ask why direction encodes circuit identity, but whether it is causally portable, and if so, under what conditions\.

We study this question through a decomposition of the weight vector into a radial part \(its norm\) and an angular part \(its unit direction\),θ=r⋅u\\theta=r\\cdot u\. This decomposition itself is not new: the interplay of weight norm and direction in grokking has been analyzed within single trajectories\(Liuet al\.,[2022](https://arxiv.org/html/2607.06628#bib.bib3); Varmaet al\.,[2023](https://arxiv.org/html/2607.06628#bib.bib4)\)\. Our contribution is a*cross\-trajectory*causal probe\. Given two networksAAandBBtrained from different seeds, we build a*chimera*by combiningBB’s norm withAA’s direction \(or vice versa\), continue training, and ask two questions: does the grokking*delay*of the chimera follow the run that donated the norm, and does the final*circuit identity*follow the run that donated the direction?

We find a sharp asymmetry\. Circuit identity—measured by a task\-appropriate power spectrum of the token embedding—follows the angular donor cleanly, consistently, and specifically to that donor’s content\. Grokking delay follows the radial donor only weakly, and this weak effect cannot be localized to any single layer group\. Pushing further, we interpolate the implanted direction along the geodesic fromAAtoBBat fixed norm and find that identity transfer is*threshold\-like*: the final circuit remains recipient\-like across a plateau of interpolation values and then switches to donor\-like over a narrow range\. The location of this threshold is not constant—and, revealingly, it is predicted by the recipient’s weight norm\. This last finding is only accessible*because*the intervention is cross\-trajectory: it concerns not merely what transfers, but when transfer is possible\.

#### Contributions\.

1. 1\.Method\.Cross\-trajectory chimera interventions, a causal probe that recombines the magnitude and direction of independently trained networks, together with an angle\-matched random control that isolates donor\-specific content from generic angular perturbation \(Section[3](https://arxiv.org/html/2607.06628#S3)\)\.
2. 2\.Dissociation\.The angular component carries a transferable, donor\-specific component associated with circuit identity across two tasks \(40/4040/40recombinations sign\-correct\); the radial component carries only a modest, non\-localizable delay effect \(Section[4](https://arxiv.org/html/2607.06628#S4),[6](https://arxiv.org/html/2607.06628#S6)\)\.
3. 3\.Threshold and its dependence on state\.Identity transfer is threshold\-like in the interpolation coordinate, and the threshold location is predicted by the recipient’s norm, separating perfectly by norm class across all2020pairs and both tasks\. We contribute an adaptive bisection procedure that localizes the threshold to±1/64\\pm 1/64at roughly three evaluations per pair, a reusable measurement strategy for interventions that are stable under their training protocol \(Section[5](https://arxiv.org/html/2607.06628#S5)\); this separation, and the identity\-transfer result above, both survive an optimizer\-state ablation that transplants Adam moments between donor and recipient rather than resetting them \(Section[8](https://arxiv.org/html/2607.06628#S8)\)\.
4. 4\.Supporting characterization\.Circuit spectra continue reorganizing for hundreds to thousands of steps after generalization is behaviourally complete, and no two seeds converge to near\-identical spectra \(Section[7](https://arxiv.org/html/2607.06628#S7)\)\.

## 2Related Work

#### Transferring weights between grokking runs\.

Closest to our setting is work that transfers a learned embedding from a weaker, already\-generalized network to initialize a stronger one, accelerating or inducing grokking in the target\(Xuet al\.,[2025](https://arxiv.org/html/2607.06628#bib.bib5)\)\. That line treats transfer as a method for*speeding up*generalization and measures its effect on delay and accuracy\. Our aims differ in three ways\. First, we recombine two*peer*runs by explicitly separating norm from direction, rather than transplanting a whole embedding from a privileged source\. Second, our primary outcome is*circuit identity*, not delay; nothing in the acceleration line measures which of several distinct circuits the recipient adopts\. Third, we introduce a donor\-content control \(an angle\-matched random direction\) that distinguishes “BB’s specific direction mattered” from “a perturbation of this angular size mattered\.” We also find that the identity\-carrying signal is present in multiple hidden layer groups, not the embedding alone \(Section[7](https://arxiv.org/html/2607.06628#S7)\), which situates our result outside an embedding\-centric account\.

#### Merging independently trained networks\.

A parallel literature merges or interpolates independently trained networks after aligning their permutation symmetries, seeking low\-loss connectivity or a single merged model\(Ainsworthet al\.,[2023](https://arxiv.org/html/2607.06628#bib.bib6); Entezariet al\.,[2021](https://arxiv.org/html/2607.06628#bib.bib7)\)\. We deliberately do*not*align permutations: the point of a chimera is to serve as a causal probe, and we measure a circuit\-identity signature and the dynamics of continued training rather than a loss barrier between fixed endpoints\.

#### Basin selection and mode determinism\.

Which solution a trajectory reaches has been studied through linear mode connectivity and the role of the early training phase in determining the eventual basin\(Frankleet al\.,[2020](https://arxiv.org/html/2607.06628#bib.bib8); Fortet al\.,[2019](https://arxiv.org/html/2607.06628#bib.bib9)\), and more recently through singular\-learning\-theory and exploration\-based accounts of basin selection in grokking\(Cullen and others,[2026](https://arxiv.org/html/2607.06628#bib.bib10)\)\. These analyses are largely observational—they characterize which basin is reached under natural training\. Our chimera intervention provides a causal complement: by transplanting a direction across trajectories we measure the boundary at which basin membership changes, and we show that the location of that boundary depends on the recipient’s state\.

#### Relationship to our prior work\.

Two of our earlier studies analyze single\-trajectory grokking: one establishes that norm and direction are jointly necessary to explain the delay within a run, and the other calibrates a norm–timescale relationship for the delay\. The present paper is neither an extension nor a re\-analysis of those datasets\. It concerns*cross\-trajectory*transferability and a dissociation between the two components’ portable content, using disjoint experiments; where we note that the modest delay effect is consistent with the earlier norm–timescale relationship, we say only that, and draw no stronger link\.

## 3Setup and Methods

#### Tasks and model\.

We study two tasks on which a small transformer groks: modular addition\(a\+b\)modp\(a\+b\)\\bmod pand modular multiplication\(a⋅b\)modp\(a\\cdot b\)\\bmod pon the multiplicative group\{1,…,p−1\}\\\{1,\\dots,p\-1\\\}, withp=59p=59\. Inputs are tokenized as\[a,b,=\]\[a,b,\{=\}\]and the model predicts the result token\. The architecture is a one\-layer transformer \(embedding dimension128128\) trained full\-batch with AdamW \(lr=10−3\\text\{lr\}=10^\{\-3\}, weight decay1\.01\.0,β=\(0\.9,0\.98\)\\beta=\(0\.9,0\.98\)\) at a50%50\\%train/test split\. For multiplication we restrict to the multiplicative group because including0makes every\(0,b\)\(0,b\)pair collapse to0, a degenerate shortcut that both inflates accuracy and pollutes the spectral fingerprint defined below\.

#### Radial–angular decomposition and chimeras\.

For a flattened weight vectorθ\\thetawe writeθ=r​u\\theta=r\\,uwithr=∥θ∥r=\\lVert\\theta\\rVertandu=θ/∥θ∥u=\\theta/\\lVert\\theta\\rVert\. Given a pre\-grokking checkpoint from runAA\(recipient\) and one from runBB\(donor\), we form*chimera*variants and continue training under the recipient’s optimizer state; results are reported for a reset optimizer unless noted, and we verify in Section[8](https://arxiv.org/html/2607.06628#S8)that this choice does not drive our conclusions\. The core variant isradial\(rB​uAr\_\{B\}\\,u\_\{A\}: recipient direction, donor norm\) and its mirrorreverse\_radial\(rA​uBr\_\{A\}\\,u\_\{B\}\)\. We include four controls:mid\_norm\(12​\(rA\+rB\)​uA\\tfrac\{1\}\{2\}\(r\_\{A\}\{\+\}r\_\{B\}\)\\,u\_\{A\}, isolating the effect of changing norm alone\),random\_u\(a fully random direction at the recipient’s norm\), and—critically—matched\_random\(a random direction placed at*exactly*the angular distance thatuBu\_\{B\}has fromuAu\_\{A\}\)\. In high dimensions a uniformly random direction is nearly orthogonal touAu\_\{A\}, whereas two independently trained runs on the same task are not; the matched\-random control equalizes the angular displacement so that any difference from the true donor isolates donor\-specific content rather than perturbation magnitude\.

AABBrecipient\(seedsAs\_\{A\}\)donor\(seedsBs\_\{B\}\)θA=rA​uA\\theta\_\{A\}=r\_\{A\}\\,u\_\{A\}θB=rB​uB\\theta\_\{B\}=r\_\{B\}\\,u\_\{B\}χ\\chichimerarB​uAr\_\{B\}\\,u\_\{A\}dir\.uAu\_\{A\}normrBr\_\{B\}χ⋆\\chi^\{\\star\}continuedtrainingAdamWdelay→\\tonorm donor?identity→\\todir\. donor?tt00\.51uAu\_\{A\}uBu\_\{B\}slerp​\(uA,uB,t\)\\mathrm\{slerp\}\(u\_\{A\},u\_\{B\},t\)at fixed normrAr\_\{A\}Figure 1:Cross\-trajectory chimera intervention\.Two networks are trained from different seeds on the same task\. Each weight vector is split into a normrrand a unit directionuu\. A*chimera*recombines one run’s norm with the other’s direction \(hererB​uAr\_\{B\}\\,u\_\{A\}\) and is trained onward; we ask whether its grokking delay follows the norm donor and its circuit identity follows the direction donor\. The directional dose\-response \(inset\) interpolates the implanted direction along the geodesic fromuAu\_\{A\}touBu\_\{B\}at fixed norm, isolating the angular coordinate\.
#### Circuit\-identity metric\.

Following the Fourier account of modular\-arithmetic circuits\(Nandaet al\.,[2023](https://arxiv.org/html/2607.06628#bib.bib2)\), we summarize a network’s circuit by the normalized power spectrum of its token\-embedding rows\. The domain of the transform must match the task: for addition, circuits are periodic in the token index, so we take the FFT over rows0,…,p−10,\\dots,p\-1; for multiplication, circuits are periodic in the*discrete logarithm*of the token \(multiplication moduloppis addition modulop−1p\-1undert↦logg⁡tt\\mapsto\\log\_\{g\}tfor a primitive rootgg\), so we reorder rows as\[g0,g1,…,gp−2\]\[g^\{0\},g^\{1\},\\dots,g^\{p\-2\}\]before the transform\. Using the index\-domain transform for multiplication is blind to genuine circuit differences—two synthetic multiplication circuits at different frequencies register cosine similarity0\.99980\.9998under the wrong domain and0\.00000\.0000under the correct one\. We define*circuit similarity*as the cosine similarity of two normalized power spectra and, for a chimera, reportCFS​\_​lean=sim​\(chimera,Bfinal\)−sim​\(chimera,Afinal\)\\mathrm\{CFS\\\_lean\}=\\mathrm\{sim\}\(\\text\{chimera\},B\_\{\\text\{final\}\}\)\-\\mathrm\{sim\}\(\\text\{chimera\},A\_\{\\text\{final\}\}\), so that negative values indicate anAA\-like final circuit and positive values aBB\-like one\.

#### Directional dose\-response\.

To move continuously from recipient to donor direction we interpolate along the geodesic on the unit sphere,

slerp​\(uA,uB,t\)=sin⁡\(\(1−t\)​ϕ\)sin⁡ϕ​uA\+sin⁡\(t​ϕ\)sin⁡ϕ​uB,ϕ=arccos⁡\(uA⋅uB\),\\mathrm\{slerp\}\(u\_\{A\},u\_\{B\},t\)=\\frac\{\\sin\\big\(\(1\{\-\}t\)\\phi\\big\)\}\{\\sin\\phi\}\\,u\_\{A\}\+\\frac\{\\sin\\big\(t\\phi\\big\)\}\{\\sin\\phi\}\\,u\_\{B\},\\qquad\\phi=\\arccos\(u\_\{A\}\\\!\\cdot u\_\{B\}\),and implantrA⋅slerp​\(uA,uB,t\)r\_\{A\}\\cdot\\mathrm\{slerp\}\(u\_\{A\},u\_\{B\},t\)at fixed recipient normrAr\_\{A\}\. Spherical interpolation preserves unit norm along the entire path, so the intervention varies*only*the angular coordinate; linear interpolation of unit vectors would shrink the mixture’s norm \(up to∼29%\{\\sim\}29\\%att=0\.5t=0\.5for near\-orthogonal endpoints\), simultaneously changing magnitude and direction and confounding the two axes we seek to dissociate\. The endpointt=1t\{=\}1coincides exactly withreverse\_radialand is reused rather than recomputed\.

#### Statistical protocol\.

We select checkpoint pairs so that no seed appears in more than one pair \(disjoint matching\), yielding1010fully independent pairs per task drawn from2020distinct seeds\. This avoids pseudoreplication: reusing a single seed across many pairs would make pooled statistics non\-independent\(Hurlbert,[1984](https://arxiv.org/html/2607.06628#bib.bib11)\)\. We report cluster\-level statistics accordingly\(Cameron and Miller,[2015](https://arxiv.org/html/2607.06628#bib.bib12)\), use exact sign tests for directional consistency, and bootstrap confidence intervals for pooled means\. Decision thresholds separating a usable signal from a confound were fixed before analysis\. Throughout, we distinguish “statistically distinguishable from zero” from “large”; the delay effect below is the former but not the latter\.

#### Threshold localization by bisection\.

Under our protocol \(full\-batch training, reset optimizer\) repeated continuations are sufficiently stable that the interpolation thresholdt⋆t^\{\\star\}per pair is limited by measurement*resolution*rather than by sampling noise\. We therefore localizet⋆t^\{\\star\}by bisection: the coarse grid brackets the sign change ofCFS​\_​lean\\mathrm\{CFS\\\_lean\}to an interval of width0\.250\.25, and each bisection step halves it, reaching±1/64\\pm 1/64in three evaluations per pair—an order of magnitude fewer continuations than a uniform fine grid over all pairs\. Because CUDA kernels are not bit\-deterministic, we reportt⋆t^\{\\star\}with its bisection half\-width as a resolution bound rather than as an exact real number\.

## 4The angular component carries transferable circuit identity

#### Endpoint swaps\.

Implanting the donor’s direction while keeping the recipient’s norm drives the continued run toward the donor’s eventual circuit\. On modular addition, theradialvariant \(recipient directionuAu\_\{A\}, donor normrBr\_\{B\}\) yields a final circuit that isAA\-like in every pair \(meanCFS​\_​lean=−0\.719\\mathrm\{CFS\\\_lean\}=\-0\.719,10/1010/10negative\), while its mirrorreverse\_radial\(donor directionuBu\_\{B\}\) isBB\-like in every pair \(mean\+0\.478\+0\.478,10/1010/10positive\)\. Modular multiplication replicates this exactly and more strongly on theBB\-side \(means−0\.719\-0\.719and\+0\.725\+0\.725;10/1010/10and10/1010/10\)\. Pooling the two variants across both tasks, the final circuit follows the direction donor in40/4040/40independent recombinations; each per\-task, per\-variant sign test givesp=2×10−3p=2\\\!\\times\\\!10^\{\-3\}\(Figure[2](https://arxiv.org/html/2607.06628#S4.F2)\)\.

![Refer to caption](https://arxiv.org/html/2607.06628v1/x1.png)Figure 2:The angular component carries transferable circuit identity\.Per\-pairCFS​\_​lean\\mathrm\{CFS\\\_lean\}under the endpoint swaps, for both tasks\. Theradialvariant \(keep recipient directionuAu\_\{A\}; blue\) yields anAA\-like final circuit in every pair \(negative\), whilereverse\_radial\(keep donor directionuBu\_\{B\}; orange\) yields aBB\-like circuit \(positive\)\. Pairs are fully independent \(disjoint seeds\)\. Sign is consistent in40/4040/40recombinations pooled across the two variants and two tasks\.
#### The effect is donor\-specific, not generic perturbation\.

The angle\-matched random control—a random direction at the same angular distance fromuAu\_\{A\}asuBu\_\{B\}—produces final circuits with essentially no lean \(mean−0\.031\-0\.031on addition,−0\.051\-0\.051on multiplication; leans scattered near zero\)\. Formally, the donor variants are distinguishable from the matched control on the circuit\-identity axis in58/6058/60comparisons on addition and60/6060/60on multiplication\. Thus the transfer reflects the donor’s specific direction, not merely a perturbation of that angular magnitude\.

#### Changing only the norm does not change identity\.

Themid\_normcontrol \(recipient direction, averaged norm\) leansAA\-like exactly asradialdoes \(−0\.707\-0\.707vs\.−0\.719\-0\.719on addition;−0\.736\-0\.736vs\.−0\.719\-0\.719on multiplication;10/1010/10negative in both\)\. Altering the implanted norm while holding direction fixed leaves circuit identity unchanged, completing the dissociation from the identity side\.

## 5Identity transfer is threshold\-like, and its threshold tracks recipient norm

#### A threshold, not a blend\.

Interpolating the implanted direction fromuAu\_\{A\}touBu\_\{B\}at fixed norm does not move circuit identity smoothly\. Across all2020pairs the endpoints have opposite sign as expected, but the change is concentrated: on average8282–88%88\\%of the total change inCFS​\_​lean\\mathrm\{CFS\\\_lean\}occurs within a single grid step, with plateaus on either side \(Figure[3](https://arxiv.org/html/2607.06628#S5.F3)\)\.

![Refer to caption](https://arxiv.org/html/2607.06628v1/x2.png)Figure 3:Identity transfer is threshold\-like, not a continuous blend\.CFS​\_​lean\\mathrm\{CFS\\\_lean\}as the implanted direction is interpolated along the geodesic fromuAu\_\{A\}\(t=0t\{=\}0\) touBu\_\{B\}\(t=1t\{=\}1\) at fixed recipient norm\. Grey curves are all pairs on the coarse grid; coloured curves are the representative pairs measured on a1616\-point fine grid\. Each pair stays near one plateau, then switches over a narrow range—a graded intervention with a sharply graded response\.A1616\-point fine grid on representative pairs confirms the plateau–jump–plateau shape and agrees with the bisection threshold to within one grid step\. Identity transfer thus behaves as a switch between two basins rather than a continuous mixture—a graded intervention with a sharply graded response, consistent with a winner\-take\-all selection between distinct circuits\.

#### The threshold location is predicted by recipient norm\.

The interpolation valuet⋆t^\{\\star\}at which identity flips is not constant across pairs, and it is organized by the recipient’s weight norm\. Localizingt⋆t^\{\\star\}to±1/64\\pm 1/64by bisection, pairs whose recipient has high norm \(far from grokking\) flip early—only a small move towarduBu\_\{B\}is needed—whereas pairs whose recipient has low norm \(near grokking\) flip late\. On modular addition the high\-norm \(slow\) recipients havet⋆∈\[0\.238,0\.297\]t^\{\\star\}\\in\[0\.238,0\.297\]and the low\-norm \(fast\) recipientst⋆∈\[0\.641,0\.891\]t^\{\\star\}\\in\[0\.641,0\.891\], a clean gap of0\.340\.34; on modular multiplication the corresponding ranges are\[0\.235,0\.443\]\[0\.235,0\.443\]and\[0\.678,0\.856\]\[0\.678,0\.856\], a gap of0\.230\.23\. The two norm classes separate perfectly on every pair in both tasks; pooling all2020pairs preserves perfect separation \(gap0\.200\.20\)\. Treating each task’s separation as an exact permutation test under the observed grouping gives probabilities2\.2×10−22\.2\\\!\\times\\\!10^\{\-2\}and8\.3×10−38\.3\\\!\\times\\\!10^\{\-3\}, and1\.9×10−41\.9\\\!\\times\\\!10^\{\-4\}jointly \(Figure[4](https://arxiv.org/html/2607.06628#S5.F4)\)\.

![Refer to caption](https://arxiv.org/html/2607.06628v1/x3.png)Figure 4:The transfer threshold is predicted by recipient norm\.Bisection\-localized thresholdt⋆t^\{\\star\}\(midpoint±\\pmhalf\-width,±1/64\\pm 1/64\) against the recipient’s weight normrAr\_\{A\}, for all2020pairs across both tasks \(circles: modular addition; squares: modular multiplication\)\. High\-norm \(slow\) recipients flip early; low\-norm \(fast\) recipients flip late\. The two norm classes separate without overlap; the green band marks the separation gap\. The shaded bands are a visual aid for the grouping, not a fitted relationship\.Because the intervention holds the recipient’s norm fixed and varies only direction, this is a statement about*susceptibility*: at high norm, circuit identity is easily overwritten by a donor direction; at low norm, it resists\.

#### Scope of the norm–threshold relationship\.

We report this as a separation between two norm classes, not as a fitted continuous lawt⋆​\(r\)t^\{\\star\}\(r\)\. The binary slow/fast grouping coincides with our pair\-selection design, which favours large norm differences and therefore produces two well\-separated norm groups rather than a dense sweep; establishing a functional form would require pairs sampled across a continuum of recipient norms\. What the data support is the ordinal claim—higher recipient norm, earlier threshold—which holds without exception here\.

## 6The radial component: a modest, distributed delay effect

The norm carries a genuine but weak influence on grokking*delay*, and no detectable identity information\. Under theradialvariant, the chimera’s delay shifts toward the norm donor by a normalized displacement whose per\-pair donor\-following score averages0\.280\.28on addition \(9/109/10positive\) and0\.410\.41on multiplication \(10/1010/10positive\)—reliably nonzero but far from the value that would indicate the chimera fully inherits the donor’s delay\. Expressed as a fractional displacement between recipient and donor delay, the chimera moves about30%30\\%of the way toward the donor, pooled across tasks\. The three implanted\-norm levels available per pair \(recipient, averaged, donor norm\) order delay monotonically in17/2017/20pairs, consistent with a graded but weak norm–delay relationship\. Crucially, when the norm swap is restricted to individual layer groups the delay effect vanishes for every group—it is a property of the whole weight vector, not attributable to any single layer\.

![Refer to caption](https://arxiv.org/html/2607.06628v1/x4.png)Figure 5:The radial component carries a modest, distributed delay effect\.Per\-pair delay donor\-following score under theradialvariant\. The mean is reliably positive \(dashed\) but no pair reaches the pre\-registered “clean” threshold of0\.70\.7\(dotted\): the norm shifts delay toward its donor, but only partially, and the effect vanishes when the swap is restricted to individual layer groups \(not shown\)\.Relative to the strong, localizable, threshold\-structured identity effect, the delay effect is weak, distributed, and carries no identity signal: the two geometric components play different roles\.

## 7Supporting characterization

#### Which layers carry the identity signal \(summary; full analysis in Appendix[A](https://arxiv.org/html/2607.06628#A1)\)\.

Restricting the directional swap to individual layer groups shows that the identity\-carrying signal is not the embedding’s alone: on both tasks, the attention, MLP, and unembedding groups each individually pull the final circuit toward the donor, but combining them gives a weaker lean than the strongest single group—the contributions are non\-additive\. We regard this as suggestive rather than established, given three pairs per task and a circularity that affects the embedding\-based readout specifically; the full numbers, the circularity issue, and the figure are in Appendix[A](https://arxiv.org/html/2607.06628#A1)\.

#### Circuits keep reorganizing after grokking\.

Tracking each run’s spectral similarity to its own final circuit across training, we find that50%50\\%of the post\-grokking reorganization is reached at a median of350350steps after the test\-accuracy transition, and90%90\\%at a median of12501250–18501850steps, in every one of3232seeds per task\. Circuit structure thus continues to consolidate well after generalization is behaviourally complete; our contribution here is to quantify this on independent seeds with a spectral identity measure, complementing qualitative reports of a post\-grokking cleanup phase\(Nandaet al\.,[2023](https://arxiv.org/html/2607.06628#bib.bib2)\)\.

#### No two seeds share a circuit\.

At a similarity threshold of0\.90\.9, all3232seeds per task occupy distinct circuits; mean pairwise similarity is0\.180\.18–0\.200\.20, near the1/d1/\\sqrt\{d\}baseline for the fingerprint dimension, and strictly positive for every pair\. Independently trained networks converge to distinct but not unrelated spectral structures; whether the shared positive component reflects a common substrate is left open\.

## 8Optimizer\-state ablation

AdamW carries, in addition to the weightsθ\\theta, a per\-parameter history of gradient statistics \(the first and second moment estimatesm,vm,v\)\. When we build a chimera and continue training, this history could in principle belong to either parent run rather than being a property ofθ\\thetaitself\. We test this directly: for both the endpoint\-swap intervention \(Section[4](https://arxiv.org/html/2607.06628#S4)\) and the threshold\-localization intervention \(Section[5](https://arxiv.org/html/2607.06628#S5)\), we repeat the experiment under three sources for the continuation optimizer’s moments—reset\(freshly initialized, as reported above\),recipient\(transplantAA’s moments\), anddonor\(transplantBB’s moments\)—and ask whether the reported effect survives all three\.

#### Circuit\-identity transfer \(C1\) is unaffected\.

Table[1](https://arxiv.org/html/2607.06628#S8.T1)reportsCFS​\_​lean\\mathrm\{CFS\\\_lean\}for theradialandreverse\_radialendpoint swaps under all three optimizer conditions\. The sign is consistent in10/1010/10pairs in every cell but one \(9/109/10forradialon addition underrecipient\), and mean magnitudes shift by at most0\.080\.08across conditions\. The identity\-transfer effect is a property of the weight configuration, not of which run’s gradient history the optimizer happens to carry\.

Table 1:MeanCFS​\_​lean\\mathrm\{CFS\\\_lean\}\(sign\-correct pairs / total\) for the two endpoint\-swap variants under the three optimizer\-state conditions\. Sign and magnitude are stable across all conditions\.
#### The threshold–norm relationship \(C5\) is unaffected\.

Repeating the full bisection procedure underrecipientanddonormoments, on all2020pairs across both tasks, the slow/fast recipient\-norm groups separate perfectly under every one of the three conditions \(Figure[6](https://arxiv.org/html/2607.06628#S8.F6)\)\. The separation gap narrows somewhat under moment transplantation \(e\.g\. modular addition:0\.340\.34under reset versus0\.250\.25under recipient and donor\) but never closes, and the per\-task exact permutation probability is unchanged because the same slow/fast grouping separates in every condition \(2\.2×10−22\.2\\\!\\times\\\!10^\{\-2\}for addition,8\.3×10−38\.3\\\!\\times\\\!10^\{\-3\}for multiplication, in each of the three conditions\)\. Treating the six task–condition separations as independent, the joint probability of this pattern under the null is6\.3×10−126\.3\\\!\\times\\\!10^\{\-12\}\. Per\-pair,t⋆t^\{\\star\}shifts by an average of0\.030\.03–0\.040\.04when moments are transplanted—about twice the bisection resolution \(±1/64\\pm 1/64\), and small relative to the0\.180\.18–0\.340\.34gap that separates the two recipient\-norm groups\. We conclude that the threshold’s dependence on recipient norm is a property ofθ\\theta, with optimizer moments contributing a secondary, non\-decisive perturbation\.

![Refer to caption](https://arxiv.org/html/2607.06628v1/x5.png)Figure 6:Optimizer\-state ablation for the threshold–norm relationship \(C5\)\.Bisection\-localizedt⋆t^\{\\star\}against recipient normrAr\_\{A\}, repeated under three sources for the continuation optimizer’s Adam moments: freshly reset \(left, as reported in Figure[4](https://arxiv.org/html/2607.06628#S5.F4)\), transplanted from the recipient \(middle\), and transplanted from the donor \(right\)\. All2020pairs, both tasks\. The slow/fast recipient\-norm groups separate without overlap under every condition; the gap narrows slightly under moment transplantation but never closes\.

## 9Discussion

directionuucircuit identityportabletransfers40/4040/40normrrthresholdt⋆t^\{\\star\}state\-dependentpredicts20/20 sep\.*which*solution*how susceptible*Figure 7:Division of labour between the two geometric components\.Direction carries a transferable component associated with circuit identity, i\.e\.*which*solution a chimera approaches; this is portable across independent trajectories regardless of recipient state\. Norm does not carry identity, but predicts the interpolation threshold at which a donor’s identity overwrites the recipient’s—i\.e\.*how susceptible*the recipient is to that overwrite, a property of the recipient’s dynamical state rather than of the donor’s content\.Our results map a division of labour between the two geometric components of a weight vector \(Figure[7](https://arxiv.org/html/2607.06628#S9.F7)\)\. Direction indexes*which*solution a trajectory approaches: transplanting it across independent runs transfers circuit identity, does so specifically to the donor’s content, and does so as a threshold switch between basins\. Norm plays a different role: it carries a weak, spatially distributed influence on*when*generalization occurs and no identity information, but it governs*how susceptible*an identity is to being overwritten—high\-norm recipients flip to a donor’s circuit under a small directional nudge, low\-norm recipients resist\. Interpreted through the basin picture, a chimera that is far from convergence \(high norm\) sits in a shallow region of the landscape where a modest directional change suffices to cross into a neighbouring basin, whereas a chimera near convergence \(low norm\) is already committed\. This interpretation is consistent with the norm–timescale relationship we reported previously for single\-run delay, but we do not claim it explains that relationship\.

The non\-additivity of the identity signal across hidden layers is an open puzzle\. We considered and ruled out a training\-instability explanation—the combined swaps re\-grok to full accuracy rather than destabilizing—leaving a genuine interaction among layer groups as the more likely account, which we do not resolve here\.

#### What does not follow from our results\.

Our findings do*not*show that direction alone determines circuit identity; that norm is irrelevant to representation; that basin geometry is fully characterized by weight norm; or that the observed dissociation generalizes beyond the two cyclic\-group tasks and single architecture studied here\. Each of these is a plausible next question, not a conclusion we are entitled to\.

## 10Limitations

We study a single architecture and two tasks from the same \(cyclic\-group\) family; whether the dissociation holds for non\-group tasks or larger models is untested\. We explored a non\-cyclic task \(sparse parity\) and found that it groks cleanly only under a different optimization regime \(weight decay0\.10\.1, lower learning rate\); rather than report a cross\-task “replication” confounded by mismatched hyperparameters, we leave the non\-cyclic extension to future work with the validated configuration in hand\. Our circuit\-identity metric is an embedding\-spectrum proxy for the full circuit, not a complete circuit equivalence, and the embedding\-swap localization result is circular for that reason \(Section[7](https://arxiv.org/html/2607.06628#S7)\)\. The thresholdt⋆t^\{\\star\}is localized on a grid of resolution±1/64\\pm 1/64; the norm–threshold relationship is established as a two\-class separation rather than a continuous law, because our pair selection produces well\-separated norm groups by design\. Continued\-training hyperparameters are held fixed regardless of donor and recipient origin\. The optimizer\-state ablation \(Section[8](https://arxiv.org/html/2607.06628#S8)\) covers all2020pairs for the bisection results but only two representative pairs per task for the underlying fine interpolation grid, matching the scope of the main fine\-grid analysis\.

## 11Conclusion

Cross\-trajectory chimera interventions dissociate the roles of weight magnitude and direction in grokking\. Direction carries a transferable, donor\-specific component associated with circuit identity and transfers it as a threshold switch between basins; the threshold’s location is predicted by the recipient’s norm, revealing that transferability depends on the recipient’s dynamical state\. Norm carries only a modest, distributed delay effect and no identity signal\. The adaptive bisection procedure we use to localize the threshold is a reusable tool for interventions that are stable under their training protocol\. Beyond the specific geometry of grokking, the chimera construction offers a general template for asking not just what a partially trained network contains, but what of it is portable to another\.

### Reproducibility Statement

All experiments use publicly reproducible task definitions, fixed random seeds recorded per run, and a pipeline that regenerates every table and figure from raw logs\. The full analysis \(pair selection, chimera and interpolation interventions, bisection, and statistical reporting\) is released as scripts with per\-job resume markers; each reported number is emitted directly by the analysis code from the experiment CSVs\.

## References

- S\. K\. Ainsworth, J\. Hayase, and S\. Srinivasa \(2023\)Git re\-basin: merging models modulo permutation symmetries\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2209\.04836Cited by:[§2](https://arxiv.org/html/2607.06628#S2.SS0.SSS0.Px2.p1.1)\.
- A\. C\. Cameron and D\. L\. Miller \(2015\)A practitioner’s guide to cluster\-robust inference\.Journal of Human Resources50\(2\),pp\. 317–372\.External Links:[Document](https://dx.doi.org/10.3368/jhr.50.2.317)Cited by:[§3](https://arxiv.org/html/2607.06628#S3.SS0.SSS0.Px5.p1.2)\.
- B\. Cullenet al\.\(2026\)A basin\-selection perspective on grokking via singular learning theory\.arXiv preprint arXiv:2603\.01192\.Cited by:[§2](https://arxiv.org/html/2607.06628#S2.SS0.SSS0.Px3.p1.1)\.
- R\. Entezari, H\. Sedghi, O\. Saukh, and B\. Neyshabur \(2021\)The role of permutation invariance in linear mode connectivity of neural networks\.arXiv preprint arXiv:2110\.06296\.Cited by:[§2](https://arxiv.org/html/2607.06628#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Fort, H\. Hu, and B\. Lakshminarayanan \(2019\)Deep ensembles: a loss landscape perspective\.arXiv preprint arXiv:1912\.02757\.Cited by:[§2](https://arxiv.org/html/2607.06628#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Frankle, G\. K\. Dziugaite, D\. M\. Roy, and M\. Carbin \(2020\)Linear mode connectivity and the lottery ticket hypothesis\.InInternational Conference on Machine Learning \(ICML\),pp\. 3259–3269\.Note:arXiv:1912\.05671Cited by:[§2](https://arxiv.org/html/2607.06628#S2.SS0.SSS0.Px3.p1.1)\.
- S\. H\. Hurlbert \(1984\)Pseudoreplication and the design of ecological field experiments\.Ecological Monographs54\(2\),pp\. 187–211\.External Links:[Document](https://dx.doi.org/10.2307/1942661)Cited by:[§3](https://arxiv.org/html/2607.06628#S3.SS0.SSS0.Px5.p1.2)\.
- Z\. Liu, O\. Kitouni, N\. Nolte, E\. J\. Michaud, M\. Tegmark, and M\. Williams \(2022\)Towards understanding grokking: an effective theory of representation learning\.arXiv preprint arXiv:2205\.10343\.Cited by:[§1](https://arxiv.org/html/2607.06628#S1.p2.5)\.
- N\. Nanda, L\. Chan, T\. Lieberum, J\. Smith, and J\. Steinhardt \(2023\)Progress measures for grokking via mechanistic interpretability\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2301\.05217Cited by:[§1](https://arxiv.org/html/2607.06628#S1.p1.1),[§3](https://arxiv.org/html/2607.06628#S3.SS0.SSS0.Px3.p1.11),[§7](https://arxiv.org/html/2607.06628#S7.SS0.SSS0.Px2.p1.6)\.
- A\. Power, Y\. Burda, H\. Edwards, I\. Babuschkin, and V\. Misra \(2022\)Grokking: generalization beyond overfitting on small algorithmic datasets\.arXiv preprint arXiv:2201\.02177\.Cited by:[§1](https://arxiv.org/html/2607.06628#S1.p1.1)\.
- V\. Varma, R\. Shah, Z\. Kenton, J\. Kramár, and R\. Kumar \(2023\)Explaining grokking through circuit efficiency\.arXiv preprint arXiv:2309\.02390\.Cited by:[§1](https://arxiv.org/html/2607.06628#S1.p1.1),[§1](https://arxiv.org/html/2607.06628#S1.p2.5)\.
- Z\. Xu, E\. Ni, Y\. Wang, and W\. Hu \(2025\)GrokTransfer: transferring features accelerates grokking in modular arithmetic\.arXiv preprint arXiv:2504\.13292\.Cited by:[§2](https://arxiv.org/html/2607.06628#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix ALayer\-localization of the identity signal

This appendix gives the full layer\-group analysis summarized in Section[7](https://arxiv.org/html/2607.06628#S7)\. Restricting the*directional*swap to individual layer groups shows that the identity\-carrying signal is not the embedding’s alone\. On modular addition, the attention, MLP, and unembedding groups each individually pull the final circuit toward the donor \(CFS​\_​lean≈−0\.6\\mathrm\{CFS\\\_lean\}\\approx\-0\.6each when swapped in theAA\-direction test\), yet swapping all of them together produces a markedly weaker lean \(−0\.145\-0\.145\): the contributions are non\-additive\. Modular multiplication reproduces the qualitative pattern—each hidden group carries the correct\-sign signal \(attention−0\.29\-0\.29, MLP−0\.45\-0\.45, unembedding−0\.32\-0\.32\) and the combination is again non\-additive—though per\-task magnitudes differ \(Figure[8](https://arxiv.org/html/2607.06628#A1.F8)\)\.

We flag one confound explicitly: because the circuit metric is computed*on*the embedding spectrum, directly swapping the embedding’s direction manipulates the very object being measured, so the embedding and “all\-but\-embedding” results are circular and we do not interpret them; the non\-circular hidden\-layer results are the basis for the redundancy claim above, which we regard as suggestive rather than established, given three pairs per task\.

![Refer to caption](https://arxiv.org/html/2607.06628v1/x6.png)Figure 8:The identity component is carried by multiple hidden groups, non\-additively\.AngularCFS​\_​lean\\mathrm\{CFS\\\_lean\}when the directional swap is restricted to one layer group \(mean over three pairs\)\. Attention, MLP, and unembedding each individually carry the correct\-sign signal, but combining them \(“attn\+mlp”\) gives a weaker lean than the strongest single group—the contributions are non\-additive\. The embedding\-based groups \(marked∗\*\) are circular because the circuit metric is computed on the embedding spectrum, and are not interpreted\.