Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics

# Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics
Source: [https://arxiv.org/html/2605.20441](https://arxiv.org/html/2605.20441)
###### Abstract

Grokking transformers trained on modular arithmetic exhibit sharp transitions between memorization, generalization, and collapse regimes\. We show that weight decay acts as a scalar empirical control parameter for these regimes, and we introduce two cheap online diagnostics, mean pairwise attention\-head cosine similarity and entropy standard deviation, that track the underlying training dynamics in these decoder\-only modular\-arithmetic settings from attention activations alone and complement loss\-landscape diagnostics at lower compute cost\. Across eleven experimental conditions and three model scales \(0\.82M to 85M parameters\), the weight\-decay axis separates memorization \(λ<λc\\lambda<\\lambda\_\{c\}, near\-zero grokking\), developmental grokking \(λ≥λc\\lambda\\geq\\lambda\_\{c\}, reaching∼\\sim100% grok rate byλ∼0\.1\\lambda\\sim 0\.1with time\-to\-grok decreasing1090→831090\\to 83epochs overλ∈\[0\.1,2\.0\]\\lambda\\in\[0\.1,2\.0\]\), and collapse \(λ=10\\lambda=10, identical attention patterns\)\. A near\-transition logistic fit localizes the memorization\-to\-developmental boundary atλc=0\.0158\\lambda\_\{c\}=0\.0158\(95% CI\[0\.0109,0\.0200\]\[0\.0109,0\.0200\],N=210N\{=\}210\); a power\-law fit to time\-to\-grok gives an empirical exponentν=0\.757\\nu=0\.757\(CI\[0\.725,0\.799\]\[0\.725,0\.799\]\)\. Tested reference exponentsν=1/2\\nu=1/2and 3D Isingν≈0\.63\\nu\\approx 0\.63lie outside the empirical CI under our four\-bin grid; we reportν\\nuas empirical and defer universality\-class identification to denser finite\-size\-scaling data collapse\(Bi et al\.,[2026](https://arxiv.org/html/2605.20441#bib.bib4)\)\. A horizon\-matched multi\-task replication \(n=280n\{=\}280, four modular operations, two scales\) preserves the WD\-control pattern beyond addition; a paired attention\-head re\-initialization experiment changes Phase\-2 amplitude at canonical post\-transitionλ=0\.05\\lambda=0\.05\(Cohen’sd=−1\.190d=\-1\.190,n=10n\{=\}10paired,pt=4\.5×10−3p\_\{t\}=4\.5\{\\times\}10^\{\-3\}\), while matched weight\-norm clipping does not, isolating the effect to head\-pattern structure rather than weight magnitude\. Per\-head dimensiond/Hd/Hmodulates differentiation amplitude saturating\-monotonically\. Three cross\-architecture scope probes \(4L MLPh=512h\{=\}512, 4L LSTMh=512h\{=\}512, and 4L Mambad=128d\{=\}128; each canonicaln=70n\{=\}70\) replicate the WD\-controlled transition in non\-attention architectures, with per\-architectureλc\\lambda\_\{c\}spanning roughly an order of magnitude \(MLPλc=0\.0511\\lambda\_\{c\}\{=\}0\.0511\[0\.0495,0\.0591\]\[0\.0495,0\.0591\]; LSTMλc=0\.0365\\lambda\_\{c\}\{=\}0\.0365\[0\.0299,0\.0473\]\[0\.0299,0\.0473\]; Mambaλc=0\.0144\\lambda\_\{c\}\{=\}0\.0144\[0\.0106,0\.0159\]\[0\.0106,0\.0159\], whose CI overlaps the transformer canonical CI rather than sitting above it\)\. Claims are scoped to modular arithmetic in small transformer attention models; architecture\-wide, language\-model, and universality\-class claims are out of scope\.

## 1Introduction

When transformers are trained on modular arithmetic, they exhibit*grokking*: a sudden transition from memorization to generalization long after training accuracy saturates\(Power et al\.,[2022](https://arxiv.org/html/2605.20441#bib.bib35)\)\. Mechanistic analysis reveals that this transition corresponds to the formation of internal circuits \(Fourier\-based representations\) that emerge gradually but manifest abruptly\(Nanda et al\.,[2023](https://arxiv.org/html/2605.20441#bib.bib29)\)\. Similarly, the formation of induction heads during transformer training constitutes a “phase change” visible from two\-layer models to 70B\+ parameter systems\(Olsson et al\.,[2022](https://arxiv.org/html/2605.20441#bib.bib32)\)\.

Across individual training phenomena \(grokking\(Power et al\.,[2022](https://arxiv.org/html/2605.20441#bib.bib35); Nanda et al\.,[2023](https://arxiv.org/html/2605.20441#bib.bib29); Kumar et al\.,[2024](https://arxiv.org/html/2605.20441#bib.bib18)\), emergent abilities\(Wei et al\.,[2022](https://arxiv.org/html/2605.20441#bib.bib56)\), lottery tickets\(Frankle & Carbin,[2019](https://arxiv.org/html/2605.20441#bib.bib10)\), neural collapse\(Papyan et al\.,[2020](https://arxiv.org/html/2605.20441#bib.bib33)\)\), recent learning\-mechanics perspectives frame staged learning, empirical laws, limiting models, and cheap measurable diagnostics as central objects for a future theory of deep learning\(Simon et al\.,[2026](https://arxiv.org/html/2605.20441#bib.bib42)\)\. The statistical mechanics community has connected training to phase transitions\(Bahri et al\.,[2020](https://arxiv.org/html/2605.20441#bib.bib3); Saxe et al\.,[2014](https://arxiv.org/html/2605.20441#bib.bib39); Ziyin & Ueda,[2023](https://arxiv.org/html/2605.20441#bib.bib67); Žunkovič & Ilievski,[2024](https://arxiv.org/html/2605.20441#bib.bib68)\), and recent work applies oscillator\-style synchronization models to neural activations\(Miyato et al\.,[2025](https://arxiv.org/html/2605.20441#bib.bib26)\)\. However, existing grokking studies rarely provide a cheap online activation\-space diagnostic and regime map for weight\-decay\-controlled attention dynamics\.

We make four contributions:

1. 1\.Quantitative weight\-decay critical threshold with bootstrap CIs and formal well\-formedness checks\.A dense weight\-decay sweep plus a sparse three\-size scale probe gives a horizon\-matched logistic transition estimateλc=0\.0158\\lambda\_\{c\}=0\.0158\(95% CI\[0\.0109,0\.0200\]\[0\.0109,0\.0200\],N=210N\{=\}210\) for the canonicalmod\+\\mathrm\{mod\}\_\{\+\}dense WD cohort, and a power\-law exponentν=0\.757\\nu=0\.757\(CI\[0\.725,0\.799\]\[0\.725,0\.799\]\) on time\-to\-grok aboveλc\\lambda\_\{c\}\. The two\-axis\(λ,N\)\(\\lambda,N\)map exhibits three qualitatively distinct regimes \(memorization, developmental, collapse\) with both “too little” and “too much” failure modes, complementing finite\-size\-scaling and weight\-geometry work that also treats weight decay as a phase\-relevant variable but does not pin down a numerical threshold\. Four diagnostic well\-formedness identities \(A1, B1, C1, E1\) are machine\-checked in Lean 4; they certify bounds and algebraic identities of the diagnostics, not the empirical regime claims\.
2. 2\.Cheap online order parameters for attention\-head coordination\.We define two scalar quantities computable at every training step: mean pairwise cosine similaritys¯​\(t\)\\bar\{s\}\(t\)and entropy standard deviationσH​\(t\)\\sigma\_\{H\}\(t\)\. These track attention\-head coordination through training, complement the refined local learning coefficient\(Wang et al\.,[2024](https://arxiv.org/html/2605.20441#bib.bib51)\)while requiring only forward\-pass attention weights, and we benchmark head\-to\-head on identical checkpoints\.
3. 3\.Two sequential phases plus seed\-dependent late\-stage retention failure\.Along the canonical training trajectory we document: Phase 1 \(attention\-head coordination, near grokking\) where heads converge ands¯\\bar\{s\}rises0\.93→0\.9950\.93\\to 0\.995; Phase 2 \(differentiation, post\-grokking\) where heads diverges¯→0\.88\\bar\{s\}\\to 0\.88while accuracy holds\. At twenty thousand training epochs the canonical seed\-42 trajectory extends into a five\-stage pattern exhibiting late\-stage accuracy collapse reminiscent of anti\-grokking; the 20K\-epoch long\-horizon retention cohort \(E8,n=20n\{=\}20\) shows this is a seed\-dependent fragility, not a universal cycle \(retention rates5/55/5,4/54/5,3/53/5,4/54/5atλ∈\{0\.1,0\.5,1\.0,2\.0\}\\lambda\\in\\\{0\.1,0\.5,1\.0,2\.0\\\}\)\. The collapse trajectory is qualitatively similar to the late\-stage cycle reported byPrakash & Martin \([2026a](https://arxiv.org/html/2605.20441#bib.bib36)\), though our spectral evidence \(§[4\.5](https://arxiv.org/html/2605.20441#S4.SS5)\) differs from theirs in timing\.
4. 4\.Per\-head dimension as amplitude modulator\.At fixed model dimension, varying the number of heads isolates per\-head dimensiond/Hd/Has an empirically dominant variable for differentiation amplitude: peakσH\\sigma\_\{H\}increases saturating\-monotonically withd/Hd/H\(monotone overd/H∈\{2,4,8,16\}d/H\\in\\\{2,4,8,16\\\}, plateauing atd/H≥16d/H\{\\geq\}16where bin means atd/H=16d/H\{=\}16andd/H=32d/H\{=\}32have overlapping 95% bin\-CIs\), reaching the same order of magnitude as random\-label null controls atd/H≈2d/H\\approx 2while remaining statistically distinguishable \(permutation\-testp=0\.009p\{=\}0\.009, Cohen’sd=1\.11d\{=\}1\.11\)\. We therefore label this an empirical architectural threshold in this setting and defer causal\-mechanism claims to follow\-up work\.

## 2Related Work

#### Grokking and training phase transitions\.

Power et al\. \([2022](https://arxiv.org/html/2605.20441#bib.bib35)\)discovered that small transformers generalize on algorithmic tasks long after memorization, a phenomenon explained mechanistically byNanda et al\. \([2023](https://arxiv.org/html/2605.20441#bib.bib29)\)as circuit formation followed by cleanup, and theoretically byVarma et al\. \([2023](https://arxiv.org/html/2605.20441#bib.bib49)\)as competition between memorizing and generalizing circuits driven by weight decay\.Kumar et al\. \([2024](https://arxiv.org/html/2605.20441#bib.bib18)\)frame grokking as a lazy\-to\-rich regime transition, whileLiu et al\. \([2023](https://arxiv.org/html/2605.20441#bib.bib19)\)extend grokking beyond algorithmic data\. Work from 2025 and 2026 has converged on framing grokking as a quantitative phase transition:Bi et al\. \([2026](https://arxiv.org/html/2605.20441#bib.bib4)\)apply finite\-size scaling with Binder\-cumulant crossings and spectral head\-tail contrast as order parameter;Wang \([2026a](https://arxiv.org/html/2605.20441#bib.bib52)\)andWang \([2026b](https://arxiv.org/html/2605.20441#bib.bib53)\)analyze effective dimensionality and self\-organized criticality via cascade\-dimension exponents;Acharya & Dhakal \([2026](https://arxiv.org/html/2605.20441#bib.bib1)\)connect grokking to variance\-limited spectral gating;Truong Xuan Khanh et al\. \([2026b](https://arxiv.org/html/2605.20441#bib.bib48)\)propose normalized spectral entropy of the representation covariance as a scalar threshold \(crossing∼0\.61\\sim\\\!0\.61before generalization\);Hennick & Corlouer \([2026](https://arxiv.org/html/2605.20441#bib.bib16)\)study reduced\-density\-matrix spectra as early warnings;Golwala \([2026](https://arxiv.org/html/2605.20441#bib.bib12)\)uses held\-out representation\-centroid geometry for early detection;Tian \([2025](https://arxiv.org/html/2605.20441#bib.bib46)\)provide provable three\-stage scaling laws in two\-layer networks\.Xu et al\. \([2026](https://arxiv.org/html/2605.20441#bib.bib57)\)prove delayed generalization in ridge regression under weight decay, whileZhang et al\. \([2026](https://arxiv.org/html/2605.20441#bib.bib65)\)give an SLT/algorithmic\-complexity abstraction view of grokking;Song & Ye \([2026](https://arxiv.org/html/2605.20441#bib.bib44)\)relate grokking on modular arithmetic to competing memorisation and generalisation timescales as functions of parameter count, complementary to our empirical WD×N\\times Nmap under a fixed protocol\. These results reinforce the value of cheap order parameters and weight\-decay\-controlled regimes, but do not supply the WD×N\\times Nattention\-head regime map studied here\.Lyu et al\. \([2024](https://arxiv.org/html/2605.20441#bib.bib21)\)establish the canonical early/late\-phase implicit\-bias dichotomy;Musat \([2025](https://arxiv.org/html/2605.20441#bib.bib27)\)characterize grokking as norm minimization on the zero\-loss manifold driven by weight decay;Manir & Rupa \([2026](https://arxiv.org/html/2605.20441#bib.bib22)\)find empirically that grokking is primarily determined by regularization and optimization rather than architecture\.Prakash & Martin \([2026a](https://arxiv.org/html/2605.20441#bib.bib36)\)report a “previously unreported third phase” \(late\-stage generalization collapse\) diagnosed via spectral density heavy\-tailedness, with follow\-up RMT/Correlation\-Trap framing inPrakash & Martin \([2026b](https://arxiv.org/html/2605.20441#bib.bib37)\)treating anti\-grokking as a long\-horizon overfitting phase\. We observe a qualitatively similar late\-collapse trajectory using attention\-similarity diagnostics and characterize its weight\-decay dependence, though our spectral evidence \(§[4\.5](https://arxiv.org/html/2605.20441#S4.SS5)\) differs from theirs because heavy\-tail structure forms during grokking onset rather than during the late cycle\.

#### Concurrent work on grokking geometry\.

Xu \([2026d](https://arxiv.org/html/2605.20441#bib.bib61)\)studies multi\-task grokking on modular arithmetic and identifies weight decay as a*phase parameter*that governs grokking timescale and curvature depth, reporting two qualitative regimes \(λ≥0\.5\\lambda\\\!\\geq\\\!0\.5fast vs\.λ≤0\.3\\lambda\\\!\\leq\\\!0\.3slow\) with complementary results inXu \([2026c](https://arxiv.org/html/2605.20441#bib.bib60);[a](https://arxiv.org/html/2605.20441#bib.bib58);[e](https://arxiv.org/html/2605.20441#bib.bib62);[f](https://arxiv.org/html/2605.20441#bib.bib63);[b](https://arxiv.org/html/2605.20441#bib.bib59)\)\. Our work differs in three respects\. First,Xu \([2026d](https://arxiv.org/html/2605.20441#bib.bib61)\)holds model sizeNNfixed throughout; we sweep bothλ\\lambdaandNNjointly, exposing a horizon\-matched transition estimateλc\\lambda\_\{c\}stable across the small/medium pair tested with overlapping 95% CIs \(fitλc=0\.0158\\lambda\_\{c\}=0\.0158, 95% CI\[0\.0109,0\.0200\]\[0\.0109,0\.0200\], from our dense WD\-sweep cohort ofN=210N\{=\}210runs post Phase A\) and a third*collapse*regime atλ\>5\\lambda\\\!\>\\\!5absent from Xu’s diagram\. Second, Xu’s diagnostics operate in weight and update space \(PCA trajectory variance, the commutator norm‖\[WQ,WK\]‖F\\\|\[W\_\{Q\},W\_\{K\}\]\\\|\_\{F\}, spectral\-edge gradient/decay decomposition, functional\-mode spectra\) and require full checkpoint access; our order parameters \(mean pairwise cosine similarity of attention activations and entropy standard deviation across heads\) are activation\-space diagnostics computable online inO​\(H2\)O\(H^\{2\}\)per evaluation step\. Third, Xu does not invoke synchronization or permutation\-symmetry\-reduction dynamics; the Phase 1 synchronization and Phase 2 head\-specialization framing, the five\-stage anti\-grokking trajectory, and thed/Hd/Hamplitude modulation are not present in their work\.Yıldırım \([2026](https://arxiv.org/html/2605.20441#bib.bib64)\)presents a counter\-framing in which grokking is bypassable via a uniform\-attention architectural ablation without weight decay; we do not claim that weight decay is*necessary*for generalization, only that within the standard attention architecture weight decay behaves as an empirical control parameter with a well\-defined regime diagram\.Tang et al\. \([2026](https://arxiv.org/html/2605.20441#bib.bib45)\)report a sharp rise inH1H\_\{1\}persistent\-homology features at grokking onset on modular arithmetic, an offline post\-hoc geometric diagnostic complementary to our online attention\-coordination order parameters; the two diagnostic classes target the same regime transitions at different signal locations \(representation topology vs head\-pattern coordination\) and different evaluation cost\.Wang et al\. \([2026](https://arxiv.org/html/2605.20441#bib.bib55)\)likewise localize grokking transitions via distributional spectral coordinates \(Wasserstein/quantile, Hankel DMD residual, effective rank\) on modular\-addition Transformer trajectories; this is another transition\-localization diagnostic at a different signal location \(weight/activation spectra\) rather than an attention\-head activation order parameter\.Ali \([2026](https://arxiv.org/html/2605.20441#bib.bib2)\)examine WD\-placement timing on compositional tasks and report a critical training window phenomenon explicitly absent on modular arithmetic, an orthogonal axis \(when\-WD\) to our amount\-based regime mapping \(how\-much\-WD\)\.Gomezjurado Gonzalez \([2026](https://arxiv.org/html/2605.20441#bib.bib13)\)attribute delayed generalization in encoder\-decoder Collatz prediction to a representation\-access bottleneck rather than feature\-acquisition failure, complementary to our weight\-decay\-controlled attention\-coordination signal in decoder\-only modular\-arithmetic models\.Lyle et al\. \([2025](https://arxiv.org/html/2605.20441#bib.bib20)\)connect grokking\-style feature\-learning dynamics to nonstationary continual\-learning primacy bias via effective learning rate, framing grokking as a pump for plasticity orthogonal to our weight\-decay regime diagram\.

#### Emergent abilities in LLMs\.

Wei et al\. \([2022](https://arxiv.org/html/2605.20441#bib.bib56)\)documented∼\\sim100 abilities appearing sharply with scale, thoughSchaeffer et al\. \([2023](https://arxiv.org/html/2605.20441#bib.bib41)\)argued some are metric artifacts\. The quantization model ofMichaud et al\. \([2023](https://arxiv.org/html/2605.20441#bib.bib24)\)hypothesizes discrete skill acquisition, whileNam et al\. \([2024](https://arxiv.org/html/2605.20441#bib.bib28)\)provide analytically tractable models of ordered emergence\.

#### Attention head specialization\.

Voita et al\. \([2019](https://arxiv.org/html/2605.20441#bib.bib50)\)showed that few heads carry most computation, whileMichel et al\. \([2019](https://arxiv.org/html/2605.20441#bib.bib25)\)demonstrated 70 to 90% of heads are prunable\.Olsson et al\. \([2022](https://arxiv.org/html/2605.20441#bib.bib32)\)documented a “phase change” during training where induction heads form abruptly\.Chen et al\. \([2024](https://arxiv.org/html/2605.20441#bib.bib6)\)prove that induction\-head formation follows gradient flow in the infinite\-time limit \(continuous convergence; not a claim of discrete loss drops in finite\-step SGD\)\. Most directly related to our Phase 2 finding,Wang et al\. \([2024](https://arxiv.org/html/2605.20441#bib.bib51)\)introduce the refined local learning coefficient \(rLLC\) as an SGLD\-estimated local\-learning\-coefficient diagnostic for staged head differentiation during training; we benchmark our cosine\-similarity\-based order parameter against rLLC on the identical 11\-checkpoint canonical 4L8H trajectory \(epochs 100 to 20 000, devinterp 1\.0\.0 SGLD, 3 chains×\\times200 draws per ckpt\) and find Pearson correlationr=0\.46r\{=\}0\.46betweenLLC​\(t\)\\mathrm\{LLC\}\(t\)andσH​\(t\)\\sigma\_\{H\}\(t\)\(then=11n\{=\}11ckpt benchmark gives a wide Fisher\-zz95% CI\[−0\.20,0\.82\]\[\-0\.20,0\.82\]that is consistent with anything from no correlation to a strong one\), indicating the two diagnostics are*complementary rather than equivalent*:σH\\sigma\_\{H\}tracks attention\-entropy dispersion across heads \(a representation\-space signal\), while rLLC tracks local\-loss\-landscape geometry, and the two correlate only moderately across the five phases at the ckpt\-count tested\. We therefore do not claimσH\\sigma\_\{H\}replaces rLLC; instead it provides a cheap, online\-computable complement that captures a related but distinct aspect of Phase\-2 dynamics\.Sagitova et al\. \([2026](https://arxiv.org/html/2605.20441#bib.bib38)\)provide a theoretical account of softmax\-head specialization as a high\-dimensional staged transition in a single\-location model\.

#### Statistical mechanics of deep learning\.

Bahri et al\. \([2020](https://arxiv.org/html/2605.20441#bib.bib3)\)connected loss landscapes to spin glasses and dynamical phase transitions\.Saxe et al\. \([2014](https://arxiv.org/html/2605.20441#bib.bib39)\)derived exact learning dynamics for deep linear networks showing plateau\-transition structure\.Saxe et al\. \([2019](https://arxiv.org/html/2605.20441#bib.bib40)\)used these dynamics to model semantic development with stage\-like transitions matching developmental psychology data, though restricted to linear networks and toy datasets\.Poole et al\. \([2016](https://arxiv.org/html/2605.20441#bib.bib34)\)showed networks at the “edge of chaos” achieve exponential expressivity\.

#### Synchronization physics in neural networks\.

Miyato et al\. \([2025](https://arxiv.org/html/2605.20441#bib.bib26)\)replaced threshold neurons with Kuramoto oscillators \(AKOrN\), demonstrating improved robustness and reasoning through phase\-based synchronization\.Nguyen et al\. \([2024](https://arxiv.org/html/2605.20441#bib.bib30)\)applied the Kuramoto model to prevent over\-smoothing in GNNs\.Hays \([2026](https://arxiv.org/html/2605.20441#bib.bib15)\)replaces standard self\-attention with a closed\-form Kuramoto steady\-state operator that turns each token into a learnable\-frequency oscillator and reads attention weights from phase\-locking strength\. These approaches all apply Kuramoto to*representations*at inference time; we apply it as a qualitative analogy to*training dynamics*over time and explicitly disclaim a quantitative Kuramoto\-model fit \(§[3\.3](https://arxiv.org/html/2605.20441#S3.SS3)\)\.

#### Developmental landscapes and weight decay mechanism\.

Boukacem et al\. \([2024](https://arxiv.org/html/2605.20441#bib.bib5)\)demonstrated Waddington\-like sequential bifurcations in generalized Hopfield networks on MNIST; we extend this staged\-dynamics framing to transformer training dynamics while noting that our observed post\-grokking transition is an empirical head\-specialization event, distinct from the saddle\-node\-of\-saddles bifurcation ofBoukacem et al\. \([2024](https://arxiv.org/html/2605.20441#bib.bib5)\)\. The mechanistic role of weight decay has been clarified by recent work:D’Angelo et al\. \([2023](https://arxiv.org/html/2605.20441#bib.bib8)\)provide a modern account of why weight decay is needed in deep learning;Galanti et al\. \([2022](https://arxiv.org/html/2605.20441#bib.bib11)\)show that SGD with weight decay secretly minimizes effective rank;Singh et al\. \([2026](https://arxiv.org/html/2605.20441#bib.bib43)\)demonstrate that architectural choices such as layer\-normalization placement strongly modulate grokking dynamics;Wang & Aitchison \([2024](https://arxiv.org/html/2605.20441#bib.bib54)\)derive scaling rules for AdamW weight decay across model and dataset size by treating learned weights as an exponential moving average of recent updates, complementing our trajectory\-based amplification fit\.Truong Xuan Khanh et al\. \([2026a](https://arxiv.org/html/2605.20441#bib.bib47)\)derive a quantitative scaling law for the grokking delayTgrok−Tmem=Θ​\(\(1/γeff\)​log⁡\(‖θmem‖2/‖θpost‖2\)\)T\_\{\\text\{grok\}\}\-T\_\{\\text\{mem\}\}=\\Theta\(\(1/\\gamma\_\{\\text\{eff\}\}\)\\log\(\\\|\\theta\_\{\\text\{mem\}\}\\\|^\{2\}/\\\|\\theta\_\{\\text\{post\}\}\\\|^\{2\}\)\)via Lyapunov contraction, withγeff≥η​λ\\gamma\_\{\\text\{eff\}\}\\geq\\eta\\lambdafor AdamW; our §[C\.5](https://arxiv.org/html/2605.20441#A3.SS5)provides the empirical calibration of the AdamW amplification \(κ\\kappa\) for the canonical 4L8H mod\-add cohort, plus the inverted formulation as aλc\\lambda\_\{c\}threshold under a fixed training horizon\.Zhang et al\. \([2025](https://arxiv.org/html/2605.20441#bib.bib66)\)frame grokking as a glass\-physics relaxation analogically \(memorisation as rapid cooling into a glassy state, generalisation as slow relaxation\), which is a different abstraction from the mechanistic AdamW\-update relaxation argument we deploy in §[C\.5](https://arxiv.org/html/2605.20441#A3.SS5)\.

#### Biological cardiac synchronization \(terminology origin only\)\.

The staged\-coordination vocabulary used in this paper was originally motivated by cardiac synchronization literature:Jia et al\. \([2023](https://arxiv.org/html/2605.20441#bib.bib17)\)showed the first vertebrate heartbeat is a saddle\-node\-on\-invariant\-circle \(SNIC\) bifurcation;Nitsan et al\. \([2016](https://arxiv.org/html/2605.20441#bib.bib31)\)demonstrated mechanical extracellular\-matrix\-mediated synchronization;Chiou et al\. \([2016](https://arxiv.org/html/2605.20441#bib.bib7)\)showed embryonic hearts coordinate mechanically before electrical infrastructure matures\. We make no biological claim; §[5](https://arxiv.org/html/2605.20441#S5)explains why the empirical scaling exponent rules out the SNIC analogy\.

## 3Methods

### 3\.1Task and Model

We train small decoder\-only transformers on modular arithmetic followingPower et al\. \([2022](https://arxiv.org/html/2605.20441#bib.bib35)\)\. Formod\+\\mathrm\{mod\}\_\{\+\},mod×\\mathrm\{mod\}\_\{\\times\}, andmod−\\mathrm\{mod\}\_\{\-\}, inputs are pairs\(a,b\)∈\{0,…,p−1\}2\(a,b\)\\in\\\{0,\\dots,p\-1\\\}^\{2\}withp=97p=97\(sop2=9409p^\{2\}=9409total examples\), half used for training and half held out under a seed\-controlled permutation\. Formod÷\\mathrm\{mod\}\_\{\\div\}, pairs withb=0b=0are excluded before the same split, because division by zero is undefined modulo primepp\. The canonical architecture is a 4\-layer, 8\-head transformer withdmodel=128d\_\{\\text\{model\}\}\{=\}128, FFN width 512, dropout 0, pre\-LayerNorm\. We additionally sweep layers∈\{2,4,6,12\}\\in\\\{2,4,6,12\\\}, heads∈\{4,8,16,32,64\}\\in\\\{4,8,16,32,64\\\}, anddmodel∈\{32,64,128,256,512,768\}d\_\{\\text\{model\}\}\\in\\\{32,64,128,256,512,768\\\}for the finite\-size\-scaling and scale\-axis experiments, yielding parameter counts from 0\.82 M \(small,4×8×1284\{\\times\}8\{\\times\}128,dff=512d\_\{\\text\{ff\}\}\{=\}512\) through 19 M \(medium,6×8×5126\{\\times\}8\{\\times\}512,dff=2048d\_\{\\text\{ff\}\}\{=\}2048\) to 85 M \(large,12×12×76812\{\\times\}12\{\\times\}768,dff=3072d\_\{\\text\{ff\}\}\{=\}3072\)\. All runs use AdamW atlr=10−3\\mathrm\{lr\}\{=\}10^\{\-3\}, batch 512, cross\-entropy loss\. The canonical\(p,L,H,d,lr,batch\)=\(97,4,8,128,10−3,512\)\(p,L,H,d,\\mathrm\{lr\},\\mathrm\{batch\}\)=\(97,4,8,128,10^\{\-3\},512\)choice follows the Power 2022 modular\-arithmetic grokking baseline used byNanda et al\. \([2023](https://arxiv.org/html/2605.20441#bib.bib29)\); Kumar et al\. \([2024](https://arxiv.org/html/2605.20441#bib.bib18)\); Bi et al\. \([2026](https://arxiv.org/html/2605.20441#bib.bib4)\), ensuring direct comparability with the 2026 grokking\-wave literature; we sweepL,H,dmodelL,H,d\_\{\\mathrm\{model\}\}for finite\-size\-scaling but holdpp,lr\\mathrm\{lr\}, and batch fixed\. Sensitivity to alternative regularizers \(dropout, label smoothing\) and optimizers \(SGD, Lion, Sophia\) is out of scope and deferred\. Seeds are drawn from a 10\-seed base set𝒮0=\{7,11,31,42,73,97,123,199,401,977\}\\mathcal\{S\}\_\{0\}=\\\{7,11,31,42,73,97,123,199,401,977\\\}atn=10n\{=\}10per cell\. A 20\-seed Phase\-A extension𝒮1=\{2,3,5,17,23,29,53,67,89,103,113,137,149,167,181,197,211,229,241,263\}\\mathcal\{S\}\_\{1\}=\\\{2,\\allowbreak 3,\\allowbreak 5,\\allowbreak 17,\\allowbreak 23,\\allowbreak 29,\\allowbreak 53,\\allowbreak 67,\\allowbreak 89,\\allowbreak 103,\\allowbreak 113,\\allowbreak 137,\\allowbreak 149,\\allowbreak 167,\\allowbreak 181,\\allowbreak 197,\\allowbreak 211,\\allowbreak 229,\\allowbreak 241,\\allowbreak 263\\\}adds replication atλ∈\{0\.01,0\.1,1\.0\}\\lambda\\in\\\{0\.01,0\.1,1\.0\\\}for then=30n\{=\}30near\-critical and canonical cells\.

### 3\.2Order Parameters

LetAl​h​\(t\)∈ℝB×T×TA\_\{lh\}\(t\)\\in\\mathbb\{R\}^\{B\\times T\\times T\}be the attention\-weight tensor of headhhin layerllat training steptt, whereBBis the evaluation batch size andT=3T\{=\}3is the token sequence length after appending the output\-query token\. We define two scalar order parameters, evaluated every 10 training steps:

s¯​\(t\)\\displaystyle\\bar\{s\}\(t\)≔𝔼l​\[2H​\(H−1\)​∑i<jcos⁡\(vec​\(Al​i\),vec​\(Al​j\)\)\]\\displaystyle\\coloneqq\\mathbb\{E\}\_\{l\}\\\!\\left\[\\tfrac\{2\}\{H\(H\-1\)\}\\sum\_\{i<j\}\\cos\\\!\\big\(\\mathrm\{vec\}\(A\_\{li\}\),\\mathrm\{vec\}\(A\_\{lj\}\)\\big\)\\right\]\\quad\(pairwise head cosine\)\(1\)σH​\(t\)\\displaystyle\\sigma\_\{H\}\(t\)≔𝔼l​\[Stdh​\(H​\[Al​h\]\)\]\\displaystyle\\coloneqq\\mathbb\{E\}\_\{l\}\\\!\\left\[\\mathrm\{Std\}\_\{h\}\\\!\\left\(H\[A\_\{lh\}\]\\right\)\\right\]\\quad\(entropy std over heads\)\(2\)HereH​\[⋅\]H\[\\cdot\]is Shannon entropy\. Both diagnostics use only the attention weights already produced by the forward pass;s¯\\bar\{s\}andσH\\sigma\_\{H\}cost𝒪​\(L​H2​B​T2\)\\mathcal\{O\}\(LH^\{2\}BT^\{2\}\)and𝒪​\(L​H​B​T2\)\\mathcal\{O\}\(LHBT^\{2\}\)respectively per evaluation step, compared with checkpoint\-level SGLD estimation for the rLLC diagnostic ofWang et al\. \([2024](https://arxiv.org/html/2605.20441#bib.bib51)\)\(dominated by repeated model evaluations and sampling chains rather than head\-count\)\. SinceT=3T=3is fixed in these modular\-arithmetic runs, the diagnostic overhead is dominated by batch size and head count rather than model parameters\. On the canonical 4L8H model, evaluatings¯\\bar\{s\}andσH\\sigma\_\{H\}adds approximately3%3\\%wall\-clock overhead amortized over the every\-10\-step cadence used here\. Two complementary controls \(Kuramoto coherencerϕr\_\{\\phi\}and similarity\-matrix spectral gapλg\\lambda\_\{g\}\) are tracked in supplementary trace JSONs but not used for the regime map or causal contrasts; their definitions and selection rationale are in Appendix[B](https://arxiv.org/html/2605.20441#A2)\.

### 3\.3Synchronization Analogy and Tanh Fit

As a loose synchronization analogy, one can treat each head as an oscillator whose phase is the principal attention\-pattern direction\. Empirically we fit the pre\-grokking rise ofs¯​\(t\)\\bar\{s\}\(t\)to the overdamped tanh forms¯​\(t\)=s0\+A​tanh⁡\(\(t−tc\)/τ\)\\bar\{s\}\(t\)=s\_\{0\}\+A\\tanh\(\(t\-t\_\{c\}\)/\\tau\), which matches the shape of a uniform\-coupling mean\-field synchronization model with time\-independent drive\. Ofn=50n\{=\}50canonical runs \(post Phase A; expanded fromn=22n\{=\}22pre\-Phase\-A\), the unconstrained tanh fit achievesR2\>0\.9R^\{2\}\{\>\}0\.9in 10 runs andR2\>0\.7R^\{2\}\{\>\}0\.7in 33 runs \(medianR2=0\.843R^\{2\}\{=\}0\.843\)\. However,16/5016/50of those unconstrained fits achieve high apparentR2R^\{2\}viatanh\\tanhsaturating to a constant \(s0,As\_\{0\},Adiverging with opposite signs over the fit window\), which is a fit\-form degeneracy on a near\-flat Phase 1 plateau rather than a successful synchronization fit\. Restricting to physically interpretable parameters \(s0∈\[0,1\.5\]s\_\{0\}\\in\[0,1\.5\],A∈\[0,1\]A\\in\[0,1\],\|tc\|,τ<5000\|t\_\{c\}\|,\\tau<5000\) yields34/5034/50valid fits with medianR2=0\.63R^\{2\}\{=\}0\.63and only4/344/34atR2\>0\.9R^\{2\}\{\>\}0\.9\. This supports only a qualitative synchronization analogy for Phase 1; it is not quantitative validation of a Kuramoto model, and explicit coupling\-constant extraction from the AdamW update is deferred\.

### 3\.4Finite\-Size Scaling

Following finite\-size\-scaling analyses of grokking\(Bi et al\.,[2026](https://arxiv.org/html/2605.20441#bib.bib4); Wang,[2026a](https://arxiv.org/html/2605.20441#bib.bib52)\), we test whether time\-to\-grok nearλc\\lambda\_\{c\}followstgrok∝\(λ−λc\)−νt\_\{\\text\{grok\}\}\\propto\(\\lambda\-\\lambda\_\{c\}\)^\{\-\\nu\}\. We estimateλc\\lambda\_\{c\}by fitting a logistic toP​\(grok\)P\(\\mathrm\{grok\}\)vslog⁡λ\\log\\lambdaacrossN=210N\{=\}210small\-scale runs \(logistic cohort post Phase A; expanded fromN=150N\{=\}150pre\-Phase\-A vian=30n\{=\}30replication atλ∈\{0\.01,0\.1,1\.0\}\\lambda\\in\\\{0\.01,0\.1,1\.0\\\}\), then fitν\\nuby linear regression oflog⁡tgrok\\log t\_\{\\text\{grok\}\}onlog⁡\(λ−λc\)\\log\(\\lambda\-\\lambda\_\{c\}\)restricted to grokking runs\. The scale axisN∈\{0\.82,19,85\}N\\in\\\{0\.82,19,85\\\}\\,M is sampled atλ∈\{0\.01,0\.1,1\.0\}\\lambda\\in\\\{0\.01,0\.1,1\.0\\\}withn=10n\{=\}10seeds per cell\. This supports the empiricalλ×N\\lambda\\times Nphase map but remains insufficient for full finite\-size data collapse because the weight\-decay grid has only three bins \(instrumented but underpowered; denser grids andn≥20n\{\\geq\}20per cell targeted for future work\)\. Confidence intervals are reported from nonparametric bootstraps \(1,500 resamples for the logisticλc\\lambda\_\{c\}and power\-lawν\\nufits\) and from leave\-one\-seed\-out jackknife; the residual\-bootstrap sensitivity check onν\\nuuses 5,000 resamples\.

## 4Results

### 4\.1Two\-axis regime diagram

Figure[1](https://arxiv.org/html/2605.20441#S4.F1)maps the empiricalλ×N\\lambda\\times Nregime diagram fromN=210N\{=\}210WD\-sweep runs \(post Phase A\) andn=90n\{=\}90three\-scale sweep runs \(E5; 3 sizes×\\times3 WDs×\\times10 seeds\)\. Along the WD axis at smallNN:λ<0\.0158\\lambda\{<\}0\.0158yields near\-zero grokking;λ∈\[0\.1,2\.0\]\\lambda\\in\[0\.1,2\.0\]gives roughly 90 to 100% grokking with monotone time\-to\-grok decrease1090→831090\\to 83epochs;λ=10\\lambda=10collapses all heads to identical patterns \(s¯=1\.000\\bar\{s\}=1\.000exact\)\. Along theNNaxis atλ=1\.0\\lambda=1\.0: small \(0\.82 M\) groks reliably; large \(85 M\) collapses into a null state \(s¯=1\.000\\bar\{s\}=1\.000,σH=0\.000\\sigma\_\{H\}=0\.000\) by epoch 3000\. The available scale grid therefore supports scale dependence under the stated training horizons, but not a monotoneλc​\(N\)\\lambda\_\{c\}\(N\)law; the completed horizon\-matched small/medium follow\-up \(E7\) and multi\-task pooled refit \(E9\) are used only to rule out a clean small\-to\-medium monotone\-threshold claim, not to identify a new scale law\.

![Refer to caption](https://arxiv.org/html/2605.20441v1/x1.png)Figure 1:Two\-axis empirical regime diagram acrossn=300n\{=\}300runs post Phase A \(210 WD\-sweep logistic cohort \+ 90 three\-scale runs\)\. Filled black circles: grokking runs\. Blue crosses: non\-grokking \(memorization or collapse\)\. Shaded regions: regimes along theλ\\lambdaaxis\. Dashed vertical line:λc=0\.0158\\lambda\_\{c\}\{=\}0\.0158logistic fit\.
### 4\.2Two\-phase dynamics and five\-phase long\-horizon cycle

Figure[2](https://arxiv.org/html/2605.20441#S4.F2)shows the replicated canonical cohort \(n=50n\{=\}50\) exhibiting sharp Phase 1 synchronization \(medians¯\\bar\{s\}rises during epochs 100 to 200\) followed by Phase 2 differentiation \(IQR\-widening ins¯\\bar\{s\}andσH\\sigma\_\{H\}during epochs 1000 to 3000\) while test accuracy remains near plateau\. At 20 000 epochs \(Figure[3](https://arxiv.org/html/2605.20441#S4.F3)\) the canonical seed\-42 trajectory elongates into five annotated stages terminating in anti\-grokking collapse \(s¯→0\.99\\bar\{s\}\\to 0\.99, test accuracy→0\.46\\to 0\.46\); a44\-seed cohort underlay \(seeds\{7,11,31,123\}\\\{7,11,31,123\\\}at matched 4L8Hλ=1\.0\\lambda\{=\}1\.020 00020\\,000\-epoch configuration via the cross\-seed checkpoint cohort\) plotted alongside the canonical detail discloses cross\-seed variability directly\. The horizon\-matched long\-horizon retention cohort \(E8;n=5n\{=\}5seeds at each ofλ∈\{0\.1,0\.5,1\.0,2\.0\}\\lambda\\in\\\{0\.1,0\.5,1\.0,2\.0\\\},2020runs total at 20 000 epochs\) confirms the five\-stage pattern is a*seed\-dependent fragility*rather than a universal cycle: retention rates are5/55/5atλ=0\.1\\lambda\{=\}0\.1,4/54/5atλ=0\.5\\lambda\{=\}0\.5,3/53/5atλ=1\.0\\lambda\{=\}1\.0, and4/54/5atλ=2\.0\\lambda\{=\}2\.0, so collapse concentrates at higher WD but a non\-trivial fraction of seeds retains generalization across the full horizon at every WD tested\. The five\-stage canonical figure shows the strongest cycle pronouncement, occurring in roughly one third of canonical\-WD seeds; the cross\-seed overlay is the direct visual confirmation\. The pattern is qualitatively consistent with the third\-phase phenomenology ofPrakash & Martin \([2026a](https://arxiv.org/html/2605.20441#bib.bib36)\)\.

![Refer to caption](https://arxiv.org/html/2605.20441v1/x2.png)Figure 2:Canonical two\-phase dynamics atλ=1\.0\\lambda\{=\}1\.0overn=50n\{=\}50replicated 4L8H runs\. Solid curves show cohort medians; bands show interquartile ranges\. Phase 1 \(yellow shading\): synchronization \+ grokking\. Phase 2 \(blue shading\): differentiation \+ test\-accuracy plateau\.![Refer to caption](https://arxiv.org/html/2605.20441v1/x3.png)Figure 3:Long\-horizon canonical seed\-4242trajectory with a44\-seed matched underlay \(\{7,11,31,123\}\\\{7,11,31,123\\\}\) at 4L8H mod\-addλ=1\.0\\lambda\{=\}1\.020 00020\\,000\-epoch configuration\. Bold colored traces \(canonical seed4242\): raw per\-epoch values at low alpha plus moving\-average\-smoothed overlay\. Gray traces: smoothed cross\-seed trajectories underlaid to disclose cohort variability\. P1–P5 bands are canonical seed\-4242landmark windows, not per\-seed fitted boundaries\. P1: attention\-head coordination\. P2: first differentiation\. P3: re\-sync\. P4: second differentiation \(canonical rawσH\\sigma\_\{H\}peak0\.2010\.201at epoch∼\\sim10 500; smoothed peak∼\\sim0\.10\)\. P5: collapse with canonical test\-accuracy decay to0\.460\.46\. The five\-stage pattern is most pronounced for the canonical seed; the cross\-seed underlay shows that 1–2 of the four additional seeds exhibit terminal late\-cycle accuracy collapse and the remaining seeds recover or retain generalization by the full horizon, consistent with the seed\-dependent fragility documented in the E8 retention cohort\.
### 4\.3Per\-head dimension as amplitude modulator

At fixed total model dimensiondd, varying head countHHscans per\-head dimensiond/Hd/H\. Figure[4](https://arxiv.org/html/2605.20441#S4.F4)shows peakσH\\sigma\_\{H\}following a saturating\-exponential profileσHmax=c\+a​\(1−e−b​\(d/H\)\)\\sigma\_\{H\}^\{\\max\}=c\+a\(1\-e^\{\-b\(d/H\)\}\)acrossn=44n\{=\}44small\-scale runs \(Akaike information criterion, AIC, preferred over linear, log, and power\-law alternatives\), approaching the random\-label\-null scale0\.087±0\.0250\.087\\pm 0\.025atd/H≈2d/H\\approx 2\.

![Refer to caption](https://arxiv.org/html/2605.20441v1/x4.png)Figure 4:PeakσH\\sigma\_\{H\}vs per\-head dimensiond/Hd/H\. Saturating\-exponential AIC\-preferred\. Dotted line: random\-label null\-control scale reference, not an equivalence claim\.
### 4\.4Scaling exponent and universality\-class limits

Figure[5](https://arxiv.org/html/2605.20441#S4.F5)A shows the logisticP​\(grok\)P\(\\mathrm\{grok\}\)fit locatingλc=0\.0158\\lambda\_\{c\}=0\.0158\(95% CI\[0\.0109,0\.0200\]\[0\.0109,0\.0200\]\) acrossN=210N\{=\}210WD\-axis runs \(logistic cohort, 13 WD bins withn=10n\{=\}10to3030seeds per bin, Phase A replication atλ∈\{0\.01,0\.1,1\.0\}\\lambda\\in\\\{0\.01,0\.1,1\.0\\\}\)\. Figure[5](https://arxiv.org/html/2605.20441#S4.F5)B shows the power\-lawtgrok∝\(λ−λc\)−νt\_\{\\text\{grok\}\}\\propto\(\\lambda\-\\lambda\_\{c\}\)^\{\-\\nu\}fit withν=0\.757\\nu=0\.757\(95% CI\[0\.725,0\.799\]\[0\.725,0\.799\],n=140n\{=\}140grok\-positive runs in theN=210N\{=\}210logistic cohort\); a jackknife on the multi\-task\-extended grok\-positive cohort \(n=148n\{=\}148\) gives bias\-correctedν=0\.761\\nu\{=\}0\.761\(\[0\.723,0\.799\]\[0\.723,0\.799\]\) with residual\-bootstrap CI\[0\.738,0\.786\]\[0\.738,0\.786\]\. Theseν\\nuintervals are conditional on the fitted point estimateλc=0\.0158\\lambda\_\{c\}=0\.0158and do not propagate uncertainty inλc\\lambda\_\{c\}itself; a fully joint\(λc,ν\)\(\\lambda\_\{c\},\\nu\)bootstrap is deferred to the denser\-grid finite\-size\-scaling work cited next\. The measured exponent does not match tested reference exponents such asν=1/2\\nu\{=\}1/2or three\-dimensional Isingν≈0\.63\\nu\{\\approx\}0\.63\(both outside CI under our four\-bin grid\)\. We report the value as an*empirical exponent*and explicitly defer universality\-class identification to future finite\-size\-scaling data\-collapse work with denser weight\-decay grids and larger per\-cell replication\.

![Refer to caption](https://arxiv.org/html/2605.20441v1/x5.png)Figure 5:\(A\) LogisticP​\(grok\)P\(\\mathrm\{grok\}\)vs weight decay acrossN=210N\{=\}210runs post Phase A\. Points are WD bins with Wilson 95% CIs, the orange curve is the logistic fit, and the dashed vertical line marksλc=0\.0158\\lambda\_\{c\}\{=\}0\.0158\(95% CI\[0\.0109,0\.0200\]\[0\.0109,0\.0200\]\)\. \(B\) Power\-law divergence oftgrokt\_\{\\text\{grok\}\}aboveλc\\lambda\_\{c\};ν=0\.757\\nu\{=\}0\.757, CI\[0\.725,0\.799\]\[0\.725,0\.799\]\. Tested reference exponents such asν=1/2\\nu\{=\}1/2and 3D Isingν≈0\.63\\nu\{\\approx\}0\.63are outside CI under the four\-bin grid; we do not identify a universality class\.
### 4\.5Permutation\-symmetry reduction: direct test on canonical checkpoints

Head indices are exchangeable at initialization \(permutation groupSHS\_\{H\}acts transitively on heads\); post\-Phase\-2, head\-pattern covariance becomes lower\-participation and permutation\-distinguishable\. We instrument this directly on the 11 saved canonical\-trajectory checkpoints \(4L8H,d=128d\{=\}128,λ=1\.0\\lambda\{=\}1\.0, 20 000 epochs, seed4242\) using the affine\-normalized participation ratio of the head\-covariance spectrum:

Rl​\(t\)≔\(∑k=1Hal​k​\(t\)\)2∑k=1Hal​k​\(t\)2,PRnorm​\(t\)≔𝔼l​\[Rl​\(t\)−1H−1\],R\_\{l\}\(t\)\\coloneqq\\frac\{\\big\(\\sum\_\{k=1\}^\{H\}a\_\{lk\}\(t\)\\big\)^\{2\}\}\{\\sum\_\{k=1\}^\{H\}a\_\{lk\}\(t\)^\{2\}\},\\qquad\\mathrm\{PR\}\_\{\\text\{norm\}\}\(t\)\\coloneqq\\mathbb\{E\}\_\{l\}\\\!\\left\[\\frac\{R\_\{l\}\(t\)\-1\}\{H\-1\}\\right\],\(3\)whereal​k​\(t\)a\_\{lk\}\(t\)are the nonnegative eigenvalues of the layer\-llhead\-pattern covariance matrix used in the direct test\.PRnorm=1\\mathrm\{PR\}\_\{\\text\{norm\}\}=1when all spectral directions participate equally, andPRnorm=0\\mathrm\{PR\}\_\{\\text\{norm\}\}=0under rank\-1 concentration\.

The canonical seed\-42 trace \(Figure[6](https://arxiv.org/html/2605.20441#S4.F6), 11 ckpts,PRnorm\\mathrm\{PR\}\_\{\\text\{norm\}\}reported to 2 s\.f\.\): initialization0\.86→0\.86\\toPhase 1 attention\-head coordination0\.710\.71at epochs 100 to 500 \(grokking onset\)→\\toPhases 2 to 4 differentiation oscillation \(epochs 1000 to 12 500\)→\\toPhase 5 collapse0\.130\.13at epoch 20 000 \(low\-participation head\-covariance structure, well below the random\-initialization baseline\)\. A55\-seed cohort \(canonical seed4242\+ cross\-seed\{7,11,31,123\}\\\{7,11,31,123\\\}\) plotted in Figure[6](https://arxiv.org/html/2605.20441#S4.F6)confirms the same qualitative trajectory across all five seeds with seed\-level scatter at the late\-Phase nadir; the IQR band uses finite values at each checkpoint \(validn=4n\{=\}4at epoch17 50017\\,500andn=5n\{=\}5otherwise\)\. This replaces the proxyMpermM\_\{\\text\{perm\}\}of the pre\-checkpoint manuscript with a directSHS\_\{H\}\-breaking diagnostic\. Appendix[C\.4](https://arxiv.org/html/2605.20441#A3.SS4)verifies the raw participation\-ratio/CV identity and the released affine normalization on 183 layer\-epoch rows from the same five seeds, with maximum absolute PR error1\.73×10−61\.73\{\\times\}10^\{\-6\}\.

![Refer to caption](https://arxiv.org/html/2605.20441v1/x6.png)Figure 6:Permutation\-symmetry test via affine\-normalizedPRnorm\\mathrm\{PR\}\_\{\\mathrm\{norm\}\}on 11 canonical\-trajectory checkpoints across a55\-seed cohort \(canonical seed4242\+ cross\-seed cohort\{7,11,31,123\}\\\{7,11,31,123\\\}, all at 4L8Hλ=1\.0\\lambda\{=\}1\.020 00020\\,000\-epoch matched configuration\)\. Blue circles \+ bold line: canonical seed4242focus trace\. Gray thin lines: cross\-seed individual traces\. Blue shaded band: cohort interquartile range over finite values at each checkpoint \(validn=4n\{=\}4at epoch17 50017\\,500andn=5n\{=\}5otherwise\)\. Gray dotted: random\-init baseline0\.860\.86\. Vermilion dashed: rank\-1 floor0\. The canonical median\-across\-layers trace falls to0\.710\.71at Phase 1 onset, oscillates between0\.180\.18and0\.550\.55across Phases 2 to 4, and collapses to0\.130\.13at Phase 5; the cross\-seed cohort confirms the qualitative trajectory across all five seeds with seed\-level scatter at the late\-Phase nadir\. Cross\-seed C1\-identity validation appears in Appendix[C\.4](https://arxiv.org/html/2605.20441#A3.SS4); Table[3](https://arxiv.org/html/2605.20441#A3.T3)reports the layer\-averaged values\.We additionally run spectral empirical\-density \(Weightwatcher\) analysis on the same 11 ckpts, tracking the heavy\-tail exponentα\\alphaof the weight\-matrix spectral density, following the Weightwatcher heavy\-tail framework\(Martin et al\.,[2021](https://arxiv.org/html/2605.20441#bib.bib23)\)as applied to anti\-grokking byPrakash & Martin \([2026a](https://arxiv.org/html/2605.20441#bib.bib36)\); the trace is shown in Figure[7](https://arxiv.org/html/2605.20441#S4.F7)\.α\\alphafalls from2\.072\.07at initialization to1\.391\.39by epoch 500 \(Phase 1 grokking onset\) and remains≲1\.5\\lesssim 1\.5through Phase 5\. Heavy\-tail structure therefore*forms during grokking onset, not during late\-stage collapse*, complicating direct identification of the “third phase” signature fromPrakash & Martin \([2026a](https://arxiv.org/html/2605.20441#bib.bib36)\)with our trajectory: the two phenomena co\-occur butα\\alphais not itself monotone with generalization collapse\. We report this as a partial parallel rather than a reproduction\.

![Refer to caption](https://arxiv.org/html/2605.20441v1/x7.png)Figure 7:ESD heavy\-tail exponentα\\alpha\(Weightwatcher, layer\-median\) across the 11 canonical\-trajectory checkpoints \(seed4242only\)\. Cross\-seed Weightwatcher at the matched 11\-checkpoint grid is deferred; a coarser 3\-seed supplementary trace provides a qualitative onset check\.α\\alphadrops from2\.072\.07at random init to1\.391\.39at Phase 1 grokking onset and remains in the heavy\-tail regime \(α<2\\alpha<2\) through Phase 5\. The third\-phase signature fromPrakash & Martin \([2026a](https://arxiv.org/html/2605.20441#bib.bib36)\)co\-occurs with grokking onset rather than developing during late\-stage collapse, complicating its use as a direct collapse\-onset detector\.
### 4\.6Causal intervention: head re\-initialization vs weight clipping

#### Pre\-specified hypotheses\.

The causal\-intervention study \(E12\) was evaluated against three planned hypotheses \(set before outcome inspection, not externally registered\): H1, head re\-initialization reduces peakσH\\sigma\_\{H\}relative to the paired control; H2, the intervention does not prevent grokking; and H3, the weight\-clipping control distinguishes head\-pattern disruption from a matched weight\-norm perturbation\. The pooled paired contrasts are confirmatory for these hypotheses\. WD\-stratified contrasts, including the sub\-criticalλ=0\.015\\lambda\{=\}0\.015cell, are exploratory scope checks\.

The order\-parameter results in Sections[4\.2](https://arxiv.org/html/2605.20441#S4.SS2)through[4\.5](https://arxiv.org/html/2605.20441#S4.SS5)are correlational: peakσH\\sigma\_\{H\},s¯\\bar\{s\}, andPRnorm\\mathrm\{PR\}\_\{\\mathrm\{norm\}\}all evolve together during Phase 2\. To probe causal structure we run a paired\-design intervention experiment \(n=60n\{=\}60runs, 3 groups×\\times10 seeds×\\times2 interventionλ\\lambdavalues\): group A is unintervened control, group B re\-initializes 2/8 attention heads at the per\-seed peak\-σH\\sigma\_\{H\}epoch \(typically iterations 1,000 to 2,000\), group C clips‖W‖2\\\|W\\\|\_\{2\}to the per\-seed median at the same epoch\. All other hyperparameters are matched\.

All three groups grok at100%100\\%rate \(10/10 in groups A and B at bothλ\\lambdavalues; 9/9 in group C atλ=0\.05\\lambda\{=\}0\.05, where one weight\-clip run was excluded because the post\-clip optimizer state produced a non\-finite gradient on the next backward pass; the exclusion was logged before downstream analysis and the decision was applied identically across cells\)\. The intervention does not prevent generalization\. Phase 2 amplitude does respond causally: paired across seeds andλ\\lambda\(n=20n\{=\}20\), the peakσH\\sigma\_\{H\}differenceB−AB\-Ahas mean−0\.038±0\.043\-0\.038\\pm 0\.043\(pairedtt,pt=9\.3×10−4p\_\{t\}\{=\}9\.3\{\\times\}10^\{\-4\}, Wilcoxonp=2\.5×10−3p\{=\}2\.5\{\\times\}10^\{\-3\}, Cohen’sd=−0\.876d\{=\}\-0\.876\)\.111Pairedtttests assume approximately normal within\-pair differences and can be affected by outliers; Wilcoxon signed\-rank tests are the non\-parametric paired check and give the same headline decision here\. Cohen’sddis computed on paired differences, with0\.2/0\.5/0\.80\.2/0\.5/0\.8as small/medium/large reference magnitudes\. Residual\-bootstrap CIs for theν\\nusensitivity fit use 5 000 resamples; the jackknife reports a leave\-one\-out bias\-correctedν\\nu\.The peak per\-head dimension differentialr¯diff\\bar\{r\}\_\{\\mathrm\{diff\}\}shows a smaller paired effect \(Δ=−0\.014\\Delta\{=\}\-0\.014,pt=0\.020p\_\{t\}\{=\}0\.020,d=−0\.569d\{=\}\-0\.569\), final test accuracy andmin⁡\(s¯\)\\min\(\\bar\{s\}\)are unchanged, and grokking epoch shifts by377±1 282377\{\\pm\}1\\,282iterations \(pt=0\.20p\_\{t\}\{=\}0\.20, statistically null at thisnn\)\.

Stratified byλ\\lambda\(n=10n\{=\}10per cell\), the head\-re\-initialization effect on peakσH\\sigma\_\{H\}is concentrated at the canonical post\-transitionλ=0\.05\\lambda\{=\}0\.05\(Δ=−0\.055±0\.046\\Delta\{=\}\-0\.055\\pm 0\.046,pt=4\.5×10−3p\_\{t\}\{=\}4\.5\{\\times\}10^\{\-3\},d=−1\.190d\{=\}\-1\.190\) and is non\-significant at sub\-criticalλ=0\.015\\lambda\{=\}0\.015\(Δ=−0\.021±0\.034\\Delta\{=\}\-0\.021\\pm 0\.034,pt=0\.086p\_\{t\}\{=\}0\.086,d=−0\.609d\{=\}\-0\.609\)\. Group C \(weight clipping\) yields no significant peakσH\\sigma\_\{H\}change relative to group A at eitherλ\\lambda\(pairedd=−0\.406d\{=\}\-0\.406,pt=0\.094p\_\{t\}\{=\}0\.094pooled\), localising the effect to head\-pattern structure rather than weight norm\. The C\-vs\-B contrast at canonicalλ=0\.05\\lambda\{=\}0\.05is significant \(Δpeak​σH=\+0\.048±0\.030\\Delta\_\{\\mathrm\{peak\}~\\sigma\_\{H\}\}\{=\}\+0\.048\\pm 0\.030,pt=1\.4×10−3p\_\{t\}\{=\}1\.4\{\\times\}10^\{\-3\},d=\+1\.586d\{=\}\+1\.586\): replacing head structure suppresses Phase 2 amplitude in a way that scaling weight norm does not reproduce\. Across the 15\-test paired\-outcome family \(three pooled pairwise contrastsB−AB\{\-\}A,C−AC\{\-\}A,C−BC\{\-\}Bacross five outcomes: peakσH\\sigma\_\{H\}, minimums¯\\bar\{s\}, peakr¯diff\\bar\{r\}\_\{\\mathrm\{diff\}\}, grokking\-onset epoch, and final test accuracy\), the strict Bonferroni cutoff isα/15=0\.0033\\alpha/15=0\.0033; the pooledB−AB\-Apeak\-σH\\sigma\_\{H\}result remains below this cutoff, while the planned WD\-stratified follow\-up retains the canonicalλ=0\.05\\lambda\{=\}0\.05contrast under Holm\-Bonferroni and the sub\-criticalλ=0\.015\\lambda\{=\}0\.015contrast does not survive correction\.

![Refer to caption](https://arxiv.org/html/2605.20441v1/x8.png)Figure 8:Causal\-intervention paired forest plot for peakσH\\sigma\_\{H\}, pooled acrossλ∈\{0\.015,0\.05\}\\lambda\\in\\\{0\.015,0\.05\\\}\. Points show mean paired change and bars show 95% CIs:B−A=−0\.038B\-A=\-0\.038\(\[−0\.057,−0\.020\]\[\-0\.057,\-0\.020\],n=20n\{=\}20\),C−A=−0\.017C\-A=\-0\.017\(\[−0\.036,0\.0005\]\[\-0\.036,0\.0005\],n=19n\{=\}19\), andC−B=\+0\.023C\-B=\+0\.023\(\[−0\.004,0\.044\]\[\-0\.004,0\.044\],n=19n\{=\}19\)\. WD\-stratified breakdown in §[4\.6](https://arxiv.org/html/2605.20441#S4.SS6)body: at canonical post\-transitionλ=0\.05\\lambda\{=\}0\.05,B−A=−0\.055B\-A=\-0\.055\(pt=4\.5×10−3p\_\{t\}=4\.5\{\\times\}10^\{\-3\},d=−1\.190d=\-1\.190\) andC−B=\+0\.048C\-B=\+0\.048\(pt=1\.4×10−3p\_\{t\}=1\.4\{\\times\}10^\{\-3\},d=\+1\.586d=\+1\.586\) are both highly significant\.We read this as causal evidence \(forest plot in Figure[8](https://arxiv.org/html/2605.20441#S4.F8)\) that Phase 2 amplitude is sensitive to head\-pattern structure rather than weight magnitude, at canonical post\-transitionλ\\lambdain the studied 4L8Hd=128d\{=\}128transformer onmod\+\\mathrm\{mod\}\_\{\+\}\(p=97p\{=\}97\)\. We do not claim the head\-perturbation effect generalizes across architectures, tasks, or WD values outside\{0\.015,0\.05\}\\\{0\.015,0\.05\\\}, nor that intervention rescues any retention metric\.

### 4\.7Controls and predictions

A random\-label null control \(n=15n\{=\}15extended random\-label control runs, superseding the smaller E8 pilot\) yields peakσH=0\.087±0\.025\\sigma\_\{H\}\{=\}0\.087\\pm 0\.025, the same order of magnitude as thed/H=2d/H\{=\}2low\-amplitude regime; the add\-vs\-random amplitude contrast is significant but small \(permutation\-testp=0\.009p\{=\}0\.009, Cohen’sd=1\.11d\{=\}1\.11,na=12n\_\{a\}\{=\}12vsnb=15n\_\{b\}\{=\}15\), so the null control is a meaningful lower bound on amplitude rather than statistically equivalent tod/H=2d/H\{=\}2\.

#### Multi\-task replication atn=280n\{=\}280\.

The earliern=28n\{=\}28task\-control cohort \(E3\) is now superseded by a horizon\-matched multi\-task sweep \(E9\) \(Figure[9](https://arxiv.org/html/2605.20441#S4.F9)\) across all four operationsmod\+\\mathrm\{mod\}\_\{\+\},mod−\\mathrm\{mod\}\_\{\-\},mod×\\mathrm\{mod\}\_\{\\times\},mod÷\\mathrm\{mod\}\_\{\\div\}at two scales \(4L8Hd=128d\{=\}128and 6L8Hd=512d\{=\}512\) and seven near\-criticalλ\\lambdavalues withn=5n\{=\}5seeds each, yieldingn=280n\{=\}280runs at 10 K epochs\. Per\-WD pooled grok rate \(across 4 ops×\\times2 scales\) increases monotonically from7/407/40\(0\.1750\.175\) atλ=0\.003\\lambda\{=\}0\.003to40/4040/40\(1\.001\.00\) atλ=0\.07\\lambda\{=\}0\.07, reproducing the canonical WD\-control structure\. Per\-task grok rate \(pooled WDs and scales\) ismod\+​0\.73\\mathrm\{mod\}\_\{\+\}\\,0\.73,mod÷​0\.71\\mathrm\{mod\}\_\{\\div\}\\,0\.71,mod×​0\.70\\mathrm\{mod\}\_\{\\times\}\\,0\.70,mod−​0\.49\\mathrm\{mod\}\_\{\-\}\\,0\.49, recovering themod−\\mathrm\{mod\}\_\{\-\}slow\-grok ordering of the original E3 cohort at much largernn\. Pooled\-4\-ops logistic refit yieldsλc\\lambda\_\{c\}small0\.00770\.0077\(95% CI\[0\.0056,0\.0106\]\[0\.0056,0\.0106\],n=102/140n\{=\}102/140grok\) andλc\\lambda\_\{c\}medium0\.01280\.0128\(\[0\.0076,0\.0194\]\[0\.0076,0\.0194\],n=82/140n\{=\}82/140\)\. The small and medium CIs overlap, consistent with our horizon\-matched verdict that no monotoneλc​\(N\)\\lambda\_\{c\}\(N\)law is supported within the small/medium pair\. The E9 pooled\-4\-ops small CI does not overlap with the WD\-axis canonical CIλc=0\.0158\\lambda\_\{c\}\{=\}0\.0158\(\[0\.0109,0\.0200\]\[0\.0109,0\.0200\]\); this is a protocol difference, not a contradiction\. The canonical headline is fit on themod\+\\mathrm\{mod\}\_\{\+\}dense WD\-sweep cohort with 13 WD bins at the canonical training horizon; E9 is the horizon\-matched pooled four\-operation cohort with 7 WD bins at10,00010\{,\}000epochs\. The two estimates agree on the qualitative monotone WD\-control structure and on the order of magnitude ofλc\\lambda\_\{c\}; they differ in numerical point estimate because they pool different tasks and use different WD\-bin densities\. We treatλc=0\.0158\\lambda\_\{c\}\{=\}0\.0158as the canonical mod\-add reference and the E9 pooled values as task\-pool consistency probes, not as competing point estimates of a single underlying scalar\. Per\-task medium\-scale spread is suggestive but individually under\-powered atn=5n\{=\}5per cell, with mutually overlapping 95% CIs \(mod÷​0\.003\\mathrm\{mod\}\_\{\\div\}\\,0\.003,mod\+​0\.012\\mathrm\{mod\}\_\{\+\}\\,0\.012,mod×​0\.019\\mathrm\{mod\}\_\{\\times\}\\,0\.019,mod−​0\.025\\mathrm\{mod\}\_\{\-\}\\,0\.025\); the pooled monotone WD\-control pattern rather than the task\-ordering carries the inference\. The eight per\-task logistic fits \(4 tasks×\\times2 scales\) are treated as a single multiple\-comparison family; the active inference is the pooled monotone WD\-control pattern rather than an individually significant task\-ordering test\.

![Refer to caption](https://arxiv.org/html/2605.20441v1/x9.png)Figure 9:Multi\-task grok\-rate heatmap \(4 modular operations×\\times2 model scales×\\times7 weight decays,n=5n\{=\}5seeds per cell,N=280N\{=\}280total\)\. Each cell showsgroked/n\\mathrm\{groked\}/n; color encodes the same fraction on a cividis sequential scale\. Rows are mostly monotone left\-to\-right: weight decay is the dominant axis\. Pooled task orderingmod\+​0\.73\\mathrm\{mod\}\_\{\+\}\\,0\.73,mod÷​0\.71\\mathrm\{mod\}\_\{\\div\}\\,0\.71,mod×​0\.70\\mathrm\{mod\}\_\{\\times\}\\,0\.70,mod−​0\.49\\mathrm\{mod\}\_\{\-\}\\,0\.49reproduces the sub\-hardness ranking of the originaln=28n\{=\}28task\-control cohort\.

### 4\.8Order parameters as correlational discriminators of long\-horizon retention

We tested whether the cheap online order parameters defined in §[3\.2](https://arxiv.org/html/2605.20441#S3.SS2)predict long\-horizon outcome \(stable retention vs\. never\-grok vs\. collapse\) using a 50\-run held\-out cohort \(held\-out retention test\) at WD values not seen in the training set \(λ∈\{0\.025,0\.04\}\\lambda\\in\\\{0\.025,0\.04\\\}\) with new seeds\. Train cohort:998998runs from the Phase\-A pool; features per run at epoch 1 K:σH\\sigma\_\{H\},s¯\\bar\{s\}, weight norm,entstd\\mathrm\{ent\}\_\{\\mathrm\{std\}\}, peakr¯diff\\bar\{r\}\_\{\\mathrm\{diff\}\},d/Hd/H, scale, WD, test accuracy\. Two classifiers \(logistic regression and random forest\) were grouped\-CV\-trained at the \(scale, WD\) level\.

Random forest holdout area under the ROC curve \(AUC\) is0\.7990\.799with Brier score0\.0980\.098on the canonical scikit\-learn 1\.5\.x training environment; across recent scikit\-learn versions \(1\.4\.x to 1\.6\.x\) on identical inputs the holdout AUC ranges from0\.790\.79to0\.810\.81and Brier from0\.0980\.098to0\.1020\.102due to default\-parameter changes inRandomForestClassifier, with both AUC extrema below the pre\-specified0\.850\.85predictor gate\. Logistic regression holdout AUC is0\.6780\.678\. Train\-time AUC was0\.8340\.834, so holdout generalization gap is−0\.035\-0\.035\(roughly4%4\\%relative drop\)\. The verdict from our pre\-specified acceptance gate \(AUC≥0\.85\\geq 0\.85for predictor status, set before outcome inspection and not externally registered\) iscorrelational only: the order parameters at epoch 1 K carry retention signal but do not reach the predictor threshold on out\-of\-distribution WDs\. Top\-five RF feature importances are WD \(0\.2470\.247\), weight norm \(0\.2200\.220\),s¯\\bar\{s\}\(0\.1450\.145\),entstd\\mathrm\{ent\}\_\{\\mathrm\{std\}\}\(0\.1300\.130\), test accuracy \(0\.1230\.123\); the order\-parameter features \(s¯\\bar\{s\},entstd\\mathrm\{ent\}\_\{\\mathrm\{std\}\}\) together account for0\.2750\.275of feature importance, comparable to weight\-norm magnitude alone\.

### 4\.9Cross\-architecture scope probes: 4L MLP, 4L LSTM, and 4L Mamba grids

To bound architecture\-specificity we ran a horizon\-matched cross\-architecture sweep \(E10\) \(Figure[10](https://arxiv.org/html/2605.20441#S4.F10)\) on a 4\-layer MLP \(hidden dimension512512, no attention\) formod\+\\mathrm\{mod\}\_\{\+\},p=97p\{=\}97, with 7 WD values×\\times10 seeds=70=70runs at 10 K epochs\. The MLP groks atλ≥0\.05\\lambda\{\\geq\}0\.05in3/103/10seeds,λ=0\.07\\lambda\{=\}0\.07in10/1010/10, andλ<0\.05\\lambda\{<\}0\.05in0/100/10\. Logistic refit givesλc=0\.0511\\lambda\_\{c\}\{=\}0\.0511\(95% CI\[0\.0495,0\.0591\]\[0\.0495,0\.0591\],n=13/70n\{=\}13/70grok\), shifted upward by roughly 3 to 7×\\timesrelative to the transformer pooled values0\.00770\.0077small and0\.01280\.0128medium\. Grokking is therefore not attention\-specific in our setting, but the transition\-scaleλ\\lambdadepends strongly on architecture; the valueλc=0\.0158\\lambda\_\{c\}\{=\}0\.0158in the rest of this paper applies to the 4L8Hd=128d\{=\}128transformer cohort and should not be transferred to other architectures without architecture\-specific recalibration\. Our attention\-specific order parameterss¯\\bar\{s\}andσH\\sigma\_\{H\}are not directly defined for an MLP without head structure; cross\-architecture diagnostics are follow\-up work\.

A second cross\-architecture probe \(E14\) on a 4\-layer LSTM \(hidden dimensionh=512h\{=\}512, no attention, no per\-head structure,7\.687\.68M parameters; comparable scale to the transformer\-medium cohort and roughly9×9\\timeslarger than transformer\-small\) formod\+\\mathrm\{mod\}\_\{\+\},p=97p\{=\}97, with the same 7 WD values×\\times10 seeds=70=70runs at 10 K epochs yields22/7022/70grok\. Logistic fit givesλc=0\.0365\\lambda\_\{c\}\{=\}0\.0365\(95% bootstrap CI\[0\.0299,0\.0473\]\[0\.0299,0\.0473\],10001000resamples\)\. Per\-WD grok rates with Wilson 95% CIs:0/100/10forλ≤0\.015\\lambda\{\\leq\}0\.015\(Wilson\[0\.0,0\.278\]\[0\.0,0\.278\]\),1/101/10atλ=0\.02\\lambda\{=\}0\.02\(\[0\.018,0\.404\]\[0\.018,0\.404\]\),2/102/10atλ=0\.03\\lambda\{=\}0\.03\(\[0\.057,0\.510\]\[0\.057,0\.510\]\),9/109/10atλ=0\.05\\lambda\{=\}0\.05\(\[0\.596,0\.982\]\[0\.596,0\.982\]\),10/1010/10atλ=0\.07\\lambda\{=\}0\.07\(\[0\.722,1\.000\]\[0\.722,1\.000\]\)\. The LSTMλc\\lambda\_\{c\}sits between the transformer pooled values \(0\.00770\.0077small,0\.01280\.0128medium\) and the MLP probe \(0\.05110\.0511\); we report this monotone ordering as an empirical observation and do not claim a mechanism for it\. The LSTM probe is not strictly parameter\-matched to the canonical transformer; per\-architecture canonicalλ\\lambdashould be recalibrated rather than transferred\.

A third cross\-architecture probe \(E15\) on a 4\-layer Mamba selective\-state\-space network\(Gu & Dao,[2023](https://arxiv.org/html/2605.20441#bib.bib14)\)\(canonical configurationdmodel=128d\_\{\\mathrm\{model\}\}\{=\}128, expand factor44,dstate=16d\_\{\\mathrm\{state\}\}\{=\}16,dconv=4d\_\{\\mathrm\{conv\}\}\{=\}4;0\.960\.96M parameters, comparable scale to the transformer\-small cohort\) formod\+\\mathrm\{mod\}\_\{\+\},p=97p\{=\}97, with the same 7 WD values×\\times10 seeds=70=70runs at 10 K epochs yields46/7046/70grok\. Logistic fit on Laplace\-smoothed per\-bin grok rates givesλc=0\.0144\\lambda\_\{c\}\{=\}0\.0144\(95% bootstrap CI\[0\.0106,0\.0159\]\[0\.0106,0\.0159\],15001500resamples\)\. Per\-WD grok rates with Wilson 95% CIs:0/100/10forλ≤0\.006\\lambda\{\\leq\}0\.006\(Wilson\[0\.0,0\.278\]\[0\.0,0\.278\]\),6/106/10atλ=0\.015\\lambda\{=\}0\.015\(\[0\.313,0\.832\]\[0\.313,0\.832\]\),10/1010/10atλ≥0\.02\\lambda\{\\geq\}0\.02\(Wilson lower bound\[0\.722,1\.000\]\[0\.722,1\.000\]atn=10n\{=\}10\)\. The transition is sharper than in the LSTM and MLP probes \(fit steepness parameterk=20\.6k\{=\}20\.6versus LSTMk=6\.7k\{=\}6\.7and MLPkkunreported in E10\), driving the tightλc\\lambda\_\{c\}CI\. The Mambaλc\\lambda\_\{c\}point estimate sits near the transformer\-medium fit and overlaps its CI; we report this proximity as an empirical observation and do not claim that state\-space and attention architectures share a transition mechanism\. Two scope\-probe\-within\-scope\-probe sweeps further measured Mamba sensitivity to expand factor and task: a smaller expand\-22variant onmod\+\\mathrm\{mod\}\_\{\+\}\(0\.490\.49M parameters;42/7042/70grok;λc=0\.0163\\lambda\_\{c\}\{=\}0\.0163, CI\[0\.0138,0\.0182\]\[0\.0138,0\.0182\]\) and an expand\-44probe onmod×\\mathrm\{mod\}\_\{\\times\}\(0\.960\.96M parameters;37/7037/70grok;λc=0\.0191\\lambda\_\{c\}\{=\}0\.0191, CI\[0\.0169,0\.0219\]\[0\.0169,0\.0219\]\)\. Per\-architecture canonicalλ\\lambdashould be recalibrated rather than transferred\.

![Refer to caption](https://arxiv.org/html/2605.20441v1/x10.png)Figure 10:Cross\-architecture transition comparison from logistic grok\-rate fits\. Bars showλc\\lambda\_\{c\}and 95% bootstrap CIs for transformer small \(0\.00770\.0077, CI\[0\.0056,0\.0106\]\[0\.0056,0\.0106\],102/140102/140grok\), transformer medium \(0\.01280\.0128, CI\[0\.0076,0\.0194\]\[0\.0076,0\.0194\],82/14082/140\), 4L MLPh=512h\{=\}512\(0\.05110\.0511, CI\[0\.0495,0\.0591\]\[0\.0495,0\.0591\],13/7013/70\), 4L LSTMh=512h\{=\}512\(0\.03650\.0365, CI\[0\.0299,0\.0473\]\[0\.0299,0\.0473\],22/7022/70\), and 4L Mambad=128d\{=\}128\(0\.01440\.0144, CI\[0\.0106,0\.0159\]\[0\.0106,0\.0159\],46/7046/70\)\. Theλc\\lambda\_\{c\}values span roughly an order of magnitude across the five architecture configurations; reported as empirical observation, no mechanism claimed\.
### 4\.10Reproducibility and artifact trail

All headline numerical claims trace through a machine\-readable provenance manifest that maps each figure or table cell to an aggregate JSON and then to per\-run records \(the canonical transformer cohort comprises 1,120 paper\-accepted runs traced through 1,442 raw transformer\-cohort JSONs; the additional 322 raw JSONs are pilot or filtered records retained for cross\-check, with 861 legacy aggregate JSONs preserved alongside\)\. The three cross\-architecture scope probes contribute 350 additional raw per\-run JSONs \(70 MLP for E10, 70 LSTM for E14, 210 Mamba for E15\) traced via the same Appendix[D](https://arxiv.org/html/2605.20441#A4)map and the per\-grid rich aggregates undereval/\(separate logistic\-fit JSONs for the LSTM cross\-architecture probe, the canonical 4L Mamba probe onmod\+\\mathrm\{mod\}\_\{\+\}, the Mamba expand\-22variant, and the Mambamod×\\mathrm\{mod\}\_\{\\times\}probe\)\. Figures and aggregate JSONs are generated by the paper build pipeline rather than hand\-edited\. The public artifact surface is deliberately smaller than the internal training pipeline: it ships the rendered figures, aggregate JSONs, selected aggregation/figure scripts, a coverage manifest, the Lean target, and lightweight numerical\-verification scripts; raw per\-run JSONs live in the companion dataset\. Full retraining and full end\-to\-end regeneration of every figure from raw runs are not claimed as one\-command public artifacts\. The full claim→\\tocomponent→\\todata map appears in Appendix[D](https://arxiv.org/html/2605.20441#A4)\.

#### Formal verification\.

We formalise A1 \(three regime parts\), B1, C1, and E1 in Lean 4 against mathlib v4\.29\.0; all four proofs compile with nosorry\. C1 \(the raw participation\-ratio/CV identity and the affine\-normalizedPRnorm\\mathrm\{PR\}\_\{\\mathrm\{norm\}\}form used in the JSON artifacts\) is reduced to an elementary variance decomposition∑ixi2=∑i\(xi−x¯\)2\+n​x¯2\\sum\_\{i\}x\_\{i\}^\{2\}=\\sum\_\{i\}\(x\_\{i\}\-\\bar\{x\}\)^\{2\}\+n\\bar\{x\}^\{2\}, supplied as a standalone lemma in the same file\. These proofs establish*well\-formedness*of the diagnostic identities \(bounds, identities, rank properties\); they do not formalise the empirical experimental claims, which remain JSON\-traced via the provenance map in Appendix[D](https://arxiv.org/html/2605.20441#A4)\.

#### Release\.

Code, aggregate JSONs, and the Lean formalisation are released alongside the manuscript under Apache\-2\.0 \(code\) and CC\-BY\-4\.0 \(data\) at[https://github\.com/lucky\-verma/grokking\-diagnostics](https://github.com/lucky-verma/grokking-diagnostics); the 1,792 per\-run JSONs \(1,442 transformer\-cohort containing 1,120 paper\-accepted runs; 350 cross\-architecture scope\-probe, all paper\-accepted\) and the per\-grid aggregate fits are deposited at[https://huggingface\.co/datasets/lucky\-verma/grokking\-diagnostics\-runs](https://huggingface.co/datasets/lucky-verma/grokking-diagnostics-runs)\. A citable Zenodo archive of the v1 source tarball is minted post\-arXiv\-identifier assignment\.

## 5Discussion

### 5\.1Limitations

The empirical scope is modular arithmetic in transformer attention models up to 85 M parameters\. We do not claim a formal thermodynamic derivation, membership in a canonical universality class, or generality beyond the tested modular operations, model sizes, and attention architectures\. Language\-model, larger\-scale, and non\-attention studies are follow\-up work rather than prerequisites for the claims made here\.

We use control\-parameter language in a specific, limited sense: weight decayλ\\lambdais a single scalar whose variation sweeps the system across qualitatively distinct empirical regimes separated by sharp boundaries\. We do*not*derive a Hamiltonian, a free\-energy functional, or an equilibrium partition function for transformer training, and we do not claim that SGD trajectories constitute a thermal ensemble\. Our contribution is the empirical identification of the 2D\(λ,N\)\(\\lambda,N\)regime diagram, the three regimes, the transition estimateλc=0\.0158\\lambda\_\{c\}=0\.0158withν=0\.757\\nu=0\.757power\-law fit to time\-to\-grok, and the activation\-level order parameters that make the transition observable online\. The measured exponent does not match the tested reference exponents, so universality\-class identification is deferred to finite\-size\-scaling data\-collapse work with larger per\-cell replication and theoretical backing\. Rigorous statistical\-mechanics treatments of grokking with finite\-size scaling and Binder\-cumulant crossings are concurrent work byBi et al\. \([2026](https://arxiv.org/html/2605.20441#bib.bib4)\), which we position as complementary: we supply phenomenology and cheap diagnostics, they supply the falsifiability test\.

The three horizon\-matched cross\-architecture probes in §[4\.9](https://arxiv.org/html/2605.20441#S4.SS9)\(4L MLPn=70n\{=\}70,λc=0\.0511\\lambda\_\{c\}\{=\}0\.0511\[0\.0495,0\.0591\]\[0\.0495,0\.0591\]; 4L LSTMh=512h\{=\}512n=70n\{=\}70,λc=0\.0365\\lambda\_\{c\}\{=\}0\.0365\[0\.0299,0\.0473\]\[0\.0299,0\.0473\]; 4L Mambad=128d\{=\}128n=70n\{=\}70,λc=0\.0144\\lambda\_\{c\}\{=\}0\.0144\[0\.0106,0\.0159\]\[0\.0106,0\.0159\]\) confirm that the WD\-controlled grokking transition on this task is not attention\-specific, withλc\\lambda\_\{c\}values spanning roughly an order of magnitude across the five architecture configurations and the Mamba probe sitting near the transformer\-medium fit\. The LSTM probe is at a different parameter scale \(7\.68M vs 0\.82M for the transformer\-small cohort\) and the Mamba probe \(0\.96M\) is roughly parameter\-matched to transformer\-small but uses selective scan rather than attention; so the cross\-architecture comparison is qualitative rather than strictly parameter\-matched across all five points\. The present paper’ss¯\\bar\{s\}andσH\\sigma\_\{H\}are attention\-head order parameters and are not directly defined for non\-attention architectures; cross\-architecture order parameters for MLP, recurrent, and state\-space models, and parameter\-matched probes at each architecture, are follow\-up work\.

The AdamW\-relaxation calibration in App[C\.5](https://arxiv.org/html/2605.20441#A3.SS5)exhibits cross\-seed bimodalκ\\kappa: 2 of 5 cohorts \(seeds 31, 123\) sit outside the empiricalλc\\lambda\_\{c\}95% CI under the early\-window fit, with slower early\-relaxation rates the one\-parameter relaxation argument does not yet explain\. Full SDE\-based derivation ofκ\\kappafrom the AdamW second\-moment dynamics is the natural follow\-up\.

#### Figure\-specific seed coverage\.

Figures 1, 2, 4, 5, 8, 9, 10 are derived from multi\-seed cohorts \(n≥5n\{\\geq\}5per WD bin where applicable\)\. Figures 3 and 6 plot a55\-seed cohort \(canonical seed4242\+ cross\-seed\{7,11,31,123\}\\\{7,11,31,123\\\}at matched 4L8Hλ=1\.0\\lambda\{=\}1\.020 00020\\,000\-epoch configuration\) with the canonical seed as the focus trace and cross\-seed traces as underlay\. Figure 7 \(ESD heavy\-tailα\\alpha\) currently displays only the canonical seed\-42 trajectory because Weightwatcher analysis on cross\-seed checkpoints at the matched 11\-checkpointλ=1\.0\\lambda\{=\}1\.0epoch grid is not complete in this version; a coarser\-epoch 3\-seed cohort logged in supplementaryprakash\_out/corroborates the qualitative onset signature but at a different epoch grid\. Full cross\-seed Weightwatcher computation at the canonical\-trajectory grid is deferred to follow\-up work\.

### 5\.2Numerical comparison with concurrent grokking\-theory work

Table[1](https://arxiv.org/html/2605.20441#S5.T1)positions the present work against the recent \(2024 to 2026\) grokking\-theory wave on four dimensions: framework abstraction, primary numerical prediction, optimizer support, and validation regime\. The present paper’s empirical scope \(1 120 runs, 5 cross\-seed trajectories at canonical 4L8H, formally verified diagnostic well\-formedness properties\) is complementary to the analytic Lyapunov contraction ofTruong Xuan Khanh et al\. \([2026a](https://arxiv.org/html/2605.20441#bib.bib47)\), the finite\-size\-scaling framework ofBi et al\. \([2026](https://arxiv.org/html/2605.20441#bib.bib4)\), the dimensionality scaling ofWang \([2026a](https://arxiv.org/html/2605.20441#bib.bib52)\), and the provable two\-layer scaling ofTian \([2025](https://arxiv.org/html/2605.20441#bib.bib46)\)\. No single framework dominates across all four dimensions\. The numerical agreement within an order of magnitude across ourλc=0\.0158\\lambda\_\{c\}=0\.0158logistic fit, the calibrated AdamW amplificationκ=18\.6\\kappa=18\.6, and the Khanhγeff≥η​λ\\gamma\_\{\\text\{eff\}\}\\geq\\eta\\lambdainequality is a calibrated consistency check with documented seed\-level failure modes \(§[C\.5](https://arxiv.org/html/2605.20441#A3.SS5)\), not an independent prediction ofλc\\lambda\_\{c\}from optimizer mechanics\.

Table 1:Comparison with concurrent grokking\-theory work\. This paper supplies an empirical calibration of the Khanh et al\. contraction relation \(γeff≥η​λ\\gamma\_\{\\text\{eff\}\}\{\\geq\}\\eta\\lambda, fitκ=18\.6\\kappa\{=\}18\.6on the canonical 4L8H mod\-add seed\-42 trajectory; cross\-seedκ\\kappais bimodal with 3 of 5 cohorts inside the empiricalλc\\lambda\_\{c\}CI, see §[C\.5](https://arxiv.org/html/2605.20441#A3.SS5)\) plus machine\-verified diagnostic well\-formedness properties under mathlib v4\.29\.0\. No work identifies a universality class forν≈0\.76\\nu\{\\approx\}0\.76; tested reference exponents \(1/21/2, 3D Ising0\.630\.63\) lie outside the empirical CI under our four\-bin grid\.#### Concurrent work and open questions raised inPower et al\. \([2022](https://arxiv.org/html/2605.20441#bib.bib35)\)\.

The original grokking paper\(Power et al\.,[2022](https://arxiv.org/html/2605.20441#bib.bib35)\)§4 left open whether minimum\-flatness or implicit\-bias measures correlate with the memorization\-to\-generalization transition\. We supply two cheap online diagnostics \(s¯,σH\\bar\{s\},\\sigma\_\{H\}; see §[3\.2](https://arxiv.org/html/2605.20441#S3.SS2)for evaluation cost\) that complement SGLD\-estimated rLLC \(r=0\.46r\{=\}0\.46correlation, §[2](https://arxiv.org/html/2605.20441#S2)\), characterise the transition atλc=0\.0158\\lambda\_\{c\}=0\.0158with empirical exponentν=0\.757\\nu=0\.757, and provide a one\-parameter relaxation bound onλc\\lambda\_\{c\}\(§[C\.5](https://arxiv.org/html/2605.20441#A3.SS5)\) that empirically calibrates theγeff\\gamma\_\{\\text\{eff\}\}amplification factor introduced analytically byTruong Xuan Khanh et al\. \([2026a](https://arxiv.org/html/2605.20441#bib.bib47)\)\. Where Khanh et al\. predict the grokking*delay*via Lyapunov contraction, we predict the critical*threshold*via the dual horizon constraint, with cross\-seed bimodalκ\\kappadocumenting a remaining gap that motivates the SDE\-based refinement they leave open\. This contribution is complementary to the finite\-size\-scaling\(Bi et al\.,[2026](https://arxiv.org/html/2605.20441#bib.bib4)\), dimensionality\(Wang,[2026a](https://arxiv.org/html/2605.20441#bib.bib52)\), and glass\-relaxation\(Zhang et al\.,[2025](https://arxiv.org/html/2605.20441#bib.bib66)\)frameworks: each operates at a different abstraction \(Binder\-cumulant finite\-size scaling, effective dimensionality, glass physics analogy, AdamW update mechanics\) and the same empirical phenomenon supports them all to within an order of magnitude\.

#### Mathematical properties of the diagnostics and a one\-parameter bound onλc\\lambda\_\{c\}\.

Appendix[C](https://arxiv.org/html/2605.20441#A3)records five elementary results that confirm the diagnostics are mathematically well defined: a regularized competition model can produce memorization, developmental, and collapse regimes \(A1\); dominant weight decay selects minimum\-complexity predictors when loss differences are bounded \(B1\); the participation\-ratio diagnostic is an affine transform of an exact coefficient\-of\-variation statistic over the head\-covariance spectrum \(C1\); online similarity estimates concentrate with batch size \(D1\); and per\-head dimension bounds attention\-score rank \(E1\)\. Section[C\.5](https://arxiv.org/html/2605.20441#A3.SS5)narrows the predictive gap with an AdamW\-relaxation argument that combines A1 with one empirically\-fit constant \(the AdamW amplificationκ=18\.6\\kappa=18\.6from the canonical seed\-42 trajectory\), boundingλcbound=0\.0124\\lambda\_\{c\}^\{\\text\{bound\}\}=0\.0124, inside the empirical 95% CI\[0\.0109,0\.0200\]\[0\.0109,0\.0200\]for the canonical\-trajectoryκ\\kappa\. We label this a calibrated consistency check rather than a first\-principles derivation:κ\\kappais fit per\-seed, the cross\-seed cohort exhibits bimodalκ\\kappawith 3 of 5 seeds inside the empirical CI under the early\-window fit, and a full SDE\-based derivation ofκ\\kappafrom optimizer mechanics is deferred\. These results explain why the diagnostics are mathematically well defined and provide an order\-of\-magnitude calibration of the M→\\toG transition; they do not establish transformer\-grokking universality or identify a universality class\.

#### Developmental analogy is not evidence\.

Developmental biology motivated the vocabulary of staged coordination, but no biological analogy supports the results in this paper\. In particular, the Phase\-A replication in §[4\.4](https://arxiv.org/html/2605.20441#S4.SS4)places our fitted exponent far from theν=1/2\\nu\{=\}1/2scaling associated with the SNIC heartbeat model ofJia et al\. \([2023](https://arxiv.org/html/2605.20441#bib.bib17)\); the mathematical bridge does not hold\.

#### What Phase 2 differentiation does, mechanistically\.

Our cosine\-similarity and entropy\-standard\-deviation diagnostics measure head redundancy, not the specific function each head computes\. To probe Phase\-2 structure we computed, on each pilot checkpoint \(24 ckpts×\\times3 seeds, 32 heads per model\), the number of distinct head*roles*as hierarchical single\-linkage clusters of per\-head output vectors at cosine\-distance threshold 0\.5\. The cluster counts are noisy but support the same coarse picture: at the first checkpoint the three seeds have1111,1515, and88clusters; seeds 42 and 123 then compress around epochs100100to200200\(seed 42 reaches44clusters at epoch160160; seed 123 stays in the99to1212range\), followed by broader post\-transition counts reaching2424and2020clusters, respectively, by epochs20002000to49994999\. Seed 7, which also exhibits the anti\-grokking late\-collapse anomaly discussed in §[4](https://arxiv.org/html/2605.20441#S4), is noisier and does not show a clean U\-shape; we therefore treat this cluster analysis as mechanistic support for the synchronization/differentiation interpretation rather than as an additional cross\-setting phase criterion\. Mechanistic identification of*which*algorithm each cluster computes \(for instance Fourier\-basis multiplication as inNanda et al\. \([2023](https://arxiv.org/html/2605.20441#bib.bib29)\), or monosemantic feature circuits as inElhage et al\. \([2022](https://arxiv.org/html/2605.20441#bib.bib9)\)\) is left to follow\-up work; our diagnostics flag*when*differentiation occurs and its magnitude, not*what*the differentiated heads individually do\.

## 6Conclusion

Training in these modular\-arithmetic transformers exhibits staged dynamics \(synchronization followed by differentiation\) that are not explicitly designed but arise from the interaction of optimization, regularization, and architecture\. Whether staged transition cascades generalize beyond this setting is open; the present results establish their presence in small\-transformer grokking on modular arithmetic, not their universality across learning systems\.

## References

- Acharya & Dhakal \(2026\)Pratyush Acharya and Habish Dhakal\.Grokking as a variance\-limited phase transition: Spectral gating and the epsilon\-stability threshold\.*arXiv preprint arXiv:2603\.15492*, 2026\.URL[https://arxiv\.org/abs/2603\.15492](https://arxiv.org/abs/2603.15492)\.
- Ali \(2026\)Sarwan Ali\.Critical windows of complexity control: When transformers decide to reason or memorize\.*arXiv preprint arXiv:2605\.04396*, 2026\.URL[https://arxiv\.org/abs/2605\.04396](https://arxiv.org/abs/2605.04396)\.
- Bahri et al\. \(2020\)Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam S Schoenholz, Jascha Sohl\-Dickstein, and Surya Ganguli\.Statistical mechanics of deep learning\.*Annual Review of Condensed Matter Physics*, 11:501–528, 2020\.
- Bi et al\. \(2026\)Yuda Bi, Chenyu Zhang, Qiheng Wang, and Vince D\. Calhoun\.Grokking as a falsifiable finite\-size transition\.*arXiv preprint arXiv:2603\.24746*, 2026\.
- Boukacem et al\. \(2024\)Nacer Eddine Boukacem, Allen Leary, Robin Thériault, Felix Gottlieb, Madhav Mani, and Paul François\.Waddington landscape for prototype learning in generalized Hopfield networks\.*Physical Review Research*, 6:033098, 2024\.doi:10\.1103/PhysRevResearch\.6\.033098\.URL[https://arxiv\.org/abs/2312\.03012](https://arxiv.org/abs/2312.03012)\.
- Chen et al\. \(2024\)Siyu Chen, Heejune Sheen, Tianhao Wang, and Zhuoran Yang\.Unveiling induction heads: Provable training dynamics and feature learning in transformers\.In*NeurIPS*, 2024\.
- Chiou et al\. \(2016\)Kevin K\. Chiou, Jason W\. Rocks, Christina Yingxian Chen, Sangkyun Cho, Koen E\. Merkus, Anjali Rajaratnam, Patrick Robison, Manorama Tewari, Kenneth Vogel, Stephanie F\. Majkut, Benjamin L\. Prosser, Dennis E\. Discher, and Andrea J\. Liu\.Mechanical signaling coordinates the embryonic heartbeat\.*Proceedings of the National Academy of Sciences of the United States of America*, 113\(32\):8939–8944, 2016\.doi:10\.1073/pnas\.1520428113\.
- D’Angelo et al\. \(2023\)Francesco D’Angelo, Maksym Andriushchenko, Aditya Varre, and Nicolas Flammarion\.Why do we need weight decay in modern deep learning?*arXiv preprint arXiv:2310\.04415*, 2023\.URL[https://arxiv\.org/abs/2310\.04415](https://arxiv.org/abs/2310.04415)\.
- Elhage et al\. \(2022\)Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield\-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah\.Toy models of superposition\.*Transformer Circuits Thread, Anthropic*, 2022\.URL[https://transformer\-circuits\.pub/2022/toy\_model/index\.html](https://transformer-circuits.pub/2022/toy_model/index.html)\.
- Frankle & Carbin \(2019\)Jonathan Frankle and Michael Carbin\.The lottery ticket hypothesis: Finding sparse, trainable neural networks\.In*ICLR*, 2019\.
- Galanti et al\. \(2022\)Tomer Galanti, Zachary S\. Siegel, Aparna Gupte, and Tomaso Poggio\.SGD and weight decay secretly minimize the rank of your neural network\.*arXiv preprint arXiv:2206\.05794*, 2022\.URL[https://arxiv\.org/abs/2206\.05794](https://arxiv.org/abs/2206.05794)\.
- Golwala \(2026\)Shreel Golwala\.ILDR: Geometric early detection of grokking\.*arXiv preprint arXiv:2604\.20923*, 2026\.URL[https://arxiv\.org/abs/2604\.20923](https://arxiv.org/abs/2604.20923)\.
- Gomezjurado Gonzalez \(2026\)Laura Gomezjurado Gonzalez\.The long delay to arithmetic generalization: When learned representations outrun behavior\.*arXiv preprint arXiv:2604\.13082*, 2026\.URL[https://arxiv\.org/abs/2604\.13082](https://arxiv.org/abs/2604.13082)\.
- Gu & Dao \(2023\)Albert Gu and Tri Dao\.Mamba: Linear\-time sequence modeling with selective state spaces\.*arXiv preprint arXiv:2312\.00752*, 2023\.
- Hays \(2026\)Hasi Hays\.Selective synchronization attention\.*arXiv preprint arXiv:2602\.14445*, 2026\.URL[https://arxiv\.org/abs/2602\.14445](https://arxiv.org/abs/2602.14445)\.
- Hennick & Corlouer \(2026\)Max Hennick and Guillaume Corlouer\.From density matrices to phase transitions in deep learning: Spectral early warnings and interpretability\.*arXiv preprint arXiv:2603\.29805*, 2026\.URL[https://arxiv\.org/abs/2603\.29805](https://arxiv.org/abs/2603.29805)\.
- Jia et al\. \(2023\)Bill Z Jia, Yitong Qi, J David Wong\-Campos, Sean G Megason, and Adam E Cohen\.A bioelectrical phase transition patterns the first vertebrate heartbeats\.*Nature*, 622\(7981\):149–155, 2023\.doi:10\.1038/s41586\-023\-06561\-z\.
- Kumar et al\. \(2024\)Tanishq Kumar, Blake Bordelon, Samuel J Gershman, and Cengiz Pehlevan\.Grokking as the transition from lazy to rich training dynamics\.In*ICLR*, 2024\.
- Liu et al\. \(2023\)Ziming Liu, Eric J Michaud, and Max Tegmark\.Omnigrok: Grokking beyond algorithmic data\.In*ICLR*, 2023\.
- Lyle et al\. \(2025\)Clare Lyle, Gharda Sokar, Razvan Pascanu, and Andras Gyorgy\.What can grokking teach us about learning under nonstationarity?*arXiv preprint arXiv:2507\.20057*, 2025\.URL[https://arxiv\.org/abs/2507\.20057](https://arxiv.org/abs/2507.20057)\.
- Lyu et al\. \(2024\)Kaifeng Lyu, Jikai Jin, Zhiyuan Li, Simon S\. Du, Jason D\. Lee, and Wei Hu\.Dichotomy of early and late phase implicit biases can provably induce grokking\.In*International Conference on Learning Representations*, 2024\.URL[https://arxiv\.org/abs/2311\.18817](https://arxiv.org/abs/2311.18817)\.
- Manir & Rupa \(2026\)Shalima Binta Manir and Anamika Paul Rupa\.A systematic empirical study of grokking: Depth, architecture, activation, and regularization\.*arXiv preprint arXiv:2603\.25009*, 2026\.URL[https://arxiv\.org/abs/2603\.25009](https://arxiv.org/abs/2603.25009)\.
- Martin et al\. \(2021\)Charles H\. Martin, Tongsu Peng, and Michael W\. Mahoney\.Predicting trends in the quality of state\-of\-the\-art neural networks without access to training or testing data\.*Nature Communications*, 12\(1\):4122, 2021\.doi:10\.1038/s41467\-021\-24025\-8\.URL[https://arxiv\.org/abs/2002\.06716](https://arxiv.org/abs/2002.06716)\.
- Michaud et al\. \(2023\)Eric J Michaud, Ziming Liu, Uzay Girit, and Max Tegmark\.The quantization model of neural scaling\.In*NeurIPS*, 2023\.
- Michel et al\. \(2019\)Paul Michel, Omer Levy, and Graham Neubig\.Are sixteen heads really better than one?In*NeurIPS*, 2019\.
- Miyato et al\. \(2025\)Takeru Miyato, Sindy Löwe, Andreas Geiger, and Max Welling\.Artificial kuramoto oscillatory neurons\.In*ICLR \(Oral\)*, 2025\.URL[https://openreview\.net/forum?id=nwDRD4AMoN](https://openreview.net/forum?id=nwDRD4AMoN)\.
- Musat \(2025\)Tiberiu Musat\.The geometry of grokking: Norm minimization on the zero\-loss manifold\.*arXiv preprint arXiv:2511\.01938*, 2025\.URL[https://arxiv\.org/abs/2511\.01938](https://arxiv.org/abs/2511.01938)\.
- Nam et al\. \(2024\)Yoonsoo Nam, Nayara Fonseca, Seok Hyeong Lee, Chris Mingard, and Ard A\. Louis\.An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem\.In*NeurIPS*, 2024\.URL[https://arxiv\.org/abs/2404\.17563](https://arxiv.org/abs/2404.17563)\.
- Nanda et al\. \(2023\)Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt\.Progress measures for grokking via mechanistic interpretability\.In*ICLR*, 2023\.
- Nguyen et al\. \(2024\)Tuan Nguyen, Hirotada Honda, Takashi Sano, Vinh Nguyen, Shugo Nakamura, and Tan M\. Nguyen\.From coupled oscillators to graph neural networks: Reducing over\-smoothing via a Kuramoto model\-based approach\.In*International Conference on Artificial Intelligence and Statistics*, 2024\.URL[https://arxiv\.org/abs/2311\.03260](https://arxiv.org/abs/2311.03260)\.
- Nitsan et al\. \(2016\)Ido Nitsan, Stavit Drori, Yair E\. Lewis, Shlomi Cohen, and Shelly Tzlil\.Mechanical communication in cardiac cell synchronized beating\.*Nature Physics*, 12\(5\):472–477, 2016\.doi:10\.1038/nphys3619\.
- Olsson et al\. \(2022\)Catherine Olsson, Nelson Elhage, Neel Nanda, et al\.In\-context learning and induction heads\.*Transformer Circuits Thread, Anthropic*, 2022\.
- Papyan et al\. \(2020\)Vardan Papyan, X Y Han, and David L Donoho\.Prevalence of neural collapse during the terminal phase of deep learning training\.*PNAS*, 117\(40\):24652–24663, 2020\.
- Poole et al\. \(2016\)Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl\-Dickstein, and Surya Ganguli\.Exponential expressivity in deep neural networks through transient chaos\.In*NeurIPS*, 2016\.
- Power et al\. \(2022\)Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra\.Grokking: Generalization beyond overfitting on small algorithmic datasets\.*arXiv preprint arXiv:2201\.02177*, 2022\.
- Prakash & Martin \(2026a\)Hari K\. Prakash and Charles H\. Martin\.Late\-stage generalization collapse in grokking: Detecting anti\-grokking with Weightwatcher\.*arXiv preprint arXiv:2602\.02859*, 2026a\.URL[https://arxiv\.org/abs/2602\.02859](https://arxiv.org/abs/2602.02859)\.
- Prakash & Martin \(2026b\)Hari K\. Prakash and Charles H\. Martin\.Detecting overfitting in neural networks during long\-horizon grokking using random matrix theory\.*arXiv preprint arXiv:2605\.12394*, 2026b\.URL[https://arxiv\.org/abs/2605\.12394](https://arxiv.org/abs/2605.12394)\.
- Sagitova et al\. \(2026\)M\. Sagitova, O\. Duranthon, and L\. Zdeborová\.Specialization of softmax attention heads: Insights from the high\-dimensional single\-location model\.*arXiv preprint arXiv:2603\.03993*, 2026\.URL[https://arxiv\.org/abs/2603\.03993](https://arxiv.org/abs/2603.03993)\.
- Saxe et al\. \(2014\)Andrew M Saxe, James L McClelland, and Surya Ganguli\.Exact solutions to the nonlinear dynamics of learning in deep linear neural networks\.In*ICLR*, 2014\.
- Saxe et al\. \(2019\)Andrew M Saxe, James L McClelland, and Surya Ganguli\.A mathematical theory of semantic development in deep neural networks\.*Proceedings of the National Academy of Sciences*, 116\(23\):11537–11546, 2019\.doi:10\.1073/pnas\.1820226116\.URL[https://arxiv\.org/abs/1810\.10531](https://arxiv.org/abs/1810.10531)\.
- Schaeffer et al\. \(2023\)Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo\.Are emergent abilities of large language models a mirage?In*NeurIPS*, 2023\.URL[https://arxiv\.org/abs/2304\.15004](https://arxiv.org/abs/2304.15004)\.
- Simon et al\. \(2026\)Jamie Simon, Daniel Kunin, Alexander Atanasov, Enric Boix\-Adserà, Blake Bordelon, Jeremy Cohen, Nikhil Ghosh, Florentin Guth, Arthur Jacot, Mason Kamb, Dhruva Karkada, Eric J\. Michaud, Berkan Ottlik, and Joseph Turnbull\.There will be a scientific theory of deep learning\.*arXiv preprint arXiv:2604\.21691*, 2026\.
- Singh et al\. \(2026\)Jaisidh Singh, Diganta Misra, and Antonio Orvieto\.Explaining grokking in transformers through the lens of inductive bias\.*arXiv preprint arXiv:2602\.06702*, 2026\.URL[https://arxiv\.org/abs/2602\.06702](https://arxiv.org/abs/2602.06702)\.
- Song & Ye \(2026\)Yiding Song and Hanming Ye\.Model capacity determines grokking through competing memorisation and generalisation speeds\.*arXiv preprint arXiv:2605\.09724*, 2026\.URL[https://arxiv\.org/abs/2605\.09724](https://arxiv.org/abs/2605.09724)\.
- Tang et al\. \(2026\)Yifan Tang, Qiquan Wang, Inés García\-Redondo, and Anthea Monod\.Topological signatures of grokking\.*arXiv preprint arXiv:2605\.06352*, 2026\.URL[https://arxiv\.org/abs/2605\.06352](https://arxiv.org/abs/2605.06352)\.
- Tian \(2025\)Yuandong Tian\.Provable scaling laws of feature emergence from learning dynamics of grokking\.*arXiv preprint arXiv:2509\.21519*, 2025\.doi:10\.48550/arXiv\.2509\.21519\.URL[https://arxiv\.org/abs/2509\.21519](https://arxiv.org/abs/2509.21519)\.
- Truong Xuan Khanh et al\. \(2026a\)Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung, and Phan Thanh Duc\.The norm\-separation delay law of grokking: A first\-principles theory of delayed generalization\.*arXiv preprint arXiv:2603\.13331*, 2026a\.
- Truong Xuan Khanh et al\. \(2026b\)Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung, and Phan Thanh Duc\.Spectral entropy collapse as a phase transition in delayed generalisation: An interventional and predictive framework for grokking\.*arXiv preprint arXiv:2604\.13123*, 2026b\.
- Varma et al\. \(2023\)Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar\.Explaining grokking through circuit efficiency\.*arXiv preprint arXiv:2309\.02390*, 2023\.
- Voita et al\. \(2019\)Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov\.Analyzing multi\-head self\-attention: Specialized heads do the heavy lifting, the rest can be pruned\.In*ACL*, 2019\.
- Wang et al\. \(2024\)George Wang, Jesse Hoogland, Stan van Wingerden, Zach Furman, and Daniel Murfet\.Differentiation and specialization of attention heads via the refined local learning coefficient\.*arXiv preprint arXiv:2410\.02984*, 2024\.
- Wang \(2026a\)Ping Wang\.Grokking as dimensional phase transition in neural networks\.*arXiv preprint arXiv:2604\.04655*, 2026a\.URL[https://arxiv\.org/abs/2604\.04655](https://arxiv.org/abs/2604.04655)\.
- Wang \(2026b\)Ping Wang\.Dimensional criticality at grokking across MLPs and transformers\.*arXiv preprint arXiv:2604\.16431*, 2026b\.
- Wang & Aitchison \(2024\)Xi Wang and Laurence Aitchison\.How to set adamw’s weight decay as you scale model and dataset size\.*arXiv preprint arXiv:2405\.13698*, 2024\.
- Wang et al\. \(2026\)Ziyue Wang, Yufeng Ying, and Takafumi Kanamori\.Distributional spectral diagnostics for localizing grokking transitions\.*arXiv preprint arXiv:2605\.08237*, 2026\.URL[https://arxiv\.org/abs/2605\.08237](https://arxiv.org/abs/2605.08237)\.
- Wei et al\. \(2022\)Jason Wei, Yi Tay, Rishi Bommasani, et al\.Emergent abilities of large language models\.*TMLR*, 2022\.
- Xu et al\. \(2026\)Mingyue Xu, Gal Vardi, and Itay Safran\.To grok grokking: Provable grokking in ridge regression\.*arXiv preprint arXiv:2601\.19791*, 2026\.URL[https://arxiv\.org/abs/2601\.19791](https://arxiv.org/abs/2601.19791)\.
- Xu \(2026a\)Yongzhong Xu\.Early\-warning signals of grokking via loss\-landscape geometry\.*arXiv preprint arXiv:2602\.16967*, 2026a\.
- Xu \(2026b\)Yongzhong Xu\.Spectral edge dynamics reveal functional modes of learning\.*arXiv preprint arXiv:2604\.06256*, 2026b\.URL[https://arxiv\.org/abs/2604\.06256](https://arxiv.org/abs/2604.06256)\.
- Xu \(2026c\)Yongzhong Xu\.Low\-dimensional and transversely curved optimization dynamics in grokking\.*arXiv preprint arXiv:2602\.16746*, 2026c\.
- Xu \(2026d\)Yongzhong Xu\.The geometry of multi\-task grokking: Transverse instability, superposition, and weight decay phase structure\.*arXiv preprint arXiv:2602\.18523*, 2026d\.
- Xu \(2026e\)Yongzhong Xu\.Spectral edge dynamics: An analytical\-empirical study of phase transitions in neural network training\.*arXiv preprint arXiv:2603\.28964*, 2026e\.doi:10\.48550/arXiv\.2603\.28964\.URL[https://arxiv\.org/abs/2603\.28964](https://arxiv.org/abs/2603.28964)\.
- Xu \(2026f\)Yongzhong Xu\.The lifecycle of the spectral edge: From gradient learning to weight\-decay compression\.*arXiv preprint arXiv:2604\.07380*, 2026f\.URL[https://arxiv\.org/abs/2604\.07380](https://arxiv.org/abs/2604.07380)\.
- Yıldırım \(2026\)Alper Yıldırım\.The geometric inductive bias of grokking: Bypassing phase transitions via architectural topology\.*arXiv preprint arXiv:2603\.05228*, 2026\.URL[https://arxiv\.org/abs/2603\.05228](https://arxiv.org/abs/2603.05228)\.
- Zhang et al\. \(2026\)Junjie Zhang, Zhen Shen, Gang Xiong, and Xisong Dong\.Grokking from abstraction to intelligence\.*arXiv preprint arXiv:2603\.29262*, 2026\.doi:10\.48550/arXiv\.2603\.29262\.URL[https://arxiv\.org/abs/2603\.29262](https://arxiv.org/abs/2603.29262)\.
- Zhang et al\. \(2025\)Xiaotian Zhang, Yue Shang, Entao Yang, and Ge Zhang\.Is grokking a computational glass relaxation?*arXiv preprint arXiv:2505\.11411*, 2025\.
- Ziyin & Ueda \(2023\)Liu Ziyin and Masahito Ueda\.Zeroth, first, and second\-order phase transitions in deep neural networks\.*Physical Review Research*, 5:043243, 2023\.
- Žunkovič & Ilievski \(2024\)Bojan Žunkovič and Enej Ilievski\.Grokking phase transitions in learning local rules with gradient descent\.*Journal of Machine Learning Research*, 25\(199\):1–52, 2024\.URL[https://jmlr\.org/papers/v25/22\-1228\.html](https://jmlr.org/papers/v25/22-1228.html)\.

## Appendix AExperiment Index

Throughout the paper we refer to specific experimental cohorts by short codes \(E⋅\\cdot, M⋅\\cdot, X⋅\\cdot\)\. Table[2](https://arxiv.org/html/2605.20441#A1.T2)maps each code to its scope, configuration, and seed count\.

Table 2:Experiment\-cohort index\. Body refers to each cohort by itsE⋅E\\cdotlabel in parentheses on first mention within each subsection\. Aggregate run counts in the manuscript text are the union of the listed cohorts\.

## Appendix BComplementary Order Parameters

Two complementary controls are logged alongside the headline diagnosticss¯\\bar\{s\}andσH\\sigma\_\{H\}\(§[3\.2](https://arxiv.org/html/2605.20441#S3.SS2)\) but not used for the regime map or causal contrasts:

rϕ​\(t\)\\displaystyle r\_\{\\phi\}\(t\)≔\|𝔼l,h​\[ei​ϕl​h​\(t\)\]\|\\displaystyle\\coloneqq\\left\|\\,\\mathbb\{E\}\_\{l,h\}\\\!\\left\[e^\{i\\phi\_\{lh\}\(t\)\}\\right\]\\,\\right\|\\quad\(Kuramoto coherence\)\(4\)λg​\(t\)\\displaystyle\\lambda\_\{g\}\(t\)≔λ1​\(S\)−λ2​\(S\),Si​j=cos⁡\(Ai,Aj\)\\displaystyle\\coloneqq\\lambda\_\{1\}\(S\)\-\\lambda\_\{2\}\(S\),\\;S\_\{ij\}\\\!=\\\!\\cos\(A\_\{i\},A\_\{j\}\)\\quad\(similarity\-matrix spectral gap\)\(5\)whereϕl​h\\phi\_\{lh\}is the principal attention\-pattern direction\. We observe thatrϕr\_\{\\phi\}trackss¯\\bar\{s\}closely under the canonical configuration, andλg\\lambda\_\{g\}adds noise without resolving Phase 2 better thanσH\\sigma\_\{H\}, so neither is reported in the main results\. Both are available in the supplementary trace artifacts for downstream analyses requiring oscillator\-style or spectral\-gap framings\.

## Appendix CDiagnostic Properties and AdamW\-Relaxation Bound

This appendix records mathematical properties of the diagnostics introduced in §[3\.2](https://arxiv.org/html/2605.20441#S3.SS2)and a one\-parameter relaxation bound on the empirical critical weight decayλc\\lambda\_\{c\}\. The five elementary results \(A1, B1, C1, D1, E1\) confirm the order parameters are well\-defined and bounded; the Section[C\.5](https://arxiv.org/html/2605.20441#A3.SS5)relaxation argument boundsλc\\lambda\_\{c\}via the AdamW decoupled\-WD update with one empirically\-fit constant\. Four custom Lean 4 proofs \(mathlib v4\.29\.0\) compile without anysorry: A1 \(parts 1 to 3, three\-regime competition\), B1 \(large\-WD collapse\), C1 \(raw participation\-ratio/CV identity plus affine\-normalized PRnormform via an explicit variance lemma\), and E1 \(rank bound\)\. D1 \(Hoeffding\) follows from mathlib’s standard concentration bound and is not re\-derived here \(consistent with the standard concentration result\)\.

### C\.1Three\-Regime Regularized Competition

#### Theorem A1\.

Let

Jλ​\(f\)=Ltr​\(f\)\+λ​Ω​\(f\)J\_\{\\lambda\}\(f\)=L\_\{\\mathrm\{tr\}\}\(f\)\+\\lambda\\Omega\(f\)and suppose three candidate solution families, memorizingMM, generalizingGG, and collapsedCC, have representative losses and complexities satisfying

ℓM≤ℓG<ℓC,ωC<ωG<ωM\.\\ell\_\{M\}\\leq\\ell\_\{G\}<\\ell\_\{C\},\\qquad\\omega\_\{C\}<\\omega\_\{G\}<\\omega\_\{M\}\.Define

λM​G=ℓG−ℓMωM−ωG,λG​C=ℓC−ℓGωG−ωC\.\\lambda\_\{MG\}=\\frac\{\\ell\_\{G\}\-\\ell\_\{M\}\}\{\\omega\_\{M\}\-\\omega\_\{G\}\},\\qquad\\lambda\_\{GC\}=\\frac\{\\ell\_\{C\}\-\\ell\_\{G\}\}\{\\omega\_\{G\}\-\\omega\_\{C\}\}\.If0≤λM​G<λG​C0\\leq\\lambda\_\{MG\}<\\lambda\_\{GC\}, thenMMis preferred toGGforλ<λM​G\\lambda<\\lambda\_\{MG\},GGis preferred to bothMMandCCforλM​G<λ<λG​C\\lambda\_\{MG\}<\\lambda<\\lambda\_\{GC\}, andCCis preferred toGGforλ\>λG​C\\lambda\>\\lambda\_\{GC\}\.

#### Proof sketch\.

The objective gapsJλ​\(fM\)−Jλ​\(fG\)J\_\{\\lambda\}\(f\_\{M\}\)\-J\_\{\\lambda\}\(f\_\{G\}\)andJλ​\(fG\)−Jλ​\(fC\)J\_\{\\lambda\}\(f\_\{G\}\)\-J\_\{\\lambda\}\(f\_\{C\}\)are affine functions ofλ\\lambdawith slopesωM−ωG\>0\\omega\_\{M\}\-\\omega\_\{G\}\>0andωG−ωC\>0\\omega\_\{G\}\-\\omega\_\{C\}\>0\. They cross zero atλM​G\\lambda\_\{MG\}andλG​C\\lambda\_\{GC\}, respectively\. The strict ordering of the crossings gives a nonempty intermediate interval in which the generalizing candidate beats both alternatives\.

### C\.2Large\-Weight\-Decay Collapse

#### Theorem B1\.

Assume0≤Ltr​\(f\)≤Lmax0\\leq L\_\{\\mathrm\{tr\}\}\(f\)\\leq L\_\{\\max\}on the hypothesis class andΩ​\(f\)≥0\\Omega\(f\)\\geq 0\. LetΩmin=minf⁡Ω​\(f\)\\Omega\_\{\\min\}=\\min\_\{f\}\\Omega\(f\)and

ℱϵ=\{f:Ω​\(f\)≥Ωmin\+ϵ\}\.\\mathcal\{F\}\_\{\\epsilon\}=\\\{f:\\Omega\(f\)\\geq\\Omega\_\{\\min\}\+\\epsilon\\\}\.Ifλ\>Lmax/ϵ\\lambda\>L\_\{\\max\}/\\epsilon, no minimizer ofJλ​\(f\)=Ltr​\(f\)\+λ​Ω​\(f\)J\_\{\\lambda\}\(f\)=L\_\{\\mathrm\{tr\}\}\(f\)\+\\lambda\\Omega\(f\)lies inℱϵ\\mathcal\{F\}\_\{\\epsilon\}\.

#### Proof sketch\.

Choosef0f\_\{0\}withΩ​\(f0\)=Ωmin\\Omega\(f\_\{0\}\)=\\Omega\_\{\\min\}\. For anyf∈ℱϵf\\in\\mathcal\{F\}\_\{\\epsilon\},

Jλ​\(f\)−Jλ​\(f0\)≥−Lmax\+λ​ϵ\.J\_\{\\lambda\}\(f\)\-J\_\{\\lambda\}\(f\_\{0\}\)\\geq\-L\_\{\\max\}\+\\lambda\\epsilon\.The right\-hand side is positive whenλ\>Lmax/ϵ\\lambda\>L\_\{\\max\}/\\epsilon, so such anffcannot minimize the regularized objective\. In this paper, the link from this minimum\-complexity region to uniform or collapsed attention is empirical\.

### C\.3Participation Ratio and Head Dispersion

#### Theorem C1\.

Letah≥0a\_\{h\}\\geq 0be the nonnegative spectral weights of the head\-covariance matrix at a fixed layer and checkpoint\. If at least oneaha\_\{h\}is nonzero, define the raw participation ratio

R=\(∑h=1Hah\)2∑h=1Hah2R=\\frac\{\\big\(\\sum\_\{h=1\}^\{H\}a\_\{h\}\\big\)^\{2\}\}\{\\sum\_\{h=1\}^\{H\}a\_\{h\}^\{2\}\}and the released affine\-normalized diagnostic

PRnorm=R−1H−1\.\\mathrm\{PR\}\_\{\\mathrm\{norm\}\}=\\frac\{R\-1\}\{H\-1\}\.Then1≤R≤H1\\leq R\\leq Hand0≤PRnorm≤10\\leq\\mathrm\{PR\}\_\{\\mathrm\{norm\}\}\\leq 1\. With the population mean and variance

μ=1H​∑hah,σ2=1H​∑h\(ah−μ\)2,\\mu=\\frac\{1\}\{H\}\\sum\_\{h\}a\_\{h\},\\qquad\\sigma^\{2\}=\\frac\{1\}\{H\}\\sum\_\{h\}\(a\_\{h\}\-\\mu\)^\{2\},the raw ratio satisfies the exact identity

RH=μ2μ2\+σ2=11\+CV2,PRnorm=H/\(1\+CV2\)−1H−1\.\\frac\{R\}\{H\}=\\frac\{\\mu^\{2\}\}\{\\mu^\{2\}\+\\sigma^\{2\}\}=\\frac\{1\}\{1\+\\mathrm\{CV\}^\{2\}\},\\qquad\\mathrm\{PR\}\_\{\\mathrm\{norm\}\}=\\frac\{H/\(1\+\\mathrm\{CV\}^\{2\}\)\-1\}\{H\-1\}\.

#### Proof sketch\.

Cauchy\-Schwarz gives\(∑hah\)2≤H​∑hah2\(\\sum\_\{h\}a\_\{h\}\)^\{2\}\\leq H\\sum\_\{h\}a\_\{h\}^\{2\}, yieldingR≤HR\\leq H, and\(∑hah\)2≥∑hah2\(\\sum\_\{h\}a\_\{h\}\)^\{2\}\\geq\\sum\_\{h\}a\_\{h\}^\{2\}for nonnegativeaha\_\{h\}, yieldingR≥1R\\geq 1\. Substituting∑hah=H​μ\\sum\_\{h\}a\_\{h\}=H\\muand∑hah2=H​\(μ2\+σ2\)\\sum\_\{h\}a\_\{h\}^\{2\}=H\(\\mu^\{2\}\+\\sigma^\{2\}\)into the definition gives the coefficient\-of\-variation identity and then the affine\-normalized form\.

### C\.4Empirical Validation of C1 Identity

The C1 identity is also checked on saved checkpoint data\. The validation uses the canonical seed\-42 trajectory plus cross\-seed cohort seeds 7, 11, 31, and 123\. The analysis compares measured participation ratio against the value predicted from the eigenvalue coefficient of variation after correcting the stored sample standard deviation to a population standard deviation, then applies the affine normalization used by the released JSON artifacts\. Across 183 valid layer\-epoch rows, the mean absolute raw\-PR error is2\.10×10−72\.10\{\\times\}10^\{\-7\}, the maximum raw\-PR error is1\.73×10−61\.73\{\\times\}10^\{\-6\}, and the maximum affine\-normalized PR error is2\.56×10−72\.56\{\\times\}10^\{\-7\}\.

Table 3:C1 empirical validation on the canonical seed\-42 trajectory, averaged across four layers per checkpoint\. The displayed eigenvalue\-dispersion statistic is the sampleCV​\(λ\)\\mathrm\{CV\}\(\\lambda\); C1 prediction first applies the sample\-to\-population variance correction layerwise and then the released affine normalization before averaging\. The statistic rises from 0\.392 at epoch 100 to 1\.880 at epoch 20 000, with phase\-scale oscillations; the C1\-predicted affinePRnorm\\mathrm\{PR\}\_\{\\mathrm\{norm\}\}matches the measured value to the displayed precision\.This check only validates the algebraic equivalence between the participation\-ratio order parameter and eigenvalue dispersion for the tested checkpoint stack\. It does not establish a causal threshold for symmetry breaking, nor does it transfer the diagnostic to non\-attention architectures\.

### C\.5AdamW\-Relaxation Argument Boundingλc\\lambda\_\{c\}

The minimal\-model thresholds in Theorem A1 are existence statements; they do not predict the numerical value ofλc\\lambda\_\{c\}\. Section[C\.5](https://arxiv.org/html/2605.20441#A3.SS5)narrows the predictive gap with a relaxation argument that combines A1 with the AdamW decoupled\-weight\-decay update rule and one empirically\-fit constant\. We do not claim a first\-principles derivation: the AdamW amplificationκ\\kappais fit from one canonical training trajectory rather than derived from the optimizer’s second\-moment dynamics\. The argument therefore predictsλc\\lambda\_\{c\}as a function of\(η,Tmax,prelax,κ\)\(\\eta,T\_\{\\max\},p\_\{\\text\{relax\}\},\\kappa\)withκ\\kappaas the single remaining empirical parameter\.

#### Relaxation argument\.

Under decoupled weight decay\(D’Angelo et al\.,[2023](https://arxiv.org/html/2605.20441#bib.bib8)\), the AdamW update isθt\+1=θt−η​m^t/\(v^t\+ϵ\)−η​λ​θt\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\eta\\,\\hat\{m\}\_\{t\}/\(\\sqrt\{\\hat\{v\}\_\{t\}\}\+\\epsilon\)\-\\eta\\lambda\\theta\_\{t\}, whereη\\etais the learning rate andλ\\lambdathe weight\-decay coefficient\. The deterministic component of the parameter norm therefore decays as

Ω​\(t\)=Ω∞\+\(Ω0−Ω∞\)​e−η​λeff​t,λeff=κ​λ,\\Omega\(t\)=\\Omega\_\{\\infty\}\+\(\\Omega\_\{0\}\-\\Omega\_\{\\infty\}\)\\,e^\{\-\\eta\\lambda\_\{\\mathrm\{eff\}\}\\,t\},\\qquad\\lambda\_\{\\mathrm\{eff\}\}=\\kappa\\,\\lambda,\(6\)whereΩ0\\Omega\_\{0\}is the initial Frobenius norm,Ω∞\\Omega\_\{\\infty\}is the asymptote of the post\-grokking trajectory, andκ\\kappais the AdamW amplification factor that absorbs adaptive\-step rescaling\. Theorem A1 placesΩ0\\Omega\_\{0\}in the M\-basin andΩ∞\\Omega\_\{\\infty\}in the G\-basin; Theorem B1 ensures the trajectory cannot escape the regularized\-minimum region for sufficiently largeλ\\lambda\.

#### Relation to Khanh et al\. \(2026\)\.

Truong Xuan Khanh et al\. \([2026a](https://arxiv.org/html/2605.20441#bib.bib47)\)independently derive a closely related result via a discrete Lyapunov contraction argument:Tgrok−Tmem=Θ​\(\(1/γeff\)​log⁡\(‖θmem‖2/‖θpost‖2\)\)T\_\{\\text\{grok\}\}\-T\_\{\\text\{mem\}\}=\\Theta\(\(1/\\gamma\_\{\\text\{eff\}\}\)\\,\\log\(\\\|\\theta\_\{\\text\{mem\}\}\\\|^\{2\}/\\\|\\theta\_\{\\text\{post\}\}\\\|^\{2\}\)\), withγeff=η​λ\\gamma\_\{\\text\{eff\}\}=\\eta\\lambdafor SGD andγeff≥η​λ\\gamma\_\{\\text\{eff\}\}\\geq\\eta\\lambdafor AdamW\. Theirγeff\\gamma\_\{\\text\{eff\}\}corresponds to ourη​κ​λ\\eta\\,\\kappa\\,\\lambda, and the norm ratio‖θmem‖2/‖θpost‖2\\\|\\theta\_\{\\text\{mem\}\}\\\|^\{2\}/\\\|\\theta\_\{\\text\{post\}\}\\\|^\{2\}corresponds to our\(Ω0/Ω∞\)2\(\\Omega\_\{0\}/\\Omega\_\{\\infty\}\)^\{2\}\. Khanh et al\. provide an analytic lower bound on AdamW’s effective contraction rate \(γeff≥η​λ\\gamma\_\{\\text\{eff\}\}\\geq\\eta\\lambda\) and a forward prediction of grokking delay; we provide the empirical*calibration*ofκ\\kappaon five canonical\-architecture trajectories \(full\-fit canonical\-trajectoryκ=18\.6\\kappa\{=\}18\.6; early\-window cross\-seed meanκ=14\.4\\kappa\{=\}14\.4, range8\.18\.1to19\.219\.2\) and the dual*inverted*formulationλc\\lambda\_\{c\}as a function ofTmaxT\_\{\\max\}\. Section[C\.5](https://arxiv.org/html/2605.20441#A3.SS5)is the empirical instance of the Khanh framework, restricted to the canonical 4L8H mod\-add cohort, with cross\-seed bimodalκ\\kappadocumented as a target for the SDE refinement they leave open\.

#### Horizon constraint\.

Grokking within training requires the relaxed norm to reach within a fraction1−prelax1\-p\_\{\\text\{relax\}\}of the G\-basin asymptote by stepTmaxT\_\{\\max\}:

Ω​\(Tmax\)−Ω∞≤\(1−prelax\)​\(Ω0−Ω∞\)\.\\Omega\(T\_\{\\max\}\)\-\\Omega\_\{\\infty\}\\leq\(1\-p\_\{\\text\{relax\}\}\)\(\\Omega\_\{0\}\-\\Omega\_\{\\infty\}\)\.\(7\)Solving for the smallestλ\\lambdasatisfying this constraint gives

λc=−ln⁡\(1−prelax\)η​κ​Tmax\.\\boxed\{\\lambda\_\{c\}\\;=\\;\\frac\{\-\\ln\(1\-p\_\{\\text\{relax\}\}\)\}\{\\eta\\,\\kappa\\,T\_\{\\max\}\}\.\}\(8\)

#### Numerical instantiation\.

For the canonical\-trajectory cohort \(4L8H,d=128d\{=\}128,p=97p\{=\}97,η=10−3\\eta\{=\}10^\{\-3\},Tmax=20 000T\_\{\\max\}\{=\}20\\,000\), an exponential fit to theΩtotal​\(t\)\\Omega\_\{\\text\{total\}\}\(t\)trajectory across1111saved checkpoints \(trainingλ=1\.0\\lambda\{=\}1\.0\) givesΩ∞=28\.4\\Omega\_\{\\infty\}=28\.4,Ω0≈200\\Omega\_\{0\}\\approx 200\(fit upper bound, trajectory starts at55\.255\.2\), andη​λeff=1\.86×10−2\\eta\\lambda\_\{\\text\{eff\}\}=1\.86\\times 10^\{\-2\}, henceκ=18\.6\\kappa=18\.6\.*Thisκ\\kappavalue is fit on the single canonical seed\-42 trajectory*; the cross\-seed cohort mean reported in the next paragraph \(κ=14\.4\\kappa\{=\}14\.4, range8\.18\.1to19\.219\.2\) is the honest summary across the five\-seed cohort\. Table[4](https://arxiv.org/html/2605.20441#A3.T4)records the sensitivity ofλcbound\\lambda\_\{c\}^\{\\text\{bound\}\}to the relaxation criterionprelaxp\_\{\\text\{relax\}\}\. The natural physical choiceprelax=0\.99p\_\{\\text\{relax\}\}\{=\}0\.99\(matching the test\-accuracy≥0\.99\\geq 0\.99grokking criterion\) givesλcbound=0\.0124\\lambda\_\{c\}^\{\\text\{bound\}\}=0\.0124, inside the empirical 95% CI\[0\.0109,0\.0200\]\[0\.0109,0\.0200\]forλc=0\.0158\\lambda\_\{c\}=0\.0158\.

#### Cross\-seed stability ofκ\\kappa\.

Repeating the fit across the44available cross\-seed checkpoint trajectories \(seeds77,1111,3131,123123\) at the same architecture andλ=1\.0\\lambda\{=\}1\.0, plus the canonical trajectory \(seed4242\), reveals a fit\-window dependence: when fit on the full20 00020\\,000\-epoch trajectory the late\-stage anti\-grok cycle \(§[4\.2](https://arxiv.org/html/2605.20441#S4.SS2)\) contaminates the single\-exponential and produces a bimodalκ\\kappadistribution \(\{18\.6,19\.0\}\\\{18\.6,19\.0\\\}for seeds\{42,7\}\\\{42,7\\\}vs\.\{5\.5,6\.9,6\.4\}\\\{5\.5,6\.9,6\.4\\\}for seeds\{11,31,123\}\\\{11,31,123\\\},2/52/5in CI\)\. Restricting the fit to the M→\\toG transition window \(t≤5 000t\\leq 5\\,000\) eliminates contamination from the late cycle and yieldsκ\\kappavalues\{18\.4,19\.2,17\.9,8\.1,8\.6\}\\\{18\.4,19\.2,17\.9,8\.1,8\.6\\\}across the same55seeds, with boundλc∈\{0\.0125,0\.0120,0\.0128,0\.0284,0\.0268\}\\lambda\_\{c\}\\in\\\{0\.0125,0\.0120,0\.0128,0\.0284,0\.0268\\\}\. Three of five cohorts now fall in the empirical 95% CI; the across\-cohort mean shifts toλcbound=0\.0185±0\.0074\\lambda\_\{c\}^\{\\text\{bound\}\}=0\.0185\\pm 0\.0074\(range\[0\.012,0\.028\]\[0\.012,0\.028\]\), with the mean itself inside the empirical CI\. The remaining out\-of\-CI seeds \(3131,123123\) exhibit slower early\-relaxation rates that the model does not yet explain; full SDE refinement is the natural follow\-up\. Figure[11](https://arxiv.org/html/2605.20441#A3.F11)plots the trajectories and early\-fit residuals\.

![Refer to caption](https://arxiv.org/html/2605.20441v1/x11.png)Figure 11:\(a\)Ωtotal​\(t\)\\Omega\_\{\\text\{total\}\}\(t\)Frobenius\-norm trajectories for55cross\-seed cohorts \(canonical seed4242\+ cross\-seed cohort seeds77,1111,3131,123123\) at the same architecture, trainingλ=1\.0\\lambda\{=\}1\.0,20 00020\\,000epochs\. Dashed lines show single\-exponential AdamW\-relaxation fits restricted to the M→\\toG transition window \(t≤5 000t\\leq 5\\,000, vertical gray\)\. Late\-stage cycle visible att\>10 000t\\\!\>\\\!10\\,000explains the full\-fit bimodalκ\\kappa\. \(b\) Early\-window residuals are mostly within a few Frobenius units but include seed\-level excursions from−4\.30\-4\.30to\+7\.67\+7\.67, so the fit is a useful calibration rather than a tight mechanistic law\.Table 4:Sensitivity of the AdamW\-relaxationλcbound\\lambda\_\{c\}^\{\\text\{bound\}\}to the relaxation criterionprelaxp\_\{\\text\{relax\}\}\. Atprelax=0\.99p\_\{\\text\{relax\}\}\{=\}0\.99\(the criterion physically matched to the empirical grokking threshold\) the bound lies inside the empirical 95% CI forλc=0\.0158\\lambda\_\{c\}\{=\}0\.0158\.
#### Two complementary bounds bracket the developmental regime\.

The relaxation derivation provides the lower bound \(M→\\toG transition\)\. An independent symbolic\-regression analysis on the E9 multi\-task amplitude data \(PySR, joint formσHmax≈c​\(d/H\)/\(wd\+d/H\)\\sigma\_\{H\}^\{\\max\}\\approx c\\,\(d/H\)/\(\\mathrm\{wd\}\+\\sqrt\{d/H\}\)preferred over the saturating\-exponential ansatz withΔ​AIC\\Delta\\text\{AIC\}from 215 to 273 across cohorts\) gives an upper bound: the half\-maximum amplitude crossover sits atλc≈d/H\\lambda\_\{c\}\\approx\\sqrt\{d/H\}, equal to4\.04\.0for the canonicald/H=16d/H=16cohort\. This is the G→\\toC anti\-grokking boundary at highwd\\mathrm\{wd\}, distinct from the M→\\toG transition and consistent with the empirical observation thatλ=10\\lambda\{=\}10collapses heads to identical patterns\. The two bounds bracket the developmental regime\[0\.012,4\]\[0\.012,4\]within an order of magnitude of the empirical\[0\.0158,∼5\]\[0\.0158,\\sim 5\]\.

#### What this derivation does not establish\.

The derivation does not predict the critical exponentν\\nu, which requires finite\-size\-scaling data collapse with denser grids \(deferred\)\. It does not establish transformer\-grokking universality, and the AdamW amplification factorκ=18\.6\\kappa=18\.6is fit, not derived from the optimizer’s second\-moment dynamics \(a full SDE derivation is left for follow\-up\)\. The single canonical seed\-42 trajectory used to pin the canonicalκ\\kappadoes not span the model\-scale axis; cross\-cohort calibration \(3 of 5 seeds inside the empiricalλc\\lambda\_\{c\}CI under the early\-window fit\) is reported but full validation is future work\.

### C\.6Online Order\-Parameter Concentration

#### Theorem D1\.

For a fixed checkpoint and layer, let

Zb=2H​\(H−1\)​∑i<jcos⁡\(vec​\(Ab,i\),vec​\(Ab,j\)\)Z\_\{b\}=\\frac\{2\}\{H\(H\-1\)\}\\sum\_\{i<j\}\\cos\(\\mathrm\{vec\}\(A\_\{b,i\}\),\\mathrm\{vec\}\(A\_\{b,j\}\)\)be the per\-example mean pairwise head similarity\. SinceZb∈\[−1,1\]Z\_\{b\}\\in\[\-1,1\], the batch estimators^B=B−1​∑b=1BZb\\hat\{s\}\_\{B\}=B^\{\-1\}\\sum\_\{b=1\}^\{B\}Z\_\{b\}satisfies

Pr⁡\(\|s^B−s\|≥ϵ\)≤2​exp⁡\(−B​ϵ22\)\.\\Pr\(\|\\hat\{s\}\_\{B\}\-s\|\\geq\\epsilon\)\\leq 2\\exp\\\!\\left\(\-\\frac\{B\\epsilon^\{2\}\}\{2\}\\right\)\.

#### Proof sketch\.

Apply Hoeffding’s inequality to independent bounded variables with range length22\. The same bounded\-variable logic applies to per\-head entropy estimates because attention entropy lies in\[0,log⁡T\]\[0,\\log T\]for sequence lengthTT; the across\-head standard deviation is a Lipschitz function of the vector of head entropies\.

### C\.7Head\-Dimension Capacity Bound

#### Proposition E1\.

For a single attention head withQ,K∈ℝT×dhQ,K\\in\\mathbb\{R\}^\{T\\times d\_\{h\}\}, the unnormalized score matrixS=Q​K⊤S=QK^\{\\top\}has rank at mostdhd\_\{h\}\.

#### Corollary E2\.

If a target attention\-score kernelS⋆∈ℝT×TS\_\{\\star\}\\in\\mathbb\{R\}^\{T\\times T\}has rankrr, then one dot\-product attention head can represent it exactly only ifdh≥rd\_\{h\}\\geq r\.

#### Proof sketch\.

Matrix rank submultiplicativity gives

rank​\(Q​K⊤\)≤min⁡\{rank​\(Q\),rank​\(K⊤\)\}≤dh\.\\mathrm\{rank\}\(QK^\{\\top\}\)\\leq\\min\\\{\\mathrm\{rank\}\(Q\),\\mathrm\{rank\}\(K^\{\\top\}\)\\\}\\leq d\_\{h\}\.IfS⋆=Q​K⊤S\_\{\\star\}=QK^\{\\top\}exactly, its rank cannot exceeddhd\_\{h\}, giving the corollary\. The observedd/Hd/Hthreshold in this paper is therefore consistent with a low\-rank capacity bottleneck, but the exact target\-kernel rank of the modular\-arithmetic circuit is not identified here\.

## Appendix DCode and Data Provenance

Table[5](https://arxiv.org/html/2605.20441#A4.T5)maps each manuscript claim or figure to the paper\-build analysis component and data artifact\. The public repository ships the executable reviewer\-facing subset underscripts/andeval/scripts/, together with aggregate JSONs undereval/, the coverage manifest underdocs/, and the Lean 4 formalisation underlean\_proofs/\. Some rows name build\-pipeline components whose public counterpart is the shipped aggregate artifact plus verifier rather than a raw\-training driver\.

Table 5:Provenance map: manuscript claim or figure→\\toanalysis component→\\todata artifact\. Raw per\-run JSONs are released in the companion dataset; the public code repository ships aggregate JSONs, selected scripts, the coverage manifest, and the Lean target\. For rows whose full build component is not part of the lightweight public code release, the released aggregate artifact and numerical verifier provide the reviewer\-facing check\.