What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy

# What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy
Source: [https://arxiv.org/html/2606.18465](https://arxiv.org/html/2606.18465)
###### Abstract

Grokking is generalization that appears long after a network has fit its training data\. The grokking delay grows with the weight norm, and holding the norm fixed with a clamp reproduces a clean dose response, which has invited the reading that the weight norm sets the timescale\. We show that this reading is incomplete\. The clamp holds the scalar norm fixed by rescaling the weights, but rescaling also raises the logit scale, so the norm and the logit scale move together and the clamp does not separate them\. We separate them with a non\-trainable output temperature that divides the logits before the loss and is not part of the norm\. Holding the total weight norm fixed and varying only this temperature slides the delay across the full range produced by raising the norm, and matching the effective logit scale back to its baseline value recovers0\.830\.83\(95% CI\[0\.82,0\.85\]\[0\.82,0\.85\]\) and0\.890\.89\(\[0\.88,0\.91\]\[0\.88,0\.91\]\) of the norm\-induced delay at two moduli\. Across a grid of norms and temperatures the delay collapses onto the effective logit scale, which alone explains97%97\\%of its variance while the norm dose adds11–2%2\\%\. Under cross\-entropy, then, the weight\-norm effect on the delay runs through the logit scale and the softmax saturation it causes, not through the scalar norm itself\. The same intervention does nothing of the sort under mean\-squared error: there the effective logit scale is pinned near one and cannot be moved, the norm effect is about half the size, and it is carried by a separate route\. The mechanism is therefore loss\-dependent\. A control shows the temperature acts on the delayed\-generalization phase and not on memorization time, ruling out a gradient\-magnitude artifact, an independent float64 audit reaches the same softmax\-saturation channel by precision rather than by intervention, and a no\-LayerNorm transformer reproduces the effect\. Forking arms from one identical state shows the delay tracks the held norm value, not the rescaling operation, closing the clamp\-artifact objection\. Claims are confined to theℓ2\\ell\_\{2\}/weight\-decay regime on modular\-arithmetic networks\.

## 1Introduction

When a neural network keeps training after it has fit its training set, it usually stops improving\. Grokking is the striking exception: on certain algorithmic tasks, test accuracy stays at chance for many thousands of steps after the training loss has collapsed, then rises sharply to near perfect\(Poweret al\.,[2022](https://arxiv.org/html/2606.18465#bib.bib1)\)\. The phenomenon is robust and reproducible; the open question is no longer*whether*it happens but*what controls the delay*\.

One variable is repeatedly implicated: the weight norm\. Grokking coincides with weight\-norm decay\(Liuet al\.,[2023](https://arxiv.org/html/2606.18465#bib.bib2)\), the delay lengthens when the norm is held high, and a recent line treats norm minimization on the zero\-loss manifold as the organizing principle\(Musat,[2025](https://arxiv.org/html/2606.18465#bib.bib8); Boursieret al\.,[2025](https://arxiv.org/html/2606.18465#bib.bib9)\)\. If the norm is clamped to a chosen value throughout training, the grokking delay follows a clean exponential dose response in the held norm\. It is tempting to read this as the weight norm setting the timescale of grokking\.

We argue that this reading conflates two things the clamp changes together\. Holding‖W‖\\\|W\\\|fixed requires rescaling the weight matrices, and rescaling also raises the logits, which in cross\-entropy pushes the softmax toward saturation\. So when a higher held norm lengthens the delay, we cannot yet tell whether the scalar norm is the operative variable or whether it is the logit scale that the rescaling drags along\. The distinction matters: the first says grokking is governed by a geometric quantity, the second points at a function\-space quantity already implicated by the numerical\-stability account of grokking\(Prietoet al\.,[2025](https://arxiv.org/html/2606.18465#bib.bib6)\)\.

We separate the two with an intervention\. We hold the total weight norm fixed with the clamp and add a non\-trainable output temperatureτ\\tauthat divides the logits before the loss\. Becauseτ\\tauis not a weight, it is not part of‖W‖\\\|W\\\|; at a clamped norm it tunes the effective logit scale while leaving the norm fixed\. The result, across two moduli, is unambiguous under cross\-entropy: varyingτ\\taualone slides the grokking delay across the entire range produced by raising the norm, and the baseline and the norm\-raised runs lie on a single curve of delay against effective logit scale\. Matching the logit scale back to baseline recovers0\.830\.83and0\.890\.89of the norm\-induced delay\. The weight\-norm effect on the delay is, to that extent, the logit\-scale effect\.

The mechanism is loss\-dependent\. Under mean\-squared error the effective logit scale at grokking is pinned near one because the regression target fixes it,τ\\taucannot move it, and the norm effect, while still present, is about half the size and is not carried by the logit scale\. A control rules out the obvious artifact:τ\\tauleaves the memorization time essentially unchanged and acts almost entirely on the delayed\-generalization phase, so the effect is about grokking specifically and not about training speed\.

Our contributions:

- •A temperature\-mediation test that separates the scalar weight norm from the effective logit scale at a fixed clamped norm, and the finding that under cross\-entropy the norm\-induced grokking delay is recovered primarily \(∼\\sim0\.85, tight bootstrap CI, two moduli\) by restoring the logit scale\.
- •A data collapse: across a grid of norms and temperatures, the delay is a function of the effective logit scale alone \(R2=0\.97R^\{2\}=0\.97\), with the norm dose adding11–2%2\\%beyond it\.
- •A clean loss dissociation: the logit\-scale channel is active under cross\-entropy and absent under mean\-squared error, so the weight\-norm dependence of grokking is not a single mechanism\.
- •A memorization control showing the temperature acts on the delay and not on memorization, a float64 audit that reaches the same softmax\-saturation channel independently, and qualitative corroboration in a no\-LayerNorm transformer\.
- •A same\-state test that forks arms from one identical state and shows the delay tracks the held norm value, not the clamp’s rescaling operation, closing the rescaling\-artifact objection for this setting\.
- •Honest scope: the result localizes the proximal variable under cross\-entropy; it does not characterize the mean\-squared\-error route, and the quantitative estimates are for MLPs at two moduli\.

## 2Setup

Task and model\.We study modular addition, inputs\(a,b\)∈\{0,…,p−1\}2\(a,b\)\\in\\\{0,\\dots,p\-1\\\}^\{2\}and target\(a\+b\)modp\(a\+b\)\\bmod p, app\-way classification, forp∈\{43,59,67,97,113\}p\\in\\\{43,59,67,97,113\\\}\. The dataset is allp2p^\{2\}pairs; a fixed fractionα=0\.40\\alpha=0\.40is the training split, drawn per seed by a seeded permutation\. The model is a two\-layer MLP: each ofa,ba,bis embedded through a sharedE∈ℝp×dE\\in\\mathbb\{R\}^\{p\\times d\}\(d=128d=128\), the two embeddings are concatenated and passed through a linear layerW1∈ℝ2​d×HW\_\{1\}\\in\\mathbb\{R\}^\{2d\\times H\}\(H=256H=256\) with a GeLU nonlinearity, then a linear readoutW2∈ℝH×pW\_\{2\}\\in\\mathbb\{R\}^\{H\\times p\}\. Optimizer is AdamW \(β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999, learning rate10−310^\{\-3\}, weight decayλ=1\.0\\lambda=1\.0unless varied\), full\-batch unless noted,1212seeds per cell\. The weight norm is‖W‖=‖E‖F2\+‖W1‖F2\+‖W2‖F2\\\|W\\\|=\\sqrt\{\\\|E\\\|\_\{F\}^\{2\}\+\\\|W\_\{1\}\\\|\_\{F\}^\{2\}\+\\\|W\_\{2\}\\\|\_\{F\}^\{2\}\}, biases excluded\.

The clamp\.After each optimizer step fromt≥tintt\\geq t\_\{\\mathrm\{int\}\}, the weight matrices are rescaled by a single scalar so that‖W‖=ρ​wc\\\|W\\\|=\\rho\\,w\_\{c\}exactly, wherewcw\_\{c\}is the norm at grokking measured in a free control run andρ\\rhois the dose\. The clamp is engaged before memorization in every cell used here, so the intervention is active throughout the relevant dynamics\.

Effective logit scale\.The central variable of this paper needs a precise definition\. For a configuration with temperatureτ\\tau, the effective logits are the post\-τ\\taulogits the loss actually sees\. The effective logit scale is the mean, over allp2p^\{2\}input pairs and over seeds, of theL2L\_\{2\}norm of the per\-example effective logit vector, read at the grokking step \(the logged step nearest the medianTgrokT\_\{\\mathrm\{grok\}\}of that cell\)\. In the mediation analysis it is normalized by the baseline \(ρ=1\\rho=1,τ=1\\tau=1\) value so the axis is dimensionless and comparable across moduli\.

Temperatures and metrics\.The temperatureτ\\taudivides the logits before the softmax \(CE\) or the squared error \(MSE\);τ=1\\tau=1recovers the standard model andτ\\tauis held fixed within a run\. We log test accuracy \(grokking timeTgrokT\_\{\\mathrm\{grok\}\}is the first step with test accuracy≥0\.90\\geq 0\.90, median over seeds\), training accuracy \(memorization timeTmemT\_\{\\mathrm\{mem\}\},≥0\.99\\geq 0\.99\), the per\-group weight norms, the effective logit scale, and the softmax\-collapse rate ofPrietoet al\.\([2025](https://arxiv.org/html/2606.18465#bib.bib6)\)\.

## 3The fixed\-norm exponential law \(the dose response\)

Holding the norm fixed and sweeping the doseρ\\rhoyields a clean dose response between the held norm and the grokking time\. Across all five moduli the relationship is exponential,Tgrok∝exp⁡\(α​‖W‖\)T\_\{\\mathrm\{grok\}\}\\propto\\exp\(\\alpha\\\|W\\\|\), with a per\-modulus slopeα\\alpha\. We compare this against a power law and against a saddle\-node formT∝\(w0−‖W‖\)−1/2T\\propto\(w\_\{0\}\-\\\|W\\\|\)^\{\-1/2\}\(the signature a fold or critical\-slowing\-down bifurcation would produce\) by AIC\. The exponential is generally selected \(Table[1](https://arxiv.org/html/2606.18465#S3.T1)\): decisively at four of the five moduli \(Δ\\DeltaAIC4\.64\.6–6\.46\.4\), and weakly atp=43p=43\(Δ\\DeltaAIC0\.40\.4, essentially tied with the power\-law\), which we report rather than smooth over\. We read this as a robust empirical regularity of the fixed\-norm dose response, not a law derived from first principles, and we use it here only as the readout against which the interventions below are measured\. The slopes are fit on the softmax\-collapse\-free range \(ρ≤1\.15\\rho\\leq 1\.15; §[5](https://arxiv.org/html/2606.18465#S5)\)\.

This dose response is the phenomenon to be explained\. It establishes that raising the held norm lengthens the delay; it does not, on its own, say what the norm acts through\. That is the question of the next section\.

Table 1:Fixed\-norm exponential lawln⁡Tgrok=c\+α​‖W‖\\ln T\_\{\\mathrm\{grok\}\}=c\+\\alpha\\\|W\\\|per modulus, fit on the softmax\-collapse\-free range \(ρ≤1\.15\\rho\\leq 1\.15\), majority\-grok cells only\.Δ\\DeltaAIC is the exponential’s margin over the better of the power\-law and saddle\-node forms \(positive favors the exponential\)\.
## 4What the weight norm acts through

### 4\.1A temperature\-mediation test

The clamp holds‖W‖\\\|W\\\|fixed by rescaling the weights, but that rescaling raises the logits, and in cross\-entropy larger logits saturate the softmax\. So the dose response of §3 does not separate the scalar norm from the logit scale it sets\. We separate them with a non\-trainable output temperatureτ\\tauthat divides the logits before the loss and is not counted in‖W‖\\\|W\\\|\. At a clamped norm,τ\\tautunes the effective logit scale while leaving the norm fixed\.

The design is a three\-condition mediation\. Baseline \(ρ=1\.0\\rho=1\.0,τ=1\\tau=1\) gives delayT0T\_\{0\}; raising the norm \(ρ=1\.15\\rho=1\.15,τ=1\\tau=1\) givesT1T\_\{1\}; and atρ=1\.15\\rho=1\.15we sweepτ\\tauupward, which lowers the effective logit scale at fixed norm\. If the norm effect runs through the logit scale, settingτ\\tauso that the effective logit scale matches the baseline should recover the baseline delay\.

Under cross\-entropy this is what happens \(Fig\.[1](https://arxiv.org/html/2606.18465#S4.F1)a, Table[2](https://arxiv.org/html/2606.18465#S4.T2)\)\. The norm\-up delay is3\.4×3\.4\\timesthe baseline at both moduli\. Sweepingτ\\taufrom1\.01\.0to1\.71\.7slides the delay monotonically back down, from3\.4×3\.4\\timesto below baseline, and the cells trace a single curve of delay against effective logit scale on which the baseline and norm\-up points both lie\. The effective logit scale itself moves by about1\.6×1\.6\\timesacross the sweep\. Reading off the delay where the effective logit scale equals its baseline value gives a*logit\-scale recovery fraction*\(T1−T2\)/\(T1−T0\)\(T\_\{1\}\-T\_\{2\}\)/\(T\_\{1\}\-T\_\{0\}\)of0\.830\.83atp=59p=59\(95% bootstrap CI\[0\.82,0\.85\]\[0\.82,0\.85\]\) and0\.890\.89atp=97p=97\(\[0\.88,0\.91\]\[0\.88,0\.91\]\); the CI is a paired resample over the twelve seeds and reflects seed variability inTgrokT\_\{\\mathrm\{grok\}\}, with the logit curve held fixed\. We call it a recovery fraction rather than a mediation fraction deliberately: it is the share of the norm\-up delay that the temperature intervention recovers by restoring the effective logit scale, an intervention\-level quantity, not a regression\-based natural\-indirect\-effect estimate\. To that extent the weight\-norm effect on the delay is the logit\-scale effect: the norm matters because it sets the logit scale, which sets the softmax saturation\. This is the same channel the audit of §[5](https://arxiv.org/html/2606.18465#S5)probes, reached here by intervention rather than by precision\. The next subsection shows the relationship is not specific to this one norm level: across a grid of norms and temperatures, the delay collapses onto the effective logit scale\.

Table 2:Temperature mediation atρ=1\.15\\rho=1\.15\(medians over 12 seeds\)\. The logit\-scale recovery fraction is the share of the norm\-up delay recovered by matching the effective logit scale back to baseline; it is reported only for cross\-entropy, where the logit scale is a usable knob\.![Refer to caption](https://arxiv.org/html/2606.18465v1/x1.png)Figure 1:At a fixed clamped norm, varying only the output temperatureτ\\tau\.\(a\)Under cross\-entropy the cells trace one curve of delay against effective logit scale: the baseline \(star\) and the norm\-up point \(τ=1\\tau=1, top right\) lie on it, and increasingτ\\tauslides the delay back down toward baseline\. About8383–89%89\\%of the norm\-up delay is recovered by matching the logit scale \(two moduli\)\.\(b\)Under mean\-squared error the effective logit scale at grokking is pinned near baseline andτ\\taucannot move it; the residual norm effect \(vertical spread\) is not along the logit\-scale axis\. Both moduli overlaid; axes normalized by the baseline cell\.
### 4\.2A data collapse across the norm–temperature grid

The single\-norm recovery fraction asks what happens at one norm level\. A stronger question is whether the delay is a function of the effective logit scale*regardless*of how that scale was reached, whether by raising the norm or by lowering the temperature\. We test this on a grid of four norm doses \(ρ∈\{1\.00,1\.05,1\.10,1\.15\}\\rho\\in\\\{1\.00,1\.05,1\.10,1\.15\\\}\) crossed with three temperatures \(τ∈\{1\.0,1\.3,1\.7\}\\tau\\in\\\{1\.0,1\.3,1\.7\\\}\), twelve cells per modulus, all grokking 12/12\. Figure[2](https://arxiv.org/html/2606.18465#S4.F2)plotsTgrokT\_\{\\mathrm\{grok\}\}against the effective logit scale at grokking, colored byρ\\rho: the cells fall on a single log\-linear curve, and cells of differentρ\\rhoat the same effective logit scale have the same delay\.

We quantify the collapse by regression\. A least\-squares fit ofln⁡Tgrok\\ln T\_\{\\mathrm\{grok\}\}on the effective logit scale explainsR2=0\.97R^\{2\}=0\.97of the variance at both moduli; addingρ\\rhoas a second predictor raisesR2R^\{2\}by only0\.020\.02\(p=59p=59\) and0\.010\.01\(p=97p=97\), so once the effective logit scale is known the norm dose carries almost no additional information about the delay \(Table[3](https://arxiv.org/html/2606.18465#S4.T3)\)\. The delay is, to this precision, a function of the effective logit scale alone\. We are careful about the causal reading: thexx\-axis is the*realized*logit scale at grokking and is therefore endogenous, so the collapse is strong organizing evidence, not an intervention on its own; the causal weight rests on the temperature manipulation of §[4\.1](https://arxiv.org/html/2606.18465#S4.SS1), which sets the logit scale directly at fixed norm\. The two are complementary: the intervention shows the channel is causal, and the collapse shows it accounts for essentially all of the norm dependence\.

![Refer to caption](https://arxiv.org/html/2606.18465v1/x2.png)Figure 2:Data collapse across theρ×τ\\rho\\times\\taugrid \(12 cells per modulus, all grok 12/12\)\.TgrokT\_\{\\mathrm\{grok\}\}against the effective logit scale at grokking, colored by the norm doseρ\\rho\. Cells of differentρ\\rhobut matched effective logit scale share the same delay, so the delay collapses onto the logit scale; the norm dose adds11–2%2\\%of explained variance beyond it\.Table 3:Collapse regression across theρ×τ\\rho\\times\\taugrid\.R2R^\{2\}ofln⁡Tgrok\\ln T\_\{\\mathrm\{grok\}\}against the effective logit scale alone, againstρ\\rhoalone, and against both;Δ​R2​\(ρ\|logit\)\\Delta R^\{2\}\(\\rho\\,\|\\,\\text\{logit\}\)is the varianceρ\\rhoadds beyond the logit scale \(near zero indicates collapse\)\.
### 4\.3The mechanism is loss\-dependent

Mean\-squared error dissociates cleanly \(Fig\.[1](https://arxiv.org/html/2606.18465#S4.F1)b\)\. There the effective logit scale at grokking sits near one and barely moves withτ\\tau, because the regression target fixes the output scale, so the temperature cannot act on a logit\-scale channel: acrossτ∈\[1\.0,1\.7\]\\tau\\in\[1\.0,1\.7\]the effective logit scale moves about1\.1×1\.1\\times, against1\.6×1\.6\\timesunder cross\-entropy\. The norm\-up effect under MSE is real but about half the size \(1\.71\.7–1\.9×1\.9\\timesversus3\.4×3\.4\\times\) and is carried by a route that does not pass through the logit scale\. So the weight\-norm dependence of the delay is not one mechanism\. Under cross\-entropy it is logit\-scale saturation; under mean\-squared error it is something else\.

We are deliberately conservative about the MSE residual\. The temperature still shortens the MSE delay somewhat, but a temperature that divides the logits also changes the scale of the squared\-error target, so we do not read this residual as a clean second channel\. We report MSE only to establish the dissociation that the logit\-scale channel is absent there; we do not assign the residual norm effect to a specific mechanism\. This connects to an asymmetry noted byPrietoet al\.\([2025](https://arxiv.org/html/2606.18465#bib.bib6)\), who observed that grokking under softmax cross\-entropy behaves differently from the MSE setting ofLiuet al\.\([2023](https://arxiv.org/html/2606.18465#bib.bib2)\), with the softmax\-collapse rate approaching one under cross\-entropy; our intervention gives that asymmetry a mechanism on the cross\-entropy side\.

### 4\.4The effect is on the delay, not on memorization

A temperature that divides the logits changes both the softmax saturation and the gradient magnitude, so a skeptic can ask whetherτ\\tausimply speeds up training in general\. It does not, under cross\-entropy\. Splitting the per\-seed times \(Fig\.[3](https://arxiv.org/html/2606.18465#S4.F3)\), the memorization timeTmemT\_\{\\mathrm\{mem\}\}is essentially flat acrossτ\\tau\(175→200175\\to 200steps atp=59p=59,200→250200\\to 250atp=97p=97\), while the delayTgrok−TmemT\_\{\\mathrm\{grok\}\}\-T\_\{\\mathrm\{mem\}\}carries the entireτ\\taueffect \(100% and 101% of the change inTgrokT\_\{\\mathrm\{grok\}\}\)\. The temperature acts on the delayed\-generalization phase, not on how fast the network fits the training set, so the gradient\-magnitude reading does not explain it\. \(Under MSE the split is less clean: atp=97p=97about two thirds of theτ\\taueffect falls onTmemT\_\{\\mathrm\{mem\}\}, a further reason we do not interpret the MSE residual\.\)

![Refer to caption](https://arxiv.org/html/2606.18465v1/x3.png)Figure 3:Under cross\-entropy, the temperature changes the delayed\-generalization phase, not memorization\.TmemT\_\{\\mathrm\{mem\}\}\(bottom\) is flat acrossτ\\tauwhileTgrokT\_\{\\mathrm\{grok\}\}and the delayTgrok−TmemT\_\{\\mathrm\{grok\}\}\-T\_\{\\mathrm\{mem\}\}fall together\.yy\-axis is log scale\.
### 4\.5Corroboration across allocation and architecture

A second intervention corroborates that the scalar norm is not the sole driver\. Holding the total norm fixed and shifting mass toward the readout layer \(raising‖W2‖\\\|W\_\{2\}\\\|at the expense of‖E‖\\\|E\\\|\) changes the delay by up to2\.2×2\.2\\timesatp=59p=59and1\.7×1\.7\\timesatp=97p=97\. We treat this as supporting rather than primary, because reallocating mass also changes embedding capacity, so it is not a clean logit\-scale move; the temperature test is the clean manipulation\.

The mechanism also reproduces in a different architecture\. We ran the temperature test on a no\-LayerNorm transformer atp=59p=59: raising the held norm lengthens the delay, and at the raised norm, increasing the temperature shortens the delay monotonically by4\.7×4\.7\\times\(16,100→3,50016\{,\}100\\to 3\{,\}500steps\), confirming that a variable outside the norm controls the delay under cross\-entropy in a transformer as it does in the MLP\. We report this as qualitative corroboration and deliberately do not quote a recovery fraction for the transformer: its logits grow far larger and more erratically than the MLP’s\. The highest\-stress cell shows the uncontrolled logit growth ofPrietoet al\.\([2025](https://arxiv.org/html/2606.18465#bib.bib6)\), with the logit scale climbing past10001000after grokking, so the effective logit scale at grokking is a noisier proxy there, and matching it back to baseline would require extrapolation\. The direction and size of the effect transfer; the precise mediation estimate is MLP\-specific\. That the transformer’s logits are harder to hold down is itself consistent with the saturation account\.

We are also explicit about what the temperature test does not do on its own\. It shows that, at a fixed clamped norm, a variable outside the norm controls the delay, which localizes the proximal variable under cross\-entropy to the logit scale, and we claim the logit scale is the proximal mediator of the norm effect\. The remaining question is whether the clamp’s rescaling operation, rather than the held value, drives the delay; the next subsection settles it\.

### 4\.6A same\-state test of the clamp

The clamp sets the norm by rescaling the weights, so one objection survives the temperature test: perhaps the rescaling operation, applied every step, distorts the trajectory, and the delay reflects the operation rather than the held value\. We separate the two by forking four arms from one identical state\. We train freely to a fork step after memorization and before grokking, then continue from the saved state under four conditions atρ=1\.15\\rho=1\.15: free \(no clamp\), clamp at the forked normN0N\_\{0\}\(the operation is active but the value is unchanged\), clamp atρ​N0\\rho N\_\{0\}, and clamp atN0/ρN\_\{0\}/\\rho\.

From the identical state, the delay rises monotonically with the held norm value, in all twelve seeds at both moduli \(Fig\.[4](https://arxiv.org/html/2606.18465#S4.F4)\)\. The decisive contrast is clamp\-at\-N0N\_\{0\}against clamp\-at\-ρ​N0\\rho N\_\{0\}\. Both apply the same clamp operation from the same state and differ only in the held value, and the raised arm takes3\.36×3\.36\\timeslonger atp=59p=59\(95% CI\[3\.27,3\.43\]\[3\.27,3\.43\]\) and3\.27×3\.27\\timesatp=97p=97\(\[3\.21,3\.38\]\[3\.21,3\.38\]\)\. Since the operation is identical in the two arms, the difference is not an artifact of the operation: the delay follows the held value\. The ordering across lower, hold, and raise also rules out a one\-time rescaling jump, which would perturb the up and down arms the same way instead of ordering them by value\. The free arm groks faster than clamp\-at\-N0N\_\{0\}because its norm decays after the fork, which is the norm decay that the generalizing\-shell account associates with grokking\(Liuet al\.,[2023](https://arxiv.org/html/2606.18465#bib.bib2)\)\. This closes the rescaling\-artifact objection for the cross\-entropy MLP setting\. Training full\-batch leaves the dose response intact, so the effect is not a sampling\-noise phenomenon either\.

![Refer to caption](https://arxiv.org/html/2606.18465v1/x4.png)Figure 4:Four arms forked from one identical post\-memorization state \(t=800t=800\)\. The grokking delay \(from the fork\) rises monotonically with the held norm value across all twelve seeds\. The labelled contrast, clamp\-at\-N0N\_\{0\}versus clamp\-at\-ρ​N0\\rho N\_\{0\}, applies the identical clamp operation and differs only in the held value \(3\.33\.3–3\.4×3\.4\\times\), so the delay tracks the value, not the rescaling operation\. Free \(dotted\) groks faster because its norm decays after the fork\.

## 5A softmax\-collapse audit

Because the cross\-entropy clamp operates at large logits, the regime in which softmax collapse can occur\(Prietoet al\.,[2025](https://arxiv.org/html/2606.18465#bib.bib6)\), we audit the law for floating\-point artifacts\. We re\-run the two highest\-dose cross\-entropy cells atp=59p=59in float64, holding the norm at the identical absolute values used in float32, and record the softmax\-collapse rate \(Table[4](https://arxiv.org/html/2606.18465#S5.T4)\)\. Atρ=1\.15\\rho=1\.15the collapse rate is zero in both precisions and the grokking times agree, so the law is precision\-robust there\. Atρ=1\.25\\rho=1\.25the float32 run shows a 31% collapse rate and groks at 25k steps, whereas the float64 run does not grok within 60k, so the float32 collapse accelerates a spurious transition at extreme norm\. We therefore fit the law on the collapse\-free range and exclude the affected cell\. This audit is convergent with §[4\.1](https://arxiv.org/html/2606.18465#S4.SS1): softmax collapse is the extreme of the same logit\-saturation channel that the temperature test isolates, reached by precision without any intervention\.

Table 4:Softmax\-collapse audit atp=59p=59: the two highest\-dose cross\-entropy cells re\-run in float64 at held norms identical to float32\.TT= median grok step; sc = softmax\-collapse rate\.
## 6Related work

Weight norm and norm minimization\.The weight norm has long been central to grokking: Omnigrok ties grokking to the norm decaying into a generalizing shell\(Liuet al\.,[2023](https://arxiv.org/html/2606.18465#bib.bib2)\), and a recent line characterizes the late dynamics as norm minimization on the zero\-loss manifold\(Musat,[2025](https://arxiv.org/html/2606.18465#bib.bib8); Boursieret al\.,[2025](https://arxiv.org/html/2606.18465#bib.bib9)\), including beyond the Euclidean norm\(Tikeng Notsawoet al\.,[2025](https://arxiv.org/html/2606.18465#bib.bib10)\)\. These accounts concern the*endpoint*\(which configuration the optimizer selects once training loss is zero\)\. Our question is about the*timescale*and the proximal variable that sets it; we find that under cross\-entropy the norm acts on the timescale through the logit scale, so the two are complementary rather than competing\.

Logit growth, softmax saturation, and optimization starvation\.The account closest to our result is the numerical\-stability view ofPrietoet al\.\([2025](https://arxiv.org/html/2606.18465#bib.bib6)\): cross\-entropy training without regularization drives uncontrolled logit growth \(“naive loss minimization”\), whose extreme is softmax collapse, where floating\-point errors zero the gradient and stall learning\. Our temperature test isolates the upstream of that channel, the logit scale, and shows it mediates most of the norm\-induced delay under cross\-entropy, with the collapse ofPrietoet al\.\([2025](https://arxiv.org/html/2606.18465#bib.bib6)\)as its endpoint \(§[5](https://arxiv.org/html/2606.18465#S5)\)\. Independently,Becket al\.\([2026](https://arxiv.org/html/2606.18465#bib.bib18)\)show that regularizing logits directly biases linear classifiers toward clustered logits and can induce grokking, further evidence that logit space, not weight norm alone, is the operative arena under cross\-entropy\. The intervention itself, dividing logits by a fixed temperature, is the temperature scaling ofGuoet al\.\([2017](https://arxiv.org/html/2606.18465#bib.bib26)\); we use it not for calibration but as a handle on the effective logit scale at fixed norm\.

Phase transitions and scaling\.A parallel line characterizes grokking as a phase transition along model or data axes: a first\-order transition in two\-layer networks\(Rubinet al\.,[2024](https://arxiv.org/html/2606.18465#bib.bib14)\), an information\-theoretic transition\(Clauwet al\.,[2024](https://arxiv.org/html/2606.18465#bib.bib15)\), Ising\-style and local\-rule descriptions\(Hutchison and Yevick,[2025](https://arxiv.org/html/2606.18465#bib.bib17); Žunkovič and Ilievski,[2024](https://arxiv.org/html/2606.18465#bib.bib13)\), and finite\-size or dimensional scaling\(Biet al\.,[2026](https://arxiv.org/html/2606.18465#bib.bib22); Wang,[2026b](https://arxiv.org/html/2606.18465#bib.bib23),[a](https://arxiv.org/html/2606.18465#bib.bib24)\)\. These concern the nature of the transition and its scaling with model or data size, a different axis from our intervention\-level question of which variable gates the delay at fixed data and model\.

Other proposed drivers\.Grokking has also been attributed to noise\-assisted escape and slingshot dynamics\(Thilaket al\.,[2022](https://arxiv.org/html/2606.18465#bib.bib11); Lopatinet al\.,[2025](https://arxiv.org/html/2606.18465#bib.bib12)\), to circuit\-efficiency reallocation\(Nandaet al\.,[2023](https://arxiv.org/html/2606.18465#bib.bib3); Varmaet al\.,[2023](https://arxiv.org/html/2606.18465#bib.bib4)\), and to a lazy\-to\-rich transition\(Kumaret al\.,[2024](https://arxiv.org/html/2606.18465#bib.bib5); Lyu and Li,[2020](https://arxiv.org/html/2606.18465#bib.bib7)\)\. Our full\-batch result places the delay outside the noise\-activated family for this setting, and our structure measure is the Fourier progress measure ofNandaet al\.\([2023](https://arxiv.org/html/2606.18465#bib.bib3)\)\.

Most of this work asks what*produces*grokking and answers with a mechanism that suffices\. Our question is narrower: with the weight norm held at a chosen value, which mechanism still moves the delay? The temperature handle lets us put one candidate, the logit scale, in and take it out directly, so the contribution here is not another mechanism but a way to decide, under cross\-entropy, what the norm acts through\.

## 7Limitations

The claims are confined to theℓ2\\ell\_\{2\}/weight\-decay regime on modular\-arithmetic networks\. The quantitative results \(the recovery fraction and the data collapse\) are established on MLPs at two moduli; a no\-LayerNorm transformer reproduces the effect in direction and size but with noisier logit dynamics, so we do not extend the precise recovery estimate to it, and we have not tested larger\-scale settings or other task families \(sparse parity, group tasks beyond modular arithmetic\)\. The same\-state test \(§[4\.6](https://arxiv.org/html/2606.18465#S4.SS6)\) addresses the clamp\-rescaling confound for the cross\-entropy MLP setting, but the matched contrast is at one fork point and oneρ\\rho; we have not mapped the full fork\-time dependence\. The data collapse uses the realized logit scale at grokking, which is endogenous, so it is organizing evidence and the causal weight rests on the temperature intervention\. The mean\-squared\-error route is identified only negatively: it does not pass through the logit scale, and we do not characterize it further; the residual temperature effect under MSE is partly a memorization\-speed effect and is not interpreted\. The recovery\-fraction CI reflects seed variability inTgrokT\_\{\\mathrm\{grok\}\}with the logit curve held fixed, and does not include interpolation uncertainty on the logit axis\. Two directions follow directly: characterizing the mean\-squared\-error route, which is left here as not\-the\-logit\-scale, and testing whether the logit\-scale mediation holds at transformer scale, where the logit dynamics we saw are already noisier\.

## 8Conclusion

The weight\-norm dependence of the grokking delay is not, under cross\-entropy, an effect of the scalar norm\. Holding the norm fixed and varying only an output temperature slides the delay across the full norm\-up range, and matching the effective logit scale back to baseline recovers about85%85\\%of the delay at two moduli, with the effect falling on the delayed\-generalization phase and not on memorization\. Across a grid of norms and temperatures the delay collapses onto the effective logit scale, which alone explains97%97\\%of its variance\. The weight norm is an upstream handle; the proximal variable is the logit scale, and the channel is the softmax saturation that the numerical\-stability account of grokking also points to\. The mechanism is loss\-dependent: mean\-squared error pins the logit scale and the norm effect runs elsewhere\. Forking arms from one identical state confirms that the delay follows the held norm value and not the rescaling operation, so the dose response is not a clamp artifact\. Beyond grokking, the methodological point is that a causal claim about a scalar quantity such as the weight norm has to be separated from the function\-space variables that rescaling the weights changes along with it; here, once that separation is made, much of what looked like a weight\-norm law is a logit\-scale law\.

Appendix

## Appendix AModel, task, and exact definitions

Task\.Modular addition: inputs\(a,b\)∈\{0,…,p−1\}2\(a,b\)\\in\\\{0,\\dots,p\-1\\\}^\{2\}, target\(a\+b\)modp\(a\+b\)\\bmod p, app\-way classification, forp∈\{43,59,67,97,113\}p\\in\\\{43,59,67,97,113\\\}\(the dose\-response law\) andp∈\{59,97\}p\\in\\\{59,97\\\}\(the interventions\)\. The dataset is allp2p^\{2\}pairs; a fixed fractionα=0\.40\\alpha=0\.40is the training split, drawn per seed by a seeded permutation, the remainder held out for test\.

Model\.A two\-layer MLP\. Each ofa,ba,bis mapped through a shared embeddingE∈ℝp×dE\\in\\mathbb\{R\}^\{p\\times d\}\(d=128d=128\); the two embeddings are concatenated and passed through a linear layerW1∈ℝ2​d×HW\_\{1\}\\in\\mathbb\{R\}^\{2d\\times H\}\(H=256H=256\) with biasb1b\_\{1\}and a GeLU nonlinearity, then a linear readoutW2∈ℝH×pW\_\{2\}\\in\\mathbb\{R\}^\{H\\times p\}with biasb2b\_\{2\}\. Weight matrices are initialized𝒩​\(0,1\)/fan​\-​in\\mathcal\{N\}\(0,1\)/\\sqrt\{\\mathrm\{fan\\text\{\-\}in\}\}; biases initialized to zero\.

Optimizer\.AdamW \(β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999,ϵ=10−8\\epsilon=10^\{\-8\}\), learning rate10−310^\{\-3\}, weight decayλ=1\.0\\lambda=1\.0unless varied\. Full\-batch gradients unless a minibatch noise axis is specified\. Weight decay and the clamp act onE,W1,W2E,W\_\{1\},W\_\{2\}; biases are unregularized and excluded from‖W‖\\\|W\\\|\. Twelve seeds per cell\.

Weight norm\.‖W‖=‖E‖F2\+‖W1‖F2\+‖W2‖F2\\\|W\\\|=\\sqrt\{\\\|E\\\|\_\{F\}^\{2\}\+\\\|W\_\{1\}\\\|\_\{F\}^\{2\}\+\\\|W\_\{2\}\\\|\_\{F\}^\{2\}\}\. This is the quantity the clamp holds and thexx\-axis of the dose response\.

The clamp\.After each optimizer step witht≥tintt\\geq t\_\{\\mathrm\{int\}\}\(tint=500t\_\{\\mathrm\{int\}\}=500for the cross\-entropy runs,5050for the mean\-squared\-error runs, chosen below memorization so the intervention is active throughout the relevant dynamics\), the weight matrices are rescaled by a single scalar so that‖W‖=ρ​wc\\\|W\\\|=\\rho\\,w\_\{c\}exactly, wherewcw\_\{c\}is the median norm at grokking in a free control run andρ\\rhois the dose\. The control calibrateswc=54\.49w\_\{c\}=54\.49\(p=59p=59\) and65\.8565\.85\(p=97p=97\) under cross\-entropy\.

Temperature\.A non\-trainable scalarτ\\taudivides the logits before the loss;τ=1\\tau=1recovers the standard model\.τ\\tauis not a parameter and is not part of‖W‖\\\|W\\\|, so at a clamped norm it changes the effective logit scale while leaving‖W‖\\\|W\\\|fixed\.

Effective logit scale\.For a configuration with temperatureτ\\tau, the effective logits are the post\-τ\\taulogits the loss sees\. The effective logit scale is1S​∑s1p2​∑\(a,b\)‖zs,\(a,b\)/τ‖2\\frac\{1\}\{S\}\\sum\_\{s\}\\frac\{1\}\{p^\{2\}\}\\sum\_\{\(a,b\)\}\\\|z\_\{s,\(a,b\)\}/\\tau\\\|\_\{2\}, the mean over allp2p^\{2\}input pairs and over theS=12S=12seeds of theL2L\_\{2\}norm of the per\-example effective logit vectorz/τz/\\tau, read at the grokking step \(the logged step nearest the medianTgrokT\_\{\\mathrm\{grok\}\}of the cell\)\. In the mediation analysis it is normalized by the baseline \(ρ=1\\rho=1,τ=1\\tau=1\) value\.

Times\.TmemT\_\{\\mathrm\{mem\}\}is the first step with train accuracy≥0\.99\\geq 0\.99;TgrokT\_\{\\mathrm\{grok\}\}is the first step with test accuracy≥0\.90\\geq 0\.90\. Both are per seed and aggregated by the median over seeds\. A cell is counted only if a majority of seeds grok\.

## Appendix BModel selection for the dose\-response law

For a dose response ofnncells with held norms‖W‖i\\\|W\\\|\_\{i\}and log grokking timesℓi=ln⁡Ti\\ell\_\{i\}=\\ln T\_\{i\}, each candidate form is fit by least squares and scored byAIC=n​ln⁡\(RSS/n\)\+2​k\\mathrm\{AIC\}=n\\ln\(\\mathrm\{RSS\}/n\)\+2kwithkkthe parameter count\. The compared forms are the exponentialℓ=c\+α​‖W‖\\ell=c\+\\alpha\\\|W\\\|\(k=2k=2\), a power law with free offsetℓ=c\+β​ln⁡\(‖W‖−w0\)\\ell=c\+\\beta\\ln\(\\\|W\\\|\-w\_\{0\}\)\(k=3k=3\), and a saddle\-node formℓ=c−12​ln⁡\(w0−‖W‖\)\\ell=c\-\\tfrac\{1\}\{2\}\\ln\(w\_\{0\}\-\\\|W\\\|\),w0\>max⁡‖W‖w\_\{0\}\>\\max\\\|W\\\|\(k=2k=2\), the critical\-slowing\-down signature a fold bifurcation would produce\. Only majority\-grokked cells on the softmax\-collapse\-free range \(ρ≤1\.15\\rho\\leq 1\.15\) enter the fit\.Δ\\DeltaAIC in Table[1](https://arxiv.org/html/2606.18465#S3.T1)is the exponential’s margin over the better of the power\-law and saddle\-node forms; positive favors the exponential\.

## Appendix CMediation: per\-cell data and the recovery fraction

Tables[5](https://arxiv.org/html/2606.18465#A3.T5)–[6](https://arxiv.org/html/2606.18465#A3.T6)give every cell behind §[4\.1](https://arxiv.org/html/2606.18465#S4.SS1)\(medians over 12 seeds; all cells grok 12/12\)\. Each block holds the baseline \(ρ=1\.0\\rho=1\.0,τ=1\\tau=1\) and the norm\-up sweep \(ρ=1\.15\\rho=1\.15,τ\\tauincreasing, which lowers the effective logit scale at fixed norm\)\. The logit\-scale recovery fraction is\(T1−T2\)/\(T1−T0\)\(T\_\{1\}\-T\_\{2\}\)/\(T\_\{1\}\-T\_\{0\}\), whereT0T\_\{0\}is the baseline delay,T1T\_\{1\}the norm\-up delay atτ=1\\tau=1, andT2T\_\{2\}the delay read where the effective logit scale, interpolated over theτ\\tausweep, equals the baseline value\. The 95% CI is a paired bootstrap over the twelve seeds \(4000 resamples\): each resample recomputes the per\-cell medianTgrokT\_\{\\mathrm\{grok\}\}and the interpolatedT2T\_\{2\}, with the effective\-logit curve held at its full\-sample value; the interval is the 2\.5–97\.5 percentile range\. It therefore reflects seed variability inTgrokT\_\{\\mathrm\{grok\}\}, the dominant source, and not interpolation uncertainty on the logit axis\.

Table 5:Temperature sweep, cross\-entropy\.LL= effective logit scale at grok\. All cells grok 12/12\.Table 6:Temperature sweep, mean\-squared error\. The effective logit scaleLLis pinned near baseline and does not move withτ\\tau, so no recovery fraction is defined\. All cells grok 12/12\.Under cross\-entropy the recovery fraction is0\.830\.83atp=59p=59\(CI\[0\.82,0\.85\]\[0\.82,0\.85\]\) and0\.890\.89atp=97p=97\(CI\[0\.88,0\.91\]\[0\.88,0\.91\]\)\. The memorization control reads directly from these tables:TmemT\_\{\\mathrm\{mem\}\}moves by2525–5050steps across the fullτ\\tausweep while the delay moves by thousands, so the entireτ\\taueffect is on the delay\. Under mean\-squared error the effective logit scale moves only∼1\.1×\\sim 1\.1\\timesacrossτ\\tau\(against∼1\.6×\\sim 1\.6\\timesunder cross\-entropy\), so no logit\-scale channel is available forτ\\tauto act on; atp=97p=97roughly two thirds of the smallτ\\taueffect falls onTmemT\_\{\\mathrm\{mem\}\}, which is a further reason the mean\-squared\-error residual is not interpreted\.

Collapse grid\.Table[7](https://arxiv.org/html/2606.18465#A3.T7)lists the twelveρ×τ\\rho\\times\\taucells per modulus behind the collapse of §[4\.2](https://arxiv.org/html/2606.18465#S4.SS2)\(all grok 12/12\)\. The regression statistics are in Table[3](https://arxiv.org/html/2606.18465#S4.T3)\.

Table 7:Theρ×τ\\rho\\times\\taugrid behind Fig\.[2](https://arxiv.org/html/2606.18465#S4.F2):TgrokT\_\{\\mathrm\{grok\}\}\(median over 12 seeds\) and the effective logit scaleLLat grok, cross\-entropy\. Cells of differentρ\\rhobut matchedLLshare the delay\.No\-LayerNorm transformer\.Table[8](https://arxiv.org/html/2606.18465#A3.T8)gives the transformer temperature sweep atp=59p=59\(all grok 12/12\)\. At the raised norm, increasingτ\\taushortens the delay monotonically by4\.7×4\.7\\times\. The effective logit scale at grok is large and non\-monotonic inτ\\tau\(the highest\-stressτ=1\.0\\tau=1\.0cell exhibits post\-grokking logit growth past10001000\), so we report this as qualitative corroboration and do not compute a recovery fraction for the transformer\.

Table 8:No\-LayerNorm transformer, cross\-entropy,p=59p=59\(medians over 12 seeds\)\. The delay responds to the held norm and toτ\\tauas in the MLP; the effective logit scaleLLis larger and noisier\.
## Appendix DSoftmax\-collapse audit and the layer\-allocation control

Softmax\-collapse audit\.The audit of §[5](https://arxiv.org/html/2606.18465#S5)re\-runs the two highest\-dose cross\-entropy cells atp=59p=59in float64, at held norms identical to the float32 run, and records the softmax\-collapse rate \(the fraction of training points whose correct\-class softmax probability reaches one in working precision\)\. Details in Table[4](https://arxiv.org/html/2606.18465#S5.T4)\.

Layer allocation \(Arm A\)\.As a second intervention we hold the total norm fixed atρ=1\.0\\rho=1\.0\(‖W‖=wc\\\|W\\\|=w\_\{c\}\) and shift mass between the embedding and the readout by a parameterγ\\gamma: the readout share of theEE–W2W\_\{2\}power budget is scaled by\(1\+γ\)\(1\+\\gamma\)\(clipped\), withW1W\_\{1\}held at its baseline share, soγ\>0\\gamma\>0moves mass intoW2W\_\{2\}at fixed total norm\. Table[9](https://arxiv.org/html/2606.18465#A4.T9)reports the result\. The grokking time changes by2\.2×2\.2\\times\(p=59p=59\) and1\.7×1\.7\\times\(p=97p=97\) across the sweep at fixed total norm, which corroborates that the scalar norm is not the sole driver\. We treat it as supporting, not primary for two reasons: the effective logit scale at grok barely moves acrossγ\\gamma\(it is the*total*readout magnitude, not its share, that the temperature test varies cleanly\), so the change is driven by the embedding\-capacity shift rather than by a clean logit\-scale move; and atγ=\+0\.6\\gamma=\+0\.6,p=59p=59, three of twelve seeds fail to grok within budget\. The temperature test of §[4\.1](https://arxiv.org/html/2606.18465#S4.SS1)is the clean manipulation\.

Table 9:Layer allocation at fixed total norm \(ρ=1\.0\\rho=1\.0\)\.γ\>0\\gamma\>0shifts mass into the readoutW2W\_\{2\}\.LL= effective logit scale at grok\. Medians over 12 seeds unless the grok count notes otherwise\.†9 of 12 seeds grok within budget; the median is over the grokked seeds\.

## Appendix ESame\-state arms

Table[10](https://arxiv.org/html/2606.18465#A5.T10)gives the four arms forked from one identical state att=800t=800\(§[4\.6](https://arxiv.org/html/2606.18465#S4.SS6)\), all grokking 12/12\. The delay is measured from the fork\. The recovery contrast is clamp\-at\-N0N\_\{0\}against clamp\-at\-ρ​N0\\rho N\_\{0\}, both applying the same clamp operation from the same state\.

Table 10:Same\-state arms, cross\-entropy, fork att=800t=800\(median delay from fork over 12 seeds\)\. The raised/same ratio is a paired bootstrap over the 12 seeds\. The monotonic ordering holds in all 12 seeds at both moduli\.
## Code and data availability

The clamp and temperature\-mediation runner, the per\-cell metric outputs used for every fit, table, and figure, the AIC model\-selection script, and the float64 softmax\-collapse audit are available at[https://github\.com/ClevixLab/grokking\-logit\-scale](https://github.com/ClevixLab/grokking-logit-scale), with seeds fixed so that every number in Tables[1](https://arxiv.org/html/2606.18465#S3.T1)–[4](https://arxiv.org/html/2606.18465#S5.T4)and the figures can be regenerated\.

## References

- A\. Beck, Y\. Bar\-Sinai, and N\. Levi \(2026\)The implicit bias of logit regularization\.arXiv preprint arXiv:2602\.12039\.Cited by:[§6](https://arxiv.org/html/2606.18465#S6.p2.1)\.
- Y\. Bi, C\. Zhang, Q\. Wang, and V\. D\. Calhoun \(2026\)Grokking as a falsifiable finite\-size transition\.arXiv preprint arXiv:2603\.24746\.Cited by:[§6](https://arxiv.org/html/2606.18465#S6.p3.1)\.
- E\. Boursier, S\. Pesme, and R\. Dragomir \(2025\)A theoretical framework for grokking: interpolation followed by riemannian norm minimisation\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2505\.20172Cited by:[§1](https://arxiv.org/html/2606.18465#S1.p2.1),[§6](https://arxiv.org/html/2606.18465#S6.p1.1)\.
- K\. Clauw, S\. Stramaglia, and D\. Marinazzo \(2024\)Information\-theoretic progress measures reveal grokking is an emergent phase transition\.arXiv preprint arXiv:2408\.08944\.Cited by:[§6](https://arxiv.org/html/2606.18465#S6.p3.1)\.
- C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger \(2017\)On calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning \(ICML\),PMLR, Vol\.70,pp\. 1321–1330\.Cited by:[§6](https://arxiv.org/html/2606.18465#S6.p2.1)\.
- K\. Hutchison and D\. Yevick \(2025\)Grokking in the ising model\.arXiv preprint arXiv:2510\.25966\.Cited by:[§6](https://arxiv.org/html/2606.18465#S6.p3.1)\.
- T\. Kumar, B\. Bordelon, S\. J\. Gershman, and C\. Pehlevan \(2024\)Grokking as the transition from lazy to rich training dynamics\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2310\.06110Cited by:[§6](https://arxiv.org/html/2606.18465#S6.p4.1)\.
- Z\. Liu, E\. J\. Michaud, and M\. Tegmark \(2023\)Omnigrok: grokking beyond algorithmic data\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2210\.01117Cited by:[§1](https://arxiv.org/html/2606.18465#S1.p2.1),[§4\.3](https://arxiv.org/html/2606.18465#S4.SS3.p2.1),[§4\.6](https://arxiv.org/html/2606.18465#S4.SS6.p2.9),[§6](https://arxiv.org/html/2606.18465#S6.p1.1)\.
- I\. A\. Lopatin, S\. V\. Kozyrev, and A\. N\. Pechen \(2025\)Predator\-prey model: driven hunt for accelerated grokking\.arXiv preprint arXiv:2509\.10562\.Cited by:[§6](https://arxiv.org/html/2606.18465#S6.p4.1)\.
- K\. Lyu and J\. Li \(2020\)Gradient descent maximizes the margin of homogeneous neural networks\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:1906\.05890Cited by:[§6](https://arxiv.org/html/2606.18465#S6.p4.1)\.
- T\. Musat \(2025\)The geometry of grokking: norm minimization on the zero\-loss manifold\.arXiv preprint arXiv:2511\.01938\.Cited by:[§1](https://arxiv.org/html/2606.18465#S1.p2.1),[§6](https://arxiv.org/html/2606.18465#S6.p1.1)\.
- N\. Nanda, L\. Chan, T\. Lieberum, J\. Smith, and J\. Steinhardt \(2023\)Progress measures for grokking via mechanistic interpretability\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2301\.05217Cited by:[§6](https://arxiv.org/html/2606.18465#S6.p4.1)\.
- A\. Power, Y\. Burda, H\. Edwards, I\. Babuschkin, and V\. Misra \(2022\)Grokking: generalization beyond overfitting on small algorithmic datasets\.arXiv preprint arXiv:2201\.02177\.Cited by:[§1](https://arxiv.org/html/2606.18465#S1.p1.1)\.
- L\. Prieto, M\. Barsbey, P\. A\. M\. Mediano, and T\. Birdal \(2025\)Grokking at the edge of numerical stability\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2501\.04697Cited by:[§1](https://arxiv.org/html/2606.18465#S1.p3.1),[§2](https://arxiv.org/html/2606.18465#S2.p4.7),[§4\.3](https://arxiv.org/html/2606.18465#S4.SS3.p2.1),[§4\.5](https://arxiv.org/html/2606.18465#S4.SS5.p2.4),[§5](https://arxiv.org/html/2606.18465#S5.p1.3),[§6](https://arxiv.org/html/2606.18465#S6.p2.1)\.
- N\. Rubin, I\. Seroussi, and Z\. Ringel \(2024\)Grokking as a first order phase transition in two layer networks\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2310\.03789Cited by:[§6](https://arxiv.org/html/2606.18465#S6.p3.1)\.
- V\. Thilak, E\. Littwin, S\. Zhai, O\. Saremi, R\. Paiss, and J\. Susskind \(2022\)The slingshot mechanism: an empirical study of adaptive optimizers and the grokking phenomenon\.arXiv preprint arXiv:2206\.04817\.Cited by:[§6](https://arxiv.org/html/2606.18465#S6.p4.1)\.
- P\. J\. Tikeng Notsawo, G\. Dumas, and G\. Rabusseau \(2025\)Grokking beyond the euclidean norm of model parameters\.InInternational Conference on Machine Learning \(ICML\),Note:arXiv:2506\.05718, PMLR 267:28552–28618Cited by:[§6](https://arxiv.org/html/2606.18465#S6.p1.1)\.
- V\. Varma, R\. Shah, Z\. Kenton, J\. Kramár, and R\. Kumar \(2023\)Explaining grokking through circuit efficiency\.arXiv preprint arXiv:2309\.02390\.Cited by:[§6](https://arxiv.org/html/2606.18465#S6.p4.1)\.
- P\. Wang \(2026a\)Dimensional criticality at grokking across mlps and transformers\.arXiv preprint arXiv:2604\.16431\.Cited by:[§6](https://arxiv.org/html/2606.18465#S6.p3.1)\.
- P\. Wang \(2026b\)Grokking as a dimensional phase transition in neural networks\.arXiv preprint arXiv:2604\.04655\.Cited by:[§6](https://arxiv.org/html/2606.18465#S6.p3.1)\.
- B\. Žunkovič and E\. Ilievski \(2024\)Grokking phase transitions in learning local rules with gradient descent\.Journal of Machine Learning Research25,pp\. 1–52\.Cited by:[§6](https://arxiv.org/html/2606.18465#S6.p3.1)\.