Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

# Quantifying the Memorization-to-Generalization Transition:Scaling Laws and Phase Structure in Grokking
Source: [https://arxiv.org/html/2609.10657](https://arxiv.org/html/2609.10657)
Anish Kataria

###### Abstract

Neural networks trained past memorization frequently undergo a delayed transition to generalization, a phenomenon known as grokking\. Despite theoretical progress on*why*this transition occurs, the quantitative structure of*when*it occurs in hyperparameter space remains uncharacterized\. We map the memorization\-to\-generalization boundary across 384 configurations of two\-hidden\-layer MLPs on modular arithmetic, fitting a power\-law scaling relation for generalization onset time:Tgrok∝H−0\.27​D−2\.04​η−0\.50​λ−0\.64T\_\{\\mathrm\{grok\}\}\\propto H^\{\-0\.27\}\\,D^\{\-2\.04\}\\,\\eta^\{\-0\.50\}\\,\\lambda^\{\-0\.64\}\(R2=0\.732R^\{2\}=0\.732;0\.8210\.821with interactions\)\. The exponent hierarchy reveals that data complexity \(D−2\.04D^\{\-2\.04\}\) is the dominant driver of regime transition, not model capacity \(H−0\.27H^\{\-0\.27\}\): doubling data accelerates generalization by∼4×\{\\sim\}4\\times, while doubling width yields only∼1\.2×\{\\sim\}1\.2\\times\. A sharp phase boundary at weight decayλ≳1\.0\\lambda\\gtrsim 1\.0separates grokking from non\-grokking configurations, and weight norm trajectories show monotonic compression during the transition, consistent with implicit regularization selecting low\-complexity solutions\. These results provide a quantitative foundation for predicting and controlling regime transitions in overparameterized networks\.

###### Keywords:

grokking, scaling laws, phase transitions, memorization, generalization, deep learning theory

## 1Introduction

When do overparameterized neural networks transition from memorizing their training data to genuinely generalizing? This question sits at the center of modern deep learning theory\. The double descent phenomenon\([Belkin et al\., 2019](https://arxiv.org/html/2609.10657#bib.bib2);[Nakkiran et al\., 2020](https://arxiv.org/html/2609.10657#bib.bib11)\)shows that generalization can improve beyond the interpolation threshold, and the lazy\-to\-rich transition\([Woodworth et al\., 2020](https://arxiv.org/html/2609.10657#bib.bib15);[Chizat et al\., 2019](https://arxiv.org/html/2609.10657#bib.bib5)\)identifies feature learning as the mechanism that separates kernel\-regime memorization from structured generalization\. Yet these frameworks characterize the transition qualitatively; they do not predict*when*, in training time, a given configuration will cross from one regime to the other\.

Grokking\([Power et al\., 2022](https://arxiv.org/html/2609.10657#bib.bib13)\)provides a clean experimental window into this question\. Networks trained on modular arithmetic first memorize the training set perfectly, then, after continued training with weight decay, abruptly generalize\. The delay between memorization and generalization can span orders of magnitude depending on hyperparameters\. Mechanistic studies have identified*what*happens during this transition: Fourier\-feature circuits replace memorization lookup tables\([Nanda et al\., 2023](https://arxiv.org/html/2609.10657#bib.bib12)\), competing subnetworks resolve in favor of sparse generalizing solutions\([Merrill et al\., 2023](https://arxiv.org/html/2609.10657#bib.bib10)\), and weight norms compress toward low\-complexity basins\([Liu et al\., 2023](https://arxiv.org/html/2609.10657#bib.bib8)\)\.

What is missing is a*quantitative scaling theory*: which hyperparameters govern the transition time, in what proportion, and does a phase boundary exist in hyperparameter space? We address this with a 384\-configuration sweep, fitting a power\-law scaling law for generalization onset timeTgrokT\_\{\\mathrm\{grok\}\}\.

#### Contributions\.

1. 1\.A power\-law scaling relationTgrok∝H−0\.27​D−2\.04​η−0\.50​λ−0\.64T\_\{\\mathrm\{grok\}\}\\propto H^\{\-0\.27\}\\,D^\{\-2\.04\}\\,\\eta^\{\-0\.50\}\\,\\lambda^\{\-0\.64\}\(R2=0\.821R^\{2\}=0\.821with interactions\), revealing that data complexity dominates model capacity by an order of magnitude in exponent\.
2. 2\.A sharp phase boundary atλ≳1\.0\\lambda\\gtrsim 1\.0in\(η,λ\)\(\\eta,\\lambda\)\-space, below which generalization is structurally suppressed regardless of training duration\.
3. 3\.Weight norm dynamics showing monotonic compression \(median ratio∥θ⁡\(Tgrok\)∥/∥θ⁡\(Tmem\)∥=0\.42\\lVert\\theta\(T\_\{\\mathrm\{grok\}\}\)\\rVert/\\lVert\\theta\(T\_\{\\mathrm\{mem\}\}\)\\rVert=0\.42\) as a measurable proxy for regime state\.

The central finding overturns a natural intuition: it is data coverage and regularization strength, not model capacity, that primarily control when networks escape the memorizing regime\.

## 2Experimental Setup

#### Tasks\.

We study addition mod 113 \(1132=12,769113^\{2\}=12\{,\}769examples\) and division mod 97 \(97×96=9,31297\\times 96=9\{,\}312examples\), canonical grokking benchmarks\([Power et al\., 2022](https://arxiv.org/html/2609.10657#bib.bib13)\)where the ground\-truth generalizing solution \(discrete Fourier transform\) is known\([Nanda et al\., 2023](https://arxiv.org/html/2609.10657#bib.bib12)\)\.

#### Architecture\.

A two\-hidden\-layer MLP with learned embeddings:f⁡\(a,b\)=W3​σ​\(W2​σ​\(W1​\[ea;eb\]\)\)f\(a,b\)=W\_\{3\}\\,\\sigma\(W\_\{2\}\\,\\sigma\(W\_\{1\}\[e\_\{a\};\\,e\_\{b\}\]\)\), whereea,eb∈ℝHe\_\{a\},e\_\{b\}\\in\\mathbb\{R\}^\{H\},σ\\sigmais ReLU, andH∈\{128,256,512\}H\\in\\\{128,256,512\\\}\(100K–960K parameters\)\.

#### Hyperparameter sweep\.

We vary data fractionD∈\{0\.3,0\.5,0\.7,0\.97\}D\\in\\\{0\.3,0\.5,0\.7,0\.97\\\}, learning rateη∈\{0\.001,0\.003,0\.01,0\.03\}\\eta\\in\\\{0\.001,0\.003,0\.01,0\.03\\\}, and weight decayλ∈\{0\.1,0\.3,1\.0,3\.0\}\\lambda\\in\\\{0\.1,0\.3,1\.0,3\.0\\\}, yielding2×3×43=3842\\times 3\\times 4^\{3\}=384configurations\. All models use AdamW \(β1=0\.9\\beta\_\{1\}\{=\}0\.9,β2=0\.98\\beta\_\{2\}\{=\}0\.98\), full\-batch training, up to 150K steps, each with a single random seed \(median seed variance CV≈\\approx8%; up to 18% near the phase boundary, quantified in[SectionA\.1](https://arxiv.org/html/2609.10657#A1.SS1)\)\. Of 356 completed runs \(28 diverged at highη\\eta, lowλ\\lambda\), 297 \(83\.4%\) grokked\.

#### Regime operationalization\.

We defineTmemT\_\{\\mathrm\{mem\}\}as the first step with training accuracy\>\>99% andTgrokT\_\{\\mathrm\{grok\}\}as the first step with test accuracy\>\>95%\. A configuration is*non\-grokking*ifTgrok\>150,000T\_\{\\mathrm\{grok\}\}\>150\{,\}000\. This operationalization cleanly separates the memorization plateau from the generalization transition: the grokking gapΔ​T=Tgrok−Tmem\\Delta T=T\_\{\\mathrm\{grok\}\}\-T\_\{\\mathrm\{mem\}\}spans from 100 to over 100,000 steps \(∼1,000×\{\\sim\}1\{,\}000\\timesrange\)\.

## 3Scaling Law for Generalization Onset

We fit a log\-linear model over the 297 grokked runs:

log⁡Tgrok=α​log⁡H\+β​log​D\+γ​log​η\+δ​log​λ\+c,\\log T\_\{\\mathrm\{grok\}\}=\\alpha\\log H\+\\beta\\log D\+\\gamma\\log\\eta\+\\delta\\log\\lambda\+c\\,,\(1\)yielding the power\-law form

Tgrok∝H−0\.27​D−2\.04​η−0\.50​λ−0\.64T\_\{\\mathrm\{grok\}\}\\;\\propto\\;H^\{\-0\.27\}\\;D^\{\-2\.04\}\\;\\eta^\{\-0\.50\}\\;\\lambda^\{\-0\.64\}\(2\)withR2=0\.732R^\{2\}=0\.732\. All exponents are negative: increasing any hyperparameter reducesTgrokT\_\{\\mathrm\{grok\}\}\.

Figure 1:Grokking dynamics across 356 runs, two tasks, and three model sizes\.\(a\) Representative training curves showing the memorization plateau followed by delayed generalization\. \(b\)TgrokT\_\{\\mathrm\{grok\}\}vs\. FLOPs by model width: wider models grok faster in steps but at higher FLOP cost\. \(c\) Phase diagram in\(η,λ\)\(\\eta,\\lambda\)space: grokking rate approaches 100% forλ≥1\.0\\lambda\\geq 1\.0\. \(d\) Grokking gap vs\.λ\\lambdaby width\. \(e\) Data scaling: medianTgrokT\_\{\\mathrm\{grok\}\}vs\.DD\. \(f\) Compute\-optimal frontier\.### 3\.1Exponent Hierarchy: Data Dominates Capacity

The exponent magnitudes in[Table1](https://arxiv.org/html/2609.10657#S3.T1)establish a clear hierarchy among the drivers of generalization onset\.

Table 1:Fitted exponents with standard errors and Spearman correlations\. Data fractionDDhas the steepest exponent and second\-highest rank correlation; weight decayλ\\lambdahas the highest rank correlation despite a smaller exponent, reflecting its role in the phase boundary\.#### Data complexity \(D−2\.04D^\{\-2\.04\}\)\.

Doubling the training fraction cutsTgrokT\_\{\\mathrm\{grok\}\}by22\.04≈4\.1×2^\{2\.04\}\\approx 4\.1\\times\. This superlinear dependence suggests that additional data does more than provide redundant examples: it simultaneously strengthens the signal for the generalizing circuit and destabilizes the memorizing solution by reducing its effective capacity advantage\. In the Fourier\-feature picture of[Nanda et al\. \(2023\)](https://arxiv.org/html/2609.10657#bib.bib12), each training pair constrains the phase of a Fourier component; at highDD, the constraint set becomes overdetermined for the memorizing lookup table but remains consistent for the algebraic circuit\.

#### Implicit regularization \(λ−0\.64\\lambda^\{\-0\.64\}\)\.

Weight decay is the second lever\. Its Spearman correlation withTgrokT\_\{\\mathrm\{grok\}\}\(ρ=−0\.52\\rho=\-0\.52\) is higher than that of any other hyperparameter, reflecting its dual role:λ\\lambdaboth accelerates the transition and determines whether it occurs at all \([Section4](https://arxiv.org/html/2609.10657#S4)\)\. Theλ−0\.64\\lambda^\{\-0\.64\}exponent is consistent with implicit regularization theory\([Lyu and Li, 2020](https://arxiv.org/html/2609.10657#bib.bib9)\): weight decay biases gradient descent toward low\-norm solutions, and the rate of this bias scales sublinearly withλ\\lambda\.

#### Learning rate \(η−0\.50\\eta^\{\-0\.50\}\)\.

Theη−0\.50\\eta^\{\-0\.50\}scaling is consistent with SGD convergence rates: faster optimization traverses the loss landscape more quickly, reducing the time to reach the generalizing basin\. The interaction termlog⁡H×log⁡η\\log H\\times\\log\\eta\(t=6\.2t=6\.2\) indicates wider models benefit more from higher learning rates, suggesting the optimization landscape becomes more navigable with increased capacity\.

#### Model capacity \(H−0\.27H^\{\-0\.27\}\)\.

Width has the weakest effect\. DoublingHHreducesTgrokT\_\{\\mathrm\{grok\}\}by only20\.27≈1\.2×2^\{0\.27\}\\approx 1\.2\\times\. This contradicts the naive expectation that larger models should generalize faster and supports a view where the transition is governed by*optimization dynamics*\(how quickly weight decay compresses the network\) rather than*representational capacity*\(how many parameters are available\)\. The low Spearman correlation \(ρ=−0\.08\\rho=\-0\.08\) confirms that width’s contribution is further muted by interactions with other hyperparameters\.

Figure 2:Scaling law fit quality\.Predicted vs\. actuallog⁡Tgrok\\log T\_\{\\mathrm\{grok\}\}under the base power\-law model \(R2=0\.732R^\{2\}=0\.732\)\. Points near the diagonal indicate good fit; spread reflects residual variance from initialization noise, task structure, and unswept hyperparameters\.Figure 3:Per\-model\-size scaling\.TgrokT\_\{\\mathrm\{grok\}\}vs\. FLOPs stratified by model width\. Wider models grok in fewer steps but consume more FLOPs per step, producing the characteristic rightward shift\.

### 3\.2Interaction Effects

Adding all six pairwise interactions plus a binary task indicator raisesR2R^\{2\}to0\.8210\.821\(adjustedR2=0\.813R^\{2\}=0\.813; leave\-one\-run\-outR2=0\.799R^\{2\}=0\.799\)\. Four interactions are significant \(all\|t\|\>4\|t\|\>4\):

- •log⁡D×log⁡η\\log D\\times\\log\\eta:\+0\.50\+0\.50\(t=6\.3t\{=\}6\.3\)\. At high data fractions, faster learning rates accelerate grokking more\.
- •log⁡H×log⁡η\\log H\\times\\log\\eta:\+0\.35\+0\.35\(t=6\.2t\{=\}6\.2\)\. Wider models benefit more from higher learning rates\.
- •log⁡D×log⁡λ\\log D\\times\\log\\lambda:\+0\.38\+0\.38\(t=5\.3t\{=\}5\.3\)\. Larger datasets partially substitute for strong regularization\.
- •log⁡H×log⁡λ\\log H\\times\\log\\lambda:−0\.23\-0\.23\(t=−4\.1t\{=\}\{\-\}4\.1\)\. Wider models are less sensitive to weight decay\.

Figure 4:Model–data interaction\.Visualization of thelog⁡D×log⁡η\\log D\\times\\log\\etaandlog⁡H×log⁡η\\log H\\times\\log\\etainteraction terms, showing how the effects of learning rate are modulated by data fraction and model width\.TheD×λD\\times\\lambdainteraction \(\+0\.38\+0\.38\) is notable: when data is abundant \(D→1D\\to 1\), the marginal effect of weight decay is reduced\. This supports a picture where data coverage and regularization are partially substitutable mechanisms for driving the memorization\-to\-generalization transition, both acting to destabilize the high\-norm memorizing solution\.

### 3\.3Robustness

The exponents are stable across tasks \(addition:D=−1\.95D=\-1\.95; division:D=−2\.12D=\-2\.12; pooled:−2\.04\-2\.04\) and across generalization thresholds \([SectionA\.2](https://arxiv.org/html/2609.10657#A1.SS2)\)\. A Weibull accelerated failure time model fit to all 356 runs \(59 right\-censored\) achieves concordance index 0\.71 with coefficient signs consistent with OLS\. The Weibull shape parameterk=1\.4k=1\.4implies a mildly increasing hazard: once a network begins transitioning, it accelerates, consistent with the positive\-feedback picture of norm compression \([Section5](https://arxiv.org/html/2609.10657#S5)\)\.

## 4Phase Structure of the Grokking Boundary

The scaling law describes howTgrokT\_\{\\mathrm\{grok\}\}varies*within*the grokking regime\. Equally important is the boundary*between*regimes: which configurations grok at all?

Figure 5:Left:Phase diagram in\(η,λ\)\(\\eta,\\lambda\)space\. Color indicates grokking rate across dataset fractions and widths\. Atλ≥1\.0\\lambda\\geq 1\.0, nearly all configurations grok; atλ=0\.1\\lambda=0\.1, fewer than 60% do\. The boundary is sharp, not gradual\.Right:Compute\-optimal frontier\. Total FLOPs=Tgrok×Cstep​\(H\)=T\_\{\\mathrm\{grok\}\}\\times C\_\{\\mathrm\{step\}\}\(H\), whereCstep∝H2C\_\{\\mathrm\{step\}\}\\propto H^\{2\}\. Wider models grok faster in steps but at higher FLOP cost, mirroring Chinchilla\-style trade\-offs\([Hoffmann et al\., 2022](https://arxiv.org/html/2609.10657#bib.bib6)\)\.[Figure5](https://arxiv.org/html/2609.10657#S4.F5)\(left\) shows a sharp phase boundary in\(η,λ\)\(\\eta,\\lambda\)\-space\. Atλ≥1\.0\\lambda\\geq 1\.0, nearly all configurations grok regardless of learning rate\. Atλ=0\.1\\lambda=0\.1, fewer than 60% do, and grokking becomes sensitive toη\\eta\. The boundary is not gradual: the transition from<<60% to\>\>95% grokking occurs within a factor of3×3\\timesinλ\\lambda\.

![Refer to caption](https://arxiv.org/html/2609.10657v1/grok_rate_heatmaps.png)Figure 6:Grokking rate heatmapsacross the\(η,λ\)\(\\eta,\\lambda\)grid, stratified by model width and task\. The sharp transition atλ≈1\.0\\lambda\\approx 1\.0is consistent across all settings\.###### Conjecture 1\(Phase boundary\)\.

There exists a critical regularization strengthλ∗≈1\.0\\lambda^\{\*\}\\approx 1\.0\(relative to the AdamW scale used here\) such that generalization onset is structurally suppressed forλ<λ∗\\lambda<\\lambda^\{\*\}\. Belowλ∗\\lambda^\{\*\}, the memorizing solution is a stable fixed point of the training dynamics; aboveλ∗\\lambda^\{\*\}, weight decay destabilizes the memorizing basin, and the network converges to a lower\-norm generalizing solution\.

This conjecture is consistent with the Omnigrok framework of[Liu et al\. \(2023\)](https://arxiv.org/html/2609.10657#bib.bib8), where grokking occurs when the weight norm trajectory crosses a critical threshold at which the generalizing loss landscape basin becomes accessible\. Our phase diagram provides the first quantitative characterization of where this threshold sits in hyperparameter space\.

#### The grokking gap\.

The delayΔ​T=Tgrok−Tmem\\Delta T=T\_\{\\mathrm\{grok\}\}\-T\_\{\\mathrm\{mem\}\}drops by more than an order of magnitude asλ\\lambdaincreases from 0\.1 to 3\.0\. This gap measures the time spent in the memorizing regime after the training objective is satisfied; it is the “wasted” computation from a generalization perspective\. The scaling law predicts this gap, enabling practitioners to estimate whether a given configuration will exhibit delayed generalization or rapid transition\.

## 5Weight Norm Dynamics as a Regime Indicator

Existing accounts of grokking emphasize that norm compression drives the transition\([Liu et al\., 2023](https://arxiv.org/html/2609.10657#bib.bib8);[Merrill et al\., 2023](https://arxiv.org/html/2609.10657#bib.bib10)\)\. We quantify this directly\.

Figure 7:Weight norm dynamics\.Norm trajectories across representative configurations, showing monotonic compression between memorization and generalization onset\. The median ratio∥θ⁡\(Tgrok\)∥/∥θ⁡\(Tmem\)∥=0\.42\\lVert\\theta\(T\_\{\\mathrm\{grok\}\}\)\\rVert/\\lVert\\theta\(T\_\{\\mathrm\{mem\}\}\)\\rVert=0\.42\.From 20 configurations stratified byTgrokT\_\{\\mathrm\{grok\}\}\(fast<<1K, medium 1K–10K, slow\>\>10K; draws within each stratum\), we track weight norms at 100\-step intervals\. Among the 14 of 20 runs with non\-trivial grokking gap, the norm at generalization is lower than at memorization in all 14 cases\. The median ratio∥θ⁡\(Tgrok\)∥/∥θ⁡\(Tmem\)∥=0\.42\\lVert\\theta\(T\_\{\\mathrm\{grok\}\}\)\\rVert/\\lVert\\theta\(T\_\{\\mathrm\{mem\}\}\)\\rVert=0\.42\(IQR: 0\.31–0\.54\)\. While the sample is modest, the monotonic compression pattern is consistent across all three strata \(fast, medium, slow\) and both tasks\.

###### Conjecture 2\(Norm compression threshold\)\.

The memorization\-to\-generalization transition occurs when the weight norm ratio∥θ⁡\(t\)∥/∥θ⁡\(Tmem\)∥\\lVert\\theta\(t\)\\rVert/\\lVert\\theta\(T\_\{\\mathrm\{mem\}\}\)\\rVertcrosses a thresholdr∗r^\{\*\}in the range0\.30\.3–0\.50\.5\. Ifr∗r^\{\*\}is approximately configuration\-independent, then the time to reach it is governed by the scaling law \([2](https://arxiv.org/html/2609.10657#S3.E2)\), with weight decayλ\\lambdacontrolling the compression rate and data fractionDDcontrolling how much compression is needed\.

Two observations support this conjecture\. First, the norm ratio is monotonically decreasing in all 14 runs during the memorization phase: the network progressively compresses before the transition\. Second, the absolute norm atTmemT\_\{\\mathrm\{mem\}\}does not predictTgrokT\_\{\\mathrm\{grok\}\}\(Spearmanρ=0\.06\\rho=0\.06,p=0\.83p=0\.83\), but the*rate*of norm decrease does: configurations with faster compression grok sooner\. This is consistent with the view that what matters is not the starting point but the trajectory through weight space, governed by the interplay ofλ\\lambda\(compression force\) andDD\(landscape structure\)\.

## 6Related Work

#### Grokking\.

[Power et al\. \(2022\)](https://arxiv.org/html/2609.10657#bib.bib13)discovered grokking in modular arithmetic\.[Nanda et al\. \(2023\)](https://arxiv.org/html/2609.10657#bib.bib12)reverse\-engineered the Fourier circuits that form during the transition\.[Liu et al\. \(2023\)](https://arxiv.org/html/2609.10657#bib.bib8)showed grokking extends beyond algorithmic data and proposed the norm\-based LU mechanism\.[Barak et al\. \(2022\)](https://arxiv.org/html/2609.10657#bib.bib1)demonstrated hidden progress in parity learning with SGD\. These works characterize*what*happens; we characterize*when*, providing the first quantitative scaling law\.

#### Implicit regularization and regime transitions\.

[Lyu and Li \(2020\)](https://arxiv.org/html/2609.10657#bib.bib9)showed gradient descent converges to max\-margin solutions for homogeneous networks\.[Woodworth et al\. \(2020\)](https://arxiv.org/html/2609.10657#bib.bib15)identified the lazy\-to\-rich transition as a function of initialization scale\. Ourλ−0\.64\\lambda^\{\-0\.64\}exponent quantifies how explicit regularization modulates this transition in practice\.

#### Phase transitions in learning\.

[Belkin et al\. \(2019\)](https://arxiv.org/html/2609.10657#bib.bib2)and[Nakkiran et al\. \(2020\)](https://arxiv.org/html/2609.10657#bib.bib11)identified the interpolation threshold as a phase boundary\. Our phase diagram \([Figure5](https://arxiv.org/html/2609.10657#S4.F5)\) identifies an analogous boundary in\(η,λ\)\(\\eta,\\lambda\)\-space governing delayed generalization\.

#### Scaling laws\.

[Kaplan et al\. \(2020\)](https://arxiv.org/html/2609.10657#bib.bib7)and[Hoffmann et al\. \(2022\)](https://arxiv.org/html/2609.10657#bib.bib6)established power\-law scaling for language model loss\.[Bordelon et al\. \(2024\)](https://arxiv.org/html/2609.10657#bib.bib3)derived scaling laws from a solvable model connecting lazy\-to\-rich transitions with exponent values\. We extend the scaling law framework to predict not final loss but the timing of a qualitative regime transition\.

#### Memorization in generative models\.

[Somepalli et al\. \(2023\)](https://arxiv.org/html/2609.10657#bib.bib14)and[Carlini et al\. \(2023\)](https://arxiv.org/html/2609.10657#bib.bib4)documented memorization in diffusion models, finding that dataset scale modulates replication rates\. OurD−2\.04D^\{\-2\.04\}exponent quantifies an analogous data\-complexity effect on memorization persistence\.

## 7Discussion

#### The data complexity result\.

TheD−2\.04D^\{\-2\.04\}exponent, nearly eight times the capacity exponentH−0\.27H^\{\-0\.27\}, suggests that the memorization\-to\-generalization transition is fundamentally governed by how well the training data constrains the solution space, not by how many parameters are available\. To halveTgrokT\_\{\\mathrm\{grok\}\}, one can either double width \(reducingTgrokT\_\{\\mathrm\{grok\}\}by1\.2×1\.2\\times\) or increase data fraction by 41% \(reducingTgrokT\_\{\\mathrm\{grok\}\}by2×2\\times\)\. Data is the cheaper lever\.

#### Connections to generative models\.

Although our experiments use MLPs on modular arithmetic, the memorization\-to\-generalization transition we characterize is relevant to generative model training\. Diffusion models memorize training images at early stages before generalizing to novel samples\([Somepalli et al\., 2023](https://arxiv.org/html/2609.10657#bib.bib14);[Carlini et al\., 2023](https://arxiv.org/html/2609.10657#bib.bib4)\), and dataset scale modulates replication rates\. Our scaling methodology—hyperparameter sweep plus power\-law fit to transition timing—offers a template for characterizing these transitions in larger\-scale generative models\. The key prediction is structural: data complexity should dominate capacity in determining transition timing, regardless of architecture\. Testing this prediction on transformers, which also grok on modular arithmetic\([Nanda et al\., 2023](https://arxiv.org/html/2609.10657#bib.bib12);[Power et al\., 2022](https://arxiv.org/html/2609.10657#bib.bib13)\), is a natural next step; quantitative data\-fraction exponents for attention architectures remain to be measured\.

#### Limitations\.

\(1\) The scaling law is fitted on two\-hidden\-layer MLPs on modular arithmetic; exponents may differ for transformers or natural\-language tasks\. \(2\) The width exponent \(−0\.27±0\.10\-0\.27\\pm 0\.10\) is estimated from only three discrete levels\. \(3\)R2=0\.732R^\{2\}=0\.732\(base model\) leaves 27% variance unexplained, likely from initialization noise \(∼2%\{\\sim\}2\\%\), task\-specific structure \(∼3%\{\\sim\}3\\%\), and unswept hyperparameters \(∼13%\{\\sim\}13\\%\)\. \(4\) The phase boundary \([Conjecture1](https://arxiv.org/html/2609.10657#Thmconjecture1)\) is empirically observed, not derived from first principles\.

#### Open questions\.

\(i\) Does theD−2D^\{\-2\}exponent generalize to other group\-structured tasks, or is it specific to modular arithmetic? \(ii\) Can the phase boundary atλ∗\\lambda^\{\*\}be derived from a PAC\-Bayes or minimum description length argument about the relative complexity of the memorizing vs\. generalizing solutions? \(iii\) Is the norm compression trajectory \([Conjecture2](https://arxiv.org/html/2609.10657#Thmconjecture2)\) a Lyapunov function for the memorizing regime, withλ\\lambdacontrolling the rate of descent? These questions connect the empirical findings reported here to the theoretical foundations that the community is well\-positioned to develop\.

#### Code availability\.

## References

- Barak et al\. \[2022\]Boaz Barak, Benjamin L Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang\.Hidden progress in deep learning: SGD learns parities near the computational limit\.In*NeurIPS*, 2022\.
- Belkin et al\. \[2019\]Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal\.Reconciling modern machine\-learning practice and the classical bias–variance trade\-off\.*Proceedings of the National Academy of Sciences*, 116\(32\):15849–15854, 2019\.
- Bordelon et al\. \[2024\]Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan\.A dynamical model of neural scaling laws\.In*ICML*, 2024\.
- Carlini et al\. \[2023\]Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace\.Extracting training data from diffusion models\.In*USENIX Security Symposium*, 2023\.
- Chizat et al\. \[2019\]Lénaïc Chizat, Edouard Oyallon, and Francis Bach\.On lazy training in differentiable programming\.In*NeurIPS*, 2019\.
- Hoffmann et al\. \[2022\]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al\.Training compute\-optimal large language models\.In*NeurIPS*, 2022\.
- Kaplan et al\. \[2020\]Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei\.Scaling laws for neural language models\.*arXiv:2001\.08361*, 2020\.
- Liu et al\. \[2023\]Ziming Liu, Eric J Michaud, and Max Tegmark\.Omnigrok: Grokking beyond algorithmic data\.In*ICLR*, 2023\.
- Lyu and Li \[2020\]Kaifeng Lyu and Jian Li\.Gradient descent maximizes the margin of homogeneous neural networks\.In*ICLR*, 2020\.
- Merrill et al\. \[2023\]William Merrill, Nikolaos Tsilivis, and Aman Shukla\.A tale of two circuits: Grokking as competition of sparse and dense subnetworks\.*arXiv:2303\.11873*, 2023\.
- Nakkiran et al\. \[2020\]Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever\.Deep double descent: Where bigger models and more data hurt\.In*ICLR*, 2020\.
- Nanda et al\. \[2023\]Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt\.Progress measures for grokking via mechanistic interpretability\.In*ICLR*, 2023\.
- Power et al\. \[2022\]Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra\.Grokking: Generalization beyond overfitting on small algorithmic datasets\.In*ICLR Workshop on Mathematics of Deep Learning*, 2022\.
- Somepalli et al\. \[2023\]Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein\.Diffusion art or digital forgery? Investigating data replication in diffusion models\.In*CVPR*, 2023\.
- Woodworth et al\. \[2020\]Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro\.Kernel and rich regimes in overparametrized models\.In*COLT*, 2020\.

## Appendix ARobustness and Variance Analysis

### A\.1Seed Variance

We rerun 10 configurations spanning theTgrokT\_\{\\mathrm\{grok\}\}range with 5 random seeds each\. The median within\-configuration CV is 8%\. The maximum CV is 18% for a configuration near the grokking phase boundary \(λ=0\.1\\lambda=0\.1,η=0\.001\\eta=0\.001\), where stochastic fluctuations can determine whether the network grokks within the 150K step budget\. Initialization noise accounts for roughly 1–2% of variance inlog⁡Tgrok\\log T\_\{\\mathrm\{grok\}\}\.

Table 2:Seed variance analysis\. Each configuration is trained 5 times with different random initializations\.
### A\.2Threshold Sensitivity

TheD−2D^\{\-2\}exponent is stable across generalization thresholds:−2\.15\-2\.15at 85%,−2\.05\-2\.05at 90%,−2\.04\-2\.04at 95%,−1\.88\-1\.88at 99%\. The width exponent varies more \(−0\.28\-0\.28to−0\.19\-0\.19\), reflecting its weaker signal\.

Table 3:Fitted exponents under different generalization\-accuracy thresholds\.NNis the number of runs classified as grokked\.
### A\.3Cross\-Validation

Leave\-one\-level\-out cross\-validation yieldsRCV2=0\.67R^\{2\}\_\{\\mathrm\{CV\}\}=0\.67–0\.760\.76across held\-out hyperparameter levels, with median multiplicative prediction errors of1\.41\.4–1\.6×1\.6\\times\. The hardest level to extrapolate isDD\(RCV2=0\.67R^\{2\}\_\{\\mathrm\{CV\}\}=0\.67\) because theD−2D^\{\-2\}power law must extrapolate over a wider dynamic range\.

Table 4:Leave\-one\-level\-out cross\-validation of the interaction model\.

## Appendix BInteraction Model Details

The full interaction model is:

log⁡Tgrok\\displaystyle\\log T\_\{\\mathrm\{grok\}\}=−0\.27​log⁡H−2\.04​log⁡D−0\.50​log⁡η\\displaystyle=\-0\.27\\log H\-2\.04\\log D\-0\.50\\log\\eta−0\.64​log⁡λ\+0\.50​log⁡D⋅log⁡η\\displaystyle\\quad\-0\.64\\log\\lambda\+0\.50\\log D\\cdot\\log\\eta\+0\.35logH⋅logη\+0\.38logD⋅logλ\\displaystyle\\quad\+0\.35\\log H\\cdot\\log\\eta\+0\.38\\log D\\cdot\\log\\lambda−0\.23logH⋅logλ\+\(minor terms\)\+ε\.\\displaystyle\\quad\-0\.23\\log H\\cdot\\log\\lambda\+\\text\{\(minor terms\)\}\+\\varepsilon\\,\.\(3\)The two remaining interactions \(log⁡H×log⁡D\\log H\\times\\log Dandlog⁡η×log⁡λ\\log\\eta\\times\\log\\lambda\) are not significant \(\|t\|<2\|t\|<2\)\.

## Appendix CCompute\-Optimal Frontier

The per\-step FLOP cost for a two\-hidden\-layer MLP with widthHH, input dimension2​H2H\(embeddings\), andCCoutput classes is:Cstep​\(H\)=2​\(2​H2\+H2\+C​H\)=2​H​\(3​H\+C\)C\_\{\\mathrm\{step\}\}\(H\)=2\(2H^\{2\}\+H^\{2\}\+CH\)=2H\(3H\+C\)\. For our architectures \(C∈\{97,113\}C\\in\\\{97,113\\\},H∈\{128,256,512\}H\\in\\\{128,256,512\\\}\), theH2H^\{2\}terms dominate:Cstep≈6​H2C\_\{\\mathrm\{step\}\}\\approx 6H^\{2\}\. Total FLOPs to grok:F=Tgrok×Cstep​\(H\)∝H−0\.27×H2=H1\.73F=T\_\{\\mathrm\{grok\}\}\\times C\_\{\\mathrm\{step\}\}\(H\)\\propto H^\{\-0\.27\}\\times H^\{2\}=H^\{1\.73\}\. Since total FLOPs grow with width \(1\.73\>01\.73\>0\), wider models are less FLOP\-efficient despite grokking in fewer steps\. The compute\-optimal width for a given FLOP budgetFFsatisfiesH∗∝F1/1\.73≈F0\.58H^\{\*\}\\propto F^\{1/1\.73\}\\approx F^\{0\.58\}, meaning roughly 58% of additional budget should go to width\.

## Appendix DFalsifiable Predictions

The scaling law and norm dynamics generate three testable predictions:

1. 1\.Fourier mode onset scales asD−2D^\{\-2\}\.If theD−2D^\{\-2\}exponent reflects data coverage requirements for circuit formation, then the step at which specific Fourier features emerge in the weight matrices \(measurable via the methodology of[Nanda et al\. \[2023\]](https://arxiv.org/html/2609.10657#bib.bib12)\) should scale asD−2D^\{\-2\}when data fraction is varied\.
2. 2\.Wider models form higher\-rank memorization solutions\.If the weak width exponent \(H−0\.27H^\{\-0\.27\}\) reflects wider models building more complex memorization solutions, then the singular value spectrum of weight matrices atTmemT\_\{\\mathrm\{mem\}\}should have higher effective rank for largerHH\.
3. 3\.Norm compression rate predictsTgrokT\_\{\\mathrm\{grok\}\}better than absolute norm\.The derivative∥d​θ/d​t∥\\lVert d\\theta/dt\\rVertatTmemT\_\{\\mathrm\{mem\}\}should correlate more strongly withTgrokT\_\{\\mathrm\{grok\}\}than∥θ⁡\(Tmem\)∥\\lVert\\theta\(T\_\{\\mathrm\{mem\}\}\)\\rVertdoes\. Our data showsρ=0\.06\\rho=0\.06for the absolute norm; we predict\|ρ\|\>0\.5\|\\rho\|\>0\.5for the compression rate\.