Thermodynamic Weight Decay: Exploring Grokking Acceleration via Attention Specific Heat

# Exploring Grokking Acceleration via Attention Specific Heat
Source: [https://arxiv.org/html/2607.20552](https://arxiv.org/html/2607.20552)
## Thermodynamic Weight Decay: Exploring Grokking Acceleration via Attention Specific Heat

Chitraansh Pandey Independent Researcher chitraanshpandey@gmail\.com github\.com/baymaxbyte/cbo\_core

###### Abstract

Grokking—the delayed generalization of neural networks long after they have memorized their training data—wastes thousands of training epochs and is notoriously unpredictable\. Building on the recent result that Transformer attention is formally isomorphic to a thermodynamic system, we treat the variance of attention logits as a*specific heat*CvC\_\{v\}and show that its peak reliably precedes the generalization transition\. We introduceCvAdamW, a drop\-in AdamW variant that monitorsCvC\_\{v\}online and injects “thermal energy” by dynamically scaling weight decay when a phase transition is detected\. Through a strictly iterative development process we identify three failure modes—initialization noise, mini\-batch micro\-ripples, and “slingshot blinding”—and resolve them with a memorization gate and an exponential\-moving\-average shock absorber\. On modular arithmetic\(a\+bmod97\)\(a\+b\\bmod 97\), CvAdamW enables grokking at epoch 2802 in a 4000\-epoch budget where the baseline never groks\. We further propose a*scale\-invariant*zz\-score reformulation that removes task\-specific hyperparameters, and evaluate it across 10 paired seeds\. A paired analysis shows the cold\-start variant reduces mean grokking latency by 257 epochs \(6\.0%6\.0\\%; median166166epochs; Wilcoxonp=0\.049p=0\.049, Cohen’sd=0\.68d=0\.68, bootstrap95%95\\%CI\[53,489\]\[53,489\]\), improving 8 of 10 seeds; on this single taskCvC\_\{v\}peaks before grokking in all 10 seeds\. Our results indicate that neural networks may expose detectable precursors of impending generalization transitions, and that a physically motivated, proportional intervention can facilitate generalization within a fixed compute budget\. Code and data are public\.

## 1Introduction

Modern over\-parameterized networks frequently exhibit*grokking*\(Poweret al\.,[2022](https://arxiv.org/html/2607.20552#bib.bib2)\): training accuracy saturates within tens of epochs while validation accuracy remains at chance for hundreds or thousands of epochs, then abruptly jumps to near\-perfect\. The intervening plateau is computationally wasteful and, worse, its duration is difficult to predict in advance\. A practitioner cannot easily tell whether a model is about to generalize or has stalled permanently\.

Recent theory offers a striking reinterpretation of this phenomenon\.Kim \([2026](https://arxiv.org/html/2607.20552#bib.bib1)\)prove that the attention mechanism is formally isomorphic to a canonical thermodynamic ensemble: the softmax is the Boltzmann distribution that minimizes Helmholtz free energy, the scaling factor1/dk1/\\sqrt\{d\_\{k\}\}is an inverse temperature, the attention logitsQ​K⊤QK^\{\\top\}are energy levels, and—critically—the*variance*of those logits behaves as a*specific heat*\. In statistical mechanics, specific heat diverges at phase transitions\. If grokking is a phase transition, the network should announce it through a spike in this observable\.

We take this prediction literally and ask a control\-theoretic question:*if the network signals its own phase boundary, can we detect that signal online and supply exactly the energy needed to cross it?*The physical actuator is weight decay\. Since the effective temperature of the attention system scales asTeff∝dk/∥W∥2T\_\{\\mathrm\{eff\}\}\\propto\\sqrt\{d\_\{k\}\}/\\lVert W\\rVert^\{2\}, increasing weight decay shrinks the parameter norm and*heats*the system\. Our contribution is an optimizer that closes this loop\.

#### Contributions\.

- •We show empirically that, on modular arithmetic, the attention specific heatCv=Var​\(Q​K⊤/dk\)C\_\{v\}=\\mathrm\{Var\}\(QK^\{\\top\}/\\sqrt\{d\_\{k\}\}\)peaks before grokking in every seed we tested \(Section[6\.1](https://arxiv.org/html/2607.20552#S6.SS1)\)\.
- •We introduceCvAdamW, a thermodynamically\-aware optimizer that scales weight decay in proportion to the smoothed momentum ofCvC\_\{v\}\(Section[3](https://arxiv.org/html/2607.20552#S3)\), and document three failure modes and their fixes\.
- •We propose a*scale\-invariant*zz\-score formulation that eliminates task\-specific thresholds \(Section[4](https://arxiv.org/html/2607.20552#S4)\)\.
- •We provide a paired statistical evaluation over 10 seeds—per\-seed results, nonparametric tests, effect sizes, bootstrap confidence intervals, lead\-time statistics, and a precursor correlation analysis—rather than point estimates \(Section[6\.3](https://arxiv.org/html/2607.20552#S6.SS3)\)\.

## 2Background

### 2\.1The thermodynamic isomorphism of attention

For a single query, attention computes weights overnnkeys via

pi=exp⁡\(zi/dk\)∑j=1nexp⁡\(zj/dk\),zi=\(Q​K⊤\)i\.p\_\{i\}=\\frac\{\\exp\(z\_\{i\}/\\sqrt\{d\_\{k\}\}\)\}\{\\sum\_\{j=1\}^\{n\}\\exp\(z\_\{j\}/\\sqrt\{d\_\{k\}\}\)\},\\qquad z\_\{i\}=\(QK^\{\\top\}\)\_\{i\}\.\(1\)This is exactly the Boltzmann distributionpi=e−Ei/T/Zp\_\{i\}=e^\{\-E\_\{i\}/T\}/Zwith energy levelsEi=−ziE\_\{i\}=\-z\_\{i\}, inverse temperatureβ=1/dk\\beta=1/\\sqrt\{d\_\{k\}\}, and partition functionZ=∑je−Ej/TZ=\\sum\_\{j\}e^\{\-E\_\{j\}/T\}\(Kim,[2026](https://arxiv.org/html/2607.20552#bib.bib1)\)\. Under this mapping the specific heat of a canonical ensemble,Cv=Var​\(E\)/T2C\_\{v\}=\\mathrm\{Var\}\(E\)/T^\{2\}, corresponds to the variance of the scaled attention logits\. FollowingKim \([2026](https://arxiv.org/html/2607.20552#bib.bib1)\), we identify the specific heat of the attention “information gas” with

Cv=Var​\(Q​K⊤dk\)\.C\_\{v\}\\;=\\;\\mathrm\{Var\}\\\!\\left\(\\frac\{QK^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\right\)\.\(2\)Thermodynamically,CvC\_\{v\}measures the energy required to change the system’s temperature and diverges at a phase transition\. Our working hypothesis is that grokking is such a transition, soCvC\_\{v\}should spike as the model reorganizes its internal representations from memorization to generalization\.

### 2\.2Grokking and weight decay

Grokking was first characterized on algorithmic tasks byPoweret al\.\([2022](https://arxiv.org/html/2607.20552#bib.bib2)\)and has since been linked to weight norm and regularization\(Liuet al\.,[2023](https://arxiv.org/html/2607.20552#bib.bib6); Nandaet al\.,[2023](https://arxiv.org/html/2607.20552#bib.bib5)\)\. A common view is that memorization corresponds to a sharp, high\-curvature minimum, while generalization lives in a flatter basin; escaping the former requires either implicit or explicit pressure on the parameter norm\. Weight decay\(Loshchilov and Hutter,[2019](https://arxiv.org/html/2607.20552#bib.bib3)\)is the standard tool\. The thermodynamic picture makes this precise: with

Teff∝dk∥W∥2,T\_\{\\mathrm\{eff\}\}\\;\\propto\\;\\frac\{\\sqrt\{d\_\{k\}\}\}\{\\lVert W\\rVert^\{2\}\},\(3\)increasing weight decay reduces∥W∥\\lVert W\\rVertand therefore raisesTeffT\_\{\\mathrm\{eff\}\}\. More weight decay is, literally, more heat\.

## 3Method: CvAdamW

CvAdamW is a drop\-in replacement for AdamW that computes a*dynamic*weight decayλt\\lambda\_\{t\}each epoch as a function of theCvC\_\{v\}trajectory\. The base optimizer is unchanged; only the regularization coefficient is modulated\.

### 3\.1Proportional heat injection \(κ/τ\\kappa/\\tauformulation\)

LetCv​\(t\)C\_\{v\}\(t\)be the specific heat measured at epochtt\. We track the smoothed momentum of its velocity with an exponential moving average \(EMA\):

δt\\displaystyle\\delta\_\{t\}=Cv​\(t\)−Cv​\(t−1\),\\displaystyle=C\_\{v\}\(t\)\-C\_\{v\}\(t\-1\),\(4\)μt\\displaystyle\\mu\_\{t\}=α​μt−1\+\(1−α\)​δt,\\displaystyle=\\alpha\\,\\mu\_\{t\-1\}\+\(1\-\\alpha\)\\,\\delta\_\{t\},\(5\)λt\\displaystyle\\lambda\_\{t\}=λbase\+κ⋅max⁡\(0,μt−τ\)\.\\displaystyle=\\lambda\_\{\\mathrm\{base\}\}\+\\kappa\\cdot\\max\\\!\\left\(0,\\;\\mu\_\{t\}\-\\tau\\right\)\.\(6\)Hereα=0\.9\\alpha=0\.9gives an effective window of1/\(1−α\)=101/\(1\-\\alpha\)=10epochs,τ\\tauis a noise floor below which fluctuations are ignored, andκ\\kappaamplifies sustained positive momentum into weight\-decay units\. WhenCvC\_\{v\}is flat or falling,μt≤τ\\mu\_\{t\}\\leq\\tauand the optimizer reverts to standard AdamW withλbase=0\.1\\lambda\_\{\\mathrm\{base\}\}=0\.1\. AsCvC\_\{v\}climbs toward the transition,μt\\mu\_\{t\}grows andλt\\lambda\_\{t\}scales proportionally\.

#### The memorization gate\.

None of the above activates until the model has memorized, enforced bytrain\_acc≥0\.99\\text\{train\\\_acc\}\\geq 0\.99\. A phase transition is only meaningful once the system has reached the metastable \(memorized\) state\.

### 3\.2Three failure modes

CvAdamW was not obtained in one shot\. Each design element resolves a specific, observed failure \(full logs in the accompanying repository\)\.

\(1\) Initialization noise\.Without any gating, all detectors fired at epoch 16, mistaking the chaotic attention patterns of random initialization for a phase transition\. Injecting heat this early performed*worse*than baseline: one cannot force a system out of a minimum it has not yet entered\. The memorization gate resolves this\.

\(2\) Mini\-batch micro\-ripples\.With the gate but without smoothing, a discrete kinematic trigger fired at epoch 240 on a single\-epoch upward blip caused by batch\-sampling noise\. The one\-shot intervention was wasted\. Theα=0\.9\\alpha=0\.9EMA absorbs 1–2 epoch noise while passing the sustained,5050–200200epoch rise of a real transition \(Figure[1](https://arxiv.org/html/2607.20552#S3.F1)\)\.

\(3\) Slingshot blinding\.During major structural reorganization, competing circuits transiently degrade training accuracy—the “slingshot” effect\(Thilaket al\.,[2022](https://arxiv.org/html/2607.20552#bib.bib8)\)\. In one run a largeCvC\_\{v\}spike coincided with train\_acc dropping below0\.990\.99, closing the gate exactly at the peak and blinding a discrete trigger\. A continuous, proportional optimizer sidesteps this: it has already accumulated momentum before the gate closes, so the thermal energy is delivered as the mountain forms rather than after it has passed\.

These failures motivate the central design principle of continuity\. A proportional response that tracks signal magnitude is robust to timing errors that are fatal to a one\-shot trigger\.

![Refer to caption](https://arxiv.org/html/2607.20552v1/figures/fig2_classic_grokking_curve.png)Figure 1:Classic grokking on\(a\+b\)mod97\(a\+b\)\\bmod 97\. Training accuracy saturates within tens of epochs, while validation accuracy remains at chance for thousands of epochs \(shaded “wasted compute”\)\. Under a constrained 4000\-epoch budget the baseline never groks\.

## 4Scale\-Invariant Reformulation

Theκ/τ\\kappa/\\tauformulation has two task\-specific constants:τ\\tauis an absolute magnitude andκ\\kappaa dimensional conversion factor\. On a task whereCvC\_\{v\}occupies a different range, both require retuning\. We remove them by treating the velocityvt=Cv​\(t\)−Cv​\(t−1\)v\_\{t\}=C\_\{v\}\(t\)\-C\_\{v\}\(t\-1\)as a random variable and detecting statistical anomalies\. Using EMA estimates of its running mean and variance,

μt\\displaystyle\\mu\_\{t\}=βz​μt−1\+\(1−βz\)​vt,\\displaystyle=\\beta\_\{z\}\\,\\mu\_\{t\-1\}\+\(1\-\\beta\_\{z\}\)\\,v\_\{t\},\(7\)σt2\\displaystyle\\sigma\_\{t\}^\{2\}=βz​σt−12\+\(1−βz\)​\(vt−μt−1\)​\(vt−μt\),\\displaystyle=\\beta\_\{z\}\\,\\sigma\_\{t\-1\}^\{2\}\+\(1\-\\beta\_\{z\}\)\\,\(v\_\{t\}\-\\mu\_\{t\-1\}\)\(v\_\{t\}\-\\mu\_\{t\}\),\(8\)Zt\\displaystyle Z\_\{t\}=vt−μtσt2\+ϵ,\\displaystyle=\\frac\{v\_\{t\}\-\\mu\_\{t\}\}\{\\sqrt\{\\sigma\_\{t\}^\{2\}\}\+\\epsilon\},\(9\)λt\\displaystyle\\lambda\_\{t\}=λbase\+max⁡\(0,Zt−zthresh\)\.\\displaystyle=\\lambda\_\{\\mathrm\{base\}\}\+\\max\\\!\\left\(0,\\;Z\_\{t\}\-z\_\{\\mathrm\{thresh\}\}\\right\)\.\(10\)The only free constant,zthresh=2\.0z\_\{\\mathrm\{thresh\}\}=2\.0, is a universal2​σ2\\sigmaanomaly threshold\.

#### Cold\-start vs\. continuous sensor\.

The memorization gate creates a choice of when to begin accumulating statistics\. The*cold\-start*variant startsμ,σ2\\mu,\\sigma^\{2\}only after the gate opens \(train\_acc≥0\.99\\text\{train\\\_acc\}\\geq 0\.99\), maximizing sensitivity to the first post\-gate signal\. The*continuous sensor*variant tracks statistics from epoch 1 \(gating only the actuator\), providing a warm baseline\. As we show, the cold\-start variant is paradoxically stronger: a warm baseline dilutes the relative anomaly of a genuine transition\.

## 5Experimental Setup

All experiments use a 2\-layer decoder\-only Transformer \(dmodel=128d\_\{\\mathrm\{model\}\}=128, 4 heads,dk=32d\_\{k\}=32, RoPE positional encoding\(Suet al\.,[2021](https://arxiv.org/html/2607.20552#bib.bib7)\), GELU\) on modular addition\(a\+b\)mod97\(a\+b\)\\bmod 97, a dataset of972=940997^\{2\}=9409examples with a50/5050/50train/validation split\. The base optimizer is AdamW\(Loshchilov and Hutter,[2019](https://arxiv.org/html/2607.20552#bib.bib3)\)\(lr=3×10−4\\text\{lr\}=3\\times 10^\{\-4\},λbase=0\.1\\lambda\_\{\\mathrm\{base\}\}=0\.1, batch size 512\)\. We report the*grokking epoch*, defined as the first epoch at which validation accuracy exceeds0\.950\.95\. The scale\-invariant study uses 10 seeds\{42,123,256,512,1024,2048,3141,4096,7777,9999\}\\\{42,123,256,512,1024,2048,3141,4096,7777,9999\\\}over 7000 epochs; theκ/τ\\kappa/\\taustudy uses 5 seeds over 10,000 epochs\.

## 6Results

### 6\.1Observed Cv precursor dynamics

Across every configuration we ran on this task,CvC\_\{v\}exhibits a pronounced peak that precedes the validation\-accuracy transition \(quantified in Section[6\.3](https://arxiv.org/html/2607.20552#S6.SS3)\)\. Figure[2](https://arxiv.org/html/2607.20552#S6.F2)contrasts three regimes under a 4000\-epoch budget\. The baseline \(A\) shows a clearCvC\_\{v\}peak yet remains trapped at chance accuracy: the signal is present but the system lacks the energy to cross\. The step\-function intervention \(B\) injects a fixed weight\-decay spike \(0\.1→1\.00\.1\\\!\\to\\\!1\.0\) at the detected peak \(epoch 3049\) and groks 284 epochs later\. CvAdamW \(C\) scales weight decay smoothly to a peak of1\.611\.61and groks earliest, at epoch 2802\. Under this budget the baseline never groks, so CvAdamW’s contribution is not merely acceleration but*enabling*generalization\.

![Refer to caption](https://arxiv.org/html/2607.20552v1/figures/fig4_master_three_panel.png)Figure 2:Master comparison under a 4000\-epoch budget \(single seed\)\.\(A\)Baseline AdamW:CvC\_\{v\}\(red\) peaks but validation accuracy \(blue\) stays at chance\.\(B\)Step\-function: a binary weight\-decay spike at the detected peak forces grokking\.\(C\)CvAdamW: continuous, proportional weight\-decay scaling groks earliest\.![Refer to caption](https://arxiv.org/html/2607.20552v1/figures/fig5_wd_comparison.png)Figure 3:Weight\-decay schedules and resulting validation accuracy\. The step\-function \(red\) applies a binary spike with a fixed cooldown; CvAdamW \(green\) applies a smooth, proportional response that tracksCvC\_\{v\}momentum\. The continuous schedule generalizes earlier\.
### 6\.2Continuous vs\. discrete intervention

Figure[3](https://arxiv.org/html/2607.20552#S6.F3)overlays the two weight\-decay schedules\. The step\-function is fragile: its single spike must land precisely at the peak and is defeated by slingshot blinding\. The continuous schedule delivers energy throughout the transition and is immune to single\-epoch gate closures, consistent with the physical intuition that phase transitions are extended, not instantaneous, events\.

### 6\.3Scale\-invariant paired study

We evaluate the scale\-invariant cold\-start variant against the baseline across 10 paired seeds\. Because every seed is run under both conditions, we use a*paired*analysis\. Letdi=Baselinei−ColdStartid\_\{i\}=\\text\{Baseline\}\_\{i\}\-\\text\{ColdStart\}\_\{i\}; positive values favor our method\. Table[1](https://arxiv.org/html/2607.20552#S6.T1)gives the full per\-seed results \(including the continuous\-sensor variant\), so readers can inspect variance, outliers, and robustness directly; Table[2](https://arxiv.org/html/2607.20552#S6.T2)summarizes the paired statistics\.

Table 1:Per\-seed grokking epoch \(first epoch with validation accuracy\>0\.95\>0\.95\) for the baseline and the two scale\-invariant CvAdamW variants over 7000 epochs\. Lower is better\.Table 2:Paired statistical comparison: baseline AdamW vs\. scale\-invariant cold\-start CvAdamW over 10 seeds \(7000 epochs\)\. Positive improvement means fewer epochs to grok\. All values are computed from the released per\-seed data\.The cold\-start variant improves 8 of 10 seeds with a mean reduction of 257 epochs \(6\.0%6\.0\\%\) and a median of 166 epochs \(4\.3%4\.3\\%\)\. The evidence is suggestive but mixed atn=10n=10: the two\-sided Wilcoxon signed\-rank test is significant \(p=0\.049p=0\.049, rank\-biserialr=0\.71r=0\.71\) and the bootstrap95%95\\%confidence interval for the mean improvement excludes zero \(\[53,489\]\[53,489\]\), whereas the two\-tailed pairedtt\-test \(p=0\.060p=0\.060\), the sign test \(p=0\.109p=0\.109\), and the classical confidence interval \(\[−13,527\]\[\-13,527\]\) do not reach significance\. The effect size is medium \(Cohen’sd=0\.68d=0\.68; Cliff’sδ=0\.22\\delta=0\.22\)\. We report the disagreement between tests transparently rather than selecting the most favorable one\. The two regressions \(seeds 123 and 1024\) are small \(−140\-140and−120\-120epochs\) relative to the largest gains \(seeds 2048 and 3141:\+781\+781and\+1023\+1023epochs\)\.

#### CvC\_\{v\}is a precursor

As predicted byKim \([2026](https://arxiv.org/html/2607.20552#bib.bib1)\),CvC\_\{v\}acts as a precursor and also serves as a prerequisite for our method\. On the undisturbed baseline trajectory we locate, for each seed, the epoch of the peak of the EMA\-smoothedCvC\_\{v\}after memorization \(train\_acc≥0\.99\\geq 0\.99\) and compare it to the grokking epoch\. Consistent withKim \([2026](https://arxiv.org/html/2607.20552#bib.bib1)\), we observe thatCvC\_\{v\}peaks before grokking across all seeds \(10/10; mean lead time24002400epochs, std974974, median23962396\)\. This replication is what motivates usingCvC\_\{v\}as a control signal for adaptive intervention\.

Importantly, while theCvC\_\{v\}peak consistently precedes grokking, its timing is not strongly correlated with the eventual grokking epoch \(Pearsonr=−0\.099r=\-0\.099,p=0\.79p=0\.79; Spearmanr=0\.055r=0\.055,p=0\.88p=0\.88\)\. This suggests thatCvC\_\{v\}acts as a*qualitative*precursor signal rather than a*quantitative*predictor of remaining training time—sufficient to trigger an intervention, but not, on its own, to forecast*when*generalization will occur\.

![Refer to caption](https://arxiv.org/html/2607.20552v1/figures/fig7_scale_invariant_cold_start.png)

![Refer to caption](https://arxiv.org/html/2607.20552v1/figures/fig8_scale_invariant_continuous.png)

Figure 4:Scale\-invariant CvAdamW on seed 4096\.Left:cold\-start gate, grokking at epoch 2767\.Right:continuous sensor, grokking at epoch 3078\. Each panel shows validation accuracy, the runningzz\-score, and the dynamic weight decay\. The cold\-start variant produces a sharper anomaly and grokks earlier\.

### 6\.4Strength vs\. universality trade\-off

The scale\-invariant formulation reaches a maximum dynamic weight decay of only0\.80\.8–1\.11\.1, versus1\.51\.5–2\.62\.6for theκ/τ\\kappa/\\tauversion\. BecauseZtZ\_\{t\}rarely exceeds3\.03\.0, the extra weight decay is capped near1\.01\.0\. Theκ=5\.0\\kappa=5\.0amplification of the original formulation delivers stronger thermal shocks and is more effective on this deep phase boundary, but requires retuning per task\. Figure[4](https://arxiv.org/html/2607.20552#S6.F4)shows the cold\-start and continuous variants on a shared seed: the cold\-start anomaly is sharper and groks earlier, matching the statistics in Table[2](https://arxiv.org/html/2607.20552#S6.T2)\.

## 7Discussion

Our experiments support three conclusions\. First, we replicate—on this task—the prediction ofKim \([2026](https://arxiv.org/html/2607.20552#bib.bib1)\)thatCvC\_\{v\}peaks before the phase boundary \(10/10 seeds\), providing empirical support for one consequence of the thermodynamic interpretation rather than the full isomorphism\. This replication validatesCvC\_\{v\}as a usable control signal; however, because its peak timing does not correlate with the grokking epoch, we treat it as a qualitative precursor that can*trigger*intervention, not a quantitative forecaster of remaining training time\. Second, exploiting the signal requires care—rawCvC\_\{v\}is too noisy for discrete triggers, memorization must complete before intervention is meaningful, and a continuous response is essential to survive slingshot dynamics\. Third, the principal value of the method is*reliability*: when grokking is possible but slow, CvAdamW offers modest acceleration, but when the baseline cannot grok within the compute budget, CvAdamW facilitates it\.

#### Negative results and design lessons\.

We emphasize the failure modes as a contribution in their own right\. Three successive detector formulations failed—the first fired on initialization noise \(all triggers at epoch 16\), the second on batch\-induced single\-epoch micro\-ripples, and discrete triggers were defeated by slingshot dynamics that close the memorization gate exactly at theCvC\_\{v\}peak\. Each negative result directly motivated a design element \(the memorization gate, the EMA shock absorber, and continuous proportional scaling\)\. Reporting these failures is intended to save others the same iterations and to justify why the final design takes the form it does\.

#### Limitations\.

We flag several limitations explicitly\.*\(1\) Single task\.*All results are on modular arithmetic\(a\+b\)mod97\(a\+b\)\\bmod 97with a small 2\-layer Transformer, where grokking is unusually clean; we do not claim generality to other tasks or architectures\.*\(2\) No evidence on language or vision models\.*We have not tested GPT\-style language models or Vision Transformers, where emergence is more gradual and the precursor may be harder to detect\.*\(3\) No evidence thatCvC\_\{v\}is uniquely superior\.*We have not comparedCvC\_\{v\}head\-to\-head against other progress measures \(attention entropy, weight norm, gradient norm, loss derivative\); we therefore cannot claim it is the best available signal, only that it is a usable one\.*\(4\) Limited statistical power\.*Withn=10n=10seeds the tests disagree at the margin \(Wilcoxon and bootstrap significant;tt\-test and sign test not\), and theCvC\_\{v\}\-peak epoch does not linearly predict the grokking epoch \(Section[6\.3](https://arxiv.org/html/2607.20552#S6.SS3)\)\.*\(5\) Threshold portability\.*The scale\-invariant thresholdzthresh=2\.0z\_\{\\mathrm\{thresh\}\}=2\.0, though task\-agnostic by construction, may still need adaptation on very different problems\.

#### Future work\.

The most important next experiment is a head\-to\-head comparison ofCvC\_\{v\}against other progress measures \(attention entropy, weight norm, gradient norm, and the training\-loss derivative\) to establish whetherCvC\_\{v\}is uniquely useful\. Further priorities are breadth \(other modular operations such asa×ba\\times banda−bmodpa\-b\\bmod p, and other modulip∈\{53,67,113\}p\\in\\\{53,67,113\\\}\), component ablations, and weight\-decay schedule controls that separate “the signal matters” from “more weight decay helps\.” A natural algorithmic step is a hybrid that useszz\-score detection \(a universal trigger for*when*to act\) together withκ\\kappa\-style amplification \(a strong response for*how hard*to push\)\. Beyond weight decay, the sameCvC\_\{v\}signal could modulate learning\-rate noise, gradient clipping, or dropout\. Finally, scaling to GPT\-style language models and Vision Transformers will test whether the precursor remains actionable where grokking is harder to observe\.

## 8Conclusion

We recast grokking as a thermodynamic phase transition for which the specific heat of attention logits may act as a detectable precursor\. CvAdamW listens to this signal and supplies proportional thermal energy via dynamic weight decay, which on modular arithmetic enables generalization within a fixed compute budget where the baseline never groks\. A scale\-invariantzz\-score variant removes task\-specific tuning and yields a modest, statistically suggestive improvement across seeds \(significant under the Wilcoxon test and a bootstrap interval, though not under all tests atn=10n=10\)\. On this task, neural networks appear to expose precursors of impending generalization; whether the signal generalizes beyond modular arithmetic, and whether it is uniquely informative among progress measures, remain open questions we hope to address next\.

## References

- Thermodynamic isomorphism of transformers: a lagrangian approach to attention dynamics\.arXiv preprint arXiv:2602\.08216\.Cited by:[§1](https://arxiv.org/html/2607.20552#S1.p2.2),[§2\.1](https://arxiv.org/html/2607.20552#S2.SS1.p1.6),[§6\.3](https://arxiv.org/html/2607.20552#S6.SS3.SSS0.Px1.p1.8),[§7](https://arxiv.org/html/2607.20552#S7.p1.3)\.
- Z\. Liu, E\. J\. Michaud, and M\. Tegmark \(2023\)Omnigrok: grokking beyond algorithmic data\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2\.2](https://arxiv.org/html/2607.20552#S2.SS2.p1.3)\.
- I\. Loshchilov and F\. Hutter \(2019\)Decoupled weight decay regularization\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2\.2](https://arxiv.org/html/2607.20552#S2.SS2.p1.3),[§5](https://arxiv.org/html/2607.20552#S5.p1.10)\.
- N\. Nanda, L\. Chan, T\. Lieberum, J\. Smith, and J\. Steinhardt \(2023\)Progress measures for grokking via mechanistic interpretability\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2\.2](https://arxiv.org/html/2607.20552#S2.SS2.p1.3)\.
- A\. Power, Y\. Burda, H\. Edwards, I\. Babuschkin, and V\. Misra \(2022\)Grokking: generalization beyond overfitting on small algorithmic datasets\.InICLR MATH\-AI Workshop,Cited by:[§1](https://arxiv.org/html/2607.20552#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.20552#S2.SS2.p1.3)\.
- J\. Su, Y\. Lu, S\. Pan, A\. Murtadha, B\. Wen, and Y\. Liu \(2021\)RoFormer: enhanced transformer with rotary position embedding\.arXiv preprint arXiv:2104\.09864\.Cited by:[§5](https://arxiv.org/html/2607.20552#S5.p1.10)\.
- V\. Thilak, E\. Littwin, S\. Zhai, O\. Saremi, R\. Paiss, and J\. Susskind \(2022\)The slingshot mechanism: an empirical study of adaptive optimizers and the grokking phenomenon\.arXiv preprint arXiv:2206\.04817\.Cited by:[§3\.2](https://arxiv.org/html/2607.20552#S3.SS2.p4.2)\.