The Kinetics of Training: A Driven-Nucleation Rate Law for Emergence, Plasticity Loss, and Circuit Control in Language Models

# A Driven-Nucleation Rate Law for Emergence, Plasticity Loss, and Circuit Control in Language Models
Source: [https://arxiv.org/html/2607.27281](https://arxiv.org/html/2607.27281)
\(July 2026\)

###### Abstract

A capability appears in a language model when the last parts of its circuit align in one stochastic attempt, and getting all but one right is worth nothing\. We show this no\-partial\-credit joint alignment is therate\-limiting stepof capability formation\. Two fingerprints: in a shortcut\-free apparatus a five\-part circuit missing three waits as long as a three\-part circuit missing three \(1\.19–1\.37\), so the wait counts*missing*parts, not size; and on Pythia across seven capabilities and three scales, ablating one part leaves a median 17% of the capability in 32 of 32 discriminating cells, where partial credit predicts 50–83% \(p=2×10−10p=2\\times 10^\{\-10\}\), while a random non\-part head leaves 100%\.

One rare event whose barrier grows with missing parts yields one rate equation — sites×\\timesattempts×\\timesdrive×e−β​K\\times\\,e^\{\-\\beta K\}, minus destruction — read three ways, each preregistered with frozen constants\.*Forward*: a capability flat at baseline ignites at a step of our choosing once the mix passes a concentration floor \(10/10 above, 0/12 below\), and while still flat its arrival is datable from its precursor to 5% median error on six held\-out models\.*Backward*: the delay to learn a withheld capability grows with waiting until, past a critical step, it never ignites — yet validation loss falls smoothly throughout, so standard monitors are blind to it\. We locate the damage \(heads commit to the base data\) and isolate the cure: re\-initializing only the query–key slices restores learnability \(6/6\) while the value slices do nothing \(0/6\)\. We prove the mechanism in a controlled gated\-attention model, where occupation forces a finite deadline whose consequences need*no mixing assumption*\.*Completed*: SGD’s noise fails the fluctuation–dissipation test, so we install one and anneal, melt and pin circuits on schedule\. Frozen factors predict never\-run combinations to 1–8%\. Scope: conjunction circuits in transformers to 1\.4B, two families\.

## 1Introduction: training has a statics; it is missing its kinetics

A substantial body of theory now describes learning in*equilibrium*terms\. Singular learning theory characterizes training stages as phase transitions of a Bayesian free energy\[[1](https://arxiv.org/html/2607.27281#bib.bib1),[2](https://arxiv.org/html/2607.27281#bib.bib2),[3](https://arxiv.org/html/2607.27281#bib.bib3)\]; data\-mixing thresholds have been derived as capacity\-allocation optima\[[4](https://arxiv.org/html/2607.27281#bib.bib4)\]; grokking has been modeled as a first\-order transition\[[5](https://arxiv.org/html/2607.27281#bib.bib5)\]and as thermal escape from a metastable state\[[6](https://arxiv.org/html/2607.27281#bib.bib6)\]\. This is, in the language of materials science, the*equilibrium phase diagram*of training: which phases exist and which is preferred\.

A phase diagram, however, answers none of the questions a practitioner asks\.*When*will this capability appear?*How fast*will it grow, and can it be accelerated?*Until when*can it still be learned — and why does a network that has trained too long on the wrong data lose the ability to learn it at all\[[7](https://arxiv.org/html/2607.27281#bib.bib7)\]?*Which*circuit will a given perturbation destroy first? These are questions about*rates*, and a free energy is not a rate\. Metallurgy needed a second diagram — the time–temperature–transformation \(TTT\) diagram — before phase knowledge became process control\. Training does not yet have one\.

This paper constructs that missing kinetics\. Its single object of study is the*nucleation rate*of a computational circuit,

J=N⏟sites×ν⏟attempt freq\.×σ​\(c\)⏟supersaturation×e−β​K⏟barrier−D⏟destruction,J\\;=\\;\\underbrace\{N\}\_\{\\text\{sites\}\}\\times\\underbrace\{\\nu\}\_\{\\text\{attempt freq\.\}\}\\times\\underbrace\{\\sigma\(c\)\}\_\{\\text\{supersaturation\}\}\\times\\underbrace\{e^\{\-\\beta K\}\}\_\{\\text\{barrier\}\}\\;\-\\;\\underbrace\{D\}\_\{\\text\{destruction\}\},\(1\)derived once from the stochastic dynamics of SGD \(§[2](https://arxiv.org/html/2607.27281#S2)–§[4](https://arxiv.org/html/2607.27281#S4)\) and then read three ways\. Read*forward*— raising the driveσ\\sigmapast a floor — it is capability*emergence*, with a derived clock that dates arrival before the capability is measurable \(§[6](https://arxiv.org/html/2607.27281#S6)\)\. Read*backward*— as the site termNNis consumed by training itself — it is*loss of plasticity*, supplied with the quantitative law, horizon, mechanism, and repair that the plasticity literature identifies as missing \(§[7](https://arxiv.org/html/2607.27281#S7)\)\. And*completed*— injecting the temperature that SGD provably lacks — it becomes*process control*: annealing, selective dissolution, and quench\-and\-release schedules for individual circuits \(§[8](https://arxiv.org/html/2607.27281#S8)\)\. Every factor in Eq\. \([1](https://arxiv.org/html/2607.27281#S1.E1)\) is measured independently, frozen, and tested on never\-run combinations \(§[9](https://arxiv.org/html/2607.27281#S9)\); every section of this paper either measures one factor or reads the equation in one direction\.

Such a construction earns its keep only if it pays out on problems the field already cares about, in the field’s own language\. It pays out on three\.Loss of plasticityis a recognized open problem: the phenomenon is established in*Nature*\[[7](https://arxiv.org/html/2607.27281#bib.bib7)\], the warm\-starting puzzle\[[8](https://arxiv.org/html/2607.27281#bib.bib8)\]stands, and the causes literature is explicitly partial — intervening on any single proposed mechanism is insufficient\[[9](https://arxiv.org/html/2607.27281#bib.bib9)\]\. What exists are remedies without a law, a timetable, or a located mechanism; §[7](https://arxiv.org/html/2607.27281#S7)supplies each, quantitatively\.Capability\-level data scheduling: data\-mixing methods\[[10](https://arxiv.org/html/2607.27281#bib.bib10),[11](https://arxiv.org/html/2607.27281#bib.bib11),[12](https://arxiv.org/html/2607.27281#bib.bib12)\]optimize loss, not which capabilities form; the floor, the clock, and the shelf life answer “for this mix and this size, will capability X emerge, when, and what is the deadline to still get it\.”Emergence early\-warning: frontier safety frameworks poll evaluations and react\[[13](https://arxiv.org/html/2607.27281#bib.bib13)\]; the precursor readout dates arrival while the capability itself is flat at baseline\.

Scope, stated up front\.Two model families \(Pythia\[[14](https://arxiv.org/html/2607.27281#bib.bib14)\], OLMo\[[15](https://arxiv.org/html/2607.27281#bib.bib15)\]— the public suites with dense early checkpoints\), scales to 1\.4B, mechanism\-level capabilities plus one task\-level demonstration, causal constants from a controlled continued\-training regime whose*structure*\(not constants\) is the claim — with the central causal results additionally reproduced from random initialization; and one explicitly unproved mathematical premise \(Assumption[1](https://arxiv.org/html/2607.27281#Thmassumption1)\) that is flagged wherever it is used\.

## 2The rate equation, forced step by step

We separate throughout what is*cited*\(established results we do not re\-prove\), what is*proved here*\(under stated assumptions\), and what is*measured*\. The derivation chain is short and each step forces the next\.

#### Step 1 \(cited\): minibatch SGD, in the diffusion regime, is an SDE\.

With minibatches, the update is the full gradient plus zero\-mean noise; in the diffusion\-approximation regime \(small steps, locally Gaussian batch noise\) SGD is approximated byd​θ=−∇L​\(θ\)​d​t\+2​ε​Σ​\(θ\)​d​Wd\\theta=\-\\nabla L\(\\theta\)\\,dt\+\\sqrt\{2\\varepsilon\\,\\Sigma\(\\theta\)\}\\,dWwith explicit weak\-error bounds\[[16](https://arxiv.org/html/2607.27281#bib.bib16),[17](https://arxiv.org/html/2607.27281#bib.bib17)\]\. Production optimizers \(Adam, clipping, schedules\) deform this SDE without removing its two load\-bearing features — drift plus state\-dependent noise — but we make no stronger claim: the regime condition is part of the theory’s stated scope \(§[12](https://arxiv.org/html/2607.27281#S12)\), and all causal constants are measured under plain SGD/AdamW where the approximation is controlled\.

#### Step 2: why the barrier is exponential inKK— two independent foundations\.

*\(A\) Computational — proven complexity theory for the tested task class\.*For the family our premise tests instantiate —KK\-wise conjunctions none of whose strict subsets carries signal, a property*proved exactly*for our apparatus in Lemma[1](https://arxiv.org/html/2607.27281#Thmlemma1)— the exponential cost inKKis a theorem of learning theory, not an ansatz\. Any statistical\-query algorithm solving the\(n,k\)\(n,k\)\-sparse parity problem withTTqueries of toleranceτ\\taumust satisfyT/τ2≥Ω​\(nk\)T/\\tau^\{2\}\\geq\\Omega\(n^\{k\}\)\[[18](https://arxiv.org/html/2607.27281#bib.bib18),[19](https://arxiv.org/html/2607.27281#bib.bib19)\]; gradient methods with bounded gradient precision fall within the SQ framework \(see the reduction discussed in[20](https://arxiv.org/html/2607.27281#bib.bib20), App\. A\.1and references therein\), and SGD provably operates near this limit, learningkk\-parities innO​\(k\)n^\{O\(k\)\}steps in the architectures where the guarantee is established\[[20](https://arxiv.org/html/2607.27281#bib.bib20)\]\. Complementing the cost bound, the staircase and leap theorems\[[21](https://arxiv.org/html/2607.27281#bib.bib21),[22](https://arxiv.org/html/2607.27281#bib.bib22)\]prove the dichotomy on which our rate\-determining\-step analysis rests: targets whose monomials each add at most one new coordinate \(subset signal at every stage\) are SGD\-learnable inO​\(d\)O\(d\)samples — an exact iff\-characterization in the mean\-field regime — while a leap of sizeℓ\\ellcostsΘ~​\(dmax⁡\(ℓ−1,1\)\)\\tilde\{\\Theta\}\(d^\{\\max\(\\ell\-1,1\)\}\)online\-SGD steps where proven \(Gaussian nested\-monomial targets under layer\-wise training; CSQ lower bounds in general; conjectured for vanilla SGD, with the leap\-2 case established for standard minibatch SGD by[23](https://arxiv.org/html/2607.27281#bib.bib23)\)\. These theorems govern the task class at large ambient dimension and in stated training regimes; our apparatus is a small\-dimension member of the class whose defining no\-subset\-signal property holds exactly \(Lemma[1](https://arxiv.org/html/2607.27281#Thmlemma1)\), and training is single\-pass throughout \(batch reuse can evade the leap barrier\[[24](https://arxiv.org/html/2607.27281#bib.bib24)\]\)\.

*What the theorems fix, and what they do not\.*They fix the*form*— cost exponential in the number of jointly\-required parts — not the*constant*\. In thenkn^\{k\}scaling,nncounts candidate coordinates over which the relevant support must be found; it is not a vocabulary size, and no reading of the bound assignsβ\\betaa universal value\. Written aseβe^\{\\beta\}per part,β\\betais therefore a logarithm of an*effective*candidate pool, to be measured per system exactly as an activation energy is measured, not derived, in chemical kinetics\. The measured values are informative on precisely this point: in the toy the per\-part cost isβ^=1\.75\\hat\{\\beta\}=1\.75\(summed family\) and≥2\.40\\geq 2\.40\(parity family\), i\.e\. effective pools of≈6\\approx 6and≥11\\geq 11— the scale of the local window the design makes relevant, and far below the full context, showing the search is structured rather than blind\. On real models the clock’sβ≈0\.23\\beta\\approx 0\.23corresponds to an effective pool near unity: real circuit formation is not a blind combinatorial search at all, which is exactly what the staged picture of §[10](https://arxiv.org/html/2607.27281#S10)predicts — cheap subset\-signal\-bearing stages first, leaving a small, heavily guided residual joint step\. Consistently, per\-part costs measured on real models by other protocols are of the same modest order and vary with circuit and regime \(≈0\.23\\approx 0\.23from the clock;0\.470\.47–0\.620\.62from regeneration;≈1\.0\\approx 1\.0from direct\-KKcontrol in prior work\[[25](https://arxiv.org/html/2607.27281#bib.bib25)\]\) — a family\- and circuit\-indexed material constant, as the theory says it must be, and not a universal number the theorems could have supplied\.*\(B\) Physical — the nucleation reading, conditional on mixing\.*For a diffusion of Step 1’s form, the rate of a rare transition out of a metastable basin is, to leading exponential order,J≍exp⁡\(−Δ​S/ε\)J\\asymp\\exp\(\-\\Delta S/\\varepsilon\), withΔ​S\\Delta Sthe quasipotential action of the least\-improbable escape path — Freidlin–Wentzell\[[26](https://arxiv.org/html/2607.27281#bib.bib26),[27](https://arxiv.org/html/2607.27281#bib.bib27)\]; Kramers in the reversible case\[[28](https://arxiv.org/html/2607.27281#bib.bib28)\]\. Given mixing on the formation timescale \(Assumption[1](https://arxiv.org/html/2607.27281#Thmassumption1)\), large\-deviation theory leaves no alternative leading\-order form\.*Only this leg*requires that assumption, and only this leg supplies the thermodynamic vocabulary — temperature, supersaturation, melting — and hence the control operations of §[8](https://arxiv.org/html/2607.27281#S8)\. Rigorous escape\-time and threshold analyses for online SGD exist in adjacent settings and scale exactly as this picture requires: sample complexityNmax⁡\(1,k−1\)N^\{\\max\(1,k\-1\)\}for information exponentkk, with stretched\-exponential trapping below threshold\[[29](https://arxiv.org/html/2607.27281#bib.bib29)\]\. Three gaps between the classical theorems and production training are named rather than hidden: batch noise is not asymptotically small \(the small\-noise limit is a regime condition, whose diagnostic — memorylessness — is itself measured\); adaptive optimizers carry moment state beyond aθ\\theta\-diffusion \(our controlled measurements run under AdamW, so the measured signatures already include this deformation\); and the landscape evolves during training \(the quasi\-stationarity clause of Assumption[1](https://arxiv.org/html/2607.27281#Thmassumption1)\)\.

#### Step 3 \(proved here\): for aKK\-part circuit,Δ​S=β​K\\Delta S=\\beta K\.

This is the mathematical core of the paper — the point where the generic exponential acquires its*content*— and it occupies §[4](https://arxiv.org/html/2607.27281#S4)\.

#### Step 4 \(measured\): what multiplies the exponential\.

The prefactor factorizes as sites×\\timesattempt frequency×\\timessupersaturation under conditional independence \(a mean\-field approximation whose regime of validity, and measured breakdown at the boundary, are quantified in §[9](https://arxiv.org/html/2607.27281#S9)\)\. The supersaturation has the measured formσ​\(c\)∝\(c−c0\)γ\\sigma\(c\)\\propto\(c\-c\_\{0\}\)^\{\\gamma\}above a floor concentrationc0c\_\{0\}and is zero below it \(§[6](https://arxiv.org/html/2607.27281#S6)\)\. The destruction termDDhas two components with different supports: a*reversion drift*on sub\-critical embryos \(always present — it is what the drive must overcome, and what sets the floor\), and a*knockdown*component acting on formed structure, which is Arrhenius in an*injected*temperature and dormant at native settings and practical learning rates \(§[8](https://arxiv.org/html/2607.27281#S8)\)\.

#### The non\-equilibrium branch \(diagnosis cited; test and prescription ours\)\.

That SGD is out of equilibrium — broken detailed balance, anisotropic state\-dependent noise — is established\[[30](https://arxiv.org/html/2607.27281#bib.bib30),[31](https://arxiv.org/html/2607.27281#bib.bib31)\], and we do not claim the diagnosis\. What we add is the*operational*step: a fluctuation–dissipation*qualification test*that any candidate “temperature knob” must pass, the measured result that all six native SGD knobs fail it \(§[5](https://arxiv.org/html/2607.27281#S5)\), and the prescription that follows — the drive is data, and a true temperature, if wanted for control, must be injected \(§[8](https://arxiv.org/html/2607.27281#S8)\)\.

## 3Methods and apparatus

All experiments fall into three apparatus classes; every number in the paper traces to one of them, and every protocol below was frozen \(with pass/kill criteria\) before the corresponding formal runs\.

Controlled toys \(premise tests, EK suite; nose/melt/quench\)\.Models are 2\-layer GPT\-NeoX transformers \(hidden 256, 4–16 heads, MLP4×4\\times, rotary 25%\), trained with AdamW \(lr5×10−45\\times 10^\{\-4\},β=\(0\.9,0\.95\)\\beta=\(0\.9,0\.95\), weight decay0\.010\.01, gradient clip1\.01\.0\), batch 64, from seeded random initialization\.*Conjunction task*\(EK\-1/2/3\): sequences of length 64 over a 64\-token vocabulary follow a peaked Markov chain; with probabilityν\\nu\(the drive\) a position is replaced by the sum mod 64 of the tokens at theKKfixed offsets — theKK\-part target\. The probe is next\-token accuracy on fully\-enabled sequences \(256\-sequence batch, evaluated every 10–25 steps\);t∗t^\{\*\}= first crossing of0\.50\.5\(crossings at0\.3/0\.5/0\.70\.3/0\.5/0\.7recorded\)\.*Open\-loop query\-parity task*\(EK\-4/5\): sequences of length 96 over\{0,1\}\\\{0,1\\\}plus query tokens; at query positions \(gaps≥W\+3\\geq W\{\+\}3, guaranteeing every window contains only raw i\.i\.d\. bits — no feedback channel\) the next token is the parity of theKKbits atKKof theWWwindow slots, the remaining slots being distractors\. Parity makes every strict subset of the relevant bits exactly independent of the target\. Score = accuracy at answer positions \(chance0\.50\.5\);t∗t^\{\*\}= first crossing of0\.750\.75\(crossings at0\.6/0\.75/0\.90\.6/0\.75/0\.9recorded\)\.*Missing\-parts grid*\(EK\-5\): a neutral substrate \(8,000 steps of constant\-answer queries — builds the readout, zero parts\) is followed by a\+1\+1curriculum, each stage trained to completion \(score≥0\.9\\geq 0\.9on three consecutive evaluations, plus 500 consolidation steps\) and checkpointed; grid cells load thess\-part substrate and train theKK\-part task with a 70/30 task/maintenance query mix \(maintenance sustains the built parts; both the 30% and 0% variants are reported, with their respective identified artifacts, in §[4\.4](https://arxiv.org/html/2607.27281#S4.SS4)\); slot roles are randomly permuted per seed so that slot difficulty averages out within every cell\.*Injected bath*\(§[8](https://arxiv.org/html/2607.27281#S8)\): isotropic Gaussian weight noise of per\-step scale2​T⋅lr\\sqrt\{2T\\cdot\\mathrm\{lr\}\}added to all parameters — the qualified temperature of Definition[2](https://arxiv.org/html/2607.27281#Thmdefinition2)\.

Statistics\.Waiting times are censoring\-aware throughout: medians count censored runs as\+∞\+\\infty\(the Kaplan–Meier median, since all censoring occurs at the budget end; reported as undefined below 50% formation\), and means use the shifted\-exponential maximum\-likelihood estimator with right\-censoring\. The estimators were validated directly by rerunning every censored cell at doubled budget: the KM medians predicted the extended\-budget measurements exactly in all three cases \(the naive median\-of\-formed had been 15–19% low\)\. Calibration \(smoke\) seeds are excluded from all analyses\.

Public\-checkpoint observation \(§[6](https://arxiv.org/html/2607.27281#S6)\)\.Mechanism\-level probes are single forward passes at public Pythia and OLMo checkpoints in fp32 with eager attention \(a stated numerical precondition: bf16 softmax underflow collapses the scores\)\. Induction score = prefix\-matching attention on repeated random tokens\[[32](https://arxiv.org/html/2607.27281#bib.bib32)\]\(best head\); precursor score = previous\-token attention \(best head\); task probes are logit\-difference accuracies with train/eval\-disjoint content\.t∗t^\{\*\}= first 0\.5\-crossing, log\-interpolated between checkpoints; thresholds0\.3/0\.5/0\.7×0\.3/0\.5/0\.7\\timeslinear/log interpolation give held\-out median errors 4\.4–13\.0% in all six variants\. Clock constants were fit on five Pythia models, frozen, and only then evaluated on held\-out models\.

Controlled continued training and from\-scratch reproduction \(§[6](https://arxiv.org/html/2607.27281#S6)–§[7](https://arxiv.org/html/2607.27281#S7)\)\.Substrates are public Pythia models \(70M–410M\) or random initializations of the same architecture\. The drive is supplied as repeated\-token passages mixed into plain text at per\-batch concentrationcc; the*switch*replaces the mix at an experimenter\-chosen step;*aging*trainsSSsteps on plain data before the switch;*resets*restore named weight groups \(attention / MLP / embeddings\) of an aged substrate to young values; the*revival*sweep raisesccon a dead substrate\. Arm sizes: switch/shelf/resetn=5n=5; floor sweep 0/12 and 4/5 at the critical concentrations; othersn=2n=2–3 as flagged in §[12](https://arxiv.org/html/2607.27281#S12)\. Every number is recomputable from released per\-experiment JSON files and scripts\.

## 4The mathematical core: a combinatorial barrier and its clock

This section states the assumptions, proves the barrier and clock laws, and reports the experiments that test each assumption*separately*\. The logical status of every claim is marked, and the ledger now has three grades rather than one\.*Proved outright:*the no\-partial\-credit structure of the test apparatus \(Lemma[1](https://arxiv.org/html/2607.27281#Thmlemma1)— Assumption[2](https://arxiv.org/html/2607.27281#Thmassumption2)is exact by construction there\), and the exponential cost inKKfor the task class, which stands on the query\-complexity and leap theorems of §[2](https://arxiv.org/html/2607.27281#S2)\(A\) independently of any stochastic\-analysis premise\.*Conditional theorems:*the nucleation\-form propositions below, exact given Assumptions[1](https://arxiv.org/html/2607.27281#Thmassumption1)–[3](https://arxiv.org/html/2607.27281#Thmassumption3); of these, only the*physical reading*still leans on the unproven mixing clause, which no result of ours can currently discharge for transformer SGD\.*Measured:*every remaining premise, each falsifiable against a concrete alternative and tested in a dedicated preregistered experiment \(§[4\.4](https://arxiv.org/html/2607.27281#S4.SS4)\) independently of the conclusions it licenses\.

### 4\.1Setting and definitions

###### Definition 1\(Circuit, parts, order parameter\)\.

A*circuit*is a set ofKKjointly necessary computational parts \(attention\-head functions\) whose conjunction implements a capability\.KKis defined operationally as the size of the*minimal causal part set*: the smallest set of components each individually necessary at a just\-post\-ignition checkpoint \(single\-component ablation destroys the capability\), machine\-counted by the engine \(§[9](https://arxiv.org/html/2607.27281#S9)\), not assigned by hand\. Its*order parameter*ϕ​\(t\)∈\[0,1\]\\phi\(t\)\\in\[0,1\]is the normalized completeness \(probe\) score, and its formation timet∗t^\{\*\}is the first0\.50\.5\-crossing ofϕ\\phi\(threshold\-robustness is quantified in §[6](https://arxiv.org/html/2607.27281#S6)\)\.

###### Assumption 1\(SDE regime and mixing — the open frontier\)\.

On the formation timescale, \(a\) the SDE approximation of Step 1 holds, and \(b\) the dynamics within the pre\-formation basin is sufficiently mixing that the quasi\-stationary escape theory ofFreidlin and Wentzell \[[26](https://arxiv.org/html/2607.27281#bib.bib26)\]applies\. Part \(b\) is*not proved for transformer SGD*; it is the single unproved premise on which the propositions below are conditional, we flag it rather than hide it, and §[12](https://arxiv.org/html/2607.27281#S12)states what would change if it were established \(every proposition below becomes an unconditional theorem\)\.

###### Assumption 2\(Simultaneous joint alignment\)\.

Formation of aKK\-part circuit requires theKKsub\-alignments to be present*simultaneously*in an attempt window; partial progress on a strict subset does not accumulate into the crossing\. \(Empirically discriminated from the alternative — sequential, “archivable” assembly — in §[4\.4](https://arxiv.org/html/2607.27281#S4.SS4): the two models make opposite predictions, and the data select this one\.\)

###### Lemma 1\(Zero drift on subset\-supported directions — exact in the test apparatus\)\.

In the parity apparatus of §[3](https://arxiv.org/html/2607.27281#S3), let the window bits\(bi\)\(b\_\{i\}\)be i\.i\.d\. uniform and independent of the remaining contextZZ, letRR\(\|R\|=K\|R\|=K\) be the relevant slots, and for anySSwithR⊈SR\\not\\subseteq Slet𝒜S=σ​\(\(bi\)i∈S,Z\)\\mathcal\{A\}\_\{S\}=\\sigma\(\(b\_\{i\}\)\_\{i\\in S\},Z\)be the information available to a strict\-subset feature\. Then:\(a\)𝔼​\[Y∣𝒜S\]=12\\mathbb\{E\}\[Y\\mid\\mathcal\{A\}\_\{S\}\]=\\tfrac\{1\}\{2\}a\.s\.;\(b\)among all𝒜S\\mathcal\{A\}\_\{S\}\-measurable predictors, the population cross\-entropy is*globally minimized*at chance \(p≡12p\\equiv\\tfrac\{1\}\{2\}\) — no descent below chance exists anywhere in the subset\-supported predictor class;\(c\)\(Fourier form\) the target’s expansion over the window bits consists of the*single*characterχR\\chi\_\{R\}:𝔼​\[\(2​Y−1\)​χT\]=0\\mathbb\{E\}\[\(2Y\{\-\}1\)\\chi\_\{T\}\]=0for everyT≠RT\\neq R, so the population gradient’s correlation with any partial character vanishes identically;\(d\)\(stationarity\) on the chance manifold, the projection of the population loss gradient onto the𝒜S\\mathcal\{A\}\_\{S\}\-measurable function subspace is zero, for every strict subsetSSsimultaneously\. Assumption[2](https://arxiv.org/html/2607.27281#Thmassumption2)is therefore not an idealization in the apparatus but*exact by construction*, at the level of the dynamics and not merely of the information\.

###### Proof\.

\(a\)Y=⨁i∈S∩Rbi⊕⨁j∈R∖SbjY=\\bigoplus\_\{i\\in S\\cap R\}b\_\{i\}\\oplus\\bigoplus\_\{j\\in R\\setminus S\}b\_\{j\}; the second factor is uniform and independent of𝒜S\\mathcal\{A\}\_\{S\}sinceR∖S≠∅R\\setminus S\\neq\\emptyset\. \(b\) Conditioning on𝒜S\\mathcal\{A\}\_\{S\},𝔼​\[ℓ​\(p,Y\)∣𝒜S\]=12​ℓ​\(p,1\)\+12​ℓ​\(p,0\)\\mathbb\{E\}\[\\ell\(p,Y\)\\mid\\mathcal\{A\}\_\{S\}\]=\\tfrac\{1\}\{2\}\\ell\(p,1\)\+\\tfrac\{1\}\{2\}\\ell\(p,0\), uniquely minimized atp=12p=\\tfrac\{1\}\{2\}; take expectations\. \(c\)2​Y−1=χR2Y\-1=\\chi\_\{R\}, and characters are orthonormal under the uniform measure\. \(d\) For any𝒜S\\mathcal\{A\}\_\{S\}\-measurable function\-space directiongg, the cross\-entropy functional gradient component is𝔼​\[\(p−Y\)​g\]=𝔼​\[\(p−𝔼​\[Y∣𝒜S\]\)​g\]=𝔼​\[\(p−12\)​g\]\\mathbb\{E\}\[\(p\-Y\)g\]=\\mathbb\{E\}\[\(p\-\\mathbb\{E\}\[Y\\mid\\mathcal\{A\}\_\{S\}\]\)g\]=\\mathbb\{E\}\[\(p\-\\tfrac\{1\}\{2\}\)g\], which vanishes atp≡12p\\equiv\\tfrac\{1\}\{2\}\. ∎

###### Assumption 3\(Part independence, with a measured failure domain\)\.

Away from the capacity wall, theKKsub\-alignment events in an attempt window are approximately independent\. This is the mean\-field idealization of the theory; its failure*is itself measurable*\(Corollary[1](https://arxiv.org/html/2607.27281#Thmcorollary1)\) and is measured in §[4\.4](https://arxiv.org/html/2607.27281#S4.SS4)\.

### 4\.2The barrier law

###### Proposition 1\(Additivity of the combinatorial barrier\)\.

Under Assumptions[1](https://arxiv.org/html/2607.27281#Thmassumption1)–[3](https://arxiv.org/html/2607.27281#Thmassumption3), let theii\-th sub\-alignment be a rare event of per\-attempt probabilitypi≍e−si/εp\_\{i\}\\asymp e^\{\-s\_\{i\}/\\varepsilon\}\. Then the per\-attempt formation probability satisfies

P​\(formation in one attempt\)=∏i=1Kpi≍exp⁡\(−1ε​∑i=1Ksi\),P\(\\text\{formation in one attempt\}\)\\;=\\;\\prod\_\{i=1\}^\{K\}p\_\{i\}\\;\\asymp\\;\\exp\\\!\\Big\(\-\\tfrac\{1\}\{\\varepsilon\}\\sum\_\{i=1\}^\{K\}s\_\{i\}\\Big\),so the escape action is additive,Δ​S=∑isi\\Delta S=\\sum\_\{i\}s\_\{i\}; for exchangeable parts \(si≡ss\_\{i\}\\equiv s\),

J∝ν​e−β​K,β:=s/ε\.J\\;\\propto\\;\\nu\\,e^\{\-\\beta K\},\\qquad\\beta:=s/\\varepsilon\.

###### Proof\.

By Assumption[2](https://arxiv.org/html/2607.27281#Thmassumption2)the formation event in an attempt window is the intersection⋂iEi\\bigcap\_\{i\}E\_\{i\}of theKKsub\-alignment events; by Assumption[3](https://arxiv.org/html/2607.27281#Thmassumption3)the probability of the intersection is the product of the probabilities\. Taking logarithms, the large\-deviation exponents add — the elementary additivity of independent rare\-event costs that underlies Cramér’s and Sanov’s theorems\[[27](https://arxiv.org/html/2607.27281#bib.bib27)\]\. Exchangeability collapses the sum tos​KsK\. Under Assumption[1](https://arxiv.org/html/2607.27281#Thmassumption1), the escape rate inherits this action through the Freidlin–Wentzell principle \(Step 2\), givingJ∝ν​e−β​KJ\\propto\\nu e^\{\-\\beta K\}with the attempt frequencyν\\nuas prefactor\. ∎

###### Corollary 1\(Deviation from linearity measures part dependence\)\.

If the parts share a common enabling cause \(positive association\),P​\(⋂iEi\)≥∏iP​\(Ei\)P\(\\bigcap\_\{i\}E\_\{i\}\)\\geq\\prod\_\{i\}P\(E\_\{i\}\), soΔ​S≤s​K\\Delta S\\leq sK: the barrier grows*sublinearly*inKK\. If forming parts consume a shared resource \(negative interaction, e\.g\. capacity near the wall\), the barrier grows*superlinearly*\. The measured departure ofln⁡t∗\\ln t^\{\*\}from linearity inKKis therefore a direct, signed measure of part dependence — the theory’s own error bar, not an anomaly\.

### 4\.3The clock law

###### Proposition 2\(Memoryless waiting and the precursor clock\)\.

Under Assumption[1](https://arxiv.org/html/2607.27281#Thmassumption1), the normalized escape \(formation\) time is asymptotically exponentially distributed with mean∝1/J\\propto 1/J\[[33](https://arxiv.org/html/2607.27281#bib.bib33),[34](https://arxiv.org/html/2607.27281#bib.bib34)\]\. For a composition ofKKexchangeable parts and its single\-part precursor driven by the same stream, the exact ratio is

𝔼​tcomp∗𝔼​tprec∗=AprecAcomp​eβ​\(K−1\),\\frac\{\\mathbb\{E\}\\,t^\{\*\}\_\{\\rm comp\}\}\{\\mathbb\{E\}\\,t^\{\*\}\_\{\\rm prec\}\}\\;=\\;\\frac\{A\_\{\\rm prec\}\}\{A\_\{\\rm comp\}\}\\,e^\{\\beta\(K\-1\)\},whereA=N​νA=N\\nucollects each circuit’s prefactor \(sites×\\timesattempt frequency\)\. Under the additional prefactor\-cancellation assumptionAprec≈AcompA\_\{\\rm prec\}\\approx A\_\{\\rm comp\}\(A3, stated and tested below\), the clockC​\(K\)=eβ​\(K−1\)C\(K\)=e^\{\\beta\(K\-1\)\}is a*derived*constant: measuring the precursor’s arrival dates the composition’s arrival before the composition exists\.

###### Proof\.

The exponential limit law for exit times from a metastable basin under small noise is classical\[[33](https://arxiv.org/html/2607.27281#bib.bib33)\]; mean exit time≍\\asympinverse rate is the content of metastability theory in the same regime\[[34](https://arxiv.org/html/2607.27281#bib.bib34)\]\. The ratio of means is then the ratio of inverse rates, which is\(Aprec/Acomp\)​eβ​\(K−1\)\(\{A\_\{\\rm prec\}\}/\{A\_\{\\rm comp\}\}\)\\,e^\{\\beta\(K\-1\)\}by Proposition[1](https://arxiv.org/html/2607.27281#Thmproposition1); A3 reduces it toeβ​\(K−1\)e^\{\\beta\(K\-1\)\}\. ∎

### 4\.4Testing the premises, not just the conclusion

Each assumption above is the target of a dedicated experiment \(design frozen before running; the EK suite\):

Statistics are censoring\-aware throughout: medians count censored runs as\+∞\+\\infty\(the Kaplan–Meier median here, since all censoring occurs at the budget end\), and means use the shifted\-exponential maximum\-likelihood estimator with right\-censoring; cells with<<50% formation are reported as undefined rather than estimated\. The estimators were validated directly: rerunning every censored cell at doubled budget \(to 92–100% formation\), the censoring\-corrected KM medians predicted the extended\-budget measurements*exactly*in all three cases, and the MLE means to 0\.2–10%, where the naive median\-of\-formed had been 15–19% low\.K=1K\{=\}1cells form at the evaluation\-grid floor \(t∗≤25t^\{\*\}\\leq 25\) and are therefore reported only as bounds, never entered into fits\.

\(EK\-1\) Memorylessness of waiting times\[tests the premise of Prop\.[2](https://arxiv.org/html/2607.27281#Thmproposition2)\]\. At fixed\(K,ν\)\(K,\\nu\), the formation\-time distribution over 60 seeds per cell spans the predicted crossover \(Fig\.[2](https://arxiv.org/html/2607.27281#S4.F2)a\): at shallow barriers the waiting is nearly deterministic \(rawκ≈220\\kappa\\approx 220, CV 0\.07\), and as the barrier deepens the shifted\-tail shape parameter falls toκ=2\.0\\kappa=2\.0and then*plateaus at the memoryless limit*—κ=1\.2\\kappa=1\.2and1\.31\.3at the two deepest barriers \(60/60 formed at extended budget\) — the exponential regime, exactly where the clock law needs it, and stable once reached\. \(Cell labels: the deep\-barrier cells here are the summed design’s nominalK=3K\{=\}3, which the discriminator below shows solves the task by a two\-part shortcut\. The measured claim is unaffected — what is established is that waiting becomes memoryless once formation is barrier\-dominated, a statement about the kinetics and not about the part count — but we do not attach a part count to these cells\.\) Consistent prior evidence:κ≈80\\kappa\\approx 80at full batch→\\toκ→1\\kappa\\to 1under minibatch noise\[[25](https://arxiv.org/html/2607.27281#bib.bib25)\]\.*Logical status:*exponential waiting is a*necessary condition*of Assumption[1](https://arxiv.org/html/2607.27281#Thmassumption1)\(b\), so EK\-1 is a consistency test that could have falsified the assumption and did not — it is not, and is not offered as, a proof of mixing\.*Regime diagnosis:*κ\\kappaalso arbitrates between the two proven mechanisms available to the computational leg\.Barak et al\. \[[20](https://arxiv.org/html/2607.27281#bib.bib20)\]show that at population scale SGD amplifies a deterministic Fourier\-gap signal of sizeγ=Θ​\(n−\(k−1\)/2\)\\gamma=\\Theta\(n^\{\-\(k\-1\)/2\}\)— progress hidden but not random — while at batch sizes far below the∼nk\\sim n^\{k\}needed to resolveγ\\gammaper step, the search is fluctuation\-dominated\. The two mechanisms share the samenΘ​\(k\)n^\{\\Theta\(k\)\}cost \(the SQ bound is mechanism\-blind\) but differ in waiting\-time shape: deterministic amplification givesκ≫1\\kappa\\gg 1, fluctuation\-driven escape givesκ→1\\kappa\\to 1\. Our measuredκ\\kappaspectrum \(≈\\approx220 at shallow barriers→\\to1\.2–1\.3 at deep ones, at batch 64\) locates the apparatus in the fluctuation\-dominated regime precisely where the nucleation reading applies — and simultaneously shows the crossover toward the deterministic regime at shallow barriers, reconciling the two literatures rather than contradicting either\.

\(EK\-2\) Additivity, and a measured pathway change\[tests Prop\.[1](https://arxiv.org/html/2607.27281#Thmproposition1)and Cor\.[1](https://arxiv.org/html/2607.27281#Thmcorollary1)\]\. The*same nested part set*is assembled intoK=2,3K=2,3at fixed drive across three widths \(Fig\.[2](https://arxiv.org/html/2607.27281#S4.F2)b\)\. We report this family in full, including a contamination we found by direct probing and which withdraws part of it\.

In the summed design the target at an enabled position is the sum of theKKpreceding tokens; when two consecutive positions are both enabled, subtracting the two relations givesst=2​st−1−st−K−1s\_\{t\}=2s\_\{t\-1\}\-s\_\{t\-K\-1\}— an*algebraic shortcut requiring only two attention targets, independent ofKK*\. It is a second solution to the same task: it agrees with the trueKK\-sum wherever the recurrence holds and disagrees where it is broken\. A discriminating probe \(scoring positions whose predecessor is*not*enabled, so that the shortcut is wrong and the true sum still right\) settles which solution each cell learns\. The verdict is sharp: atK=2K\{=\}2the network learns the true sum \(broken/valid accuracy ratio0\.940\.94,n=8n\{=\}8\), while atK=3K\{=\}3andK=4K\{=\}4it learns the shortcut \(ratios0\.110\.11and0\.030\.03; broken\-position accuracy0\.0600\.060and0\.0150\.015against a chance level of0\.0160\.016\)\.*We therefore withdraw the summed family’sK≥3K\\geq 3cells as barrier measurements*:β^=ln⁡\(t3∗/t2∗\)=1\.75\\hat\{\\beta\}=\\ln\(t^\{\*\}\_\{3\}/t^\{\*\}\_\{2\}\)=1\.75compares a shortcut\-formation time to a true\-sum formation time and is not a clean per\-part increment\. The trustworthy per\-part cost is the parity family’s \(≥\\geq11×\\times; §[4\.4](https://arxiv.org/html/2607.27281#S4.SS4), EK\-4\), and the reason for the gap is now mechanistic rather than mysterious:*the summed design admits a shortcut and the parity design provably does not*\.

The pathway change is itself a measured result, and it disposes of what previously looked like an anomaly\.K=4K\{=\}4forming*faster*thanK=3K\{=\}3\(1175 vs\. 1738 in replication; 1138 vs\. 1725 originally\) is not a violation of additivity: fromK=3K\{=\}3upward both cells run on the sameKK\-independent shortcut, whose cost does not grow withKK, so the ladder flattens rather than climbing\. This is textbook parallel\-pathway competition — kinetic versus thermodynamic product — with the crossover located atK=3K\{=\}3: below it the true solution is the cheaper route, above it the two\-part shortcut is, and the system takes whichever barrier is lower, not whichever solution is more accurate\. The picture makes a sharp prediction, which we then tested: if every cell fromK=3K\{=\}3upward runs on the sameKK\-independent route, formation times must*plateau*rather than continue climbing\. Measured \(8 seeds per cell\):300300,17381738,11751175,938938,962962steps atK=2,3,4,5,6K=2,3,4,5,6, with the discriminator ratio0\.940\.94atK=2K\{=\}2and0\.11,0\.03,0\.03,0\.030\.11,0\.03,0\.03,0\.03thereafter\. The ladder flattens exactly where the discriminator says the route changes, and the size of the effect is not marginal: an additive ladder continued from the true\-sum cells would placeK=6K\{=\}6near3×1053\\times 10^\{5\}steps, against962962measured — a factor of350350\. A confirmed prediction of a pathway crossover is stronger evidence for parallel\-route kinetics than the original clean ladder would have been for additivity\. Our first hypothesis for this cell, a metastable partial\-sum intermediate \(Ostwald staging\), was tested by probing every partial sum during training and*refuted*: 0/8 seeds showed a partial\-sum plateau \(peak partial accuracy0\.030\.03against chance0\.0160\.016\)\. We report that failure because it is what forced the correct explanation\.

What the width sweep still shows, within the uncontaminatedK=2K\{=\}2cells: shrinking width toward the capacity wall inflates the barrier \(t2∗t^\{\*\}\_\{2\}:300→900→2000300\\to 900\\to 2000fromd=256d\{=\}256to9696to6464, a6\.7×6\.7\\timescapacity penalty\) — the capacity\-competition side of Cor\.[1](https://arxiv.org/html/2607.27281#Thmcorollary1), measured on cells the discriminator certifies as true\-sum solutions\.

\(EK\-3\) Product law versus bottleneck\[tests Assumption[2](https://arxiv.org/html/2607.27281#Thmassumption2)\]\. Simultaneous joint alignment predicts rate∝\\proptosites in the rare\-event regime; sequential/archivable assembly predicts a bottleneck \(max\) law whose site\-dependence should*strengthen*as the drive stops binding\. A sites\(heads\)×\\timesdrive\(ν\\nu\) grid decides \(Fig\.[2](https://arxiv.org/html/2607.27281#S4.F2)c\): at the memoryless point the log–log slope oft∗t^\{\*\}against head count is−1\.25\-1\.25\(confirmed uncensored at extended budget, 8/8\) — rate proportional to sites, the product law; in the drift\-limited cells the slope collapses to≈0\\approx 0— no site statistics where formation is not a rare event\. \(As in EK\-1, the deep\-barrier cells are the summed design’s nominalK=3K\{=\}3; the site\-proportionality is a statement about the kinetics of a barrier\-dominated formation event, with no part count attached\.\) The bottleneck law predicts the*opposite*pattern \(site\-dependence strongest at high drive\) and is refuted in both regimes\. Consistent prior evidence:J∝N⋅rJ\\propto N\\cdot r,R2R^\{2\}0\.97\[[25](https://arxiv.org/html/2607.27281#bib.bib25)\]\.

\(EK\-5\) The missing\-parts grid\[tests additivity through its sharpest fingerprint\]\. If barriers add, formation time depends only on the number of*missing*parts, never on the circuit’s total size: a substrate holdingssofKKparts \(pre\-built by staged curriculum, each stage trained to completion — partially formed parts do not count, a threshold measured in prior work\[[25](https://arxiv.org/html/2607.27281#bib.bib25)\]\) waits only for the remainingK−sK\-sto align jointly\. A\(K,s\)\(K,s\)grid overK=2\.\.5K=2\.\.5,s=0\.\.4s=0\.\.4\(46 cells, 8–16 seeds each, slot roles randomized per seed, criteria preregistered\) confirms the fingerprint where it is cleanest: at fixedK−s=3K\{\-\}s=3the cell medians forK=3,4,5K=3,4,5agree to 1\.19–1\.37 \(4550/6225/5412 steps; bootstrap 95% intervals share a common overlap,\[1275,8050\]\[1275,8050\]\), and atK−s=4K\{\-\}s=4to 1\.08–1\.34 \(common overlap\[4912,14412\]\[4912,14412\]\) —*a five\-part circuit missing three waits as long as a three\-part circuit missing three*, though its total barrier is far larger\. The alternatives fail characteristically: an energetic \(cluster\-size\) barrier requires within\-row growth withKK\(none observed; row maxima are interior\), and sequential accumulation requires arithmetic row spacing \(measured spacings are not arithmetic\)\. Two protocol artifacts were caught, mechanism\-traced, and controlled: a copy\-task maintenance stream interferes specifically withs=1s\{=\}1cells \(removing it restores 8/8 formation and a\>\>2×\\timeslower median\), while zero maintenance lets built parts decay in high\-sscells, in proportion to their exposure time — under per\-cell clean protocols the shallow\-row spread is 1\.86, and the deepest measured row is flat under*both*protocols \(1\.13 and 1\.33\)\. A window\-width sweep closed the last gap: apparent rung\-height saturation at deep rows vanishes when distractor slots are replenished \(rung ratio×2\.1→×3\.9\\times 2\.1\\to\\times 3\.9fromW=5W\{=\}5to66, exceeding the entire budget atW≥7W\\geq 7\) — a designed\-in ceiling of the smallest window, not a failure of additivity \(Fig\.[1](https://arxiv.org/html/2607.27281#S4.F1)\)\.

![Refer to caption](https://arxiv.org/html/2607.27281v1/figures/fig15_ek5_grid.png)Figure 1:The missing\-parts grid\(EK\-5; preregistered; censoring\-aware medians, uniform\-protocol matrix\)\. \(a\) Formation timet∗​\(K,s\)t^\{\*\}\(K,s\)is constant along the diagonals of fixedK−sK\{\-\}s: what has been built is kinetically invisible\. \(b\) The same data against totalKK: rows at fixed missing count are flat while the full barrier \(s=0s\{=\}0column, dotted\) climbs∼\\sim13×\\times— the direct rate\-determining\-step measurement \(§[10](https://arxiv.org/html/2607.27281#S10)\)\. \(c\) The deep\-rung saturation of theW=5W\{=\}5window is a designed\-in ceiling: replenishing distractor slots restores the rung \(×2\.1→×3\.9\\times 2\.1\\to\\times 3\.9atW=6W\{=\}6; both rungs exceed the full budget atW≥7W\\geq 7\)\.The regime map as a consistency check\.Three independent measurements draw the same boundary: waiting is memoryless \(κ→1\\kappa\\to 1\) precisely in the cells where the site\-product law holds \(slope−1\.25\-1\.25\), and deterministic \(κ≫1\\kappa\\gg 1\) precisely where site statistics vanish \(slope≈0\\approx 0\)\. The nucleation theory’s signatures appear together, in its claimed domain — the barrier\-dominated regime that real compositional circuits inhabit — and switch off together outside it\.\(A3, shared pace\)Across from\-scratch ladder seeds, successive stage increments are strongly co\-scaled \(corr=\+0\.89\\mathrm\{corr\}=\+0\.89\) — precisely the common mode the ratio clock cancels\.

![Refer to caption](https://arxiv.org/html/2607.27281v1/figures/fig14_ek_premises.png)Figure 2:Testing the premises, not the conclusion\(EK suite; 46 preregistered cells including extended\-budget reruns, censoring\-aware statistics\)\. \(a\) Waiting\-time shape crosses from deterministic \(κ≈220\\kappa\\approx 220\) to memoryless and plateaus there \(κ=1\.2,1\.3\\kappa=1\.2,1\.3at the two deepest barriers\) — the premise of Prop\.[2](https://arxiv.org/html/2607.27281#Thmproposition2)\. \(b\) The per\-part barrier increment is additive at roomy width \(β^=1\.75\\hat\{\\beta\}=1\.75, both estimators\) and rises toward the capacity wall — Prop\.[1](https://arxiv.org/html/2607.27281#Thmproposition1)and the superlinear dependence of Cor\.[1](https://arxiv.org/html/2607.27281#Thmcorollary1)\. \(c\) Rate∝\\proptosites exactly where waiting is memoryless; no site statistics in the drift\-limited regime; the bottleneck alternative predicts the opposite pattern and is refuted — Assumption[2](https://arxiv.org/html/2607.27281#Thmassumption2)\.Near\-wall breakdown of factorization \(the 112% point, error growing as a power of floor distance\) is reported with the synthesis in §[9](https://arxiv.org/html/2607.27281#S9)— by Cor\.[1](https://arxiv.org/html/2607.27281#Thmcorollary1)it is the predicted signature of part dependence at shared capacity, and it converts the law’s boundary from a caveat into a falsifiable commitment\.

## 5No intrinsic temperature: the fluctuation–dissipation qualification test

Equilibrium escape theory presumes a bath\. Before Eq\. \([1](https://arxiv.org/html/2607.27281#S1.E1)\) can be read as*thermal*nucleation, one must ask which, if any, of training’s native knobs supplies a temperature\. We make the question operational\.

###### Definition 2\(Temperature qualification\)\.

A noise source qualifies as a*temperature*for the escape dynamics if it satisfies the fluctuation–dissipation structure of an equilibrium bath\[[35](https://arxiv.org/html/2607.27281#bib.bib35)\]: its covariance is \(i\) isotropic, \(ii\) independent of the current state \(in particular, it does not vanish at loss minima\), so that \(iii\) the stationary distribution it induces is the Gibbs measure of the loss at that temperature, and Kramers/Freidlin–Wentzell escape applies withε=T\\varepsilon=T\.

This is a*criterion*, cited, not a theorem of ours; our contribution is to apply it as a qualification exam\. Two clarifications delimit the claim\. First, it concerns the*equilibrium*notion of temperature — the one under which Gibbs sampling and Kramers escape are licensed; non\-equilibrium effective temperatures \(noise temperatures, FDT\-violation ratios\) can of course be*defined*for SGD, and our statement is precisely that none of them earns the escape\-theory role\. Second, the verdict is empirical and unambiguous\. Minibatch\-gradient noise has covariance proportional to the per\-sample gradient second moment: state\-coupled \(it dies where gradients die\), highly anisotropic \(covariance rank as low as∼\\sim1%\[[30](https://arxiv.org/html/2607.27281#bib.bib30)\]\), and therefore fails \(i\)–\(iii\) structurally — the known result that SGD does not sample a Gibbs measure\[[30](https://arxiv.org/html/2607.27281#bib.bib30)\], sharpened recently to full thermodynamic irreversibility with the explicit warning that step size is not a temperature\[[31](https://arxiv.org/html/2607.27281#bib.bib31)\]\. We tested all six native knobs \(minibatch noise, learning rate, batch size, dropout, weight decay, and their clock\-rescaled combinations\) against Definition[2](https://arxiv.org/html/2607.27281#Thmdefinition2)in the controlled apparatus of §[8](https://arxiv.org/html/2607.27281#S8):*zero qualify*\. Learning rate is not even “partially” thermal: converting its noise amplification into an effective clock leaves a monotonic60×60\\timesdegradation and zero net barrier assistance — it is an*anti\-thermal amplifier*, and its noise power is strongly state\-dependent \(measured factor2\.5×2\.5\\timesbetween formed and plateau states for SGD noise;72×72\\timesfor dropout\)\. This is consistent, not in tension, with the real\-LM finding that within the igniting region learning rate simply accelerates ignition \(§[6](https://arxiv.org/html/2607.27281#S6)\): in both regimes the unified statement is that*lr is a clock, never a temperature*— it rescales time without ever supplying barrier assistance\. Anti\-thermal noise does not merely fail to help: because it is gradient\-coupled, it selectively dismantles exactly the multi\-part coincidences that Proposition[1](https://arxiv.org/html/2607.27281#Thmproposition1)prices — measured as barrier\-gated damage \(high\-KKcircuits hurt≈2\.7×\\approx 2\.7\\timesmore than low\-KKon the dropout axis\)\.

Two consequences organize the rest of the paper\. First,*the drive is data*: with no thermal term available, the only state\-decoupled generation knob in Eq\. \([1](https://arxiv.org/html/2607.27281#S1.E1)\) is the supersaturationσ​\(c\)\\sigma\(c\)— the concentration of capability\-relevant statistics in the stream\. Formation in native training is*athermal, driven nucleation*, and its control variable is the data mix \(§[6](https://arxiv.org/html/2607.27281#S6)\)\. Second,*the missing term can be installed*: an injected isotropic weight\-noise bath passes the qualification by construction, and §[8](https://arxiv.org/html/2607.27281#S8)shows that installing it buys exactly the control functions \(anneal, dissolve, pin\) that native training lacks\. In prior work the injected bath was verified to restore a single\(U,D\)\(U,D\)pair reproducing drift, occupancy, and committor simultaneously\[[25](https://arxiv.org/html/2607.27281#bib.bib25)\]\.

## 6Reading the equation forward: emergence

Forward means: hold a substrate fixed, raise the driveσ​\(c\)\\sigma\(c\), and watch the generation term\. The equation predicts a threshold \(no nucleation below a supersaturation floor\), a throttle \(rate rising steeply above it\), and a clock \(the barrier ratio of Prop\.[2](https://arxiv.org/html/2607.27281#Thmproposition2)\)\. Each is confirmed, the clock prospectively\.

Probes, protocols, and preregistration are specified in §[3](https://arxiv.org/html/2607.27281#S3); mechanism\-level probes make “emergence” unambiguous, and constants were frozen before any held\-out model was measured\.

### 6\.1The clock, prospectively validated

FitC=1\.241C=1\.241on five Pythia models \(leave\-one\-out median 3%\); froze constants; predicted six unseen models \(five deduped\-data variants and pythia\-1b, outside the fitted scale range\): 6/6 within the preregistered 25% band, median 5\.0% \(Fig\.[3](https://arxiv.org/html/2607.27281#S6.F3)\)\. Cross\-family: OLMo\-1B — different architecture, data, tokenizer — measures 1\.244 against the frozen 1\.241\. A cheap variant reads a single attention score at step 512 \(0\.36% of training, target capability at baseline everywhere\): 6/6, median 12\.8%\.*No circularity, twice over:*every prospective test used the human\-known previous\-token head from prior literature; the automatic engine \(§[9](https://arxiv.org/html/2607.27281#S9)\) is validated against these answers, not their source\. And the prospective record concerns*induction only*, with constants frozen before any held\-out model was measured; the cross\-circuit unification ofβ\\beta\(induction with IOI, §[9](https://arxiv.org/html/2607.27281#S9)\) is a separate, explicitly post\-hoc consistency check and is nowhere counted as a prospective result\.*No alternative signal:*pre\-ignition, the target’s own curve is flat \(slopes10−610^\{\-6\}–10−510^\{\-5\}/step\), and a placebo sweep shows 200 random heads predict at 42\.5% error \(= guessing\) versus the causal precursor’s 8\.7%\. WhereLavie et al\. \[[36](https://arxiv.org/html/2607.27281#bib.bib36)\]conclude softmax\-attention emergence is intrinsically unpredictable andBaherwani et al\. \[[37](https://arxiv.org/html/2607.27281#bib.bib37)\]that timing is initialization\-random, these prospective results are a direct counterexample for the capability class covered here: the timing is predictable, from the precursor, because the barrier is combinatorial\.

![Refer to caption](https://arxiv.org/html/2607.27281v1/figures/fig1_precursor_law.png)Figure 3:The clock\.\(a\) The capability is invisible while its precursor is not; the same two\-stage shape reproduces on OLMo \(dashed\)\. \(b\) Prospective prediction with frozen constants: 6/6 held\-out models within±\\pm25% \(median 5%\); OLMo\-1B at 1\.244 vs\. frozen 1\.241\.Mechanism dates behavior\.Across 15/15 model–task pairs the mechanism ignites first; real\-text in\-context learning ignites simultaneously with the mechanism \(ratios 0\.93–0\.97\), strict exact\-copy behavior lags×\\times1\.27 \(Fig\.[4](https://arxiv.org/html/2607.27281#S6.F4)\)\. A task\-level few\-shot skill \(in\-context label induction\) shows the full ladder on public checkpoints: never below the capacity wall \(70M flat through 143k steps\), late and stretched at its edge \(160M∼\\sim12,400\), scale\-consistent above it \(410M: 3,933; 1B: 3,822\),∼\\sim5×\\timesafter the induction mechanism \(Fig\.[5](https://arxiv.org/html/2607.27281#S6.F5)\)\. And on all choice\-based probes the pre\-ignition period is systematically*below chance*— an anti\-phase reported as a robust correlational early\-warning channel,*not*a proven mechanism: the direct causal test was inconclusive because the precursor is redundant \(§[11](https://arxiv.org/html/2607.27281#S11)\)\.

![Refer to caption](https://arxiv.org/html/2607.27281v1/figures/fig5_bridge.png)Figure 4:The bridge\.\(a\) At 160M, real\-text in\-context learning ignites with the mechanism; strict copy lags\. \(b\) Mechanism\-first in 15/15 pairs\.![Refer to caption](https://arxiv.org/html/2607.27281v1/figures/fig13_task_ladder.png)Figure 5:Task\-level rung: few\-shot label induction never arrives below the wall, is stretched at its edge, and ignites scale\-consistently above it\.
### 6\.2The floor: a supersaturation threshold, proved intensive

The floor is where the kinetic reading departs most sharply from the equilibrium one\.Gu et al\. \[[4](https://arxiv.org/html/2607.27281#bib.bib4)\]document a data\-mixing threshold and attribute it to capacity allocation — an equilibrium answer \(which phase wins the parameter budget\)\. Equation \([1](https://arxiv.org/html/2607.27281#S1.E1)\) makes a stronger, kinetic claim: below a critical concentration the rate is*suppressed beyond nucleation order*— operationally zero for any subexponential token budget — and the threshold is a property of the per\-step stream composition, not of any total\.

###### Proposition 3\(The floor is a fold — from structure, with no functional form assumed\)\.

Letϕ∈\[0,1\]\\phi\\in\[0,1\]be the alignment order parameter \(the normalized product of attention masses on theKKrequired slots, soϕ=1\\phi=1iff every part is in place\)\. Write the population drift asϕ˙=c​G​\(ϕ\)−D​\(ϕ\)\\dot\{\\phi\}=c\\,G\(\\phi\)\-D\(\\phi\), target minus competitor\. Then the following three properties — each*derived*, none postulated — suffice:\(D1\) the competitor’s drift is linear inϕ\\phi:D​\(ϕ\)=r​ϕD\(\\phi\)=r\\phiwithr=K​\(1−c\)​λ′\>0r=K\(1\-c\)\\lambda^\{\\prime\}\>0;\(D2\) the target’s drift vanishes super\-linearly at the origin:GGis continuous on\[0,1\]\[0,1\]withG​\(ϕ\)/ϕ→0G\(\\phi\)/\\phi\\to 0asϕ→0\+\\phi\\to 0^\{\+\};\(D3\) it vanishes again at completion and is bounded between:G​\(1\)=0G\(1\)=0,GGbounded,G\>0G\>0on\(0,1\)\(0,1\)\. Leth​\(ϕ\):=G​\(ϕ\)/ϕh\(\\phi\):=G\(\\phi\)/\\phi, extended byh​\(0\):=0h\(0\):=0; thenM:=max\[0,1\]⁡hM:=\\max\_\{\[0,1\]\}his finite and attained at an interiorϕ∗\\phi^\{\*\}, and withc0:=r/Mc\_\{0\}:=r/M:\(a\)forc<c0c<c\_\{0\}the*only*fixed point in\[0,1\]\[0,1\]isϕ=0\\phi=0, and it is asymptotically stable — the circuit direction has no resting place at all;\(b\)forc\>c0c\>c\_\{0\}at least two positive fixed pointsϕ−<ϕ\+\\phi\_\{\-\}<\\phi\_\{\+\}exist,ϕ−\\phi\_\{\-\}unstable andϕ\+\\phi\_\{\+\}stable — metastability with activated crossing overϕ−\\phi\_\{\-\};\(c\)ifG∈C1G\\in C^\{1\}andϕ∗\\phi^\{\*\}is a non\-degenerate maximum ofhh, the pair is born atc=c0c=c\_\{0\}through a tangency: a saddle–node \(fold\), not a transcritical exchange\.

###### Proof\.

*The three properties are consequences of results already established\.*\(D1\) is the replicator geometry of softmax attention plus the restoring\-force computation: the exact identity∂𝔼​\[L\]/∂sj=aj​\(ℓj−⟨ℓ⟩a\)\\partial\\mathbb\{E\}\[L\]/\\partial s\_\{j\}=a\_\{j\}\(\\ell\_\{j\}\-\\langle\\ell\\rangle\_\{a\}\)\(Lemma[8](https://arxiv.org/html/2607.27281#Thmlemma8)\) makes every drift on an attention mass*proportional to that mass*, and the competitor supplies a per\-unit\-mass restoring force bounded below byλ′\>0\\lambda^\{\\prime\}\>0uniformly over the basin \(Lemma[7](https://arxiv.org/html/2607.27281#Thmlemma7)\), weighted by its stream share1−c1\-c\. A product ofKKmasses each decaying at rate\(1−c\)​λ′\(1\-c\)\\lambda^\{\\prime\}decays atK​\(1−c\)​λ′K\(1\-c\)\\lambda^\{\\prime\}, so linearity survives the product\. Linear decay is thus forced by the simplex geometry, not chosen for convenience\. \(D2\) is no\-partial\-credit in dynamical form, and the two structures compose\. By Lemma[4](https://arxiv.org/html/2607.27281#Thmlemma4)the target’s pull*per unit mass*at a required slotrris controlled by the product over the*other*required slots,ϕ/Ar\\phi/A\_\{r\}; the replicator factor of \(D1\) then givesA˙r∝Ar⋅\(ϕ/Ar\)=ϕ\\dot\{A\}\_\{r\}\\propto A\_\{r\}\\cdot\(\\phi/A\_\{r\}\)=\\phi, andϕ˙=ϕ​∑rA˙r/Ar\\dot\{\\phi\}=\\phi\\sum\_\{r\}\\dot\{A\}\_\{r\}/A\_\{r\}yields, in the symmetric configurationAr=ϕ1/KA\_\{r\}=\\phi^\{1/K\},

G​\(ϕ\)∝K​ϕ2−1/K,G\(\\phi\)\\;\\propto\\;K\\,\\phi^\{\\,2\-1/K\},whose exponent exceeds11for everyK≥2K\\geq 2, soG​\(ϕ\)/ϕ→0G\(\\phi\)/\\phi\\to 0\. A term linear inϕ\\phiwould be precisely partial credit, which Lemma[1](https://arxiv.org/html/2607.27281#Thmlemma1)excludes exactly; note also that the exponent*rises*withKK, so deeper conjunctions have flatter origins and, by the argument below, higher floors\. \(D3\) is the cross\-entropy’s own saturation together with the simplex bound: masses lie in\[0,1\]\[0,1\], soGGis bounded; and when every part is in place the model predicts the target correctly, the error probability1−pc→01\-p\_\{c\}\\to 0, and with it the label\-dependent gradient —G​\(1\)=0G\(1\)=0\. Nothing is imposed by hand: the gain rises out of the origin, is capped by the simplex, and is switched off by being right\.*The conclusion follows\.*hhis continuous on the compact\[0,1\]\[0,1\]\(continuity at0is \(D2\), at11it is0by \(D3\)\), hence attains a finite maximumMMat someϕ∗\\phi^\{\*\}, interior becauseh\>0h\>0on\(0,1\)\(0,1\)andh​\(0\)=h​\(1\)=0h\(0\)=h\(1\)=0\. Positive fixed points solvec​h​\(ϕ\)=rc\\,h\(\\phi\)=r\. \(a\) Ifc<c0c<c\_\{0\}thenc​h​\(ϕ\)≤c​M<rc\\,h\(\\phi\)\\leq cM<reverywhere, so no positive root; and near the originϕ˙=ϕ​\(c​h​\(ϕ\)−r\)<0\\dot\{\\phi\}=\\phi\(c\\,h\(\\phi\)\-r\)<0, giving asymptotic stability ofϕ=0\\phi=0\. \(b\) Ifc\>c0c\>c\_\{0\}thenc​h​\(ϕ∗\)\>rc\\,h\(\\phi^\{\*\}\)\>rwhilec​h​\(0\)=c​h​\(1\)=0<rc\\,h\(0\)=c\\,h\(1\)=0<r, so the intermediate value theorem gives a root on each side ofϕ∗\\phi^\{\*\}; the sign ofϕ˙=ϕ​\(c​h−r\)\\dot\{\\phi\}=\\phi\(ch\-r\)changes−,\+,−\-,\+,\-across them, which is exactlyϕ−\\phi\_\{\-\}unstable,ϕ\+\\phi\_\{\+\}stable\. \(c\) Atc=c0c=c\_\{0\}the equationc​h=rc\\,h=rholds only atϕ∗\\phi^\{\*\}, whereh′​\(ϕ∗\)=0h^\{\\prime\}\(\\phi^\{\*\}\)=0: the curves are tangent, and non\-degeneracy \(h′′​\(ϕ∗\)<0h^\{\\prime\\prime\}\(\\phi^\{\*\}\)<0\) gives the standard saddle–node normal form, so the two branches are born together rather than crossing\. ∎

###### Proposition 4\(Intensivity and the effective floor \(suppression beyond nucleation order\)\)\.

Suppose \(i\) the drive enters the generation term of Eq\. \([1](https://arxiv.org/html/2607.27281#S1.E1)\) only through the per\-step concentrationccof capability\-relevant statistics \(an intensive quantity\), and \(ii\) forc<c0c<c\_\{0\}the circuit configuration is not metastable under the drift: it is disjoint from the attractor set, the destruction drift exceeding the generation drift along every formation path\. Then:\(a\)\(intensivity\) at fixedc<c0c<c\_\{0\}, increasing total tokens, steps, or batch size cannot induce formation — the floor is independent of budget;\(b\)\(beyond\-order suppression\) the per\-attempt probability of a*sustained*crossing \(ϕ\>1/2\\phi\>1/2held for a windowτ\\tau\) is bounded bye−a​τ/εe^\{\-a\\tau/\\varepsilon\}for somea\>0a\>0, so formation is suppressed beyond any nucleation\-rate order: unlike escape into a metastable target, which is a one\-shot barrier crossing, holding an unstable configuration costs action*linearly in the holding time*\. \(We do not claim a strictly zero probability for a stochastic process; the claim is the qualitative separation between one\-shot barrier crossing above the floor and time\-extensive holding cost below it — and hypothesis \(ii\) is not circular bookkeeping but the floor’s measured content: sub\-critical embryos are directly observed stalling and dissolving belowc0c\_\{0\}\.\)

###### Proof\.

\(a\) Under \(i\), the token budget enters only as the number of attemptsnn; the per\-attempt success probabilityp​\(c\)p\(c\)depends onccalone\. Ifp​\(c\)=0p\(c\)=0then1−\(1−p​\(c\)\)n=01\-\(1\-p\(c\)\)^\{n\}=0for everynn; more generally, by \(b\),p​\(c\)≤e−a​τ/εp\(c\)\\leq e^\{\-a\\tau/\\varepsilon\}uniformly innn, so any subexponential budget leaves the total probability vanishing asε→0\\varepsilon\\to 0\. \(b\) By hypothesis \(ii\) the set\{ϕ\>1/2\}\\\{\\phi\>1/2\\\}contains no attractor of the drift forc<c0c<c\_\{0\}\. By the Freidlin–Wentzell estimates for occupation of neighborhoods of non\-attracting compact sets\[[26](https://arxiv.org/html/2607.27281#bib.bib26), Ch\. 4\], a trajectory that remains in such a neighborhood for timeτ\\taumust follow a path whose action grows at least linearly inτ\\tau— there is a per\-unit\-time costa\>0a\>0to hold the state against the drift — whence the probability bounde−a​τ/εe^\{\-a\\tau/\\varepsilon\}\. \(We use only the sojourn estimate, not the full exit theory\. The drift hypothesis is no longer free\-standing: Proposition[3](https://arxiv.org/html/2607.27281#Thmproposition3)derives it from no\-partial\-credit plus saturation, and it is independently the measured content of the floor — sub\-critical embryos are directly observed stalling and dissolving belowc0c\_\{0\}\.\) ∎

The measured floor matches both clauses\. Intensivity: a4×44\\times 4concentration×\\timesbatch grid \(Fig\.[6](https://arxiv.org/html/2607.27281#S6.F6)\) showsc=0\.20c=0\.20never igniting at any batch \(b64 = 8×\\timesthe tokens of b8\) whilec≥0\.35c\\geq 0\.35always ignites; a finer sweep locates the 70M floor sharply \(0/12 forc≤0\.26c\\leq 0\.26; 4/5 atc=0\.28c=0\.28\)\. Beyond\-order suppression \(not mere slowness\): the near\-floor divergenceτ∝\(c−c0\)−γ\\tau\\propto\(c\-c\_\{0\}\)^\{\-\\gamma\},γ=0\.62±0\.01\\gamma=0\.62\\pm 0\.01\(R2R^\{2\}0\.999\), with sub\-critical embryos observed stalling and dissolving below the floor — the driving\-force law of nucleation, not a capacity story\. The floor is intensive but*not*scale\-invariant: it falls0\.27→0\.20→∼0\.170\.27\\to 0\.20\\to\{\\sim\}0\.17from 70M to 410M, consistent with a weak power law,c0≈0\.26​\(N/70​M\)−0\.26c\_\{0\}\\approx 0\.26\\,\(N/70\\mathrm\{M\}\)^\{\-0\.26\}\(R2=0\.96R^\{2\}=0\.96on three points — reported as the measured trend and an extrapolation hypothesis, not an established law, since three points cannot fix an exponent\)\. Within the igniting region, batch enters asb−0\.33±0\.05b^\{\-0\.33\\pm 0\.05\}and learning rate only accelerates:*at fixed mix, lr and batch are clocks; the mix is the drive*\. Above the floor the throttle is steep: never /∼\\sim1,400 /∼\\sim470 steps at doses 0\.15/0\.30/0\.50 \(Fig\.[7](https://arxiv.org/html/2607.27281#S6.F7)\)\.

What “concentration” has to mean: three diluents, one apparatus\.The real\-model floor leaves a definitional question open — concentration of*what*, measured against*what*? We settled it in the shortcut\-free apparatus by holding the target task, the model, and the optimizer fixed and changing only the material the target is diluted*into*, sweeping the target’s share over up to33×33\\timesin each case\.*Inert diluent*\(uniform random tokens\): formation time is flat — log–log slope0over an8×8\\timesdensity change, no floor\. The inertness is the point: unpredictable filler contributes no gradient, so the target’s share of the*gradient*never changes however its share of the*tokens*falls; and a pure gain change is in any case normalized away by AdamW’s per\-parameter scaling\.*Saturating diluent*\(a single competing copy skill\): slope−0\.22\-0\.22, still no floor, and at intermediate dilution formation is*faster*than at full strength — the competitor is learned within∼\\sim100 steps, after which it stops competing, and the offset\-selective head it built seeds the target \(the cross\-nucleation effect ofDong \[[25](https://arxiv.org/html/2607.27281#bib.bib25)\], reproduced unbidden\)\.*Non\-saturating diluent*\(a family of24\>H24\>Hcopy skills, so the background always has something left to learn\): the slope steepens past the product law and diverges \(−0\.55→−1\.67\-0\.55\\to\-1\.67\), and a floor appears\. The lesson is a definition:supersaturation is the target’s share of the gradient against a persistent competitor, not its share of the token stream— which is exactly why natural text, never exhausted, produces a floor in real models while an inert or quickly\-mastered filler cannot\.

The floor is a stability threshold, not an absence of events\.Pushing the target share down in the non\-saturating apparatus, formation events do not simply stop: at0\.5%0\.5\\%the waiting times split into two clusters \(3\.83\.8–6\.96\.9k versus8383–159159k steps\), one seed forms and then*dissolves*, and one is censored at2×1052\\times 10^\{5\}steps — a flickering band, the signature of net rate crossing zero rather than of a hard kinetic wall\. Measuring the destruction side directly resolves it\. Seeding the circuit atc=0\.25c=0\.25and then switching the stream to a maintenance concentrationcholdc\_\{\\rm hold\}gives survival8/88/8,8/88/8,8/88/8,5/85/8,2/82/8,0/80/8atchold=0\.25,0\.06,0\.02,0\.005,0\.001,0c\_\{\\rm hold\}=0\.25,0\.06,0\.02,0\.005,0\.001,0\(30k held\-out steps;2%2\\%vs0\.1%0\.1\\%Fisherp=0\.007p=0\.007, vs0p=1\.6×10−4p=1\.6\\times 10^\{\-4\}\), with the un\-reduced arm surviving8/88/8— so the collapse is caused by the concentration drop, not by continued training\. At zero drive a formed circuit dissolves to chance within 200 steps\. The two measurements agree on where the floor sits \(0\.10\.1–0\.5%0\.5\\%\), and together they say something sharper than the proposition assumed:*below the floor the capability is not merely never built, it cannot be held*— Proposition[3](https://arxiv.org/html/2607.27281#Thmproposition3)’s “no resting place” read as an experiment\. The two protocols are not identical, and this yields a prediction for real models: below their floor, budgets far longer than ours should show transient, flickering formation rather than a blank, since our0/120/12was measured at a fixed budget\.

![Refer to caption](https://arxiv.org/html/2607.27281v1/figures/fig8_floor_grid.png)Figure 6:The intensive floor: belowc≈0\.26c\\approx 0\.26\(70M\) no batch ignites — token volume cannot substitute for concentration\.![Refer to caption](https://arxiv.org/html/2607.27281v1/figures/fig3_throttle.png)Figure 7:The throttle: dose\-controlled arrival with a nucleation floor\.The switch, including from scratch\.Ignition follows an experimenter\-chosen switch of the data mix past the floor 10/10; without it, never\. From random initialization — no pretrained substrate — the same three laws reproduce: plain data never ignites \(0\.02–0\.03 through 16k steps\); the switch ignites 2/2 \(∼\\sim700 steps\); a delayed switch pays a growing tax; switching too late never ignites \(Fig\.[8](https://arxiv.org/html/2607.27281#S6.F8); the tax and deadline belong to the backward reading, §[7](https://arxiv.org/html/2607.27281#S7)\)\. Cross\-family, on OLMo\-1B: control 0/2 never, switch 2/2 at∼\\sim247, delayed switch taxed \(τ\\tau471/799\)\.

![Refer to caption](https://arxiv.org/html/2607.27281v1/figures/fig12_scratch_switch.png)Figure 8:From random initialization: ignition follows the switch; waiting pays the occupation tax; too late never ignites \(tc≈3000t\_\{c\}\\approx 3000\)\.

## 7Reading the equation backward: loss of plasticity

Backward means: hold the drive available but let training itself consume the*site*termNN\. The equation then predicts a specific, testable structure for how learnability dies — and a discriminator separating it from every drive\-side explanation\.

###### Proposition 5\(Site\-limited versus drive\-limited failure, and the horizon\)\.

In Eq\. \([1](https://arxiv.org/html/2607.27281#S1.E1)\) with factorized generationG=N​ν​σ​\(c\)​e−β​KG=N\\nu\\,\\sigma\(c\)\\,e^\{\-\\beta K\}:\(a\)∂G/∂c=N​ν​σ′​\(c\)​e−β​K\>0\\partial G/\\partial c=N\\nu\\,\\sigma^\{\\prime\}\(c\)\\,e^\{\-\\beta K\}\>0wheneverN\>0N\>0andσ′\>0\\sigma^\{\\prime\}\>0, while ifN=0N=0thenG=0G=0*for every*cc\. Hence raising the drive rescues a drive\-limited substrate and cannot rescue a site\-limited one: the response to a drive increase is a discriminating experiment between the two failure modes\.\(b\)LetNmin​\(c\):=D/\(ν​σ​\(c\)​e−β​K\)N\_\{\\min\}\(c\):=D/\(\\nu\\,\\sigma\(c\)\\,e^\{\-\\beta K\}\)be the fewest sites at which generation clears destruction at drivecc; it is decreasing incc\. If the site count decays with occupation timeSS\(training on other data\) as a non\-increasingN​\(S\)N\(S\), the*plasticity horizon at drivecc*istc​\(c\)=inf\{S:N​\(S\)<Nmin​\(c\)\}t\_\{c\}\(c\)=\\inf\\\{S:N\(S\)<N\_\{\\min\}\(c\)\\\}, non\-decreasing incc\. Pasttc​\(c\)t\_\{c\}\(c\), formation is impossible at that drive, and possible again only by one of exactly two routes: re\-opening sites \(raisingNN\), or raising the drive past the*escape threshold*cesc​\(S\):=inf\{c:Nmin​\(c\)≤N​\(S\)\}c\_\{\\rm esc\}\(S\):=\\inf\\\{c:N\_\{\\min\}\(c\)\\leq N\(S\)\\\}— the drive at which generation on the residual sites clears destruction again\.

###### Proof\.

\(a\) is immediate from the factorized form \(itself the mean\-field structure whose validity domain is measured in §[9](https://arxiv.org/html/2607.27281#S9)\):ccenters only throughσ​\(c\)\\sigma\(c\), multiplied byNN\. \(b\)Nmin​\(c\)N\_\{\\min\}\(c\)is decreasing becauseσ\\sigmais increasing; hencetc​\(c\)t\_\{c\}\(c\)is non\-decreasing\. ForS\>tc​\(c\)S\>t\_\{c\}\(c\),N​\(S\)<Nmin​\(c\)N\(S\)<N\_\{\\min\}\(c\)givesG=N​\(S\)​ν​σ​\(c\)​e−β​K<DG=N\(S\)\\nu\\sigma\(c\)e^\{\-\\beta K\}<D: net rate negative,ϕ\\phicannot cross\. The net rate is restored exactly by raisingNNaboveNmin​\(c\)N\_\{\\min\}\(c\)or by raisingccuntilNmin​\(c\)≤N​\(S\)N\_\{\\min\}\(c\)\\leq N\(S\), i\.e\.c≥cesc​\(S\)c\\geq c\_\{\\rm esc\}\(S\); monotonicity ofNminN\_\{\\min\}makescescc\_\{\\rm esc\}a sharp threshold\. ∎

Site dynamics, closing the loop\.The forward reading has a rate law; symmetry demands the backward reading have one too, not an exogenous “NNdecreases\.” The closure is

d​Nd​S=−λ​\(θ​\(S\)\)​N\+R,\\frac\{dN\}\{dS\}\\;=\\;\-\\,\\lambda\(\\theta\(S\)\)\\,N\\;\+\\;R,whereλ\\lambdais the occupation rate — mechanistically identified below as attention\-subspace commitment — andRRis site re\-opening, zero in native training and*exogenously actuated*by our reset and graft interventions \(they are theRRterm, performed by hand\)\. We deliberately do not impose a parametric form forλ\\lambda: with the occupancy protocol run at zero target drive \(c=0c=0during aging, so the depletion measured here is occupation by the base task alone, not consumption by target\-crystal growth — the coupled case is outside this protocol and stated as such\), the equation integrates toN​\(S\)/N​\(0\)=e−∫0SλN\(S\)/N\(0\)=e^\{\-\\int\_\{0\}^\{S\}\\lambda\}, and since formation time scales as1/N1/Nat fixed drive \(Prop\.[5](https://arxiv.org/html/2607.27281#Thmproposition5)\), the*measured, non\-parametric*depletion curve isN​\(S\)/N​\(0\)=τ​\(0\)/τ​\(S\)N\(S\)/N\(0\)=\\tau\(0\)/\\tau\(S\):1→0\.46→0\.451\\to 0\.46\\to 0\.45atS=0/600/1200S=0/600/1200, crossingNmin​\(c=0\.5\)N\_\{\\min\}\(c\{=\}0\.5\)beforeS=3600S=3600\. Three points and a bound do not fix a functional form, and we decline to overfit one; what they do fix is the*structure*— depletion is real, monotone, drive\-independent in this protocol, mechanistically located \(λ\\lambdatracks commitment atr=0\.93r=0\.93\), and reversible only through theRRterm\. The scale law enters asd​λ/d​\(size\)<0d\\lambda/d\(\\text\{size\}\)<0\(aging slope0\.69→0\.160\.69\\to 0\.16from 70M to 410M\)\.

The proposition’s content is that these are*measurable alternatives*, and every measurement lands on the site side\.The discriminator: on a substrate aged past its horizon, doubling the drive does not rescue \(0/3\), while*opening new sites does*— grafted fresh head pairs ignite with lightning kinetics \(200–500 steps\) at per\-pair successp≈0\.15p\\approx 0\.15consistent with independent\-site statistics, and sites must be opened*in pairs*across both layers \(single\-layer openings: 0/6\), exactly as a two\-layer composition requires\. Learnability death is a site\-term failure, not a drive\-term failure\.The tax and the horizon: the post\-switch delay grows with waiting \(n=5n=5seeds:τ=506±44→1108±66→1115±65\\tau=506\\pm 44\\to 1108\\pm 66\\to 1115\\pm 65atS=0/600/1200S=0/600/1200\); atS=3600S=3600no ignition in 4,400 further steps \(tail slope zero — not censoring\)\. The horizon has a scale law:tc≈3200t\_\{c\}\\approx 3200\(70M\),\>6000\>6000\(160M\),\>8000\>8000\(410M\); the aging slope falls0\.69→0\.160\.69\\to 0\.16— larger substrates age slower \(Fig\.[9](https://arxiv.org/html/2607.27281#S7.F9)\)\.Drive\-relative death: the dead substrate revives above a sharp escape drive — 0/2 at 0\.60–0\.69, 2/3 at 0\.71, 2/2 at≥\\geq0\.72; thresholdcesc=0\.705±0\.005c\_\{\\rm esc\}=0\.705\\pm 0\.005— clause \(b\)’s second route, observed, including its predicted*sharpness*\(monotoneNminN\_\{\\min\}makescescc\_\{\\rm esc\}a threshold, not a gradient\)\.The located mechanism: during waiting, every head’s attention pattern grows more committed \(peakedness\+45%\+45\\%, entropy−20%\-20\\%\), tracking the tax atr=0\.93r=0\.93\. What ages is*recruitable capacity*, not the existing parts: the precursor head itself keeps growing throughout the waiting period, and what is exhausted is the supply of uncommitted heads — which is why the failure is site\-limited \(above\) and why it is repaired by re\-opening sites rather than by rebuilding parts\. Targeted resets make this causal — restoring only the*attention*weights of a dead model rescues ignition 5/5 \(vs\. 0/5 un\-reset controls\), restoring MLP rescues weakly, restoring embeddings does not \(0/2\), replicating at 160M and 410M\. The two reset arms separate with non\-overlapping Wilson 95% intervals \(5/55/5:\[0\.57,1\.00\]\[0\.57,1\.00\];0/50/5:\[0\.00,0\.43\]\[0\.00,0\.43\]; Fisher exactp=0\.008p=0\.008\), as do the floor arms \(0/120/12below the floor:\[0\.00,0\.24\]\[0\.00,0\.24\];4/54/5above it:\[0\.38,0\.96\]\[0\.38,0\.96\]\)\.The active ingredient, isolated\.Theorem[1](https://arxiv.org/html/2607.27281#Thmtheorem1)\(iii\) says occupation lives in the attention*pattern*, so the score channel should carry the entire cure and the value channel none of it; the apparatus confirmed this \(App\.[A\.7](https://arxiv.org/html/2607.27281#A1.SS7)\), and the prediction was then tested on the real substrate by slicing the fused QKV projection head\-wise\. It separates completely \(six seeds, one aged substrate per seed, all four arms run from it\): un\-reset0/60/6; re\-initializing only the*query–key*slices6/66/6\(Fisherp=0\.002p=0\.002against control\); re\-initializing only the*value*slices and the output projection0/60/6— indistinguishable from doing nothing \(induction score0\.2520\.252vs0\.2630\.263un\-reset, against0\.7000\.700for the QK arm\); and the whole\-block reset6/66/6, exactly equal to the QK arm\.*The block reset’s entire therapeutic effect comes from the score channel\.*This converts the repair from a coarse intervention into a prescription — reset QK, keep OV — and supplies the sharpest causal evidence in the paper that what ages is attention*allocation*rather than the content the heads carry\. \(Note the distinction: this reset*re\-opens sites*for a capability that has never formed on this substrate; it does not act on formed structure\. The melt hierarchy of §[8](https://arxiv.org/html/2607.27281#S8), in which formed compositions dissolve before their parts, is a different axis — destruction of what exists versus recovery of the ability to build — and the two results are consistent, not in tension\.\)Loss\-blindness: throughout the aging that kills inducibility, validation loss falls smoothly \(5\.46→1\.635\.46\\to 1\.63\) — standard monitoring is structurally blind to the site term, which is why the phenomenon has stayed invisible to ordinary telemetry \(Fig\.[10](https://arxiv.org/html/2607.27281#S7.F10)\)\.

![Refer to caption](https://arxiv.org/html/2607.27281v1/figures/fig9_shelf_life.png)Figure 9:Shelf life grows with scale; the occupation tax accumulates∼\\sim4×\\timesslower at 410M than 70M\.![Refer to caption](https://arxiv.org/html/2607.27281v1/figures/fig11_loss_blind.png)Figure 10:Loss monitoring is blind: validation loss falls healthily during exactly the waiting period in which the capability becomes permanently uninducible at fixed drive\.### 7\.1The deadline as a theorem: occupation in a controlled gated\-attention model

Everything above is measured\. This subsection reports that the site story is also*provable*— as a theorem in a controlled model — with a division of labor that deserves emphasis: the deadline’s*consequences*\(irrecoverability by data, repair by reset, invisibility in the loss\) requireno ergodicity assumption at all\. Confinement is the easy direction of stochastic analysis — a drift inequality plus bounded increments \(Hajek’s drift theorem\[[38](https://arxiv.org/html/2607.27281#bib.bib38)\]\) — in contrast to the rate law’s upper\-bound side, which carries Assumption[1](https://arxiv.org/html/2607.27281#Thmassumption1)\. The paper’s backward reading thus stands on the unconditional side of the mathematics\.

*The model*\(full detail in App\.[A](https://arxiv.org/html/2607.27281#A1)\):HHposition\-attention heads \(softmax scores overWWwindow offsets\); a data stream mixing*background skills*\(cue tokengi→g\_\{i\}\\tocopy the token at offsetoio\_\{i\}; traffic shareπi\\pi\_\{i\}\),*target queries*\(sharecc; theKK\-slot no\-partial\-credit task of Lemma[1](https://arxiv.org/html/2607.27281#Thmlemma1)\), and neutral substrate; a readout*gated by the current token*, with the substrate channel closed\.The gate is the model’s most idealized ingredient — it hands the model an oracle task identity\.The point of retaining it is sharp: in this model the difficulty of late learning is provably*pure capacity occupation*— even knowing exactly which task is which, a substrate short ofKKuncommitted heads cannot form the capability\. Each structural assumption has the empirical counterpart measured above \(commitment atr=0\.93r=0\.93; control ablations retaining 99% — the readout does approximate “read only the carrier”; reset 5/5\), which licenses the abstraction without claiming it as a theorem\.

###### Theorem 1\(Occupation: the silent deadline, in the gated model\)\.

Under assumptions \(G1\)–\(G8\) and feasibility/scale conditions C0–C4 of App\.[A](https://arxiv.org/html/2607.27281#A1), with formation requiringKKheads on distinct target slots:\(i\) \(deadline; uses the recruitment assumption\)the number of uncommitted\-or\-loosely\-pinned heads is non\-increasing with high probability over the recruitment window, and the deadlinetdeadt\_\{\\rm dead\}\(fewer thanKKsuch heads\) is a\.s\. finite, with𝔼​\[tdead\]≲\(H−K\+1\)/\(ρmin​p0\)\\mathbb\{E\}\[t\_\{\\rm dead\}\]\\lesssim\(H\-K\+1\)/\(\\rho\_\{\\min\}p\_\{0\}\)\.*We do not claim a constant per\-head recruitment hazard*: the traffic\-share structure of C0 \(μi∝πi\\mu\_\{i\}\\propto\\pi\_\{i\}\) implies the opposite, and we measure it below\.\(ii\) \(data cannot rescue; no mixing\)pasttdeadt\_\{\\rm dead\},P​\(formation within​T​steps\)≤H​K​T​e−θ​ΔP\(\\text\{formation within \}T\\text\{ steps\}\)\\leq HKT\\,e^\{\-\\theta\\Delta\}withθ=Ω​\(μ/\(η​G2\)\)\\theta=\\Omega\\\!\\big\(\\mu/\(\\eta G^\{2\}\)\\big\); the target shareccentersμ\\muonly linearly, so rescue by data requirescclarge enough to break the background’s own pinning margin — the catastrophic\-forgetting dual, not a gentle scaling\.\(iii\) \(reset rescues; no mixing\)re\-initializing the attention scores of anyKKheads restores the fresh\-substrate formation rate\.\(iv\) \(silence; no mixing\)pre\-formation the target loss equalsln⁡V\\ln Vplus a bounded junk term held near zero, and the background loss decreases throughtdeadt\_\{\\rm dead\}in steps of the same order at every pinning event: the deadline produces*no discontinuity, kink, or necessary precursor in any loss\-derived scalar*\. The step that kills learnability looks, in the loss, like every other step of ordinary background learning\.

*Proof chain*\(full proofs in App\.[A](https://arxiv.org/html/2607.27281#A1)\): pre\-formation, the target exerts on any single head only a pull proportional to a product of attention masses on the required slots — an iterated\-oscillation bound on full\-support Fourier coefficients, valid because the readout sees window contents only through head outputs — together with an exact decomposition of the cross\-entropy gradient into a*uniformizing potential*\(label\-free\) and a signal covariance; the attention simplex makes carried and target patterns exclusive; a linear\-response computation, every block carrying the\(1−pc\)\(1\-p\_\{c\}\)factor, gives a restoring drift toward the carried offset; an exact score\-difference identity turns the force ledger into a drift inequality; Hajek’s theorem converts drift into exponential confinement with*no ergodicity input*; and stochastic domination of the recruitment process yields the deadline\.

The theorem’s predictions, tested — including one we had to withdraw\.We ran the theorem’s own falsifiable consequences in the apparatus \(App\.[A\.7](https://arxiv.org/html/2607.27281#A1.SS7)\)\.*Confirmed, and sharper than the real\-model result that motivated it*: the repair is specific to the attention*score*channel\. From one aged substrate, re\-initializing the QK slices recovers formation 4\.3×\\times\(6/6 formed\), MLP does little \(0\.81×0\.81\\times\), and re\-initializing the OV slices is*worse than no reset at all*\(1\.76×1\.76\\timesslower\) — occupation lives in the attention pattern, so freeing the pattern is the repair, while resetting the value channel destroys learned structure and frees nothing\. This turns §[7](https://arxiv.org/html/2607.27281#S7)’s block reset \(5/5 vs 0/5\) into a prescription — reset QK, keep OV — which we then confirmed prospectively on the real substrate: on aged Pythia\-70M the QK arm ignites 6/6, the OV arm 0/6 \(identical to no reset\), and the block reset 6/6, i\.e\. exactly the QK arm and nothing more \(§[7](https://arxiv.org/html/2607.27281#S7)\)\.*Falsified*: we had additionally assumed a constant per\-head recruitment hazard, which would make occupation decay exponentially and the deadline scale aslog⁡H\\log H\. It does not: the hazard falls≈\\approx15×\\timesacross the sweep \(32→18→8→532\\to 18\\to 8\\to 5available slots\), and extrapolating the initial rate misses by∼\\sim1500×\\times\. The failed clause was an extra simplification that*contradicted our own traffic\-share model*— with pullμi∝πi\\mu\_\{i\}\\propto\\pi\_\{i\}, high\-traffic skills take heads first and the remainder recruit slowly, so the hazard must decline\. We withdraw the clause and thelog⁡H\\log Hprediction with it; the mechanism, the finiteness of the deadline, and clauses \(ii\)–\(iv\) are untouched\. The correction also resolves a tension we had flagged: Pythia’s saturating depletion profile \(1→0\.46→0\.451\\to 0\.46\\to 0\.45\) is the same declining\-hazard family, not a counterexample\.

Position in the plasticity literature\.The phenomenon is established\[[7](https://arxiv.org/html/2607.27281#bib.bib7),[8](https://arxiv.org/html/2607.27281#bib.bib8)\], the remedies are empirical\[[39](https://arxiv.org/html/2607.27281#bib.bib39)\], and the causes literature is explicitly partial — no single proposed mechanism suffices\[[9](https://arxiv.org/html/2607.27281#bib.bib9)\]\.Lyle et al\. \[[40](https://arxiv.org/html/2607.27281#bib.bib40)\]conjecture, via effective learning rate, that the dynamics of emergence and of plasticity loss are linked; we cite that as the conceptual precursor and supply the quantitative form the conjecture lacks: the*same*rate equation whose forward reading is §[6](https://arxiv.org/html/2607.27281#S6), with a law \(τ​\(S\)\\tau\(S\)\), a horizon \(tct\_\{c\}and its scale law\), a located and causally verified mechanism \(attention\-subspace commitment\), a repair \(targeted reset\), and a discriminator \(drive versus sites\) that rules out the drive\-side alternatives\. Theorem[1](https://arxiv.org/html/2607.27281#Thmtheorem1)adds the tier the causes literature is missing: a proof, in a controlled model, that occupation*must*produce exactly this phenomenology — deadline, data\-irrecoverability, reset\-repair, and silence\.

## 8Completing the equation: injected temperature and process control

The knockdown component ofDDin Eq\. \([1](https://arxiv.org/html/2607.27281#S1.E1)\) is dormant in native training \(§[5](https://arxiv.org/html/2607.27281#S5): nothing qualifies as a temperature, and at practical learning rates weight drift, not noise, is the only destroyer of formed structure\)\. Installing the missing term — an isotropic weight\-noise bath of magnitudeTT, a few lines of code — completes the equation and turns it into a control surface\. Everything in this section is measured in a controlled apparatus \(a conjunction\-circuit toy withK=2\.\.4K=2\.\.4, plus real\-Pythia induction circuits\), with constants frozen and out\-of\-sample tests preregistered\.

#### Why inject a temperature at all?

Since emergence needs no bath \(§[6](https://arxiv.org/html/2607.27281#S6): the drive is data\), one may ask whether this section adds anything data cannot\. It adds exactly the operations that a*monotone*drive cannot express\. Data can only push formation forward, in barrier order; it cannot*pause*a capability below threshold and release it on command \(pinning atϕs​s\\phi\_\{ss\}\), cannot*selectively erase*one circuit while sparing its parts \(the melt line — the primitive machine unlearning lacks\), and cannot*accelerate*a high\-barrier circuit past its own kinetics \(the nose, where annealing beats a gradient\-filter baseline\)\. The bath is also the theory’s*instrument*: melt points, the hysteresis window, dose–damage bookkeeping, andϕs​s​\(T\)\\phi\_\{ss\}\(T\)are measurements of the destruction term and of basin depths that native training simply cannot perform — no thermometer, no thermometry\. We therefore present these schedules as existence proofs that every term of Eq\. \([1](https://arxiv.org/html/2607.27281#S1.E1)\) is independently actuatable — the completion of the program — and as measurement instruments, not as a claim that production training should run with a bath on\.

### 8\.1The nose: barrier\-gated annealing

With a qualified temperature, generation gains a barrier\-assistance factor \(crossing aided by total noiseε0\+T\\varepsilon\_\{0\}\+T\) while destruction turns on above a damage onset\. These two monotonicities force the shape of the formation\-time curve:

###### Proposition 6\(Existence of a TTT nose, if and only if barrier\-gated\)\.

Let the generation rateg​\(T\)g\(T\)be continuous and strictly increasing inTT\(barrier assistance\), and the destruction rated​\(T\)d\(T\)be continuous, non\-decreasing,d≡0d\\equiv 0on\[0,Tdam\]\[0,T\_\{\\rm dam\}\]for some damage onsetTdam\>0T\_\{\\rm dam\}\>0, withg​\(Tceil\)=d​\(Tceil\)g\(T\_\{\\rm ceil\}\)=d\(T\_\{\\rm ceil\}\)at some finite ceilingTceil\>TdamT\_\{\\rm ceil\}\>T\_\{\\rm dam\}\. Define the formation timeτ​\(T\)=C/\(g​\(T\)−d​\(T\)\)\\tau\(T\)=C/\(g\(T\)\-d\(T\)\)on\[0,Tceil\)\[0,T\_\{\\rm ceil\}\)\. Then:\(a\)τ\\tauis strictly decreasing on\[0,Tdam\]\[0,T\_\{\\rm dam\}\]andτ​\(T\)→∞\\tau\(T\)\\to\\inftyasT→Tceil−T\\to T\_\{\\rm ceil\}^\{\-\}; henceτ\\tauattains an interior minimum \(a*nose*\) in\[Tdam,Tceil\)\[T\_\{\\rm dam\},T\_\{\\rm ceil\}\)\.\(b\)If instead the barrier is negligible —ggconstant inTT— thenτ\\tauis non\-decreasing on all of\[0,Tceil\)\[0,T\_\{\\rm ceil\}\)and the minimum is atT=0T=0: cold is fastest, and no nose exists\. Thus an interior nose exists precisely when there is a barrier for temperature to assist\.

###### Proof\.

\(a\) On\[0,Tdam\]\[0,T\_\{\\rm dam\}\],d=0d=0andggis strictly increasing, soτ=C/g\\tau=C/gis strictly decreasing there; andg−d→0\+g\-d\\to 0^\{\+\}asT→Tceil−T\\to T\_\{\\rm ceil\}^\{\-\}, soτ→∞\\tau\\to\\infty\. Sinceτ\\tauis continuous on\[Tdam,Tceil\)\[T\_\{\\rm dam\},T\_\{\\rm ceil\}\)and diverges at the right end, it attains a minimum on that interval at some finiteT∗T^\{\*\}; by the strict decrease on\[0,Tdam\]\[0,T\_\{\\rm dam\}\],T∗T^\{\*\}is the global minimizer over\[0,Tceil\)\[0,T\_\{\\rm ceil\}\)and satisfiesTdam≤T∗<TceilT\_\{\\rm dam\}\\leq T^\{\*\}<T\_\{\\rm ceil\}— an interior point of the temperature domain, i\.e\. a nose\. \(b\) Withggconstant,g−dg\-dis non\-increasing, soτ\\tauis non\-decreasing; the minimum is atT=0T=0\. ∎

The measured curves realize both branches exactly as the proposition sorts them:K=2K=2\(barrier assist capped at1\.11×1\.11\\times\) shows no nose in six curves — cold fastest, the \(b\) branch;K=3K=3\(assist2\.17×2\.17\\times\) shows a clean nose atT≈0\.003T\\approx 0\.003, non\-overlapping across seeds, depth 23–41%;K=4K=4confirms\. The full C\-curve formula \(attempt×\\timesassistance×\\timesceiling proximity\), with constants frozen on\-axis, predicts a held\-out drive lane at median 13\.9% error and a prospective double\-nose position hit at 14\.3% — and yields the intrinsic noise scale of SGD itself,ε0≈5×10−4\\varepsilon\_\{0\}\\approx 5\\times 10^\{\-4\}, as a fit constant \(the quantitative version of “the cold arm does not freeze”\)\. As an application, nose\-temperature annealing accelerates a chosen high\-barrier circuit∼\\sim2×\\timesover the Grokfast baseline\[[41](https://arxiv.org/html/2607.27281#bib.bib41)\]— a mechanism\-guided schedule versus a gradient filter\. And the construction is a genuine TTT diagram, not only a formation\-time curve: post\-nucleation growth follows Johnson–Mehl–Avrami–Kolmogorov kinetics with measured Avrami exponentn≈1\.16n\\approx 1\.16\(single\-front growth\)\[[42](https://arxiv.org/html/2607.27281#bib.bib42),[25](https://arxiv.org/html/2607.27281#bib.bib25)\], supplying the transformed\-fraction axisX​\(t,T\)X\(t,T\)that the metallurgical analogy requires\.

### 8\.2The melt line: destruction is a weakest\-link law

Formation and destruction read the same part countKKthrough opposite logical gates, and this duality is provable\.

###### Proposition 7\(Weakest\-link melting, and the build/break duality\)\.

Let a circuit function if and only if allKKof its links function \(the defining conjunction\), and let linkiifail when the perturbation exceeds a thresholdσi\\sigma\_\{i\}\. Then:\(a\)the circuit’s melt threshold isσ∗=mini⁡σi\\sigma^\{\*\}=\\min\_\{i\}\\sigma\_\{i\}: it fails as soon as any link fails;\(b\)for nested circuitsS⊂S′S\\subset S^\{\\prime\},σ∗​\(S′\)≤σ∗​\(S\)\\sigma^\{\*\}\(S^\{\\prime\}\)\\leq\\sigma^\{\*\}\(S\): composition depth can only lower, never raise, the melt point;\(c\)\(duality\) over the same part set, the formation cost is a*sum*\(Prop\.[1](https://arxiv.org/html/2607.27281#Thmproposition1): all parts must co\-occur — an AND for building\) while the destruction threshold is a*min*\(any part’s failure suffices — an OR for breaking\)\. Compositional circuits are therefore simultaneously hard to build and easy to break, with both facts derived from the sameKK\.

###### Proof\.

\(a\) Failure of the circuit is the union⋃i\{link​i​fails\}\\bigcup\_\{i\}\\\{\\text\{link \}i\\text\{ fails\}\\\}, which first occurs at perturbationmini⁡σi\\min\_\{i\}\\sigma\_\{i\}\. \(b\) The minimum over a superset is at most the minimum over the subset\. \(c\) is \(a\) juxtaposed with Prop\.[1](https://arxiv.org/html/2607.27281#Thmproposition1): the intersection event prices formation as a sum of exponents; the union event prices destruction as a minimum of thresholds\. ∎

Measured, at circuit grain and across scale: melt thresholds under injected weight noise order parts≫\\ggcompositions in 20/21 model–pair tests across both families \(2\.5–6×\\times; one 8% local inversion reported as such; Fig\.[11](https://arxiv.org/html/2607.27281#S8.F11)\); on real Pythia the previous\-token head melts at 5\.3–6\.4×\\timesthe induction head’s threshold at every scale — both are near scale\-invariant*material constants*\(induction: 0\.0017–0\.0025 across 31M–410M synthetic; 0\.00028–0\.00063 real\-text\), and their ratio is a constant of the composition\. The parts survive to perturbations where the composition’s inter\-layer*wiring*has long failed — the weakest link is the composition itself, clause \(b\) in the wild\. Melting is first\-order\-like \(cliff, not slope\), sits above the formation ceiling \(a hysteresis window, measured \(0\.03, 0\.05\) in the toy\), and melt points scale with maintenance strength \(3–8×\\timesbetween data regimes\) — a dynamic, driven melting point, as the non\-equilibrium branch requires\. Applications:*selective dissolution*— atT=0\.001T=0\.001, induction dies while the previous\-token part survives, demonstrated on real Pythia at three scales; and a*risk rule*for practitioners — assess a model’s fragility by its longest chain, not its parts\.

![Refer to caption](https://arxiv.org/html/2607.27281v1/figures/fig6_lifo.png)Figure 11:The reverse ladder: melt thresholds order parts≫\\ggcompositions at every scale and both families; last formed, first destroyed — Prop\.[7](https://arxiv.org/html/2607.27281#Thmproposition7)\(b\) measured\.
### 8\.3Glass, quench, and the release protocol

Between the nose and the melt line lies the third control primitive:*pinning*\. Holding a bath at window temperature pins every sub\-threshold structure at a steady\-state order parameterϕs​s​\(T\)\\phi\_\{ss\}\(T\)— measured as a clean decreasing spectrum \(0\.373/0\.303/0\.204/0\.164/0\.1060\.373/0\.303/0\.204/0\.164/0\.106atT=0\.003\.\.0\.02T=0\.003\.\.0\.02, seed spread±0\.02\\pm 0\.02\) with∼\\sim100×\\timesretardation of ordering, the driven\-steady\-state picture of irradiated alloys\[[43](https://arxiv.org/html/2607.27281#bib.bib43)\]\. Release is*not*merely removing the bath: on real text, pure\-temperature release fails \(0/16\) because quenching freezes the disorder in — the burned state has weights inflated 5–24×\\timesand attention entropy collapsed to saturation\. The working protocol is quench*plus mechanism surgery*: rescaling attention\-weight RMS to healthy values at release \(direction preserved\) recovers 6/6, and attention\-only surgery re\-ignites in∼\\sim500 steps — faster than a virgin substrate, the held heat converted into a head start\. Damage bookkeeping is a dose law: below the knockdown line, zero accumulated damage; above it, damage∝\\proptoknockdown rate×\\timesduration \(R2R^\{2\}0\.952\) — discrete Poisson\-avalanche knockdowns, not diffusive wear, consistent with the jump\-type destruction term\.

The control palette, then:*seeds and sites choose which*\(a transplanted sub\-critical seed re\-orders barriers — the only lawful way to invert the temperature\-forced order, 8/8\);*drive chooses whether and how fast*\(floor and throttle\);*temperature chooses when*\(pin atϕs​s\\phi\_\{ss\}, release by quench\-plus\-surgery\);*the melt line chooses what survives*\. Each primitive is a term of Eq\. \([1](https://arxiv.org/html/2607.27281#S1.E1)\)\.

## 9The engine, the synthesis, and the boundary

Two duties remain to the spine: measuring the equation’s one non\-obvious*input*\(KK, the part count\), and testing its*structure*\(the factorization\) where all axes move at once\.

The engine\.Given only a behavioral probe, single\-head ablation at a just\-post\-ignition checkpoint returns the circuit’s constituent parts; tracking each constituent’s attention\-pattern crystallinity across checkpoints reveals which arrived first — that part is the precursor, and its count isKK\(Fig\.[12](https://arxiv.org/html/2607.27281#S9.F12)\)\. Blind runs recover the known circuits in 10/11 capability–scale pairs\. The count is physically load\-bearing, not descriptive: induction \(K=2K=2,C=1\.24C=1\.24\) and IOI away from the wall \(K=4K=4,C=2\.1C=2\.1\) shareβ≈0\.23\\beta\\approx 0\.23, whereas the textbook depth assignment \(K=3K=3for IOI\) is inconsistent with the clock — the machine count is the one the physics accepts\. Nor is the count circular:KKis measured by ablation at a*single*checkpoint, with no access to formation times, and only then inserted into the clock — an independently measured input that could have failed to unify the constants \(the textbookK=3K\{=\}3*does*fail\), not a parameter tuned to make them agree\.

The count’s domain of validity, measured\.We delimited it directly, by sweeping single\-head ablation of the copy capability across four scales and two checkpoints \(behavioural probe; chance≈0\\approx 0\)\. Two facts emerge, and the second is a real limitation\.*First*, the count is right where the theory uses it: at a just\-post\-ignition checkpoint in the smallest model the individually\-necessary set is exactly\{\\\{previous\-token head, induction head\}\\\}—K=2K=2, both known parts recovered by a criterion that had no access to their identity — and at 160M it is again of size 2\.*Second*, the criterion degrades with redundancy: by the final checkpoint at 410M and 1B*no single head*clears the necessity threshold \(K→0K\\to 0; ablating any one head leaves accuracy at0\.950\.95against a0\.950\.95baseline\), because the capability has delocalized into interchangeable copies\. A redundancy\-robust variant — the smallest*group*whose joint ablation destroys the capability — grows from 1–2 in the small/early models to 3 in the large/late ones, quantifying the redundancy, but it measures the*destruction*side \(a minimum over links, Prop\.[7](https://arxiv.org/html/2607.27281#Thmproposition7)\) and therefore does not substitute for the formation\-side count\. The honest statement is thus sharper than before: ablation\-based part counting is valid*before delocalization*— which is exactly the engine’s stated operating rule — and beyond it the formation\-sideKKmust be carried by formation\-time evidence \(the clock’s cross\-circuit consistency\), not by ablation\. That the causal precursor \(the previous\-token head\) remains in the necessary set at every scale and checkpoint where the criterion is defined is what the clock actually relies on, and it is measured to hold throughout\.

Relatedly, the same sweep measures the AND\-gate structure of real circuits directly: removing*one*identified part collapses the capability to a median5%5\\%of baseline \(range 2–42%\), whereas removing a randomly chosen non\-part head leaves it at 99–102% of baseline\. Graceful, proportional degradation — the signature of a partial\-credit architecture — is absent; the collapse is what a conjunction predicts, and it is not an artifact of ablation*per se*, since the control ablations do nothing\. The engine’s one systematic failure is itself confirmatory: at late checkpoints, formed circuits delocalize into redundant copies and single\-head ablation loses signal — fixing the operating rule \(dissect shortly after ignition\) and independently evidencing the redundancy physics\.

The AND\-gate test at scale, on the engine’s own operating rule\.Applying that rule turns the test from anecdote into statistics\. Sweeping three scales \(70M/160M/410M\)×\\timesfive checkpoints \(1k–16k steps, i\.e\. near ignition rather than at the end of training\)×\\timesseven capabilities gives 120 cells, of which 32 are*discriminating*— capability present, and at least two individually necessary parts, so that conjunction and partial credit predict different things \(K=1K\{=\}1cells are excluded a priori: with one part both models predict collapse\)\. Removing one part leaves a median16\.7%16\.7\\%of the capability; removing a random non\-part head leaves100%100\\%\. Against each cell’s own partial\-credit prediction \(1−1/K1\-1/K, i\.e\. 50–83% retained\),32 of 32 cells fall below, sign testp=2\.3×10−10p=2\.3\\times 10^\{\-10\}; the stricter preregistered criterion \(≤25%\\leq 25\\%retained*and*controls≥80%\\geq 80\\%\) passes 22 of 32, the remainder landing at 25–48% — still below partial credit, not above it\. Part counts spanK=2K=2–66across IOI, subject–verb agreement, greater\-than, in\-context learning, word\-pair recall, counting, and copying\. Six of the 32 were recovered by a preregistered secondary read\-out: where baseline accuracy saturates at 100% the instrument has no dynamic range, so we score the unthresholded margin \(accuracy is the sign of the margin\) — a resolution fix of the same kind as measuring probes in fp32, recorded per cell so the two read\-outs are never mixed\.

![Refer to caption](https://arxiv.org/html/2607.27281v1/figures/fig7_engine.png)Figure 12:The engine: ablation finds the parts; the earliest\-crystallizing constituent is the precursor\.The synthesis, and the boundary as a signed prediction\.Full frozen constants \(continued\-training regime\): sites decay slope0\.61±0\.090\.61\\pm 0\.09; batch exponent−0\.329±0\.046\-0\.329\\pm 0\.046\(R2R^\{2\}0\.96\); drive exponentγ=0\.62±0\.01\\gamma=0\.62\\pm 0\.01\(R2R^\{2\}0\.999\); barrierβ=0\.23±0\.05\\beta=0\.23\\pm 0\.05\(Pythia, off\-wall\); melt thresholds per Fig\.[11](https://arxiv.org/html/2607.27281#S8.F11)\. Freezing all single\-axis terms and predicting three never\-run combinations in which all three axes deviate at once: 8\.4% and 1\.3% error in the bulk; 112% at the near\-floor point, the error growing monotonically with floor proximity \(error∝\(c−c0\)−3\.2±0\.9\\mathrm\{error\}\\propto\(c\-c\_\{0\}\)^\{\-3\.2\\pm 0\.9\},R2R^\{2\}0\.92 over three points, stated as such\)\. By Corollary[1](https://arxiv.org/html/2607.27281#Thmcorollary1)this is not an unexplained failure but the predicted signature of part dependence where capacity is shared: the factorized law holds where independence holds and couples at the boundary — the standard structure of nucleation kinetics away from versus near a spinodal — and it is a standing falsifiable commitment: any new combination’s factorization error should be set by its floor distance\.

## 10Why one law survives a complex model: the rate\-determining step

A fair objection runs: a language model contains many circuits, coupled far more richly than a bag ofKKparts — dataflow chains across layers, fan\-in, cooperative and competitive interactions — so why should a single exponential law with a singleKKdescribe any real capability? The answer is not ours to claim\. It is the*rate\-determining step*principle of chemical kinetics\[[44](https://arxiv.org/html/2607.27281#bib.bib44)\]: a multi\-step mechanism, however elaborate, exhibits the rate law of its slowest step, and the complexity of the remaining steps collapses at the bottleneck\. One Arrhenius form underwrites thousands of distinct mechanisms for exactly this reason\. What is ours is the demonstration, measurement by measurement, that capability formation*has*such a bottleneck, that the bottleneck is the subset\-signal\-free joint alignment, and that its dimensionality is observable:

*\(i\) The cheap steps are cheap and happen early\.*Circuit stages with independent utility — parts that carry their own gradient signal — form first and fast: the mechanism precedes the behavior in 15/15 model–task pairs \(§[6](https://arxiv.org/html/2607.27281#S6)\), and an exact\-depth from\-scratch ladder ignites strictly in order, depths1→41\\to 4, in 6/6 seeds\.*\(ii\) The expensive step is the joint alignment — and this dichotomy is itself a theorem\.*Where no strict subset carries signal, a single added part multiplies formation time by\>\>11×\\times\(the parity family\), against 5\.8×\\timeswhen subset signal exists \(the summed family\) — the bottleneck’s cost dwarfs the stages’\. The staircase/leap theorems give this measured dichotomy its mathematical backbone: subset\-signal\-at\-every\-stage targets are SGD\-learnable inO​\(d\)O\(d\)samples \(an iff in the mean\-field regime\), while anℓ\\ell\-wise joint step costsΘ~​\(dmax⁡\(ℓ−1,1\)\)\\tilde\{\\Theta\}\(d^\{\\max\(\\ell\-1,1\)\}\)where proven\[[21](https://arxiv.org/html/2607.27281#bib.bib21),[22](https://arxiv.org/html/2607.27281#bib.bib22)\]— cheap stages and an expensive bottleneck are not our metaphor but the proven structure of SGD learning on this function class\.*\(iii\) Everything upstream of the bottleneck is kinetically invisible\.*This is the direct rate\-determining\-step measurement: in the missing\-parts grid \(EK\-5\), waiting time depends only on the size of the remaining joint step, not on how much circuit already stands — rows flat to 1\.1–1\.4 across totalKK\.*\(iv\) The wait is governed by one barrier, not a chain of comparable ones\.*The waiting\-time shape is exponential \(κ→1\.2\\kappa\\to 1\.2, EK\-1\); a sequence ofkkcomparable activated steps would instead show an Erlang shapeκ≈k\\kappa\\approx k\.*\(v\) The bottleneck’s dimensionality is measurable and load\-bearing\.*The engine’s causally\-countedKKis what unifies the clock across different circuits \(K=2K\{=\}2andK=4K\{=\}4sharing oneβ\\beta\) while the textbook count fails \(§[9](https://arxiv.org/html/2607.27281#S9)\)\.

Nor do circuit–circuit interactions reopen unbounded complexity: each measured interaction enters as a*correction with a measured form*, not a new law\. Cross\-circuit catalysis makes the rate linear in neighbor density \(secondary nucleation;R2R^\{2\}0\.88 on real models in prior work\[[25](https://arxiv.org/html/2607.27281#bib.bib25)\]\); competition for shared capacity renormalizes constants rather than the form \(in a two\-skill furnace the loser’s formation time is renormalized∼\\sim20×\\timeswhile each skill alone obeys its own law\); and the near\-wall coupling carries its own scaling,error∝\(c−c0\)−3\.2\\mathrm\{error\}\\propto\(c\-c\_\{0\}\)^\{\-3\.2\}\(§[9](https://arxiv.org/html/2607.27281#S9)\) — the theory meters its own domain of validity\.

The epistemic position this leaves us in is, again, not novel; it is the standard position of physical metallurgy\[[45](https://arxiv.org/html/2607.27281#bib.bib45)\]\. Nobody derives a steel’s TTT diagram from quantum mechanics — steel is far too complex — yet the framework fixes the diagram’s*structure*, a handful of measured points fix the alloy’s constants, and the result runs industry\. Here the structure \(floor, clock, shelf life, melt line\) transfers across families, the per\-family effectiveβ\\betais the material constant, and one cheap anchor — a single attention readout at 0\.36% of training — calibrates a family, after which frozen constants predict held\-out models to median 5% \(§[6](https://arxiv.org/html/2607.27281#S6)\)\. The framework would break, and is built to break visibly, if the rate\-limiting step showed non\-exponential kinetics, if the precursor stopped dating the composition, or if the causal part\-count stopped unifying the constants; none of these has happened in any system tested\.

## 11Negative results and reproducibility

A design lesson worth more than the result it cost us\.Synthetic sequence tasks harbour*algebraic shortcuts*: alternative solutions of much lower part count that agree with the intended target on the training distribution\. We have now hit this family three times — a closed\-loop collapse in which a fully\-driven conjunction degenerates into single\-token copying; a look\-up\-table task solved by periodic orbits; and the summed design’s two\-term recurrence documented above, which silently replaced the intendedKK\-part solution fromK=3K\{=\}3upward and cost us a per\-part barrier number we have withdrawn\. The general prescription we now follow, and recommend: for any synthetic target, \(i\) search for algebraic identities that relate the target at nearby positions, since these generate low\-part\-count alternative solutions; \(ii\) build a*discriminating probe*— evaluation positions where the shortcut and the intended solution disagree — and report both scores, never the aggregate alone; \(iii\) prefer designs that are provably shortcut\-free \(our query\-parity design isolates each query behind a gap, so consecutive targets are statistically independent and no cross\-position identity exists\)\. A task whose intended solution is never verified against its cheapest alternative is not a measurement of that solution\.

The complete failure log is part of the evidence\. IOI is un\-ignitable in every continued\-training arm \(its barrier exceeds the regime’s power, consistent with its nativet∗t^\{\*\}\); greater\-than is unwithholdable \(plain text already exceeds its floor — capabilities split into withholdable and unwithholdable by food scarcity\); a global entropy regularizer does not rescue aged substrates \(it also suppresses the peaked circuit it should enable\); newborn circuits are narrow\-band \(food at repetition period 128 does not serve a period\-64 probe — a trap hit twice, fixed, and itself evidence that young circuits are distance\-specific before delocalization\); four attempts at a precursor dose–response were all confounded by precursor redundancy \(the causal case rests on the switch and the resets instead\); pure\-temperature release fails without surgery \(§[8](https://arxiv.org/html/2607.27281#S8)\); three execution artifacts were caught by internal audits \(a mirror resolving every revision to one commit — detectable because step\-0 weights scored as trained;\.bin\-only checkpoints refused under a torch CVE policy; an ignition\-detector edge case\), each with detection criteria for reproducers; and one publicly failed frozen prediction \(the occupation tax was frozen as saturating and measured as divergent\) is retained because a fitted story would not contain it\. Every number is recomputable from released per\-experiment JSON files and scripts; the EK premise\-test suite \(§[4\.4](https://arxiv.org/html/2607.27281#S4.SS4)\) was likewise designed and its budgets frozen before running\.

## 12Honest scope

≤\\leq1\.4B parameters; two families \(the only ones with dense public checkpoints\); causal constants belong to a continued\-training regime whose token flux is far below native pretraining \(the from\-scratch reproduction removes the pretrained\-substrate confound but not the flux gap\); capabilities are mechanism\-level plus one task\-level demonstration — no benchmark\-suite prediction yet; switch/shelf/reset arms atn=5n=5, several others atn=2n=2–3; single hardware/framework stack; fp32 is required for*probe evaluation only*\(softmax scores collapse under bf16 underflow\) — models trained in low precision are probed in fp32, so the constraint is on the measurement instrument, not on training practice; the diffusion\-approximation regime of Step 1 is itself a scope condition \(production optimizer stacks deform the SDE\); the engine’s count for induction is less crisp than for IOI; direct\-KKmeasurements spanK=2K=2–44, and the wide\-KKladder \(Remark[4](https://arxiv.org/html/2607.27281#Thmremark4)ff\.\) is the stated next experiment;β\\betais single\-valued only away from the capacity wall; control results are toy\-plus\-real\-Pythia, not production\-scale; the alignment\-fragility reading \(safety training as the top rung — last in, first out\[[46](https://arxiv.org/html/2607.27281#bib.bib46)\]\) is a corollary of Prop\.[7](https://arxiv.org/html/2607.27281#Thmproposition7)\(b\) plus literature, not a direct experiment\. The mathematical status is exactly as marked, in three grades\. Proved: the no\-partial\-credit structure of the apparatus \(Lemma[1](https://arxiv.org/html/2607.27281#Thmlemma1)\); the exponential\-in\-KKcost for the task class \(query\-complexity and leap theorems, §[2](https://arxiv.org/html/2607.27281#S2)A, with their stated regimes: SQ/CSQ algorithm classes, single\-pass training, large ambient dimension — and no such theorem yet exists for transformers themselves\); and the occupation theorem for the backward reading \(Theorem[1](https://arxiv.org/html/2607.27281#Thmtheorem1), App\.[A](https://arxiv.org/html/2607.27281#A1)\) — deadline, data\-irrecoverability, reset\-repair, and loss\-silence, its confinement side free of any mixing assumption, at the price of the model’s stated idealizations \(positional attention, oracle gating, the C0 feasibility condition\), each idealization flagged in the appendix with its empirical counterpart\. Conditional: the nucleation\-form propositions, exact given Assumptions[1](https://arxiv.org/html/2607.27281#Thmassumption1)–[3](https://arxiv.org/html/2607.27281#Thmassumption3); Assumption[1](https://arxiv.org/html/2607.27281#Thmassumption1)\(b\) — mixing for transformer SGD on formation timescales — now burdens*only the physical reading*\(temperature, supersaturation, melting\), is beyond present mathematics for anyone, and proving it would upgrade that reading to a theorem: an invitation, not a concession\. Measured: everything else, preregistered\.

## 13Related work: the equilibrium picture, and what a rate adds

Equilibrium theories of training structure\.Singular learning theory and developmental interpretability characterize training stages as Bayesian free\-energy transitions and detect them post hoc via the local learning coefficient\[[1](https://arxiv.org/html/2607.27281#bib.bib1),[2](https://arxiv.org/html/2607.27281#bib.bib2),[3](https://arxiv.org/html/2607.27281#bib.bib3)\]; grokking has an equilibrium first\-order model\[[5](https://arxiv.org/html/2607.27281#bib.bib5)\]and a Kramers\-escape model in linear networks\[[6](https://arxiv.org/html/2607.27281#bib.bib6)\]\. These supply the statics\. None yields a formation\-*rate*law, a precursor clock, an aging horizon, or a control schedule — a free energy is not a rate\.Predicting emergence\.Aoyama et al\. \[[47](https://arxiv.org/html/2607.27281#bib.bib47)\]predict induction\-head arrival from data statistics with a throughput equation; the prediction is real but mechanism\-free\. Our clock is derived from a barrier law, reads an*internal*precursor, and itsKK\-dependence is forced, not fit; and whereLavie et al\. \[[36](https://arxiv.org/html/2607.27281#bib.bib36)\]argue softmax\-attention emergence is intrinsically unpredictable andBaherwani et al\. \[[37](https://arxiv.org/html/2607.27281#bib.bib37)\]that timing is initialization\-random, the prospective clock results are a direct counterexample at the capability class we cover\.Emergence phenomenology\.Wei et al\. \[[48](https://arxiv.org/html/2607.27281#bib.bib48)\], Schaeffer et al\. \[[49](https://arxiv.org/html/2607.27281#bib.bib49)\], Ganguli et al\. \[[13](https://arxiv.org/html/2607.27281#bib.bib13)\]debate description;Olsson et al\. \[[32](https://arxiv.org/html/2607.27281#bib.bib32)\], Elhage et al\. \[[50](https://arxiv.org/html/2607.27281#bib.bib50)\]identify the induction circuit and its phase change;Reddy \[[51](https://arxiv.org/html/2607.27281#bib.bib51)\], Singh et al\. \[[52](https://arxiv.org/html/2607.27281#bib.bib52)\], Hoogland et al\. \[[2](https://arxiv.org/html/2607.27281#bib.bib2)\]model and detect its development\. We add the quantitative ratio law, frozen\-constant prospective validation, cross\-family transfer, causal steering, and the plasticity connection\.Data mixing\.Gu et al\. \[[4](https://arxiv.org/html/2607.27281#bib.bib4)\]document a mixing\-ratio threshold and attribute it to capacity allocation; our floor carries the nucleation signatures \(intensivity, near\-floor divergence, scale trend\) in a mechanistic order parameter \(Prop\.[4](https://arxiv.org/html/2607.27281#Thmproposition4)\);Xie et al\. \[[10](https://arxiv.org/html/2607.27281#bib.bib10)\], Liu et al\. \[[11](https://arxiv.org/html/2607.27281#bib.bib11)\], Ye et al\. \[[12](https://arxiv.org/html/2607.27281#bib.bib12)\]optimize loss, not which capabilities form\.Loss of plasticity\.The phenomenon and remedies are established\[[7](https://arxiv.org/html/2607.27281#bib.bib7),[8](https://arxiv.org/html/2607.27281#bib.bib8),[39](https://arxiv.org/html/2607.27281#bib.bib39)\]; the causes literature is explicitly partial\[[9](https://arxiv.org/html/2607.27281#bib.bib9)\];Lyle et al\. \[[40](https://arxiv.org/html/2607.27281#bib.bib40)\]conjecture the emergence–plasticity link via effective learning rate\. We supply the quantitative form: one rate equation with law, horizon, located mechanism, repair, and the site/drive discriminator \(Prop\.[5](https://arxiv.org/html/2607.27281#Thmproposition5)\)\. Resets as a*remedy*are standard in that literature\[[7](https://arxiv.org/html/2607.27281#bib.bib7),[39](https://arxiv.org/html/2607.27281#bib.bib39)\]; what is new here is not the operation but the causal localization of*which*weights \(attention — with MLP and embedding controls discriminated causally\),*why*\(commitment,r=0\.93r=0\.93\), and*when*\(the horizon and its scale law\)\.Non\-equilibrium SGD\.Broken detailed balance and irreversibility are established\[[30](https://arxiv.org/html/2607.27281#bib.bib30),[31](https://arxiv.org/html/2607.27281#bib.bib31)\]; Yaida’s fluctuation–dissipation relations are a measurement tool at stationarity\[[53](https://arxiv.org/html/2607.27281#bib.bib53)\]; we add the FDT*qualification*test, its six\-knob failed verdict, and the injected\-bath prescription\.Control baselines\.Injected\-noise training is itself classical — simulated annealing, gradient noise\[[54](https://arxiv.org/html/2607.27281#bib.bib54)\], SGLD — but is framed there as optimization or exploration; none applies a fluctuation–dissipation qualification, none selects*which*circuit forms, and none demonstrates pinning or selective dissolution — the operations, not the noise, are the contribution\. Grokfast\[[41](https://arxiv.org/html/2607.27281#bib.bib41)\]accelerates grokking by gradient filtering without selecting circuits; dual\-process steering\[[55](https://arxiv.org/html/2607.27281#bib.bib55)\]switches mechanisms by discrete forgetting; our schedules are continuous, bidirectional, and derived from the rate equation\.Fragility at coarse grain\.Layer\- and module\-level sensitivity to perturbation and quantization is known; the circuit\-grain melt points, their cross\-scale constancy, and the build/break duality \(Prop\.[7](https://arxiv.org/html/2607.27281#Thmproposition7)\) are new\.Metaphor\.A crystallization framing of training has been proposed qualitatively\[[56](https://arxiv.org/html/2607.27281#bib.bib56)\]; this program makes it quantitative and falsifiable\.Our own prior workdevelops single\-circuit nucleation with measured Langevin structure\[[25](https://arxiv.org/html/2607.27281#bib.bib25)\]; this paper is the generalization to the full life cycle and the control layer\.

## 14Conclusion

Training has had a phase diagram; this paper gives it a TTT diagram\. One rate equation — sites×\\timesattempts×\\timessupersaturation×\\timesbarrier, minus destruction — derived under stated assumptions and measured factor by factor, carries a capability from its first invisible precursor through ignition, aging, and, if one chooses, controlled dissolution\. Emergence is the equation read forward: a floor, a throttle, and a clock that dates arrival to median 5% before the capability exists\. Loss of plasticity is the same equation read backward: a tax, a horizon, a located mechanism, a repair — and, in a controlled model, a theorem — the quantitative law a decade of remedies has been missing\. Control is the equation completed: install the temperature SGD lacks, and circuits can be accelerated at the nose, pinned in the glass, released on command, and dissolved selectively at their melt points — scheduled, in short, like the heat treatment of an alloy\. The mathematics is exactly as strong as its premises, and the premises are the experiments: joint alignment is a product law \(measured\), the barrier is additive \(measured, with its own deviation theory\), the waiting is memoryless \(measured\), and the one unproved premise — mixing for transformer SGD — is stated as the field’s next theorem to want\.

## Appendix AThe occupation theorem: model, assumptions, and full proofs

This appendix proves Theorem[1](https://arxiv.org/html/2607.27281#Thmtheorem1)\. Every step is graded:*exact*\(arithmetic or algebraic identity\),*in\-model theorem*\(proved under the stated assumptions\),*cited theorem*\(an external result with its conditions verified\), or*explicit assumption*\. The headline structural fact bears repeating: clauses \(ii\)–\(iv\) — the consequences of occupation — use only a drift inequality and bounded increments \(Hajek\[[38](https://arxiv.org/html/2607.27281#bib.bib38)\]\); no ergodicity or mixing enters\. Only clause \(i\), the deadline’s*arrival*, uses the recruitment assumption \(G8\), whose empirical counterpart is the measured occupation of §[7](https://arxiv.org/html/2607.27281#S7)\.

### A\.1Model

Data\.Each sequence position belongs independently to one of: a*background skill*i∈\{1\.\.M\}i\\in\\\{1\.\.M\\\}\(shareπi\\pi\_\{i\}\): a cue tokengig\_\{i\}appears and the correct next token isxt−oix\_\{t\-o\_\{i\}\}, a copy from fixed offsetoio\_\{i\}; the*target*\(sharecc\): a query tokenQQappears and the correct next token isY=\(∑j∈Rxt−j\)modVY=\\big\(\\sum\_\{j\\in R\}x\_\{t\-j\}\\big\)\\bmod VwithR⊂\{1\.\.W\}R\\subset\\\{1\.\.W\\\},\|R\|=K≥2\|R\|=K\\geq 2, window contents i\.i\.d\. uniform onℤV\\mathbb\{Z\}\_\{V\}\(the apparatus of Lemma[1](https://arxiv.org/html/2607.27281#Thmlemma1)\); or*neutral substrate*\(remaining share\): i\.i\.d\. uniform tokens\.

Architecture\.One layer ofHHheads\. Headhhcarries positional scoress\(h\)∈ℝWs^\{\(h\)\}\\in\\mathbb\{R\}^\{W\}with attentiona\(h\)=softmax​\(s\(h\)\)a^\{\(h\)\}=\\mathrm\{softmax\}\(s^\{\(h\)\}\), and outputsuh=∑jaj\(h\)​e​\(xt−j\)u\_\{h\}=\\sum\_\{j\}a^\{\(h\)\}\_\{j\}e\(x\_\{t\-j\}\)witheethe one\-hot value embedding; thusuh∈conv​\{e1,…,eV\}u\_\{h\}\\in\\mathrm\{conv\}\\\{e\_\{1\},\\dots,e\_\{V\}\\\}and the joint domain𝒰:=\{\(u1\.\.uH\)\}\\mathcal\{U\}:=\\\{\(u\_\{1\}\.\.u\_\{H\}\)\\\}is convex and compact\. The readout is*gated by the current token*:z=F​\(u;g\)z=F\(u;\\,g\), with a background channel \(value matrixBB, plus a bounded higher\-order partF2F\_\{2\}\) on cue tokens, a target channelFQF\_\{Q\}onQQ, and the substrate channel*closed*\(zzconstant, independent ofuu\)\.

Training\.SGD with learning rateη\\eta, gradients clipped in norm atGG, minibatches sampled unbiasedly from the stream\.

Definitions\.Headhh*carries*skilliiifaoi\(h\)≥1−δ0a^\{\(h\)\}\_\{o\_\{i\}\}\\geq 1\-\\delta\_\{0\}, the open\-channel value margin is positive \(G4\), andhhis the unique carrier \(zero\-ablatinghhcollapses the margin — the operational, measured definition\)\. Withμi\\mu\_\{i\}as in C0 below, a carrier is*firmly pinned*ifμi\>0\\mu\_\{i\}\>0\.Navail​\(t\):=N\_\{\\rm avail\}\(t\):=number of heads not firmly pinned;tdead:=inf\{t:Navail​\(t\)<K\}t\_\{\\rm dead\}:=\\inf\\\{t:N\_\{\\rm avail\}\(t\)<K\\\}\. Serving a target slotrrrequiresar\(h\)≥γneed\>δ0a^\{\(h\)\}\_\{r\}\\geq\\gamma\_\{\\rm need\}\>\\delta\_\{0\};*formation*==KKheads serving theKKslots ofRR\. For a carrier, the score\-gap coordinate isξj:=sj\(h\)−soi\(h\)\\xi\_\{j\}:=s^\{\(h\)\}\_\{j\}\-s^\{\(h\)\}\_\{o\_\{i\}\}\(soaj/aoi=eξja\_\{j\}/a\_\{o\_\{i\}\}=e^\{\\xi\_\{j\}\}\), andℓk:=∂𝔼​\[L\]/∂ak\\ell\_\{k\}:=\\partial\\,\\mathbb\{E\}\[L\]/\\partial a\_\{k\},⟨ℓ⟩a:=∑kak​ℓk\\langle\\ell\\rangle\_\{a\}:=\\sum\_\{k\}a\_\{k\}\\ell\_\{k\}\.

### A\.2Assumptions and conditions

###### Assumption 4\(The gated model\)\.

\(G1\) Readout locality:logits depend on window contents only through\(u1\.\.uH\)\(u\_\{1\}\.\.u\_\{H\}\)— no trainable bypass from raw tokens\.\(G2\) Positional attention:one global pattern per head\.\(G3\) Disjointness:\{oi\}∩R=∅\\\{o\_\{i\}\\\}\\cap R=\\emptyset\.\(G4\) Carrying margin:value\-channel marginγ:=minx⁡\[Bx​x−maxy≠x⁡By​x\]≥γ0\>0\\gamma:=\\min\_\{x\}\[B\_\{xx\}\-\\max\_\{y\\neq x\}B\_\{yx\}\]\\geq\\gamma\_\{0\}\>0for carried skills\.\(G5\) Regularity:row\-mean balance\|B¯y−B¯y′\|≤γ/4\|\\bar\{B\}\_\{y\}\-\\bar\{B\}\_\{y^\{\\prime\}\}\|\\leq\\gamma/4withB¯y:=1V​∑xBy​x\\bar\{B\}\_\{y\}:=\\frac\{1\}\{V\}\\sum\_\{x\}B\_\{yx\}; each channel isCK\+1​\(𝒰\)C^\{K\+1\}\(\\mathcal\{U\}\)with mixed partials≤CF\\leq C\_\{F\};β2:=sup𝒰‖∇uF2‖\\beta\_\{2\}:=\\sup\_\{\\mathcal\{U\}\}\\\|\\nabla\_\{u\}F\_\{2\}\\\|;β3:=supbasin\|∂ξ\-differential of​∇ΦQ\|\\beta\_\{3\}:=\\sup\_\{\\rm basin\}\|\\partial\_\{\\xi\}\\text\{\-differential of \}\\nabla\\Phi\_\{Q\}\|;Λ:=supbasinmaxk⁡\|ℓk−ℓoi\|\\Lambda:=\\sup\_\{\\rm basin\}\\max\_\{k\}\|\\ell\_\{k\}\-\\ell\_\{o\_\{i\}\}\|\(all finite:𝒰\\mathcal\{U\}compact\)\.\(G6\) Oracle gating:the substrate channel is closed\.*This is the model’s most idealized assumption: it presupposes perfect task identity\. Removing it breaks the theorem\. Its internal necessity: with the substrate channel open, the uniformizing force \(Lemma[5](https://arxiv.org/html/2607.27281#Thmlemma5)\) acts at the largest data share and penalizes large attention masses hardest — in the score\-gap coordinate an escape\-*assisting*force with no small factor — and background skills themselves become unlearnable at equilibrium \(junk penalty exceeds skill reward\)\. Gating is what a substrate trained to equilibrium looks like; real transformers implement it by content routing\. Empirical counterpart: trained models output near\-uniform on substrate positions, and control ablations retain 99%\.*\(G7\) Leakage bound:on skill\-i′i^\{\\prime\}positions the readout’s coupling to non\-carrier heads yields a score\-gap differential pull≤χ\\leq\\chi\.\(G8\) Recruitment \(ergodic\-flavored; used only for clause \(i\)\):while≥K\\geq Kheads are available, each uncarried high\-traffic skill has per\-step recruitment hazard≥ρmin\>0\\geq\\rho\_\{\\min\}\>0\.

Conditions\.Withλ′\\lambda^\{\\prime\}andJJdefined in Lemmas[7](https://arxiv.org/html/2607.27281#Thmlemma7)and[4](https://arxiv.org/html/2607.27281#Thmlemma4):C0 \(feasibility\):at leastH−K\+1H\-K\+1skills haveμi:=πi​λ′−c​\(J\+β3\)−χ−β2\>0\\mu\_\{i\}:=\\pi\_\{i\}\\lambda^\{\\prime\}\-c\(J\+\\beta\_\{3\}\)\-\\chi\-\\beta\_\{2\}\>0\. \(Uniform special caseπi=\(1−c\)/M\\pi\_\{i\}=\(1\-c\)/M: C0 reads “MMnot so large as to dilute the pull\-back\.” The constants insideJJare conservative; effective serving\-head counts would tighten them\.\)C2 \(small step\):2​η​G≤12\\eta G\\leq 1\.C3 \(timescale separation\):with toleranceδtol\\delta\_\{\\rm tol\}andTrec:=\(H−K\+1\)/\(ρmin​p0\)T\_\{\\rm rec\}:=\(H\{\-\}K\{\+\}1\)/\(\\rho\_\{\\min\}p\_\{0\}\):Trec≤\(δtol/2​H\)​eθ​\(2−2​η​G\)T\_\{\\rm rec\}\\leq\(\\delta\_\{\\rm tol\}/2H\)\\,e^\{\\theta\(2\-2\\eta G\)\}\(θ\\thetafrom Lemma[10](https://arxiv.org/html/2607.27281#Thmlemma10); automatic at smallη\\etasince the right side is exponentially large\)\.C4 \(small basin\):δ0​Λ≤μi\\delta\_\{0\}\\Lambda\\leq\\mu\_\{i\}\.

### A\.3Pre\-formation weak coupling

###### Lemma 2\(Conditional uniformity; exact\)\.

For anyS⊊RS\\subsetneq Rand any window coordinates outsideRR, conditioning on them leavesYYuniform onℤV\\mathbb\{Z\}\_\{V\}\.

###### Proof\.

Pickj∗∈R∖Sj^\{\*\}\\in R\\setminus S:Y=\(const\+Xj∗\+⋯\)modVY=\(\\text\{const\}\+X\_\{j^\{\*\}\}\+\\cdots\)\\bmod VwithXj∗X\_\{j^\{\*\}\}uniform and independent of the conditioning; uniformity is translation\-invariant\. Hence any predictor reading a strict subset has cross\-entropy≥ln⁡V\\geq\\ln V, with equality only for the constant uniform predictor\. ∎

###### Lemma 3\(Fourier support; exact\)\.

Withω=e2​π​i/V\\omega=e^\{2\\pi i/V\}andχmR​\(x\):=∏j∈Rωm​xj\\chi\_\{m\}^\{R\}\(x\):=\\prod\_\{j\\in R\}\\omega^\{mx\_\{j\}\}:𝟏Y=y=1V​∑mω−m​y​χmR​\(x\)\\mathbf\{1\}\_\{Y=y\}=\\frac\{1\}\{V\}\\sum\_\{m\}\\omega^\{\-my\}\\chi\_\{m\}^\{R\}\(x\); form≠0m\\neq 0the character’s support is exactlyRR; andCov​\(𝟏Y=y,φ\)≠0\\mathrm\{Cov\}\(\\mathbf\{1\}\_\{Y=y\},\\varphi\)\\neq 0iffφ\\varphihas a Fourier component with support⊇R\\supseteq R\.

###### Proof\.

Discrete inversion onℤV\\mathbb\{Z\}\_\{V\}; character orthogonality under the i\.i\.d\. uniform measure\. ∎

###### Lemma 4\(Iterated\-oscillation bound; in\-model theorem\)\.

LetAr:=∑har\(h\)A\_\{r\}:=\\sum\_\{h\}a^\{\(h\)\}\_\{r\}\. Under \(G1\) and \(G5\), every Fourier coefficient ofzzor of∂z/∂s\(h\)\\partial z/\\partial s^\{\(h\)\}with support⊇R\\supseteq Ris bounded byCF​4K​∏r∈RArC\_\{F\}\\,4^\{K\}\\prod\_\{r\\in R\}A\_\{r\}\. Consequently, pre\-formation \(some slotr∗r^\{\*\}hasAr∗≤H​α¯A\_\{r^\{\*\}\}\\leq H\\bar\{\\alpha\}\), the label\-dependent \(signal\) part of the target gradient on any head’s scores is bounded byc​JcJwithJ:=V​CF​4K​HK​α¯J:=VC\_\{F\}4^\{K\}H^\{K\}\\bar\{\\alpha\}\.

###### Proof\.

*\(Coefficient bound\.\)*Form≠0m\\neq 0,𝔼Xr​\[ω−m​Xr\]=0\\mathbb\{E\}\_\{X\_\{r\}\}\[\\omega^\{\-mX\_\{r\}\}\]=0, so replacingggbyg−𝔼Xr​\[g∣rest\]g\-\\mathbb\{E\}\_\{X\_\{r\}\}\[g\\mid\\text\{rest\}\]leaves⟨g,χ¯mR⟩\\langle g,\\bar\{\\chi\}\_\{m\}^\{R\}\\rangleunchanged while its sup\-norm is bounded by the direction\-rroscillation; iterating over theKKcoordinates ofRRbounds the coefficient by the mixedKK\-fold oscillation\. ChangingXrX\_\{r\}displacesuubyvr:=∑har\(h\)​\(e​\(xr′\)−e​\(xr\)\)v\_\{r\}:=\\sum\_\{h\}a^\{\(h\)\}\_\{r\}\(e\(x\_\{r\}^\{\\prime\}\)\-e\(x\_\{r\}\)\),‖vr‖1≤2​Ar\\\|v\_\{r\}\\\|\_\{1\}\\leq 2A\_\{r\}, and the segment\[u,u\+vr\]\[u,u\+v\_\{r\}\]stays in the convex𝒰\\mathcal\{U\}, so the mixed difference has the iterated line\-integral representation∫01⋯∫01DKF\[vr1,\.\.,vrK\]dλ\\int\_\{0\}^\{1\}\\\!\\\!\\cdots\\\!\\int\_\{0\}^\{1\}D^\{K\}F\[v\_\{r\_\{1\}\},\.\.,v\_\{r\_\{K\}\}\]\\,d\\lambda, giving\|⋅\|≤CF∏r2Ar\|\{\\cdot\}\|\\leq C\_\{F\}\\prod\_\{r\}2A\_\{r\}— the mean\-value step is legitimate precisely because𝒰\\mathcal\{U\}is convex\. For∂z/∂sj\(h\)=⟨∇uhF,aj\(h\)​\(e​\(xj\)−uh\)⟩\\partial z/\\partial s^\{\(h\)\}\_\{j\}=\\langle\\nabla\_\{u\_\{h\}\}F,\\,a\_\{j\}^\{\(h\)\}\(e\(x\_\{j\}\)\-u\_\{h\}\)\\rangle, both factors depend on contents; the discrete Leibniz ruleΔr​\(f​g\)=\(Δr​f\)​g\+\(Sr​f\)​\(Δr​g\)\\Delta\_\{r\}\(fg\)=\(\\Delta\_\{r\}f\)g\+\(S\_\{r\}f\)\(\\Delta\_\{r\}g\)iterated overKKcoordinates yields2K2^\{K\}terms, each bounded byCF​∏r2​ArC\_\{F\}\\prod\_\{r\}2A\_\{r\}, for a totalCF​4K​∏rArC\_\{F\}4^\{K\}\\prod\_\{r\}A\_\{r\}\.*\(Signal bound\.\)*By Lemma[3](https://arxiv.org/html/2607.27281#Thmlemma3)the label\-dependent part flows only through support\-⊇R\\supseteq Rcomponents; summingVVfrequencies and bounding∏rAr≤HK−1⋅Ar∗≤HK​α¯\\prod\_\{r\}A\_\{r\}\\leq H^\{K\-1\}\\cdot A\_\{r^\{\*\}\}\\leq H^\{K\}\\bar\{\\alpha\}gives the claim\. A linear readout \(KK\-th mixed derivative zero\) has*exactly zero*signal drift\. ∎

###### Lemma 5\(Exact gradient decomposition; in\-model theorem\)\.

On target positions, for any parameter block,𝔼​\[∇LT\]=∇Φ\+S\\mathbb\{E\}\[\\nabla L\_\{T\}\]=\\nabla\\Phi\+S, whereΦ:=𝔼​\[−1V​∑yln⁡py\]\\Phi:=\\mathbb\{E\}\\big\[\-\\tfrac\{1\}\{V\}\\sum\_\{y\}\\ln p\_\{y\}\\big\]\(the*uniformizing potential*, label\-free, globally minimized exactly on the chance manifoldp≡uniformp\\equiv\\mathrm\{uniform\}by Gibbs’ inequality\) andS:=−∑yCov​\(P​\(Y=y∣X\),∇zy\)S:=\-\\sum\_\{y\}\\mathrm\{Cov\}\\big\(P\(Y\{=\}y\\mid X\),\\,\\nabla z\_\{y\}\\big\)with\|S\|≤c​J\|S\|\\leq cJpre\-formation\.

###### Proof\.

𝔼​\[𝟏Y=y​∇zy\]=𝔼​\[P​\(Y=y\|X\)​∇zy\]\\mathbb\{E\}\[\\mathbf\{1\}\_\{Y=y\}\\nabla z\_\{y\}\]=\\mathbb\{E\}\[P\(Y\{=\}y\|X\)\\nabla z\_\{y\}\]\(tower property; the model’s forward pass never seesYY\)\. SplitP​\(Y=y\|X\)=1V\+\(full\-support part\)P\(Y\{=\}y\|X\)=\\frac\{1\}\{V\}\+\(\\text\{full\-support part\}\)by Lemmas[2](https://arxiv.org/html/2607.27281#Thmlemma2)–[3](https://arxiv.org/html/2607.27281#Thmlemma3); the1V\\frac\{1\}\{V\}part joins∑y𝔼​\[py​∇zy\]\\sum\_\{y\}\\mathbb\{E\}\[p\_\{y\}\\nabla z\_\{y\}\]to form∑y𝔼​\[\(py−1V\)​∇zy\]=∇Φ\\sum\_\{y\}\\mathbb\{E\}\[\(p\_\{y\}\-\\frac\{1\}\{V\}\)\\nabla z\_\{y\}\]=\\nabla\\Phi\(softmax identity∂zk\(−1V​∑yln⁡py\)=pk−1V\\partial\_\{z\_\{k\}\}\(\-\\frac\{1\}\{V\}\\sum\_\{y\}\\ln p\_\{y\}\)=p\_\{k\}\-\\frac\{1\}\{V\}\); the remainder isSS, bounded by Lemma[4](https://arxiv.org/html/2607.27281#Thmlemma4)\.*Level separation:*this lemma establishes the decomposition and the force’s form only;*where*the uniformizing force acts is governed by \(G6\)/\(G7\) and is book\-kept in Lemma[9](https://arxiv.org/html/2607.27281#Thmlemma9)\. ∎

### A\.4Exclusivity and restoring drift

###### Lemma 6\(Exclusivity; exact\)\.

Under \(G2\),a\(h\)∈ΔWa^\{\(h\)\}\\in\\Delta\_\{W\}is a single global vector: carrying \(aoi≥1−δ0a\_\{o\_\{i\}\}\\geq 1\-\\delta\_\{0\}\) forces total target\-slot mass≤δ0\\leq\\delta\_\{0\}; serving a slot \(≥γneed\>δ0\\geq\\gamma\_\{\\rm need\}\>\\delta\_\{0\}\) requires first shedding mass fromoio\_\{i\}\.*\(Rank\-1 remark, supporting evidence only: content\-dependent scores of rank one,sj​\(cue\)=φ​\(cue\)​κjs\_\{j\}\(\\mathrm\{cue\}\)=\\varphi\(\\mathrm\{cue\}\)\\kappa\_\{j\}, admit at most two sharp patterns per head —arg⁡max⁡κ\\arg\\max\\kappaandarg⁡min⁡κ\\arg\\min\\kappa— so the theorem survives withH→2​HH\\to 2H\. General rank is not claimed; extrapolation is carried by the experiments\.\)*

###### Lemma 7\(Restoring drift; in\-model theorem\)\.

Let headhhcarry skillii\(G4\)\. On skill\-iipositions, moving attention massε\\varepsilonfromoio\_\{i\}to any other slot obeysd​Li/d​ε≥\(1−p¯c\)​\(3​γ/4−C1​δ0\)dL\_\{i\}/d\\varepsilon\\geq\(1\-\\bar\{p\}\_\{c\}\)\\,\(3\\gamma/4\-C\_\{1\}\\delta\_\{0\}\)withC1:=16​‖B‖∞2C\_\{1\}:=16\\\|B\\\|\_\{\\infty\}^\{2\}, under4​δ0​‖B‖∞≤144\\delta\_\{0\}\\\|B\\\|\_\{\\infty\}\\leq\\tfrac\{1\}\{4\}\. Since softmax confidence is strictly below 1,\(1−p¯c\)≥κ:=1V​e−\(2​‖B‖∞\+‖ζ‖∞\)\>0\(1\-\\bar\{p\}\_\{c\}\)\\geq\\kappa:=\\tfrac\{1\}\{V\}e^\{\-\(2\\\|B\\\|\_\{\\infty\}\+\\\|\\zeta\\\|\_\{\\infty\}\)\}\>0\(weight decay bounds‖B‖\\\|B\\\|, makingκ\\kappaa bona fide constant\)\. Writeλ′:=κ​\(3​γ/4−C1​δ0\)\\lambda^\{\\prime\}:=\\kappa\\,\(3\\gamma/4\-C\_\{1\}\\delta\_\{0\}\)\.

###### Proof\.

With correct tokenc∗c^\{\*\}and perturbationd​zy=d​ε​\(By,X′−By,c∗\)dz\_\{y\}=d\\varepsilon\\,\(B\_\{y,X^\{\\prime\}\}\-B\_\{y,c^\{\*\}\}\), the identity∑y\(py−𝟏y=c∗\)​vy=∑y≠c∗py​\(vy−vc∗\)\\sum\_\{y\}\(p\_\{y\}\-\\mathbf\{1\}\_\{y=c^\{\*\}\}\)v\_\{y\}=\\sum\_\{y\\neq c^\{\*\}\}p\_\{y\}\(v\_\{y\}\-v\_\{c^\{\*\}\}\)places the*entire*first variation on error\-probability weight\. Three blocks: the diagonal block contributes≥γ​∑y≠c∗𝔼​\[py\]=γ​\(1−p¯c\)\\geq\\gamma\\sum\_\{y\\neq c^\{\*\}\}\\mathbb\{E\}\[p\_\{y\}\]=\\gamma\(1\-\\bar\{p\}\_\{c\}\)\(G4\); the row\-mean block is bounded by\(γ/4\)​\(1−p¯c\)\(\\gamma/4\)\(1\-\\bar\{p\}\_\{c\}\)\(G5 — note it, too, carries the\(1−p¯c\)\(1\-\\bar\{p\}\_\{c\}\)weight, which is what forbids a sign flip at high confidence\); the correlation error \(the dependence ofpponX′X^\{\\prime\}through theaj′≤δ0a\_\{j^\{\\prime\}\}\\leq\\delta\_\{0\}channel\) is bounded along the logit\-interpolation path byC1​δ0​\(1−p¯c\)C\_\{1\}\\delta\_\{0\}\(1\-\\bar\{p\}\_\{c\}\)\. Summing gives the claim: the restoring force weakens with confidence but never reverses\. ∎

### A\.5Score\-difference dynamics and the force ledger

###### Lemma 8\(Exact score\-gap dynamics\)\.

\(a\)∂𝔼​\[L\]/∂sj=aj​\(ℓj−⟨ℓ⟩a\)\\partial\\,\\mathbb\{E\}\[L\]/\\partial s\_\{j\}=a\_\{j\}\(\\ell\_\{j\}\-\\langle\\ell\\rangle\_\{a\}\)— the softmax chain identity, with no approximation\. \(b\) Score\-gap updates are exactly linear in the stochastic gradient, so𝔼​\[Δ​ξj∣ℱ\]=−η​\[aj​\(ℓj−⟨ℓ⟩a\)−aoi​\(ℓoi−⟨ℓ⟩a\)\]\\mathbb\{E\}\[\\Delta\\xi\_\{j\}\\mid\\mathcal\{F\}\]=\-\\eta\\big\[a\_\{j\}\(\\ell\_\{j\}\-\\langle\\ell\\rangle\_\{a\}\)\-a\_\{o\_\{i\}\}\(\\ell\_\{o\_\{i\}\}\-\\langle\\ell\\rangle\_\{a\}\)\\big\], with*no second\-order remainder*\.

###### Proof\.

∂ak/∂sj=ak​\(δk​j−aj\)\\partial a\_\{k\}/\\partial s\_\{j\}=a\_\{k\}\(\\delta\_\{kj\}\-a\_\{j\}\), so∂L/∂sj=∑kℓk​ak​\(δk​j−aj\)=aj​ℓj−aj​⟨ℓ⟩a\\partial L/\\partial s\_\{j\}=\\sum\_\{k\}\\ell\_\{k\}a\_\{k\}\(\\delta\_\{kj\}\-a\_\{j\}\)=a\_\{j\}\\ell\_\{j\}\-a\_\{j\}\\langle\\ell\\rangle\_\{a\}\.Δ​ξj=Δ​sj−Δ​soi=−η​\(g^j−g^oi\)\\Delta\\xi\_\{j\}=\\Delta s\_\{j\}\-\\Delta s\_\{o\_\{i\}\}=\-\\eta\(\\hat\{g\}\_\{j\}\-\\hat\{g\}\_\{o\_\{i\}\}\); take conditional expectations\. ∎

###### Lemma 9\(Net drift; in\-model theorem\)\.

Under C0 and C4, inside the basin \(aj≤δ0a\_\{j\}\\leq\\delta\_\{0\}for allj∈Rj\\in R\), for a firmly pinned head:𝔼​\[Δ​ξj∣ℱ\]≤−η​aoi​\(1−aoi\)​μi\\mathbb\{E\}\[\\Delta\\xi\_\{j\}\\mid\\mathcal\{F\}\]\\leq\-\\eta\\,a\_\{o\_\{i\}\}\(1\-a\_\{o\_\{i\}\}\)\\,\\mu\_\{i\}exactly; on the stripaj∈\[δ0/e2,δ0\]a\_\{j\}\\in\[\\delta\_\{0\}/e^\{2\},\\delta\_\{0\}\],𝔼​\[Δ​ξj\|ℱ\]≤−η​μstrip\\mathbb\{E\}\[\\Delta\\xi\_\{j\}\|\\mathcal\{F\}\]\\leq\-\\eta\\,\\mu\_\{\\rm strip\}withμstrip:=\(1−δ0\)​\(δ0/e2\)​μi\\mu\_\{\\rm strip\}:=\(1\-\\delta\_\{0\}\)\(\\delta\_\{0\}/e^\{2\}\)\\mu\_\{i\}, and\|Δ​ξj\|≤2​η​G\|\\Delta\\xi\_\{j\}\|\\leq 2\\eta G\(C2\)\.

###### Proof\.

The pairwise gapsℓj′−ℓoi\\ell\_\{j^\{\\prime\}\}\-\\ell\_\{o\_\{i\}\}are book\-kept by position class, with*no symmetry\-cancellation argument anywhere*: skill\-iipositions give≥λ′\\geq\\lambda^\{\\prime\}for everyj′≠oij^\{\\prime\}\\neq o\_\{i\}at shareπi\\pi\_\{i\}\(Lemma[7](https://arxiv.org/html/2607.27281#Thmlemma7)\); other skills contribute\|⋅\|≤χ\|\{\\cdot\}\|\\leq\\chi\(G7\); substrate contributes zero \(G6\); target positions contribute≤c​\(J\+β3\)\\leq c\(J\+\\beta\_\{3\}\)\(Lemmas[4](https://arxiv.org/html/2607.27281#Thmlemma4)–[5](https://arxiv.org/html/2607.27281#Thmlemma5)\); the higher\-order readout≤β2\\leq\\beta\_\{2\}\(G5\)\. Hence⟨ℓ⟩a−ℓoi≥\(1−aoi\)​μi\\langle\\ell\\rangle\_\{a\}\-\\ell\_\{o\_\{i\}\}\\geq\(1\-a\_\{o\_\{i\}\}\)\\mu\_\{i\}, making the second term of Lemma[8](https://arxiv.org/html/2607.27281#Thmlemma8)\(b\) at most−η​aoi​\(1−aoi\)​μi\-\\eta a\_\{o\_\{i\}\}\(1\-a\_\{o\_\{i\}\}\)\\mu\_\{i\}; for the first term,ℓj−⟨ℓ⟩a≥μi−\(1−aoi\)​Λ≥μi−δ0​Λ≥0\\ell\_\{j\}\-\\langle\\ell\\rangle\_\{a\}\\geq\\mu\_\{i\}\-\(1\-a\_\{o\_\{i\}\}\)\\Lambda\\geq\\mu\_\{i\}\-\\delta\_\{0\}\\Lambda\\geq 0by C4, so it is favorable and can be discarded\. ∎

### A\.6Confinement, occupation, and the theorem

###### Lemma 10\(Exponential confinement; cited theorem, self\-contained proof\)\.

Under C0, C2, C4, withθ:=min⁡\(μstrip/\(8​η​G2\),1/\(2​η​G\)\)\\theta:=\\min\\\!\\big\(\\mu\_\{\\rm strip\}/\(8\\eta G^\{2\}\),\\,1/\(2\\eta G\)\\big\):P\(∃t≤T:aj\(t\)≥δ0\)≤Te−θ​\(2−2​η​G\)P\\big\(\\exists\\,t\\leq T:a\_\{j\}\(t\)\\geq\\delta\_\{0\}\\big\)\\leq T\\,e^\{\-\\theta\(2\-2\\eta G\)\}, so the expected escape time is≥eθ​\(2−2​η​G\)/2\\geq e^\{\\theta\(2\-2\\eta G\)\}/2\.

###### Proof\.

Themin\\minmakesθ⋅2​η​G≤1\\theta\\cdot 2\\eta G\\leq 1always; at the boundaryθ​\|Δ​ξ\|=1\\theta\\,\|\\Delta\\xi\|=1the inequalityex≤1\+x\+x2e^\{x\}\\leq 1\+x\+x^\{2\}still holds \(e≤3e\\leq 3\), so on the strip𝔼​\[eθ​Δ​ξ\|ℱ\]≤1−θ​η​μstrip\+θ2​\(2​η​G\)2<1\\mathbb\{E\}\[e^\{\\theta\\Delta\\xi\}\|\\mathcal\{F\}\]\\leq 1\-\\theta\\eta\\mu\_\{\\rm strip\}\+\\theta^\{2\}\(2\\eta G\)^\{2\}<1andeθ​ξe^\{\\theta\\xi\}is a supermartingale\. Escape must traverse the strip, ofξ\\xi\-widthln⁡\(δ0/\(δ0/e2\)\)=2\\ln\(\\delta\_\{0\}/\(\\delta\_\{0\}/e^\{2\}\)\)=2; entries overshoot the bottom by at most2​η​G2\\eta Gper step; optional stopping bounds each crossing attempt bye−θ​\(2−2​η​G\)e^\{\-\\theta\(2\-2\\eta G\)\}, and at most one attempt begins per step, giving the union factorTT\. This is Hajek’s drift theorem\[[38](https://arxiv.org/html/2607.27281#bib.bib38)\];*no ergodicity or mixing is used*— only drift and bounded increments\. The bound is conservative: it certifies difficulty of reachingδ0<γneed\\delta\_\{0\}<\\gamma\_\{\\rm need\}, which reachingγneed\\gamma\_\{\\rm need\}requires first\. ∎

###### Lemma 11\(Occupation is monotone; the deadline is finite\)\.

\(a\) Under C3, on the window\[0,2​Trec\]\[0,\\,2T\_\{\\rm rec\}\], the probability that*any*firmly pinned head escapes is≤H⋅2​Trec​e−θ​\(2−2​η​G\)≤δtol\\leq H\\cdot 2T\_\{\\rm rec\}\\,e^\{\-\\theta\(2\-2\\eta G\)\}\\leq\\delta\_\{\\rm tol\}:NavailN\_\{\\rm avail\}is non\-increasing with probability≥1−δtol\\geq 1\-\\delta\_\{\\rm tol\}\. \(b\) Under \(G8\) and C0: recruitment events are*not*independent \(skills compete for heads\); the argument uses only “whileNavail≥KN\_\{\\rm avail\}\\geq K, at least one new firm pinning occurs per window with probability≥p0\\geq p\_\{0\},” soTrecT\_\{\\rm rec\}is stochastically dominated by a sum ofH−K\+1H\-K\+1geometric variables:tdeadt\_\{\\rm dead\}is a\.s\. finite with𝔼​\[tdead\]≤Trec\+C\\mathbb\{E\}\[t\_\{\\rm dead\}\]\\leq T\_\{\\rm rec\}\+C, inside the window of \(a\) by C3\. \(c\)*\(Constant\-hazard clause: stated only to record that we tested and rejected it\.\)*If in addition each free head faced a constant hazard≥ρ\\geq\\rho, one would get𝔼​\[tdead\]≤ρ−1​ln⁡\(H/\(K−1\)\)\+C\\mathbb\{E\}\[t\_\{\\rm dead\}\]\\leq\\rho^\{\-1\}\\ln\(H/\(K\{\-\}1\)\)\+Cand exponential site decay\.This extra clause is inconsistent with the model’s own traffic\-share structure\(μi∝πi\\mu\_\{i\}\\propto\\pi\_\{i\}: once the high\-traffic skills have taken their heads, the remaining skills pull weakly, so the hazard must fall\),and it is falsified by measurement\(App\.[A\.7](https://arxiv.org/html/2607.27281#A1.SS7)\): the hazard declines by≈\\approx15×\\timesacross the sweep\. We therefore drop it; only \(a\) and \(b\) are used\.

###### Proof of Theorem[1](https://arxiv.org/html/2607.27281#Thmtheorem1)\.

\(i\) is Lemma[11](https://arxiv.org/html/2607.27281#Thmlemma11)\. \(ii\): pasttdeadt\_\{\\rm dead\}, formation requires some firmly pinned head to reachγneed\>δ0\\gamma\_\{\\rm need\}\>\\delta\_\{0\}on some slot; Lemma[10](https://arxiv.org/html/2607.27281#Thmlemma10)plus a union overHHheads,KKslots,TTsteps gives the bound\. The shareccentersμi\\mu\_\{i\}only through−c​\(J\+β3\)\-c\(J\+\\beta\_\{3\}\): an8×8\\timesincrease inccshiftsμi\\mu\_\{i\}linearly and the exponent barely moves; overturning C0 by data requiresccso large thatπi​λ′<c​\(J\+β3\)\\pi\_\{i\}\\lambda^\{\\prime\}<c\(J\+\\beta\_\{3\}\)— destroying the background margin itself\. \(iii\): re\-initialized scores give near\-uniform attention \(mass≈1/W<1−δ0\\approx 1/W<1\-\\delta\_\{0\}on every offset\), so the head carries nothing;Navail≥KN\_\{\\rm avail\}\\geq Kis restored and the fresh\-substrate rate applies; only attention*scores*need resetting\. \(iv\): pre\-formation,LT=ln⁡V\+ΔjunkL\_\{T\}=\\ln V\+\\Delta\_\{\\rm junk\}withΔjunk:=Φ−ln⁡V≥0\\Delta\_\{\\rm junk\}:=\\Phi\-\\ln V\\geq 0\(Gibbs\); the uniformizing force pushes it toward zero while leakage \(G7\) and the target coupling \(β3\\beta\_\{3\}\) hold it in a bounded neighborhood of zero — direction and boundedness are proved, a convergence rate is not claimed; empirically the neighborhood is below experimental resolution, hence the flatln⁡V\\ln Vplateau\. The background loss is a function of the pinning countn​\(t\)n\(t\), decreasing by one same\-order step per pinning event;tdeadt\_\{\\rm dead\}is the hitting time ofnn, and its step is distributionally of the same order as every neighboring step: no discontinuity, kink, or necessary precursor in any loss\-derived scalar\. Gradient\-level second\-order statistics and the attention\-occupation meterNavailN\_\{\\rm avail\}are*not*covered by this clause — the latter is precisely the early\-warning quantity of §[7](https://arxiv.org/html/2607.27281#S7)\. ∎

Corollary \(the waiting tax\)\.If the target is introduced att0t\_\{0\}, the available\-site count is non\-increasing int0t\_\{0\}, so the expected formation time rises witht0t\_\{0\}and saturates near the deadline — the measured506→1108→1115506\\to 1108\\to 1115profile\. The saturation is itself informative: it is what a*declining*recruitment hazard produces, and is inconsistent with the constant\-hazard clause dropped above\.

### A\.7Testing the theorem’s own predictions: two confirmed, one falsified

The theorem fixes falsifiable predictions; we ran them in the apparatus of App\.[A\.1](https://arxiv.org/html/2607.27281#A1.SS1)\(H=16H=16heads×\\times2 layers==32 head\-slots, background skill family ofM=24M=24copy skills so thatM\>HM\>H, target drivec=0\.06c=0\.06, 6 seeds per cell\)\. Aging occupies heads atc=0c=0; the target is then switched on and formation timed\. Committed heads are counted directly from attention peakedness on background cue positions\.

Falsified — exponential occupation decay \(and with it thelog⁡H\\log Hdeadline\)\.Available head\-slots fall32→18→8→532\\to 18\\to 8\\to 5at agingS=0/1000/4000/16000S=0/1000/4000/16000\. The implied per\-head hazard is not constant but falls5\.8→2\.7→0\.395\.8\\to 2\.7\\to 0\.39\(per10410^\{4\}steps\), a≈\\approx15×\\timesdecline; extrapolating the initial rate exponentially predicts0\.0030\.003free slots atS=16000S=16000against 5 measured, a∼\\sim1500×\\timesmiss\.*The failure is of an extra simplifying clause, not of the mechanism, and its direction is the one the theorem’s own traffic\-share structure demands*: high\-traffic skills take their heads first, leaving low\-traffic skills whose pullμi∝πi\\mu\_\{i\}\\propto\\pi\_\{i\}is weak and whose recruitment is correspondingly slow\. Dropping the clause makes the model more coherent, not less\. It also resolves an apparent tension with the real\-model measurement: Pythia’s depletion profile \(1→0\.46→0\.451\\to 0\.46\\to 0\.45\) saturates rather than decaying exponentially — the same declining\-hazard family, measured on the other substrate\.

Confirmed and sharpened — the repair is specific to the score channel\.From a single aged substrate \(S=40,000S=40\{,\}000, so identical damage across arms\), re\-initializing different weight groups gives: attention*scores*\(the QK slices\)t∗=1050t^\{\*\}=1050, 6/6 formed —4\.3×\\timesfasterthan no reset \(t∗=4500t^\{\*\}=4500, 5/6\); MLP36503650\(0\.81×0\.81\\times\); and value/output \(the OV slices\)79007900, i\.e\.1\.76×\\times*slower*than no reset at all\. Occupation lives in the attention*pattern*, exactly as Lemma[6](https://arxiv.org/html/2607.27281#Thmlemma6)requires: freeing the pattern restores site availability, while resetting the value channel destroys learned structure without freeing any site and is therefore worse than doing nothing\. This refines the real\-model repair of §[7](https://arxiv.org/html/2607.27281#S7)\(which re\-initialized attention weights as a block, 5/5 vs 0/5\) into a testable prescription — reset QK, keep OV\.

Also observed\.The tax curve is non\-monotone at its start: brief aging*helps*\(S=0S=0: 4/6 formed, median28502850;S=1000S=1000: 6/6, median15001500\) before it taxes, because the background first builds offset\-selective heads that seed the target — the cross\-nucleation effect reported for real circuits, reproduced here without being asked for\.Not tested\.The deadline’s dependence on head count: at a single aging depth \(S=16,000S=16\{,\}000, head dimension held fixed at 16\) formation times are non\-monotone inHH\(13400/9250/1450013400/9250/14500forH=4/8/16H=4/8/16\), so a per\-HHdeadline sweep would be required; with thelog⁡H\\log Hclaim withdrawn, nothing in the paper now rests on it\.

Site mediation, causally\.Aging changes more than the site count, so the aging sweep alone cannot attribute the slowdown to sites \(available slots18→8\.5→518\\to 8\.5\\to 5againstt∗=1500→2000→14500t^\{\*\}=1500\\to 2000\\to 14500is a log–log slope of−1\.68\-1\.68, steeper than the−1\-1a pure site\-product law would give\)\. The reset arms supply what the sweep cannot: at*fixed*aging, restoring only the attention scores — the one intervention that restores availability without undoing any other consequence of aging — recovers formation speed 4\.3×\\times\. That is the causal statement the site reading requires\.

What remains open\.Whether gating/junk\-suppression \(G6/G7\) can be*proved*to be the training equilibrium of the substrate rather than assumed — the sharpest remaining question; content\-dependent attention of rank≥2\\geq 2\(the rank\-1 case is covered as supporting evidence\); and the conservative constants \(HKH^\{K\},4K4^\{K\}\), which tighten under effective\-serving\-head counts\.

## References

- Watanabe \[2009\]Sumio Watanabe\.*Algebraic Geometry and Statistical Learning Theory*\.Cambridge University Press, 2009\.
- Hoogland et al\. \[2024\]Jesse Hoogland, George Wang, Matthew Farrugia\-Roberts, Liam Carroll, Susan Wei, and Daniel Murfet\.The developmental landscape of in\-context learning\.*arXiv preprint arXiv:2402\.02364*, 2024\.
- Chen et al\. \[2023\]Zhongtian Chen, Edmund Lau, Jake Mendel, Susan Wei, and Daniel Murfet\.Dynamical versus bayesian phase transitions in a toy model of superposition\.*arXiv preprint arXiv:2310\.06301*, 2023\.
- Gu et al\. \[2026\]Xinran Gu, Kaifeng Lyu, Jiazheng Li, and Jingzhao Zhang\.Data mixing can induce phase transitions in knowledge acquisition\.*arXiv preprint arXiv:2505\.18091*, 2026\.
- Rubin et al\. \[2024\]Noa Rubin, Inbar Seroussi, and Zohar Ringel\.Grokking as a first order phase transition in two layer networks\.In*International Conference on Learning Representations*, 2024\.
- Ersoy and Wiesner \[2026\]Ibrahim Talha Ersoy and Karoline Wiesner\.Noise\-driven escape from metastable phases explains grokking in deep neural networks\.*arXiv preprint arXiv:2606\.17120*, 2026\.
- Dohare et al\. \[2024\]Shibhansh Dohare, J\. Fernando Hernandez\-Garcia, Qingfeng Lan, Parash Rahman, A\. Rupam Mahmood, and Richard S\. Sutton\.Loss of plasticity in deep continual learning\.*Nature*, 632:768–774, 2024\.
- Ash and Adams \[2020\]Jordan T\. Ash and Ryan P\. Adams\.On warm\-starting neural network training\.In*Advances in Neural Information Processing Systems*, 2020\.
- Lyle et al\. \[2024\]Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney\.Disentangling the causes of plasticity loss in neural networks\.In*International Conference on Machine Learning*, 2024\.
- Xie et al\. \[2023\]Sang Michael Xie, Hieu Pham, Xuanyi Dong, et al\.Doremi: Optimizing data mixtures speeds up language model pretraining\.In*Advances in Neural Information Processing Systems*, 2023\.
- Liu et al\. \[2024\]Qian Liu, Xiaosen Zheng, Niklas Muennighoff, et al\.Regmix: Data mixture as regression for language model pre\-training\.*arXiv preprint arXiv:2407\.01492*, 2024\.
- Ye et al\. \[2024\]Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu\.Data mixing laws: Optimizing data mixtures by predicting language modeling performance\.*arXiv preprint arXiv:2403\.16952*, 2024\.
- Ganguli et al\. \[2022\]Deep Ganguli, Danny Hernandez, Liane Lovitt, et al\.Predictability and surprise in large generative models\.In*ACM Conference on Fairness, Accountability, and Transparency*, 2022\.
- Biderman et al\. \[2023\]Stella Biderman, Hailey Schoelkopf, Quentin Anthony, et al\.Pythia: A suite for analyzing large language models across training and scaling\.In*International Conference on Machine Learning*, 2023\.
- Groeneveld et al\. \[2024\]Dirk Groeneveld, Iz Beltagy, Pete Walsh, et al\.Olmo: Accelerating the science of language models\.*arXiv preprint arXiv:2402\.00838*, 2024\.
- Li et al\. \[2017\]Qianxiao Li, Cheng Tai, and Weinan E\.Stochastic modified equations and adaptive stochastic gradient algorithms\.In*International Conference on Machine Learning*, 2017\.
- Mandt et al\. \[2017\]Stephan Mandt, Matthew D\. Hoffman, and David M\. Blei\.Stochastic gradient descent as approximate bayesian inference\.*Journal of Machine Learning Research*, 18\(134\):1–35, 2017\.
- Kearns \[1998\]Michael Kearns\.Efficient noise\-tolerant learning from statistical queries\.*Journal of the ACM*, 45\(6\):983–1006, 1998\.
- Blum et al\. \[1994\]Avrim Blum, Merrick Furst, Jeffrey Jackson, Michael Kearns, Yishay Mansour, and Steven Rudich\.Weakly learning dnf and characterizing statistical query learning using fourier analysis\.In*Proceedings of the 26th Annual ACM Symposium on Theory of Computing \(STOC\)*, pages 253–262, 1994\.
- Barak et al\. \[2022\]Boaz Barak, Benjamin L\. Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang\.Hidden progress in deep learning: Sgd learns parities near the computational limit\.In*Advances in Neural Information Processing Systems*, 2022\.
- Abbe et al\. \[2022\]Emmanuel Abbe, Enric Boix\-Adserà, and Theodor Misiakiewicz\.The merged\-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two\-layer neural networks\.In*Conference on Learning Theory \(COLT\)*, 2022\.
- Abbe et al\. \[2023\]Emmanuel Abbe, Enric Boix\-Adserà, and Theodor Misiakiewicz\.Sgd learning on neural networks: leap complexity and saddle\-to\-saddle dynamics\.In*Conference on Learning Theory \(COLT\)*, 2023\.
- Glasgow \[2024\]Margalit Glasgow\.Sgd finds then tunes features in two\-layer neural networks with near\-optimal sample complexity: A case study in the xor problem\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.
- Dandi et al\. \[2024\]Yatin Dandi, Emanuele Troiani, Luca Arnaboldi, Luca Pesce, Lenka Zdeborová, and Florent Krzakala\.The benefit of reusing batches for gradient descent in two\-layer networks: Breaking the curse of information and leap exponents\.In*International Conference on Machine Learning \(ICML\)*, 2024\.
- Dong \[2026\]Lei Dong\.Circuits crystallize: A nucleation theory of whether, when, and how conjunction circuits form, compete, and regenerate\.Zenodo, doi:10\.5281/zenodo\.21253471, 2026\.
- Freidlin and Wentzell \[2012\]Mark I\. Freidlin and Alexander D\. Wentzell\.*Random Perturbations of Dynamical Systems*\.Springer, 3rd edition, 2012\.
- Dembo and Zeitouni \[1998\]Amir Dembo and Ofer Zeitouni\.*Large Deviations Techniques and Applications*\.Springer, 2nd edition, 1998\.
- Kramers \[1940\]Hendrik A\. Kramers\.Brownian motion in a field of force and the diffusion model of chemical reactions\.*Physica*, 7\(4\):284–304, 1940\.
- Ben Arous et al\. \[2021\]Gérard Ben Arous, Reza Gheissari, and Aukosh Jagannath\.Online stochastic gradient descent on non\-convex losses from high\-dimensional inference\.*Journal of Machine Learning Research*, 22\(106\):1–51, 2021\.
- Chaudhari and Soatto \[2018\]Pratik Chaudhari and Stefano Soatto\.Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks\.*International Conference on Learning Representations*, 2018\.
- Ziyin et al\. \[2026\]Liu Ziyin, Yizhou Ren, Herbert Levine, and Isaac Chuang\.Thermodynamic irreversibility of training algorithms\.*arXiv preprint arXiv:2605\.21933*, 2026\.
- Olsson et al\. \[2022\]Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, et al\.In\-context learning and induction heads\.*Transformer Circuits Thread*, 2022\.
- Day \[1983\]Martin V\. Day\.On the exponential exit law in the small parameter exit problem\.*Stochastics*, 8\(4\):297–323, 1983\.
- Bovier and den Hollander \[2015\]Anton Bovier and Frank den Hollander\.*Metastability: A Potential\-Theoretic Approach*\.Springer, 2015\.
- Kubo \[1966\]Rep Kubo\.The fluctuation\-dissipation theorem\.*Reports on Progress in Physics*, 29\(1\):255–284, 1966\.
- Lavie et al\. \[2026\]Itamar Lavie, Kirsten Fischer, Stefan Lekov, Ward Van Maele, Zohar Ringel, and Moritz Helias\.Phase transitions in attention: A bayesian theory of copy head emergence\.*arXiv preprint arXiv:2606\.12058*, 2026\.
- Baherwani et al\. \[2026\]Vedang Baherwani, Yifei Chen, Shikai Qiu, Andrew Gordon Wilson, and Pavel Izmailov\.Emergent capabilities arise randomly from learning sparse attention patterns\.*arXiv preprint arXiv:2606\.25010*, 2026\.
- Hajek \[1982\]Bruce Hajek\.Hitting\-time and occupation\-time bounds implied by drift analysis with applications\.*Advances in Applied Probability*, 14\(3\):502–525, 1982\.
- Sokar et al\. \[2023\]Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci\.The dormant neuron phenomenon in deep reinforcement learning\.In*International Conference on Machine Learning*, 2023\.
- Lyle et al\. \[2026\]Clare Lyle, Ghada Sokar, Razvan Pascanu, and András György\.What can grokking teach us about learning under nonstationarity?*arXiv preprint arXiv:2507\.20057*, 2026\.
- Lee et al\. \[2024\]Jaerin Lee, Bong Gyun Kang, Kihoon Kim, and Kyoung Mu Lee\.Grokfast: Accelerated grokking by amplifying slow gradients\.In*arXiv preprint arXiv:2405\.20233*, 2024\.
- Avrami \[1939\]Melvin Avrami\.Kinetics of phase change\. i: General theory\.*Journal of Chemical Physics*, 7\(12\):1103–1112, 1939\.
- Martin and Bellon \[1996\]Georges Martin and Pascal Bellon\.Driven alloys\.*Solid State Physics*, 50:189–331, 1996\.
- Laidler \[1987\]Keith J\. Laidler\.*Chemical Kinetics*\.Harper & Row, 3rd edition, 1987\.
- Porter and Easterling \[1992\]David A\. Porter and Kenneth E\. Easterling\.*Phase Transformations in Metals and Alloys*\.Chapman & Hall, 2nd edition, 1992\.
- Qi et al\. \[2024\]Xiangyu Qi, Yi Zeng, Tinghao Xie, et al\.Fine\-tuning aligned language models compromises safety, even when users do not intend to\!*International Conference on Learning Representations*, 2024\.
- Aoyama et al\. \[2025\]Tatsuya Aoyama, Ethan Wilcox, and Nathan Schneider\.Predicting the emergence of induction heads in language model pretraining\.*arXiv preprint arXiv:2511\.16893*, 2025\.
- Wei et al\. \[2022\]Jason Wei, Yi Tay, Rishi Bommasani, et al\.Emergent abilities of large language models\.*Transactions on Machine Learning Research*, 2022\.
- Schaeffer et al\. \[2023\]Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo\.Are emergent abilities of large language models a mirage?In*Advances in Neural Information Processing Systems*, 2023\.
- Elhage et al\. \[2021\]Nelson Elhage, Neel Nanda, Catherine Olsson, et al\.A mathematical framework for transformer circuits\.*Transformer Circuits Thread*, 2021\.
- Reddy \[2024\]Gautam Reddy\.The mechanistic basis of data dependence and abrupt learning in an in\-context classification task\.*International Conference on Learning Representations*, 2024\.
- Singh et al\. \[2023\]Aaditya K\. Singh, Stephanie C\. Y\. Chan, Ted Moskovitz, Erin Grant, Andrew M\. Saxe, and Felix Hill\.The transient nature of emergent in\-context learning in transformers\.*Advances in Neural Information Processing Systems*, 2023\.
- Yaida \[2019\]Sho Yaida\.Fluctuation\-dissipation relations for stochastic gradient descent\.In*International Conference on Learning Representations*, 2019\.
- Neelakantan et al\. \[2015\]Arvind Neelakantan, Luke Vilnis, Quoc V\. Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens\.Adding gradient noise improves learning for very deep networks\.*arXiv preprint arXiv:1511\.06807*, 2015\.
- Singh et al\. \[2024\]Aaditya K\. Singh, Ted Moskovitz, Felix Hill, Stephanie C\. Y\. Chan, and Andrew M\. Saxe\.What needs to go right for an induction head? a mechanistic study of in\-context learning circuits and their formation\.*International Conference on Machine Learning*, 2024\.
- Hallgren \[2025\]Jonas Hallgren\.Have you tried thinking about it as crystals?LessWrong, December 2025, 2025\.