Same Loss, Same Noise, Opposite Schedules: Noise Structure and Optimizer Normalization Jointly Determine Whether Learning-Rate Cooldown Helps

# Same Loss, Same Noise, Opposite Schedules: Noise Structure and Optimizer Normalization Jointly Determine Whether Learning-Rate Cooldown Helps
Source: [https://arxiv.org/html/2607.12360](https://arxiv.org/html/2607.12360)
###### Abstract

The cooldown phase of a warmup–stable–decay \(WSD\) learning\-rate schedule, now a default in large\-model pretraining, lowers the final training loss in some settings and does nothing in others\. We give a provable account of which case obtains, and it turns on two properties together: the structure of the gradient noise and whether the optimizer normalizes its update\. On a strongly convex objective with multiplicative \(gradient\-proportional\) noise, stochastic gradient descent contracts geometrically at a constant learning rate, so cooldown has nothing to improve\. Under the same objective and noise, sign\-based and normalized methods, the standard surrogates for adaptive optimizers, settle on a noise floor of orderη2\\eta^\{2\}and reach the minimizer only as the learning rate is driven to zero; any additive noise then reinstates a floor for every method\. The mechanism is elementary: an SGD step shrinks in proportion to the gradient and so anneals itself, whereas a normalized step keeps unit scale and cannot\. We solve the signSGD stationary law on the quadratic exactly and obtain the floor constant in closed form, prove a local form of the dissociation under\(L0,L1\)\(L\_\{0\},L\_\{1\}\)\-smoothness, extend the floor to normalized SGD in dimensiond\>1d\>1by a scale\-invariance argument, and establish robustness to momentum and heavy\-tailed noise\. Simulation confirms every prediction, and we demonstrate the resulting noise\-regime diagnostic on a real classification task with directly measured gradient noise\. The mechanism explains*whether*cooldown helps; the*interior*cooldown fraction used at scale lies outside stationary landscape\-and\-noise geometry\.

## 1Introduction

The learning\-rate schedule is among the most consequential and least understood choices in neural network training\. Warmup–stable–decay \(WSD\) schedules hold the rate constant for most of training and then cool it down to a small final value\(Huet al\.[2024](https://arxiv.org/html/2607.12360#bib.bib8)\)\. They have become a default for large language model pretraining, where they match or exceed cosine schedules while letting training resume from any stable\-phase checkpoint\(Hägeleet al\.[2024](https://arxiv.org/html/2607.12360#bib.bib10)\), and their qualitative dynamics are strikingly consistent across architectures\(Belloniet al\.[2026](https://arxiv.org/html/2607.12360#bib.bib1)\)\. What remains missing is a theoretical account of*when*the cooldown phase actually helps\.

Part of the difficulty is that cooldown is a last\-iterate effect: it lowers the loss of the final iterate without improving the best loss seen so far\. It therefore leaves no trace in the min\-over\-iterates and randomly\-sampled\-iterate guarantees that dominate nonconvex optimization theory\(Monzio Compagnoniet al\.[2026](https://arxiv.org/html/2607.12360#bib.bib3); Shulginet al\.[2026](https://arxiv.org/html/2607.12360#bib.bib18)\), and even sharp last\-iterate analyses of SGD\(Shamir and Zhang[2013](https://arxiv.org/html/2607.12360#bib.bib9)\)concern the additive\-noise regime\. Convex last\-iterate bounds do reproduce the shape of practical schedules remarkably well\(Schaippet al\.[2025](https://arxiv.org/html/2607.12360#bib.bib16)\), but they too treat \(sub\)gradient methods under additive\-type noise\. Nothing in either line of work predicts that changing the*optimizer*, with the loss and the noise held fixed, should change whether cooldown helps at all\.

This paper shows that it does\. We do not try to resolve the WSD cooldown fraction observed at scale\. We isolate a narrower question that admits sharp answers: with the loss and the noise held fixed, does the optimizer alone decide whether cooldown helps? It does, and the deciding property is whether the magnitude of the update shrinks as the gradient shrinks\.

Our setting is the affine\-variance noise model ofMonzio Compagnoniet al\.\([2026](https://arxiv.org/html/2607.12360#bib.bib3), Def\. 3\.2\), which bounds the gradient\-noise variance byσ02\+σ12​‖∇f‖2\\sigma\_\{0\}^\{2\}\+\\sigma\_\{1\}^\{2\}\\\|\\nabla f\\\|^\{2\}\. Theσ02\\sigma\_\{0\}^\{2\}term is the usual bounded additive noise\. Theσ12\\sigma\_\{1\}^\{2\}term scales with the gradient and vanishes at the optimum, as in the interpolation regime of over\-parameterized models\(Schmidt and Le Roux[2013](https://arxiv.org/html/2607.12360#bib.bib17); Vaswaniet al\.[2019](https://arxiv.org/html/2607.12360#bib.bib19)\)\. Our contributions are as follows\.

- •SGD self\-anneals under multiplicative noise\.Withσ0=0\\sigma\_\{0\}=0the second moment obeys an exactly multiplicative recursion with no additive term \(Prop\.[1](https://arxiv.org/html/2607.12360#Thmproposition1)\): the optimal fixed\-horizon schedule is the constant rateη⋆=1/\(λ​\(1\+σ12\)\)\\eta^\{\\star\}=1/\(\\lambda\(1\+\\sigma\_\{1\}^\{2\}\)\), convergence is geometric, and the optimal cooldown fraction is0\. This half restates classical interpolation results in schedule language, and we claim no novelty for it\.
- •Normalized methods cannot self\-anneal\.Under the same noise, the probability that signSGD takes a correct step is independent of the iterate, for*any*noise law \(Lemma[1](https://arxiv.org/html/2607.12360#Thmlemma1)\)\. On the quadratic the induced dynamics form an explicitly solvable lattice walk: the last\-iterate loss is floored atΘ​\(η2\)\\Theta\(\\eta^\{2\}\), with the floor constant in closed form \(Prop\.[2](https://arxiv.org/html/2607.12360#Thmproposition2), Remark[3](https://arxiv.org/html/2607.12360#Thmremark3)\)\. Cooldown is necessary\. Any additive component \(σ0\>0\\sigma\_\{0\}\>0\) restores anη\\eta\-dependent floor for every method \(Prop\.[3](https://arxiv.org/html/2607.12360#Thmproposition3)\)\.
- •Robustness\.A local version of the dissociation holds under\(L0,L1\)\(L\_\{0\},L\_\{1\}\)\-smoothness \(Sec\.[6](https://arxiv.org/html/2607.12360#S6)\); normalized SGD ind\>1d\>1has a dimension\-free granularity floor, with a scale\-invariance argument for why it cannot anneal \(Sec\.[7](https://arxiv.org/html/2607.12360#S7)\); the classification is inherited under momentum \(Sec\.[8](https://arxiv.org/html/2607.12360#S8)\) and moment\-free under heavy tails \(Sec\.[9](https://arxiv.org/html/2607.12360#S9)\)\.
- •Verification and scope\.Every quantitative prediction is checked against simulation, including WSD sweeps on a nonconvex deep linear network, and the noise structure and schedule preferences are measured directly on a real classification task \(Sec\.[10](https://arxiv.org/html/2607.12360#S10), Figs\.[1](https://arxiv.org/html/2607.12360#S1.F1)and[2](https://arxiv.org/html/2607.12360#S10.F2)\)\. We state plainly what the mechanism does not explain \(Sec\.[11](https://arxiv.org/html/2607.12360#S11)\)\.

![Refer to caption](https://arxiv.org/html/2607.12360v1/x1.png)Figure 1:Last\-iterate loss versus cooldown fraction under WSD schedules, with the peak rate tuned separately for every point \(mean over512512runs for \(a\)–\(b\),88runs for \(c\)\)\. \(a\) Anisotropic quadratic \(d=20d=20\), pure multiplicative noise: cooldown strictly hurts SGD and improves signSGD and normalized SGD by about six orders of magnitude\. \(b\) Same quadratic with an added additive component \(σ0=0\.2\\sigma\_\{0\}=0\.2\): every method now benefits from cooldown\. \(c\) Deep linear network with a planted teacher, multiplicative noise: SGD sits at numerical zero \(≈10−32\\approx 10^\{\-32\}\) for every cooldown fraction while signSGD requires strong cooldown\.
## 2Setup and Noise Model

We minimize the separable quadraticf​\(x\)=12​∑i=1dλi​xi2f\(x\)=\\tfrac\{1\}\{2\}\\sum\_\{i=1\}^\{d\}\\lambda\_\{i\}x\_\{i\}^\{2\}withλi\>0\\lambda\_\{i\}\>0; Section[6](https://arxiv.org/html/2607.12360#S6)treats an\(L0,L1\)\(L\_\{0\},L\_\{1\}\)\-smooth extension\. Gradients are observed with noise,g^t=∇f​\(xt\)\+ξt\\hat\{g\}\_\{t\}=\\nabla f\(x\_\{t\}\)\+\\xi\_\{t\}\. FollowingMonzio Compagnoniet al\.\([2026](https://arxiv.org/html/2607.12360#bib.bib3), Def\. 3\.2\), who bound the gradient\-noise variance byσ02\+σ12​‖∇f‖2\\sigma\_\{0\}^\{2\}\+\\sigma\_\{1\}^\{2\}\\\|\\nabla f\\\|^\{2\}, we work with the coordinatewise instance of that model in which the bound is met exactly:

ξi=σ0​Zi0\+σ1​\|∂if​\(x\)\|​Zi,\\xi\_\{i\}=\\sigma\_\{0\}Z^\{0\}\_\{i\}\+\\sigma\_\{1\}\\,\|\\partial\_\{i\}f\(x\)\|\\,Z\_\{i\},\(1\)where the joint law of\(Z0,Z\)\(Z^\{0\},Z\)does not depend onxxand, whenever the moments exist,𝔼​\[Zi\]=𝔼​\[Zi0\]=0\\mathbb\{E\}\[Z\_\{i\}\]=\\mathbb\{E\}\[Z^\{0\}\_\{i\}\]=0and𝔼​\[Zi2\]=𝔼​\[\(Zi0\)2\]=1\\mathbb\{E\}\[Z\_\{i\}^\{2\}\]=\\mathbb\{E\}\[\(Z^\{0\}\_\{i\}\)^\{2\}\]=1\. “Multiplicative” meansσ0=0\\sigma\_\{0\}=0\. In that case the noise is a*scale family*: the standardized noiseZZis iterate\-independent, and the noise vanishes at the optimum\. This structural property, not Gaussianity, is what our results use; Lemma[1](https://arxiv.org/html/2607.12360#Thmlemma1)and the floors below hold for arbitraryZZ, including laws without any finite moments \(Sec\.[9](https://arxiv.org/html/2607.12360#S9)\)\.

We study the last\-iterate loss𝔼​\[f​\(xT\)\]\\mathbb\{E\}\[f\(x\_\{T\}\)\]under WSD schedules:ηt=η\\eta\_\{t\}=\\etafort<\(1−cf\)​Tt<\(1\-\\mathrm\{cf\}\)T, after whichηt\\eta\_\{t\}decays linearly to0, with peak rateη\\etaand cooldown fractioncf∈\[0,1\]\\mathrm\{cf\}\\in\[0,1\]\.

## 3SGD Self\-Anneals under Multiplicative Noise

###### Proposition 1\(SGD, pure multiplicative noise\)\.

Consider one coordinate with curvatureλ\\lambdaandσ0=0\\sigma\_\{0\}=0, and writemt:=𝔼​\[xt2\]m\_\{t\}:=\\mathbb\{E\}\[x\_\{t\}^\{2\}\]\. Then

mt\+1=r​\(ηt\)​mt,mT=m0​∏t<Tr​\(ηt\),m\_\{t\+1\}=r\(\\eta\_\{t\}\)\\,m\_\{t\},\\qquad m\_\{T\}=m\_\{0\}\\textstyle\\prod\_\{t<T\}r\(\\eta\_\{t\}\),\(2\)withr​\(η\)=\(1−η​λ\)2\+η2​σ12​λ2r\(\\eta\)=\(1\-\\eta\\lambda\)^\{2\}\+\\eta^\{2\}\\sigma\_\{1\}^\{2\}\\lambda^\{2\}, and: \(i\)rris minimized atη⋆=1/\(λ​\(1\+σ12\)\)\\eta^\{\\star\}=1/\(\\lambda\(1\+\\sigma\_\{1\}^\{2\}\)\), wherer​\(η⋆\)=σ12/\(1\+σ12\)<1r\(\\eta^\{\\star\}\)=\\sigma\_\{1\}^\{2\}/\(1\+\\sigma\_\{1\}^\{2\}\)<1; \(ii\) over schedules\{ηt∈\[0,ηmax\]\}t<T\\\{\\eta\_\{t\}\\in\[0,\\eta\_\{\\max\}\]\\\}\_\{t<T\}withTTfixed,mTm\_\{T\}is minimized by the constant scheduleηt≡min⁡\(η⋆,ηmax\)\\eta\_\{t\}\\equiv\\min\(\\eta^\{\\star\},\\eta\_\{\\max\}\), so the optimal cooldown fraction within the WSD family is0; \(iii\) atη⋆\\eta^\{\\star\},𝔼​\[f​\(xT\)\]=f​\(x0\)​\(σ12/\(1\+σ12\)\)T→0\\mathbb\{E\}\[f\(x\_\{T\}\)\]=f\(x\_\{0\}\)\\big\(\\sigma\_\{1\}^\{2\}/\(1\+\\sigma\_\{1\}^\{2\}\)\\big\)^\{T\}\\to 0geometrically, with no noise floor\. Indddimensions every coordinate contracts geometrically at any constant rate withmaxi⁡ri​\(η\)<1\\max\_\{i\}r\_\{i\}\(\\eta\)<1, so the loss again decays geometrically to zero\.

###### Proof\.

Fromg^t=λ​xt\+ξt\\hat\{g\}\_\{t\}=\\lambda x\_\{t\}\+\\xi\_\{t\}we getxt\+1=\(1−ηt​λ\)​xt−ηt​ξtx\_\{t\+1\}=\(1\-\\eta\_\{t\}\\lambda\)x\_\{t\}\-\\eta\_\{t\}\\xi\_\{t\}; conditioning onxtx\_\{t\}and using𝔼​\[ξt∣xt\]=0\\mathbb\{E\}\[\\xi\_\{t\}\\mid x\_\{t\}\]=0,𝔼​\[ξt2∣xt\]=σ12​λ2​xt2\\mathbb\{E\}\[\\xi\_\{t\}^\{2\}\\mid x\_\{t\}\]=\\sigma\_\{1\}^\{2\}\\lambda^\{2\}x\_\{t\}^\{2\}gives \([2](https://arxiv.org/html/2607.12360#S3.E2)\)\. For \(i\),r′​\(η\)=−2​λ\+2​η​λ2​\(1\+σ12\)r^\{\\prime\}\(\\eta\)=\-2\\lambda\+2\\eta\\lambda^\{2\}\(1\+\\sigma\_\{1\}^\{2\}\)vanishes atη⋆\\eta^\{\\star\}, and substituting1−η⋆​λ=σ12/\(1\+σ12\)1\-\\eta^\{\\star\}\\lambda=\\sigma\_\{1\}^\{2\}/\(1\+\\sigma\_\{1\}^\{2\}\)givesr​\(η⋆\)r\(\\eta^\{\\star\}\)\. For \(ii\),log⁡mT=log⁡m0\+∑tlog⁡r​\(ηt\)\\log m\_\{T\}=\\log m\_\{0\}\+\\sum\_\{t\}\\log r\(\\eta\_\{t\}\)is minimized term by term, andrris decreasing on\[0,η⋆\]\[0,\\eta^\{\\star\}\]\. Part \(iii\) applies \([2](https://arxiv.org/html/2607.12360#S3.E2)\) coordinatewise and sums12​λi​𝔼​\[xi,T2\]\\tfrac\{1\}\{2\}\\lambda\_\{i\}\\mathbb\{E\}\[x\_\{i,T\}^\{2\}\]\. ∎

The noise contributionηt2​σ12​λ2​mt\\eta\_\{t\}^\{2\}\\sigma\_\{1\}^\{2\}\\lambda^\{2\}m\_\{t\}is proportional tomtm\_\{t\}, so it forms part of the geometric contraction rather than an additive floor\. Near the optimum the SGD stepηt​g^t∝ηt​xt\\eta\_\{t\}\\hat\{g\}\_\{t\}\\propto\\eta\_\{t\}x\_\{t\}shrinks on its own, and the schedule has nothing further to contribute\.

## 4Normalized Methods Cannot Self\-Anneal

###### Lemma 1\(Iterate\-independent sign reliability\)\.

Under scale\-family multiplicative noise on one coordinate,

ℙ​\(sign⁡\(g^\)=sign⁡\(x\)\)=\{ℙ​\(Z\>−1/σ1\),x\>0,ℙ​\(Z<1/σ1\),x<0,\\mathbb\{P\}\\big\(\\operatorname\{sign\}\(\\hat\{g\}\)=\\operatorname\{sign\}\(x\)\\big\)=\\begin\{cases\}\\mathbb\{P\}\(Z\>\-1/\\sigma\_\{1\}\),&x\>0,\\\\\[2\.0pt\] \\mathbb\{P\}\(Z<\\phantom\{\-\}1/\\sigma\_\{1\}\),&x<0,\\end\{cases\}\(3\)independent of\|x\|\|x\|for every law ofZZ\. IfZZis symmetric the two values coincide; call thempp\. GaussianZZgivesp=Φ​\(1/σ1\)p=\\Phi\(1/\\sigma\_\{1\}\), and in generalp\>12p\>\\tfrac\{1\}\{2\}if and only ifℙ​\(Z≤−1/σ1\)<12\\mathbb\{P\}\(Z\\leq\-1/\\sigma\_\{1\}\)<\\tfrac\{1\}\{2\}, which holds in particular wheneverZZhas median zero and positive mass on\(−1/σ1,0\]\(\-1/\\sigma\_\{1\},0\]\.

###### Proof\.

Forx\>0x\>0,g^=λ​x​\(1\+σ1​Z\)\\hat\{g\}=\\lambda x\(1\+\\sigma\_\{1\}Z\), sosign⁡\(g^\)=sign⁡\(1\+σ1​Z\)\\operatorname\{sign\}\(\\hat\{g\}\)=\\operatorname\{sign\}\(1\+\\sigma\_\{1\}Z\); the casex<0x<0is analogous\. \(Ties1\+σ1​Z=01\+\\sigma\_\{1\}Z=0have probability zero for continuousZZ\.\) ∎

The probability of a correct, restoring step does not improve as the iterate approaches the optimum\. Taking the sign discards precisely the gradient\-magnitude information that produces SGD’s self\-annealing\.

###### Proposition 2\(signSGD floor\)\.

Run signSGD,xt\+1=xt−η​sign⁡\(g^t\)x\_\{t\+1\}=x\_\{t\}\-\\eta\\,\\operatorname\{sign\}\(\\hat\{g\}\_\{t\}\), with constantη\\etaon one coordinate with curvatureλ\>0\\lambda\>0, pure multiplicative noise with symmetricZZ, andx0=\(k\+φ\)​η∉η​ℤx\_\{0\}=\(k\+\\varphi\)\\eta\\notin\\eta\\mathbb\{Z\}with offsetφ∈\(0,1\)\\varphi\\in\(0,1\)\(any continuous initialization satisfies this almost surely\)\. Then:

\(i\)*\(Granularity\.\)*Every step changesxxby exactly±η\\pm\\eta, somax⁡\(\|xt\|,\|xt\+1\|\)≥η/2\\max\(\|x\_\{t\}\|,\|x\_\{t\+1\}\|\)\\geq\\eta/2for alltt, and

𝔼​\[f​\(xT\)\]\+𝔼​\[f​\(xT\+1\)\]≥18​λ​η2for every​T,\\mathbb\{E\}\[f\(x\_\{T\}\)\]\+\\mathbb\{E\}\[f\(x\_\{T\+1\}\)\]\\;\\geq\\;\\tfrac\{1\}\{8\}\\lambda\\eta^\{2\}\\quad\\text\{for every \}T,hencelim supT𝔼​\[f​\(xT\)\]≥116​λ​η2\\limsup\_\{T\}\\mathbb\{E\}\[f\(x\_\{T\}\)\]\\geq\\tfrac\{1\}\{16\}\\lambda\\eta^\{2\}: the loss cannot converge to zero at constantη\\eta\.

\(ii\)*\(Exact floor\.\)*\|xt\|\|x\_\{t\}\|is a reflected random walk on the lattice\{\(j\+φ\)​η\}j≥0∪\{\(j\+1−φ\)​η\}j≥0\\\{\(j\+\\varphi\)\\eta\\\}\_\{j\\geq 0\}\\cup\\\{\(j\+1\-\\varphi\)\\eta\\\}\_\{j\\geq 0\}with iterate\-independent inward probabilitypp\(Lemma[1](https://arxiv.org/html/2607.12360#Thmlemma1)\)\. Forp∈\(12,1\]p\\in\(\\tfrac\{1\}\{2\},1\]its invariant law is explicit, and under it, withq:=1−pq:=1\-p,

𝔼​\[x2\]=c​\(φ,p\)​η2,c​\(φ,p\)=φ2\+\(1−φ\)22\+q2​p−1\+q\(2​p−1\)2,\\begin\{gathered\}\\mathbb\{E\}\[x^\{2\}\]=c\(\\varphi,p\)\\,\\eta^\{2\},\\\\ c\(\\varphi,p\)=\\frac\{\\varphi^\{2\}\+\(1\-\\varphi\)^\{2\}\}\{2\}\+\\frac\{q\}\{2p\-1\}\+\\frac\{q\}\{\(2p\-1\)^\{2\}\},\\end\{gathered\}\(4\)so the stationary loss is12​λ​c​\(φ,p\)​η2=Θ​\(η2\)\\tfrac\{1\}\{2\}\\lambda\\,c\(\\varphi,p\)\\,\\eta^\{2\}=\\Theta\(\\eta^\{2\}\)\. Driving the loss to zero requiresηt→0\\eta\_\{t\}\\to 0\.

###### Proof sketch\.

Sincex0∉η​ℤx\_\{0\}\\notin\\eta\\mathbb\{Z\}, the iterate never hits0,g^t≠0\\hat\{g\}\_\{t\}\\neq 0almost surely, and each step is exactly±η\\pm\\eta; if\|xt\|<η/2\|x\_\{t\}\|<\\eta/2then\|xt\+1\|=η−\|xt\|\>η/2\|x\_\{t\+1\}\|=\\eta\-\|x\_\{t\}\|\>\\eta/2, which givesxt2\+xt\+12≥η2/4x\_\{t\}^\{2\}\+x\_\{t\+1\}^\{2\}\\geq\\eta^\{2\}/4pointwise and hence \(i\)\. For \(ii\), Lemma[1](https://arxiv.org/html/2607.12360#Thmlemma1)makes the inward/outward indicators i\.i\.d\. Bernoulli\(pp\), so\|xt\|\|x\_\{t\}\|is a birth–death chain on two half\-lattices joined at the bottom \(the walk crosses0betweenφ​η\\varphi\\etaand\(1−φ\)​η\(1\-\\varphi\)\\eta\)\. A chain on a tree is reversible; detailed balance gives geometric level weights proportional to\(q/p\)j\(q/p\)^\{j\}on both half\-lattices with equal weight at the two bottom states, and summingx2x^\{2\}against these weights yields \([4](https://arxiv.org/html/2607.12360#S4.E4)\); the full computation is in the technical appendix\. ∎

###### Corollary 1\(Coordinatewise signSGD,d≥1d\\geq 1\)\.

On the separable quadratic withx0,i∉η​ℤx\_\{0,i\}\\notin\\eta\\mathbb\{Z\}for everyii, coordinatewise signSGD moves each coordinate by exactly±η\\pm\\etaper step, so Proposition[2](https://arxiv.org/html/2607.12360#Thmproposition2)\(i\) applies coordinatewise and𝔼​\[f​\(xT\)\]\+𝔼​\[f​\(xT\+1\)\]≥η28​∑i=1dλi\\mathbb\{E\}\[f\(x\_\{T\}\)\]\+\\mathbb\{E\}\[f\(x\_\{T\+1\}\)\]\\geq\\tfrac\{\\eta^\{2\}\}\{8\}\\sum\_\{i=1\}^\{d\}\\lambda\_\{i\}for everyTT\. The floor grows with the trace of the curvature and holds for any noise law\.

###### Proof\.

g^i=λi​xi​\(1\+σ1​Zi\)≠0\\hat\{g\}\_\{i\}=\\lambda\_\{i\}x\_\{i\}\(1\+\\sigma\_\{1\}Z\_\{i\}\)\\neq 0almost surely, so each coordinate is a±η\\pm\\etawalk on its own lattice; apply the pair bound of Proposition[2](https://arxiv.org/html/2607.12360#Thmproposition2)\(i\) per coordinate and sum\. ∎

### Mechanism\.

The SGD stepη​g^∝η​x\\eta\\hat\{g\}\\propto\\eta xshrinks with the iterate, whereas the signSGD stepη​sign⁡\(g^\)\\eta\\,\\operatorname\{sign\}\(\\hat\{g\}\)keeps a fixed magnitude however close the iterate is to the optimum\. Any annealing the update does not provide on its own must instead come from the schedule\.

## 5The Dissociation and the Additive Boundary

Table 1:Optimal schedule by noise structure and optimizer\.###### Proposition 3\(Additive regime\)\.

Letσ0\>0\\sigma\_\{0\}\>0and0<η<2/\(λ​\(1\+σ12\)\)0<\\eta<2/\(\\lambda\(1\+\\sigma\_\{1\}^\{2\}\)\), so thata​\(η\):=\(1−η​λ\)2\+η2​σ12​λ2<1a\(\\eta\):=\(1\-\\eta\\lambda\)^\{2\}\+\\eta^\{2\}\\sigma\_\{1\}^\{2\}\\lambda^\{2\}<1\. The SGD second moment obeys the affine recursionmt\+1=a​\(η\)​mt\+η2​σ02m\_\{t\+1\}=a\(\\eta\)m\_\{t\}\+\\eta^\{2\}\\sigma\_\{0\}^\{2\}, whose fixed point

m∞=η2​σ021−a​\(η\)=η​σ022​λ​\(1\+O​\(η\)\)m\_\{\\infty\}=\\frac\{\\eta^\{2\}\\sigma\_\{0\}^\{2\}\}\{1\-a\(\\eta\)\}=\\frac\{\\eta\\,\\sigma\_\{0\}^\{2\}\}\{2\\lambda\}\\big\(1\+O\(\\eta\)\\big\)\(5\)is strictly positive and strictly increasing inη\\etaon the admissible range\. Reducing the terminal rate therefore strictly reduces the last\-iterate floor: cooldown helps*every*optimizer\.

###### Proof\.

Iterating the affine map givesmt=at​m0\+η2​σ02​∑k<tak→η2​σ02/\(1−a\)m\_\{t\}=a^\{t\}m\_\{0\}\+\\eta^\{2\}\\sigma\_\{0\}^\{2\}\\sum\_\{k<t\}a^\{k\}\\to\\eta^\{2\}\\sigma\_\{0\}^\{2\}/\(1\-a\), and1−a​\(η\)=η​λ​\(2−η​λ​\(1\+σ12\)\)1\-a\(\\eta\)=\\eta\\lambda\(2\-\\eta\\lambda\(1\+\\sigma\_\{1\}^\{2\}\)\)\. Differentiatingm∞m\_\{\\infty\}inη\\etagives2​σ02​λ/\[λ​\(2−η​λ​\(1\+σ12\)\)\]2\>02\\sigma\_\{0\}^\{2\}\\lambda/\[\\lambda\(2\-\\eta\\lambda\(1\+\\sigma\_\{1\}^\{2\}\)\)\]^\{2\}\>0\. ∎

The dissociation is thus specific to the multiplicative regime: an additive component reintroduces a floor that only decay removes, and the two optimizer families agree once more \(Table[1](https://arxiv.org/html/2607.12360#S5.T1)\)\. The same objective under the same noise can call for opposite schedules, and the property that decides between them is whether the update self\-anneals\. One caveat on Table[1](https://arxiv.org/html/2607.12360#S5.T1): for the anisotropic quadratic under a single shared rate, the constant\-schedule optimality for SGD is proven per coordinate and confirmed numerically \(Table[3](https://arxiv.org/html/2607.12360#S10.T3)\); the termwise argument of Proposition[1](https://arxiv.org/html/2607.12360#Thmproposition1)\(ii\) is scalar\.

One limitation is visible already here\. In the pure quadratic, the additive\-regime optimum within the WSD family is full decay, not an interior cooldown fraction, and no model in this paper produces an interior optimum \(Sec\.[11](https://arxiv.org/html/2607.12360#S11)\)\.

## 6A Local Theorem under\(L0,L1\)\(L\_\{0\},L\_\{1\}\)\-Smoothness

Letf​\(x\)=λ2​x2\+c4​x4f\(x\)=\\tfrac\{\\lambda\}\{2\}x^\{2\}\+\\tfrac\{c\}\{4\}x^\{4\}withc≥0c\\geq 0, sof′​\(x\)=λ​x\+c​x3f^\{\\prime\}\(x\)=\\lambda x\+cx^\{3\}and\|f′′​\(x\)\|=λ\+3​c​x2≤4​λ\+3​c/λ​\|f′​\(x\)\|\|f^\{\\prime\\prime\}\(x\)\|=\\lambda\+3cx^\{2\}\\leq 4\\lambda\+3\\sqrt\{c/\\lambda\}\\,\|f^\{\\prime\}\(x\)\|: the objective is\(L0,L1\)\(L\_\{0\},L\_\{1\}\)\-smooth in the sense ofZhanget al\.\([2020](https://arxiv.org/html/2607.12360#bib.bib21)\), withL0=4​λL\_\{0\}=4\\lambdaandL1=3​c/λL\_\{1\}=3\\sqrt\{c/\\lambda\}, but notLL\-smooth forc\>0c\>0\. Writes​\(x\):=f′​\(x\)/x=λ\+c​x2≥λs\(x\):=f^\{\\prime\}\(x\)/x=\\lambda\+cx^\{2\}\\geq\\lambda\.

###### Proposition 4\(SGD: local geometric convergence, no floor\)\.

Assume scale\-family multiplicative noise with\|Z\|≤M\|Z\|\\leq Malmost surely andσ1​M≤1\\sigma\_\{1\}M\\leq 1\. Fix a basinBρ=\{\|x\|≤ρ\}B\_\{\\rho\}=\\\{\|x\|\\leq\\rho\\\}and letsρ=λ\+c​ρ2s\_\{\\rho\}=\\lambda\+c\\rho^\{2\}\. If

η≤min⁡\{2sρ​\(1\+σ1​M\),1sρ​\(1\+σ12\)\},\\eta\\;\\leq\\;\\min\\\!\\Big\\\{\\frac\{2\}\{s\_\{\\rho\}\(1\+\\sigma\_\{1\}M\)\},\\ \\frac\{1\}\{s\_\{\\rho\}\(1\+\\sigma\_\{1\}^\{2\}\)\}\\Big\\\},thenBρB\_\{\\rho\}is forward\-invariant and𝔼​\[xt\+12∣xt\]≤r​\(λ,η\)​xt2\\mathbb\{E\}\[x\_\{t\+1\}^\{2\}\\mid x\_\{t\}\]\\leq r\(\\lambda,\\eta\)\\,x\_\{t\}^\{2\}withr​\(λ,η\)=\(1−η​λ\)2\+η2​σ12​λ2<1r\(\\lambda,\\eta\)=\(1\-\\eta\\lambda\)^\{2\}\+\\eta^\{2\}\\sigma\_\{1\}^\{2\}\\lambda^\{2\}<1\. Hence𝔼​\[xt2\]≤r​\(λ,η\)t​x02→0\\mathbb\{E\}\[x\_\{t\}^\{2\}\]\\leq r\(\\lambda,\\eta\)^\{t\}x\_\{0\}^\{2\}\\to 0geometrically, with no floor; the rate is governed by the curvature at the minimum, and its infimum over admissibleη\\etaapproachesσ12/\(1\+σ12\)\\sigma\_\{1\}^\{2\}/\(1\+\\sigma\_\{1\}^\{2\}\)asρ→0\\rho\\to 0, recovering Proposition[1](https://arxiv.org/html/2607.12360#Thmproposition1)\.

###### Proof\.

The update isxt\+1=xt​\(1−η​s​\(xt\)​\(1\+σ1​Zt\)\)x\_\{t\+1\}=x\_\{t\}\\big\(1\-\\eta s\(x\_\{t\}\)\(1\+\\sigma\_\{1\}Z\_\{t\}\)\\big\)\. For\|xt\|≤ρ\|x\_\{t\}\|\\leq\\rhoand\|Zt\|≤M\|Z\_\{t\}\|\\leq M, the factorη​s​\(xt\)​\(1\+σ1​Zt\)\\eta s\(x\_\{t\}\)\(1\+\\sigma\_\{1\}Z\_\{t\}\)lies in\[η​s​\(xt\)​\(1−σ1​M\),η​sρ​\(1\+σ1​M\)\]⊆\[0,2\]\[\\,\\eta s\(x\_\{t\}\)\(1\-\\sigma\_\{1\}M\),\\ \\eta s\_\{\\rho\}\(1\+\\sigma\_\{1\}M\)\\,\]\\subseteq\[0,2\], usingσ1​M≤1\\sigma\_\{1\}M\\leq 1for the lower end and the step\-size condition for the upper end\. Hence\|1−η​s​\(xt\)​\(1\+σ1​Zt\)\|≤1\|1\-\\eta s\(x\_\{t\}\)\(1\+\\sigma\_\{1\}Z\_\{t\}\)\|\\leq 1and\|xt\+1\|≤\|xt\|≤ρ\|x\_\{t\+1\}\|\\leq\|x\_\{t\}\|\\leq\\rho: the basin is forward\-invariant\. Next,

𝔼​\[xt\+12∣xt\]=xt2​r​\(s​\(xt\),η\),\\mathbb\{E\}\[x\_\{t\+1\}^\{2\}\\mid x\_\{t\}\]=x\_\{t\}^\{2\}\\,r\(s\(x\_\{t\}\),\\eta\),wherer​\(s,η\)=1−2​η​s\+η2​s2​\(1\+σ12\)r\(s,\\eta\)=1\-2\\eta s\+\\eta^\{2\}s^\{2\}\(1\+\\sigma\_\{1\}^\{2\}\)\. Forη≤1/\(sρ​\(1\+σ12\)\)\\eta\\leq 1/\(s\_\{\\rho\}\(1\+\\sigma\_\{1\}^\{2\}\)\)we have∂sr=−2​η\+2​η2​s​\(1\+σ12\)≤0\\partial\_\{s\}r=\-2\\eta\+2\\eta^\{2\}s\(1\+\\sigma\_\{1\}^\{2\}\)\\leq 0on\[λ,sρ\]\[\\lambda,s\_\{\\rho\}\], sor​\(s​\(xt\),η\)≤r​\(λ,η\)r\(s\(x\_\{t\}\),\\eta\)\\leq r\(\\lambda,\\eta\), and iterating gives the claim\. Asρ→0\\rho\\to 0,sρ→λs\_\{\\rho\}\\to\\lambdaand the admissible range extends toη⋆=1/\(λ​\(1\+σ12\)\)\\eta^\{\\star\}=1/\(\\lambda\(1\+\\sigma\_\{1\}^\{2\}\)\), wherer​\(λ,η⋆\)=σ12/\(1\+σ12\)r\(\\lambda,\\eta^\{\\star\}\)=\\sigma\_\{1\}^\{2\}/\(1\+\\sigma\_\{1\}^\{2\}\)\. ∎

The boundedness assumption withσ1​M≤1\\sigma\_\{1\}M\\leq 1keeps the noisy gradient sign\-consistent inside the basin, enabling almost\-sure forward invariance; the contraction inequality itself is algebraic\. In\-basin contraction is*faster*thanr​\(λ,η\)r\(\\lambda,\\eta\)while\|x\|\|x\|is large, sincerrdecreases inss\.

###### Proposition 5\(signSGD: floor persists\)\.

For the quartic family above \(anyc≥0c\\geq 0\), signSGD with constantη\\eta, symmetricZZ, andx0∉η​ℤx\_\{0\}\\notin\\eta\\mathbb\{Z\}satisfies𝔼​\[f​\(xT\)\]\+𝔼​\[f​\(xT\+1\)\]≥18​λ​η2\\mathbb\{E\}\[f\(x\_\{T\}\)\]\+\\mathbb\{E\}\[f\(x\_\{T\+1\}\)\]\\geq\\tfrac\{1\}\{8\}\\lambda\\eta^\{2\}for everyTT, hencelim supT𝔼​\[f​\(xT\)\]≥116​λ​η2\\limsup\_\{T\}\\mathbb\{E\}\[f\(x\_\{T\}\)\]\\geq\\tfrac\{1\}\{16\}\\lambda\\eta^\{2\}\.

###### Proof\.

Sincef′​\(x\)=s​\(x\)​xf^\{\\prime\}\(x\)=s\(x\)xwiths\>0s\>0,sign⁡\(g^\)=sign⁡\(x\)​sign⁡\(1\+σ1​Z\)\\operatorname\{sign\}\(\\hat\{g\}\)=\\operatorname\{sign\}\(x\)\\operatorname\{sign\}\(1\+\\sigma\_\{1\}Z\)and every step is exactly±η\\pm\\eta, so the granularity argument of Proposition[2](https://arxiv.org/html/2607.12360#Thmproposition2)\(i\) applies verbatim; thenf​\(x\)≥12​λ​x2f\(x\)\\geq\\tfrac\{1\}\{2\}\\lambda x^\{2\}pointwise\. ∎

The theorem is genuinely local: outside the basin the\(L0,L1\)\(L\_\{0\},L\_\{1\}\)curvature makes a fixed large step unstable, and SGD with too large anη\\etadiverges \(Sec\.[10](https://arxiv.org/html/2607.12360#S10)\)\. This is precisely the role of warmup; the result is a statement about the post\-warmup phase\.

## 7Normalized SGD in Higher Dimension

Letf​\(x\)=12​x⊤​Λ​xf\(x\)=\\tfrac\{1\}\{2\}x^\{\\top\}\\Lambda xwithΛ=diag​\(λi\)≻0\\Lambda=\\mathrm\{diag\}\(\\lambda\_\{i\}\)\\succ 0and scale\-family multiplicative noise \([1](https://arxiv.org/html/2607.12360#S2.E1)\) withσ0=0\\sigma\_\{0\}=0\. Consider normalized SGD,xt\+1=xt−η​g^t/‖g^t‖x\_\{t\+1\}=x\_\{t\}\-\\eta\\,\\hat\{g\}\_\{t\}/\\\|\\hat\{g\}\_\{t\}\\\|\(withxt\+1=xtx\_\{t\+1\}=x\_\{t\}on the probability\-zero eventg^t=0\\hat\{g\}\_\{t\}=0\)\.

###### Lemma 2\(Scale\-invariant update geometry\)\.

Forα\>0\\alpha\>0, replacingxxbyα​x\\alpha xscales∇f\\nabla fandξ\\xijointly byα\\alpha, so the joint law of the unit vectors\(g^/‖g^‖,∇f/‖∇f‖,x/‖x‖\)\\big\(\\hat\{g\}/\\\|\\hat\{g\}\\\|,\\ \\nabla f/\\\|\\nabla f\\\|,\\ x/\\\|x\\\|\\big\)depends onxxonly through its directionx^=x/‖x‖\\hat\{x\}=x/\\\|x\\\|\. Every angular statistic of the update, in particularc¯​\(x^\):=𝔼​\[cos⁡∠​\(g^,x\)\]\\bar\{c\}\(\\hat\{x\}\):=\\mathbb\{E\}\[\\cos\\angle\(\\hat\{g\},x\)\]ands¯​\(x^\):=𝔼​\[sin2⁡∠​\(g^,x\)\]\\bar\{s\}\(\\hat\{x\}\):=\\mathbb\{E\}\[\\sin^\{2\}\\angle\(\\hat\{g\},x\)\], is independent of the radius along a fixed ray\.

###### Proposition 6\(Normalized\-SGD floor,d\>1d\>1\)\.

Letλmin=mini⁡λi\\lambda\_\{\\min\}=\\min\_\{i\}\\lambda\_\{i\}andRt=‖xt‖R\_\{t\}=\\\|x\_\{t\}\\\|\.

\(i\)*\(Granularity\.\)*The step norm is exactlyη\\eta, somax⁡\(Rt,Rt\+1\)≥η/2\\max\(R\_\{t\},R\_\{t\+1\}\)\\geq\\eta/2for alltt, and𝔼​\[f​\(xT\)\]\+𝔼​\[f​\(xT\+1\)\]≥18​λmin​η2\\mathbb\{E\}\[f\(x\_\{T\}\)\]\+\\mathbb\{E\}\[f\(x\_\{T\+1\}\)\]\\geq\\tfrac\{1\}\{8\}\\lambda\_\{\\min\}\\eta^\{2\}for everyTT; the loss cannot converge to zero at constantη\\eta\.

\(ii\)*\(Equilibrium scale\.\)*Withθt=∠​\(g^t,xt\)\\theta\_\{t\}=\\angle\(\\hat\{g\}\_\{t\},x\_\{t\}\), forRt≫ηR\_\{t\}\\gg\\eta,

𝔼​\[Rt\+1−Rt∣xt\]=−η​c¯​\(x^t\)\+η2​s¯​\(x^t\)2​Rt\+O​\(η3/Rt2\),\\mathbb\{E\}\[R\_\{t\+1\}\-R\_\{t\}\\mid x\_\{t\}\]=\-\\eta\\,\\bar\{c\}\(\\hat\{x\}\_\{t\}\)\+\\frac\{\\eta^\{2\}\\,\\bar\{s\}\(\\hat\{x\}\_\{t\}\)\}\{2R\_\{t\}\}\+O\\\!\\big\(\\eta^\{3\}/R\_\{t\}^\{2\}\\big\),withc¯,s¯\\bar\{c\},\\bar\{s\}radius\-independent by Lemma[2](https://arxiv.org/html/2607.12360#Thmlemma2)\. Ifc−:=infx^c¯​\(x^\)\>0c\_\{\-\}:=\\inf\_\{\\hat\{x\}\}\\bar\{c\}\(\\hat\{x\}\)\>0, the drift is inward wheneverRt≫ηR\_\{t\}\\gg\\eta, so the radius is driven down to, and equilibrates at, scaleΘ​\(η\)\\Theta\(\\eta\); combined with \(i\),𝔼​\[f​\(xT\)\]=Θ​\(η2\)\\mathbb\{E\}\[f\(x\_\{T\}\)\]=\\Theta\(\\eta^\{2\}\)and cooldown is required\. For isotropicΛ=λ​I\\Lambda=\\lambda I,∇f∥x\\nabla f\\parallel xandθt\\theta\_\{t\}is also the angle to the gradient\.

###### Proof\.

\(i\)Rt\+1≥‖η​g^t/‖g^t‖‖−Rt=η−RtR\_\{t\+1\}\\geq\\\|\\eta\\hat\{g\}\_\{t\}/\\\|\\hat\{g\}\_\{t\}\\\|\\\|\-R\_\{t\}=\\eta\-R\_\{t\}, soRt<η/2R\_\{t\}<\\eta/2forcesRt\+1\>η/2R\_\{t\+1\}\>\\eta/2; thenRt2\+Rt\+12≥η2/4R\_\{t\}^\{2\}\+R\_\{t\+1\}^\{2\}\\geq\\eta^\{2\}/4andf​\(x\)≥12​λmin​‖x‖2f\(x\)\\geq\\tfrac\{1\}\{2\}\\lambda\_\{\\min\}\\\|x\\\|^\{2\}\. \(ii\) The law of cosines givesRt\+1=\(Rt2−2​Rt​η​cos⁡θt\+η2\)1/2R\_\{t\+1\}=\\big\(R\_\{t\}^\{2\}\-2R\_\{t\}\\eta\\cos\\theta\_\{t\}\+\\eta^\{2\}\\big\)^\{1/2\}; expanding inη/Rt\\eta/R\_\{t\}and taking conditional expectations yields the display, and Lemma[2](https://arxiv.org/html/2607.12360#Thmlemma2)removes the radius dependence of the coefficients\. ∎

This is thed\>1d\>1form of the mechanism: multiplicative noise makes the*angular*error of the update scale\-free, so a fixed\-norm step has an inward efficiency that does not improve as the iterate approaches the optimum, and the radius settles at a multiple of the step size\.

## 8Momentum

Momentum does not change the classification, which turns on whether the*effective*step magnitude shrinks with‖∇f‖\\\|\\nabla f\\\|\. Heavy\-ball SGD in its averaged form,mt=β​mt−1\+\(1−β\)​g^tm\_\{t\}=\\beta m\_\{t\-1\}\+\(1\-\\beta\)\\hat\{g\}\_\{t\}with step−η​mt\-\\eta m\_\{t\}, hasmt→0m\_\{t\}\\to 0as∇f→0\\nabla f\\to 0under multiplicative noise: it still self\-anneals\. Sign\- and RMS\-normalized momentum methods \(signSGD with momentum, Adam\) retainΘ​\(1\)\\Theta\(1\)update magnitude near the optimum under stationary scale\-family noise, up to theϵ\\epsilon\-damping and the non\-stationary crossover discussed in Section[10](https://arxiv.org/html/2607.12360#S10), and the floor persists\. Cooldown is thus needed exactly when the effective step does not shrink with the gradient\.

## 9Heavy\-Tailed Noise

The two sides of the dissociation make sharply different moment demands, and the comparison favors the normalized side\. The granularity bounds \(Props\.[2](https://arxiv.org/html/2607.12360#Thmproposition2)\(i\),[5](https://arxiv.org/html/2607.12360#Thmproposition5),[6](https://arxiv.org/html/2607.12360#Thmproposition6)\(i\)\) and the sign reliability of Lemma[1](https://arxiv.org/html/2607.12360#Thmlemma1)use no moments ofZZat all, andcos⁡θ∈\[−1,1\]\\cos\\theta\\in\[\-1,1\]is bounded, so the signSGD and normalized\-SGD floors areΘ​\(η2\)\\Theta\(\\eta^\{2\}\)even under infinite\-variance noise; the closed form \([4](https://arxiv.org/html/2607.12360#S4.E4)\) still applies, withppread off from the noise law\. By contrast, the second\-moment recursion of Proposition[1](https://arxiv.org/html/2607.12360#Thmproposition1)requires𝔼​\[Z2\]<∞\\mathbb\{E\}\[Z^\{2\}\]<\\infty, e\.g\. Student\-tνt\_\{\\nu\}withν\>2\\nu\>2\. Forν≤2\\nu\\leq 2theL2L^\{2\}analysis is unavailable, although the iterate can still converge almost surely \(log⁡\|xt\|\\log\|x\_\{t\}\|is a random walk whose increments have finite, typically negative mean; verified atν=1\.5\\nu=1\.5in Sec\.[10](https://arxiv.org/html/2607.12360#S10)\)\. The part of the dissociation that is new here is thus robust precisely in the heavy\-tailed regime where the classical self\-annealing analysis degrades, consistent with the heavy\-tail robustness of sign\-based methods emphasized byMonzio Compagnoniet al\.\([2026](https://arxiv.org/html/2607.12360#bib.bib3)\)\.

## 10Numerical Verification

All quantitative claims in this paper were checked against discrete\-time simulation\. Protocol:λ=1\\lambda=1,σ1=0\.5\\sigma\_\{1\}=0\.5, GaussianZZ, unless stated; stationary quantities are time averages over at least10510^\{5\}post\-burn\-in steps across at least100100independent chains; complete configurations, seeds, and code are in the supplementary material\.

### Self\-annealing \(Prop\.[1](https://arxiv.org/html/2607.12360#Thmproposition1)\)\.

The one\-step second\-moment ratio matchesr​\(η\)r\(\\eta\)to within6×10−56\\times 10^\{\-5\}acrossη∈\[0\.7,0\.9\]\\eta\\in\[0\.7,0\.9\]\(2×1072\\times 10^\{7\}samples per point\), with minimum atη⋆=0\.8\\eta^\{\\star\}=0\.8,r​\(η⋆\)=0\.2r\(\\eta^\{\\star\}\)=0\.2; atη⋆\\eta^\{\\star\}trajectories decay to numerical underflow with no floor\.

### signSGD floor \(Prop\.[2](https://arxiv.org/html/2607.12360#Thmproposition2)\)\.

With continuous initialization, the stationary loss scales asη2\.00±0\.01\\eta^\{2\.00\\pm 0\.01\}overη∈\[0\.02,0\.16\]\\eta\\in\[0\.02,0\.16\]forσ1∈\{0\.5,1,2\}\\sigma\_\{1\}\\in\\\{0\.5,1,2\\\}, with measuredc=𝔼​\[x2\]/η2=0\.38,0\.90,3\.24c=\\mathbb\{E\}\[x^\{2\}\]/\\eta^\{2\}=0\.38,\\,0\.90,\\,3\.24against the closed\-form valuesc¯​\(p\)=0\.382,0\.906,3\.243\\bar\{c\}\(p\)=0\.382,\\,0\.906,\\,3\.243from \([4](https://arxiv.org/html/2607.12360#S4.E4)\)\. Anx0∈η​ℤx\_\{0\}\\in\\eta\\mathbb\{Z\}would place the absorbing point0on the lattice and give a spuriously low floor, hence the continuous initialization\.

### Sign reliability is flat \(Lemma[1](https://arxiv.org/html/2607.12360#Thmlemma1)\)\.

ℙ​\(correct sign\)\\mathbb\{P\}\(\\text\{correct sign\}\)atx=1,0\.1,0\.01x=1,\\,0\.1,\\,0\.01:0\.9770\.977at all three scales for GaussianZZ\(Φ​\(2\)=0\.9772\\Phi\(2\)=0\.9772\);0\.9080\.908for Student\-t2t\_\{2\}\(1−Ft2​\(−2\)=0\.90821\-F\_\{t\_\{2\}\}\(\-2\)=0\.9082\);1\.0001\.000for a standardized lognormal, which is bounded below by−0\.76\>−1/σ1\-0\.76\>\-1/\\sigma\_\{1\}, sop=1p=1exactly\. The value ofppdepends on the noise law; its independence ofxxdoes not\.

### \(L0,L1\)\(L\_\{0\},L\_\{1\}\)local theorem \(Prop\.[4](https://arxiv.org/html/2607.12360#Thmproposition4)\)\.

In\-basin \(x0=0\.3x\_\{0\}=0\.3,η=0\.3\\eta=0\.3,c=1c=1\), the conditional contraction factor rises fromr​\(s​\(x0\),η\)=0\.480r\(s\(x\_\{0\}\),\\eta\)=0\.480at initialization towardr​\(λ,η\)=0\.5125r\(\\lambda,\\eta\)=0\.5125within roughly4040steps, confirming both the transient speedup \(rrdecreasing inss\) and the asymptotic rate; out of basin \(η=0\.8\\eta=0\.8,x0=1\.5x\_\{0\}=1\.5\) the iterate diverges, confirming the locality caveat and the role of warmup\.

### Additive floor \(Prop\.[3](https://arxiv.org/html/2607.12360#Thmproposition3)\)\.

Atσ0=0\.2\\sigma\_\{0\}=0\.2,η=0\.05\\eta=0\.05: simulated stationary second moment1\.0323×10−31\.0323\\times 10^\{\-3\}; predicted fixed point1\.0323×10−31\.0323\\times 10^\{\-3\}\.

### Normalized SGD,d=20d=20\(Prop\.[6](https://arxiv.org/html/2607.12360#Thmproposition6)\)\.

With eigenvalues log\-spaced from11to10−310^\{\-3\}andx∝𝟏x\\propto\\mathbf\{1\}, the alignment𝔼​\[cos⁡∠​\(g^,∇f\)\]\\mathbb\{E\}\[\\cos\\angle\(\\hat\{g\},\\nabla f\)\]equals0\.91310\.9131and the radial alignment𝔼​\[cos⁡∠​\(g^,x\)\]\\mathbb\{E\}\[\\cos\\angle\(\\hat\{g\},x\)\]equals0\.49320\.4932, each identical to four digits at radii11,0\.10\.1, and0\.010\.01\(2×1062\\times 10^\{6\}draws per radius\), as Lemma[2](https://arxiv.org/html/2607.12360#Thmlemma2)requires; the isotropic radial alignment at the samed,σ1d,\\sigma\_\{1\}is0\.89760\.8976\. The stationary loss scales asη2\.00\\eta^\{2\.00\}\.

### Momentum \(Sec\.[8](https://arxiv.org/html/2607.12360#S8)\)\.

Atη=0\.02\\eta=0\.02,β=0\.9\\beta=0\.9under multiplicative noise: heavy\-ball SGD reaches meanlog10⁡x2≈−587\\log\_\{10\}x^\{2\}\\approx\-587after2\.5×1042\.5\\times 10^\{4\}steps and is still descending \(no floor\), while signSGD\-with\-momentum stalls at𝔼​\[f\]=4\.5×10−4=Θ​\(η2\)\\mathbb\{E\}\[f\]=4\.5\\times 10^\{\-4\}=\\Theta\(\\eta^\{2\}\)\.

### AdamW\.

The floor is not an artifact of the sign surrogate\. AdamW \(β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999,ϵ=10−12\\epsilon=10^\{\-12\}\) on the quadratic under multiplicative noise has a last\-iterate floor scaling asη2\.00\\eta^\{2\.00\}acrossη∈\{0\.005,0\.01,0\.02,0\.04\}\\eta\\in\\\{0\.005,0\.01,0\.02,0\.04\\\}, essentially unchanged by decoupled weight decay \(η2\.01\\eta^\{2\.01\}at weight decay0\.010\.01\)\. The damping constant adds a predictable refinement: once gradients fall belowϵ\\epsilon, the updatem^/\(v^\+ϵ\)≈m^/ϵ\\hat\{m\}/\(\\sqrt\{\\hat\{v\}\}\+\\epsilon\)\\approx\\hat\{m\}/\\epsilonbecomes gradient\-proportional and self\-annealing, and the measured exponent steepens to2\.182\.18atϵ=10−4\\epsilon=10^\{\-4\}\. AdamW therefore needs cooldown in the multiplicative regime, withϵ\\epsilonsetting the gradient scale below which it crosses over to SGD\-like behavior\.

### Heavy tails\.

Under CauchyZZ\(no finite mean\), the signSGD floor still scales asη2\.00\\eta^\{2\.00\}, and the measured constant matchesc¯​\(p\)=0\.840\\bar\{c\}\(p\)=0\.840atp=12\+arctan⁡\(1/σ1\)/π≈0\.852p=\\tfrac\{1\}\{2\}\+\\arctan\(1/\\sigma\_\{1\}\)/\\pi\\approx 0\.852: the closed form is moment\-free, as claimed\. For SGD under Student\-t1\.5t\_\{1\.5\}multiplicative noise atη=0\.8\\eta=0\.8, the median of\|x300\|\|x\_\{300\}\|is≈10−135\\approx 10^\{\-135\}\(almost\-sure convergence\), while the sample mean ofx2x^\{2\}is≈10−193\\approx 10^\{\-193\}, far above the median≈10−270\\approx 10^\{\-270\}: rare excursions dominate, as expected when the population second moment is infinite\.

Table 2:signSGD floor constant: closed formc¯​\(p\)\\bar\{c\}\(p\)of Remark[3](https://arxiv.org/html/2607.12360#Thmremark3)versus simulation \(time\-averaged, continuous initialization\)\.
### Exact floor constant \(Remark[3](https://arxiv.org/html/2607.12360#Thmremark3)\)\.

Table[2](https://arxiv.org/html/2607.12360#S10.T2)compares the closed form against simulation across the practical range ofpp; agreement is within0\.7%0\.7\\%, the Monte Carlo error of the time averages\.

![Refer to caption](https://arxiv.org/html/2607.12360v1/x2.png)Figure 2:Real\-data check: two\-layer ReLU network on the digits dataset\. \(a\) Final training loss under WSD schedules, relative to no cooldown \(peak rate tuned per point; bands are min–max over five seeds\)\. Cooldown hurts SGD and Adam here and helps only signSGD\. \(b\) Per\-microbatch gradient variance against the full\-batch gradient norm along a reference SGD trajectory: the noise collapses together with the gradient \(log–log slope0\.560\.56against‖∇f‖2\\\|\\nabla f\\\|^\{2\}; a pure multiplicative model would give slope11\), so the vanishing\-noise regime of Sections[3](https://arxiv.org/html/2607.12360#S3)–[7](https://arxiv.org/html/2607.12360#S7)is the operative one\.
### Dissociation across models \(Table[3](https://arxiv.org/html/2607.12360#S10.T3), Fig\.[1](https://arxiv.org/html/2607.12360#S1.F1)\)\.

We ran full WSD sweeps \(cooldown fractioncf∈\{0,0\.05,…,0\.95\}\\mathrm\{cf\}\\in\\\{0,0\.05,\\dots,0\.95\\\}, peak rate tuned per point\) on the anisotropic quadratic, the\(L0,L1\)\(L\_\{0\},L\_\{1\}\)quartic, and a nonconvex deep linear networky=W2​W1​xy=W\_\{2\}W\_\{1\}xwith a planted orthogonal teacher and multiplicative gradient noise\. Under multiplicative noise, cooldown makes SGD strictly worse on the quadratic \(from6\.8×10−86\.8\\times 10^\{\-8\}atcf=0\\mathrm\{cf\}=0to1\.5×10−61\.5\\times 10^\{\-6\}atcf=0\.95\\mathrm\{cf\}=0\.95\) and is irrelevant on the quartic and the deep linear network, where SGD reaches numerical zero at everycf\\mathrm\{cf\}; signSGD and normalized SGD improve by five to seven orders of magnitude with strong cooldown, with a broad plateau overcf∈\[0\.4,0\.95\]\\mathrm\{cf\}\\in\[0\.4,0\.95\]\. Within such a plateau the argmin is not sharply determined, so Table[3](https://arxiv.org/html/2607.12360#S10.T3)should be read for the contrast betweencf⋆=0\\mathrm\{cf\}^\{\\star\}=0and a plateau near full decay rather than for the exact argmin\. Withσ0\>0\\sigma\_\{0\}\>0every method benefits from cooldown\. These controlled testbeds confirm the mechanism across convex,\(L0,L1\)\(L\_\{0\},L\_\{1\}\), and nonconvex landscapes, dimensions, anisotropy, and noise laws\.

### A real\-data check \(Fig\.[2](https://arxiv.org/html/2607.12360#S10.F2)\)\.

We also trained a two\-layer ReLU network \(6464–128128–1010\) on the scikit\-learn digits dataset \(1,5001\{,\}500training images, cross\-entropy, minibatch3232\), using the same sweep protocol \(five seeds\) together with a direct measurement of the noise structure\. The noise is of the vanishing type: the per\-microbatch gradient variance falls by two and a half orders of magnitude over training and tracks‖∇f‖2\\\|\\nabla f\\\|^\{2\}with log–log slope0\.560\.56\(Fig\.[2](https://arxiv.org/html/2607.12360#S10.F2)b\)\. A constant additive floor is rejected, since the late\-training variance sits a factor of4545below the intercept of a global affine fit; the regime is interpolation\-like but not exactly multiplicative, for which the slope would be11\. Remark[4](https://arxiv.org/html/2607.12360#Thmremark4)covers this intermediate case: the measured exponent leaves SGD a residual floor of orderη2\.3\\eta^\{2\.3\}, invisible at this horizon, while the sign floor is noise\-agnostic\. The schedule preferences follow\. SGD’s optimal cooldown is0and its final loss is2\.4×2\.4\\timesworse atcf=0\.95\\mathrm\{cf\}=0\.95; signSGD is the only method cooldown helps \(7\.7×10−27\.7\\times 10^\{\-2\}to6\.6×10−26\.6\\times 10^\{\-2\}, plateau overcf∈\[0\.8,0\.95\]\\mathrm\{cf\}\\in\[0\.8,0\.95\]\), remaining more than an order of magnitude above SGD throughout \(a factor of2828without cooldown\); Adam again preferscf⋆=0\\mathrm\{cf\}^\{\\star\}=0, ending3\.5×3\.5\\timesworse under strong cooldown\. We read Adam’s SGD\-like behavior as a non\-stationary self\-annealing channel that our stationary analysis excludes: near interpolation the gradient collapses faster than thev^\\sqrt\{\\hat\{v\}\}average forgets, som^/\(v^\+ϵ\)\\hat\{m\}/\(\\sqrt\{\\hat\{v\}\}\+\\epsilon\)shrinks with the gradient rather than stayingΘ​\(1\)\\Theta\(1\), the crossover theϵ\\epsilonexperiment isolates on the quadratic\. This is measured, not inferred: Adam’s per\-step update norm, as a fraction of a full sign stepη​d\\eta\\sqrt\{d\}, falls from0\.330\.33to0\.020\.02over training while the gradient norm drops by500×500\\times\(technical appendix\), so the update is no longerΘ​\(1\)\\Theta\(1\)\-normalized and the criterion of Section[8](https://arxiv.org/html/2607.12360#S8)classifies it correctly\. We report training loss because the claim is optimization\-level, and in single\-epoch pretraining, where cooldown originates, it is also the quantity of record; test accuracy is schedule\-insensitive here \(0\.980\.98–0\.990\.99\)\.

Table 3:WSD sweeps: optimal cooldown fractioncf⋆\\mathrm\{cf\}^\{\\star\}, with the cooldown gain𝔼​f​\(xT\)\|cf=0/mincf⁡𝔼​f​\(xT\)\\mathbb\{E\}f\(x\_\{T\}\)\|\_\{\\mathrm\{cf\}=0\}\\,/\\,\\min\_\{\\mathrm\{cf\}\}\\mathbb\{E\}f\(x\_\{T\}\)in parentheses\. Multiplicative noise unless noted\.†Loss at numerical zero \(≲10−32\\lesssim 10^\{\-32\}\) for everycf\\mathrm\{cf\}; cooldown offers no benefit\.

## 11Scope, a Diagnostic, and What This Does Not Explain

### Not the interior WSD fraction\.

In every regime studied here, SGD’s optimal cooldown is0\(multiplicative\) or maximal \(additive/mixed\), and the normalized methods’ optima sit on broad plateaus near full decay\. Nothing produces the interior cooldown fractions of roughly0\.20\.2–0\.350\.35that work well in real pretraining runs\(Hägeleet al\.[2024](https://arxiv.org/html/2607.12360#bib.bib10); Schaippet al\.[2025](https://arxiv.org/html/2607.12360#bib.bib16); Belloniet al\.[2026](https://arxiv.org/html/2607.12360#bib.bib1)\)\. This is informative: stationary landscape\-plus\-noise models, ours included, appear structurally unable to produce an interior optimum, pointing to non\-stationarity, feature learning, or finite\-data effects as the source\.

### A regime diagnostic\.

The dissociation is actionable only near the multiplicative regime\. The governing ratio isρ​\(x\)=σ02/\(σ12​‖∇f​\(x\)‖2\)\\rho\(x\)=\\sigma\_\{0\}^\{2\}/\(\\sigma\_\{1\}^\{2\}\\\|\\nabla f\(x\)\\\|^\{2\}\): forρ≪1\\rho\\ll 1the noise is effectively multiplicative, and SGD\-type methods need no cooldown while normalized methods do; forρ≳1\\rho\\gtrsim 1an additive floor dominates and every method benefits\. Bothσ02\\sigma\_\{0\}^\{2\}andσ12\\sigma\_\{1\}^\{2\}are estimable online by regressing per\-microbatch gradient variance on‖g^‖2\\\|\\hat\{g\}\\\|^\{2\}, as we do in Section[10](https://arxiv.org/html/2607.12360#S10)\. Because\(σ02,σ12\)\(\\sigma\_\{0\}^\{2\},\\sigma\_\{1\}^\{2\}\)drifts over a long trajectory, the regression should be applied locally in time; the operative question is whether the noise variance vanishes with the gradient\. Since‖∇f‖\\\|\\nabla f\\\|shrinks over training,ρ\\rhogrows late in a run whenever a genuine additive component exists, so cooldown should matter most toward the end, which is where WSD puts it\.

### What is and is not new\.

The self\-annealing half is classical interpolation theory\(Schmidt and Le Roux[2013](https://arxiv.org/html/2607.12360#bib.bib17); Vaswaniet al\.[2019](https://arxiv.org/html/2607.12360#bib.bib19)\); the contribution is the contrast \(Lemma[1](https://arxiv.org/html/2607.12360#Thmlemma1), Props\.[2](https://arxiv.org/html/2607.12360#Thmproposition2)and[6](https://arxiv.org/html/2607.12360#Thmproposition6)\): convex last\-iterate theory explains why cooldown helps \(sub\)gradient methods under additive\-type noise\(Schaippet al\.[2025](https://arxiv.org/html/2607.12360#bib.bib16)\)but does not separate optimizers at fixed noise\. A tight last\-iterate analysis of normalized methods under\(L0,L1\)\(L\_\{0\},L\_\{1\}\)\-smoothness, turning the dissociation into an optimal cooldown*shape*, remains open\.

## 12Related Work

SDE approximations of stochastic optimizers go back toLiet al\.\([2017](https://arxiv.org/html/2607.12360#bib.bib13)\), with adaptive\-method scaling rules inMalladiet al\.\([2022](https://arxiv.org/html/2607.12360#bib.bib15)\)\.Monzio Compagnoniet al\.\([2026](https://arxiv.org/html/2607.12360#bib.bib3)\)analyze distributed, compressed, and sign SGD under\(L0,L1\)\(L\_\{0\},L\_\{1\}\)\-smoothness with the affine variance model we adopt, giving randomly\-sampled\-iterate guarantees complementary to our last\-iterate question\.Shulginet al\.\([2026](https://arxiv.org/html/2607.12360#bib.bib18)\)derive hyperparameter and schedule scaling laws for modern optimizers from linear\-minimization\-oracle bounds, andLiet al\.\([2026](https://arxiv.org/html/2607.12360#bib.bib14)\)obtain optimal schedule shapes under functional scaling laws, flagging adaptivity and momentum as open\. Empirically,Hägeleet al\.\([2024](https://arxiv.org/html/2607.12360#bib.bib10)\)establish constant\-rate\-plus\-cooldown as a reliable alternative to cosine schedules,Wenet al\.\([2025](https://arxiv.org/html/2607.12360#bib.bib20)\)view WSD through a river\-valley landscape,Belloniet al\.\([2026](https://arxiv.org/html/2607.12360#bib.bib1)\)document cross\-architecture universality, andSchaippet al\.\([2025](https://arxiv.org/html/2607.12360#bib.bib16)\)match practical schedules to convex last\-iterate bounds\. Under additive\-type noise, constant\-step SGD has anη\\eta\-scaled stationary distribution\(Dieuleveutet al\.[2020](https://arxiv.org/html/2607.12360#bib.bib6)\), the regime of Proposition[3](https://arxiv.org/html/2607.12360#Thmproposition3); under multiplicative noise there is no floor to converge to\. Sign descent as a surrogate for Adam is supported byBalles and Hennig \([2018](https://arxiv.org/html/2607.12360#bib.bib4)\)andKunstneret al\.\([2023](https://arxiv.org/html/2607.12360#bib.bib12)\), who trace much of Adam’s advantage on transformers to sign\-like behavior, while schedule\-free averaging\(Defazioet al\.[2024](https://arxiv.org/html/2607.12360#bib.bib7)\)removes decay by averaging iterates instead\. Interpolation and strong growth are due toSchmidt and Le Roux \([2013](https://arxiv.org/html/2607.12360#bib.bib17)\)andVaswaniet al\.\([2019](https://arxiv.org/html/2607.12360#bib.bib19)\), signSGD toBernsteinet al\.\([2018](https://arxiv.org/html/2607.12360#bib.bib2)\), with non\-convergence counterexamples and error feedback inKarimireddyet al\.\([2019](https://arxiv.org/html/2607.12360#bib.bib11)\)and relaxed\-smoothness analyses inZhanget al\.\([2020](https://arxiv.org/html/2607.12360#bib.bib21)\)andCrawshawet al\.\([2022](https://arxiv.org/html/2607.12360#bib.bib5)\)\.

## 13Conclusion

Whether learning\-rate cooldown helps is a joint property of the gradient noise and the optimizer\. Under multiplicative noise SGD self\-anneals and prefers a constant rate, while sign\-based and normalized methods sit on anη2\\eta^\{2\}floor that only decay removes; any additive component then restores a floor for every method\. The dissociation persists under\(L0,L1\)\(L\_\{0\},L\_\{1\}\)geometry, in higher dimension, and with momentum and heavy\-tailed noise, and we confirm it on a nonconvex network and on real data\. What it does not settle, the interior cooldown fraction used at scale, appears to lie beyond stationary landscape\-and\-noise geometry\.

## Technical Appendix

Proposition, lemma, remark, equation, table, and figure numbers below refer to the numbered statements in the main text above\.

## A\. Exact Stationary Law for signSGD \(Proposition 2\(ii\)\)

### Setup\.

Consider one coordinate with curvatureλ\>0\\lambda\>0, pure multiplicative scale\-family noise with symmetricZZ, constant stepη\\eta, and initializationx0=\(k\+φ\)​ηx\_\{0\}=\(k\+\\varphi\)\\etawith offsetφ∈\(0,1\)\\varphi\\in\(0,1\)\. The signSGD update isxt\+1=xt−η​sign⁡\(g^t\)x\_\{t\+1\}=x\_\{t\}\-\\eta\\,\\operatorname\{sign\}\(\\hat\{g\}\_\{t\}\)withg^t=λ​xt​\(1\+σ1​Zt\)\\hat\{g\}\_\{t\}=\\lambda x\_\{t\}\(1\+\\sigma\_\{1\}Z\_\{t\}\), so

xt\+1=xt−η​sign⁡\(xt\)​Bt,x\_\{t\+1\}=x\_\{t\}\-\\eta\\,\\operatorname\{sign\}\(x\_\{t\}\)\\,B\_\{t\},whereBt:=sign⁡\(1\+σ1​Zt\)∈\{±1\}B\_\{t\}:=\\operatorname\{sign\}\(1\+\\sigma\_\{1\}Z\_\{t\}\)\\in\\\{\\pm 1\\\}, and by Lemma 1 the variablesBtB\_\{t\}are i\.i\.d\. withℙ​\(Bt=\+1\)=p\\mathbb\{P\}\(B\_\{t\}=\+1\)=p, independent of the iterate\. \(For symmetricZZthe sign reliability is the same on both sides of the optimum, which is the only place symmetry is used\.\) Every iterate stays on the latticex0\+η​ℤx\_\{0\}\+\\eta\\mathbb\{Z\}, which does not contain0; henceg^t≠0\\hat\{g\}\_\{t\}\\neq 0almost surely, every step has magnitude exactlyη\\eta, and the absorbing point0is never reached\.

### The absolute\-value chain\.

Write the positive lattice points as

Aj:=\(j\+φ\)​η,Bj:=\(j\+1−φ\)​η,j≥0,A\_\{j\}:=\(j\+\\varphi\)\\eta,\\qquad B\_\{j\}:=\(j\+1\-\\varphi\)\\eta,\\qquad j\\geq 0,where theAA\-ladder collects the values of\|x\|\|x\|realized whilex\>0x\>0and theBB\-ladder those realized whilex<0x<0\(ifx0\>0x\_\{0\}\>0; the labeling is symmetric otherwise\)\. A correct step \(Bt=\+1B\_\{t\}=\+1\) moves\|x\|\|x\|inward, an incorrect one outward:

Aj→Aj−1​\(j≥1\)andA0→B0w\.p\.​p,\\displaystyle A\_\{j\}\\to A\_\{j\-1\}\\ \(j\\geq 1\)\\quad\\text\{and\}\\quad A\_\{0\}\\to B\_\{0\}\\quad\\text\{w\.p\.\\ \}p,Aj→Aj\+1w\.p\.​q:=1−p,\\displaystyle A\_\{j\}\\to A\_\{j\+1\}\\quad\\text\{w\.p\.\\ \}q:=1\-p,and identically on theBB\-ladder\. The only communication between the ladders is the bottom edgeA0↔B0A\_\{0\}\\leftrightarrow B\_\{0\}: stepping inward fromA0=φ​ηA\_\{0\}=\\varphi\\etacrosses the origin and lands at−\(1−φ\)​η\-\(1\-\\varphi\)\\eta, i\.e\. atB0B\_\{0\}\.

### Invariant law by detailed balance\.

The transition graph is a tree \(two half\-lines joined by one edge\), and a Markov chain on a tree is reversible, so its invariant lawπ\\pisatisfies detailed balance edge by edge:

π​\(Aj\)​q=π​\(Aj\+1\)​p,π​\(Bj\)​q=π​\(Bj\+1\)​p,\\displaystyle\\pi\(A\_\{j\}\)\\,q=\\pi\(A\_\{j\+1\}\)\\,p,\\qquad\\pi\(B\_\{j\}\)\\,q=\\pi\(B\_\{j\+1\}\)\\,p,π​\(A0\)​p=π​\(B0\)​p\.\\displaystyle\\pi\(A\_\{0\}\)\\,p=\\pi\(B\_\{0\}\)\\,p\.Henceπ​\(Aj\)=π​\(Bj\)=κ​βj\\pi\(A\_\{j\}\)=\\pi\(B\_\{j\}\)=\\kappa\\beta^\{j\}withβ:=q/p<1\\beta:=q/p<1forp\>12p\>\\tfrac\{1\}\{2\}, and normalization2​κ​∑j≥0βj=12\\kappa\\sum\_\{j\\geq 0\}\\beta^\{j\}=1givesκ=\(1−β\)/2\\kappa=\(1\-\\beta\)/2\.

### Second moment\.

Underπ\\pi,

𝔼π​\[x2\]η2=1−β2​∑j≥0βj​\[\(j\+φ\)2\+\(j\+1−φ\)2\]\.\\frac\{\\mathbb\{E\}\_\{\\pi\}\[x^\{2\}\]\}\{\\eta^\{2\}\}=\\frac\{1\-\\beta\}\{2\}\\sum\_\{j\\geq 0\}\\beta^\{j\}\\Big\[\(j\+\\varphi\)^\{2\}\+\(j\+1\-\\varphi\)^\{2\}\\Big\]\.Expanding\(j\+φ\)2\+\(j\+1−φ\)2=2​\(j2\+j\)\+φ2\+\(1−φ\)2\(j\+\\varphi\)^\{2\}\+\(j\+1\-\\varphi\)^\{2\}=2\(j^\{2\}\+j\)\+\\varphi^\{2\}\+\(1\-\\varphi\)^\{2\}and using the geometric sums∑jβj=11−β\\sum\_\{j\}\\beta^\{j\}=\\tfrac\{1\}\{1\-\\beta\},∑jj​βj=β\(1−β\)2\\sum\_\{j\}j\\beta^\{j\}=\\tfrac\{\\beta\}\{\(1\-\\beta\)^\{2\}\},∑jj2​βj=β​\(1\+β\)\(1−β\)3\\sum\_\{j\}j^\{2\}\\beta^\{j\}=\\tfrac\{\\beta\(1\+\\beta\)\}\{\(1\-\\beta\)^\{3\}\},

𝔼π​\[x2\]η2=φ2\+\(1−φ\)22\+\(1−β\)​\[β​\(1\+β\)\(1−β\)3\+β\(1−β\)2\]\.\\frac\{\\mathbb\{E\}\_\{\\pi\}\[x^\{2\}\]\}\{\\eta^\{2\}\}=\\frac\{\\varphi^\{2\}\+\(1\-\\varphi\)^\{2\}\}\{2\}\+\(1\-\\beta\)\\Big\[\\frac\{\\beta\(1\+\\beta\)\}\{\(1\-\\beta\)^\{3\}\}\+\\frac\{\\beta\}\{\(1\-\\beta\)^\{2\}\}\\Big\]\.Substitutingβ=q/p\\beta=q/pand1−β=\(2​p−1\)/p1\-\\beta=\(2p\-1\)/psimplifies the bracket toq/\(2​p−1\)\+q/\(2​p−1\)2q/\(2p\-1\)\+q/\(2p\-1\)^\{2\}, which is Equation \(4\):

c​\(φ,p\)=φ2\+\(1−φ\)22\+q2​p−1\+q\(2​p−1\)2\.c\(\\varphi,p\)=\\frac\{\\varphi^\{2\}\+\(1\-\\varphi\)^\{2\}\}\{2\}\+\\frac\{q\}\{2p\-1\}\+\\frac\{q\}\{\(2p\-1\)^\{2\}\}\.Sinceφ2\+\(1−φ\)2≥12\\varphi^\{2\}\+\(1\-\\varphi\)^\{2\}\\geq\\tfrac\{1\}\{2\}, we getc​\(φ,p\)≥14c\(\\varphi,p\)\\geq\\tfrac\{1\}\{4\}for allφ,p\\varphi,p, and averaging overφ∼Unif​\[0,1\)\\varphi\\sim\\mathrm\{Unif\}\[0,1\)\(the offset induced by a continuous initialization\) replaces the first term by13\\tfrac\{1\}\{3\}, givingc¯​\(p\)\\bar\{c\}\(p\)of Remark 3\. Atp=1p=1the walk deterministically oscillates between the two bottom states andc=\(φ2\+\(1−φ\)2\)/2c=\\big\(\\varphi^\{2\}\+\(1\-\\varphi\)^\{2\}\\big\)/2exactly, consistent with the formula atq=0q=0\.

### Periodicity and why Proposition 2\(i\) is stated for pairs\.

The two\-ladder graph is bipartite, so the chain has period22: the law ofxTx\_\{T\}oscillates between an even and an odd profile, and𝔼​\[xT2\]\\mathbb\{E\}\[x\_\{T\}^\{2\}\]need not converge, although time averages converge to𝔼π​\[x2\]\\mathbb\{E\}\_\{\\pi\}\[x^\{2\}\]and each parity profile has the same geometric tails\. This is why the granularity bound in Proposition 2\(i\) is stated for consecutive pairs,𝔼​\[f​\(xT\)\]\+𝔼​\[f​\(xT\+1\)\]≥18​λ​η2\\mathbb\{E\}\[f\(x\_\{T\}\)\]\+\\mathbb\{E\}\[f\(x\_\{T\+1\}\)\]\\geq\\tfrac\{1\}\{8\}\\lambda\\eta^\{2\}, rather than as alim inf\\liminfof the single\-iterate loss: forppclose to11and offsetφ\\varphiclose to0or11, the single\-iterate expectation genuinely dips to12λmin\(φ,1−φ\)2η2\\tfrac\{1\}\{2\}\\lambda\\min\(\\varphi,1\-\\varphi\)^\{2\}\\eta^\{2\}on one parity class\. All floor statements in this paper use the pair form for this reason\.

## B\. Drift–Diffusion Comparison \(Remark 3\)

A drift–diffusion \(small\-step\) approximation replaces the walk by the SDEd​x=−μ​sign⁡\(x\)​d​t\+σ​d​Wdx=\-\\mu\\,\\operatorname\{sign\}\(x\)\\,dt\+\\sigma\\,dWwith per\-step driftμ=\(2​p−1\)​η\\mu=\(2p\-1\)\\etaand per\-step varianceσ2=𝔼​\[\(η​B\)2\]−μ2=4​p​q​η2\\sigma^\{2\}=\\mathbb\{E\}\[\(\\eta B\)^\{2\}\]\-\\mu^\{2\}=4pq\\,\\eta^\{2\}\. Its stationary density is Laplace,∝e−2​μ​\|x\|/σ2\\propto e^\{\-2\\mu\|x\|/\\sigma^\{2\}\}, with second momentσ4/\(2​μ2\)\\sigma^\{4\}/\(2\\mu^\{2\}\), i\.e\.

cdd​\(p\)=8​p2​q2\(2​p−1\)2\.c\_\{\\mathrm\{dd\}\}\(p\)=\\frac\{8p^\{2\}q^\{2\}\}\{\(2p\-1\)^\{2\}\}\.Because the step sizeη\\etacoincides with the stationary widthΘ​\(η\)\\Theta\(\\eta\), there is no scale separation and the approximation degrades asp→1p\\to 1:cdd→0c\_\{\\mathrm\{dd\}\}\\to 0while the exactc¯​\(p\)→13\\bar\{c\}\(p\)\\to\\tfrac\{1\}\{3\}\. Numerically, the ratioc¯/cdd\\bar\{c\}/c\_\{\\mathrm\{dd\}\}is1\.071\.07,1\.341\.34,2\.152\.15,6\.076\.07,20\.220\.2atp=0\.6,0\.7,0\.8,0\.9,0\.95p=0\.6,0\.7,0\.8,0\.9,0\.95, and88\.088\.0atp=Φ​\(2\)≈0\.9772p=\\Phi\(2\)\\approx 0\.9772\(the Gaussian caseσ1=0\.5\\sigma\_\{1\}=0\.5used throughout this paper\)\.

## C\. Expansion Details for Proposition 6\(ii\)

Withθt=∠​\(g^t,xt\)\\theta\_\{t\}=\\angle\(\\hat\{g\}\_\{t\},x\_\{t\}\)and step−η​g^t/‖g^t‖\-\\eta\\hat\{g\}\_\{t\}/\\\|\\hat\{g\}\_\{t\}\\\|, the law of cosines gives

Rt\+1=Rt​1−2​u​cos⁡θt\+u2,u:=η/Rt\.R\_\{t\+1\}=R\_\{t\}\\sqrt\{1\-2u\\cos\\theta\_\{t\}\+u^\{2\}\},\\qquad u:=\\eta/R\_\{t\}\.Foru<1u<1,1−2​u​cos⁡θ\+u2=1−u​cos⁡θ\+12​u2​sin2⁡θ\+O​\(u3\)\\sqrt\{1\-2u\\cos\\theta\+u^\{2\}\}=1\-u\\cos\\theta\+\\tfrac\{1\}\{2\}u^\{2\}\\sin^\{2\}\\theta\+O\(u^\{3\}\)uniformly inθ\\theta, so

Rt\+1−Rt=−η​cos⁡θt\+η2​sin2⁡θt2​Rt\+O​\(η3Rt2\)\.R\_\{t\+1\}\-R\_\{t\}=\-\\eta\\cos\\theta\_\{t\}\+\\frac\{\\eta^\{2\}\\sin^\{2\}\\theta\_\{t\}\}\{2R\_\{t\}\}\+O\\\!\\Big\(\\frac\{\\eta^\{3\}\}\{R\_\{t\}^\{2\}\}\\Big\)\.Taking conditional expectations and applying Lemma 2 \(the joint law of the relevant unit vectors depends onxtx\_\{t\}only throughx^t\\hat\{x\}\_\{t\}\) yields the drift display with radius\-independent coefficientsc¯​\(x^t\)\\bar\{c\}\(\\hat\{x\}\_\{t\}\)ands¯​\(x^t\)\\bar\{s\}\(\\hat\{x\}\_\{t\}\)\. Whenc−=infx^c¯​\(x^\)\>0c\_\{\-\}=\\inf\_\{\\hat\{x\}\}\\bar\{c\}\(\\hat\{x\}\)\>0, the drift is bounded above by−η​c−\+η2/\(2​Rt\)\-\\eta c\_\{\-\}\+\\eta^\{2\}/\(2R\_\{t\}\), which is negative forRt\>η/\(2​c−\)R\_\{t\}\>\\eta/\(2c\_\{\-\}\); the radius is therefore driven into a band of widthΘ​\(η\)\\Theta\(\\eta\)and, by the granularity bound of Proposition 6\(i\), cannot collapse below scaleη\\etaeither\. The main text presents part \(ii\) as a leading\-order equilibrium statement; the fully rigorous floor is part \(i\), which needs no expansion\.

## D\. Complete Simulation Configurations

Common defaults: curvatureλ=1\\lambda=1, noise scaleσ1=0\.5\\sigma\_\{1\}=0\.5, standard GaussianZZ, NumPydefault\_rngwith fixed seeds recorded in the code\. Stationary quantities are time averages after burn\-in\. All scripts run on CPU in minutes\.

### One\-step SGD ratio \(Prop\. 1\)\.

𝔼​\[\(1−η​\(1\+σ1​Z\)\)2\]\\mathbb\{E\}\[\(1\-\\eta\(1\+\\sigma\_\{1\}Z\)\)^\{2\}\]estimated from2×1072\\times 10^\{7\}draws atη∈\{0\.70,0\.75,0\.80,0\.85,0\.90\}\\eta\\in\\\{0\.70,0\.75,0\.80,0\.85,0\.90\\\}; maximum absolute deviation fromr​\(η\)r\(\\eta\)was6×10−56\\times 10^\{\-5\}\. Long\-horizon decay checked atη⋆=0\.8\\eta^\{\\star\}=0\.8until floating\-point underflow\.

### signSGD floors \(Prop\. 2, Table 2\)\.

Direct Bernoulli\-walk simulation for Table 2:η=0\.1\\eta=0\.1,x0∼Unif​\(0\.5,1\.5\)x\_\{0\}\\sim\\mathrm\{Unif\}\(0\.5,1\.5\),400400chains,4×1054\\times 10^\{5\}steps, burn\-in5×1045\\times 10^\{4\}\. Gaussian\-noise floors:η∈\{0\.02,0\.04,0\.08,0\.16\}\\eta\\in\\\{0\.02,0\.04,0\.08,0\.16\\\},σ1∈\{0\.5,1,2\}\\sigma\_\{1\}\\in\\\{0\.5,1,2\\\},300300chains,3×1053\\times 10^\{5\}steps, burn\-in6×1046\\times 10^\{4\}; exponents from a log\-log fit\. The absorption artifact was reproduced by settingx0=0\.3x\_\{0\}=0\.3withη=0\.003\\eta=0\.003\(sox0∈η​ℤx\_\{0\}\\in\\eta\\mathbb\{Z\}\): the iterate can then reach the lattice point0, where the multiplicative gradient and hence the update vanish\.

### Sign reliability \(Lemma 1\)\.

2×1062\\times 10^\{6\}draws per law and scale; laws: standard Gaussian, Student\-t2t\_\{2\}\(raw scale\), lognormal standardized to zero mean and unit variance, evaluated atx∈\{1,0\.1,0\.01\}x\\in\\\{1,0\.1,0\.01\\\}\.

### \(L0,L1\)\(L\_\{0\},L\_\{1\}\)quartic \(Props\. 4, 5\)\.

f​\(x\)=x2/2\+x4/4f\(x\)=x^\{2\}/2\+x^\{4\}/4\. In\-basin:x0=0\.3x\_\{0\}=0\.3,η=0\.3\\eta=0\.3,10510^\{5\}chains; the reported conditional contraction factor is the sample mean ofr​\(s​\(xt\),η\)r\(s\(x\_\{t\}\),\\eta\)\. Out\-of\-basin:η=0\.8\\eta=0\.8,x0=1\.5x\_\{0\}=1\.5\. WSD sweep:T=2000T=2000,x0∼Unif​\(0\.25,0\.35\)x\_\{0\}\\sim\\mathrm\{Unif\}\(0\.25,0\.35\),40004000chains×3\\times\\,3seeds, peak\-rate grids\{0\.1,0\.2,0\.3,0\.5,0\.7\}\\\{0\.1,0\.2,0\.3,0\.5,0\.7\\\}\(SGD\) and\{0\.003,0\.01,0\.03,0\.1\}\\\{0\.003,0\.01,0\.03,0\.1\\\}\(signSGD\)\.

### Additive floor \(Prop\. 3\)\.

σ0=0\.2\\sigma\_\{0\}=0\.2,η=0\.05\\eta=0\.05,20002000chains,2×1052\\times 10^\{5\}steps, burn\-in5×1045\\times 10^\{4\}\.

### Normalized SGD ind=20d=20\(Prop\. 6\)\.

Spectrumλi\\lambda\_\{i\}log\-spaced from11to10−310^\{\-3\}; directionx∝𝟏x\\propto\\mathbf\{1\}; alignments from2×1062\\times 10^\{6\}draws at radii1,0\.1,0\.011,0\.1,0\.01; the isotropic comparison usesλi≡1\\lambda\_\{i\}\\equiv 1\. Floor exponent fromη∈\{0\.02,0\.04,0\.08\}\\eta\\in\\\{0\.02,0\.04,0\.08\\\},100100chains,1\.5×1051\.5\\times 10^\{5\}steps, burn\-in3×1043\\times 10^\{4\}\.

### Momentum \(Sec\. 8\)\.

β=0\.9\\beta=0\.9,η=0\.02\\eta=0\.02\. Heavy\-ball in the averaged formmt=β​mt−1\+\(1−β\)​g^tm\_\{t\}=\\beta m\_\{t\-1\}\+\(1\-\\beta\)\\hat\{g\}\_\{t\},2×1042\\times 10^\{4\}chains,2\.5×1042\.5\\times 10^\{4\}steps, tracked in log space with periodic rescaling of the joint state\(xt,mt\)\(x\_\{t\},m\_\{t\}\), which is exact because the dynamics are linear in the state\. Sign\-with\-momentum: step−η​sign⁡\(mt\)\-\\eta\\,\\operatorname\{sign\}\(m\_\{t\}\),200200chains,2×1052\\times 10^\{5\}steps, burn\-in5×1045\\times 10^\{4\}\.

### AdamW \(Sec\. 10\)\.

β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999, bias correction on,η∈\{0\.005,0\.01,0\.02,0\.04\}\\eta\\in\\\{0\.005,0\.01,0\.02,0\.04\\\},200200chains,2×1052\\times 10^\{5\}steps, burn\-in5×1045\\times 10^\{4\}; configurations\(ϵ,wd\)∈\{\(10−12,0\),\(10−12,0\.01\),\(10−4,0\)\}\(\\epsilon,\\text\{wd\}\)\\in\\\{\(10^\{\-12\},0\),\(10^\{\-12\},0\.01\),\(10^\{\-4\},0\)\\\}with decoupled weight decay\.

### Heavy tails \(Sec\. 9\)\.

signSGD under standard CauchyZZ: same protocol as the Gaussian floors\. SGD under Student\-t1\.5t\_\{1\.5\}:η=0\.8\\eta=0\.8,2×1052\\times 10^\{5\}runs of300300steps, evolved aslog⁡\|xt\|\\log\|x\_\{t\}\|; the reported mean ofx2x^\{2\}is the log\-sum\-exp sample mean\.

### WSD sweeps \(Table 3, Fig\. 1\)\.

Schedule:ηt=η\\eta\_\{t\}=\\etafort<\(1−cf\)​Tt<\(1\-\\mathrm\{cf\}\)T, then linear decay to0;cf∈\{0,0\.05,…,0\.95\}\\mathrm\{cf\}\\in\\\{0,0\.05,\\dots,0\.95\\\}; the reported loss at eachcf\\mathrm\{cf\}is the minimum over the peak\-rate grid of the mean last\-iterate loss\. For SGD under multiplicative noise this min\-over\-grid loss increases monotonically incf\\mathrm\{cf\}across the entire sweep \(Fig\. 1a\), socf⋆=0\\mathrm\{cf\}^\{\\star\}=0is not an artifact of grid resolution\. Quadratic:d=20d=20, spectrum as above,x0=𝟏/dx\_\{0\}=\\mathbf\{1\}/\\sqrt\{d\},T=2000T=2000,512512chains per configuration; grids\{0\.1,0\.2,0\.4,0\.8,1\.2,1\.5\}\\\{0\.1,0\.2,0\.4,0\.8,1\.2,1\.5\\\}\(SGD\) and\{0\.003,0\.01,0\.03,0\.1,0\.3\}\\\{0\.003,0\.01,0\.03,0\.1,0\.3\\\}\(signSGD, normalized SGD\); mixed regime addsσ0=0\.2\\sigma\_\{0\}=0\.2\. Deep linear network:y=W2​W1​xy=W\_\{2\}W\_\{1\}xwithW1,W2∈ℝ10×10W\_\{1\},W\_\{2\}\\in\\mathbb\{R\}^\{10\\times 10\}, planted orthogonal teacherW⋆W^\{\\star\}\(QR factor of a Gaussian matrix\), population loss12​‖W2​W1−W⋆‖F2\\tfrac\{1\}\{2\}\\\|W\_\{2\}W\_\{1\}\-W^\{\\star\}\\\|\_\{F\}^\{2\}, exact population gradients with coordinatewise multiplicative noise, initializationI\+0\.1​𝒩​\(0,1\)10×10I\+0\.1\\,\\mathcal\{N\}\(0,1\)^\{10\\times 10\},T=4000T=4000,88runs per configuration; grids\{0\.02,0\.05,0\.1,0\.2,0\.4\}\\\{0\.02,0\.05,0\.1,0\.2,0\.4\\\}\(SGD\) and\{0\.001,0\.003,0\.01,0\.03\}\\\{0\.001,0\.003,0\.01,0\.03\\\}\(signSGD\)\.

## E\. Real\-Data Experiment \(Fig\. 2\)

### Task and model\.

Scikit\-learndigits\(8×88\\times 8grayscale images, features scaled to\[0,1\]\[0,1\]\), shuffled with a fixed seed and split1,5001\{,\}500train /297297test\. Two\-layer ReLU network64→128→1064\\to 128\\to 10\(He initialization, biases zero;9,6109\{,\}610parameters\), softmax cross\-entropy, minibatches of size3232drawn i\.i\.d\. with replacement\.

### WSD sweep\.

T=4,000T=4\{,\}000steps; cooldown fractionscf∈\{0,0\.1,…,0\.9,0\.95\}\\mathrm\{cf\}\\in\\\{0,0\.1,\\dots,0\.9,0\.95\\\}; peak\-rate grids\{0\.05,0\.1,0\.2,0\.4\}\\\{0\.05,0\.1,0\.2,0\.4\\\}\(SGD\),\{10−4,3×10−4,10−3,3×10−3,10−2\}\\\{10^\{\-4\},3\\times 10^\{\-4\},10^\{\-3\},3\\times 10^\{\-3\},10^\{\-2\}\\\}\(signSGD\),\{3×10−4,10−3,3×10−3,10−2\}\\\{3\\times 10^\{\-4\},10^\{\-3\},3\\times 10^\{\-3\},10^\{\-2\}\\\}\(Adam,β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999,ϵ=10−8\\epsilon=10^\{\-8\}, bias correction\); five seeds controlling both initialization and minibatch order\. Reported: mean over seeds of the final full\-batch training loss, minimized over the peak\-rate grid per\(optimizer,cf\)\(\\text\{optimizer\},\\mathrm\{cf\}\); bands in Fig\. 2a are min–max over seeds at the selected peak rate\. Final test accuracies at the per\-optimizer best configuration:0\.9910\.991\(SGD\),0\.9860\.986\(signSGD\),0\.9940\.994\(Adam\); accuracy varies by less than0\.010\.01acrosscf\\mathrm\{cf\}for every optimizer\.

### Noise measurement\.

Along a reference run \(SGD, constantη=0\.1\\eta=0\.1\), at checkpointst∈\{0,50,100,200,400,800,1600,3200,6400,12800\}t\\in\\\{0,50,100,200,400,800,1600,3200,6400,12800\\\}we compute the full\-batch gradientGGand the empirical varianceV=164​∑b=164‖gb−G‖2V=\\frac\{1\}\{64\}\\sum\_\{b=1\}^\{64\}\\\|g\_\{b\}\-G\\\|^\{2\}over6464fresh microbatches of size3232\(gradients flattened over all parameters\)\. Measured values \(tt, training loss,‖G‖2\\\|G\\\|^\{2\},VV\):\(0,2\.46,9\.0×10−1,1\.31\)\(0,\\,2\.46,\\,9\.0\\times 10^\{\-1\},\\,1\.31\);\(50,0\.94,2\.9×10−1,0\.70\)\(50,\\,0\.94,\\,2\.9\\times 10^\{\-1\},\\,0\.70\);\(100,0\.50,1\.1×10−1,0\.50\)\(100,\\,0\.50,\\,1\.1\\times 10^\{\-1\},\\,0\.50\);\(200,0\.28,8\.2×10−2,0\.40\)\(200,\\,0\.28,\\,8\.2\\times 10^\{\-2\},\\,0\.40\);\(400,0\.17,5\.0×10−2,0\.32\)\(400,\\,0\.17,\\,5\.0\\times 10^\{\-2\},\\,0\.32\);\(800,0\.10,1\.1×10−2,0\.18\)\(800,\\,0\.10,\\,1\.1\\times 10^\{\-2\},\\,0\.18\);\(1600,0\.056,3\.6×10−3,0\.12\)\(1600,\\,0\.056,\\,3\.6\\times 10^\{\-3\},\\,0\.12\);\(3200,0\.028,4\.3×10−3,6\.8×10−2\)\(3200,\\,0\.028,\\,4\.3\\times 10^\{\-3\},\\,6\.8\\times 10^\{\-2\}\);\(6400,0\.011,2\.7×10−4,2\.1×10−2\)\(6400,\\,0\.011,\\,2\.7\\times 10^\{\-4\},\\,2\.1\\times 10^\{\-2\}\);\(12800,4\.3×10−3,3\.9×10−5,3\.7×10−3\)\(12800,\\,4\.3\\times 10^\{\-3\},\\,3\.9\\times 10^\{\-5\},\\,3\.7\\times 10^\{\-3\}\)\. The log–log regression ofVVon‖G‖2\\\|G\\\|^\{2\}has slope0\.560\.56\. A global affine fitV=a\+b​‖G‖2V=a\+b\\\|G\\\|^\{2\}givesa=0\.166a=0\.166,b=1\.36b=1\.36; the final measuredVVis a factor of4545below that intercept, which is how a constant additive floor is rejected: the affine model with trajectory\-constant\(σ02,σ12\)\(\\sigma\_\{0\}^\{2\},\\sigma\_\{1\}^\{2\}\)is only locally valid, and the operative fact is that the noise variance collapses together with the gradient\.

### Adam update\-magnitude measurement\.

To test the explanation offered in the main text for Adam’s SGD\-like schedule preference, we track the per\-step update norm‖θt\+1−θt‖\\\|\\theta\_\{t\+1\}\-\\theta\_\{t\}\\\|as a fraction of a full sign stepη​d\\eta\\sqrt\{d\}\(d=9,610d=9\{,\}610parameters\), averaged over a1010\-step window, along an Adam run at constantη=3×10−3\\eta=3\\times 10^\{\-3\}\(the peak rate selected by the sweep atcf=0\\mathrm\{cf\}=0\), together with the full\-batch gradient norm\. Measured values \(tt, fraction,‖∇f‖\\\|\\nabla f\\\|\):\(50,0\.33,0\.58\)\(50,\\,0\.33,\\,0\.58\);\(100,0\.22,0\.28\)\(100,\\,0\.22,\\,0\.28\);\(200,0\.14,0\.15\)\(200,\\,0\.14,\\,0\.15\);\(400,0\.12,0\.18\)\(400,\\,0\.12,\\,0\.18\);\(800,0\.12,0\.087\)\(800,\\,0\.12,\\,0\.087\);\(1600,0\.074,0\.038\)\(1600,\\,0\.074,\\,0\.038\);\(3200,0\.032,0\.0050\)\(3200,\\,0\.032,\\,0\.0050\);\(4000,0\.022,0\.0012\)\(4000,\\,0\.022,\\,0\.0012\)\. Atη=10−3\\eta=10^\{\-3\}the fraction falls from0\.430\.43to0\.070\.07over the same horizon\. AΘ​\(1\)\\Theta\(1\)\-normalized method would hold this fraction roughly constant; instead it declines by an order of magnitude as the gradient collapses, confirming that on this task thev^\\sqrt\{\\hat\{v\}\}memory converts Adam into an effectively gradient\-proportional, self\-annealing update\.

## F\. Code

Thecode/directory is a self\-contained package \(NumPy/SciPy/Matplotlib/scikit\-learn\) with a one\-command entry point,run\_all\.sh, that regenerates every result and figure intoresults/\. Shared utilities live incommon\.py\(deterministic seeding and the WSD schedule\) andmlp\.py\(the two\-layer network\); all randomness is seeded deterministically, so every number reproduces on rerun\. The experiment scripts areverify\_claims\.py\(all stationary checks, exponents, and constants\),wsd\_sweeps\.py\(quadratic and deep\-linear WSD sweeps\),quartic\_wsd\.py\(quartic sweep and the one\-step ratio check\),mlp\_realdata\.py\(real\-data experiment of Fig\. 2\), andadam\_step\_norm\.py\(the Adam update\-magnitude measurement above\);make\_figure1\.pyandmake\_figure2\.pyrender the figures from the saved outputs\. Total runtime is under forty minutes on a laptop CPU\.

## References

- L\. Balles and P\. Hennig \(2018\)Dissecting Adam: the sign, magnitude and variance of stochastic gradients\.InProceedings of the 35th International Conference on Machine Learning \(ICML\),Cited by:[§12](https://arxiv.org/html/2607.12360#S12.p1.2)\.
- A\. Belloni, L\. Noci, and A\. Orvieto \(2026\)Universal dynamics of warmup stable decay: understanding WSD beyond transformers\.Note:ICML 2025 Workshops on High\-dimensional Learning Dynamics \(HiLD\) and Methods and Opportunities at Small Scale \(MOSS\); arXiv:2601\.09000Cited by:[§1](https://arxiv.org/html/2607.12360#S1.p1.1),[§11](https://arxiv.org/html/2607.12360#S11.SS0.SSS0.Px1.p1.3),[§12](https://arxiv.org/html/2607.12360#S12.p1.2)\.
- J\. Bernstein, Y\.\-X\. Wang, K\. Azizzadenesheli, and A\. Anandkumar \(2018\)signSGD: compressed optimisation for non\-convex problems\.InProceedings of the 35th International Conference on Machine Learning \(ICML\),Cited by:[§12](https://arxiv.org/html/2607.12360#S12.p1.2),[Remark 2](https://arxiv.org/html/2607.12360#Thmremark2.p1.1)\.
- M\. Crawshaw, M\. Liu, F\. Orabona, W\. Zhang, and Z\. Zhuang \(2022\)Robustness to unbounded smoothness of generalized SignSGD\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§12](https://arxiv.org/html/2607.12360#S12.p1.2)\.
- A\. Defazio, X\. A\. Yang, H\. Mehta, K\. Mishchenko, A\. Khaled, and A\. Cutkosky \(2024\)The road less scheduled\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§12](https://arxiv.org/html/2607.12360#S12.p1.2)\.
- A\. Dieuleveut, A\. Durmus, and F\. Bach \(2020\)Bridging the gap between constant step size stochastic gradient descent and Markov chains\.Annals of Statistics48\(3\)\.Cited by:[§12](https://arxiv.org/html/2607.12360#S12.p1.2)\.
- A\. Hägele, E\. Bakouch, A\. Kosson, L\. Ben Allal, L\. von Werra, and M\. Jaggi \(2024\)Scaling laws and compute\-optimal training beyond fixed training durations\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2607.12360#S1.p1.1),[§11](https://arxiv.org/html/2607.12360#S11.SS0.SSS0.Px1.p1.3),[§12](https://arxiv.org/html/2607.12360#S12.p1.2)\.
- S\. Hu, Y\. Tu, X\. Han, C\. He, G\. Cui,et al\.\(2024\)MiniCPM: unveiling the potential of small language models with scalable training strategies\.arXiv preprint arXiv:2404\.06395\.Cited by:[§1](https://arxiv.org/html/2607.12360#S1.p1.1)\.
- S\. P\. Karimireddy, Q\. Rebjock, S\. U\. Stich, and M\. Jaggi \(2019\)Error feedback fixes SignSGD and other gradient compression schemes\.InProceedings of the 36th International Conference on Machine Learning \(ICML\),Cited by:[§12](https://arxiv.org/html/2607.12360#S12.p1.2),[Remark 2](https://arxiv.org/html/2607.12360#Thmremark2.p1.1)\.
- F\. Kunstner, J\. Chen, J\. W\. Lavington, and M\. Schmidt \(2023\)Noise is not the main factor behind the gap between SGD and Adam on transformers, but sign descent might be\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§12](https://arxiv.org/html/2607.12360#S12.p1.2)\.
- B\. Li, Z\. Wang, F\. Chen, S\. Zhao, R\. Zheng, and L\. Wu \(2026\)Optimal learning\-rate schedules under functional scaling laws: power decay and warmup–stable–decay\.InProceedings of the 39th Conference on Learning Theory \(COLT\),Note:arXiv:2602\.06797Cited by:[§12](https://arxiv.org/html/2607.12360#S12.p1.2)\.
- Q\. Li, C\. Tai, and W\. E \(2017\)Stochastic modified equations and adaptive stochastic gradient algorithms\.InProceedings of the 34th International Conference on Machine Learning \(ICML\),Cited by:[§12](https://arxiv.org/html/2607.12360#S12.p1.2)\.
- S\. Malladi, K\. Lyu, A\. Panigrahi, and S\. Arora \(2022\)On the SDEs and scaling rules for adaptive gradient algorithms\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§12](https://arxiv.org/html/2607.12360#S12.p1.2)\.
- E\. Monzio Compagnoni, R\. Islamov, F\. N\. Proske, A\. Lucchi, A\. Orvieto, and E\. Gorbunov \(2026\)On the interaction of batch noise, adaptivity, and compression, under\(L0,L1\)\(L\_\{0\},L\_\{1\}\)\-smoothness: an SDE approach\.InProceedings of the 43rd International Conference on Machine Learning \(ICML\),Note:arXiv:2506\.00181Cited by:[§1](https://arxiv.org/html/2607.12360#S1.p2.1),[§1](https://arxiv.org/html/2607.12360#S1.p4.3),[§12](https://arxiv.org/html/2607.12360#S12.p1.2),[§2](https://arxiv.org/html/2607.12360#S2.p1.5),[§9](https://arxiv.org/html/2607.12360#S9.p1.11)\.
- F\. Schaipp, A\. Hägele, A\. Taylor, U\. Simsekli, and F\. Bach \(2025\)The surprising agreement between convex optimization theory and learning\-rate scheduling for large model training\.arXiv preprint arXiv:2501\.18965\.Cited by:[§1](https://arxiv.org/html/2607.12360#S1.p2.1),[§11](https://arxiv.org/html/2607.12360#S11.SS0.SSS0.Px1.p1.3),[§11](https://arxiv.org/html/2607.12360#S11.SS0.SSS0.Px3.p1.1),[§12](https://arxiv.org/html/2607.12360#S12.p1.2)\.
- M\. Schmidt and N\. Le Roux \(2013\)Fast convergence of stochastic gradient descent under a strong growth condition\.arXiv preprint arXiv:1308\.6370\.Cited by:[§1](https://arxiv.org/html/2607.12360#S1.p4.3),[§11](https://arxiv.org/html/2607.12360#S11.SS0.SSS0.Px3.p1.1),[§12](https://arxiv.org/html/2607.12360#S12.p1.2),[Remark 1](https://arxiv.org/html/2607.12360#Thmremark1.p1.3)\.
- O\. Shamir and T\. Zhang \(2013\)Stochastic gradient descent for non\-smooth optimization: convergence results and optimal averaging schemes\.InProceedings of the 30th International Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2607.12360#S1.p2.1)\.
- E\. Shulgin, D\. von Rütte, T\. H\. Zhang, N\. Ajroldi, B\. Schölkopf, and A\. Orvieto \(2026\)Deriving hyperparameter scaling laws via modern optimization theory\.arXiv preprint arXiv:2603\.15958\.Cited by:[§1](https://arxiv.org/html/2607.12360#S1.p2.1),[§12](https://arxiv.org/html/2607.12360#S12.p1.2)\.
- S\. Vaswani, F\. Bach, and M\. Schmidt \(2019\)Fast and faster convergence of SGD for over\-parameterized models and an accelerated perceptron\.InProceedings of the 22nd International Conference on Artificial Intelligence and Statistics \(AISTATS\),Cited by:[§1](https://arxiv.org/html/2607.12360#S1.p4.3),[§11](https://arxiv.org/html/2607.12360#S11.SS0.SSS0.Px3.p1.1),[§12](https://arxiv.org/html/2607.12360#S12.p1.2),[Remark 1](https://arxiv.org/html/2607.12360#Thmremark1.p1.3)\.
- K\. Wen, Z\. Li, J\. Wang, D\. Hall, P\. Liang, and T\. Ma \(2025\)Understanding warmup–stable–decay learning rates: a river valley loss landscape view\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§12](https://arxiv.org/html/2607.12360#S12.p1.2)\.
- J\. Zhang, T\. He, S\. Sra, and A\. Jadbabaie \(2020\)Why gradient clipping accelerates training: a theoretical justification for adaptivity\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§12](https://arxiv.org/html/2607.12360#S12.p1.2),[§6](https://arxiv.org/html/2607.12360#S6.p1.10)\.