Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks

# Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks
Source: [https://arxiv.org/html/2607.07845](https://arxiv.org/html/2607.07845)
\\pdfximage

hessian\_supplement\.pdf

Marcel KühnInstitut für Theoretische Physik, Universität Leipzig, 04103 Leipzig, GermanyScaDS\.AI Dresden/Leipzig, Universität Leipzig, 04105 Leipzig, Germany

###### Abstract

The Hessian of the training loss governs the local geometry of the loss landscape, yet despite existing explanations for its largest eigenvalues, the origin of the vast multitude of vanishingly small eigenvalues remains elusive\. We argue that the bulk consists of the weakly lifted pseudo\-Goldstone modes of the continuous symmetries of the network parametrization\. In deep linear networks these symmetries are exact: they generate flat directions and hence exact zero modes, whose eigenvectors we construct explicitly\. Introducing a ReLU nonlinearity as a perturbation, we show that it breaks these symmetries weakly and explicitly\. Resolving the spectrum at the level of eigenvectors, we find that the high\-curvature directions are orthogonal to the symmetry subspace, while the bulk lies almost entirely within it\. We demonstrate the mechanism in a two\-layer ReLU student–teacher model and in a network trained on CIFAR\-10\. A convolutional example demonstrates that the same diagnostic extends beyond fully connected layers\. Together, these results link the Hessian bulk to weakly broken symmetries and clarify the origin of near\-zero modes\.

*Introduction—*The geometry of the loss landscape, encoded in the Hessian𝖧\\sf Hof the training loss, underlies much of our understanding of how deep networks optimize and generalize\[[41](https://arxiv.org/html/2607.07845#bib.bib44),[21](https://arxiv.org/html/2607.07845#bib.bib42),[7](https://arxiv.org/html/2607.07845#bib.bib43),[6](https://arxiv.org/html/2607.07845#bib.bib24),[13](https://arxiv.org/html/2607.07845#bib.bib47)\]\. It sets the local curvature relevant to second\-order optimization\[[3](https://arxiv.org/html/2607.07845#bib.bib2),[46](https://arxiv.org/html/2607.07845#bib.bib1),[45](https://arxiv.org/html/2607.07845#bib.bib3),[62](https://arxiv.org/html/2607.07845#bib.bib4)\], is repeatedly tied to generalization through the flatness of minima\[[27](https://arxiv.org/html/2607.07845#bib.bib6),[35](https://arxiv.org/html/2607.07845#bib.bib7),[15](https://arxiv.org/html/2607.07845#bib.bib8),[12](https://arxiv.org/html/2607.07845#bib.bib9),[29](https://arxiv.org/html/2607.07845#bib.bib10)\], and appears throughout pruning, robustness, and mode\-connectivity analyses\[[40](https://arxiv.org/html/2607.07845#bib.bib11),[25](https://arxiv.org/html/2607.07845#bib.bib12),[56](https://arxiv.org/html/2607.07845#bib.bib13),[17](https://arxiv.org/html/2607.07845#bib.bib14),[61](https://arxiv.org/html/2607.07845#bib.bib15),[48](https://arxiv.org/html/2607.07845#bib.bib16),[58](https://arxiv.org/html/2607.07845#bib.bib17),[20](https://arxiv.org/html/2607.07845#bib.bib22),[11](https://arxiv.org/html/2607.07845#bib.bib19),[2](https://arxiv.org/html/2607.07845#bib.bib23),[28](https://arxiv.org/html/2607.07845#bib.bib21)\]\.

Empirically, the Hessian spectrum of a deep network is highly anisotropic: a dense bulk of small and nearly vanishing eigenvalues sits below a few large outliers\[[53](https://arxiv.org/html/2607.07845#bib.bib25),[54](https://arxiv.org/html/2607.07845#bib.bib26),[22](https://arxiv.org/html/2607.07845#bib.bib27)\], the latter linked to data and class structure\[[49](https://arxiv.org/html/2607.07845#bib.bib28),[50](https://arxiv.org/html/2607.07845#bib.bib29)\]\. Analytic studies of shallow ReLU Hessians exploiting permutation symmetries\[[4](https://arxiv.org/html/2607.07845#bib.bib33),[5](https://arxiv.org/html/2607.07845#bib.bib34)\]and random\-matrix analyses of the closely related Fisher matrix\[[52](https://arxiv.org/html/2607.07845#bib.bib30),[31](https://arxiv.org/html/2607.07845#bib.bib32),[32](https://arxiv.org/html/2607.07845#bib.bib31)\]confirm that nonlinear networks generically carry many small but nonzero eigenvalues\. What has been missing is a single, identifiable origin for this bulk\.

We propose that, in fully connected networks in particular, the bulk is the spectral signature of weakly broken continuous symmetries of the specific architecture\. Linear multilayer networks possess exact symmetries—inter\-layer transformations that leave the function unchanged—which generate flat directions and thus exact zero modes\. A nonlinearity like standard ReLU breaks these symmetries only weakly: the would\-be flat directions acquire small curvature, exactly as an explicitly but weakly broken continuous symmetry produces parametrically light pseudo\-Goldstone modes; in this analogy, the spontaneous symmetry breaking occurs already at network initialization\. Additional flat directions may arise from low input variance\[[41](https://arxiv.org/html/2607.07845#bib.bib44)\], but often in far smaller number\. The picture connects naturally to the local geometry of loss landscapes\[[16](https://arxiv.org/html/2607.07845#bib.bib53)\]and to gradient descent’s confinement to a tiny top subspace\[[24](https://arxiv.org/html/2607.07845#bib.bib59)\], since gradients are orthogonal to these symmetry directions\.

Continuous and discrete network symmetries and their consequences for gradients and learning dynamics have been studied before\[[14](https://arxiv.org/html/2607.07845#bib.bib56),[11](https://arxiv.org/html/2607.07845#bib.bib19),[55](https://arxiv.org/html/2607.07845#bib.bib18),[38](https://arxiv.org/html/2607.07845#bib.bib35),[59](https://arxiv.org/html/2607.07845#bib.bib48),[64](https://arxiv.org/html/2607.07845#bib.bib20),[44](https://arxiv.org/html/2607.07845#bib.bib49)\], and degenerate Fisher matrices from parameter nonidentifiability are central to singular learning theory\[[60](https://arxiv.org/html/2607.07845#bib.bib62),[18](https://arxiv.org/html/2607.07845#bib.bib61)\]\. In particular, the general identity between symmetries and Hessian eigenvectors\[[38](https://arxiv.org/html/2607.07845#bib.bib35)\]and the count of finite eigenvalues at the minimum of a deep linear network\[[57](https://arxiv.org/html/2607.07845#bib.bib36),[9](https://arxiv.org/html/2607.07845#bib.bib37)\]are established\. In this Letter, we move from*counting*to*eigenvectors*, and from linear to nonlinear networks\. We give an explicit orthogonal basis of symmetry generators in multilayer linear networks; we show, at the eigenvector level, that these symmetries persist approximately in nonlinear networks and after training; we establish an adiabatic connection between the nonlinear and linear spectra; and we extend the mechanism to a convolutional network\.

*Symmetries and zero eigenvalues—*We consider a model𝒇\{\\bm\{f\}\}mapping inputs𝐱∈ℝN\{\\bf x\}\\in\\mathbb\{R\}^\{N\}to outputs𝒇​\(𝜽,𝐱\)∈ℝC\{\\bm\{f\}\}\(\{\\bm\{\\theta\}\},\{\\bf x\}\)\\in\\mathbb\{R\}^\{C\}, with parameters𝜽∈ℝd\{\\bm\{\\theta\}\}\\in\\mathbb\{R\}^\{d\}collecting all weights, and lossℒ​\(𝜽\)≔⟨ℓ​\(𝒇​\(𝜽,𝐱i\),𝒚i\)⟩i\\mathcal\{L\}\(\{\\bm\{\\theta\}\}\)\\coloneqq\\langle\\ell\(\\bm\{f\}\(\{\\bm\{\\theta\}\},\{\\bf x\}\_\{i\}\),\{\\bm\{y\}\}\_\{i\}\)\\rangle\_\{i\}averaged over a dataset\. The Hessian is𝖧=∂𝜽2ℒ\{\\sf H\}=\\partial^\{2\}\_\{\\bm\{\\theta\}\}\\mathcal\{L\}\. A continuous symmetry is a transformationϕ​\(α,𝜽\)\\bm\{\\phi\}\(\\alpha,\{\\bm\{\\theta\}\}\)withℒ​\(ϕ​\(α,𝜽\)\)=ℒ​\(𝜽\)\\mathcal\{L\}\(\{\\bm\{\\phi\}\}\(\\alpha,\{\\bm\{\\theta\}\}\)\)=\\mathcal\{L\}\(\{\\bm\{\\theta\}\}\)and, for simplicity,ϕ​\(0,𝜽\)=𝜽\{\\bm\{\\phi\}\}\(0,\{\\bm\{\\theta\}\}\)=\{\\bm\{\\theta\}\}\. Its generator isϕ′≔∂αϕ\|α=0\{\\bm\{\\phi\}\}^\{\\prime\}\\coloneqq\\left\.\\partial\_\{\\alpha\}\{\\bm\{\\phi\}\}\\right\|\_\{\\alpha=0\}\. We allow nonlinear symmetries \(e\.g\. parameter products and inverses\), as needed for the convolutional case below\. For symmetries linear in the weights,ϕ=𝖱​\(α\)​𝜽\{\\bm\{\\phi\}\}=\\mathsf\{R\}\(\\alpha\)\\bm\{\\theta\}with𝖱​\(δ​α\)≈𝟣\+δ​α​𝖦\\mathsf\{R\}\(\\delta\\alpha\)\\approx\\mathsf\{1\}\+\\delta\\alpha\\,\\mathsf\{G\}, one hasϕ′=𝖦​𝜽\{\\bm\{\\phi\}\}^\{\\prime\}=\\mathsf\{G\}\\bm\{\\theta\}\.

Differentiating the invariance condition∂αℒ\|α=0=0\\left\.\\partial\_\{\\alpha\}\\mathcal\{L\}\\right\|\_\{\\alpha=0\}=0with respect to𝜽\{\\bm\{\\theta\}\}gives\[[38](https://arxiv.org/html/2607.07845#bib.bib35)\]

0=𝖧​ϕ′\+\[∂𝜽\(ϕ′\)⊤\]​∂𝜽ℒ\.\\displaystyle 0=\{\\sf H\}\{\\bm\{\\phi\}\}^\{\\prime\}\+\\left\[\\partial\_\{\\bm\{\\theta\}\}\(\{\\bm\{\\phi\}\}^\{\\prime\}\)^\{\\top\}\\right\]\\partial\_\{\\bm\{\\theta\}\}\\mathcal\{L\}\\ \.\(1\)At critical points the second term vanishes andϕ′\{\\bm\{\\phi\}\}^\{\\prime\}is a zero\-eigenvalue eigenvector of𝖧\\sf H: every continuous symmetry of the loss is a flat direction\. Away from critical points, curvature alongϕ′\{\\bm\{\\phi\}\}^\{\\prime\}may be finite \(see the rotation\-invariant Mexican\-hat potential, End Matter,[Fig\.˜6](https://arxiv.org/html/2607.07845#S1.F6)\)\.

*Hessian, Gauss–Newton, and Fisher—*The architectural symmetries we construct are stronger than generic loss invariances: they leave the function𝒇\{\\bm\{f\}\}itself unchanged, so the network gradient is orthogonal to the symmetry generator,∂𝜽⊤𝒇​ϕ′=0\\partial\_\{\\bm\{\\theta\}^\{\\\!\\top\}\}\\\!\{\\bm\{f\}\}\\,\{\\bm\{\\phi\}\}^\{\\prime\}=0, at*every*𝜽\{\\bm\{\\theta\}\}\. This matters because of the Gauss–Newton decomposition,

𝖧=⟨∂𝜽𝒇i⊤​\(∂𝒇2ℓi\)​∂𝜽⊤𝒇i⟩i\+⟨\(∂𝒇⊤ℓi\)​∂𝜽2𝒇i⟩i,\\displaystyle\{\\sf H\}=\\big\\langle\\partial\_\{\\bm\{\\theta\}^\{\\vphantom\{\\top\}\}\}\{\\bm\{f\}\}^\{\\\!\\top\}\_\{i\}\\\!\\\!\\left\(\\partial\_\{\\\!\\bm\{f\}\}^\{2\}\\ell\_\{i\}\\right\)\\partial\_\{\\bm\{\\theta\}^\{\\\!\\top\}\}\\\!\{\\bm\{f\}\}\_\{i\}\\big\\rangle\_\{i\}\+\\big\\langle\\\!\\left\(\\partial\_\{\\\!\\bm\{f\}^\{\\\!\\top\}\}\\ell\_\{i\}\\right\)\\partial^\{2\}\_\{\\bm\{\\theta\}\}\{\\bm\{f\}\}\_\{i\}\\big\\rangle\_\{i\}\\ ,\(2\)with𝒇i≔𝒇​\(𝜽,𝐱i\)\{\\bm\{f\}\}\_\{i\}\\coloneqq\{\\bm\{f\}\}\(\{\\bm\{\\theta\}\},\{\\bf x\}\_\{i\}\)andℓi≔ℓ​\(𝒇i,𝒚i\)\\ell\_\{i\}\\coloneqq\\ell\(\{\\bm\{f\}\}\_\{i\},\{\\bm\{y\}\}\_\{i\}\)\. The second term is a residual that vanishes when the outputs interpolate the labels and is small near critical points\[[54](https://arxiv.org/html/2607.07845#bib.bib26)\]\. The first term, the generalized Gauss–Newton matrix𝖥\{\\sf F\}, is positive semidefinite for losses convex in the output and obeys𝖥​ϕ′=0\{\\sf F\}\\,\{\\bm\{\\phi\}\}^\{\\prime\}=0at*arbitrary*parameters, whereas[Eq\.˜1](https://arxiv.org/html/2607.07845#S0.E1)guarantees Hessian zero modes only at critical points\. For mean\-squared error and softmax cross\-entropy,𝖥\{\\sf F\}is the label\-independent Fisher information matrix\[[10](https://arxiv.org/html/2607.07845#bib.bib51),[26](https://arxiv.org/html/2607.07845#bib.bib50)\]and equals the Hessian of a self\-target loss whose labels are the network’s own outputs,

𝖥=∂𝜽′2⟨ℓ​\(𝒇i​\(𝜽′\),𝒇i​\(𝜽\)\)⟩i\|𝜽′=𝜽\.\\displaystyle\{\\sf F\}=\\partial^\{2\}\_\{\\bm\{\\theta\}^\{\\prime\}\}\\big\\langle\\ell\\big\(\{\\bm\{f\}\}\_\{i\}\(\{\\bm\{\\theta\}\}^\{\\prime\}\),\{\\bm\{f\}\}\_\{i\}\(\{\\bm\{\\theta\}\}\)\\big\)\\big\\rangle\_\{i\}\\Big\|\_\{\{\\bm\{\\theta\}\}^\{\\prime\}=\{\\bm\{\\theta\}\}\}\\ \.\(3\)𝖥\{\\sf F\}is in its own right the metric of natural\-gradient optimization and a standard curvature probe\[[3](https://arxiv.org/html/2607.07845#bib.bib2),[47](https://arxiv.org/html/2607.07845#bib.bib5),[52](https://arxiv.org/html/2607.07845#bib.bib30)\]\. The Hessian inherits these zero modes whenever the residual in[Eq\.˜2](https://arxiv.org/html/2607.07845#S0.E2)vanishes, as in the interpolating student–teacher experiments below\.

*Linear networks: exact symmetries and their eigenvectors—*Consider the two\-layer linear network𝒇lin​\(𝜽,𝐱\)=𝖶\(2\)​𝖶\(1\)​𝐱\{\\bm\{f\}\}\_\{\\rm lin\}\(\\bm\{\\theta\},\{\\bf x\}\)=\{\\sf W\}^\{\(2\)\}\{\\sf W\}^\{\(1\)\}\{\\bf x\}, with𝖶\(1\)∈ℝN×N\{\\sf W\}^\{\(1\)\}\\in\\mathbb\{R\}^\{N\\times N\},𝖶\(2\)∈ℝC×N\{\\sf W\}^\{\(2\)\}\\in\\mathbb\{R\}^\{C\\times N\}, and𝜽\\bm\{\\theta\}stacking both \(d=N2\+N​Cd=N^\{2\}\+NC\)\. Inserting any invertible𝖬\{\\sf M\}and its inverse between the layers leaves the output unchanged, so the network has a symmetry under the general linear groupGLN​\(ℝ\)\{\\rm GL\}\_\{N\}\(\\mathbb\{R\}\)\. We restrict to the positive\-determinant subgroup,GLN\+​\(ℝ\)\{\\rm GL\}\_\{N\}^\{\+\}\(\\mathbb\{R\}\), containing the identity\[[14](https://arxiv.org/html/2607.07845#bib.bib56),[38](https://arxiv.org/html/2607.07845#bib.bib35)\]\. Its generators are the matrices𝖠∈ℝN×N\{\\sf A\}\\in\\mathbb\{R\}^\{N\\times N\}, acting through𝖬=eα​𝖠\{\\sf M\}=\{\\rm e\}^\{\\alpha\{\\sf A\}\}as

𝖶\(2\)​e−α​𝖠​eα​𝖠​𝖶\(1\)​𝐱=𝖶\(2\)​𝖶\(1\)​𝐱,\\displaystyle\{\\sf W\}^\{\(2\)\}\{\\rm e\}^\{\-\\alpha\{\\sf A\}\}\{\\rm e\}^\{\\alpha\{\\sf A\}\}\{\\sf W\}^\{\(1\)\}\{\\bf x\}=\{\\sf W\}^\{\(2\)\}\{\\sf W\}^\{\(1\)\}\{\\bf x\}\\ ,\(4\)which maps the weights toeα​𝖠​𝖶\(1\)\{\\rm e\}^\{\\alpha\{\\sf A\}\}\{\\sf W\}^\{\(1\)\},𝖶\(2\)​e−α​𝖠\{\\sf W\}^\{\(2\)\}\{\\rm e\}^\{\-\\alpha\{\\sf A\}\}and gives the generators

ϕ𝖠′=\(vec​\(𝖠𝖶\(1\)\)vec​\(−𝖶\(2\)​𝖠\)\)\.\\displaystyle\\bm\{\\phi\}\_\{\\sf A\}^\{\\prime\}=\\begin\{pmatrix\}\{\\rm vec\}\\\!\\left\(\{\\sf A\}\{\\sf W\}^\{\(1\)\}\\right\)\\\\ \{\\rm vec\}\\\!\\left\(\-\{\\sf W\}^\{\(2\)\}\{\\sf A\}\\right\)\\end\{pmatrix\}\\ \.\(5\)TheN2N^\{2\}independent𝖠\{\\sf A\}can be organized into canceling rescalings \(diagonal𝖠\{\\sf A\}\), rotations \(antisymmetric𝖠\{\\sf A\}\), and hyperbolic rotations \(symmetric off\-diagonal𝖠\{\\sf A\}\)\. The discrete permutation symmetry present even with nonlinear activations \([Fig\.˜1](https://arxiv.org/html/2607.07845#S0.F1)\) is not continuous and produces no zero eigenvalues\.

Going beyond the count, we exhibit the eigenvectors explicitly\. With the singular value decompositions𝖶\(k\)=𝖴\(k\)​𝖲\(k\)​\(𝖵\(k\)\)⊤\{\\sf W\}^\{\(k\)\}=\{\\sf U\}^\{\(k\)\}\{\\sf S\}^\{\(k\)\}\(\{\\sf V\}^\{\(k\)\}\)^\{\\\!\\top\},k=1,2k=1,2, the choice𝖠\(n,m\)≔𝐯n\(2\)​\(𝐮m\(1\)\)⊤\{\\sf A\}^\{\(n,m\)\}\\coloneqq\{\\bf v\}\_\{n\}^\{\(2\)\}\(\{\\bf u\}\_\{m\}^\{\(1\)\}\)^\{\\\!\\top\}, consisting of the corresponding singular vectors, renders the generators \([5](https://arxiv.org/html/2607.07845#S0.E5)\) mutually orthogonal \(End Matter\)\. TheN2N^\{2\}orthogonal generators span exactly the null space of𝖧\\sf H\. The symmetry thus induces preciselyN2N^\{2\}zero modes, which carry over to𝖥\{\\sf F\}at any parameters and any input distribution\. The remainingN​CNCeigenvalues are generically finite, matching both the overparametrization \(the input–output map is a singleC×NC\\times Nmatrix\) and the known finite\-eigenvalue count at the minimum of deep linear networks\[[57](https://arxiv.org/html/2607.07845#bib.bib36),[9](https://arxiv.org/html/2607.07845#bib.bib37)\]\. Biases, absorbed by appending a constant input component, merely render this map affine, soN​C\+CNC\+Ceigenvalues stay finite, with details in the Supplemental Material \(SM\)\[[1](https://arxiv.org/html/2607.07845#bib.bib63)\]\. The construction extends to arbitrary layer sizes and additional layers\. In particular, for hidden dimensionMMwithN\>M\>CN\>M\>C, the two\-layer network hasM2\+\(N−M\)​\(M−C\)M^\{2\}\+\(N\-M\)\(M\-C\)architectural invariances and stillN​CNCfinite modes, the additional\(N−M\)​\(M−C\)\(N\-M\)\(M\-C\)zero modes correspond to directions of𝖶\(1\)\{\\sf W\}^\{\(1\)\}that map onto the kernel of the downstream map𝖶\(2\)\{\\sf W\}^\{\(2\)\}\[[1](https://arxiv.org/html/2607.07845#bib.bib63)\]\.

![Refer to caption](https://arxiv.org/html/2607.07845v1/x1.png)Figure 1:Symmetries in fully connected layers\. Left: original network, two hidden neurons highlighted\. Middle: discrete permutation symmetry, swapping incoming and outgoing connections of two hidden units\. Holds for any activation\. Right: continuous symmetry of linear networks, an infinitesimal rotation of the incoming weights compensated on the outgoing weights\. Preserves the output and generates a zero mode\.*Weak breaking: pseudo\-Goldstone modes—*While multilayer linear networks offer valuable theoretical insights\[[34](https://arxiv.org/html/2607.07845#bib.bib38),[57](https://arxiv.org/html/2607.07845#bib.bib36),[9](https://arxiv.org/html/2607.07845#bib.bib37)\], they are of limited practical relevance\. To reach realistic networks while keeping analytic control, we switch on a nonlinearity as a perturbation, defining

𝒇LReLU\(ε\)​\(𝜽,𝐱\)≔𝖶\(2\)​g\(ε\)​\(𝖶\(1\)​𝐱\),\\displaystyle\\bm\{f\}\_\{\\rm LReLU\}^\{\(\\varepsilon\)\}\(\\bm\{\\theta\},\{\\bf x\}\)\\coloneqq\{\\sf W\}^\{\(2\)\}\\,g^\{\(\\varepsilon\)\}\\\!\\left\(\{\\sf W\}^\{\(1\)\}\{\\bf x\}\\right\)\\ ,\(6\)with the Leaky\-ReLU activation\[[43](https://arxiv.org/html/2607.07845#bib.bib46)\]g\(ε\)​\(z\)=z​\[1−ε​Θ​\(−z\)\]g^\{\(\\varepsilon\)\}\(z\)=z\\,\[1\-\\varepsilon\\,\\Theta\(\-z\)\], whereΘ\\Thetais the Heaviside step\. Atε=0\\varepsilon=0the network is linear, atε=1\\varepsilon=1it is the standard ReLU\. For positively homogeneous activations like Leaky\-ReLU, the diagonal rescaling subgroup \(NNgenerators\) survives exactly for allε\\varepsilon\[[38](https://arxiv.org/html/2607.07845#bib.bib35)\]\. The remainingN2−NN^\{2\}\-Nsymmetry directions are the ones the nonlinearity can break\.

We study the student–teacher loss

ℒ\(ε\)​\(𝜽\)=12​C​⟨‖𝒇LReLU\(ε\)​\(𝜽,𝐱\)−𝝉\(ε\)​\(𝐱\)‖2⟩𝐱,\\displaystyle\\mathcal\{L\}^\{\(\\varepsilon\)\}\(\\bm\{\\theta\}\)=\\frac\{1\}\{2C\}\\left\\langle\\left\\lVert\\bm\{f\}\_\{\\rm LReLU\}^\{\(\\varepsilon\)\}\(\\bm\{\\theta\},\{\\bf x\}\)\-\\bm\{\\tau\}^\{\(\\varepsilon\)\}\(\{\\bf x\}\)\\right\\rVert^\{2\}\\right\\rangle\_\{\\bf x\}\\ ,\(7\)over Gaussian inputs𝐱∼𝒩​\(0,𝟣N\)\{\\bf x\}\\sim\\mathcal\{N\}\(0,\{\\sf 1\}\_\{N\}\)with teacher𝝉\(ε\)≔𝒇LReLU\(ε\)​\(𝜽0,⋅\)\\bm\{\\tau\}^\{\(\\varepsilon\)\}\\coloneqq\\bm\{f\}\_\{\\rm LReLU\}^\{\(\\varepsilon\)\}\(\\bm\{\\theta\}\_\{0\},\\cdot\), and evaluate the Hessian at the global minimum𝜽=𝜽0\\bm\{\\theta\}=\\bm\{\\theta\}\_\{0\}, where𝖧\(ε\)=𝖥\(ε\)\{\\sf H\}^\{\(\\varepsilon\)\}=\{\\sf F\}^\{\(\\varepsilon\)\}\. The family interpolates continuously between𝖧\(0\)=𝖧lin\{\\sf H\}^\{\(0\)\}=\{\\sf H\}\_\{\\rm lin\}with exact symmetries and the fully nonlinear𝖧\(1\)=𝖧ReLU\{\\sf H\}^\{\(1\)\}=\{\\sf H\}\_\{\\rm ReLU\}\.

*The lifted eigenvalues scale asε2\\varepsilon^\{2\}—*[Fig\.˜2](https://arxiv.org/html/2607.07845#S0.F2)tracks the full spectrum asε\\varepsilongrows from0to11\. The eigenvalues that are exactly zero atε=0\\varepsilon=0rise smoothly and—over almost the entire bulk—trackε2\\varepsilon^\{2\}all the way to the ReLU point\. This is sharper than it may look: since𝖧\(ε\)=𝖧lin\+𝒪​\(ε\)\{\\sf H\}^\{\(\\varepsilon\)\}=\{\\sf H\}\_\{\\rm lin\}\+\\mathcal\{O\}\(\\varepsilon\), one would naively expect a*linear*rise, but the projection onto the symmetry subspace cancels the linear term, leavingε2\\varepsilon^\{2\}\(End Matter\)\. TheNNrescaling modes remain pinned at zero for allε\\varepsilon\. The lifted eigenvalues thus behave as pseudo\-Goldstone modes: would\-be flat directions of a continuous symmetry, that is explicitly broken\.

![Refer to caption](https://arxiv.org/html/2607.07845v1/x2.png)Figure 2:Ranked Hessian eigenvalues of the two\-layer Leaky\-ReLU student–teacher model \(N=20N=20,C=3C=3, single draw of Gaussian teacher weights𝖶i​j\(k\)∼𝒩​\(0,σ2\)\{\\sf W\}^\{\(k\)\}\_\{ij\}\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}\)withσ2=1/N\\sigma^\{2\}=1/N\) as a function of the symmetry\-breaking strengthε\\varepsilon, from the linear network \(ε→0\\varepsilon\\to 0\) to ReLU \(ε=1\\varepsilon=1\)\. “N​C\+CNC\{\+\}C” and “N​C\+C\+1NC\{\+\}C\{\+\}1” are the modes bordering the gap\. The bulk eigenvalues, zero atε=0\\varepsilon=0, rise asε2\\varepsilon^\{2\}and follow the eigenvalues of the projected block𝖯𝖧\(ε\)​𝖯\{\\sf P\}\{\\sf H\}^\{\(\\varepsilon\)\}\{\\sf P\}\(dashed lines\)\. TheNNrescaling zero modes stay at zero for allε\\varepsilon, below the plotted range\. A small group ofCCmodes \(orange, the mean\-gradient outliers\) bends nearε≈0\.5\\varepsilon\\approx 0\.5to merge with the large block\. The cleanε2\\varepsilon^\{2\}trend withoutε4\\varepsilon^\{4\}bending indicates eigenvectors that stay within the symmetry subspace\.*Bulk eigenvectors stay in the symmetry subspace—*Anε2\\varepsilon^\{2\}law has a direct eigenvector consequence, which we both derive and measure\. Splitting Leaky\-ReLU into a linear and an absolute\-value part, the Hessian of this two\-layer model separates exactly,𝖧\(ε\)=\(1−ε2\)2​𝖧lin\+\(ε2\)2​𝖧\|⋅\|\{\\sf H\}^\{\(\\varepsilon\)\}=\(1\-\\tfrac\{\\varepsilon\}\{2\}\)^\{2\}\{\\sf H\}\_\{\\rm lin\}\+\(\\tfrac\{\\varepsilon\}\{2\}\)^\{2\}\{\\sf H\}\_\{\|\\cdot\|\}, the cross term vanishing by parity of the Gaussian input average \(End Matter\)\. Beyond two layers and sign\-symmetric inputs, the coupling between linear symmetry subspace and curvature subspace is generically𝒪​\(ε\)\\mathcal\{O\}\(\\varepsilon\), and only theε2\\varepsilon^\{2\}eigenvalue scaling survives\[[1](https://arxiv.org/html/2607.07845#bib.bib63)\]\. Here, however, the inter\-space coupling is𝒪​\(ε2\)\\mathcal\{O\}\(\\varepsilon^\{2\}\), so a bulk eigenvalue receives a mixing correction only at𝒪​\(ε4\)\\mathcal\{O\}\(\\varepsilon^\{4\}\), with a coefficient that is a nonnegative quadratic form in the leakage of the mode into the curvature subspace\. A mode that tracksε2\\varepsilon^\{2\}with*no*ε4\\varepsilon^\{4\}correction therefore has zero leakage: its eigenvector is confined to the symmetry subspace\. The cleanε2\\varepsilon^\{2\}scaling in[Fig\.˜2](https://arxiv.org/html/2607.07845#S0.F2)is thus spectral evidence for confinement and yields an adiabatic connection between the ReLU Hessian atε=1\\varepsilon=1and the linear one atε=0\\varepsilon=0\.

We confirm this directly at the ReLU point\. For each eigenvector𝐩i\{\\bf p\}\_\{i\}of𝖧ReLU\{\\sf H\}\_\{\\rm ReLU\}we measure its overlap with the symmetry subspace,

oi≔∥𝖯​𝐩i∥2=∑j:EV=0\(𝐩i⋅𝐩j\(lin\)\)2,\\displaystyle o\_\{i\}\\coloneqq\\lVert\{\\sf P\}\\,\{\\bf p\}\_\{i\}\\rVert^\{2\}\\ =\\sum\_\{j\\,:\\,\\text\{EV\}=0\}\\\!\\left\(\{\\bf p\}\_\{i\}\\cdot\{\\bf p\}\_\{j\}^\{\(\\rm lin\)\}\\right\)^\{\\\!2\},\(8\)with𝖯\{\\sf P\}the projector onto the symmetry subspace\.oi=1o\_\{i\}=1means𝐩i\{\\bf p\}\_\{i\}lies entirely within it\. Because the subspace can be large, the random\-vector baselineorandom=rank​\(𝖯\)/do\_\{\\rm random\}=\\mathrm\{rank\}\(\{\\sf P\}\)/dwithrank​\(𝖯\)=N2\\mathrm\{rank\}\(\{\\sf P\}\)=N^\{2\}is high, and the informative signal is notoi≃1o\_\{i\}\\simeq 1in the tail but the*suppression*ofoio\_\{i\}for the high\-curvature modes\.[Fig\.˜3](https://arxiv.org/html/2607.07845#S0.F3)shows the resulting two\-tier structure: the ReLU spectrum hasN​C\+C≈N​CNC\+C\\approx NClarge eigenvalues separated by a small gap from a bulk of small ones, and the overlap is near zero for theN​CNCleading modes and near unity for the bulk\. The symmetry directions of the linear network persist, approximately, as the bulk of the ReLU spectrum\. An analogous three\-layer student–teacher analysis in the SM\[[1](https://arxiv.org/html/2607.07845#bib.bib63)\]reproduces this two\-tier separation: the high\-curvature modes remain clearly suppressed in the overlap, albeit less completely than in the shallow case, and theε2\\varepsilon^\{2\}scaling holds only approximately toward the ReLU point\.

![Refer to caption](https://arxiv.org/html/2607.07845v1/x3.png)

![Refer to caption](https://arxiv.org/html/2607.07845v1/x4.png)

Figure 3:Two\-layer student–teacher model of[Fig\.˜2](https://arxiv.org/html/2607.07845#S0.F2), Hessian at the global minimum\. Top: ReLU \(green\) and linear \(orange\) spectra\. The linear network hasN​C=60NC=60finite eigenvalues, the ReLU network hasC=3C=3additional large eigenvalues due to nonzero⟨∂𝜽𝒇⊤⟩𝐱\\langle\\partial\_\{\\bm\{\\theta\}\}\{\\bm\{f\}\}^\{\\top\}\\rangle\_\{\\bf x\}\. Bottom: overlapoio\_\{i\}\([8](https://arxiv.org/html/2607.07845#S0.E8)\) between ReLU eigenvectors and the linear symmetry subspace\. Small\-eigenvalue modes haveoi≈1o\_\{i\}\\approx 1, revealing alignment with linear symmetry directions\. The leadingN​C\+CNC\+Cmodes are strongly suppressed \(random baselineorandom≈0\.87o\_\{\\rm random\}\\approx 0\.87\)\.*Additional outliers—*TheCCextra large eigenvalues of the ReLU Hessian—the modes peeling off in[Fig\.˜2](https://arxiv.org/html/2607.07845#S0.F2)—are the known mean\-gradient outliers\[[30](https://arxiv.org/html/2607.07845#bib.bib39),[49](https://arxiv.org/html/2607.07845#bib.bib28)\]\. ReLU units have positive average activations, so the network gradient has a finite mean⟨∂𝜽𝒇⊤⟩𝐱\\langle\\partial\_\{\\bm\{\\theta\}\}\{\\bm\{f\}\}^\{\\top\}\\rangle\_\{\\bf x\}\. Since𝖧\(ε\)=𝖥\(ε\)\{\\sf H\}^\{\(\\varepsilon\)\}=\{\\sf F\}^\{\(\\varepsilon\)\}is the input\-averaged gradient outer product, this mean contributes a rank\-CCterm producing the largest eigenvalues\. In a tractable large\-NNsetting the leading ReLU eigenvector aligns with the mean gradient while retaining a substantial component along the linear symmetry directions\[[1](https://arxiv.org/html/2607.07845#bib.bib63)\]\.

*A trained network on CIFAR\-10—*We now test the mechanism away from the idealized minimum, on a network trained on real data\. A three\-layer fully connected ReLU network—30723072\-dimensional input, two hidden layers of128128units,1010\-way softmax, no biases,410,880410\{,\}880parameters—reaches53%53\\%test accuracy on CIFAR\-10\[[36](https://arxiv.org/html/2607.07845#bib.bib45)\], as expected for a plain MLP\[[63](https://arxiv.org/html/2607.07845#bib.bib55)\]\. We compute the Hessian of the cross\-entropy training loss at the endpoint of training\[[1](https://arxiv.org/html/2607.07845#bib.bib63)\]\. While the endpoint is not an exact critical point, we still compare the training loss Hessian eigenvectors with the zero modes of a linear comparison\.

For the linear comparison model we copy the trained ReLU parameters and remove the hidden nonlinearities but keep the softmax, whose invariance under uniform shifts of theCCpreactivations\[[38](https://arxiv.org/html/2607.07845#bib.bib35)\]gives an effective output dimensionCeff≔C−1=9C\_\{\\rm eff\}\\coloneqq C\-1=9\(End Matter\)\. The input data carries a second, data\-dependent source of flatness: low\-variance input directions produce small Hessian eigenvalues\[[41](https://arxiv.org/html/2607.07845#bib.bib44)\]\. To control this, we whiten the inputs\[[1](https://arxiv.org/html/2607.07845#bib.bib63)\], rescaling the top100100data covariance directions \(90%90\\%of the cumulative variance\) to unity and leaving the rest at much smaller variance\[[8](https://arxiv.org/html/2607.07845#bib.bib41),[42](https://arxiv.org/html/2607.07845#bib.bib40)\], which yields an effective input dimensionNeff=100N\_\{\\rm eff\}=100\. The comparison subspace, collectively termed the symmetry subspace𝖯\{\\sf P\}, is the null space of the linear model’s Fisher with the low\-variance directions set to zero variance\. It combines the architectural symmetry directions with the suppressed\-variance directions\. Dimension counting makes this composition explicit: of thed−Neff​Ceff=409,980d\-N\_\{\\rm eff\}C\_\{\\rm eff\}=409\{,\}980directions in𝖯\{\\sf P\}, thed−N​Ceff=383,232d\-NC\_\{\\rm eff\}=383\{,\}232architectural symmetries leave the model function unchanged for*arbitrary*inputs\. The much smaller subset of\(N−Neff\)​Ceff=26,748\(N\-N\_\{\\rm eff\}\)C\_\{\\rm eff\}=26\{,\}748zero modes alters the function only along suppressed\-variance directions\.

![Refer to caption](https://arxiv.org/html/2607.07845v1/x5.png)Figure 4:Three\-layer ReLU network trained on CIFAR\-10, with effective dimensionsNeff=100N\_\{\\rm eff\}=100,Ceff=9C\_\{\\rm eff\}=9\. Overlapoio\_\{i\}\([8](https://arxiv.org/html/2607.07845#S0.E8)\) between the largest5,0005\{,\}000ReLU Hessian eigenvectors and the symmetry subspace𝖯\{\\sf P\}of the linear comparison model\. Apart fromCeff=9C\_\{\\rm eff\}=9leading outliers, only modes beyond theNeff​Ceff=900N\_\{\\rm eff\}C\_\{\\rm eff\}=900largest reachoi≈1o\_\{i\}\\approx 1\. The random baseline isorandom≈0\.998o\_\{\\rm random\}\\approx 0\.998\(standard deviation≈10−4\\approx 10^\{\-4\}\), so the leading modes are suppressed by hundreds to thousands of standard deviations\. The cumulative coverage \([9](https://arxiv.org/html/2607.07845#S0.E9)\) reachesO5000≈0\.95O\_\{5000\}\\approx 0\.95, implying the unmeasured tail is almost entirely symmetric\.Combining effective dimensions, the linear model predicts onlyNeff​Ceff=900N\_\{\\rm eff\}C\_\{\\rm eff\}=900high\-curvature directions, i\.e\.rank​\(𝖰\)=900\\mathrm\{rank\}\(\{\\sf Q\}\)=900with𝖰=𝟣−𝖯\{\\sf Q\}=\{\\sf 1\}\-\{\\sf P\}\.[Fig\.˜4](https://arxiv.org/html/2607.07845#S0.F4)shows the same two\-tier structure as the student–teacher model, where the leadingNeff​Ceff=900N\_\{\\rm eff\}C\_\{\\rm eff\}=900ReLU modes have vanishing overlap with𝖯\{\\sf P\}, the smaller eigenvalues have overlap near unity, and a set ofCeff=9C\_\{\\rm eff\}=9outliers again shows raised overlap \(mean\-gradient mechanism\)\. Here the suppression is dramatic: the random baseline isorandom≈0\.998o\_\{\\rm random\}\\approx 0\.998with standard deviation≈10−4\\approx 10^\{\-4\}, so the leading modes are suppressed by thousands of standard deviations\. With original data variances retained, the sharp transition is smoothed by covariance\-induced eigenvalue spread, and the symmetry mechanism entangles with data\-covariance flatness, although the leading high\-curvature modes still show substantially reduced overlap\[[1](https://arxiv.org/html/2607.07845#bib.bib63)\]\.

Numerical constraints restrict us to the leading5,0005\{,\}000eigenvectors of the410,880410\{,\}880\-dimensional Hessian, seemingly leaving the overlap of the remaining∼4×105\\sim\\\!4\\times 10^\{5\}modes unmeasured\. We close this gap without diagonalizing them by defining the*cumulative curvature coverage*

Ok≔1rank​\(𝖰\)​∑i=1k\(1−oi\)=Tr​\(𝖰​Πk\)Tr​\(𝖰\)≤1,\\displaystyle O\_\{k\}\\;\\coloneqq\\;\\frac\{1\}\{\\mathrm\{rank\}\(\{\\sf Q\}\)\}\\sum\_\{i=1\}^\{k\}\\,\(1\-o\_\{i\}\)\\;=\\;\\frac\{\\mathrm\{Tr\}\\\!\\left\(\{\\sf Q\}\\,\{\\sf\\Pi\}\_\{k\}\\right\)\}\{\\mathrm\{Tr\}\(\{\\sf Q\}\)\}\\ \\leq\\ 1\\ ,\(9\)whereΠk=∑i=1k𝐩i​𝐩i⊤\{\\sf\\Pi\}\_\{k\}=\\sum\_\{i=1\}^\{k\}\{\\bf p\}\_\{i\}\{\\bf p\}\_\{i\}^\{\\top\}projects onto the top\-kkeigenspace\.OkO\_\{k\}is the fraction of the entire high\-curvature subspace already spanned by the leadingkkHessian eigenvectors\. We findO5000≈0\.95O\_\{5000\}\\approx 0\.95: the first5,0005\{,\}000modes account for95%95\\%of the900900finite curvature directions of the linear network, so the remaining∼4×105\\sim\\\!4\\times 10^\{5\}modes share only the leftover5%5\\%and must, on average, have overlap with the symmetry subspace very close to unity\. Up to this5%5\\%residual, the bulk is symmetry\-organized in its entirety\.

*A convolutional network—*Convolutional networks are a standard architecture for extracting local features, most notably in image processing\[[19](https://arxiv.org/html/2607.07845#bib.bib58),[39](https://arxiv.org/html/2607.07845#bib.bib57),[23](https://arxiv.org/html/2607.07845#bib.bib54)\]\. To demonstrate that the same diagnostic applies beyond fully connected layers and beyond symmetries linear in the weights, we consider a one\-dimensional convolutional network

fconv​\(𝜽,𝐱\)=\(𝐰\(2\)\)⊤​\(𝐰\(1\)∗𝐱\),\\displaystyle f\_\{\\rm conv\}\(\\bm\{\\theta\},\{\\bf x\}\)=\\big\(\{\\bf w\}^\{\(2\)\}\\big\)\{\\\!\}^\{\\top\}\\\!\\left\(\{\\bf w\}^\{\(1\)\}\*\{\\bf x\}\\right\)\\ ,\(10\)with kernel𝐰\(1\)∈ℝK\{\\bf w\}^\{\(1\)\}\\in\\mathbb\{R\}^\{K\}and readout𝐰\(2\)∈ℝN\{\\bf w\}^\{\(2\)\}\\in\\mathbb\{R\}^\{N\}\. With periodic boundary conditions the convolution is a circulant matrix𝖢​\(𝐰\(1\)\)\{\\sf C\}\(\{\\bf w\}^\{\(1\)\}\), and inserting canceling exponentials as in[Eq\.˜4](https://arxiv.org/html/2607.07845#S0.E4)yields symmetries\. The generator must now map the circulant to another circulant, which because𝐰\(1\)\{\\bf w\}^\{\(1\)\}has onlyKKentries leaves exactlyKKcontinuous symmetry directions\. These symmetries are*nonlinear*in the parameters \(they involve the kernel through𝖢−1\{\\sf C\}^\{\-1\}\), which is precisely why we allowed nonlinearϕ\\bm\{\\phi\}from the outset\. The explicit construction and its two\-dimensional extension are in the SM\[[1](https://arxiv.org/html/2607.07845#bib.bib63)\]\.

Here the symmetry is broken not by a nonlinear activation but by a different mechanism: nonperiodic boundary conditions, which enlarge the input width toN\+K−1N\+K\-1and thereby reduce the overparametrization\. This makes the convolutional model a test of the generality of the diagnostic rather than of the specific nonlinear mechanism\. The conclusion is the same: in a nonperiodic student–teacher setup theKKsmallest\-eigenvalue eigenvectors retain near\-unity overlap with the symmetry subspace of the periodic network \([Fig\.˜5](https://arxiv.org/html/2607.07845#S0.F5)\)\. Broken symmetries again leave identifiable low\-curvature directions\.

![Refer to caption](https://arxiv.org/html/2607.07845v1/x6.png)Figure 5:Convolutional student–teacher network \([10](https://arxiv.org/html/2607.07845#S0.E10)\), input width200200, hidden widthN=190N=190, kernel widthK=11K=11, mean\-squared error, Hessian at identical student/teacher parameters drawn once from a Gaussian\. Overlap between the eigenvectors of the nonperiodic\-network Hessian and theK=11K=11symmetry directions of the corresponding periodic network, for the6060smallest eigenvalues\. The random baseline isK/d≈0\.055K/d\\approx 0\.055\(dashed\)\. Although the boundary conditions break the exact symmetry, the smallestKKmodes retain near\-unity overlap\.*Conclusion—*We have proposed a mechanism for the Hessian bulk\. The near\-zero modes are the weakly lifted pseudo\-Goldstone modes of architectural continuous symmetries: exact in linear networks, with explicitly constructed eigenvectors; weakly and explicitly broken by nonlinearity, with eigenvectors that remain largely confined to the symmetry subspace, adiabatically connected to the linear limit\. Because the symmetry zero modes survive in the Fisher at arbitrary parameters, the same directions organize the Fisher and Gauss–Newton spectra, with consequences for natural\-gradient and second\-order methods\[[3](https://arxiv.org/html/2607.07845#bib.bib2),[51](https://arxiv.org/html/2607.07845#bib.bib52),[47](https://arxiv.org/html/2607.07845#bib.bib5),[9](https://arxiv.org/html/2607.07845#bib.bib37)\]\. The mechanism is expected to generalize beyond fully connected architectures to any model containing such layers, including the fully connected blocks of modern transformers\. Altogether, these results provide theoretical grounding and practical intuition for interpreting the Hessian spectrum and optimization dynamics\.

*Acknowledgments—*M\.K\. acknowledges financial support by the Federal Ministry of Research, Technology and Space of Germany and by the Sächsische Staatsministerium für Wissenschaft, Kultur und Tourismus in the programme Center of Excellence for AI\-research “Center for Scalable Data Analytics and Artificial Intelligence Dresden/Leipzig”, project identification number: ScaDS\.AI\. The work of B\.R\. is supported in part by the Simons Foundation Grant No\. 12574\.

*Data availability—*The data that support the findings of this article are openly available\[[37](https://arxiv.org/html/2607.07845#bib.bib64)\]\.

## References

- \[1\]Note:See Supplemental Material for derivations, model extensions, and numerical details\.Cited by:[§I](https://arxiv.org/html/2607.07845#S1.p2.13),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p10.17),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p14.11),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p15.12),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p16.5),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p17.5),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p18.10),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p19.8),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p21.8)\.
- \[2\]S\. Ainsworth, J\. Hayase, and S\. Srinivasa\(2023\)Git Re\-Basin: Merging Models modulo Permutation Symmetries\.InThe Eleventh International Conference on Learning Representations,Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[3\]S\. Amari\(1998\)Natural Gradient Works Efficiently in Learning\.Neural Computation10\(2\),pp\. 251–276\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p23.1),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p8.9)\.
- \[4\]Y\. Arjevani and M\. Field\(2020\)Analytic Characterization of the Hessian in Shallow ReLU Models: A Tale of Symmetry\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 5441–5452\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p3.1)\.
- \[5\]Y\. Arjevani and M\. Field\(2021\)Analytic Study of Families of Spurious Minima in Two\-Layer ReLU Neural Networks: A Tale of Symmetry II\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 15162–15174\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p3.1)\.
- \[6\]A\. Atanasov, A\. Meterez, J\. Simon, and C\. Pehlevan\(2025\)The Optimization Landscape of SGD Across the Feature Learning Strength\.InInternational Conference on Learning Representations,Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[7\]S\. Becker, Y\. Zhang, and A\. A\. Lee\(2020\-03\)Geometry of Energy Landscapes and the Optimizability of Deep Neural Networks\.Phys\. Rev\. Lett\.124,pp\. 108301\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[8\]A\. J\. Bell and T\. J\. Sejnowski\(1997\)The “independent components” of natural scenes are edge filters\.Vision Research37\(23\),pp\. 3327–3338\.External Links:ISSN 0042\-6989Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p18.10)\.
- \[9\]A\. Bernacchia, M\. Lengyel, and G\. Hennequin\(2018\)Exact natural gradient in deep linear networks and its application to the nonlinear case\.InAdvances in Neural Information Processing Systems,Vol\.31\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p10.17),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p11.8),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p23.1),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p5.1)\.
- \[10\]C\. M\. Bishop and N\. M\. Nasrabadi\(2006\)Pattern recognition and machine learning\.Vol\.4,Springer\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p8.8)\.
- \[11\]J\. Brea, B\. Simsek, B\. Illing, and W\. Gerstner\(2019\)Weight\-space symmetry in deep networks gives rise to permutation saddles, connected by equal\-loss valleys across the loss landscape\.External Links:1907\.02911Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p5.1)\.
- \[12\]P\. Chaudhari, A\. Choromanska, S\. Soatto, Y\. LeCun, C\. Baldassi, C\. Borgs, J\. Chayes, L\. Sagun, and R\. Zecchina\(2019\)Entropy\-SGD: biasing gradient descent into wide valleys\.Journal of Statistical Mechanics: Theory and Experiment\(12\),pp\. 124018\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[13\]J\. Cohen, S\. Kaur, Y\. Li, J\. Z\. Kolter, and A\. Talwalkar\(2021\)Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability\.InInternational Conference on Learning Representations,Cited by:[§I](https://arxiv.org/html/2607.07845#S1.p7.11),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[14\]S\. S\. Du, W\. Hu, and J\. D\. Lee\(2018\)Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced\.Advances in neural information processing systems31\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p5.1),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p9.10)\.
- \[15\]P\. Foret, A\. Kleiner, H\. Mobahi, and B\. Neyshabur\(2021\)Sharpness\-aware Minimization for Efficiently Improving Generalization\.InInternational Conference on Learning Representations,Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[16\]S\. Fort and S\. Ganguli\(2019\)Emergent properties of the local geometry of neural loss landscapes\.External Links:1910\.05929Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p4.1)\.
- \[17\]E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh\(2023\)OPTQ: Accurate Quantization for Generative Pre\-trained Transformers\.InThe Eleventh International Conference on Learning Representations,Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[18\]K\. Fukumizu and S\. Amari\(2000\)Local minima and plateaus in hierarchical structures of multilayer perceptrons\.Neural Networks13\(3\),pp\. 317–327\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p5.1)\.
- \[19\]K\. Fukushima\(1980\)Neocognitron: A self\-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position\.Biological cybernetics36\(4\),pp\. 193–202\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p21.9)\.
- \[20\]T\. Garipov, P\. Izmailov, D\. Podoprikhin, D\. P\. Vetrov, and A\. G\. Wilson\(2018\)Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs\.InAdvances in Neural Information Processing Systems,S\. Bengio, H\. Wallach, H\. Larochelle, K\. Grauman, N\. Cesa\-Bianchi, and R\. Garnett \(Eds\.\),Vol\.31\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[21\]M\. Geiger, S\. Spigler, S\. d’Ascoli, L\. Sagun, M\. Baity\-Jesi, G\. Biroli, and M\. Wyart\(2019\)Jamming transition as a paradigm to understand the loss landscape of deep neural networks\.Phys\. Rev\. E100,pp\. 012115\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[22\]B\. Ghorbani, S\. Krishnan, and Y\. Xiao\(2019\)An Investigation into Neural Net Optimization via Hessian Eigenvalue Density\.InProceedings of the 36th International Conference on Machine Learning,Vol\.97,pp\. 2232–2241\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p3.1)\.
- \[23\]I\. Goodfellow, Y\. Bengio, and A\. Courville\(2016\)Deep Learning\.MIT press Cambridge, MA, USA\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p21.9)\.
- \[24\]G\. Gur\-Ari, D\. A\. Roberts, and E\. Dyer\(2018\)Gradient Descent Happens in a Tiny Subspace\.External Links:1812\.04754Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p4.1)\.
- \[25\]B\. Hassibi and D\. Stork\(1992\)Second order derivatives for network pruning: Optimal Brain Surgeon\.InAdvances in Neural Information Processing Systems,Vol\.5\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[26\]T\. Heskes\(2000\)On “natural” learning and pruning in multilayered perceptrons\.Neural Computation12\(4\),pp\. 881–901\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p8.8)\.
- \[27\]S\. Hochreiter and J\. Schmidhuber\(1997\)Flat Minima\.Neural Computation9\(1\),pp\. 1–42\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[28\]A\. Ito, M\. Yamada, and A\. Kumagai\(2025\)Linear Mode Connectivity between Multiple Models modulo Permutation Symmetries\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 26611–26626\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[29\]S\. Jastrzębski, Z\. Kenton, D\. Arpit, N\. Ballas, A\. Fischer, Y\. Bengio, and A\. Storkey\(2018\)Three Factors Influencing Minima in SGD\.External Links:1711\.04623Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[30\]R\. Karakida, S\. Akaho, and S\. Amari\(2019\)The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p16.5)\.
- \[31\]R\. Karakida, S\. Akaho, and S\. Amari\(2019\)Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach\.InProceedings of the Twenty\-Second International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.89,pp\. 1032–1041\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p3.1)\.
- \[32\]R\. Karakida, S\. Akaho, and S\. Amari\(2021\)Pathological Spectra of the Fisher Information Metric and Its Variants in Deep Neural Networks\.Neural Computation33\(8\),pp\. 2274–2307\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p3.1)\.
- \[33\]T\. Kato\(1995\-02\)Perturbation theory for linear operators\.2 edition,Springer\.Cited by:[§I](https://arxiv.org/html/2607.07845#S1.p5.10),[§I](https://arxiv.org/html/2607.07845#S1.p6.11)\.
- \[34\]K\. Kawaguchi\(2016\)Deep Learning without Poor Local Minima\.InAdvances in Neural Information Processing Systems,Vol\.29\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p11.8)\.
- \[35\]N\. S\. Keskar, D\. Mudigere, J\. Nocedal, M\. Smelyanskiy, and P\. T\. P\. Tang\(2017\)On Large\-Batch Training for Deep Learning: Generalization Gap and Sharp Minima\.InInternational Conference on Learning Representations,Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[36\]A\. Krizhevsky\(2009\)Learning multiple layers of features from tiny images\.Technical reportTechnical report, University of Toronto,University of Toronto,Toronto, Ontario\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p17.5)\.
- \[37\]M\. Kühn and B\. Rosenow\(2026\)Github repository,[https://github\.com/RosenowGroup/approximate\-symmetries\-hessian](https://github.com/RosenowGroup/approximate-symmetries-hessian)\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p25.1)\.
- \[38\]D\. Kunin, J\. Sagastuy\-Brena, S\. Ganguli, D\. L\. Yamins, and H\. Tanaka\(2021\)Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics\.InInternational Conference on Learning Representations,Cited by:[§I](https://arxiv.org/html/2607.07845#S1.p7.11),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p11.7),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p18.10),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p5.1),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p7.2),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p9.10)\.
- \[39\]Y\. LeCun, L\. Bottou, Y\. Bengio, and P\. Haffner\(1998\)Gradient\-based learning applied to document recognition\.Proceedings of the IEEE86\(11\),pp\. 2278–2324\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p21.9)\.
- \[40\]Y\. LeCun, J\. Denker, and S\. Solla\(1989\)Optimal Brain Damage\.InAdvances in Neural Information Processing Systems,Vol\.2\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[41\]Y\. LeCun, I\. Kanter, and S\. A\. Solla\(1991\-05\)Eigenvalues of covariance matrices: Application to neural\-network learning\.Phys\. Rev\. Lett\.66,pp\. 2396–2399\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p18.10),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p4.1)\.
- \[42\]J\. Lee, S\. Schoenholz, J\. Pennington, B\. Adlam, L\. Xiao, R\. Novak, and J\. Sohl\-Dickstein\(2020\)Finite Versus Infinite Neural Networks: an Empirical Study\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 15156–15172\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p18.10)\.
- \[43\]A\. L\. Maas, A\. Y\. Hannun, A\. Y\. Ng,et al\.\(2013\)Rectifier nonlinearities improve neural network acoustic models\.InProceedings of the 30th International Conference on Machine Learning,Vol\.28\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p11.7)\.
- \[44\]S\. Marcotte, R\. Gribonval, and G\. Peyré\(2023\)Abide by the law and follow the flow: conservation laws for gradient flows\.InThirty\-seventh Conference on Neural Information Processing Systems,Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p5.1)\.
- \[45\]J\. Martens and R\. Grosse\(2015\)Optimizing Neural Networks with Kronecker\-factored Approximate Curvature\.InProceedings of the 32nd International Conference on Machine Learning,Vol\.37,pp\. 2408–2417\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[46\]J\. Martens\(2010\)Deep learning via Hessian\-free optimization\.InProceedings of the 27th International Conference on International Conference on Machine Learning,pp\. 735–742\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[47\]J\. Martens\(2020\)New Insights and Perspectives on the Natural Gradient Method\.Journal of Machine Learning Research21\(146\),pp\. 1–76\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p23.1),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p8.9)\.
- \[48\]S\. Moosavi\-Dezfooli, A\. Fawzi, J\. Uesato, and P\. Frossard\(2019\)Robustness via Curvature Regularization, and Vice Versa\.In2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 9070–9078\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[49\]V\. Papyan\(2019\)Measurements of Three\-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians\.InProceedings of the 36th International Conference on Machine Learning,Vol\.97,pp\. 5012–5021\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p16.5),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p3.1)\.
- \[50\]V\. Papyan\(2020\)Traces of Class/Cross\-Class Structure Pervade Deep Learning Spectra\.Journal of Machine Learning Research21\(252\),pp\. 1–64\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p3.1)\.
- \[51\]R\. Pascanu and Y\. Bengio\(2013\)Revisiting natural gradient for deep networks\.arXiv preprint arXiv:1301\.3584\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p23.1)\.
- \[52\]J\. Pennington and P\. Worah\(2018\)The Spectrum of the Fisher Information Matrix of a Single\-Hidden\-Layer Neural Network\.InAdvances in Neural Information Processing Systems,Vol\.31\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p3.1),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p8.9)\.
- \[53\]L\. Sagun, L\. Bottou, and Y\. LeCun\(2017\)Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond\.External Links:1611\.07476Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p3.1)\.
- \[54\]L\. Sagun, U\. Evci, V\. U\. Guney, Y\. Dauphin, and L\. Bottou\(2018\)Empirical Analysis of the Hessian of Over\-Parametrized Neural Networks\.External Links:1706\.04454Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p3.1),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p8.8)\.
- \[55\]B\. Simsek, F\. Ged, A\. Jacot, F\. Spadaro, C\. Hongler, W\. Gerstner, and J\. Brea\(2021\)Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 9722–9732\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p5.1)\.
- \[56\]S\. P\. Singh and D\. Alistarh\(2020\)WoodFisher: Efficient Second\-Order Approximation for Neural Network Compression\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 18098–18109\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[57\]S\. P\. Singh, G\. Bachmann, and T\. Hofmann\(2021\)Analytic Insights into Structure and Rank of Neural Network Hessian Maps\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 23914–23927\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p10.17),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p11.8),[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p5.1)\.
- \[58\]S\. Singla and S\. Feizi\(2020\)Second\-Order Provable Defenses against Adversarial Attacks\.InProceedings of the 37th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.119,pp\. 8981–8991\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[59\]H\. Tanaka and D\. Kunin\(2021\)Noether’s Learning Dynamics: Role of Symmetry Breaking in Neural Networks\.InAdvances in Neural Information Processing Systems,A\. Beygelzimer, Y\. Dauphin, P\. Liang, and J\. W\. Vaughan \(Eds\.\),Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p5.1)\.
- \[60\]S\. Watanabe\(2009\)Algebraic geometry and statistical learning theory\.Vol\.25,Cambridge university press\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p5.1)\.
- \[61\]Z\. Yao, A\. Gholami, Q\. Lei, K\. Keutzer, and M\. W\. Mahoney\(2018\)Hessian\-based Analysis of Large Batch Training and Robustness to Adversaries\.InAdvances in Neural Information Processing Systems,Vol\.31\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[62\]Z\. Yao, A\. Gholami, S\. Shen, M\. Mustafa, K\. Keutzer, and M\. Mahoney\(2021\)Adahessian: An adaptive second order optimizer for machine learning\.Inproceedings of the AAAI conference on artificial intelligence,Vol\.35,pp\. 10665–10673\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p2.1)\.
- \[63\]C\. Zhang, S\. Bengio, M\. Hardt, B\. Recht, and O\. Vinyals\(2021\)Understanding deep learning \(still\) requires rethinking generalization\.Communications of the ACM64\(3\)\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p17.5)\.
- \[64\]B\. Zhao, N\. Dehmamy, R\. Walters, and R\. Yu\(2022\)Symmetry Teleportation for Accelerated Optimization\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 16679–16690\.Cited by:[Explaining Near\-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks](https://arxiv.org/html/2607.07845#p5.1)\.

## IEnd Matter

*Appendix A: Curvature despite symmetry in the Mexican\-hat picture—*The mechanism of[Eq\.˜1](https://arxiv.org/html/2607.07845#S0.E1)is seen most simply in the rotation\-invariant potentialℒ​\(x,y\)=\(x2\+y2−1\)2\\mathcal\{L\}\(x,y\)=\(x^\{2\}\+y^\{2\}\-1\)^\{2\}\([Fig\.˜6](https://arxiv.org/html/2607.07845#S1.F6)\)\. At the minimum\(1,0\)\(1,0\)the symmetry direction\(0,1\)\(0,1\)is a flat zero mode\. Away from the minimum, at\(x,0\)\(x,0\)withx\>1x\>1, an infinitesimal move alongyyis a rotation combined with an𝒪​\(ϵ2\)\\mathcal\{O\}\(\\epsilon^\{2\}\)radial shift\. Because the radial gradient is nonzero there, the loss increases at𝒪​\(ϵ2\)\\mathcal\{O\}\(\\epsilon^\{2\}\)and the curvature along the symmetry direction becomes finite\. Hence, symmetry directions are exactly flat at critical points but may acquire curvature off them, controlled by the residual term in[Eq\.˜1](https://arxiv.org/html/2607.07845#S0.E1)\.

![Refer to caption](https://arxiv.org/html/2607.07845v1/x7.png)

![Refer to caption](https://arxiv.org/html/2607.07845v1/x8.png)

Figure 6:Zero Hessian eigenvalue from rotational symmetry,ℒ​\(x,y\)=\(x2\+y2−1\)2\\mathcal\{L\}\(x,y\)=\(x^\{2\}\+y^\{2\}\-1\)^\{2\}\. Left: equipotential lines \(green\)\. Right: zoom on the minimum\. Atθ=\(1,0\)\\theta=\(1,0\)the symmetry direction𝐩=\(0,1\)\{\\bf p\}=\(0,1\)gives rise toΔ​ℒ=𝒪​\(ϵ4\)\\Delta\\mathcal\{L\}=\\mathcal\{O\}\(\\epsilon^\{4\}\)with vanishing second derivative forε→0\\varepsilon\\to 0\. At\(x,0\)\(x,0\)withx\>1x\>1, a shiftΔ​y=ϵ\\Delta y=\\epsilon\(blue\) is a rotation \(green\) plus a radial shiftΔ​x∝ϵ2\\Delta x\\propto\\epsilon^\{2\}\(red\), raising the loss by𝒪​\(ϵ2\)\\mathcal\{O\}\(\\epsilon^\{2\}\)with nonzero second derivative \(finite Hessian eigenvalue\)\.*Appendix B: Orthogonal generator basis—*For the two\-layer linear network, the choice𝖠\(n,m\)≔𝐯n\(2\)​\(𝐮m\(1\)\)⊤\{\\sf A\}^\{\(n,m\)\}\\coloneqq\{\\bf v\}\_\{n\}^\{\(2\)\}\(\{\\bf u\}\_\{m\}^\{\(1\)\}\)^\{\\\!\\top\}in[Eq\.˜5](https://arxiv.org/html/2607.07845#S0.E5), with the singular value decompositions𝖶\(1\)=𝖴\(1\)​𝖲\(1\)​\(𝖵\(1\)\)⊤\{\\sf W\}^\{\(1\)\}=\{\\sf U\}^\{\(1\)\}\{\\sf S\}^\{\(1\)\}\(\{\\sf V\}^\{\(1\)\}\)^\{\\\!\\top\}and𝖶\(2\)=𝖴\(2\)​𝖲\(2\)​\(𝖵\(2\)\)⊤\{\\sf W\}^\{\(2\)\}=\{\\sf U\}^\{\(2\)\}\{\\sf S\}^\{\(2\)\}\(\{\\sf V\}^\{\(2\)\}\)^\{\\\!\\top\}, gives

𝖠\(n,m\)​𝖶\(1\)\\displaystyle\{\\sf A\}^\{\(n,m\)\}\{\\sf W\}^\{\(1\)\}=sm\(1\)​𝐯n\(2\)​\(𝐯m\(1\)\)⊤,\\displaystyle=s\_\{m\}^\{\(1\)\}\\,\{\\bf v\}\_\{n\}^\{\(2\)\}\\big\(\{\\bf v\}\_\{m\}^\{\(1\)\}\\big\)^\{\\\!\\top\},\(B1a\)𝖶\(2\)​𝖠\(n,m\)\\displaystyle\{\\sf W\}^\{\(2\)\}\{\\sf A\}^\{\(n,m\)\}=sn\(2\)​𝐮n\(2\)​\(𝐮m\(1\)\)⊤,\\displaystyle=s\_\{n\}^\{\(2\)\}\\,\{\\bf u\}\_\{n\}^\{\(2\)\}\\big\(\{\\bf u\}\_\{m\}^\{\(1\)\}\\big\)^\{\\\!\\top\},\(B1b\)wheresm\(i\)s\_\{m\}^\{\(i\)\}is a singular value,𝐯n\(i\)\{\\bf v\}\_\{n\}^\{\(i\)\}a right singular vector and𝐮n\(i\)\{\\bf u\}\_\{n\}^\{\(i\)\}a left singular vector\. It is straightforward to show that the corresponding generator vectors are mutually orthogonal\. Note that𝐮n\(2\)\{\\bf u\}\_\{n\}^\{\(2\)\}may be undefined forn\>Cn\>CwhenC<NC<N, but thensn\(2\)=0s\_\{n\}^\{\(2\)\}=0and the block vanishes\. TheN2N^\{2\}orthogonal generators span the null eigenspace of𝖧\\sf H, establishing exactlyN2N^\{2\}zero eigenvalues\. The generalization to arbitrary input, hidden, and output dimensions, including the bottleneck regimes where naive parameter counting fails, is given in the SM\[[1](https://arxiv.org/html/2607.07845#bib.bib63)\]\.

*Appendix C:ε2\\varepsilon^\{2\}scaling and confinement of the bulk—*We make precise the two statements used in the main text\. Write the Leaky\-ReLU activation as

g\(ε\)​\(z\)=\(1−ε2\)​z\+ε2​\|z\|,\\displaystyle g^\{\(\\varepsilon\)\}\(z\)=\\big\(1\-\\tfrac\{\\varepsilon\}\{2\}\\big\)\\,z\+\\tfrac\{\\varepsilon\}\{2\}\\,\|z\|\\ ,\(C1\)so that the two\-layer network splits into a linear and an absolute\-value part,

𝒇LReLU\(ε\)​\(𝜽,𝐱\)=\(1−ε2\)​𝒇lin​\(𝜽,𝐱\)\+ε2​𝒇\|⋅\|​\(𝜽,𝐱\)\.\\displaystyle\\bm\{f\}\_\{\\rm LReLU\}^\{\(\\varepsilon\)\}\(\\bm\{\\theta\},\{\\bf x\}\)=\\big\(1\-\\tfrac\{\\varepsilon\}\{2\}\\big\)\\bm\{f\}\_\{\\rm lin\}\(\\bm\{\\theta\},\{\\bf x\}\)\+\\tfrac\{\\varepsilon\}\{2\}\\,\\bm\{f\}\_\{\|\\cdot\|\}\(\\bm\{\\theta\},\{\\bf x\}\)\\ \.\(C2\)At the interpolating minimum𝖧\(ε\)≡𝖥\(ε\)=⟨\(∂𝜽⊤𝒇\(ε\)\)⊤​∂𝜽⊤𝒇\(ε\)⟩𝐱\{\\sf H\}^\{\(\\varepsilon\)\}\\equiv\{\\sf F\}^\{\(\\varepsilon\)\}=\\langle\(\\partial\_\{\\bm\{\\theta\}^\{\\top\}\\\!\}\\bm\{f\}^\{\(\\varepsilon\)\}\)^\{\\top\}\\partial\_\{\\bm\{\\theta\}^\{\\top\}\\\!\}\\bm\{f\}^\{\(\\varepsilon\)\}\\rangle\_\{\\bf x\}\. With𝖩0=∂𝜽𝒇lin⊤\{\\sf J\}\_\{0\}=\\partial\_\{\\bm\{\\theta\}\}\\bm\{f\}\_\{\\rm lin\}^\{\\top\}and𝖩1=∂𝜽𝒇\|⋅\|⊤\{\\sf J\}\_\{1\}=\\partial\_\{\\bm\{\\theta\}\}\\bm\{f\}\_\{\|\\cdot\|\}^\{\\top\},

𝖥\(ε\)\\displaystyle\{\\sf F\}^\{\(\\varepsilon\)\}=\(1−ε2\)2​⟨𝖩0​𝖩0⊤⟩\+\(1−ε2\)​ε2​⟨𝖩0​𝖩1⊤\+𝖩1​𝖩0⊤⟩\\displaystyle=\\big\(1\-\\tfrac\{\\varepsilon\}\{2\}\\big\)^\{2\}\\langle\{\\sf J\}^\{\\vphantom\{\\top\}\}\_\{0\}\{\\sf J\}\_\{0\}^\{\\top\}\\rangle\+\\big\(1\-\\tfrac\{\\varepsilon\}\{2\}\\big\)\\tfrac\{\\varepsilon\}\{2\}\\langle\{\\sf J\}^\{\\vphantom\{\\top\}\}\_\{0\}\{\\sf J\}^\{\\top\}\_\{1\}\+\{\\sf J\}^\{\\vphantom\{\\top\}\}\_\{1\}\{\\sf J\}^\{\\top\}\_\{0\}\\rangle\+\(ε2\)2​⟨𝖩1​𝖩1⊤⟩\.\\displaystyle\\quad\+\\big\(\\tfrac\{\\varepsilon\}\{2\}\\big\)^\{2\}\\langle\{\\sf J\}\_\{1\}^\{\\vphantom\{\\top\}\}\{\\sf J\}\_\{1\}^\{\\top\}\\rangle\\ \.\(C3\)The cross term vanishes identically\. Under𝐱→−𝐱\{\\bf x\}\\to\-\{\\bf x\}the centered Gaussian measure is invariant,𝖩0​\(𝐱\)\{\\sf J\}\_\{0\}\(\{\\bf x\}\)is odd and𝖩1​\(𝐱\)\{\\sf J\}\_\{1\}\(\{\\bf x\}\)is even, so⟨𝖩0​𝖩1⊤⟩=0\\langle\{\\sf J\}\_\{0\}^\{\\vphantom\{\\top\}\}\{\\sf J\}\_\{1\}^\{\\top\}\\rangle=0\. The Fisher is therefore given by

𝖥\(ε\)=\(1−ε2\)2​𝖥lin\+\(ε2\)2​𝖥\|⋅\|,\\displaystyle\{\\sf F\}^\{\(\\varepsilon\)\}=\\big\(1\-\\tfrac\{\\varepsilon\}\{2\}\\big\)^\{2\}\\,\{\\sf F\}\_\{\\rm lin\}\+\\big\(\\tfrac\{\\varepsilon\}\{2\}\\big\)^\{2\}\\,\{\\sf F\}\_\{\|\\cdot\|\}\\ ,\(C4\)with𝖥lin=⟨𝖩0​𝖩0⊤⟩\{\\sf F\}\_\{\\rm lin\}=\\langle\{\\sf J\}\_\{0\}^\{\\vphantom\{\\top\}\}\{\\sf J\}\_\{0\}^\{\\top\}\\rangleand𝖥\|⋅\|=⟨𝖩1​𝖩1⊤⟩\{\\sf F\}\_\{\|\\cdot\|\}=\\langle\{\\sf J\}\_\{1\}^\{\\vphantom\{\\top\}\}\{\\sf J\}^\{\\top\}\_\{1\}\\rangle\.

Let𝖯\{\\sf P\}project onto the symmetry subspace—the null space of𝖥lin\{\\sf F\}\_\{\\rm lin\}, spanned by theN2N^\{2\}generators—and𝖰=𝟣−𝖯\{\\sf Q\}=\{\\sf 1\}\-\{\\sf P\}\. Since𝖥lin​𝖯=0\{\\sf F\}\_\{\\rm lin\}\{\\sf P\}=0, the symmetry block and the off\-diagonal block are both𝒪​\(ε2\)\\mathcal\{O\}\(\\varepsilon^\{2\}\),

\(𝖥\(ε\)\)𝖯𝖯=\(ε2\)2​𝖥\|⋅\|𝖯𝖯,\(𝖥\(ε\)\)𝖰𝖯=\(ε2\)2​𝖥\|⋅\|𝖰𝖯,\\displaystyle\\big\(\{\\sf F\}^\{\(\\varepsilon\)\}\\big\)\{\\vphantom\{\\sf F\}\}^\{\\sf PP\}=\\big\(\\tfrac\{\\varepsilon\}\{2\}\\big\)^\{2\}\{\\sf F\}^\{\\sf PP\}\_\{\|\\cdot\|\},\\qquad\\big\(\{\\sf F\}^\{\(\\varepsilon\)\}\\big\)\{\\vphantom\{\\sf F\}\}^\{\\sf QP\}=\\big\(\\tfrac\{\\varepsilon\}\{2\}\\big\)^\{2\}\{\\sf F\}^\{\\sf QP\}\_\{\|\\cdot\|\}\\ ,\(C5\)while the curvature block\(𝖥\(ε\)\)𝖰𝖰=\(1−ε2\)2​𝖥lin𝖰𝖰\+𝒪​\(ε2\)\\big\(\{\\sf F\}^\{\(\\varepsilon\)\}\\big\)\{\\vphantom\{\\sf F\}\}^\{\\sf QQ\}=\(1\-\\tfrac\{\\varepsilon\}\{2\}\)^\{2\}\{\\sf F\}\_\{\\rm lin\}^\{\\sf QQ\}\+\\mathcal\{O\}\(\\varepsilon^\{2\}\)stays𝒪​\(1\)\\mathcal\{O\}\(1\)\. Here,\(⋅\)𝖯𝖯∈ℝrank​\(𝖯\)×rank​\(𝖯\)\(\\cdot\)^\{\\sf PP\}\\\!\\in\\mathbb\{R\}^\{\{\\rm rank\}\(\{\\sf P\}\)\\times\{\\rm rank\}\(\{\\sf P\}\)\},rank​\(𝖯\)=N2\{\\rm rank\}\(\{\\sf P\}\)=N^\{2\}, denotes the block of a matrix restricted to the symmetry subspace, with analogous restrictions\(⋅\)𝖰𝖯\(\\cdot\)^\{\\sf QP\},\(⋅\)𝖯𝖰\(\\cdot\)^\{\\sf PQ\}and\(⋅\)𝖰𝖰\(\\cdot\)^\{\\sf QQ\}\. Two conclusions follow\.

![Refer to caption](https://arxiv.org/html/2607.07845v1/x9.png)

![Refer to caption](https://arxiv.org/html/2607.07845v1/x10.png)

Figure 7:Two\-layer student–teacher model \(N=20N=20,C=3C=3\) with softmax output, single draw of Gaussian teacher weights𝖶i​j\(k\)∼𝒩​\(0,1/N\)\{\\sf W\}^\{\(k\)\}\_\{ij\}\\sim\\mathcal\{N\}\(0,1/N\), Hessian at the global minimum\. Softmax reduces the effective output width toCeff=2C\_\{\\rm eff\}=2, givingN​Ceff=40NC\_\{\\rm eff\}=40instead ofN​C=60NC=60finite eigenvalues in the linear comparison network \(top\)\. The overlapoio\_\{i\}\(bottom\) is near unity for the correspondingly increased number of small\-eigenvalue ReLU modes\.*\(i\) Scaling\.*An eigenvalue branch leaving a degenerate level is analytic in the perturbation strength—here\(ε/2\)2\(\\varepsilon/2\)^\{2\}—so to leading order the bulk eigenvalues areλi​\(ε\)=ε24​μi\+𝒪​\(ε4\)\\lambda\_\{i\}\(\\varepsilon\)=\\tfrac\{\\varepsilon^\{2\}\}\{4\}\\mu\_\{i\}\+\\mathcal\{O\}\(\\varepsilon^\{4\}\)\[[33](https://arxiv.org/html/2607.07845#bib.bib60)\], withμi\\mu\_\{i\}the eigenvalues of𝖥\|⋅\|𝖯𝖯\{\\sf F\}^\{\\sf PP\}\_\{\|\\cdot\|\}\. The naive expectation of a linear rise \(from𝖥\(ε\)=𝖥lin\+𝒪​\(ε\)\{\\sf F\}^\{\(\\varepsilon\)\}=\{\\sf F\}\_\{\\rm lin\}\+\\mathcal\{O\}\(\\varepsilon\)\) is removed by the projection, which eliminates the linear term inside the symmetry block\. Note that the𝒪​\(ε2\)\\mathcal\{O\}\(\\varepsilon^\{2\}\)scaling of the eigenvalues persists even when the cross term⟨𝖩0​𝖩1⊤\+𝖩1​𝖩0⊤⟩\\langle\{\\sf J\}\_\{0\}^\{\\vphantom\{\\top\}\}\{\\sf J\}\_\{1\}^\{\\top\}\+\{\\sf J\}\_\{1\}^\{\\vphantom\{\\top\}\}\{\\sf J\}\_\{0\}^\{\\top\}\\rangleis nonzero—for non\-Gaussian data, more than two weight layers, or different activation functions—since𝖯𝖩0=0\{\\sf P\}\{\\sf J\}\_\{0\}^\{\\vphantom\{\\top\}\}=0still holds\. Due to the positive homogeneity of Leaky\-ReLU, theNNdiagonal\-rescaling generators remain exact symmetries and stay pinned at zero for allε\\varepsilon\.

*\(ii\) Confinement\.*The second\-order coefficient of the eigenvalue branch leaving the degenerate level is given by an inter\-block term\[[33](https://arxiv.org/html/2607.07845#bib.bib60)\]

λi​\(ε\)=ε24​μi−ε416​∥\(𝖥lin𝖰𝖰\)−1/2​𝖥\|⋅\|𝖰𝖯​𝐩𝖯,i∥2\+𝒪​\(ε6\),\\displaystyle\\lambda\_\{i\}\(\\varepsilon\)=\\tfrac\{\\varepsilon^\{2\}\}\{4\}\\mu\_\{i\}\-\\tfrac\{\\varepsilon^\{4\}\}\{16\}\\,\\big\\lVert\(\{\\sf F\}\_\{\\rm lin\}^\{\\sf QQ\}\)^\{\-1/2\}\{\\sf F\}^\{\\sf QP\}\_\{\|\\cdot\|\}\\,\{\\bf p\}\_\{\{\\sf P\},i\}\\big\\rVert^\{2\}\+\\mathcal\{O\}\(\\varepsilon^\{6\}\)\\ ,\(C6\)where𝐩𝖯,i∈ℝrank​\(𝖯\)\{\\bf p\}\_\{\{\\sf P\},i\}\\in\\mathbb\{R\}^\{\{\\rm rank\}\(\{\\sf P\}\)\}is the corresponding eigenvector of𝖥\|⋅\|𝖯𝖯\{\\sf F\}^\{\\sf PP\}\_\{\|\\cdot\|\}\. Theε4\\varepsilon^\{4\}coefficient is a non\-negative quadratic form in the leakage𝖥\|⋅\|𝖰𝖯​𝐩𝖯,i\{\\sf F\}^\{\\sf QP\}\_\{\|\\cdot\|\}\\,\{\\bf p\}\_\{\{\\sf P\},i\}\(the curvature block𝖥lin𝖰𝖰\{\\sf F\}\_\{\\rm lin\}^\{\\sf QQ\}is positive definite\), and vanishes*if and only if*the leakage is zero, i\.e\. iff the mode is confined to the symmetry subspace\. Hence a bulk eigenvalue that followsε2\\varepsilon^\{2\}with noε4\\varepsilon^\{4\}correction signifies a confined eigenvector\. The empiricalε2\\varepsilon^\{2\}trend across the bulk in[Fig\.˜2](https://arxiv.org/html/2607.07845#S0.F2), up toε=1\\varepsilon=1, is the spectral counterpart of the direct overlap measurement in[Fig\.˜3](https://arxiv.org/html/2607.07845#S0.F3)\. TheCCmodes bending away are exactly those with significant leakage—the mean\-gradient outliers\.

*Appendix D: Softmax and the effective output dimension—*For a linear classifier the number of finite Hessian eigenvalues is given by the input dimensionNNmultiplied by the output dimensionCC\. A softmax output activation normalizes the outputs and is invariant under uniform shifts of theCCpreactivations\[[38](https://arxiv.org/html/2607.07845#bib.bib35)\], an exact symmetry that reduces the effective output dimension toCeff=C−1C\_\{\\rm eff\}=C\-1and the finite\-eigenvalue count toN​CeffNC\_\{\\rm eff\}\. This exact rank reduction is distinct from the overall curvature suppression of softmax cross\-entropy at high confidence late in training\[[13](https://arxiv.org/html/2607.07845#bib.bib47)\]\.[Fig\.˜7](https://arxiv.org/html/2607.07845#S1.F7)shows the effect in a modified version of the two\-layer student–teacher model of[Eq\.˜7](https://arxiv.org/html/2607.07845#S0.E7): with softmax, the linear network hasN​CeffNC\_\{\\rm eff\}finite eigenvalues, the ReLU network has≈N​Ceff\\approx NC\_\{\\rm eff\}large modes with the rest at near\-unity overlap, and the number of mean\-gradient outliers isCeffC\_\{\\rm eff\}rather thanCC\. For the two\-layer mean\-squared error model of[Eq\.˜7](https://arxiv.org/html/2607.07845#S0.E7), the Hessian is obtained analytically by evaluating the Gaussian integrals over the input distribution in closed form\. Here, with softmax and cross\-entropy, we approximate the Hessian using a finite sample of32,76832\{,\}768Gaussian inputs\. After sampling, the empirical mean is subtracted and the covariance is normalized to the identity, so that the finite dataset reproduces𝐱∼𝒩​\(0,𝟣N\)\{\\bf x\}\\sim\\mathcal\{N\}\(0,\{\\sf 1\}\_\{N\}\)in its first two moments\.