How the Hessian-Spectrum of Neural Networks Depends on Data
Summary
This paper derives the eigenvalues of the Hessian for linear neural networks of arbitrary width and depth, showing that sharpness relates to maximum class proportion for classification tasks with MSE loss, and empirically validates the predictions.
View Cached Full Text
Cached at: 07/16/26, 04:23 AM
# How the Hessian-Spectrum of Neural Networks Depends on Data Source: [https://arxiv.org/abs/2607.13631](https://arxiv.org/abs/2607.13631) [View PDF](https://arxiv.org/pdf/2607.13631) > Abstract:The Hessian matrix is an important quantity of interest when it comes to studying the loss landscape and optimization dynamics in deep learning, as well as designing measures of generalization, second\-order learning algorithms, etc\. Prior works have focused on empirical results or pursued a theoretical treatment under overly simplified settings\. In this work, we derive the eigenvalues of the Hessian of linear networks with arbitrary widths and depths, and datasets with an arbitrary number of samples, features, and labels\. Importantly, for classification tasks with MSE loss, we identify that the sharpness of the solution is directly related to the maximum proportion of samples belonging to any class\. We empirically validate our predictions and systematically analyze the effects of shedding the impractical assumptions one at a time, as well as incorporating nonlinearities\. We observe that our predictions are considerably robust in most cases, allowing us to extend our conclusions to more practical learning setups\. ## Submission history From: Jasraj Singh \[[view email](https://arxiv.org/show-email/5d893aae/2607.13631)\] **\[v1\]**Wed, 15 Jul 2026 09:25:49 UTC \(919 KB\)
Similar Articles
Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks
Derives the closed-form gradient of the Wolkowicz-Styan upper bound on the loss Hessian eigenspectrum to guide neural network training toward flat minima, and introduces Hessian Spectral Range (HSR) Regularization. Numerical experiments show that HSR narrows the Hessian eigenvalue range, avoids sharp minima and saddle points, and achieves flat solutions comparable to Sharpness-Aware Minimization (SAM).
Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks
This paper argues that the bulk of near-zero Hessian eigenvalues in neural networks arises from weakly broken symmetries of the network parametrization, showing that high-curvature directions are orthogonal to the symmetry subspace while the bulk lies within it.
Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent
This paper presents an exact decomposition of the curvature exponent α in neural network loss landscapes, explaining why it varies across layer types. It introduces the spectral alignment decomposition and derives a spectral transfer identity linking curvature, gradient rank decay, and Hessian exponents, validated across architectures and datasets.
Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuations
This paper argues that the dominant subspace of the Hessian, while contributing little to loss reduction, plays a key role in reducing sharpness during mini-batch SGD. It derives a sharpness correction term induced by mini-batch noise in the dominant directions.
Edge of Stability Selectively Shapes Learning Across the Data Distribution
MIT researchers show that the edge of stability (EoS) in neural network training is not merely a global optimization phenomenon but selectively redistributes learning across subsets of the training distribution, amplifying progress on some data groups while suppressing others. They identify two key conditions governing this allocation: gradient alignment with the top Hessian eigenvector and sustained non-vanishing gradient magnitude.