Tag
This paper derives the eigenvalues of the Hessian for linear neural networks of arbitrary width and depth, showing that sharpness relates to maximum class proportion for classification tasks with MSE loss, and empirically validates the predictions.
This paper argues that the bulk of near-zero Hessian eigenvalues in neural networks arises from weakly broken symmetries of the network parametrization, showing that high-curvature directions are orthogonal to the symmetry subspace while the bulk lies within it.
This paper diagnoses the loss landscape of gradient-based inversion for the Gray-Scott reaction-diffusion system, showing that direct backpropagation fails due to flat plateaus and sharp cliffs, while PINN components like residual loss smooth the landscape. The findings provide design implications for PINN-type methods.
MIT researchers show that the edge of stability (EoS) in neural network training is not merely a global optimization phenomenon but selectively redistributes learning across subsets of the training distribution, amplifying progress on some data groups while suppressing others. They identify two key conditions governing this allocation: gradient alignment with the top Hessian eigenvector and sustained non-vanishing gradient magnitude.
This paper presents an exact decomposition of the curvature exponent α in neural network loss landscapes, explaining why it varies across layer types. It introduces the spectral alignment decomposition and derives a spectral transfer identity linking curvature, gradient rank decay, and Hessian exponents, validated across architectures and datasets.
This paper identifies a consistent three-regime structure in scientific machine learning models, showing that optimization effectiveness is regime-specific and can challenge conventional loss-landscape interpretations. It proposes a regime-aware diagnostic framework validated across PINNs, neural operators, and neural ODEs.