hessian

This paper presents an exact decomposition of the curvature exponent α in neural network loss landscapes, explaining why it varies across layer types. It introduces the spectral alignment decomposition and derives a spectral transfer identity linking curvature, gradient rank decay, and Hessian exponents, validated across architectures and datasets.

Explains the mathematical concepts of gradient, Jacobian, and Hessian as fundamental tools in AI model training, describing how they measure change and their roles in optimization.

After 8 years, the author rewrote the open-source pytorch-hessian-eigenthings library, providing efficient eigendecomposition of Hessian and other curvature matrices for PyTorch models using iterative methods like Lanczos.