loss-landscape

该论文基于等变度数理论（equivariant degree theory）提出了一种通用的拓扑对称性指数，用于刻画神经网络损失景观中临界轨道的各向同性及其净拓扑贡献，适用于任意网络架构并给出软激活 MLP、Transformer 和等变神经网络的计算示例。

This paper classifies when the represented mobility map of parameter gradient flow is fully determined by the representation, proving an initialization classification for wide two-factor matrix representations and relating ordinary closure to invariant-level metric factorization in analytic and smooth regimes.

A new paper proposes TMLN (Trajectory-Modulatory Landscape Navigation), which treats continual learning as an optimal control problem over a curved loss landscape, using a diagonal empirical Fisher Information Matrix and trajectory-based gradient preconditioning to mitigate catastrophic forgetting without additive regularization penalties.

The paper introduces Sharp Mode Connectivity (SMC) to enhance the robustness of weight-space interpolation in neural networks by optimizing for flatness along entire paths, achieving significant accuracy improvements under distribution shifts.

This thesis elucidates mode connectivity in neural network loss landscapes through spectral dynamics, explaining phenomena like grokking and memorization, and applies insights to enhance diverse feature learning and practical neural network training.

This paper derives the eigenvalues of the Hessian for linear neural networks of arbitrary width and depth, showing that sharpness relates to maximum class proportion for classification tasks with MSE loss, and empirically validates the predictions.

This paper argues that the bulk of near-zero Hessian eigenvalues in neural networks arises from weakly broken symmetries of the network parametrization, showing that high-curvature directions are orthogonal to the symmetry subspace while the bulk lies within it.

This paper diagnoses the loss landscape of gradient-based inversion for the Gray-Scott reaction-diffusion system, showing that direct backpropagation fails due to flat plateaus and sharp cliffs, while PINN components like residual loss smooth the landscape. The findings provide design implications for PINN-type methods.

MIT researchers show that the edge of stability (EoS) in neural network training is not merely a global optimization phenomenon but selectively redistributes learning across subsets of the training distribution, amplifying progress on some data groups while suppressing others. They identify two key conditions governing this allocation: gradient alignment with the top Hessian eigenvector and sustained non-vanishing gradient magnitude.

This paper presents an exact decomposition of the curvature exponent α in neural network loss landscapes, explaining why it varies across layer types. It introduces the spectral alignment decomposition and derives a spectral transfer identity linking curvature, gradient rank decay, and Hessian exponents, validated across architectures and datasets.

This paper identifies a consistent three-regime structure in scientific machine learning models, showing that optimization effectiveness is regime-specific and can challenge conventional loss-landscape interpretations. It proposes a regime-aware diagnostic framework validated across PINNs, neural operators, and neural ODEs.