Tag
The paper introduces the Dominant Aligned Subspace (DAS), a low-dimensional subspace built from Hessian eigenvectors selected by gradient alignment rather than eigenvalue magnitude, and shows that DAS-restricted training with line search matches full-space training on PINN benchmarks (Poisson, heat, wave, Burgers), suggesting convergence only needs a small gradient-aligned subset of Hessian directions.
该论文提出使用输入损失曲率(扩散模型训练损失Hessian的Frobenius范数)作为无需采样的代理指标,用于量化模型层面和样本层面的记忆化现象,并证明了反事实记忆化由留一模型的输入曲率所控制。
提出了SparseDecoding,一种面向解码阶段的训练无关剪枝框架,通过在自回归生成时收集校准数据对齐Hessian分布,并开发优化的N:M稀疏矩阵向量核,在A100上实现最高1.48倍端到端解码加速。
This paper re-examines the edge of stability in deep learning optimization, proposing a new formulation based on directional Hessian and gradient-alignment score for more accurate predictions and diagnostic tools.
This paper argues that the dominant subspace of the Hessian, while contributing little to loss reduction, plays a key role in reducing sharpness during mini-batch SGD. It derives a sharpness correction term induced by mini-batch noise in the dominant directions.
This paper derives the eigenvalues of the Hessian for linear neural networks of arbitrary width and depth, showing that sharpness relates to maximum class proportion for classification tasks with MSE loss, and empirically validates the predictions.
This paper argues that the bulk of near-zero Hessian eigenvalues in neural networks arises from weakly broken symmetries of the network parametrization, showing that high-curvature directions are orthogonal to the symmetry subspace while the bulk lies within it.
This paper investigates how well sharpness and complexity together explain generalization in deep neural networks, introducing a Pareto-based analysis and function-oriented definitions to expand the explanatory scope.
Derives the closed-form gradient of the Wolkowicz-Styan upper bound on the loss Hessian eigenspectrum to guide neural network training toward flat minima, and introduces Hessian Spectral Range (HSR) Regularization. Numerical experiments show that HSR narrows the Hessian eigenvalue range, avoids sharp minima and saddle points, and achieves flat solutions comparable to Sharpness-Aware Minimization (SAM).
Gefen is a memory-efficient optimizer that reduces AdamW's memory footprint by ~8x by automatically sharing second-moment estimates and quantizing the first moment using a learned codebook, while maintaining AdamW-level performance.
This paper presents an exact decomposition of the curvature exponent α in neural network loss landscapes, explaining why it varies across layer types. It introduces the spectral alignment decomposition and derives a spectral transfer identity linking curvature, gradient rank decay, and Hessian exponents, validated across architectures and datasets.
Explains the mathematical concepts of gradient, Jacobian, and Hessian as fundamental tools in AI model training, describing how they measure change and their roles in optimization.
After 8 years, the author rewrote the open-source pytorch-hessian-eigenthings library, providing efficient eigendecomposition of Hessian and other curvature matrices for PyTorch models using iterative methods like Lanczos.