hessian

Tag

Cards List
#hessian

A Systematic Analysis of a Small Optimal Subspace for Physics-Informed Neural Networks: $n$-DAS $\subset \mathbb{R}^p$

arXiv cs.LG ↗ · 2d ago Cached

The paper introduces the Dominant Aligned Subspace (DAS), a low-dimensional subspace built from Hessian eigenvectors selected by gradient alignment rather than eigenvalue magnitude, and shows that DAS-restricted training with line search matches full-space training on PINN benchmarks (Poisson, heat, wave, Burgers), suggesting convergence only needs a small gradient-aligned subset of Hessian directions.

0 favorites 0 likes
#hessian

Loss Curvature as a Sampling-Free Proxy for Memorization in Diffusion Models

arXiv cs.LG ↗ · 2026-10-15 Cached

该论文提出使用输入损失曲率(扩散模型训练损失Hessian的Frobenius范数)作为无需采样的代理指标,用于量化模型层面和样本层面的记忆化现象,并证明了反事实记忆化由留一模型的输入曲率所控制。

0 favorites 0 likes
#hessian

SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference

Hugging Face Daily Papers ↗ · 2026-10-08 Cached

提出了SparseDecoding,一种面向解码阶段的训练无关剪枝框架,通过在自回归生成时收集校准数据对齐Hessian分布,并开发优化的N:M稀疏矩阵向量核,在A100上实现最高1.48倍端到端解码加速。

0 favorites 0 likes
#hessian

The Road Taken: The Role of Optimizers at the Edge of Stability

arXiv cs.LG ↗ · 2026-08-20 Cached

This paper re-examines the edge of stability in deep learning optimization, proposing a new formulation based on directional Hessian and gradient-alignment score for more accurate predictions and diagnostic tools.

0 favorites 0 likes
#hessian

Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuations

arXiv cs.LG ↗ · 2026-07-28 Cached

This paper argues that the dominant subspace of the Hessian, while contributing little to loss reduction, plays a key role in reducing sharpness during mini-batch SGD. It derives a sharpness correction term induced by mini-batch noise in the dominant directions.

0 favorites 0 likes
#hessian

How the Hessian-Spectrum of Neural Networks Depends on Data

arXiv cs.LG ↗ · 2026-07-16 Cached

This paper derives the eigenvalues of the Hessian for linear neural networks of arbitrary width and depth, showing that sharpness relates to maximum class proportion for classification tasks with MSE loss, and empirically validates the predictions.

0 favorites 0 likes
#hessian

Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks

arXiv cs.LG ↗ · 2026-07-10 Cached

This paper argues that the bulk of near-zero Hessian eigenvalues in neural networks arises from weakly broken symmetries of the network parametrization, showing that high-curvature directions are orthogonal to the symmetry subspace while the bulk lies within it.

0 favorites 0 likes
#hessian

How Far Can Sharpness and Complexity Jointly Explain Generalization?

arXiv cs.LG ↗ · 2026-06-30 Cached

This paper investigates how well sharpness and complexity together explain generalization in deep neural networks, introducing a Pareto-based analysis and function-oriented definitions to expand the explanatory scope.

0 favorites 0 likes
#hessian

Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks

arXiv cs.LG ↗ · 2026-06-30 Cached

Derives the closed-form gradient of the Wolkowicz-Styan upper bound on the loss Hessian eigenspectrum to guide neural network training toward flat minima, and introduces Hessian Spectral Range (HSR) Regularization. Numerical experiments show that HSR narrows the Hessian eigenvalue range, avoids sharp minima and saddle points, and achieves flat solutions comparable to Sharpness-Aware Minimization (SAM).

0 favorites 0 likes
#hessian

Gefen: Optimized Stochastic Optimizer

arXiv cs.LG ↗ · 2026-06-15 Cached

Gefen is a memory-efficient optimizer that reduces AdamW's memory footprint by ~8x by automatically sharing second-moment estimates and quantizing the first moment using a learned codebook, while maintaining AdamW-level performance.

0 favorites 0 likes
#hessian

Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent

arXiv cs.LG ↗ · 2026-06-03 Cached

This paper presents an exact decomposition of the curvature exponent α in neural network loss landscapes, explaining why it varies across layer types. It introduces the spectral alignment decomposition and derives a spectral transfer identity linking curvature, gradient rank decay, and Hessian exponents, validated across architectures and datasets.

0 favorites 0 likes
#hessian

@techNmak: This math sits underneath every AI model being trained right now. Gradient. Jacobian. Hessian. Three words that look in…

X AI KOLs Timeline ↗ · 2026-05-23 Cached

Explains the mathematical concepts of gradient, Jacobian, and Hessian as fundamental tools in AI model training, describing how they measure change and their roles in optimization.

0 favorites 0 likes
#hessian

After 8 years, I rewrote my open-source PyTorch curvature library

Hacker News Top ↗ · 2026-05-14 Cached

After 8 years, the author rewrote the open-source pytorch-hessian-eigenthings library, providing efficient eigendecomposition of Hessian and other curvature matrices for PyTorch models using iterative methods like Lanczos.

0 favorites 0 likes
← Back to home

Submit Feedback