Tag
This paper analyzes Adam as a diagonal empirical Fisher approximation to natural gradient descent, measuring its geometric deviation across linear, logistic, and small neural network loss landscapes. Results show deviation grows under ill-conditioning but Adam still reaches low loss, suggesting its effectiveness comes from approximation-error balance and momentum smoothing rather than true NGD tracking.
This paper introduces DP-NGD, a practical framework that integrates natural gradient descent with differential privacy by decoupling curvature estimation from private data and reconciling isotropic DP constraints with anisotropic second-order optimization, achieving state-of-the-art accuracy and up to 10x convergence speedup under the same privacy budget.